Faux' blog

xlines: stdin round-robiner

xlines is a combination of xargs and split. It takes a bunch of lines, and sends them to a number of child processes. Each process sees only one of the lines.

e.g.

seq 16 | xlines -c 'cat > $(mktemp)'

...will give you 8 temporary files (on an 8-core machine) containing:

1
9

and:

2
10

etc.

Why would you care?

You have a bunch of INSERT statements coming off a stream, but your database will only use a single core if you run them in series:

zcat sql.gz | xlines -P 32 -- psql

Some speed-up.

zcat sql.gz | xlines -P 32 -c 'buffer | psql'

Zoom.

A specific tool to fix a specific job. I still don't think it makes up for the lack of limited parallelism in shell, however. Still thinking about that one...