Parallel Execution and xargs

Build the mental model

Appending & to a command launches it in the background as a separate child process and hands control straight back to the script, letting it launch more commands before the earlier ones finish — process 1 &, process 2 &, process 3 & starts all three essentially at once, not one after another. wait, called with no arguments, blocks until every background job the script started has finished; you can also wait on a single job with wait "$!" (the PID of the most recently backgrounded process) if you only need to synchronize on one. This suits a handful of genuinely independent tasks, but doesn't scale to hundreds or thousands of inputs — you'd need that many & launches, with no easy way to cap how many run at once. That's what xargs -P N solves: it reads a stream of inputs (one per line, by default) and runs up to N copies of a command in parallel, launching a new invocation the moment a worker frees up, until every input is processed. The speed gain is real — work that would take N times as long serially can finish in roughly 1/N the time — but so is the cost: with several workers writing to the terminal at once, output interleaves unpredictably, and you lose any guarantee input 1's job finishes, or even starts, before input 2's does. Anything logged from parallel workers should be tagged with context — a job ID, the input's own name — to make sense of afterward, and N is worth tuning deliberately: too high relative to your CPU cores or a remote API's rate limits, and you're causing contention rather than speeding anything up.

Connect it to a real scenario

Say you need to resize a few thousand image files before uploading them to a CDN. Resizing them one at a time in a for loop is safe but slow — with each conversion taking, say, half a second, a few thousand files serially adds up to tens of minutes of pure waiting. Piping the file list through find . -name '*.jpg' | xargs -P 4 -I{} convert {} -resize 50% resized/{} changes that: xargs reads each filename from find's output and substitutes it for {} in the convert command, but instead of waiting for each conversion to finish before starting the next, it keeps up to 4 workers running at once, launching a new one as soon as one finishes. On a machine with 4 or more CPU cores, this can cut the wall-clock time roughly to a quarter of the serial version, since image resizing is CPU-bound work that genuinely benefits from running on multiple cores simultaneously. The trade-off you take on is that the four workers' "processing {}" log lines will land on the terminal in whatever order their conversions happen to finish, not the order find listed the files — which is exactly why relying on log order to confirm which files were processed, rather than checking the resized/ directory's actual contents afterward, would be a mistake here.

• & backgrounds a command; wait blocks until background jobs finish
• xargs -P N runs a command against many inputs using up to N parallel workers
• Parallel output interleaves unpredictably — tag log lines with context instead of relying on order

Try the working example

#!/bin/bash
set -euo pipefail

process() {
  local id=$1
  echo "worker $id: done"
}

echo "--- running jobs in the background with & and waiting with wait ---"
process 1 &
process 2 &
process 3 &
wait
echo "all background jobs finished"

echo "--- xargs -P runs a command against many inputs in parallel ---"
printf "a\nb\nc\n" | xargs -I{} -P 2 echo "processing {}" | sort

Running the script above prints:

--- running jobs in the background with & and waiting with wait ---
worker 1: done
worker 2: done
worker 3: done
all background jobs finished
--- xargs -P runs a command against many inputs in parallel ---
processing a
processing b
processing c

xargs(1) — Linux manual page — Bash / Shell Scripting