Project — Concurrent Batch Processing

Build the mental model

Scaling Lesson 21's CLI content analyzer to hundreds of files exposes a real limitation of sequential processing (one file after another): it can't use a multi-core CPU's other cores, so time grows linearly — this project directly applies Lesson 18's `std::thread`/`mpsc` channel into a production-shaped parallel-processing pipeline. A key design decision is splitting the file list into chunks matched to thread count (say, the CPU core count, queryable via `std::thread::available_parallelism()`) and moving ownership of each chunk into its own thread — every thread is independent, needing no shared mutable state (locks) at all, extending Lesson 18's "message passing instead of shared state" philosophy to real scale. Each thread's partial result (file count, total word count) is sent to the main thread through a channel, and only the main thread aggregates (sums) them — a design that's free of race conditions by construction. Comparing the sequential and concurrent versions by timing (`std::time::Instant`) shows the speedup varies with CPU core count, file size, and thread-spawn overhead (too many threads and the overhead itself can outweigh the benefit) — a practical lesson in testing the "concurrency is always faster" assumption against real data.

Connect it to a real scenario

Split Lesson 21's file list into `chunks(file_count / num_threads)` and move ownership of each chunk into `thread::spawn(move || { ... })` — each thread sequentially reads and counts the files in its chunk, then sends a `(files_processed, total_words)` tuple result to the main thread with `tx.send(...)`. The main thread collects every result with `rx.iter().take(num_threads)` and aggregates the final total with `.fold((0, 0), |acc, r| (acc.0 + r.0, acc.1 + r.1))` — also keeping a vector of handles to `.join()` each thread. Compare timing between the sequential baseline (a plain loop, no threads) and the concurrent version using `Instant::now()`/`.elapsed()`, benchmarking across different file counts and thread counts.

• Split the file list into chunks matched to thread count, moving ownership — no shared locks needed
• Send each thread's partial result to the main thread via a channel; only it aggregates
• Concurrency isn't always faster — verify thread overhead against real timing data

Try the working example

use std::sync::mpsc;
use std::thread;
use std::time::Instant;

fn word_count_for(files: &[&str]) -> (usize, usize) {
    let total: usize = files.iter().map(|f| f.split_whitespace().count()).sum();
    (files.len(), total)
}

fn main() {
    let files = vec!["rust ownership basics", "borrow checker rules", "traits and generics", "async runtime tokio"];
    let (tx, rx) = mpsc::channel();
    let start = Instant::now();

    for chunk in files.chunks(2) {
        let chunk: Vec<&str> = chunk.to_vec();
        let tx = tx.clone();
        thread::spawn(move || {
            tx.send(word_count_for(&chunk)).unwrap();
        });
    }
    drop(tx);

    let (files_done, words_done) = rx.iter().fold((0, 0), |acc, r| (acc.0 + r.0, acc.1 + r.1));
    println!("{files_done} files, {words_done} words in {:?}", start.elapsed());
}

You print "4 files, 12 words in <duration>", showing multiple threads aggregated results back through the channel to the main thread.

The Rust Programming Language — Using Message Passing to Transfer Data — Rust