Concurrent Shell Command Queues

(emmer.dev)

11 points | by speckx 2 hours ago

3 comments

js2 1 hour ago
> How to run shell commands in parallel while limiting their concurrency.
Depending upon the task, generate a Makefile and run it with:
```
  make -j <concurrency>
```
> My original use case had to do with making a large number of web requests
This is in fact the use case that caused me to think about using a Makefile for concurrency probably 20 years ago. A rule for downloading each URL to a file. Then a simple `make -j <jobs>` with whatever concurrency I desired and bonus: automatically skipping already downloaded files. e.g.:
```
  BASE  := https://example.com/files
  FILES := a.txt b.txt c.txt d.txt e.txt

  .DELETE_ON_ERROR:

  all: $(FILES)

  $(FILES):
        curl -sS --fail --max-time 30 -o $@ $(BASE)/$@
```
Or, to emulate one of the examples in the blog post:
```
  SHELL := bash
  .SHELLFLAGS := -o pipefail -c

  N    := 1 2 3 4 5
  REQS := $(N:%=req-%)

  .PHONY: all $(REQS)
  all: $(REQS)

  $(REQS): req-%:
        @curl -sS --fail-with-body --max-time 10 -X POST -H 'Content-Type: application/json' \
        -d '{"request": $*}' https://postman-echo.com/post | jq -c .json
```
Run with:
```
  $ make -O -j 3
```
> You do not need to know all the commands that you want executed up-front, you can submit commands to it over time.
Ah, yeah, that does put Make in the same bucket as `xargs -P` / GNU `parallel` and requires something more clever.
tacostakohashi 1 hour ago
I guess that works.
I've always been fond of using make for things to get free parallelism / ability to interrupt / restart / incremental updates.
kernc 1 hour ago
> language with poor concurrency controls
Few other languages have as many builtins/keywords for concurrency control: &, wait, jobs ...
[-]
- Joker_vD 52 minutes ago
  Yeah, and all those concurrency controls of the shell suck. Here's a snippet to run N commands in parallel, wait for all of them, while supporting interrupting them with simple Ctrl+C:
```
    (
        trap 'kill 0' SIGINT;
        for i in {1..$N}
        do
            whatever_command $i &
        done
        wait
    )
```
  The parentheses are absolutely crucial there for some reason, the { } won't give you pass-through of SIGINT. No wonder people keep re-inventing PID files, manually tracking $! and all kinds of other crazy bullshit that work 90% of the time, but sometimes fail to properly terminate the runaway children.
- nagaiaida 10 minutes ago
  perhaps that's more an indictment of most programming languages