parallel-processing-patterns — скилл для Claude Code
Скилл parallel-processing-patterns для Claude Code подсказывает, как запускать Bash-задачи параллельно через GNU Parallel, xargs -P и фоновые процессы. Описывает параллельную обработку файлов и управление числом одновременных задач.
MIT 181 установок ★ 66 обновлён 2026-06-18
npx skills add JosiahSiegel/claude-plugin-marketplace --skill parallel-processing-patterns -a claude-codeЧто делает parallel-processing-patterns
Скилл разбирает способы распараллелить команды в Bash: от xargs -P и GNU Parallel до фоновых процессов с & и wait. В нём есть варианты пула работников с ограничением числа одновременных задач, сбора кодов завершения и очередей на FIFO.
Также рассматриваются обработка потоков и входных файлов, журналирование результатов, повтор неудачных задач и запуск команд на удалённых серверах. Для простых сценариев подойдёт xargs -P; GNU Parallel предлагает больше настроек, но его нужно установить отдельно. Если несколько процессов пишут в один файл, важно предусмотреть защиту от гонок.
Какие задачи решает
- распараллелить обработку множества файлов
- ограничить число одновременно работающих команд
- собрать коды завершения дочерних процессов
- организовать очередь задач через FIFO
- запустить задания на нескольких серверах
Примеры запросов агенту
- Покажи, как обработать все
.txt-файлы параллельно с помощьюxargs -P. - Собери пул из 4 фоновых работников и покажи, как получить коды завершения.
- Подскажи, как распределить обработку
*.datмежду серверами изservers.txt.
Как установить parallel-processing-patterns
1Через Skills CLI (Vercel)
npx skills add JosiahSiegel/claude-plugin-marketplace --skill parallel-processing-patterns -a claude-code2Вручную: положить файлы
~/.claude/skills/parallel-processing-patterns/SKILL.mdПапка скилла целиком; для одного проекта — .claude/skills/parallel-processing-patterns/
1Через Skills CLI (Vercel)
npx skills add JosiahSiegel/claude-plugin-marketplace --skill parallel-processing-patterns -a codex2Вручную: положить файлы
.agents/skills/parallel-processing-patterns/SKILL.mdДля всех проектов — ~/.agents/skills/
1Через Skills CLI (Vercel)
npx skills add JosiahSiegel/claude-plugin-marketplace --skill parallel-processing-patterns -a cursor2Вручную: положить файлы
.cursor/skills/parallel-processing-patterns/SKILL.mdCursor читает и .agents/skills/, и .claude/skills/
1GitHub CLI
gh skill install JosiahSiegel/claude-plugin-marketplace plugins/bash-master/skills/parallel-processing-patterns2Вручную: положить файлы
.github/skills/parallel-processing-patterns/SKILL.md1Через Skills CLI (Vercel)
npx skills add JosiahSiegel/claude-plugin-marketplace --skill parallel-processing-patterns -a gemini-cli2Командой платформы
gemini skills install https://github.com/JosiahSiegel/claude-plugin-marketplace --path plugins/bash-master/skills/parallel-processing-patternsВручную: положить файлы
.devin/skills/parallel-processing-patterns/SKILL.mdСтарый путь .windsurf/skills/ тоже читается
Через Skills CLI (Vercel)
npx skills add JosiahSiegel/claude-plugin-marketplace --skill parallel-processing-patterns -a opencodeClaude.ai
Упакуйте папку в zip и загрузите в Settings → Capabilities → Skills
По шагам, с командами для каждой платформы: Как установить скилл в Claude Code: пять способов
Вопросы о parallel-processing-patterns
Как установить parallel-processing-patterns в Claude Code?
Выполните npx skills add JosiahSiegel/claude-plugin-marketplace --skill parallel-processing-patterns -a claude-code. Файл скилла появится по пути ~/.claude/skills/parallel-processing-patterns/SKILL.md.
Что делает parallel-processing-patterns?
Он объясняет, как запускать Bash-команды параллельно, ограничивать число работников и собирать результаты дочерних процессов. Рассматриваются GNU Parallel, xargs -P, фоновые задачи и FIFO-очереди.
Нужно ли устанавливать GNU Parallel?
Для примеров с GNU Parallel он нужен, но скилл также показывает параллельный запуск через xargs -P и Bash. GNU Parallel можно установить через apt-get, Homebrew или из исходников.
Исходный текст
SKILL.md · MIT
---
name: parallel-processing-patterns
description: |
Parallel and concurrent processing patterns in bash including GNU Parallel, xargs -P, job pools, and async patterns (2025).
PROACTIVELY activate for: (1) running tasks in parallel from bash, (2) xargs -P for parallel command execution, (3) GNU parallel for complex distribution, (4) background jobs with & and wait, (5) implementing a concurrency cap (job pool of N workers), (6) collecting exit codes from parallel children, (7) preventing race conditions in shared output files, (8) parallel file processing across many cores.
Provides: xargs -P recipes, GNU parallel quick reference, job-pool implementations in pure bash, exit-code aggregation patterns, and FIFO-based work queues.
---
## CRITICAL GUIDELINES
### Windows File Path Requirements
**MANDATORY: Always Use Backslashes on Windows for File Paths**
When using Edit or Write tools on Windows, you MUST use backslashes (`\`) in file paths, NOT forward slashes (`/`).
---
# Parallel Processing Patterns in Bash (2025)
## Overview
Comprehensive guide to parallel and concurrent execution in bash, covering GNU Parallel, xargs parallelization, job control, worker pools, and modern async patterns for maximum performance.
## GNU Parallel (Recommended)
### Installation
```bash
# Debian/Ubuntu
sudo apt-get install parallel
# macOS
brew install parallel
# From source
wget https://ftp.gnu.org/gnu/parallel/parallel-latest.tar.bz2
tar -xjf parallel-latest.tar.bz2
cd parallel-*
./configure && make && sudo make install
```
### Basic Usage
```bash
#!/usr/bin/env bash
set -euo pipefail
# Process multiple files in parallel
parallel gzip ::: *.txt
# Equivalent to:
# for f in *.txt; do gzip "$f"; done
# But runs in parallel!
# Using find with parallel
find . -name "*.jpg" | parallel convert {} -resize 50% resized/{}
# Specify number of jobs
parallel -j 8 process_file ::: *.dat
# From stdin
cat urls.txt | parallel -j 10 wget -q
# Multiple inputs
parallel echo ::: A B C ::: 1 2 3
# Output: A 1, A 2, A 3, B 1, B 2, B 3, C 1, C 2, C 3
# Paired inputs with :::+
parallel echo ::: A B C :::+ 1 2 3
# Output: A 1, B 2, C 3
```
### Input Handling
```bash
#!/usr/bin/env bash
set -euo pipefail
# Input from file
parallel -a input.txt process_line
# Multiple input files
parallel -a file1.txt -a file2.txt 'echo {1} {2}'
# Column-based input
cat data.tsv | parallel --colsep '\t' 'echo Name: {1}, Value: {2}'
# Named columns
cat data.csv | parallel --header : --colsep ',' 'echo {name}: {value}'
# Null-delimited for safety with special characters
find . -name "*.txt" -print0 | parallel -0 wc -l
# Line-based chunking
cat huge_file.txt | parallel --pipe -N1000 'wc -l'
```
### Replacement Strings
```bash
#!/usr/bin/env bash
set -euo pipefail
# {} - Full input
parallel echo 'Processing: {}' ::: file1.txt file2.txt
# {.} - Remove extension
parallel echo '{.}' ::: file.txt file.csv
# Output: file, file
# {/} - Basename
parallel echo '{/}' ::: /path/to/file.txt
# Output: file.txt
# {//} - Directory path
parallel echo '{//}' ::: /path/to/file.txt
# Output: /path/to
# {/.} - Basename without extension
parallel echo '{/.}' ::: /path/to/file.txt
# Output: file
# {#} - Job number (1-based)
parallel echo 'Job {#}: {}' ::: A B C
# {%} - Slot number (recycled job slot)
parallel -j 2 'echo "Slot {%}: {}"' ::: A B C D E
# Combined
parallel 'convert {} -resize 50% {//}/thumb_{/.}.jpg' ::: *.png
```
### Progress and Logging
```bash
#!/usr/bin/env bash
set -euo pipefail
# Show progress bar
parallel --bar process_item ::: {1..100}
# Progress with ETA
parallel --progress process_item ::: {1..100}
# Verbose output
parallel --verbose gzip ::: *.txt
# Log to file
parallel --joblog jobs.log gzip ::: *.txt
# Resume from where it left off (skip completed jobs)
parallel --joblog jobs.log --resume gzip ::: *.txt
# Results logging
parallel --results results_dir 'echo {1} + {2}' ::: 1 2 3 ::: 4 5 6
# Creates: results_dir/1/4/stdout, results_dir/1/4/stderr, etc.
```
### Resource Management
```bash
#!/usr/bin/env bash
set -euo pipefail
# CPU-based parallelism (number of cores)
parallel -j "$(nproc)" process_item ::: {1..1000}
# Leave some cores free
parallel -j '-2' process_item ::: {1..1000} # nproc - 2
# Percentage of cores
parallel -j '50%' process_item ::: {1..1000}
# Load-based throttling
parallel --load 80% process_item ::: {1..1000}
# Memory-based throttling
parallel --memfree 2G process_item ::: {1..1000}
# Rate limiting (max jobs per second)
parallel -j 4 --delay 0.5 wget ::: url1 url2 url3 url4
# Timeout per job
parallel --timeout 60 long_process ::: {1..100}
# Retry failed jobs
parallel --retries 3 flaky_process ::: {1..100}
```
### Distributed Execution
```bash
#!/usr/bin/env bash
set -euo pipefail
# Run on multiple servers
parallel --sshloginfile servers.txt process_item ::: {1..1000}
# servers.txt format:
# 4/server1.example.com (4 jobs on server1)
# 8/server2.example.com (8 jobs on server2)
# : (local machine)
# Transfer files before execution
parallel --sshloginfile servers.txt --transferfile {} process {} ::: *.dat
# Return results
parallel --sshloginfile servers.txt --return {.}.result process {} ::: *.dat
# Cleanup after transfer
parallel --sshloginfile servers.txt --transfer --return {.}.out --cleanup \
'process {} > {.}.out' ::: *.dat
# Environment variables
export MY_VAR="value"
parallel --env MY_VAR --sshloginfile servers.txt 'echo $MY_VAR' ::: A B C
```
### Complex Pipelines
```bash
#!/usr/bin/env bash
set -euo pipefail
# Pipe mode - distribute stdin across workers
cat huge_file.txt | parallel --pipe -N1000 'sort | uniq -c'
# Block size for pipe mode
cat data.bin | parallel --pipe --block 10M 'process_chunk'
# Keep order of output
parallel --keep-order 'sleep $((RANDOM % 3)); echo {}' ::: A B C D E
# Group output (don't mix output from different jobs)
parallel --group 'for i in 1 2 3; do echo "Job {}: line $i"; done' ::: A B C
# Tag output with job identifier
parallel --tag 'echo "output from {}"' ::: A B C
# Sequence output (output as they complete, but grouped)
parallel --ungroup 'echo "Starting {}"; sleep 1; echo "Done {}"' ::: A B C
```
## xargs Parallelization
### Basic Parallel xargs
```bash
#!/usr/bin/env bash
set -euo pipefail
# -P for parallel jobs
find . -name "*.txt" | xargs -P 4 -I {} gzip {}
# -n for items per command
echo {1..100} | xargs -n 10 -P 4 echo "Batch:"
# Null-delimited for safety
find . -name "*.txt" -print0 | xargs -0 -P 4 -I {} process {}
# Multiple arguments per process
cat urls.txt | xargs -P 10 -n 5 wget -q
# Limit max total arguments
echo {1..1000} | xargs -P 4 --max-args=50 echo
```
### xargs with Complex Commands
```bash
#!/usr/bin/env bash
set -euo pipefail
# Use sh -c for complex commands
find . -name "*.jpg" -print0 | \
xargs -0 -P 4 -I {} sh -c 'convert "$1" -resize 50% "thumb_$(basename "$1")"' _ {}
# Multiple placeholders
paste file1.txt file2.txt | \
xargs -P 4 -n 2 sh -c 'diff "$1" "$2" > "diff_$(basename "$1" .txt).patch"' _
# Process in batches
find . -name "*.log" -print0 | \
xargs -0 -P 4 -n 100 tar -czvf logs_batch.tar.gz
# With failure handling
find . -name "*.dat" -print0 | \
xargs -0 -P 4 -I {} sh -c 'process "$1" || echo "Failed: $1" >> failures.log' _ {}
```
## Job Control Patterns
### Background Job Management
```bash
#!/usr/bin/env bash
set -euo pipefail
# Track background jobs
declare -a PIDS=()
# Start jobs
for item in {1..10}; do
process_item "$item" &
PIDS+=($!)
done
# Wait for all
for pid in "${PIDS[@]}"; do
wait "$pid"
done
echo "All jobs complete"
# Or wait for any to complete
wait -n # Bash 4.3+
echo "At least one job complete"
```
### Job Pool with Semaphore
```bash
#!/usr/bin/env bash
set -euo pipefail
# Maximum concurrent jobs
MAX_JOBS=4
# Simple semaphore using a counter
job_count=0
run_with_limit() {
local cmd=("$@")
# Wait if at limit
while ((job_count >= MAX_JOBS)); do
wait -n 2>/dev/null || true
((job_count--))
done
# Start new job
"${cmd[@]}" &
((job_count++))
}
# Usage
for item in {1..20}; do
run_with_limit process_item "$item"
done
# Wait for remaining
wait
```
### FIFO-Based Job Pool
```bash
#!/usr/bin/env bash
set -euo pipefail
MAX_JOBS=4
JOB_FIFO="/tmp/job_pool_$$"
# Create job slots
mkfifo "$JOB_FIFO"
trap 'rm -f "$JOB_FIFO"' EXIT
# Initialize slots
exec 3<>"$JOB_FIFO"
for ((i=0; i<MAX_JOBS; i++)); do
echo >&3
done
# Run with slot
run_with_slot() {
local cmd=("$@")
read -u 3 # Acquire slot (blocks if none available)
{
"${cmd[@]}"
echo >&3 # Release slot
} &
}
# Usage
for item in {1..20}; do
run_with_slot process_item "$item"
done
wait
exec 3>&-
```
### Worker Pool Pattern
```bash
#!/usr/bin/env bash
set -euo pipefail
WORK_QUEUE="/tmp/work_queue_$$"
RESULT_QUEUE="/tmp/result_queue_$$"
NUM_WORKERS=4
mkfifo "$WORK_QUEUE" "$RESULT_QUEUE"
trap 'rm -f "$WORK_QUEUE" "$RESULT_QUEUE"' EXIT
# Worker function
worker() {
local id="$1"
while read -r task; do
[[ "$task" == "STOP" ]] && break
# Process task
local result
result=$(process_task "$task" 2>&1)
echo "RESULT:$id:$task:$result"
done
}
# Start workers
for ((i=0; i<NUM_WORKERS; i++)); do
worker "$i" < "$WORK_QUEUE" > "$RESULT_QUEUE" &
done
# Result collector (background)
collect_results() {
while read -r line; do
[[ "$line" == "DONE" ]] && break
echo "$line" >> results.txt
done < "$RESULT_QUEUE"
} &
COLLECTOR_PID=$!
# Producer - send work
{
for task in "${TASKS[@]}"; do
echo "$task"
done
# Stop signals for workers
for ((i=0; i<NUM_WORKERS; i++)); do
echo "STOP"
done
} > "$WORK_QUEUE"
# Signal end of results
wait # Wait for workers
echo "DONE" > "$RESULT_QUEUE"
wait "$COLLECTOR_PID"
```
## Modern Async Patterns & Performance Optimization
Longer examples for PID collection, `wait -n`, bounded concurrency, fail-fast job pools, progress reporting, signal-safe cleanup, batch sizing, CPU-count tuning, memory-aware concurrency, and benchmarking live in `references/modern-async-patterns.md` and `references/performance-optimization.md`. Load those references for complex orchestration or tuning beyond the core GNU Parallel / xargs / job-control patterns.
## Error Handling
### Graceful Failure Handling
```bash
#!/usr/bin/env bash
set -euo pipefail
# Track failures
declare -A FAILURES
parallel_with_retry() {
local max_retries=3
local items=("$@")
for item in "${items[@]}"; do
local retries=0
local success=false
while ((retries < max_retries)) && ! $success; do
if process_item "$item"; then
success=true
else
((retries++))
echo "Retry $retries for $item" >&2
sleep $((retries * 2)) # Exponential backoff
fi
done
if ! $success; then
FAILURES["$item"]="Failed after $max_retries retries"
fi
done &
wait
}
# Report failures
report_failures() {
if ((${#FAILURES[@]} > 0)); then
echo "Failures:" >&2
for item in "${!FAILURES[@]}"; do
echo " $item: ${FAILURES[$item]}" >&2
done
return 1
fi
}
```
### Cancellation Support
```bash
#!/usr/bin/env bash
set -euo pipefail
# Global cancellation flag
CANCELLED=false
declare -a WORKER_PIDS=()
cancel_all() {
CANCELLED=true
for pid in "${WORKER_PIDS[@]}"; do
kill "$pid" 2>/dev/null || true
done
}
trap cancel_all SIGINT SIGTERM
cancellable_worker() {
local id="$1"
while ! $CANCELLED; do
# Check for work
if work=$(get_next_work); then
process_work "$work"
else
sleep 0.1
fi
done
}
# Start workers
for ((i=0; i<NUM_WORKERS; i++)); do
cancellable_worker "$i" &
WORKER_PIDS+=($!)
done
# Wait with interrupt support
wait || true
```
## Resources
- [GNU Parallel Tutorial](https://www.gnu.org/software/parallel/parallel_tutorial.html)
- [GNU Parallel Manual](https://www.gnu.org/software/parallel/man.html)
- [Bash Job Control](https://www.gnu.org/software/bash/manual/html_node/Job-Control.html)
- [Advanced Bash-Scripting Guide - Process Substitution](https://tldp.org/LDP/abs/html/process-sub.html)
---
**Master parallel processing for efficient multi-core utilization and faster script execution.**
© JosiahSiegel · MIT · JosiahSiegel/claude-plugin-marketplace