Shell scripting that scales

Robust shell scripts feel boring. That is the point. A script you can read at 3 a.m. during an incident, that fails loudly instead of corrupting state, beats a clever one-liner every time.

The first three lines that matter

Most broken shell scripts share one trait: they keep going after something has already failed. The default behaviour is to shrug off an error and march on, which is how a backup script "succeeds" while writing zero bytes.

Start every script the same way:

#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'

Here is what that buys:

• set -e aborts on the first command that returns non-zero, so a failure stops instead of cascading into damage.
• set -u treats unset variables as errors. Typo a variable name and you find out immediately, instead of rm -rf "$PREFX/data" expanding to rm -rf "/data".
• set -o pipefail makes a pipeline fail if any stage fails, not only the last one. Without it, curl ... | tar x reports success even when curl died.
• Setting IFS to newline and tab kills a whole class of word-splitting bugs around filenames with spaces.

This is not paranoia. It is the difference between a script that stops and a script that does damage at scale.

Quote everything

Word splitting and glob expansion are the two footguns that ruin shell scripts. An unquoted variable gets split on whitespace and expanded as a glob, which means a file named my report.txt arrives as two arguments, and a variable containing * matches your entire directory.

The rule is simple: if it is a variable or a command substitution, it goes in double quotes.

# wrong: breaks on spaces, expands globs
cp $src $dst

# right
cp "$src" "$dst"

Use "$@" (quoted) to forward arguments, never $* or a bare $@. Use an array for a list of things rather than a space-delimited string:

files=("$dir"/*.log)
gzip "${files[@]}"

If you find yourself building a command in a string variable and running it with eval, stop. You almost certainly want an array. eval is where quoting goes to die.

Functions, not flat scripts

A 300-line script that runs top to bottom is a script nobody can test. Break the work into functions, name them for what they do, and keep a main at the bottom. Declare locals with local so a variable inside one function cannot silently clobber another.

log() { printf '%s %s\n' "$(date +%H:%M:%S)" "$*" >&2; }
die() { log "FATAL: $*"; exit 1; }

upload_artifact() {
  local file="$1" bucket="$2"
  [[ -f "$file" ]] || die "missing artifact: $file"
  aws s3 cp "$file" "s3://$bucket/" || die "upload failed: $file"
}

main() {
  upload_artifact "$1" "${BUCKET:?BUCKET must be set}"
}

main "$@"

Three things there are worth stealing. log writes to stderr, so it never pollutes the data the script emits on stdout. ${BUCKET:?BUCKET must be set} fails loudly with your own message when the variable is missing, instead of quietly expanding to nothing. And calling main "$@" on the last line means the whole file parses before anything executes, so a script that gets saved mid-edit does not run half of itself.

Clean up with trap

The hard part of a script is not the happy path, it is what happens when it dies in the middle. Temp files, lock directories and half-written output all leak unless exit is handled explicitly.

trap runs a function when the script exits for any reason: success, error, or Ctrl-C. Combine it with mktemp and the cleanup actually fires.

workdir="$(mktemp -d)"
cleanup() { rm -rf "$workdir"; }
trap cleanup EXIT

# do the work in "$workdir". It is gone no matter how we leave

One trap on EXIT covers a normal exit and most signals. If you need to tell a crash from a clean run, trap ERR separately and log there. The point is that cleanup lives in one place, declared up front, instead of being copy-pasted in front of every exit.

The footguns that survive set -e

set -e is not a force field. It has well-known holes, and pretending otherwise is how you get bitten by the thing you thought you had covered.

• A failing command in an if condition does not trigger an exit. That is by design, but it means if grep -q foo file; then will not abort even when grep fails for a reason other than "no match".
• A command inside a $(...) substitution can swallow its failure depending on context. Assign first, check after.
• local x="$(cmd)" masks the exit code of cmd, because local itself succeeds. Split the declaration: local x; x="$(cmd)".
• Arithmetic like ((count++)) returns non-zero when the result is 0, which under set -e kills the script. Use count=$((count + 1)) instead.

None of that is an argument for abandoning set -e. It is an argument for knowing its edges. A tool whose limits you understand beats a tool you trust blindly.

Know when to stop

The scripts that survive are the ones treated like real code: reviewed, linted with shellcheck, and kept small enough to hold in your head. The bar for "just a shell script" is the bar for anything else that runs in production, because production does not know the difference.

When a script grows past a few hundred lines, or starts juggling real data structures, that is the signal to reach for a language with types and a test runner. Shell is glue. Respect what it is genuinely good at, refuse to push it past that, and run shellcheck in CI so the footguns above never reach the main branch.