awk

awk transforms text as a stream of records. Each record is tested against ordered pattern-action rules, which makes the language suited to reports, validation, and small data transformations. POSIX awk is widely available; GNU awk (gawk) adds features that scripts should declare when they require them.

Records, Fields, and Counters #

RS separates input records and defaults to newline. FS separates fields; its default treats runs of spaces and tabs specially. $0 is the whole record, $1 through $NF are fields, and NF is the field count. NR counts records across all inputs, whereas FNR restarts for each input file.

awk -F ':' '$3 >= 1000 { print $1, $6 }' /etc/passwd
awk -v limit=10 'NF > limit { print FILENAME ":" FNR }' input

Set output separators explicitly when generating a structured result:

BEGIN { FS = OFS = "," }
NF >= 3 { total += $3 }
END { print "total", total }

GNU awk treats a multi-character RS as a regular expression. Portable awk requires a single-character record separator, so scripts handling NUL-delimited names should require gawk explicitly, for example gawk -v RS='\0'.

Write Pattern-Action Programs #

A rule has the form pattern { action }. A regular expression such as /error/ tests $0; an expression tests according to awk’s truth rules; and an omitted pattern matches every record. BEGIN runs before input and END runs after the final record.

BEGIN { FS = ":" }
$7 ~ /nologin$/ { next }
{ users[$1] = $6 }
END {
    for (name in users)
        print name, users[name]
}

Associative-array keys are strings, and for (key in array) has no portable iteration order. Sort output externally or use a documented GNU awk ordering feature when deterministic output is required.

Values and Regular Expressions #

awk values have string and numeric forms. Context selects a conversion: total += $3 needs a number, while concatenation needs a string. Validate input before relying on conversion because a nonnumeric string converts to zero in numeric context.

Regular expressions use POSIX extended regular-expression syntax. ~ and !~ test a value; match(), sub(), gsub(), and split() operate inside actions. sub() changes the first match and gsub() changes every match.

awk '
  /^[[:space:]]*#/ || NF == 0 { next }
  $2 ~ /^[0-9]+$/ { count[$1] += $2 }
  END { for (key in count) print key, count[key] }
' metrics.txt

Invoke awk Safely #

Single-quote the program so the shell does not expand $1, backslashes, or glob characters before awk receives them. Pass shell data with -v rather than interpolating it into program source:

awk -v prefix="$prefix" '$1 == prefix { print $0 }' input
awk -f report.awk -- --filename-starting-with-dash

The -- option ends awk option processing on implementations that support it; use it when input names can begin with -. Treat -v data as data, not awk source. For untrusted CSV, use a CSV-aware parser or GNU awk’s documented --csv mode because FS="," does not implement CSV quoting rules.