AWK () is a scripting language designed for text processing and typically used as a data extraction and reporting tool. Like sed and grep, it is a filter, and it is a standard feature of most Unix-like operating systems. The shell command that runs the AWK processor is named .
The AWK language is a data-driven scripting language consisting of a set of actions to be taken against streams of textual data – either run directly on files or used as part of a pipeline – for purposes of extracting or transforming text, such as producing formatted reports. The language extensively uses the string datatype, associative arrays (that is, arrays indexed by key strings), and regular expressions. While AWK has a limited intended application domain and was especially designed to support one-liner programs, the language is Turing-complete, and even the early Bell Labs users of AWK often wrote well-structured large AWK programs.
AWK was created at Bell Labs in the 1970s, and its name is derived from the surnames of its authors: Alfred Aho (author of egrep), Peter Weinberger (who worked on tiny relational databases), and Brian Kernighan. The acronym is pronounced the same as the name of the bird species auk, which is illustrated on the cover of The AWK Programming Language.
History According to Brian Kernighan, one of the goals of AWK was to have a tool that would easily manipulate both numbers and strings. AWK was also inspired by Marc Rochkind's programming language that was used to search for patterns in input data, and was implemented using yacc.
As one of the early tools to appear in Version 7 Unix, AWK added computational features to a Unix pipeline besides the Bourne shell, the only scripting language available in a standard Unix environment. It is one of the mandatory utilities of the Single UNIX Specification, and is required by the Linux Standard Base specification.
In 1983, AWK was one of several UNIX tools available for Charles River Data Systems' UNOS operating system under Bell Laboratories license.
AWK was significantly revised and expanded in 1985–88, resulting in the GNU AWK implementation written by Paul Rubin, Jay Fenlason, and Richard Stallman, released in 1988. GNU AWK may be the most widely deployed version because it is included with GNU-based Linux packages. GNU AWK has been maintained solely by Arnold Robbins since 1994. Brian Kernighan's nawk (New AWK) source was first released in 1993 unpublicized, and publicly since the late 1990s; many BSD systems use it to avoid the GPL license.
AWK was preceded by sed (1974). Both were designed for text processing. They share the line-oriented, data-driven paradigm, and are particularly suited to writing one-liner programs, due to the implicit main loop and current line variables. The power and terseness of early AWK programs – notably the powerful regular expression handling and conciseness due to implicit variables, which facilitate one-liners – together with the limitations of AWK at the time, were important inspirations for the Perl language (1987). In the 1990s, Perl became very popular, competing with AWK in the niche of Unix text-processing languages.
Structure of AWK programs
An AWK program is a series of pattern action pairs, written as:
condition { action } condition { action } ...
where condition is typically an expression and action is a series of commands. The input is split into records, where by default records are separated by newline characters so that the input is split into lines. The program tests each record against each of the conditions in turn, and executes the action for each expression that is true. Either the condition or the action may be omitted. The condition defaults to matching every record. The default action is to print the record. This is the same pattern-action structure as sed.
In addition to a simple AWK expression, such as foo == 1 or /^foo/, the condition can be BEGIN or END causing the action to be executed before or after all records have been read, or pattern1, pattern2 which matches the range of records starting with a record that matches pattern1 up to and including the record that matches pattern2 before again trying to match against pattern1 on subsequent lines.
In addition to normal arithmetic and logical operators, AWK expressions include the tilde operator, ~, which matches a regular expression against a string. As handy syntactic sugar, /regexp/ without using the tilde operator matches against the current record; this syntax derives from sed, which in turn inherited it from the ed editor, where / is used for searching. This syntax of using slashes as delimiters for regular expressions was subsequently adopted by Perl and ECMAScript, and is now common. The tilde operator was also adopted by Perl.
Commands
AWK commands are the statements that are substituted for action in the examples above. AWK commands can include function calls, variable assignments, calculations, or any combination thereof. AWK contains built-in support for many functions; many more are provided by the various flavors of AWK. Also, some flavors support the inclusion of dynamically linked libraries, which can also provide more functions.
The print command
The print command is used to output text. The output text is always terminated with a predefined string called the output record separator (ORS) whose default value is a newline. The simplest form of this command is:
print This displays the contents of the current record. In AWK, records are broken down into fields, and these can be displayed separately: print $1 Displays the first field of the current record print $1, $3 Displays the first and third fields of the current record, separated by a predefined string called the output field separator (OFS) whose default value is a single space character
Although these fields ($X) may bear resemblance to variables (the $ symbol indicates variables in the usual Unix shells and in Perl), they actually refer to the fields of the current record. A special case, $0, refers to the entire record. In fact, the commands "print" and "print $0" are identical in functionality.
The print command can also display the results of calculations and/or function calls: /regex_pattern/ { # Actions to perform in the event the record (line) matches the above regex_pattern print 3+2 print foobar(3) print foobar(variable) print sin(3-2) }
Output may be sent to a file: /regex_pattern/ { # Actions to perform in the event the record (line) matches the above regex_pattern print "expression" > "file name" }
or through a pipe: /regex_pattern/ { # Actions to perform in the event the record (line) matches the above regex_pattern print "expression" | "command" }
Built-in variables
AWK's built-in variables include the field variables: , and so on ( represents the entire record). They hold the text or values in the individual text-fields in a record.
Other variables include: NR: Number of Records. Keeps a current count of the number of input records read so far from all data files. It starts at zero, but is never automatically reset to zero. FNR: File Number of Records. Keeps a current count of the number of input records read so far in the current file. This variable is automatically reset to zero each time a new file is started. NF: Number of Fields. Contains the number of fields in the current input record. The last field in the input record can be designated by , the 2nd-to-last field by , the 3rd-to-last field by , etc. FILENAME: Contains the name of the current input-file. FS: Field Separator. Contains the "field separator" used to divide fields in the input record. The default, "white space", allows any sequence of space and tab characters. FS can be reassigned with another character or character sequence to change the field separator. RS: Record Separator. Stores the current "record separator" character. Since, by default, an input line is the input record, the default record separator character is a "newline". OFS: Output Field Separator. Stores the "output field separator", which separates the fields when awk prints them. The default is a "space" character. ORS: Output Record Separator. Stores the "output record separator", which separates the output records when awk prints them. The default is a "newline" character. OFMT: Output Format. Stores the format for numeric output. The default format is .
Variables and syntax
With the exception of language keywords, variable names can use any of these characters provided it doesn't begin with a numeric digit :
The operators + - * / ^ % represent addition, subtraction, multiplication, division, exponentiation, and remainder, respectively. For division and remainder, the same operators apply to both integer and floating point divisions. To obtain an integer quotient from integer operands, any remainder must be pre-subtracted out of the dividend, as shown below : $ awk ' > BEGIN { > > OFMT = "%.14g" > > print 331 % -79 > print -3.31e-11 % 7.9e-23 > > print 331 / 79 > print (331 - 331 % 79) / 79 > }' 15 -1.1999020635259e-23 4.1898734177215 4 The int( dividend / divisor ) approach may lead to edge cases that round incorrectly.
For string concatenation, simply place two variables (or string constants) next to each other. It is optional to use a space in between if string constants are involved, but two variable names placed adjacent to each other require a space in between. Unlike Unix shells or other languages like Perl, the sigil-less single underscore variable name (_) is neither reserved nor special, and can be freely utilized by the user. Since its default value is the special AWK NULL value that is simultaneously numeric zero and the empty string, it could even mimic Perl's $_ as an alias for $0 :jot 30 | perl -nle ' print $_ << $_ ' | awk '{ print 2^$_ * $_ }'
Double quotes ("..."), and only double quotes, delimit string constants. Statements and / or expressions need not end with semicolons. In fact, the only place where the semi-colon ; is absolutely unavoidable is the 3-argument variant of the for () loop that largely resembles its counterpart in C/C++ . All non-array arguments to functions are passed "by-value", making them freely mutable by the function while guaranteeing being side-effect free to the function's upstream caller.
Unlike Python, all augmented assignments in awk such as += -= *= /= ^= %= are expressions. This allows for effective unlimited-length chaining. One of its biggest benefits is demonstrated by the algorithm for calculating Fibonacci numbers having complexity of because of the reason that it keeps adding back on itself: For the first 77 or so Fibonacci numbers, this approach is highly effective, since it bypasses having to examine bits, swap variables, or incur recursion overhead.
Finally, comments can be added to programs by using # as the first character on a line, or behind a command or sequence of commands.
User-defined functions
In a format similar to C, function definitions consist of the keyword function, the function name, argument names and the function body. Here is an example of a function. function add_three(number) { return number + 3 }
This statement can be invoked as follows: (pattern) { print add_three(36) # Outputs '''39''' }
Functions can have variables that are in the local scope. The names of these are added to the end of the argument list, though values for these should be omitted when calling the function. It is convention to add some whitespace in the argument list before the local variables, to indicate where the parameters end and the local variables begin.
Examples
Hello, World!
Here is the customary "Hello, World!" program written in AWK: BEGIN { print "Hello, world!" exit }
Print lines longer than 80 characters
Print all lines longer than 80 characters. The default action is to print the current line. length($0) > 80
Count words