Flex

Flex generates a scanner from rules that map input text to C or C++ actions. A scanner normally supplies tokens to a parser, but it can also transform text or validate a regular-language format. The Flex manual defines generated interfaces and option-specific behavior.

Specification Structure #

A .l file has three sections separated by %%: definitions, rules, and user code. Definitions hold C declarations, named regular expressions, and start conditions. Rules pair patterns with actions. The final section supplies helper functions and, when needed, a driver.

%option noyywrap
%{
#include "parser.h"
%}

identifier  [[:alpha:]_][[:alnum:]_]*
%%
[[:space:]]+  ;
{identifier}   return IDENTIFIER;
"+"            return '+';
.              return INVALID;
%%

Flex selects the rule matching the longest input text. When several rules match the same length, it selects the rule written first. The default rule copies an unmatched character to standard output; use an explicit catch-all or -s/%option nodefault when unmatched input must be reported instead.

Regular Expressions and Scanner State #

Flex patterns describe regular languages. Named definitions reduce duplication, character classes express locale-aware groups, and quoted text matches literal strings. Escape or quote metacharacters when a language token needs them literally.

Start conditions partition rule sets. Declare inclusive conditions with %s and exclusive conditions with %x; qualify a rule with <STATE>.

%x COMMENT
%%
"/*"             BEGIN(COMMENT);
<COMMENT>"*/"    BEGIN(INITIAL);
<COMMENT>\n      ++line;
<COMMENT>.       ;
<COMMENT><<EOF>> return UNTERMINATED_COMMENT;

Inclusive conditions retain unqualified rules, while exclusive conditions run only rules explicitly qualified for that condition or for all states. Flex’s generated default rule remains active in every state unless %option nodefault or an explicit catch-all replaces it. Use BEGIN to change state and YY_START when actions need to inspect it. Handle <<EOF>> in states that can end prematurely. Model strings, comments, indentation, or lexical modes as explicit states rather than attempting to express context-sensitive structure in one pattern.

Generated APIs, Buffers, and Reentrancy #

By default, Flex generates yylex() and globals such as yytext, yyleng, and input-buffer state. %option prefix="project_" renames generated public symbols, which permits multiple scanners in one program. %option outfile="scanner.c" chooses the generated file; command-line options can provide the same configuration.

The scanner reads through buffers. yyrestart(FILE*) restarts input, and yy_scan_string or yy_scan_bytes scan memory supplied by the caller. Delete buffers created by the yy_scan_* functions with yy_delete_buffer; switch among managed buffers with yy_switch_to_buffer when nested inputs require it.

%option reentrant creates a scanner handle (yyscan_t) and changes generated APIs to accept it. Initialize and destroy the handle with yylex_init/yylex_destroy (or the _extra variant), then carry per-scanner state through yyextra rather than globals. Use this mode when parsers or threads must operate independently.

Parser Integration and Diagnostics #

For Bison integration, configure matching token declarations and include the generated parser header. %option bison-bridge passes semantic values through the scanner interface; %option bison-locations additionally passes locations. Update locations for every consumed character, including whitespace and comments, or diagnostics will drift from the source.

Keep tokenization errors distinct from parser syntax errors. A scanner should return or report an invalid token with its source span; a parser should describe the unexpected token in grammar context and recover only at deliberate synchronization points.