Practical Uses for the PHP Tokenizer

Saturday, August 23. 2008

Practical Uses for the PHP Tokenizer

When PHP has to process a request, the engine goes through several passes of parsing until the code is expressed as a set of instructions that the interpreter can execute. The first such step is “lexical scanning”, which splits the code into smaller strings called “tokens”. The token is the smallest meaningful unit of your source code, and it can represent a reserved word (for, while, class, if, etc.), operator (+, -, *, /, && etc.), value literals (integers, floats, strings etc.) and other special symbols.

The same lexical scanner which PHP uses, is also available to userspace PHP developers via the function token_get_all().