This module contains the functions and structures that are used by ScriptBasic to perform lexical analysis of the source code. The module was originally developed for ScriptBasic but was developed to be general enough to be used in other projects.