TokenInit()

Initializes a token environment

Syntax

      TokenInit( <[@]cString>], [<cTokenizer>], [<nSkipWidth>],
                 [<@cTokenEnvironment>] ) -> lState

Arguments

<[@]cString> is the processed string

<cTokenizer> is a list of characters separating the tokens in <cString> Default: chr(0) + chr(9) + chr(10) + chr(13) + chr(26) +  chr(32) + chr(32) + chr(138) + chr(141) +  “, .;:!\?/\\<>()#&%+-*”

<nSkipWidth> specifies the maximum number of successive tokenizing characters that are combined as ONE token stop, e.g. specifying 1 can yield to empty token Default: 0, any number of successive tokenizing characters are combined as ONE token stop

<@cTokenEnvironment> is a token environment stored in a binary encoded string

Returns

<lState> success of the initialization

Description

The TokenInit() function initializes a token environment. A token environment is the information about how a string is to be tokenized. This information is created in the process of tokenization of the string <cString> – equal to the one used in the TOKEN() function with the help of the <cTokenizer> and <nSkipWidth> parameters.

This token environment can be very useful when large strings have to be tokenized since the tokenization has to take place only once whereas the TOKEN() function must always start the tokenizing process from scratch.

Unlike CT3, this function provides two mechanisms of storing the resulting token environment. If a variable is passed by reference as 4th parameter, the token environment is stored in this variable, otherwise the global token environment is used. Do not modify the token environment string directly !

Additionally, a counter is stored in the token environment, so that the tokens can successivly be obtained. This counter is first set to 1. When the TokenInit() function is called without a string a tokenize, the counter of either the global environment or the environment given by reference in the 4th parameter is rewind to 1.

Additionally, unlike CT3, TokenInit() does not need the string <cString> to be passed by reference, since one must provide the string in calls to TOKENNEXT() again.

Examples

  TokenInit( cString )             // tokenize the string <cString> with 
                                   // default rules and store the token 
                                   // environment globally and eventually 
                                   // delete an old global token environment
  TokenInit( @cString )            // no difference in result, but eventually 
                                   // faster, since the string must not be 
  TokenInit()                      // copied rewind counter of global TE to 1
  TokenInit( "1,2,3", "," , 1 )    // tokenize constant string, store in 
                                   // global token environment  
  TokenInit( cString, , 1, @cTE1)  // tokenize cString and store token 
                                   // environment in cTE1 only without 
                                   // overriding global token environment
  TokenInit( cString, , 1, cTE1 )  // tokenize cString and store token 
                                   // environment in GLOBAL token environment 
                                   // since 4th parameter is not given by 
                                   // reference !!!
  TokenInit( ,,, @cTE1 )           // set counter in TE stored in cTE1 to 1

Compliance

TokenInit() is compatible with CT3’s TokenInit(), but there is an additional parameter featuring local token environments.

Platforms

All

Files

Source is token2.c, library is libct.

Seealso

TOKEN(), TOKENEXIT(), TOKENNEXT(), TOKENNUM(), TOKENAT(), SAVETOKEN(), RESTTOKEN(), TOKENEND()