TokenNext()

Successivly obtains tokens from a string

Syntax

      TokenNext( <[@]cString>, [<nToken>],
                 [<@cTokenEnvironment>] ) -> cToken

Arguments

<[@]cString> the processed string <nToken> a token number

<@cTokenEnvironment> a token environment

Returns

<cToken> a token from <cString>

Description

With TokenNext(), the tokens determined with the TOKENINIT() functions can be retrieved. To do this, TokenNext() uses the information stored in either the global token environment or the local one supplied by <cTokenEnvironment>. Note that, is supplied, this 3rd parameter has always to be passed by reference.

If the 2nd parameter, <nToken> is given, TokenNext() simply returns the <nToken>th token without manipulating the TE counter. Otherwise the token pointed to by the TE counter is returned and the counter is incremented by one. Like this, a simple loop with TOKENEND() can be used to retrieve all tokens of a string successivly.

Note that <cString> does not have to be the same used in TOKENINIT(), so that one can do a “correlational tokenization”, i.e. tokenize a string as if it was another! E.G. using TOKENINIT() with the string “AA, BBB” but calling TokenNext() with “CCCEE” would give first “CC” and then “EE” (because “CCCEE” is not long enough).

Examples

      // default behavhiour
      tokeninit( cString ) // initialize a token environment
      DO WHILE ! tokenend()
         ? TokenNext( cString )  // get all tokens successivly
      ENDDO
      ? TokenNext( cString, 3 )  // get the 3rd token, counter will remain 
                                 // the same
      tokenexit()                // free the memory used for the global 
                                 // token environment

Compliance

TokenNext() is compatible with CT3’s TokenNext(), but there are two additional parameters featuring local token environments and optional access to tokens.

Platforms

All

Files

Source is token2.c, library is libct.

Seealso

TOKENINIT(), TOKENEXIT(), TOKENNUM(), TOKENAT(), SAVETOKEN(), RESTTOKEN(), TOKENEND()