TokenNext()
Successivly obtains tokens from a string
Syntax
TokenNext( <[@]cString>, [<nToken>],
[<@cTokenEnvironment>] ) -> cToken
Arguments
<[@]cString> the processed string <nToken> a token number
<@cTokenEnvironment> a token environment
Returns
<cToken> a token from <cString>
Description
With TokenNext(), the tokens determined with the TOKENINIT() functions can be retrieved. To do this, TokenNext() uses the information stored in either the global token environment or the local one supplied by <cTokenEnvironment>. Note that, is supplied, this 3rd parameter has always to be passed by reference.
If the 2nd parameter, <nToken> is given, TokenNext() simply returns the <nToken>th token without manipulating the TE counter. Otherwise the token pointed to by the TE counter is returned and the counter is incremented by one. Like this, a simple loop with TOKENEND() can be used to retrieve all tokens of a string successivly.
Note that <cString> does not have to be the same used in TOKENINIT(), so that one can do a “correlational tokenization”, i.e. tokenize a string as if it was another! E.G. using TOKENINIT() with the string “AA, BBB” but calling TokenNext() with “CCCEE” would give first “CC” and then “EE” (because “CCCEE” is not long enough).
Examples
// default behavhiour
tokeninit( cString ) // initialize a token environment
DO WHILE ! tokenend()
? TokenNext( cString ) // get all tokens successivly
ENDDO
? TokenNext( cString, 3 ) // get the 3rd token, counter will remain
// the same
tokenexit() // free the memory used for the global
// token environment
Compliance
TokenNext() is compatible with CT3’s TokenNext(), but there are two additional parameters featuring local token environments and optional access to tokens.
Platforms
All
Files
Source is token2.c, library is libct.
Seealso
TOKENINIT(), TOKENEXIT(), TOKENNUM(), TOKENAT(), SAVETOKEN(), RESTTOKEN(), TOKENEND()