Token()

Tokens of a string

Syntax

      TOKEN( <cString>, [<cTokenizer>],
             [<nTokenCount], [<nSkipWidth>],
             [<@cPreTokenSep>], [<@cPostTokenSep>] ) -> cToken

Arguments

<cString> is the processed string

[<cTokenizer>] is a list of characters separating the tokens in <cString> Default: chr(0)+chr(9)+chr(10)+chr(13)+chr(26)+ chr(32)+chr(32)+chr(138)+chr(141)+ “, .;:!\?/\\<>()#&%+-*”

[<nTokenCount>] specifies the count of the token that should be extracted Default: last token

[<nSkipWidth>] specifies the maximum number of successive tokenizing characters that are combined as ONE token stop, e.g. specifying 1 can yield to empty token Default: 0, any number of successive tokenizing characters are combined as ONE token stop

[<@cPreTokenSep>] If given by reference, the tokenizer before the actual token will be stored

[<@cPostTokenSep>] If given by reference, the tokenizer after the actual token will be stored

Returns

<cToken> the token specified by the parameters given above

Description

The TOKEN() function extracts the <nTokenCount>th token from the string <cString>. In the course of this, the tokens in the string are separated by the character(s) specified in <cTokenizer>. The function may also extract empty tokens, if you specify a skip width other than zero.

Be aware of the new 5th and 6th parameter there the TOKEN() function stores the tokenizing character before and after the extracted token. Therefore, additional calls to the TOKENSEP() function are not necessary.

Examples

      ? token( "Hello, World!" )            -->  "World"
      ? token( "Hello, World!",, 2, 1 )     --> ""
      ? token( "Hello, World!", ",", 2, 1 ) --> " World!"
      ? token( "Hello, World!", " ", 2, 1 ) --> "World!"

Tests

      token( "Hello, World!" )            == "World"
      token( "Hello, World!",, 2, 1 )     == ""
      token( "Hello, World!", ",", 2, 1 ) == " World!"
      token( "Hello, World!", " ", 2, 1 ) == "World!"

Compliance

TOKEN() is compatible with CT3’s TOKEN, but two additional parameters have been added there the TOKEN() function can store the tokenizers before and after the current token.

Platforms

All

Files

Source is token1.c, library is libct.

Seealso

NUMTOKEN(), ATTOKEN(), TOKENLOWER(), TOKENUPPER(), TOKENSEP()