Token streams

mutable struct TokenStream

Tokenstream allows to read tokenized data from file without keeping the file ocntent in memory.

• input::IOStream: Input stream

• tokens::Vector{SubString{String}}: Array of current tokens kept in memory.

• itoken::Int64: Position of actual token in tokens array

• lineno::Int64: Line number in IOStream

• comment::Char: Comment character

• dlm::Function: Function telling if given character is a delimiter.

TokenStream(input::IOStream; comment, dlm) -> TokenStream

Create Tokenstream with IOStream argument.

TokenStream(filename::String; comment, dlm) -> TokenStream

Create Tokenstream with file name argument.

struct UnexpectedTokenError <: Exception

Error thrown when the token expected in expect! is not there.

• found::String

• expected::String

• lineno::Int64

eof(tks::TokenStream) -> Bool

Check if all tokens have been consumed.

destruct!(tks::TokenStream)

Tokenstream destructor should close input

expecttoken(tks::TokenStream, expected::String) -> Bool

Expect keyword token.

If token is missing, an UnexpectedTokenError is thrown If the token has been found, reading will continue at the position after the token found.

gettoken(
    tks::TokenStream
) -> Union{Nothing, SubString{String}}

Get next token from tokenstream.

trytoken(tks::TokenStream, expected::String) -> Bool

Try for keyword token.

It token is missing, the token read is put back into stream, a value of false is returned and the next try/gettoken command continues at the same position,

Otherwise, true is returned, and reading continues after the token found.