c++quex

Problems porting lexer to newer version of Quex

I used to feed my Quex powered lexer with a std::basic_stringstream<char> being sent to the constructor, as such:

typedef std::basic_stringstream<char> UStringStream;
UStringStream tokenStream(sCode);
quex::ecmascript_lexer ecmascript_lexer(&tokenStream);

This worked using Quex 0.64.8, but it seems the API has changed since then since it doesn't seem to work with version 0.67.4.

Any pointer regarding how I should port my code to the new version would be much appreciated.

Best regards,

Patrik J

EDIT: Removed the question regarding the string accumulator, it will be re-posted as a separate question.

Solution

  • These are actually two questions. Let's assume you edit your text so that this question is about the new constructor API. For the accumulator, please, post a separate question.

    The new API for constructor, include_push, and reset has been completely redone so that it can cope with any type of input. The key to that is the abstraction of a 'ByteLoader'. ByteLoader are implemented for C-Standard files and streams, POSIX file descriptors (->use for sockets), and direct loading from memory (good check for large sets of scenarous in unit tests). Now, how is this used?

    Create a ByteLoader for your input scenario, namely 'std::stream':

    QUEX_NAME(ByteLoader)* byte_loader = QUEX_NAME(ByteLoader_stream_new)(&my_stream);

    Then pass it to the constructor (or placement new):

    MyLexer lexer(byte_loader, /* converter */0);

    If you want to work on converted content, you might have to create a converter and pass it as second argument. Thus, the complete code becomes something like:

    QUEX_NAME(ByteLoader)* byte_loader = QUEX_NAME(ByteLoader_stream_new)(&my_stream);
QUEX_NAME(Converter)*  converter   = QUEX_NAME(Converter_IConv_new)("UTF-8", NULL);
MyLexer                lexer(byte_loader, converter);

    This type of construction of a lexical analyzer allows for a maximum of flexibility. There is no restriction to C-Standard file interfaces, input can come from any source, such as sockets or the serial line. And the conversion of incoming data is done behind the scenes while the synchronization with the buffer's content is done completely by the lexer's eninge.

    Note, that the constructor transfers ownership of the ByteLoader and Converter to the lexical analyzer. There is no need to delete them manually.

    Headers and implementations for ByteLoader-s and Converter-s are located in 

    $QUEX_PATH/quex/code_base/buffer/bytes/...

    and 

    $QUEX_PATH/quex/code_base/buffer/lexatoms/converter/...

    If you have new types of ByteLoader-s or Converter-s , submit them as a patch. As soon as they are tested, they are likely to be entered into the code base.