Chapter 9: Multiple input streams

Some scanners (such as those which support `include' files) require reading from several input streams. Usually, one switches streams when a directive like #include is encountered. Flexc++ offers two ways of switching streams:

First, a directive like #include may be defined. In this case, processing the current file is suspended and processing continues at the file mentioned in the directive. When finished processing this included file, the scanner switches back to the suspended input stream. In this case streams are pushed on a stack when encountering a stream switching directive and popped off the stack once the current file has completely been processed.
Second, the user may request the scanner to switch to another stream. Optionally, the user may request to switch back. In this case, the user is completely responsible for all stream-related bookkeeping. The scanner merely provides a means to switch from one stream to another.

Users may want to store information related to a certain stream, beside the information strictly required by flexc++ (an istream pointer and size_t for the line numbering). For example, a user may want to use the name of the file that is currently being processed. Flexc++ provides the following mechanism for storing this additional information.

The user must supply a class with certain characteristics (see below) and tell flexc++ the name and location of this class. When switching streams, a dynamically allocated pointer to this class must be provided.

9.1: StreamInfo

The class which must be provided by the user is used to instantiate the template class ScannerTemplate, it's template type parameter is called StreamInfo. The Scanner class is derived from ScannerBase, which is derived from this instantiated ScannerTemplate.

Classes which can be used as a StreamInfo must have at least the following properties:

Have a default constructor. By convention, the default constructor should associate the StreamInfo with cin
Have a member connect(StreamAdapter *). This member ...
Have a member resume(StreamInfo, size_t). ...
Close the stream upon destruction.

Flexc++ provides two StreamInfo classes for common use cases, but the user is given the option to provide it's own class. The classes flexc++ provides are in the namespace StreamSwitching (which is optionally in a user-configured namespace with the namespace option).

StreamInfo: This class is used by default. It has a default constructor, a constructor expecting the name of a file to open or a pointer to an istream (which StreamInfo will delete). The connect member will set the line number to 1.
LineStreamInfo: This class is derived from StreamInfo and has the same constructors. It furthermore makes sure that line numbering is not reset: when switching streams, the new file will begin numbering lines where the old, suspended stream left off. When this file is processed completely, the resume method sets the line number of the suspended stream to the line number of the current file.

For easy access to the current StreamInfo object, ScannerTemplate defines a typedef StreamInfoType. This has the advantage that the user has only one place where the name of the StreamInfo class is configured.

9.2: Public members

The following members are available for switching streams:

void pushStreamInfo(StreamInfo *si)

This member pushes the current stream on the scanner's stream stack and initializes the scanner's buffer with the new stream.

void popStreamInfo()

This member pops a previously pushed stream off the scanner's stream stack and resets the scanner's buffer to the popped info.
Called by default on EOF, unless the user provides an action for the <<EOF>> pattern.

StreamBuffer const switchStream(StreamInfo *si)

This member returns the current StreamBuffer and continues processing the stream defined by the StreamInfo object. All bookkeeping with regards to streams is now handled by the user.

void switchStream(StreamBuffer const &sb)

This member reverts processing to a stream buffer saved previously by switchStream's caller.

9.3: Option directives

There are two option directives which configure the name and location of the class:

streaminfoclassname: Name of the StreamInfo class.
streaminfoinclude: The file containing the StreamInfo class (included in scannerbase.h).

9.4: Examples

9.4.1: Default StreamInfo

The following example uses the default StreamSwitching::StreamInfo class: line numbering will be set to 1 when processing the include.

%x include
%%
"#include " begin(include);
[^#\n]*     std::cout << streamInfo().name() << ":" << lineno() << ": " << d_match << std::endl;
\n          // ignore

<include>[^ \n]* {
                std::cout << "Including " << d_match << ".\n";
                begin(INITIAL);
                pushStreamInfo(new Scanner::StreamInfoType(d_match.c_str()));
            }

Should one desire not to reset line numbering, include the following line at the top of the lexer file:

%option streaminfoclassname="StreamSwitching::LineStreamInfo"

The interesting stuff happens in the include start condition. A new StreamInfoType is pushed onto the streams stack. Flexc++ will automatically pop the stream once EOF is reached.

9.4.2: User supplied StreamInfo

The next example uses a StreamInfo which is supplied by the user: MyStreamInfo. It acts the same as StreamSwitching::LineStreamInfo, but shows how to include ones own StreamInfo class.

#ifndef INCLUDED_MYSTREAMINFO_H
#define INCLUDED_MYSTREAMINFO_H

#include "scannerbase.h"

class MyStreamInfo: public StreamSwitching::StreamInfo
{
    ScannerTemplate<MyStreamInfo>::StreamAdapter *d_ptr;

    public:
        MyStreamInfo()
        {}

        MyStreamInfo(char const *name)
        :
            StreamInfo(name)
        {}

        MyStreamInfo(std::istream *in)
        :
            StreamInfo(in)
        {}

        void connect(ScannerTemplate<MyStreamInfo>::StreamAdapter *ptr)
        {
            d_ptr = ptr; // store for resume
            ptr->in = d_in;
            ptr->lineno = ptr->lineno;
        }

        void resume(MyStreamInfo const &streamInfo, size_t lineno)
        {
            d_ptr->lineno = lineno;
        }
};

#endif

The corresponding lexer file should then be:

%option streaminfoclassname="MyStreamInfo" streaminfoinclude="mystreaminfo.h"
%x include
%%
"#include " begin(include);
[^#\n]*     std::cout << streamInfo().name() << ":" << lineno() << ": " << d_match << std::endl;
\n          // ignore

<include>[^ \n]* {
                std::cout << "Including " << d_match << ".\n";
                begin(INITIAL);
                pushStreamInfo(new Scanner::StreamInfoType(d_match.c_str()));
            }

As can be seen, the difference with the previous example is only the first line with the two option-directives.

9.4.3: Switching streams with switchStream

To use the raw stream switching facilities, the scanner has two members available: switchStream(StreamInfo *) and switchStream(StreamBuffer const &). The following example shows how to use these members. It maintains a stack itself, the following member should be added to scanner.h once it is generated:

std::stack<StreamBuffer> d_stack;

The lexer file should then look like:

%x include
%%
"#include " begin(include);
[^#\n]*     std::cout << streamInfo().name() << ":" << lineno() << ": " << d_match << std::endl;
\n          // ignore
<<EOF>>     {
    if (d_stack.size() < 1)
        return 0;
    std::cout << "Popping!\n";
    switchStream(d_stack.top());
    d_stack.pop();
}

<include>[^ \n]* {
    std::cout << "Including " << d_match << ".\n";
    begin(INITIAL);
    StreamBuffer const sb = switchStream(new Scanner::StreamInfoType(d_match.c_str()));
    d_stack.push(sb);
}

Now the interesting stuff happens both in the action for the <<EOF>> pattern as well as in the include start condition. In <include>, the stream is switched to the new file and the old StreamBuffer is saved. Note that switchStream returns a const copy: one should not alter this object in any way, it contains information required for lex(). This copy is then pushed onto a stack.

Once EOF is reached, and there is nothing left on the stack, scanning is finished. When there are suspended streams however, the current stream is switched to the last suspended file and the stack is reduced. Scanning continues.