Module syntax::tokenstream [−] [src]

🔬 This is a nightly-only experimental API. (rustc_private)

this crate is being loaded from the sysroot, an unstable location; did you mean to load this crate from crates.io via Cargo.toml instead?

Token Streams

TokenStreams represent syntactic objects before they are converted into ASTs. A TokenStream is, roughly speaking, a sequence (eg stream) of TokenTrees, which are themselves a single Token or a Delimited subsequence of tokens.

Ownership

TokenStreams are persistent data structures constructed as ropes with reference counted-children. In general, this means that calling an operation on a TokenStream (such as slice) produces an entirely new TokenStream from the borrowed reference to the original. This essentially coerces TokenStreams into 'views' of their subparts, and a borrowed TokenStream is sufficient to build an owned TokenStream without taking ownership of the original.

Structs

Cursor	[ Experimental ]
Delimited	[ Experimental ] A delimited sequence of token trees
ThinTokenStream	[ Experimental ] The `TokenStream` type is large enough to represent a single `TokenTree` without allocation. `ThinTokenStream` is smaller, but needs to allocate to represent a single `TokenTree`. We must use `ThinTokenStream` in `TokenTree::Delimited` to avoid infinite size due to recursion.
TokenStream	[ Experimental ] Token Streams A `TokenStream` is an abstract sequence of tokens, organized into `TokenTree`s. The goal is for procedural macros to work with `TokenStream`s and `TokenTree`s instead of a representation of the abstract syntax tree. Today's `TokenTree`s can still contain AST via `Token::Interpolated` for back-compat.
TokenStreamBuilder	[ Experimental ]

Enums

TokenTree

[

Experimental

]

When the main rust parser encounters a syntax-extension invocation, it parses the arguments to the invocation as a token-tree. This is a very loose structure, such that all sorts of different AST-fragments can be passed to syntax extensions using a uniform type.