token

Imports

Imports # 

"cmp"
"fmt"
"slices"
"strconv"
"sync"
"sync/atomic"
"strconv"
"unicode"
"unicode/utf8"

Constants & Variables

ADD const #

Operators and delimiters

const ADD

ADD_ASSIGN const #

The list of tokens.

const ADD_ASSIGN

AND const #

The list of tokens.

const AND

AND_ASSIGN const #

The list of tokens.

const AND_ASSIGN

AND_NOT const #

The list of tokens.

const AND_NOT

AND_NOT_ASSIGN const #

The list of tokens.

const AND_NOT_ASSIGN

ARROW const #

The list of tokens.

const ARROW

ASSIGN const #

The list of tokens.

const ASSIGN

BREAK const #

Keywords

const BREAK

CASE const #

The list of tokens.

const CASE

CHAN const #

The list of tokens.

const CHAN

CHAR const #

The list of tokens.

const CHAR

COLON const #

The list of tokens.

const COLON

COMMA const #

The list of tokens.

const COMMA

COMMENT const #

The list of tokens.

const COMMENT

CONST const #

The list of tokens.

const CONST

CONTINUE const #

The list of tokens.

const CONTINUE

DEC const #

The list of tokens.

const DEC

DEFAULT const #

The list of tokens.

const DEFAULT

DEFER const #

The list of tokens.

const DEFER

DEFINE const #

The list of tokens.

const DEFINE

ELLIPSIS const #

The list of tokens.

const ELLIPSIS

ELSE const #

The list of tokens.

const ELSE

EOF const #

The list of tokens.

const EOF

EQL const #

The list of tokens.

const EQL

FALLTHROUGH const #

The list of tokens.

const FALLTHROUGH

FLOAT const #

The list of tokens.

const FLOAT

FOR const #

The list of tokens.

const FOR

FUNC const #

The list of tokens.

const FUNC

GEQ const #

The list of tokens.

const GEQ

GO const #

The list of tokens.

const GO

GOTO const #

The list of tokens.

const GOTO

GTR const #

The list of tokens.

const GTR

HighestPrec const #

A set of constants for precedence-based expression parsing. Non-operators have lowest precedence, followed by operators starting with precedence 1 up to unary operators. The highest precedence serves as "catch-all" precedence for selector, indexing, and other operator and delimiter tokens.

const HighestPrec = 7

IDENT const #

Identifiers and basic type literals (these tokens stand for classes of literals)

const IDENT

IF const #

The list of tokens.

const IF

ILLEGAL const #

Special tokens

const ILLEGAL Token = iota

IMAG const #

The list of tokens.

const IMAG

IMPORT const #

The list of tokens.

const IMPORT

INC const #

The list of tokens.

const INC

INT const #

The list of tokens.

const INT

INTERFACE const #

The list of tokens.

const INTERFACE

LAND const #

The list of tokens.

const LAND

LBRACE const #

The list of tokens.

const LBRACE

LBRACK const #

The list of tokens.

const LBRACK

LEQ const #

The list of tokens.

const LEQ

LOR const #

The list of tokens.

const LOR

LPAREN const #

The list of tokens.

const LPAREN

LSS const #

The list of tokens.

const LSS

LowestPrec const #

A set of constants for precedence-based expression parsing. Non-operators have lowest precedence, followed by operators starting with precedence 1 up to unary operators. The highest precedence serves as "catch-all" precedence for selector, indexing, and other operator and delimiter tokens.

const LowestPrec = 0

MAP const #

The list of tokens.

const MAP

MUL const #

The list of tokens.

const MUL

MUL_ASSIGN const #

The list of tokens.

const MUL_ASSIGN

NEQ const #

The list of tokens.

const NEQ

NOT const #

The list of tokens.

const NOT

NoPos const #

The zero value for [Pos] is NoPos; there is no file and line information associated with it, and NoPos.IsValid() is false. NoPos is always smaller than any other [Pos] value. The corresponding [Position] value for NoPos is the zero value for [Position].

const NoPos Pos = 0

OR const #

The list of tokens.

const OR

OR_ASSIGN const #

The list of tokens.

const OR_ASSIGN

PACKAGE const #

The list of tokens.

const PACKAGE

PERIOD const #

The list of tokens.

const PERIOD

QUO const #

The list of tokens.

const QUO

QUO_ASSIGN const #

The list of tokens.

const QUO_ASSIGN

RANGE const #

The list of tokens.

const RANGE

RBRACE const #

The list of tokens.

const RBRACE

RBRACK const #

The list of tokens.

const RBRACK

REM const #

The list of tokens.

const REM

REM_ASSIGN const #

The list of tokens.

const REM_ASSIGN

RETURN const #

The list of tokens.

const RETURN

RPAREN const #

The list of tokens.

const RPAREN

SELECT const #

The list of tokens.

const SELECT

SEMICOLON const #

The list of tokens.

const SEMICOLON

SHL const #

The list of tokens.

const SHL

SHL_ASSIGN const #

The list of tokens.

const SHL_ASSIGN

SHR const #

The list of tokens.

const SHR

SHR_ASSIGN const #

The list of tokens.

const SHR_ASSIGN

STRING const #

The list of tokens.

const STRING

STRUCT const #

The list of tokens.

const STRUCT

SUB const #

The list of tokens.

const SUB

SUB_ASSIGN const #

The list of tokens.

const SUB_ASSIGN

SWITCH const #

The list of tokens.

const SWITCH

TILDE const #

additional tokens, handled in an ad-hoc manner

const TILDE

TYPE const #

The list of tokens.

const TYPE

UnaryPrec const #

A set of constants for precedence-based expression parsing. Non-operators have lowest precedence, followed by operators starting with precedence 1 up to unary operators. The highest precedence serves as "catch-all" precedence for selector, indexing, and other operator and delimiter tokens.

const UnaryPrec = 6

VAR const #

The list of tokens.

const VAR

XOR const #

The list of tokens.

const XOR

XOR_ASSIGN const #

The list of tokens.

const XOR_ASSIGN

additional_beg const #

The list of tokens.

const additional_beg

additional_end const #

The list of tokens.

const additional_end

debug const #

If debug is set, invalid offset and position values cause a panic (go.dev/issue/57490).

const debug = false

keyword_beg const #

The list of tokens.

const keyword_beg

keyword_end const #

The list of tokens.

const keyword_end

keywords var # 

var keywords map[string]Token

literal_beg const #

The list of tokens.

const literal_beg

literal_end const #

The list of tokens.

const literal_end

operator_beg const #

The list of tokens.

const operator_beg

operator_end const #

The list of tokens.

const operator_end

tokens var # 

var tokens = [...]string{...}

Type Aliases

Pos type #

Pos is a compact encoding of a source position within a file set. It can be converted into a [Position] for a more convenient, but much larger, representation. The Pos value for a given file is a number in the range [base, base+size], where base and size are specified when a file is added to the file set. The difference between a Pos value and the corresponding file base corresponds to the byte offset of that position (represented by the Pos value) from the beginning of the file. Thus, the file base offset is the Pos value representing the first byte in the file. To create the Pos value for a specific source offset (measured in bytes), first add the respective file to the current file set using [FileSet.AddFile] and then call [File.Pos](offset) for that file. Given a Pos value p for a specific file set fset, the corresponding [Position] value is obtained by calling fset.Position(p). Pos values can be compared directly with the usual comparison operators: If two Pos values p and q are in the same file, comparing p and q is equivalent to comparing the respective source file offsets. If p and q are in different files, p < q is true if the file implied by p was added to the respective file set before the file implied by q.

type Pos int

Token type #

Token is the set of lexical tokens of the Go programming language.

type Token int

Structs

File struct #

A File is a handle for a file belonging to a [FileSet]. A File has a name, size, and line offset table.

type File struct {
name string
base int
size int
mutex sync.Mutex
lines []int
infos []lineInfo
}

FileSet struct #

A FileSet represents a set of source files. Methods of file sets are synchronized; multiple goroutines may invoke them concurrently. The byte offsets for each file in a file set are mapped into distinct (integer) intervals, one interval [base, base+size] per file. [FileSet.Base] represents the first byte in the file, and size is the corresponding file size. A [Pos] value is a value in such an interval. By determining the interval a [Pos] value belongs to, the file, its file base, and thus the byte offset (position) the [Pos] value is representing can be computed. When adding a new file, a file base must be provided. That can be any integer value that is past the end of any interval of any file already in the file set. For convenience, [FileSet.Base] provides such a value, which is simply the end of the Pos interval of the most recently added file, plus one. Unless there is a need to extend an interval later, using the [FileSet.Base] should be used as argument for [FileSet.AddFile]. A [File] may be removed from a FileSet when it is no longer needed. This may reduce memory usage in a long-running application.

type FileSet struct {
mutex sync.RWMutex
base int
files []*File
last *ast.IndexExpr
}

Position struct #

Position describes an arbitrary source position including the file, line, and column location. A Position is valid if the line number is > 0.

type Position struct {
Filename string
Offset int
Line int
Column int
}

lineInfo struct #

A lineInfo object describes alternative file, line, and column number information (such as provided via a //line directive) for a given file offset.

type lineInfo struct {
Offset int
Filename string
Line int
Column int
}

serializedFile struct # 

type serializedFile struct {
Name string
Base int
Size int
Lines []int
Infos []lineInfo
}

serializedFileSet struct # 

type serializedFileSet struct {
Base int
Files []serializedFile
}

Functions

AddFile method #

AddFile adds a new file with a given filename, base offset, and file size to the file set s and returns the file. Multiple files may have the same name. The base offset must not be smaller than the [FileSet.Base], and size must not be negative. As a special case, if a negative base is provided, the current value of the [FileSet.Base] is used instead. Adding the file will set the file set's [FileSet.Base] value to base + size + 1 as the minimum base value for the next file. The following relationship exists between a [Pos] value p for a given file offset offs: int(p) = base + offs with offs in the range [0, size] and thus p in the range [base, base+size]. For convenience, [File.Pos] may be used to create file-specific position values from a file offset.

func (s *FileSet) AddFile(filename string, base int, size int) *File

AddLine method #

AddLine adds the line offset for a new line. The line offset must be larger than the offset for the previous line and smaller than the file size; otherwise the line offset is ignored.

func (f *File) AddLine(offset int)

AddLineColumnInfo method #

AddLineColumnInfo adds alternative file, line, and column number information for a given file offset. The offset must be larger than the offset for the previously added alternative line info and smaller than the file size; otherwise the information is ignored. AddLineColumnInfo is typically used to register alternative position information for line directives such as //line filename:line:column.

func (f *File) AddLineColumnInfo(offset int, filename string, line int, column int)

AddLineInfo method #

AddLineInfo is like [File.AddLineColumnInfo] with a column = 1 argument. It is here for backward-compatibility for code prior to Go 1.11.

func (f *File) AddLineInfo(offset int, filename string, line int)

Base method #

Base returns the base offset of file f as registered with AddFile.

func (f *File) Base() int

Base method #

Base returns the minimum base offset that must be provided to [FileSet.AddFile] when adding the next file.

func (s *FileSet) Base() int

File method #

File returns the file that contains the position p. If no such file is found (for instance for p == [NoPos]), the result is nil.

func (s *FileSet) File(p Pos) (f *File)

IsExported function #

IsExported reports whether name starts with an upper-case letter.

func IsExported(name string) bool

IsIdentifier function #

IsIdentifier reports whether name is a Go identifier, that is, a non-empty string made up of letters, digits, and underscores, where the first character is not a digit. Keywords are not identifiers.

func IsIdentifier(name string) bool

IsKeyword method #

IsKeyword returns true for tokens corresponding to keywords; it returns false otherwise.

func (tok Token) IsKeyword() bool

IsKeyword function #

IsKeyword reports whether name is a Go keyword, such as "func" or "return".

func IsKeyword(name string) bool

IsLiteral method #

IsLiteral returns true for tokens corresponding to identifiers and basic type literals; it returns false otherwise.

func (tok Token) IsLiteral() bool

IsOperator method #

IsOperator returns true for tokens corresponding to operators and delimiters; it returns false otherwise.

func (tok Token) IsOperator() bool

IsValid method #

IsValid reports whether the position is valid.

func (p Pos) IsValid() bool

IsValid method #

IsValid reports whether the position is valid.

func (pos *Position) IsValid() bool

Iterate method #

Iterate calls f for the files in the file set in the order they were added until f returns false.

func (s *FileSet) Iterate(f func(*File) bool)

Line method #

Line returns the line number for the given file position p; p must be a [Pos] value in that file or [NoPos].

func (f *File) Line(p Pos) int

LineCount method #

LineCount returns the number of lines in file f.

func (f *File) LineCount() int

LineStart method #

LineStart returns the [Pos] value of the start of the specified line. It ignores any alternative positions set using [File.AddLineColumnInfo]. LineStart panics if the 1-based line number is invalid.

func (f *File) LineStart(line int) Pos

Lines method #

Lines returns the effective line offset table of the form described by [File.SetLines]. Callers must not mutate the result.

func (f *File) Lines() []int

Lookup function #

Lookup maps an identifier to its keyword token or [IDENT] (if not a keyword).

func Lookup(ident string) Token

MergeLine method #

MergeLine merges a line with the following line. It is akin to replacing the newline character at the end of the line with a space (to not change the remaining offsets). To obtain the line number, consult e.g. [Position.Line]. MergeLine will panic if given an invalid line number.

func (f *File) MergeLine(line int)

Name method #

Name returns the file name of file f as registered with AddFile.

func (f *File) Name() string

NewFileSet function #

NewFileSet creates a new file set.

func NewFileSet() *FileSet

Offset method #

Offset returns the offset for the given file position p. If p is before the file's start position (or if p is NoPos), the result is 0; if p is past the file's end position, the result is the file size (see also go.dev/issue/57490). The following invariant, though not true for offset values in general, holds for the result offset: f.Offset(f.Pos(offset)) == offset

func (f *File) Offset(p Pos) int

Pos method #

Pos returns the Pos value for the given file offset. If offset is negative, the result is the file's start position; if the offset is too large, the result is the file's end position (see also go.dev/issue/57490). The following invariant, though not true for Pos values in general, holds for the result p: f.Pos(f.Offset(p)) == p.

func (f *File) Pos(offset int) Pos

Position method #

Position returns the Position value for the given file position p. If p is out of bounds, it is adjusted to match the File.Offset behavior. Calling f.Position(p) is equivalent to calling f.PositionFor(p, true).

func (f *File) Position(p Pos) (pos Position)

Position method #

Position converts a [Pos] p in the fileset into a Position value. Calling s.Position(p) is equivalent to calling s.PositionFor(p, true).

func (s *FileSet) Position(p Pos) (pos Position)

PositionFor method #

PositionFor returns the Position value for the given file position p. If p is out of bounds, it is adjusted to match the File.Offset behavior. If adjusted is set, the position may be adjusted by position-altering //line comments; otherwise those comments are ignored. p must be a Pos value in f or NoPos.

func (f *File) PositionFor(p Pos, adjusted bool) (pos Position)

PositionFor method #

PositionFor converts a [Pos] p in the fileset into a [Position] value. If adjusted is set, the position may be adjusted by position-altering //line comments; otherwise those comments are ignored. p must be a [Pos] value in s or [NoPos].

func (s *FileSet) PositionFor(p Pos, adjusted bool) (pos Position)

Precedence method #

Precedence returns the operator precedence of the binary operator op. If op is not a binary operator, the result is LowestPrecedence.

func (op Token) Precedence() int

Read method #

Read calls decode to deserialize a file set into s; s must not be nil.

func (s *FileSet) Read(decode func(any) error) error

RemoveFile method #

RemoveFile removes a file from the [FileSet] so that subsequent queries for its [Pos] interval yield a negative result. This reduces the memory usage of a long-lived [FileSet] that encounters an unbounded stream of files. Removing a file that does not belong to the set has no effect.

func (s *FileSet) RemoveFile(file *File)

SetLines method #

SetLines sets the line offsets for a file and reports whether it succeeded. The line offsets are the offsets of the first character of each line; for instance for the content "ab\nc\n" the line offsets are {0, 3}. An empty file has an empty line offset table. Each line offset must be larger than the offset for the previous line and smaller than the file size; otherwise SetLines fails and returns false. Callers must not mutate the provided slice after SetLines returns.

func (f *File) SetLines(lines []int) bool

SetLinesForContent method #

SetLinesForContent sets the line offsets for the given file content. It ignores position-altering //line comments.

func (f *File) SetLinesForContent(content []byte)

Size method #

Size returns the size of file f as registered with AddFile.

func (f *File) Size() int

String method #

String returns a string in one of several forms: file:line:column valid position with file name file:line valid position with file name but no column (column == 0) line:column valid position without file name line valid position without file name and no column (column == 0) file invalid position with file name - invalid position without file name

func (pos Position) String() string

String method #

String returns the string corresponding to the token tok. For operators, delimiters, and keywords the string is the actual token character sequence (e.g., for the token [ADD], the string is "+"). For all other tokens the string corresponds to the token constant name (e.g. for the token [IDENT], the string is "IDENT").

func (tok Token) String() string

Write method #

Write calls encode to serialize the file set s.

func (s *FileSet) Write(encode func(any) error) error

file method # 

func (s *FileSet) file(p Pos) *File

fixOffset method #

fixOffset fixes an out-of-bounds offset such that 0 <= offset <= f.size.

func (f *File) fixOffset(offset int) int

init function # 

func init()

position method # 

func (f *File) position(p Pos, adjusted bool) (pos Position)

searchFiles function # 

func searchFiles(a []*File, x int) int

searchInts function # 

func searchInts(a []int, x int) int

searchLineInfos function # 

func searchLineInfos(a []lineInfo, x int) int

unpack method #

unpack returns the filename and line and column number for a file offset. If adjusted is set, unpack will return the filename and line information possibly adjusted by //line comments; otherwise those comments are ignored.

func (f *File) unpack(offset int, adjusted bool) (filename string, line int, column int)

Generated with Arrow