Imports #
"fmt"
"go/token"
"io"
"sort"
"bytes"
"fmt"
"go/token"
"path/filepath"
"strconv"
"unicode"
"unicode/utf8"
scanner
Imports
Constants & Variables
ScanComments const #
const ScanComments Mode = *ast.BinaryExprbom const #
const bom = 0xFEFFdontInsertSemis const #
const dontInsertSemiseof const #
const eof = *ast.UnaryExprprefix var #
var prefix = *ast.CallExprType Aliases
ErrorHandler type #
An ErrorHandler may be provided to [Scanner.Init]. If a syntax error is encountered and a handler was installed, the handler is called with a position and an error message. The position points to the beginning of the offending token.
type ErrorHandler func(pos token.Position, msg string)ErrorList type #
ErrorList is a list of *Errors. The zero value for an ErrorList is an empty ErrorList ready to use.
type ErrorList []*ErrorMode type #
A mode value is a set of flags (or 0). They control scanner behavior.
type Mode uintStructs
Error struct #
In an [ErrorList], an error is represented by an *Error. The position Pos, if valid, points to the beginning of the offending token, and the error condition is described by Msg.
type Error struct {
Pos token.Position
Msg string
}Scanner struct #
A Scanner holds the scanner's internal state while processing a given text. It can be allocated as part of another data structure but must be initialized via [Scanner.Init] before use.
type Scanner struct {
file *token.File
dir string
src []byte
err ErrorHandler
mode Mode
ch rune
offset int
rdOffset int
lineOffset int
insertSemi bool
nlPos token.Pos
ErrorCount int
}Functions
Add method #
Add adds an [Error] with given position and error message to an [ErrorList].
func (p *ErrorList) Add(pos token.Position, msg string)Err method #
Err returns an error equivalent to this error list. If the list is empty, Err returns nil.
func (p ErrorList) Err() errorError method #
Error implements the error interface.
func (e Error) Error() stringError method #
An [ErrorList] implements the error interface.
func (p ErrorList) Error() stringInit method #
Init prepares the scanner s to tokenize the text src by setting the scanner at the beginning of src. The scanner uses the file set file for position information and it adds line information for each line. It is ok to re-use the same file when re-scanning the same file as line information which is already present is ignored. Init causes a panic if the file size does not match the src size. Calls to [Scanner.Scan] will invoke the error handler err if they encounter a syntax error and err is not nil. Also, for each error encountered, the [Scanner] field ErrorCount is incremented by one. The mode parameter determines how comments are handled. Note that Init may call err if there is an error in the first character of the file.
func (s *Scanner) Init(file *token.File, src []byte, err ErrorHandler, mode Mode)Len method #
[ErrorList] implements the sort Interface.
func (p ErrorList) Len() intLess method #
func (p ErrorList) Less(i int, j int) boolPrintError function #
PrintError is a utility function that prints a list of errors to w, one error per line, if the err parameter is an [ErrorList]. Otherwise it prints the err string.
func PrintError(w io.Writer, err error)RemoveMultiples method #
RemoveMultiples sorts an [ErrorList] and removes all but the first error per line.
func (p *ErrorList) RemoveMultiples()Reset method #
Reset resets an [ErrorList] to no errors.
func (p *ErrorList) Reset()Scan method #
Scan scans the next token and returns the token position, the token, and its literal string if applicable. The source end is indicated by [token.EOF]. If the returned token is a literal ([token.IDENT], [token.INT], [token.FLOAT], [token.IMAG], [token.CHAR], [token.STRING]) or [token.COMMENT], the literal string has the corresponding value. If the returned token is a keyword, the literal string is the keyword. If the returned token is [token.SEMICOLON], the corresponding literal string is ";" if the semicolon was present in the source, and "\n" if the semicolon was inserted because of a newline or at EOF. If the returned token is [token.ILLEGAL], the literal string is the offending character. In all other cases, Scan returns an empty literal string. For more tolerant parsing, Scan will return a valid token if possible even if a syntax error was encountered. Thus, even if the resulting token sequence contains no illegal tokens, a client may not assume that no error occurred. Instead it must check the scanner's ErrorCount or the number of calls of the error handler, if there was one installed. Scan adds line information to the file added to the file set with Init. Token positions are relative to that file and thus relative to the file set.
func (s *Scanner) Scan() (pos token.Pos, tok token.Token, lit string)Sort method #
Sort sorts an [ErrorList]. *[Error] entries are sorted by position, other errors are sorted by error message, and before any *[Error] entry.
func (p ErrorList) Sort()Swap method #
func (p ErrorList) Swap(i int, j int)digitVal function #
func digitVal(ch rune) intdigits method #
digits accepts the sequence { digit | '_' }. If base <= 10, digits accepts any decimal digit but records the offset (relative to the source start) of a digit >= base in *invalid, if *invalid < 0. digits returns a bitset describing whether the sequence contained digits (bit 0 is set), or separators '_' (bit 1 is set).
func (s *Scanner) digits(base int, invalid *int) (digsep int)error method #
func (s *Scanner) error(offs int, msg string)errorf method #
func (s *Scanner) errorf(offs int, format string, args ...any)invalidSep function #
invalidSep returns the index of the first invalid separator in x, or -1.
func invalidSep(x string) intisDecimal function #
func isDecimal(ch rune) boolisDigit function #
func isDigit(ch rune) boolisHex function #
func isHex(ch rune) boolisLetter function #
func isLetter(ch rune) boollitname function #
func litname(prefix rune) stringlower function #
func lower(ch rune) runenext method #
Read the next Unicode char into s.ch. s.ch < 0 means end-of-file. For optimization, there is some overlap between this method and s.scanIdentifier.
func (s *Scanner) next()peek method #
peek returns the byte following the most recently read character without advancing the scanner. If the scanner is at EOF, peek returns 0.
func (s *Scanner) peek() bytescanComment method #
scanComment returns the text of the comment and (if nonzero) the offset of the first newline within it, which implies a /*...*/ comment.
func (s *Scanner) scanComment() (string, int)scanEscape method #
scanEscape parses an escape sequence where rune is the accepted escaped quote. In case of a syntax error, it stops at the offending character (without consuming it) and returns false. Otherwise it returns true.
func (s *Scanner) scanEscape(quote rune) boolscanIdentifier method #
scanIdentifier reads the string of valid identifier characters at s.offset. It must only be called when s.ch is known to be a valid letter. Be careful when making changes to this function: it is optimized and affects scanning performance significantly.
func (s *Scanner) scanIdentifier() stringscanNumber method #
func (s *Scanner) scanNumber() (token.Token, string)scanRawString method #
func (s *Scanner) scanRawString() stringscanRune method #
func (s *Scanner) scanRune() stringscanString method #
func (s *Scanner) scanString() stringskipWhitespace method #
func (s *Scanner) skipWhitespace()stripCR function #
func stripCR(b []byte, comment bool) []byteswitch2 method #
func (s *Scanner) switch2(tok0 token.Token, tok1 token.Token) token.Tokenswitch3 method #
func (s *Scanner) switch3(tok0 token.Token, tok1 token.Token, ch2 rune, tok2 token.Token) token.Tokenswitch4 method #
func (s *Scanner) switch4(tok0 token.Token, tok1 token.Token, ch2 rune, tok2 token.Token, tok3 token.Token) token.TokentrailingDigits function #
func trailingDigits(text []byte) (int, int, bool)updateLineInfo method #
updateLineInfo parses the incoming comment text at offset offs as a line directive. If successful, it updates the line info table for the position next per the line directive.
func (s *Scanner) updateLineInfo(next int, offs int, text []byte)Generated with Arrow