2. Lexical analysis — Python 3.16.0a0 documentation

docs.python.org

Tin mới

Python Module Index

3.16.0a0 Documentation

The Python Language Reference

2.1. Line structure¶

A Python program is read by a parser. Input to the parser is a stream of tokens, generated by the lexical analyzer (also known as the tokenizer). This chapter describes how the lexical analyzer produces these tokens.

2.1. Line structure¶

A Python program is read by a parser. Input to the parser is a stream of tokens, generated by the lexical analyzer (also known as the tokenizer). This chapter describes how the lexical analyzer produces these tokens.

2.1.1. Logical lines¶

The end of a logical line is represented by the token NEWLINE. Statements cannot cross logical line boundaries except where NEWLINE is allowed by the syntax (e.g., between statements in compound statements). A logical li

Standard Encodings

If no encoding declaration is found, the default encoding is UTF-8. If the implicit or explicit encoding of a file is UTF-8, an initial UTF-8 byte-order mark (b'\xef\xbb\xbf') is ignored rather than being a syntax error.

builtins: The module that provides the built-in namespace.

In a case pattern within a match statement, _ is a soft keyword that denotes a wildcard.

gettext: Multilingual internationalization services.

The name _ is often used in conjunction with internationalization; refer to the documentation for the gettext module for more information on this convention.

Special method names

Identifiers (Names)

Class-private names. Names in this category, when used within the context of a class definition, are re-written to use a mangled form to help avoid name clashes between “private” attributes of base and derived classes. S

unicodedata: Access the Unicode Database.

Names that contain non-ASCII characters need additional normalization and validation beyond the rules and grammar explained above. For example, ř_1, 蛇, or साँप are valid names, but r〰2, €, or 🐍 are not.

negative lookaheads

String literals, except “f-strings” and “t-strings”, are described by the following lexical definitions.

2.5.5. Bytes literals¶

Bytes literals are always prefixed with ‘b’ or ‘B’; they produce an instance of the bytes type instead of the str type. They may only contain ASCII characters; bytes with a numeric value of 128 or greater must be express

Format Specification Mini-Language

Changed in version 3.7: The await and async for can be used in expressions within f-strings.

2.6. Numeric literals¶

NUMBER tokens represent numeric literals, of which there are three types: integers, floating-point numbers, and imaginary numbers.

2.7. Operators and delimiters¶

The following grammar defines operator and delimiter tokens, that is, the generic OP token type. A list of these tokens and their names is also available in the token module documentation.