History for Lib/tokenize.py - python/cpython · GitHub

github.com

Tin mới

[3.14] gh-63161: Fix tokenize.detect_encoding() (GH-139446) (GH-140378)

[3.14] gh-134675: Add t-string prefixes to tokenizer module, lexical analysis doc, and add a test to make sure we catch this error in the future. (GH-134734) (#134739)

[3.14] gh-134582: Fix t-strings untokenize() roundtrip removing space between braces (GH-134603) (#134659)

gh-130645: Add color to stdlib argparse CLIs (gh-133380)

gh-133306: Use \z instead of \Z in regular expressions in the stdlib (GH-133337)

gh-132661: Implement PEP 750 (#132662)

gh-131178: Add tests for `tokenize` command-line interface (#131274)

gh-125553: Fix backslash continuation in `untokenize` (#126010)

gh-128519: Align the docstring of untokenize() to match the docs (#128521)

gh-91818: Use default program name in the CLI of many modules (GH-124867)

gh-125008: Fix `tokenize.untokenize` roundtrip for `\n{{` (#125013)

gh-115154: Fix untokenize handling of unicode named literals (#115171)

gh-104169: Fix test_peg_generator after tokenizer refactoring (#110727)

gh-71299: Fix __all__ in tokenize (#105907)

gh-105390: Correctly raise TokenError instead of SyntaxError for tokenize errors (#105399)

gh-105324: Fix tokenize module main function for stdin (#105325)

gh-105069: Add a readline-like callable to the tokenizer to consume input iteratively (#105070)

gh-104976: Ensure trailing dedent tokens are emitted as the previous tokenizer (#104980)

gh-102856: Tokenize performance improvement (#104731)

gh-104719: Restore Tokenize module constants (#104722)

gh-102856: Python tokenizer implementation for PEP 701 (#104323)

bpo-46565: `del` loop vars that are leaking into module namespaces (GH-30993)

Add tests for the C tokenizer and expose it as a private module (GH-27924)

bpo-44667: Treat correctly lines ending with comments and no newlines in the Python tokenizer (GH-27499)

bpo-43014: Improve performance of tokenize.tokenize by 20-30%

bpo-5028: Fix up rest of documentation for tokenize documenting line (GH-13686)

bpo-5028: fix doc bug for tokenize (GH-11683)

bpo-36766: Typos in docs and code comments (GH-13116)

bpo-30455: Generate all token related code and docs from Grammar/Tokens. (GH-10370)

bpo-33899: Make tokenize module mirror end-of-file is end-of-line behavior (GH-7891)

bpo-12486: Document tokenize.generate_tokens() as public API (#6957)

bpo-33338: [tokenize] Minor code cleanup (#6573)

bpo-33260: Regenerate token.py after removing ASYNC and AWAIT. (GH-6447)

bpo-30406: Make async and await proper keywords (#1669)

bpo-25324: copy tok_name before changing it (#1608)