Microbenchmark on the UTF-8 encoder:

import pyperf
runner = pyperf.Runner()
runner.timeit('abc',
    setup='s="abc"',
    stmt='s.encode()')
runner.timeit('a x 1000',
    setup='s="a" * 1000',
    stmt='s.encode()')
runner.timeit('ab<surrogate> [namereplace]',
    setup=r's="ab\udc80"',
    stmt='s.encode(errors="namereplace")')
runner.timeit('ab<surrogate> [ignore]',
    setup=r's="ab\udc80"',
    stmt='s.encode(errors="ignore")')
runner.timeit('(a<surrogate>) x 1000 [namereplace]',
    setup=r's="a\udc80" * 1000',
    stmt='s.encode(errors="namereplace")')
runner.timeit('(a<surrogate>) x 1000 [ignore]',
    setup=r's="a\udc80" * 1000',
    stmt='s.encode(errors="ignore")')

Results:

cc @serhiy-storchaka

Please make benchmarks for non-ASCII strings. Consider different ranges (which represents different internal representations and lenghts of UTF-8 representation):

• 0-0x7F
• 0x80-0xFF
• 0x100-0x3FF
• 0x400-0xFFFF
• 0x10000-0x10FFFF

Consider also strings which contain one character from the higher range (for example, 0x10000 and all other characters ASCII, etc).

More benchmark:

import pyperf
runner = pyperf.Runner()
ranges = (
    (r'\0',
     r'\x7f'),
    (r'\x80',
     r'\xff'),
    (r'\u0400',
     r'\u0fff'),
    (r'\U00010000',
     r'\u0010ffff'),
)
for first, last in ranges:
    runner.timeit(f'"{first}{last}"',
        setup=f"first='{first}'; last='{last}'; s=first+last",
        stmt='s.encode()')
for length in (5, 50, 500):
    for first, last in ranges:
        runner.timeit(f'"{first}{last}" * {length}',
            setup=f"first='{first}'; last='{last}'; s=(first+last) * {length}",
            stmt='s.encode()')

Results:

Benchmark hidden because not significant (5): "\U00010000\u0010ffff", "\u0400\u0fff" * 5, "\x80\xff" * 50, "\0\x7f" * 500, "\U00010000\u0010ffff" * 500

Benchmark:

import pyperf
runner = pyperf.Runner()
for length in (5, 50, 500):
    runner.timeit(f'"x" * {length} + chr(0x10000)',
        setup=f's="x" * {length} + chr(0x10000)',
        stmt='s.encode()')

Results:

Benchmark hidden because not significant (1): "x" * 500 + chr(0x10000)

@serhiy-storchaka: The difference is about a "few nanoseconds", around +10 ns in the worst case, 1.08x faster in the best case. Do you think that it's acceptable?

Some overhead is caused by dynamic memory allocation for PyBytesWriter -- it is unavoidable.

PyBytesWriter_Create() uses a free list to avoid PyMem_Malloc() cost in the common case.

But there may be a loss due to losing fine control on overallocation -- I need to look at it closer. Maybe it can be avoided by adding a new C API.

If this loss can be measured, I would suggest adding a private API to disable overallocation.

Updated benchmark on the worst case:

import pyperf
runner=pyperf.Runner()
runner.timeit('utf8',
    setup=r's="\uFFFF"*(256//3)+"\uDC80"',
    stmt='s.encode(errors="backslashreplace")')
runner.timeit('latin1',
    setup=r"s=('a'*255+'\u0100')",
    stmt="s.encode('latin1', 'backslashreplace')")

Result:

@serhiy-storchaka: utf8_encoder() and unicode_encode_ucs1() are the last 2 functions using the private API. I would like to merge this change to be able to remove the private API, even if there is an overhead on performance. On the common cases, there is no significant impact on performance.

I updated the PR to keep the overallocate=0 optimization.

More benchmarks on the corner cases.

Benchmark: python -m pyperf timeit -s "s=('a'*100+'\u0100'*100)" "s.encode('latin1', 'backslashreplace')"

Result: Mean +- std dev: [timeit1_ref] 503 ns +- 37 ns -> [timeit1_pep782] 483 ns +- 25 ns: 1.04x faster

Benchmark: python -m pyperf timeit -s "s=(('a'*10+'\u0100')*10)" "s.encode('latin1', 'backslashreplace')"

Result: Mean +- std dev: [timeit2_ref] 243 ns +- 13 ns -> [timeit2_pep782] 248 ns +- 4 ns: 1.02x slower

I updated the PR to reimplement the min_size micro-optimization. There is no more 1.3x slowdown.

I also recomputed all benchmark results on the latest PR version. Results are now between 1.08x slower and 1.08x faster. Most benchmarks are in the [-5%, +5%] range which can be associated to noise in the benchmark (can be ignored).

@serhiy-storchaka: I plan to merge this change next week.

Merged, thanks for the review @serhiy-storchaka.

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Ubuntu Shared 3.x (tier-1) has failed when building commit 8cfd7b4.

What do you need to do:

1. Don't panic.
2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/506/builds/11478) and take a look at the build logs.
4. Check if the failure is related to this commit (8cfd7b4) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/506/builds/11478

Failed tests:

• test_interpreters

Failed subtests:

• test_keyboard_interrupt_in_thread_running_interp - test.test_interpreters.test_api.InterpreterObjectTests.test_keyboard_interrupt_in_thread_running_interp

Summary of the results of the build (if available):

==

Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_interpreters/test_api.py", line 462, in test_keyboard_interrupt_in_thread_running_interp
    self.assertEqual(retcode, 0)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^
AssertionError: -2 != 0