IMO, the existing text_factory examples look contrived; the added example where we solve the issue of invalid Unicode sequences is a more interesting and useful example.

cc. @AlexWaygood, if you have time to look at the prose.

Thanks, Alex!

It may be helpful to emphasize that the surrogate escapes cannot be passed back to sqlite through python.

The plausible real-world example -- a DB created from a CSV file or other source in legacy encoding.

Ukrainian or Korean are bad as examples, because all characters are non-ASCII, so you cannot not recognize the word in its bytes repr. "Österreich" is better, it contains only one non-Latin letter. Of course you need to use the encoding different from UTF-8 for example. It would be better if the natural legacy encoding is different from Latin1, but if there is nothing better, Latin1 works too. "Österreich" is a bit too long, some shorter Norwegian geographical name may be better.

The new example only shows the use of iterdump() and does not show the use of bytes. I think that examples with fetch() are important too. You can show it for bytes, encoding='latin1' and errors='surrogateescape', then say that iterdump() only works with text_factory producing a string and show how to save it in a file (and you can chose whether you want to recreate the DB with the originally encoded data or recode it in UTF-8 if you know the used encoding).

The CSV export from my Czech bank account uses iso-8859-2 and contains the column b'\xc8\xe1stka'.
Částka means "sum" (quantity of money). Decoding with Latin-1 this gives you Èástka which is wrong.

Or you could use b'Polo\xbeka' (položka, "entry"), which latin1 decodes to the more obviously wrong Polo¾ka.

Good examples!

@ezio-melotti, would you like to take another look?

I think this is ready to land. Shout out if you disagree. OTOH, we can easily patch up things with a new PR. I'll be merging this tomorrow morning CET.

A big thanks to everyone who helped shape this PR!

This change removes any mention of bytes. It makes the documentation less informative.

Serhiy, please see #108699 (comment). I do not intend to mention bytes unless there is a reason to do it. If you care strongly for it, I suggest you create a follow-up PR with your use-case. IMO, it does not block this PR.

It also does not resolve the original issue, in particularly, it does not document that iterdump() is not compatible with text_factory=bytes. And it does not say what to do with the result of iterdump() with custom text_factory.

It does so implicitly; it helps the user understand how to work around such encoding issues. If you disagree with this approach, please open a competing PR.

Okay. I do not fully understand what is the benefit of this change, but I do not oppose it.

Okay. I do not fully understand what is the benefit of this change, but I do not oppose it.

Why so dismissive? I've asked you twice for a concrete proposal regarding your bytes comment, but you seem to ignore this. I do not oppose such a change, I only ask for a use-case for such an example.

Sorry, I meant no disrespect. There were so many changes and comments in this PR that I was already lost. After the old examples were removed, there was a lot of discussion about new examples, but they never appeared, so I didn't see much to comment on, as this PR looked far from finished. If you consider it complete, I will perhaps continue the work in a new PR. text_factory should be fully documented (and special cases for bytes and bytearray are specially optimized in the code).

Ok, thanks for your response, Serhiy. It is ok to continue improving the docs in follow-up PRs.

Let's land this, then we can follow up your concerns in a new PR. I appreciate your comments; I just find some of them hard to address.

Thanks @erlend-aasland for the PR 🌮🎉.. I'm working now to backport this PR to: 3.11, 3.12.
🐍🍒⛏🤖 I'm not a witch! I'm not a witch!

GH-111324 is a backport of this pull request to the 3.12 branch.

GH-111325 is a backport of this pull request to the 3.11 branch.