GH-133711: Enable UTF-8 mode by default (PEP 686)

[AA-Turner]

• Issue: Enable UTF-8 mode by default (implement PEP 686) #133711

---

📚 Documentation preview 📚: https://cpython-previews--133712.org.readthedocs.build/

[StanFromIreland]

Can someone please (I don't have permissions to order it) run this buildbot to verify it clears up #133677 for 3.15.

[AA-Turner]

Investigating regular test failures first.

[methane]

test_python_legacy_windows_stdio tests pipe encoding, but it should test console I/O encoding.
cc: @zooba

[methane]

assert_python_ok uses PIPE for stdin/stdout/stderr.
spawn_python doesn't use PIPE for stderr. So this test can be rewritten like this.
But this test still requires the test is running in Console.

diff --git a/Lib/test/test_cmd_line.py b/Lib/test/test_cmd_line.py
index 1b40e0d05fe..243069aeb18 100644
--- a/Lib/test/test_cmd_line.py
+++ b/Lib/test/test_cmd_line.py
@@ -972,10 +972,12 @@ def test_python_legacy_windows_fs_encoding(self):

     @unittest.skipUnless(support.MS_WINDOWS, 'Test only applicable on Windows')
     def test_python_legacy_windows_stdio(self):
-        code = "import sys; print(sys.stdin.encoding, sys.stdout.encoding)"
+        # stdin and stdout are PIPE. So we check stderr encoding for Console I/O.
+        code = "import sys; print(sys.stderr.encoding)"
         expected = 'cp'
-        rc, out, err = assert_python_ok('-c', code, PYTHONLEGACYWINDOWSSTDIO='1')
-        self.assertIn(expected.encode(), out)
+        p = spawn_python('-c', code, env={"PYTHONLEGACYWINDOWSSTDIO": "1"})
+        out, rc = _kill_python_and_exit_code(p)
+        self.assertRegex(rb'\Acp\d+\Z', out.strip())

(I don't test it yet because I don't use Windows daily.)

[vstinner]

Is PEP 538: Coercing the legacy C locale to a UTF-8 based locale still relevant if UTF-8 mode is enabled by default? It may make disabling the UTF-8 mode more complicated. It's just an open question, I don't have the answer.

[methane]

When UTF-8 mode is disabled:

• If locale is not C or POSIX: locale encoding is used.
• If locale is C or POSIX and PYTHONCOERCELOCALE is not set, locale is changed to C.UTF-8.
  • Although UTF-8 mode is disabled, locale encoding is UTF-8.
• If locale is C or POSIX and PYTHONCOERCELOCALE is set, locale encoding will be ASCII.

[zooba]

I feel like we can probably put some more explanation in here, such as that it affects TextIOWrapper and hence open(). The current description doesn't sound as scary as it needs to, in my opinion.

Along the lines of: "Python UTF-8 mode is now enabled by default. This means that (files/console/etc.) will now use UTF-8 regardless of system settings, unless specifically overridden in code (typically with an encoding= argument)."

[zooba]

To be clear, it's nothing new. But we shouldn't assume that everyone already knows what UTF-8 mode implies. There are many more people out there who haven't ever thought about it than those who are waiting for it to be the default.

[vstinner]

Another effect of the UTF-8 Mode is that Python ignores the locale encoding.

[AA-Turner]

I've expanded the What's New entry, please see https://cpython-previews--133712.org.readthedocs.build/en/133712/whatsnew/3.15.html#other-language-changes

[vstinner]

You can add a link to the UTF-8 Mode section (utf8-mode).

[AA-Turner]

I wasn't sure what was best because the main text links to UTF 8 mode (para 3), and I think there have been recent attempts to avoid over-linking. Happy either way though.

[vstinner]

You might mention the UTF-8 Mode earlier since it has other side effects documented in the UTF-8 Mode section, such as changing sys.stdout error handler and ignoring the locale encoding.