[SOLVED] How to get both REPL/arrow key functionality and Unicode characters working with Python3 under Git Bash/WinPTY

How to get both REPL/arrow key functionality and Unicode characters working with Python3 under Git Bash/WinPTY

Windows 7, restricted user account running Git git version 2.14.1.windows.1

MINGW/MSYS MINGW64_NT-6.1 REDACTED 2.8.2(0.313/5/3) 2017-07-12 15:35 x86_64 Msys

Python Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:44:40) [MSC v.1600 64 bit (AMD64)] on win32

locale has been set:

$ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_ALL=

Python IO Encoding env variable has been set

$ env | grep "PYTHON"
PYTHONIOENCODING=utf-8

Now for the 'tests':

$ /c/Python34/python.exe -i
Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:44:40) [MSC v.1600 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.stdout.encoding
'utf-8'
>>> print('\u2660')
♠
>>>

Unicode works but the arrow key functionality has been lost in the REPL. So instead, execute using WinPTY

$ winpty /c/Python34/python.exe -i
Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:44:40) [MSC v.1600 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.stdout.encoding
'utf-8'
>>> print('\u2660')
ΓÖá
>>>

Now I can navigate the history in the REPL, but Unicode is wonky.

So seemingly I can either be able to use the arrow keys in the REPL or proper unicode encoding, but not both. Anyone else have this issue or did I miss something here that will make this all work?

Note: I am restricted to this environment; I cannot readily install/update/modify apps at will.

Solution

In the WinPTY case, the console is still using cp437 (default US OEM Windows encoding) but you've force Python to output UTF-8 via PYTHONIOENCODING. Switch to Python 3.6 since that version now uses the Windows Unicode console APIs to output Unicode and shouldn't have issues with encodings.