I have tried detecting encoding of a simple UTF-16-LE text file in Python 3 using chardet package, using following code:
rawdata = open(filename, 'rb').read()
result = chardet.detect(rawdata)
print(result['encoding'], result['confidence'])
The rawdata (contents of the file) are following:
b'N\x00O\x00T\x00 \x00Y\x00O\x00U\x00R\x00 \x00L\x00A\x00N\x00G\x00U\x00A\x00G\x00E\x00?\x00 \x00U\x00S\x00E\x00 \x00h\x00t\x00t\x00p\x00s\x00:\x00/\x00/\x00t\x00r\x00a\x00n\x00s\x00l\x00a\x00t\x00e\x00.\x00g\x00o\x00o\x00g\x00l\x00e\x00.\x00c\x00o\x00m\x00'
The result of the code above is following:
ascii 1.0
Why is chardet 100% sure the text file is in ascii, when it is clearly UTF-16?
This is a chardet issue, still waiting to be fixed: https://github.com/chardet/chardet/pull/109