pythonunicodereplacecharacterturkish

How can I replace Unicode characters with Turkish characters in a text file with Python

I am working on Twitter. I got data from Twitter with Stream API and the result of app is JSON file. I wrote tweets data in a text file and now I see Unicode characters instead of Turkish characters. I don't want to do find/replace in Notepad++ by hand. Is there any automatic option to replace characters by opening txt file, reading all data in file and changing Unicode characters with Turkish characters by Python?

Here are Unicode characters and Turkish characters which I want to replace.

I tried two different type

#!/usr/bin/env python

# -*- coding: utf-8 -*- 

import re

dosya = open('veri.txt', 'r')

for line in dosya:
    match = re.search(line, "\u011f")
    if (match):
        replace("\u011f", "ğ")

dosya.close()

and:

#!/usr/bin/env python

# -*- coding: utf-8 -*- 

f1 = open('veri.txt', 'r')
f2 = open('veri2.txt', 'w')

for line in f1:
    f2.write=(line.replace('\u011f', 'ğ')) 
    f2.write=(line.replace('\u011e', 'Ğ'))
    f2.write=(line.replace('\u0131', 'ı'))
    f2.write=(line.replace('\u0130', 'İ'))
    f2.write=(line.replace('\u00f6', 'ö'))
    f2.write=(line.replace('\u00d6', 'Ö'))
    f2.write=(line.replace('\u00fc', 'ü'))
    f2.write=(line.replace('\u00dc', 'Ü'))
    f2.write=(line.replace('\u015f', 'ş'))
    f2.write=(line.replace('\u015e', 'Ş'))
    f2.write=(line.replace('\u00e7', 'ç'))
    f2.write=(line.replace('\u00c7', 'Ç'))

f1.close()
f2.close()

Both of these didn't work. How can I make it work?

Solution

  • JSON allows both "escaped" and "unescaped" characters. The reason why the Twitter API returns only escaped characters is that it can use the ASCII encoding, which increases interoperability. For Turkish characters you need another encoding. Opening a file with the open function opens a file assuming your current locale encoding, which is probably what your editor expects. If you want the output file to have e.g. the ISO-8859-9 encoding, you can pass encoding='ISO-8859-9' as an additional parameter to the open function.

    You can read a file containing a JSON object with the json.load function. This returns a Python object with the escaped characters decoded. Writing it again with json.dump and passing ensure_ascii=False as an argument writes the object back to a file without encoding Turkish characters as escape sequences. An example:

    import json
inp = open('input.txt', 'r')
out = open('output.txt', 'w')
in_as_obj = json.load(inp)
json.dump(in_as_obj, out, ensure_ascii=False)

    Your file isn't really a JSON file, but instead a file containing multiple JSON objects. If each JSON object is on its own line, you can try the following:

    import json
inp = open('input.txt', 'r')
out = open('output.txt', 'w')
for line in inp:
    if not line.strip():
        out.write(line)
        continue
    in_as_obj = json.loads(line)
    json.dump(in_as_obj, out, ensure_ascii=False)
    out.write('\n')

    But in your case it's probably better to write unescaped JSON to the file in the first place. Try replacing your on_data method by (untested):

    def on_data(self, raw_data):
    data = json.loads(raw_data)
    print(json.dumps(data, ensure_ascii=False))