I am struggling to get special characters from an email to display correctly.
I get the message using the Gmail API like this:
msg_id = '169a8fac44fd8115'
service = build('gmail', 'v1', credentials=creds)
message = service.users().messages().get(userId='me', id=msg_id).execute()
htmlpart = message['payload']['parts'][0]['parts'][1]['body']['data']
I've then tried the following:
file_data = quopri.decodestring(base64.urlsafe_b64decode(htmlpart)).decode('iso-8859-1')
file_data = base64.urlsafe_b64decode(htmlpart.encode('UTF-8')).decode('iso-8859-1')
file_data = base64.urlsafe_b64decode(htmlpart.encode('iso-8859-1')).decode('utf-8')
file_data = base64.urlsafe_b64decode(htmlpart.encode('UTF-8')).decode('utf-8')
None of them get me the right output. Instead I get things like €2 instead of €.
For reference, the headers of this message are as follows:
'headers': [{'name': 'Content-Type', 'value': 'text/html; charset="UTF-8"'},
{'name': 'Content-Transfer-Encoding', 'value': 'quoted-printable'}]
Edit: added sample data below. I am trying to get the html of an e-mail, I am copying below just a part of it which highlights the encoding problem (You'll get).
</tr><tr><td class="m_4364729876101169671Uber18_text_p1" align="left" style="color:rgb(0,0,0);font-family:'Uber18-text-Regular','HelveticaNeue-Light','Helvetica Neue Light',Helvetica,Arial,sans-serif;font-size:16px;line-height:28px;direction:ltr;text-align:left"> Give friends free ride credit to try Uber. You'll get CN¥10 off each of your next 3 rides when they start riding. <span class="m_4364729876101169671Uber18_text_p1" style="color:#000000;font-family:'Uber18-text-Regular','HelveticaNeue-Light','Helvetica Neue Light',Helvetica,Arial,sans-serif;font-size:16px;line-height:28px">Share code: 20ccv</span></td>
The headers
'headers': [{'name': 'Content-Type', 'value': 'text/html; charset="UTF-8"'},
{'name': 'Content-Transfer-Encoding', 'value': 'quoted-printable'}]
are telling you that the message consists of text encoded as UTF-8, then quoted-printable encoded so that it can be processed by systems that only support 7-bit characters.
To decode, you need to decode from quoted-printable first, and then decode the resulting bytes from UTF-8.
Something like this ought to work:
utf8 = quopri.decodestring(htmlpart)
text = ut8.decode('utf-8')
HTML email bodies may contain character entities. These can be converted to individual characters using html.unescape (available in Python 3.4+).
>>> import html
>>> h = """</tr><tr><td class="m_4364729876101169671Uber18_text_p1" align="left" style="color:rgb(0,0,0);font-family:'Uber18-text-Regular','HelveticaNeue-Light','Helvetica Neue Light',Helvetica,Arial,sans-serif;font-size:16px;line-height:28px;direction:ltr;text-align:left"> Give friends free ride credit to try Uber. You'll get CN¥10 off each of your next 3 rides when they start riding. <span class="m_4364729876101169671Uber18_text_p1" style="color:#000000;font-family:'Uber18-text-Regular','HelveticaNeue-Light','Helvetica Neue Light',Helvetica,Arial,sans-serif;font-size:16px;line-height:28px">Share code: 20ccv</span></td>"""
>>> print(html.unescape(h))
</tr><tr><td class="m_4364729876101169671Uber18_text_p1" align="left" style="color:rgb(0,0,0);font-family:'Uber18-text-Regular','HelveticaNeue-Light','Helvetica Neue Light',Helvetica,Arial,sans-serif;font-size:16px;line-height:28px;direction:ltr;text-align:left"> Give friends free ride credit to try Uber. You'll get CN¥10 off each of your next 3 rides when they start riding. <span class="m_4364729876101169671Uber18_text_p1" style="color:#000000;font-family:'Uber18-text-Regular','HelveticaNeue-Light','Helvetica Neue Light',Helvetica,Arial,sans-serif;font-size:16px;line-height:28px">Share code: 20ccv</span></td>