pythonhtmlweb-scrapingunicodelxml

Problems with encoding while parsing html document with lxml

I am trying to get clean text from some web pages. I have read a lot of tutorials and finally ended up with python lxml + beautifulsoup + requests modules . The reason for using lxml for such a task is that it cleans html files better than beautiful soup do.

I ended up with test script like this:

from bs4 import UnicodeDammit                                                    
import re                                                                        
import requests                                                                  
import lxml                                                                      
import lxml.html                                                                 
from time import sleep                                                           

urls = [                                                                         
    "http://mathprofi.ru/zadachi_po_kombinatorike_primery_reshenij.html",        
    "http://ru.onlinemschool.com/math/assistance/statistician/",                 
    "http://mathprofi.ru/zadachi_po_kombinatorike_primery_reshenij.html",        
    "http://universarium.org/courses/info/332",                                  
    "http://compsciclub.ru/course/wordscombinatorics",                           
    "http://ru.onlinemschool.com/math/assistance/statistician/",                 
    "http://lectoriy.mipt.ru/course/Maths-Combinatorics-AMR-Lects/",             
    "http://www.youtube.com/watch?v=SLPrGWQBX0I"                                 
]                                                                                


def check(url):                                                                     
    print "That is url {}".format(url)                                           
    r = requests.get(url)                                                        
    ud = UnicodeDammit(r.content, is_html=True)                                  
    content = ud.unicode_markup.encode(ud.original_encoding, "ignore")           
    root = lxml.html.fromstring(content)                                            
    lxml.html.etree.strip_elements(root, lxml.etree.Comment,                        
                                   "script", "style")                               
    text = lxml.html.tostring(root, method="text", encoding=unicode)                
    text = re.sub('\s+', ' ', text)                                                 
    print "Text type is {}!".format(type(text))                                     
    print text[:200]                                                                
    sleep(1)                                                                        


if __name__ == '__main__':                                                          
    for url in urls:                                                                
        check(url)

Inetrmediate de- and reencoding to the original encoding is needed because the html page could contain some characters that are encoded differently from the most others. Such an occasion breaks further lxml tostring method.

However my code is not working properly with all tests. Sometimes (especially with last two urls) it outputs the mess: 

...
That is url http://ru.onlinemschool.com/math/assistance/statistician/
Text type is <type 'unicode'>!
 Онлайн решение задач по математике. Комбинаторика. Теория вероятности. Close Авторизация на сайте Введите логин: Введите пароль: Запомнить меня Регистрация Изучение математики онлайн.Изучайте математ
That is url http://lectoriy.mipt.ru/course/Maths-Combinatorics-AMR-Lects/
Text type is <type 'unicode'>!
 ÐаÑемаÑика. ÐÑÐ½Ð¾Ð²Ñ ÐºÐ¾Ð¼Ð±Ð¸Ð½Ð°ÑоÑики и ÑеоÑии ÑиÑел / ÐидеолекÑии ФизÑеÑа: ÐекÑоÑий ÐФТР- видеолекÑии по Ñизике,
That is url http://www.youtube.com/watch?v=SLPrGWQBX0I
Text type is <type 'unicode'>!
 ÐÑновнÑе ÑоÑмÑÐ»Ñ ÐºÐ¾Ð¼Ð±Ð¸Ð½Ð°ÑоÑики - bezbotvy - YouTube ÐÑопÑÑÑиÑÑ RU ÐобавиÑÑ Ð²Ð¸Ð´ÐµÐ¾ÐойÑиÐоиÑк ÐагÑÑзка... ÐÑбеÑиÑе ÑзÑк.

This mess is somehow connected with encoding ISO-8859-1, but i cannot find out how. For each of two last urls i get: 

In [319]: r = requests.get(urls[-1])

In [320]: chardet.detect(r.content)
Out[320]: {'confidence': 0.99, 'encoding': 'utf-8'}

In [321]: UnicodeDammit(r.content, is_html=True).original_encoding
Out[321]: 'utf-8'

In [322]: r = requests.get(urls[-2])

In [323]: chardet.detect(r.content)
Out[323]: {'confidence': 0.99, 'encoding': 'utf-8'}

In [324]: UnicodeDammit(r.content, is_html=True).original_encoding
Out[324]: u'utf-8'

So i guess lxml makes internal decoding based on the wrong assumptions of an input string. I think that it don't even try to make guess about input string encoding. It seems that in the core of lxml happens something like this :

In [339]: print unicode_string.encode('utf-8').decode("ISO-8859-1", "ignore")
ÑÑÑока

How could i resolve my issue and clean all urls from html tags ? Maybe i should use another python modules or do it another way ? Please, give me your suggestions.

Solution

  • I finally figured it out. The solution is not to use 

    root = lxml.html.fromstring(content)

    but configure an explicit Parser object which can be told to use particular encoding enc:

    htmlparser = etree.HTMLParser(encoding=enc)
root = etree.HTML(content, parser=htmlparser)

    I found, additionally, that even UnicodeDammit make obvious mistakes when deciding about encoding of a page. So i have added another if block:

    if (declared_enc and enc != declared_enc):

    Here is the snippet of a result:

    from lxml import html
from lxml.html import etree
import requests
from bs4 import UnicodeDammit
import chardet 


try:
    self.log.debug("Try to get content from page {}".format(url))
    r = requests.get(url)
except requests.exceptions.RequestException as e:
    self.log.warn("Unable to get page content of the url: {url}. "
                  "The reason: {exc!r}".format(url=url, exc=e))
    raise ParsingError(e.message)

ud = UnicodeDammit(r.content, is_html=True)

enc = ud.original_encoding.lower()
declared_enc = ud.declared_html_encoding
if declared_enc:
    declared_enc = declared_enc.lower()
# possible misregocnition of an encoding
if (declared_enc and enc != declared_enc):
    detect_dict = chardet.detect(r.content)
    det_conf = detect_dict["confidence"]
    det_enc = detect_dict["encoding"].lower()
    if enc == det_enc and det_conf < THRESHOLD_OF_CHARDETECT:
        enc = declared_enc
# if page contains any characters that differ from the main
# encodin we will ignore them
content = r.content.decode(enc, "ignore").encode(enc)
htmlparser = etree.HTMLParser(encoding=enc)
root = etree.HTML(content, parser=htmlparser)
etree.strip_elements(root, html.etree.Comment, "script", "style")
text = html.tostring(root, method="text", encoding=unicode)