python-3.xloggingencodingcharacter-encodingerror-logging

python 3.6 *logging modul error* UnicodeEncodeError: 'charmap' codec can't encode characters


On Windows 10 the logging module send this error (using scrapy)

# --- Logging error ---
...
# UnicodeEncodeError: 'charmap' codec can't encode characters in position 175-176: character maps to <undefined>

I have read that I should add encoding='utf-8' but I did not find how I could add it in the code below. EDIT: According to the tuto it isnt needed.

configure_logging(install_root_handler=False) #override default log settings
logging.basicConfig(
    filename='logfile.log',
    format='%(levelname)s: %(message)s',
    datefmt='%m-%d %H:%M',
    level=logging.INFO #CRITICAL ERROR WARNING  INFO    DEBUG    NOTSET 
)

I found many questions on the topics, but mostly on python 2 (or not related to the logging module). And the logging tutorial don't even talk about utf-8. (Notice that I can print UTF8 characters without any problem. The problem only occur with the logging module)


Solution

  • I'm not using Scrapy but I ran into the same issue with vanilla Python 3.6 logging of not being able to write UTF-8 characters via logging to a file (console worked just fine).

    Based on this comment I added 'utf-8' to my FileHandler initialization. I'm configuring logging via an INI file so it looks like this:

    [handler_file]
    class = FileHandler
    args = (r'log.txt', 'a', 'utf-8')
    level = NOTSET
    formatter = generic
    

    To use logging.basicConfig() I think that you should be able to do the following:

    configure_logging(install_root_handler=False) #override default log settings
    logging.basicConfig(
        handlers=[logging.FileHandler('logfile.log', 'w', 'utf-8')],
        format='%(levelname)s: %(message)s',
        datefmt='%m-%d %H:%M',
        level=logging.INFO #CRITICAL ERROR WARNING  INFO    DEBUG    NOTSET 
    )