pythoncsv

Writing to CSV, getting "Error: need to escape" for a blank string

I am probably going to feel very dumb when someone spots what I'm doing wrong here, but I am finding myself unable to defeat what looks like it should be a simple error.

I am writing some data to a CSV with Python. One of the things I want to write is a list of integers. I join the list into a string before writing it to the file:

with open('publishers.csv', 'wb') as f:
    writer = csv.writer(f, quoting=csv.QUOTE_NONE, delimiter='|', quotechar='')
    for item in big_list_of_objects:
        description = item.description
        number_list = item.number_list
        formatted_numbers = "-".join(number_list)
        writer.writerow([
            description,
            formatted_numbers
            ])

number_list may have anywhere from zero to a whole bunch of numbers in it. If it's an empty list, the join just sets formatted_numbers equal to a blank string. If it's not an empty list, I get a string made of up integers connected by hyphens.

number_list = [1,2,34,12]
formatted_numbers = '1-2-34-12'

number_list = []
formatted_numbers = ''

That's the idea, anyway. In reality, what happens is the first five rows write successfully then I get:

File "<console>", line 1, in <module>
  File "/path/path/path.py", line 500, in offending_function
    formatted_numbers
Error: need to escape, but no escapechar set

Now in this particular situation, the first five rows that write successfully have an empty number_list. The row that consistently crashes also has an empty number_list. There is nothing weird about the value being written immediately before or after number_list on this row. And there is nothing weird about the formatted_numbers being written when this error crops up - I tossed in a print statement to debug, and it's just an empty string like the five before it.

Can anyone help me figure out where I might be going wrong here?

Edit: I have added these print statements:

with open('publishers.csv', 'wb') as f:
    writer = csv.writer(f, quoting=csv.QUOTE_NONE, delimiter='|', quotechar='')
    for item in big_list_of_objects:
        description = item.description
        print "Description for %r is %r" % (item,description)
        number_list = item.number_list
        print "Now formatting %r for %r" % (number_list,item)
        formatted_numbers = "-".join(number_list)
        print repr(formatted_numbers)
        writer.writerow([
            description,
            formatted_numbers
            ])

The result:

Description for 'p89' is u''
Now formatting '' for 'p89'
''
Description for 'p88' is u''
Now formatting '' for 'p88'
''
Description for 'p83' is u''
Now formatting '' for 'p83'
''
Description for 'p82' is u'in-tr-t91411'
Now formatting '' for 'p82'
''
Description for 'p81' is u''
Now formatting '' for 'p81'
''
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/path/path/path.py", line 501, in offending_function
    formatted_numbers
Error: need to escape, but no escapechar set

p81 is not written to the CSV - this is where the crash occurs. However, as you can see, print repr(formatted_numbers) reveals it to be a blank string identical to those before it. There is no description for item p81 (just a blank string), but there is a description for the item preceding it.

Solution

  • The issue is most probably occuring because your description has | in it, which is the delimiter for your csv as well. Hence, csv is trying to escape it, but cannot since no csv.escapechars are set. Example to show same issue in my computer -

    >>> description = 'asda|sd'
>>> formatted_numbers = ''
>>> with open('a.csv','w') as f:
...     writer = csv.writer(f, quoting=csv.QUOTE_NONE, delimiter='|', quotechar='')
...     writer.writerow([
...             description,
...             formatted_numbers
...             ])
...
Traceback (most recent call last):
  File "<stdin>", line 5, in <module>
_csv.Error: need to escape, but no escapechar set

    One fix would be to provide an escapechar so that it can be escaped. Example -

    writer = csv.writer(f, quoting=csv.QUOTE_NONE, delimiter='|', quotechar='',escapechar='\\')    #Or any other appropriate escapechar

    Or another fix would be to remove the | in the description before trying to write it, if you do not really need it in the description field -

    description = description.replace('|','')

    Or you can quote all the fields , by using csv.QUOTE_ALL instead of csv.QUOTE_NONE as provide a valid quotechar .