[SOLVED] Specifying alternative datetime format "Exception: Invalid datetime. It must be bigger than that last one"

Specifying alternative datetime format "Exception: Invalid datetime. It must be bigger than that last one"

I am attempting to read in a csv that has a new data entry every 15 minutes. From what I can gather, the reason I am returning this exception is because the date doesn't change every row, but the time does. However, the feed isn't reading in the time and I'm not sure how to fix that. Here is my code:

from pyalgotrade.feed import csvfeed

feed = csvfeed.Feed("Date","%d/%m/%Y")
feed.addValuesFromCSV("eurusd-15m-july-small.csv")
for dateTime, value in feed:
    print (dateTime, value)

My csv look like this:

Date,Time,Open,High,Low,Close,Volume
07/08/2018,17:30:00,1.15994,1.15994,1.15961,1.15982,414
07/08/2018,17:45:00,1.15982,1.16001,1.15964,1.15996,485

Here is the full error:

(datetime.datetime(2018, 8, 7, 0, 0), {'High': 1.15994, 'Volume': 414.0, 
'Low': 1.15961, 'Time': '17:30:00', 'Close': 1.15982, 'Open': 1.15994})
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
<ipython-input-12-0cbdbe588a05> in <module>()
      3 feed = csvfeed.Feed("Date","%d/%m/%Y")
      4 feed.addValuesFromCSV("eurusd-15m-july-small.csv")
----> 5 for dateTime, value in feed:
      6     print (dateTime, value)

/Users/Phil/anaconda2/lib/python2.7/site- 
packages/pyalgotrade/feed/__init__.pyc in feed_iterator(feed)
     29     try:
     30         while not feed.eof():
---> 31             yield feed.getNextValuesAndUpdateDS()
     32     finally:
     33         feed.stop()

/Users/Phil/anaconda2/lib/python2.7/site- 
packages/pyalgotrade/feed/__init__.pyc in getNextValuesAndUpdateDS(self)
     88                     ds = self.createDataSeries(key, self.__maxLen)
     89                     self.__ds[key] = ds
---> 90                 ds.appendWithDateTime(dateTime, value)
     91         return (dateTime, values)
     92 

/Users/Phil/anaconda2/lib/python2.7/site- 
packages/pyalgotrade/dataseries/__init__.pyc in appendWithDateTime(self, 
dateTime, value)
    134 
    135         if dateTime is not None and len(self.__dateTimes) != 0 and 
self.__dateTimes[-1] >= dateTime:
--> 136             raise Exception("Invalid datetime. It must be bigger than 
that last one")
    137 
    138         assert(len(self.__values) == len(self.__dateTimes))

Exception: Invalid datetime. It must be bigger than that last one

Thanks in advance!

Solution

The problem is that you have the datetime in two separate columns so as @daniel mentions you have to customize parsing. Try this:

import datetime

from pyalgotrade.feed import csvfeed


# Row parser that extracts the datetime combining 2 columns
class RowParser(csvfeed.RowParser):
    def parseRow(self, csvRowDict):
        dateTimeCombined = "%s %s" % (csvRowDict["Date"], csvRowDict["Time"])
        dateTime = datetime.datetime.strptime(dateTimeCombined, "%d/%m/%Y %H:%M:%S")

        # Convert the values
        values = {}
        for key, value in csvRowDict.items():
            if key not in ["Date", "Time"]:
                values[key] = csvfeed.float_or_string(key, value)
        return (dateTime, values)

    def getFieldNames(self):
        return None

    def getDelimiter(self):
        return ","


feed = csvfeed.BaseFeed(RowParser())
feed.addValuesFromCSV("eurusd-15m-july-small.csv")
for dateTime, value in feed:
    print (dateTime, value)