Backtrader problems in resampling to daily

Using backtrader, I would like to retrieve quotes at a five minute scale and resample at a daily scale.

I am able to resample five minutes to sixty minutes, but can't do daily. Here is the code:

from __future__ import absolute_import, division, print_function, unicode_literals
import backtrader as bt
import backtrader.stores.ibstore as ibstore
import datetime
import logging
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)

_CLIENTID = 100

class St(bt.Strategy):
    def __init__(self):
        self.sma = bt.indicators.SMA(self.data)

    def logdata(self):
        txt = []
        txt.append('{}'.format(len(self)))
        txt.append('{}'.format(self.data.datetime.datetime(0).isoformat()))
        txt.append('{:.2f}'.format(self.data.open[0]))
        txt.append('{:.2f}'.format(self.data.high[0]))
        txt.append('{:.2f}'.format(self.data.low[0]))
        txt.append('{:.2f}'.format(self.data.close[0]))
        txt.append('{:.2f}'.format(self.data.volume[0]))
        logger.debug(','.join(txt))

    data_live = False

    def notify_data(self, data, status, *args, **kwargs):
        print('*' * 5, 'DATA Notification:', data._getstatusname(status), *args)
        if status == data.LIVE:
            self.data_live = True

    def next(self):
        self.logdata()
        if not self.data_live:
            return

_TICKER = "TSLA-STK-SMART-USD"
_FROMDATE = datetime.datetime(2021,1,4)
_TODATE = datetime.datetime(2021,1,29)
_HAS_STATS = False

def run(args=None):
    cerebro = bt.Cerebro(stdstats=_HAS_STATS)

    store = ibstore.IBStore(host="127.0.0.1", port=7497, clientId= _CLIENTID )
    cerebro.broker = store.getbroker()
    stockkwargs = dict(
        timeframe=bt.TimeFrame.Minutes,
        compression=10,
        rtbar=False,
        historical=True,
        qcheck=0.5,
        fromdate=_FROMDATE,
        todate=_TODATE,
        latethrough=False,
        tradename=None
    )

    data0 = store.getdata(dataname=_TICKER, **stockkwargs)
    # cerebro.resampledata(data0, timeframe=bt.TimeFrame.Minutes, compression=60)
    cerebro.resampledata(data0, timeframe=bt.TimeFrame.Days, compression=1)

    cerebro.run()

if __name__ == "__main__":
    run()

I get a connection, but no days. Here is the output (with no bars printed):

Server Version: 76
TWS Time at connection:20210303 09:39:17 EST
***** DATA Notification: DELAYED
***** DATA Notification: DISCONNECTED

However, if I go on a 60-minute resample, all is well. The code

cerebro.resampledata(data0, timeframe=bt.TimeFrame.Minutes, compression=60)
# cerebro.resampledata(data0, timeframe=bt.TimeFrame.Days, compression=1)

results in

Server Version: 76
TWS Time at connection:20210303 09:36:27 EST
***** DATA Notification: DELAYED
DEBUG:__main__:30,2021-01-05T17:00:00,749.65,754.40,735.11,753.21,6011.00
DEBUG:__main__:31,2021-01-05T18:00:00,753.39,754.18,751.60,753.34,1126.00
DEBUG:__main__:32,2021-01-05T19:00:00,753.32,753.32,750.49,752.90,1179.00
DEBUG:__main__:33,2021-01-06T04:00:00,748.00,752.55,746.66,751.88,331.00
DEBUG:__main__:34,2021-01-06T05:00:00,751.60,753.00,749.26,750.50,137.00
(omitted)
DEBUG:__main__:286,2021-01-28T17:00:00,833.00,833.25,831.36,831.61,116.00
DEBUG:__main__:287,2021-01-28T18:00:00,831.60,833.96,830.11,831.00,175.00
DEBUG:__main__:288,2021-01-28T19:00:00,830.51,832.00,829.45,829.45,358.00
***** DATA Notification: DISCONNECTED

I am using these versions:

Python 3.7.4
ib                   0.8.0
IbPy2                0.8.0
numpy                1.19.2
pandas               1.1.3

TL;DR I did several other experiments.

You are right there was a known issue a few years a back with this. There is an easy work-around though. Instead of using the backtrader resampling facility, use Interactive Brokers.

You can bypass resample and call the data directly from IB. Just remember that you must add the shortest timeframe first to Backtrader.

for tf_com in [(bt.TimeFrame.Minutes, 1), (bt.TimeFrame.Days, 1)]:
    stockkwargs = dict(
        timeframe=tf_com[0],
        compression=tf_com[1],
        rtbar=False,
        historical=True,
        qcheck=0.5,
        fromdate=_FROMDATE,
        todate=_TODATE,
        latethrough=False,
        tradename=None
    )

    data0 = store.getdata(dataname=_TICKER, **stockkwargs)

    cerebro.adddata(data0)

You can adjust your timeframe and compression in the tuples in the first line. I will adjust it to 5 and 15 minutes so you can see the output.

for tf_com in [(bt.TimeFrame.Minutes, 5), (bt.TimeFrame.Minutes, 15)]:

************ OUTPUT ************

2021-03-25 12:00:00 636.77 632.71 
2021-03-25 12:05:00 634.69 632.71 
2021-03-25 12:10:00 632.71 632.71 
2021-03-25 12:15:00 640.39 643.00 
2021-03-25 12:20:00 640.68 643.00 
2021-03-25 12:25:00 643.00 643.00 
2021-03-25 12:30:00 641.33 640.00 
2021-03-25 12:35:00 637.84 640.00 
2021-03-25 12:40:00 640.00 640.00 
2021-03-25 12:45:00 640.34 633.95 
2021-03-25 12:50:00 636.15 633.95 
2021-03-25 12:55:00 633.95 633.95 
2021-03-25 13:00:00 633.50 637.43 
2021-03-25 13:05:00 634.95 637.43 
2021-03-25 13:10:00 637.43 637.43 

EDIT: IN RESPONSE TO OP COMMENT

I beg to differ. The days are sampled by Interactive Brokers so you will have daily information. You must be sure when using 5 minute data at the same time not to go back too far with your start date. IB is not a historical data provider.

Here is the entire code for your reference.

import backtrader as bt
import backtrader.stores.ibstore as ibstore
import datetime
import os
from dotenv import load_dotenv

load_dotenv()


class St(bt.Strategy):

    def __init__(self):
        self.data_live = False
        self.timeframes = {4: "minute", 5: "day"}


    def next(self):
        time = self.data.datetime.time()
        start_day = datetime.time(4, 20, 0)
        end_day = datetime.time(19, 40, 0)
        if time < start_day or time > end_day:
            print_date = True
        else:
            print_date = False

        if not self.data_live and print_date:
            print(
                f"{self.data.datetime.datetime()}   "
                
                f"data0: tf: {self.timeframes[self.datas[0]._timeframe]} "
                f"comp: {self.datas[0]._compression},   "
                f"{self.datas[0].close[0]:5.2f} "
                
                f"data1: tf: {self.timeframes[self.datas[1]._timeframe]} "
                f"comp: {self.datas[1]._compression} "
                f"{self.datas[1].close[0]:5.2f} "
            )
            return


_TICKER = "TSLA-STK-SMART-USD"
_FROMDATE = datetime.datetime(2021, 3, 10)
_TODATE = datetime.datetime(2021, 3, 24)
_HAS_STATS = False
_CLIENTID = os.getenv("CLIENTID")
_PORT = os.getenv("SOCKET_PORT")


def run():
    cerebro = bt.Cerebro(stdstats=_HAS_STATS)
    cerebro.addstrategy(St)

    store = ibstore.IBStore(host="127.0.0.1", port=int(_PORT), clientId=_CLIENTID)
    cerebro.broker = store.getbroker()

    for tf_com in [(bt.TimeFrame.Minutes, 5), (bt.TimeFrame.Days, 1)]:
        stockkwargs = dict(
            timeframe=tf_com[0],
            compression=tf_com[1],
            rtbar=False,
            historical=True,
            qcheck=0.5,
            fromdate=_FROMDATE,
            todate=_TODATE,
            latethrough=False,
            tradename=None,
        )

        data = store.getdata(dataname=_TICKER, **stockkwargs)

        cerebro.adddata(data)

    cerebro.run()


if __name__ == "__main__":
    run()

Here is the output with the middle of the days removed.

Please note that backtrader will set the next day value at the end of the last bar of the previous day.

2021-03-10 19:55:00   data0: tf: minute comp: 5,   664.56 data1: tf: day comp: 1 664.56 
2021-03-11 04:00:00   data0: tf: minute comp: 5,   699.10 data1: tf: day comp: 1 664.56 
2021-03-11 04:05:00   data0: tf: minute comp: 5,   701.50 data1: tf: day comp: 1 664.56 
2021-03-11 04:10:00   data0: tf: minute comp: 5,   699.00 data1: tf: day comp: 1 664.56 
2021-03-11 04:15:00   data0: tf: minute comp: 5,   701.00 data1: tf: day comp: 1 664.56 
2021-03-11 19:45:00   data0: tf: minute comp: 5,   698.05 data1: tf: day comp: 1 664.56 
2021-03-11 19:50:00   data0: tf: minute comp: 5,   698.09 data1: tf: day comp: 1 664.56 
2021-03-11 19:55:00   data0: tf: minute comp: 5,   698.50 data1: tf: day comp: 1 664.56 
2021-03-11 19:55:00   data0: tf: minute comp: 5,   698.50 data1: tf: day comp: 1 698.50 
2021-03-12 04:00:00   data0: tf: minute comp: 5,   674.25 data1: tf: day comp: 1 698.50 
2021-03-12 04:05:00   data0: tf: minute comp: 5,   676.00 data1: tf: day comp: 1 698.50 
2021-03-12 04:10:00   data0: tf: minute comp: 5,   669.00 data1: tf: day comp: 1 698.50 
2021-03-12 04:15:00   data0: tf: minute comp: 5,   669.46 data1: tf: day comp: 1 698.50 
2021-03-12 19:45:00   data0: tf: minute comp: 5,   693.30 data1: tf: day comp: 1 698.50 
2021-03-12 19:50:00   data0: tf: minute comp: 5,   692.80 data1: tf: day comp: 1 698.50 
2021-03-12 19:55:00   data0: tf: minute comp: 5,   692.99 data1: tf: day comp: 1 698.50 
2021-03-12 19:55:00   data0: tf: minute comp: 5,   692.99 data1: tf: day comp: 1 692.99 
2021-03-15 04:00:00   data0: tf: minute comp: 5,   689.00 data1: tf: day comp: 1 692.99 
2021-03-15 04:05:00   data0: tf: minute comp: 5,   692.13 data1: tf: day comp: 1 692.99 
2021-03-15 04:10:00   data0: tf: minute comp: 5,   692.12 data1: tf: day comp: 1 692.99 
2021-03-15 04:15:00   data0: tf: minute comp: 5,   692.31 data1: tf: day comp: 1 692.99 
2021-03-15 19:45:00   data0: tf: minute comp: 5,   702.00 data1: tf: day comp: 1 692.99 
2021-03-15 19:50:00   data0: tf: minute comp: 5,   702.00 data1: tf: day comp: 1 692.99 
2021-03-15 19:55:00   data0: tf: minute comp: 5,   702.00 data1: tf: day comp: 1 692.99 
2021-03-15 19:55:00   data0: tf: minute comp: 5,   702.00 data1: tf: day comp: 1 702.00 
2021-03-16 04:00:00   data0: tf: minute comp: 5,   704.01 data1: tf: day comp: 1 702.00 
2021-03-16 04:05:00   data0: tf: minute comp: 5,   704.98 data1: tf: day comp: 1 702.00 
2021-03-16 04:10:00   data0: tf: minute comp: 5,   704.99 data1: tf: day comp: 1 702.00 
2021-03-16 04:15:00   data0: tf: minute comp: 5,   706.20 data1: tf: day comp: 1 702.00 
2021-03-16 19:45:00   data0: tf: minute comp: 5,   673.65 data1: tf: day comp: 1 702.00 
2021-03-16 19:50:00   data0: tf: minute comp: 5,   674.00 data1: tf: day comp: 1 702.00 
2021-03-16 19:55:00   data0: tf: minute comp: 5,   674.10 data1: tf: day comp: 1 702.00 
2021-03-16 19:55:00   data0: tf: minute comp: 5,   674.10 data1: tf: day comp: 1 674.10 
2021-03-17 04:00:00   data0: tf: minute comp: 5,   672.00 data1: tf: day comp: 1 674.10 
2021-03-17 04:05:00   data0: tf: minute comp: 5,   675.80 data1: tf: day comp: 1 674.10 
2021-03-17 04:10:00   data0: tf: minute comp: 5,   677.19 data1: tf: day comp: 1 674.10 
2021-03-17 04:15:00   data0: tf: minute comp: 5,   676.03 data1: tf: day comp: 1 674.10 
2021-03-17 19:45:00   data0: tf: minute comp: 5,   699.66 data1: tf: day comp: 1 674.10 
2021-03-17 19:50:00   data0: tf: minute comp: 5,   699.90 data1: tf: day comp: 1 674.10 
2021-03-17 19:55:00   data0: tf: minute comp: 5,   699.74 data1: tf: day comp: 1 674.10 
2021-03-17 19:55:00   data0: tf: minute comp: 5,   699.74 data1: tf: day comp: 1 699.74 
2021-03-18 04:00:00   data0: tf: minute comp: 5,   685.00 data1: tf: day comp: 1 699.74 
2021-03-18 04:05:00   data0: tf: minute comp: 5,   686.35 data1: tf: day comp: 1 699.74 
2021-03-18 04:10:00   data0: tf: minute comp: 5,   688.32 data1: tf: day comp: 1 699.74 
2021-03-18 04:15:00   data0: tf: minute comp: 5,   692.50 data1: tf: day comp: 1 699.74 
2021-03-18 19:45:00   data0: tf: minute comp: 5,   652.10 data1: tf: day comp: 1 699.74 
2021-03-18 19:50:00   data0: tf: minute comp: 5,   651.00 data1: tf: day comp: 1 699.74 
2021-03-18 19:55:00   data0: tf: minute comp: 5,   650.56 data1: tf: day comp: 1 699.74 
2021-03-18 19:55:00   data0: tf: minute comp: 5,   650.56 data1: tf: day comp: 1 650.56 
2021-03-19 04:00:00   data0: tf: minute comp: 5,   661.00 data1: tf: day comp: 1 650.56 
2021-03-19 04:05:00   data0: tf: minute comp: 5,   663.00 data1: tf: day comp: 1 650.56 
2021-03-19 04:10:00   data0: tf: minute comp: 5,   663.60 data1: tf: day comp: 1 650.56 
2021-03-19 04:15:00   data0: tf: minute comp: 5,   666.48 data1: tf: day comp: 1 650.56 
2021-03-19 19:45:00   data0: tf: minute comp: 5,   652.50 data1: tf: day comp: 1 650.56 
2021-03-19 19:50:00   data0: tf: minute comp: 5,   652.02 data1: tf: day comp: 1 650.56 
2021-03-19 19:55:00   data0: tf: minute comp: 5,   652.20 data1: tf: day comp: 1 650.56 
2021-03-19 19:55:00   data0: tf: minute comp: 5,   652.20 data1: tf: day comp: 1 652.20 
2021-03-22 04:00:00   data0: tf: minute comp: 5,   665.97 data1: tf: day comp: 1 652.20 
2021-03-22 04:05:00   data0: tf: minute comp: 5,   664.00 data1: tf: day comp: 1 652.20 
2021-03-22 04:10:00   data0: tf: minute comp: 5,   665.00 data1: tf: day comp: 1 652.20 
2021-03-22 04:15:00   data0: tf: minute comp: 5,   663.94 data1: tf: day comp: 1 652.20 
2021-03-22 19:45:00   data0: tf: minute comp: 5,   668.39 data1: tf: day comp: 1 652.20 
2021-03-22 19:50:00   data0: tf: minute comp: 5,   668.56 data1: tf: day comp: 1 652.20 
2021-03-22 19:55:00   data0: tf: minute comp: 5,   669.35 data1: tf: day comp: 1 652.20 
2021-03-22 19:55:00   data0: tf: minute comp: 5,   669.35 data1: tf: day comp: 1 669.35 
2021-03-23 04:00:00   data0: tf: minute comp: 5,   670.25 data1: tf: day comp: 1 669.35 
2021-03-23 04:05:00   data0: tf: minute comp: 5,   665.33 data1: tf: day comp: 1 669.35 
2021-03-23 04:10:00   data0: tf: minute comp: 5,   664.11 data1: tf: day comp: 1 669.35 
2021-03-23 04:15:00   data0: tf: minute comp: 5,   662.93 data1: tf: day comp: 1 669.35 
2021-03-23 19:45:00   data0: tf: minute comp: 5,   661.00 data1: tf: day comp: 1 669.35 
2021-03-23 19:50:00   data0: tf: minute comp: 5,   661.08 data1: tf: day comp: 1 669.35 
2021-03-23 19:55:00   data0: tf: minute comp: 5,   663.00 data1: tf: day comp: 1 669.35 
2021-03-23 19:55:00   data0: tf: minute comp: 5,   663.00 data1: tf: day comp: 1 663.00 

=== OP Verification ===

Here is what the data0 and data1 plot look like, adding a cerebro.plot() after the cerebro.run().

That's good looking daily data.