run_algorithm() fails because it cannot fetch data from the internet source (403 Forbidden)

I tried to follow zipline's Quickstart:

from zipline.api import order_target, record, symbol

def initialize(context):
    context.i = 0
    context.asset = symbol('AAPL')


def handle_data(context, data):
    # Skip first 300 days to get full windows
    context.i += 1
    if context.i < 300:
        return

    # Compute averages
    # data.history() has to be called with the same params
    # from above and returns a pandas dataframe.
    short_mavg = data.history(context.asset, 'price', bar_count=100, frequency="1d").mean()
    long_mavg = data.history(context.asset, 'price', bar_count=300, frequency="1d").mean()

    # Trading logic
    if short_mavg > long_mavg:
        # order_target orders as many shares as needed to
        # achieve the desired number of shares.
        order_target(context.asset, 100)
    elif short_mavg < long_mavg:
        order_target(context.asset, 0)

    # Save values for later inspection
    record(AAPL=data.current(context.asset, 'price'),
           short_mavg=short_mavg,
           long_mavg=long_mavg)

But it's failing:

$ zipline run -f dual_moving_average.py --start 2014-1-1 --end 2018-1-1 -o dma.pickle
[2020-01-06 20:31:38.548002] INFO: Loader: Cache at /home/jupyter/.zipline/data/SPY_benchmark.csv does not have data from 2014-01-02 00:00:00+00:00 to 2017-12-29 00:00:00+00:00.

[2020-01-06 20:31:38.548265] INFO: Loader: Downloading benchmark data for 'SPY' from 2013-12-31 00:00:00+00:00 to 2017-12-29 00:00:00+00:00
Traceback (most recent call last):
  File "/home/jupyter/env/bin/zipline", line 8, in <module>
    sys.exit(main())
  File "/home/jupyter/env/lib/python3.5/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/jupyter/env/lib/python3.5/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/jupyter/env/lib/python3.5/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/jupyter/env/lib/python3.5/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/jupyter/env/lib/python3.5/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/jupyter/env/lib/python3.5/site-packages/zipline/__main__.py", line 107, in _
    return f(*args, **kwargs)
  File "/home/jupyter/env/lib/python3.5/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/jupyter/env/lib/python3.5/site-packages/zipline/__main__.py", line 276, in run
    blotter=blotter,
  File "/home/jupyter/env/lib/python3.5/site-packages/zipline/utils/run_algo.py", line 159, in _run
    trading_days=trading_calendar.schedule[start:end].index,
  File "/home/jupyter/env/lib/python3.5/site-packages/zipline/finance/trading.py", line 103, in __init__
    self.bm_symbol,
  File "/home/jupyter/env/lib/python3.5/site-packages/zipline/data/loader.py", line 149, in load_market_data
    environ,
  File "/home/jupyter/env/lib/python3.5/site-packages/zipline/data/loader.py", line 216, in ensure_benchmark_data
    data = get_benchmark_returns(symbol)
  File "/home/jupyter/env/lib/python3.5/site-packages/zipline/data/benchmarks.py", line 35, in get_benchmark_returns
    data = r.json()
  File "/home/jupyter/env/lib/python3.5/site-packages/requests/models.py", line 897, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib/python3.5/json/__init__.py", line 319, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.5/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.5/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I have figured it's because it tries to fetch some data from the api.iextrading.com/1.0/stock/..., but this endpoint currently returns 403 Forbidden and compromises the zipline, i.e.:

requests.get('https://api.iextrading.com/1.0/stock/{}/chart/5y'.format('AAPL'))

I probably can extract the same data from the other internet sources... What format does zipline expect?

Solution

This issue is due to a change in IEX trading api which now requires an API key (account) to query it. I hope that it will be corrected and clarified in the documentation in the future and that retrieving data from external APIs would be much more easier.

For now, you have two options:

1. Change the benchmark code:

/home/jupyter/env/lib/python3.5/site-packages/zipline/data/benchmarks.py

Change the definition of get_benchmark_returns

...
import os 
...
def get_benchmark_returns(symbol):
     The data is provided by IEX (https://iextrading.com/), and we can
     get up to 5 years worth of data.
     """
    IEX_TOKEN = os.environ.setdefault("IEX_PUBLIC_TOKEN", "YOUR_API_KEY") 
    r = requests.get('https://cloud.iexapis.com/stable/stock/{}/chart/5y?token={}'.format(symbol, IEX_TOKEN))

    data = r.json()

    df = pd.DataFrame(data)
    ...

If you wish to amend the IEX_PUBLIC_TOKEN, you can set this env variable outside as what is usually done for Quandl.

This said, the way how the benchmark data is managed can be improved:

It should be similar to the default data ingestion (zipline ingest ...).
It should keep the old data and retrieve only the missing one. You'll see that every day the system will download again all the benchmark data from IEX instead of the new daily data for .SPX.

2. Remove the benchmark by setting it to zero (please check here)

The benchmark comparison of your trading algorithm is important as it is useful for evaluating the algorithm's performance thus I wouldn't recommend setting it to zero.

Please note as well that the default bundle from quandl is not supported anymore (data is limited to April 2018).

Let me know if you face any issue,

Thanks