python-2.7dictionarydebuggingnonetypepyalgotrade

Unexpected results when checking all values in a Python dictionary are None


Let's consider the following three dictionaries:

topByClass = {'Real Estate': 'VNO', 'Construction': 'TOL', 'Utilities': 'EXC'}

shouldPass = {'Real Estate': None, 'Construction': None, 'Utilities': 'EXC'}

shouldFail = {'Real Estate': None, 'Construction': None, 'Utilities': None}

I am looking to separate instances where all values in the dictionary are None, from everything else. (i.e. the first two should pass, while the last should fail)

I looked around online, particularly at posts such as this one. I tested various solutions out in the python console (running Python 2.7 in a virtualenv on my Mac), and the following worked:

not all(value == None for value in topByClass.values())

Both with and without "not" I can separate dictionaries like 'topByClass' from 'shouldFail'.

>>> not all(value == None for value in shouldFail.values())
>>> False
>>> not all(value == None for value in topByClass.values())
>>> True

(and vice versa for without not)

The thing is, when I go to run the python file, the if statement always evaluates as if every value is None. I have checked if possibly I am mistaking the dictionary, however I print off the dict "topByClass" in the console, and have directly pasted it above. Any ideas what this could be?

Edit:

def _getTopByClass(self, assetClass):
    # Find the instrument with the highest rank.
    ret = None
    highestRank = None
    for instrument in self.__instrumentsByClass[assetClass]:
        rank = self._getRank(instrument)
        if rank is not None and (highestRank is None or rank > highestRank):
            highestRank = rank
            ret = instrument
    return ret

def _getTop(self):
        ret = {}
        for assetClass in self.__instrumentsByClass:
            ret[assetClass] = self._getTopByClass(assetClass)
        return ret

def _rebalance(self):
    topByClass = self.getTop()
    self.info(topByClass) # where I get the output I presented above
    if any(value is not None for value in topByClass.values()):
        self.info("Not All Empty")
    else:
        self.info("All None")

Now with the above "if" all are printing ("Not All Empty")

If you would like to see getRank() or more, I would reccommend this example from PyAlgoTrade, as the core mechanics affecting the problem are similar.

Edit 2: Thought I might mention this incase someone tried to replicate the file linked above... PyAlgoTrade's modules for downloading feeds doesn't work. So you have to use this package to download the data, as well as adding bars from csv:

feed = yahoofeed.Feed()
feed.addBarsFromCSV("SPY", "data/SPY.csv")
for industry, stocks in instrumentsByClass.items():
    for stock in stocks:
        feed.addBarsFromCSV(stock, "data/"+stock+".csv")

Edit 3: Added some debug info:

self.info(isinstance(topByClass, dict))
self.info(isinstance(topByClass.values(), list))
self.info(isinstance(topByClass.values()[0], str))

returns:

>>> True
>>> True
>>> True (False when the first value is None)

Also, per a comment I thought I'd throw this in

self.info(list(topByClass.values()))
>>> [None, None, None, None]

FINAL EDIT: Many thanks to all the people who responded, thought I would go ahead and post what I figured out incase anyone runs into a similar problem... First of all the code/output that identified the problem:

self.info(list(shouldFail.values())
>>> [None, None, None]
self.info(list(topByClass.values())
>>>['VNO', 'TOL', 'EXC']
self.info(list(value is not None for value in topByClass.values()))
>>> [True, True, True]
self.info(any(value is not None for value in topByClass.values()))
>>> <generator object <genexpr> at 0x116094dc0>

I wasn't sure why it returned a generator, then I realized that it was probably using numpy's any() function, as I decalred:

import numpy as *

After changing this to:

import numpy as np

it behaved as expected.


Solution

  • Since you haven't shown us the actual code that is tickling the fail (I understand that might not be possible in a production environment), here is some philosophy about how to debug in a class hierarchy, and one theory about what might be causing this:

    Anyway, happy logging, and please post us the logger output when you pinpoint the failing comparison. It's important to track down these sort of 'existential' bugs, since they reveal something broken or blind spot in your chain of assumptions, or debugging methodology. That's how you learn.