pythonpython-3.xoptimizationline-by-line

Whey does splitting a line of code into (that does many things) into many smaller lines run *so* much slower


This is all in python 3. I have a line of code that essentially does this:

somefloat += someDict(someList(someIndex1)))(someIndex2)

In essence what I have is a list of dictionaries, each dictionary containing multiple entries that point to a list of floats. someIndex1 and someIndex2 are not related, both are integers. someList is a list of strings.

However when I split this into below it ends up much slower:

val1 = someList(someIndex1)
val2 = someDict(val1)
val3 = val2(someIndex2)
somefloat += val3

I've timed the above using kernprof (it's massively nested - over 130,000 calls). Here's the results:

Case 1 - the one liner: enter image description here

Case 2 - the multiliner: enter image description here

So you can see from the above that the one liner is slightly slower than any of the individual single lines, but the one line is almost 4 times faster than the combination of it's parts. I'm surprised by this level of slowdown - what am I missing, is this just the cost of any operation outweighing the cost of the specific operation?


Solution

  • Every time you assign a new value to val1, val2 and val3, it needs to be written somewhere. Then, it needs to be accessed to get what you want. If your lists are deeply nested and you call them 130k+ times, it needs to get the value from the said variable everytime, which can cause performance to slow down.