I currently have an ExplicitComponent that takes as an input altitude and computes atmospheric properties at that altitude. The compute function itself uses another Python library to do this. When the library calculates the atmospheric properties, it loads in a large data file to do it. As such, I have put this loading into the setup() function. To clarify, this data is not read in as an array, it's loaded into the cache to then be used by the library.
However, it seems that the trajectory.simulate method runs the setup() function for every segment in the trajectory, and so the data still gets loaded multiple times and this slows down and sometimes crashes my computation if I have over a certain number of segments.
Is there a way to have this data read in once for the entire simulation to use without doing so outside of the component? I want to do it inside the component as there are some options I have set that influence what data specifically gets loaded (e.g. the time of year and such).
Attempts: I have tried to put the loading data into an __init__ call where the options can be read from there, but the entire component gets put into its own separate problem for each segment of the simulate phase it seems, and this still results in the data being read in multiple times.
I have provided some sudo code below. Any help would be greatly appreciated!
class AtmosphereCalculator(om.ExplicitComponent):
def initialize(self):
*** define some options ***
def setup(self):
*** load data based on the options ***
*** define inputs as altitude and outputs as atmospheric properties ***
def compute(self, inputs, outputs):
*** calculate properties with the data ***
Dymos's simulation does in fact make many many separate problem instances, which themselves then instantiate and setup individual models. That is why you're seeing setup called so many times.
What you need to do is load the data one time outside the scope of the component class, then have all instances of your class reference that one shared data object.
Your pseudo code indicates that you may load different data based on options. In the most general case, I could imagine you have multiple instances with different options and data. For example, maybe half your instances have one set of data and the other have another. So you can't just blindly load the data one time during the Class definition. Instead, you need something more reactive.
My suggested solution is to define a separate DataLoader class, and make one instance of it. That class can use an internal cache to keep track of when it needs to load data or not. But since it's separate from the class, you can make as many instances as you want and it shouldn't reload data you already have.
It should look something like this:
class DataLoader():
def __init__(**kargs):
self._arg_cache = {}
def load(self, **kwargs)
if kwargs in self._arg_cache:
return self._arg_cache[kwargs]
data = *** load data based on the options ***
self._arg_cache[kwargs] = data
return data
# this is intentionally defined outside the scope of the component class
# all instances will share this one object, so they all work from the same cache
data_loader = DataLoader()
class AtmosphereCalculator(om.ExplicitComponent):
def initialize(self):
*** define some options ***
def setup(self):
data = data_loader.load(***kwargs built with options ***)
*** define inputs as altitude and outputs as atmospheric properties ***
def compute(self, inputs, outputs):
*** calculate properties with the data ***