tensorflowkerasfmipyfmi

Reinforcement Learning Agent in FMU

I want to train a reinforcement learning agent on a model which i build in OpenModelica. By using pyFMI, it is no problem to import the FMU, simulate it, and get some results.

My problem is that i don´t have a possibility to "pause" a simulation after each step, getting the states, feeding my RL-agent with it and returning his proposed action as an input.

ModelicaGym seems to be a way to solve this problem by starting a simulation, stopping, getting the results, defining the next action and starting the simulation again with the last end-time as starting time.

Reading a paper from Lund University (https://portal.research.lu.se/portal/files/7201641/pyfmi_tech.pdf) made me think about an other idea:

Creating a FMU with the Learner, and connecting the two FMUs via PyFMI.Master.

Something along these lines:

from  pyfmi  import  load_fmu
from  pyfmi.master  import  Master

controller   = load_fmu("controller.fmu")
Circuit = load_fmu("circuit.fmu")

connections = [( Circuit ,"currentSensor1.i",controller ,"feedback1.u2"),
               (controller ,"PID.y",Circuit ,"signalVoltage1.v")]

models = [Circuit , controller]
master_simulator = Master(models , connections)
res = master_simulator.simulate(final_time =1)

Controlling the circuit with an other FMU with a PID controller inside works, but is it possible to create a FMU with a Reinforcement Learning Agent, including all other requiered Libraries, packages (Keras, Tensorflow?)

According to my point of view, such an implementation could have a pretty good performance, especially for models and learners with a higher complexity, this could be an interesting approach.

Or am I just chasing some dreams, because implementing a Reinforcement Learning algorithm in a FMU is not possible or causing other troubles?

Actually, i was a little surprised of not finding other people trying to implement this.

Best regards

Henrik

Solution

  • This answer might be plentifully late, but nevertheless I found your question during my research for the exact same problem. Your question is - to my understanding - taken up in the paper

    Integration and Evaluation of Deep Reinforcement Learning Controller in a Building CoSimulation Environment [PDF found here]

    However in our context, the co-simulation environment that is used for the study of Cyber-Physical Systems is a master of co-simulation, and need to be able to call AI based Python libraries or tools from within the co-simulation. In the co-simulation, the controller requires an interaction with the controlled component at each iteration. Therefore, it is necessary to generate the AI based control components in the form of an FMU as is the case for the other components.

    They used a tool called DACCOSIM NG but later introduced their own methodology to make this approach more streamline.

    Hope you long since found your own solution.