In the example im trying to model the complexity of the model increases expontentially with every step taken in the FSM. However the rules for how the FSM grows are easy to describe in code but impossible to model by hand. What makes the model hard to create by hand is the large number of transitions that are possible from each node as the FSM progresses. The number of states are not that many but the number of transitions are huge!
I have been trying to use ModelJUnit but as far I can understand this tool only supports completly pre defined models.
So my question is if there are any Model-Based Testing tools that support dynamically creating the FSM for the system under test as described abowe?
Your problem is very typical. An example would be the controller/software which determines the position of a shifter in a vehicle - the position P R N D need to be determined from electrical transitions of the lever. If you take into account jumping over transitions and electrical faults happening you get a large/infinite space. This kind of software has the highest possible safety level, because it could cause unintended torque.
Tools like ModelJUnit, NUnit etc. are useful for educational/tool developmental purposes. You can study the code, edit it and implement your own experimental algorithms.
On the other hand unmodified they can't cope with the problem you mention : state space/transition explosion etc. That's why professional tools use very complex methods to handle large/infinite spaces:
They can often produce results like requirement coverage, mc/dc even on infinite models, because the algorithms don't need to expand the full space during exploration. Examples for tools you can look for are: