What are cost saving alternatives to Oracle Tuxedo middleware platform to whom is possible to migrate existing code base?
There are few open source alternatives like RedHat Narayana BlackTie module, and Mavimax Enduro/X. I have check the source code for both and it seems that Enduro/X is most complete, it has FML and VIEW buffers, which usually are preferred "protocol buffer" for Tuxedo apps.
BlackTie have it' own buffers like NBF, but their API is quite different from FML, thus migration might be painful.
Also BlackTie comes with Java integration, so some might benefit from that like mixing new Java apps with "Old" C. But this mean you have to boot the whole JBoss instance for your migrated app. On the other hand Enduro/X has it's own C application server, which seems to be with quite small footprint.
Regarding the licensing:
the BlackTie will comes with LGPL 2.1 license, so as long you do not change the RedHat's code you can use it for closed source/open source solutions.
Enduro/X uses GPL 2.0, which means that for in-house developments or open source projects you may use it with out acquiring the license, but if you distributed closed source solution, you need to acquire the license. However by looking on their license page, there are plenty of options, the closest license (per operating system instance vs Tuxedo's per CPU) is almost 10 times cheaper than mentioned in Oracle Global Price list.
There are other, non open source competitors, but I cannot find any pricing infos in public, thus you may need to ask direclty them:
There are some attempted XATMI implementation on Github (try to search "tpcall" or "tpadvertise" for example), but those seems to be very limited implementation.