Optimal datafile format loading on a game console

I need to load large models and other structured binary data on an older CD-based game console as efficiently as possible. What's the best way to do it? The data will be exported from a Python application. This is a pretty elaborate hobby project.

Requierements:

• no reliance on fully standard compliant STL - i might use uSTL though.
• as little overhead as possible. Aim for a solution so good. that it could be used on the original Playstation, and yet as modern and elegant as possible.
• no backward/forward compatibility necessary.
• no copying of large chunks around - preferably files get loaded into RAM in background, and all large chunks accessed directly from there later.
• should not rely on the target having the same endianness and alignment, i.e. a C plugin in Python which dumps its structs to disc would not be a very good idea.
• should allow to move the loaded data around, as with individual files 1/3 the RAM size, fragmentation might be an issue. No MMU to abuse.
• robustness is a great bonus, as my attention span is very short, i.e. i'd change saving part of the code and forget the loading one or vice versa, so at least a dumb safeguard would be nice.
• exchangeability between loaded data and runtime-generated data without runtime overhead and without severe memory management issues would be a nice bonus.

I kind of have a semi-plan of parsing in Python trivial, limited-syntax C headers which would use structs with offsets instead of pointers, and convenience wrapper structs/classes in the main app with getters which would convert offsets to properly typed pointers/references, but i'd like to hear your suggestions.

Clarification: the request is primarily about data loading framework and memory management issues.

This is a common game development pattern.

The usual approach is to cook the data in an offline pre-process step. The resulting blobs can be streamed in with minimal overhead. The blobs are platform dependent and should contain the proper alignment & endian-ness of the target platform.

At runtime, you can simply cast a pointer to the in-memory blob file. You can deal with nested structures as well. If you keep a table of contents with offsets to all the pointer values within the blob, you can then fix-up the pointers to point to the proper address. This is similar to how dll loading works.

I've been working on a ruby library, bbq, that I use to cook data for my iphone game.

Here's the memory layout I use for the blob header:

// Memory layout
//
// p begining of file in memory.
// p + 0 : num_pointers
// p + 4 : offset 0
// p + 8 : offset 1
// ...
// p + ((num_pointers - 1) * 4) : offset n-1
// p + (num_pointers * 4) : num_pointers   // again so we can figure out 
//                                            what memory to free.
// p + ((num_pointers + 1) * 4) : start of cooked data
//

Here's how I load binary blob file and fix up pointers:

void* bbq_load(const char* filename)
{
    unsigned char* p;
    int size = LoadFileToMemory(filename, &p);
    if(size <= 0)
        return 0;

    // get the start of the pointer table
    unsigned int* ptr_table = (unsigned int*)p;
    unsigned int num_ptrs = *ptr_table;
    ptr_table++;

    // get the start of the actual data
    // the 2 is to skip past both num_pointer values
    unsigned char* base = p + ((num_ptrs + 2) * sizeof(unsigned int));

    // fix up the pointers
    while ((ptr_table + 1) < (unsigned int*)base)
    {
        unsigned int* ptr = (unsigned int*)(base + *ptr_table);
        *ptr = (unsigned int)((unsigned char*)ptr + *ptr);
        ptr_table++;
    }

    return base;
}

My bbq library isn't quite ready for prime time, but it could give you some ideas on how to write one yourself in python.

Good Luck!