c++sgx

cxxrt::bad_alloc despite large EPC


I am running the following piece of code inside an SGX enclave:

void test_enclave_size() {
    unsigned int i = 0;
    const unsigned int MB = 1024 * 1024; 
    try {
        for (; i < 10000; i++) {
            char* tmp = new char[MB];
        }
    } catch (const std::exception &e) { 
        std::cout << "Crash with " << e.what() << " " << i << std::endl;
    }
}

On my dev machine with the standard 128 MB EPC this throws a bad cxxrt::bad_alloc after 118 MB, which makes sense because I believe only 96 MB is guaranteed to be available for enclave programs. However, when running this code on a Standard_DC32s_V3, which has 192 GB of EPC memory, I get the exact same result. I assumed that because the EPC is advertised to be extremely large, I should be able to allocate far more than 128 MB.

I have thought of a couple of reasons why this might be happening:

  1. While the EPC is now 192 GB in size, each process is still limited to 128 MB.
  2. There is something in the kernel that needs to be enabled for me to take advantage of this large EPC.
  3. I am misunderstanding what Azure is advertising.

I wanted to see if anyone has a good idea of what is happening before contact Azure support, since this might be a user error.

Edit: It turns out my second reason was the closest. As X99 pointed out, when developing an enclave application there is a configuration file that defines several factors such as the number of thread contexts, whether debugging is enabled, and max heap/stack size. My maximum heap size in my configuration was set to about 118 MB, which explains why I started to get bad allocations past this amount. Once I increased the number, the issue went away. Size note: if you are on Linux, the drivers support paging. This means you can use as much memory as you wish if you can afford to suffer the paging overhead.

If you are using Open Enclave as your SDK, this configuration file (example) is what you should be editing. In this example, the maximum heap and stack are 1024 pages, which is about 4MB. This page may be useful to you as well!


Solution

  • If the machine you're running your enclave on has more than 128Mb of EPC AND will allow you to go further (because of a BIOS setting), there is one more setting you must fiddle with, in your Enclave.config.xml file:

    <EnclaveConfiguration>
      <ProdID>0</ProdID>
      <ISVSVN>0</ISVSVN>
      <StackMaxSize>0x40000</StackMaxSize>
      <HeapMaxSize>0x100000</HeapMaxSize>
      <TCSNum>10</TCSNum>
      <TCSPolicy>1</TCSPolicy>
      <!-- Recommend changing 'DisableDebug' to 1 to make the enclave undebuggable for enclave release -->
      <DisableDebug>0</DisableDebug>
      <MiscSelect>0</MiscSelect>
      <MiscMask>0xFFFFFFFF</MiscMask>
    </EnclaveConfiguration>
    

    To be more specific, the HeapMaxSize value. See the SGX developper reference, page 58/59.