prevent java program from opening threads

I'm trying to run a java application , more specifically a jar compiled one, using execve() in c
something like that:

char *cmd[] = {"a.jar"};
execve("a.jar",cmd,NULL);

that is working OK but when I try to limit the number of threads that this program can open using something like that:

struct rlimit rlp;
rlp.rlim_cur = rlp.rlim_max = limit_nproc; 
setrlimit(RLIMIT_NPROC,&rlp);

I have a problem with the JVM which would open threads and I'm preventing that so I have this error:

java.lang.OutOfMemoryError: Cannot create GC thread. Out of system resources.

how can I prevent the threads opened in the java application but not those opened by the JVM ? !

please notice , the question is how to prevent user threads but not system threads , I need a restriction to the running environment like what i did in my second code "RLIMIT_NPROC"

and Thanks!

This can be achieved with JVMTI agent.

The idea is to intercept native Thread.start0() method and throw an exception whenever it is called.

Here is a sample agent written in C++:

#include <jvmti.h>

// Original native implementation of Thread.start0(), if you wish to call it
extern "C" void JNICALL JVM_StartThread(JNIEnv* env, jthread thread);

void JNICALL StartThreadHook(JNIEnv* env, jthread thread) {
    env->ThrowNew(env->FindClass("java/lang/Error"), "Threads forbidden");
}

void JNICALL VMInit(jvmtiEnv* jvmti, JNIEnv* env, jthread thread) {
    // After VM is initialized, intercept Thread.start0() with our hook function
    jclass thread_class = env->FindClass("java/lang/Thread");
    JNINativeMethod start0 = {(char*)"start0", (char*)"()V", (void*)StartThreadHook};
    env->RegisterNatives(thread_class, &start0, 1);
}

JNIEXPORT jint JNICALL Agent_OnLoad(JavaVM* vm, char* options, void* reserved) {
    jvmtiEnv* jvmti;
    vm->GetEnv((void**)&jvmti, JVMTI_VERSION_1_0);

    jvmtiEventCallbacks callbacks = {0};
    callbacks.VMInit = VMInit;
    jvmti->SetEventCallbacks(&callbacks, sizeof(callbacks));
    jvmti->SetEventNotificationMode(JVMTI_ENABLE, JVMTI_EVENT_VM_INIT, NULL);

    return 0;
}

Compile the agent:

g++ -fPIC -shared -olibnothreads.so -Wl,-soname,libnothreads.so nothreads.cpp

Run the application with the agent:

java -agentpath:/path/to/libnothreads.so -jar app.jar

Note that you may also use JVMTI to implement custom logic when to allow and when to deny starting new threads. For example, ThreadStart and ThreadEnd events will help to count created threads. GetStackTrace function will help to find which classes are trying to create a thread.