scala, spray, akka - java.lang.OutOfMemoryError: unable to create new native thread

While checking the throughput of spray api.

Scenario: 25 concurrent users

Os: Free BSD

Memory: 2GB

No Of Cores: 2

At around 13 concurrent users i was getting the following error.

[ERROR] [06/29/2015 05:01:56.407] [default-akka.actor.default-dispatcher-2]     [ActorSystem(default)] Uncaught error from thread [default-akka.actor.default-dispatcher-2] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled
java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method)
    at java.lang.Thread.start(Thread.java:714)
    at scala.concurrent.forkjoin.ForkJoinPool.tryAddWorker(ForkJoinPool.java:1672)
    at scala.concurrent.forkjoin.ForkJoinPool.deregisterWorker(ForkJoinPool.java:1795)
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:117)

Akka and Spray Conf changes from default:

akka{
    tcp{
        register-timeout = 20s
    }
}

spray.can {
    request-timeout = 30 s
    bind-timeout = 30s
    unbind-timeout = 5s
    registration-timeout = 30s
}

http.spray.can {
    server{
        pipelining-limit = 50
    }
}

What is causing the OutOfMemmoryError. The exception is thrown from the router actor

Solution

can't read mind, but probably you do blocking (Await.result or similar) inside actors. ForkJoinPool creates a new thread for every blocked one automatically. So if you have a long-time blocks count_of_threads == count_of_requests (+ every thread holds the references iside call stack), which eventually causes OutOfMemory.

See, Blocking Needs Careful Management

P.S. Here you may find why Await.result (which uses scala.concurrent.blocking inside) leads to unmanageable creation of threads in ForkJoinPool(even regardless of maxParallelism).

Or you creating a lot of ActorSystems, same page of akka documentation states:

An ActorSystem is a heavyweight structure that will allocate 1…N Threads, so create one per logical application.