Solution to API’s OutOfMemory issue which stops creating thread

Introduction

This article provides the solution/prevention to the API’s OutOfMemory error where operating system is unable to create new native thread. The operating system is limiting the number of threads your Mule process can spawn, or the memory settings are wrong (setting more memory than the physically available memory).

Memory Calculation

The usage of memory in Java is determined by the following formula:

 Max memory = [-Xmx] + [-XX:MaxPermSize] + number_of_threads * [-Xss] 

Most of the time the focus is in the Heap size determined by the JVM argument -Xmx and eventually, in Oracle JVMs, the Permanent Generation Space, specified by the argument -XX:MaxPermSize. However you must also consider how many threads will your Mule instance handle, as it also adds to the max memory usage.

For Example

Suppose you have the following parameters:

 Max Heap size: 3072 MB
 Max PermGen size: 512 MB

If your system has in total 4GB RAM, then you would think that the 512 MB left to the operating system will cause no problem. But the fact is that, for instance in a Linux 64 bits; every thread will take 1 MB of RAM from your system memory. So if your Mule instance would have 512 threads (512 x 1MB = 512 MB), then you wouldn't have memory left to the operating system. But usually that's never the case, as you will first see an OutOfMemoryError: Unable to create new native thread.

Symptom

Below is the actual log snippet which can be seen during the OutOfMemory under Mule instance - 

Exception in thread "throttling-task.13" java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method)
    at java.lang.Thread.start(Thread.java:691)
    at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
    at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1017)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1163)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
Exception in thread "[case8244].http.request.dispatch.8081.170" java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method)
    at java.lang.Thread.start(Thread.java:691)
    at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
    at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1017)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1163)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
Exception in thread "[case8244].http.request.dispatch.8081.25" java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method)
    at java.lang.Thread.start(Thread.java:691)
    at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
    at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1017)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1163)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)

Solution

Below are the steps to install the program –
1.Take a look at our MuleSoft provided Performance Tuning Guide - https://docs.mulesoft.com/mule-user-guide/v/3.7/tuning-performance
2.Determine and adjust your memory settings accordingly, taking into consideration the formula provided above.
 3.Review the operating system's resource limits for the user running Mule, like file descriptors and max user processes and adjust them accordingly to the expected usage.
 4.Load test, fine tune and load test again.

Operating system limits for Linux In Linux operating system there are several configurations that could be preventing Java of creating a new thread.
1.limits associated to the user (ulimit) user limits can be queried with the ulimit -a command
Example

$ ulimit -a

core file size (blocks, -c) 0

data seg size (kbytes, -d) unlimited

scheduling priority (-e) 0

file size (blocks, -f) unlimited

pending signals (-i) 95146

max locked memory (kbytes, -l) 64

max memory size (kbytes, -m) unlimited

open files (-n) 4096

pipe size (512 bytes, -p) 8

POSIX message queues (bytes, -q) 819200

real-time priority (-r) 0

stack size (kbytes, -s) 10240

cpu time (seconds, -t) unlimited

max user processes (-u) 4096

virtual memory (kbytes, -v) unlimited

file locks (-x) unlimited

most relevant are open files and max user process. Both should be greater than the number of threads that could be opened by the entire Mule ESB Java process

2.Operating System maximums Maximum number of files open

Example
$ cat /proc/sys/fs/file-max Maximum number of threads

Example
$ cat /proc/sys/kernel/threads-max

 Note that depending on your distribution there could be other files and configurations that could be limiting the resources allowed by the application. Check your operating system documentation for more information.

In Red Hat Linux it has been noted that the user limits can be overridden with the values from a system wide configuration. To increase the default value you'll need to modify the default configuration in /etc/security/limits.d/90-nproc.conf from the default value of 1024 process to the desired value and restart the server. Verifying actual limits for a running process. Use the following command on the running process PID: $ cat /proc/PID/limits Replace PID for the actual process identifier for the Java process that is running the ESB. Note This would not help if the application is configured to create too large number of threads for the specific system or it is actually leaking threads. In the former case further configuration tuning is recommended. In the latest you could use a thread dump and Linux ps -eLf command to analyse which threads are active. The lsof command could be used to verify if too many file descriptors are being created