How to achieve consistent throughput using Jmeter and Concurrency thread group

We are attempting to stress test our system using Jmeter and we hit a wall with inconsistent throughput, so we installed the Jmeter plugin Concurrency Thread Group with the Throughput Shaping Timer. We're currently running it on a single endpoint, to make sure we can achieve a consistent throughput. we set:

• Throughput Shaping Timer to 30 RPS for 60 seconds, which is scoped directly under the Concurrency Thread Group

• Target Concurrency using the tstFeedback function as tstFeedback(tst,13,50,10), since the max response time was around 1000

we initially achieved the expected RPS, it would reach close to 30 RPS every time we attempted the test:

However, after a couple of hours with no change to the configuration or the system, the RPS no longer reached the expected, we would consistently get lower RPS (ranging between 20 and 27 RPS):

We tried increasing the Target Concurrency, but it made no difference to the RPS and it was constantly being less than the expected. We attempted this using the GUI and CLI from a local machine and CLI on a remote instance with Jmeter installed, we are running the latest Jmeter version (5.5 at this time) locally and on the remote instance but it made no difference:

Increasing the HEAP didn’t make a difference either

When we increased the logging level on the local machine to diagnose the problem, this error would appear frequently:

o.a.j.p.h.s.HTTPHC4Impl: IOException
java.net.SocketException: Socket closed

Test instance specs

• The remote instance is set up on an AWS m6i.8xlarge instance
• Local machine has 16GB Memory with intel i7 8th gen processor
• All tests were run on Ubuntu machines

Problem/Question

We need to simulate our customer behavior as accurately as possible, so it's critical that we're able to achieve consistent throughput in our requests. How can we do that?

1. Looking at your Throughput Shaping Timer configuration I can see that it's set up to kick off 50 maximum threads.
2. You can reach 30 requests per second with 50 threads only if your response time will be less than 1.6 seconds or less.
3. Looking into response times from your reports I can see that average response time is above 2 seconds and this perfectly explains the fact you're not able to reach the throughput.

Things to check:

1. Make sure to follow JMeter Best Practices
2. Make sure that JMeter has enough headroom to operate in terms of CPU, RAM, Network, Disk, etc. It can be done using JMeter PerfMon Plugin
3. If you cannot conduct required load from the single JMeter instance - consider going for Distributed Testing
4. And finally it might be your application which cannot handle that many requests per second so it worth checking its logs, telemetry, and so on.