I would like to load test an endpoint with multiple requests at the same time. I am getting the error Argument list too long
. How can I fix this?
import subprocess
import argparse
def main(**kwargs):
my_parser = argparse.ArgumentParser()
my_parser.add_argument('-s')
my_parser.add_argument('-e')
args = my_parser.parse_args()
i = int(vars(args)['s'])
cmd = ''
f= open('test.txt')
while i <= int(vars(args)['e']):
for line in f:
cmd += f'curl -X POST -H "Content-Type: application/json" https://some_api_ep -d {line} &'
i +=1
subprocess.run(cmd, shell=True)
if __name__ == "__main__":
main()
$ python load_generator.py -s 1 -e 1
Traceback (most recent call last):
File "load_generator.py", line 20, in <module>
main()
File "load_generator.py", line 17, in main
subprocess.run(cmd, shell=True)
File "/Users/someone/.pyenv/versions/3.6.7/lib/python3.6/subprocess.py", line 403, in run
with Popen(*popenargs, **kwargs) as process:
File "/Users/someone/.pyenv/versions/3.6.7/lib/python3.6/subprocess.py", line 709, in __init__
restore_signals, start_new_session)
File "/Users/someone/.pyenv/versions/3.6.7/lib/python3.6/subprocess.py", line 1344, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 7] Argument list too long: '/bin/sh'
Number of total records
$ cat test.txt | wc -l
18732
Sample data
$ cat test.txt
'{"ErrorCode": 55201, "ErrorClass": 8, "NodeID": 32, "Params0": 1, "Params1": 70451202, "Params2": 9530, "Params3": 7, "Mid": -1, "SystemName": None, "NodeName": "MSC", "Swrelease": "SWL [CRC:3816]", "Build": "AA"}'
'{"ErrorCode": 55004, "ErrorClass": 8, "NodeID": 32, "Params0": 538990697, "Params1": 410814, "Params2": 410931, "Params3": 0, "Mid": -1, "SystemName": None, "NodeName": "MSC", "Swrelease": "SWL [CRC:3816]", "Build": "AA"}'
'{"ErrorCode": 282, "ErrorClass": 0, "NodeID": 32, "Params0": 61, "Params1": 14, "Params2": 0, "Params3": 5, "Mid": -1, "SystemName": None, "NodeName": "MSC", "Swrelease": "SWL [CRC:3816]", "Build": "AA"}'
I ended up using @barmar suggestion which is to create a .sh
file and run it via /bin/sh
.