From boto3 doc for the start_job_run, it seems like I have to create job run every time I want to trigger a job. Does it really have to work that way? Can't I take the ID of the existing job, which has already been defined with all the configuration it needs, and run it?
Reading the doc and searching on the internet
Yes that's the way it needs to be executed with boto3. You can call describe-job-run to gain the config information and then pass that to start-job-run.
This assumes that you have a short lived cluster. Executing jobs on a long lived cluster would be different.