How to run the same job multiple times in parallel with Jenkins?

I'm testing Jenkins to see if it will fit our build and testing framework. I found that Jenkins and its available plugins fit most of our needs. Except that I can't seem to find help on how to do one particular type of task.

We are creating application for embedded devices. We have 100s of tests that need to be run on these devices. If we run all the tests on one device after a build then it will take several hours to get the results. However, if we run the tests on 100 of the devices in parallel then we can get results in much shorter time.

All the tests will have very similar starting point. A test script is called with IP address of device to run the test on and user name/pw. The script would do the necessary test on the device and report back pass/fail for each test item.

I think the long/painful way of doing this is writing 100 jobs in Jenkins, each will be a different test script directly (with above parameters) and run these in parallel using available plugins. However, maintaining all these jobs will be very difficult in the long run.

So, the better way to do this would be to create a Job (let's call it child_tester) that can take parameters such as: test script name, IP address of device, user name/pw, etc. Then use another job (let's call it mother_tester) to call child_tester job 100 times with different IP addresses and run them in parallel. I would need some way of accumulating all the test results of each individual run of the child_tester jobs and report them back to mother_tester.

My question is there a plugin or any way of accomplishing this in Jenkins? I have looked into the information of the plugins called "Build Flow", "Parallel Test Executor", and "Parameterized Trigger". However, they don't seem to fit my needs.

Solution

I understand you've looked into the Build Flow plugin, but I'm not sure why you've dismissed it. Perhaps you can point out the holes in my proposal.

Assuming you have enough executors in your system to run jobs in parallel, I think that the Build Flow plugin and Build Flow Test Aggregator plugin can do what you want.

The Build Flow plugin supports running jobs in parallel. I don't see any reason why Build Flow could not schedule your "child" job to run in parallel with different parameters.
The Build Flow Test Aggregator grabs test results from the scheduled builds of a Build Flow job, so your "child" job will need to publish its own test results.
You will need to configure your "child" job so that it can run in parallel by checking the "Execute concurrent builds if necessary" in the job configuration.
Whatever set of slaves provide the connection to the embedded devices will need enough executors to run your jobs in parallel.

Update: with the simple Build Flow definition:

parallel (
  { build("dbacher flow child", VALUE: 1) },
  { build("dbacher flow child", VALUE: 2) },
  { build("dbacher flow child", VALUE: 3) },
  { build("dbacher flow child", VALUE: 4) }
)

I get the output:

parallel {
    Schedule job dbacher flow child
    Schedule job dbacher flow child
    Schedule job dbacher flow child
    Schedule job dbacher flow child
    Build dbacher flow child #5 started
    Build dbacher flow child #6 started
    Build dbacher flow child #7 started
    Build dbacher flow child #8 started
    dbacher flow child #6 completed 
    dbacher flow child #7 completed 
    dbacher flow child #5 completed 
    dbacher flow child #8 completed 
}

The job history shows that all four jobs are scheduled within seconds of each other. But the job build step contains an artificial delay (sleep) that would prevent any single build from completing that quickly.

Update 2: Here is an example of generating the list of parallel tasks dynamically from another data structure:

// create a closure for the deploy job for each server 
def paramValues = (1..4)
def testJobs = [] 
for (param in paramValues) { 
  def jobParams = [VALUE: param] 
  def testJob = { 
    // call build 
    build(jobParams, "dbacher flow child") 
  } 
  println jobParams
  testJobs.add(testJob) 
} 

parallel(testJobs)

The list passed to parallel is a list of closures that call the build with unique parameters. I had to make sure to define the job parameters outside of the closure function to ensure the jobs would be scheduled separately.

I cribbed the syntax from another answer and this thread on the Jenkins mailing list.