OpenStack's Jenkins Job Filler

OpenStack comprises of many projects, all of which have code reviews via. Gerrit and every code review is tested with many tests on Jenkins.  Whilst this is excellent for code quality this can get quite insane to manage.  In fact at time of writing OpenStack's Jenkins server now has a shade under 300 jobs to manage, I don't know about you but I really don't want to be spending my days managing all of that.  Often we need to make a small change to many jobs at the same time and human error occurs.  In the last few weeks we developed and unveiled the CI team's solution to this: Jenkins Job Filler (aka. Jenkins Job Builder when I forget what we called it).

The initial version of Jenkins Job Filler was written in a combination of Ruby and Puppet scripts but unfortunately it became unmanageable very quickly, it also required a Jenkins restart on every change.  So the new version has been writing in Python and YAML.  There is still some puppet behind it, but only to execute the Python script on every run.

How does it work?  There are two ways of specifying a project, one way is to have a YAML section in a file for every job, the other is to use a template.  With the template system this is all that is needed to specify all the jobs for Nova:

project:
 template: 'python_jobs'

values:
  name: 'nova'
  disabled: 'false'
  github_org: 'openstack'
  review_site: 'review.openstack.org'
  publisher_site: 'nova.openstack.org'

Much simpler than hacking XML files!

Behind this there are python modules which specify each part of a standard XML file and the template YAML file which specifies which modules should be used for each job and the parameters for each job.  It then injects the resulting XML straight into Jenkins using the Jenkins API.

This system is currently in use for all of StackForge and most of OpenStack (the last few remaining jobs will be converted over the next couple of weeks).  Now if we want to make a change to one job in all projects we can do so in one go as a tiny commit.  This is made much simpler for OpenStack due to the standard Project Testing Interface that projects should follow.  But it is flexible enough to work with non-standard jobs and projects.

To find out more see the Jenkins Jobs Documentation or the code in the OpenStack CI Puppet repository.