Starting point

The client had a Jenkins environment that was in active use. However, it frequently broke due to Jenkins upgrades or unmanaged changes to Jenkins configuration. People responsible for Jenkins administration were afraid of making any changes to the system to avoid breaking it. This reduced the usefulness of Jenkins in general, as it could not be quickly adapted to new developer needs.

Project

We identified lack of change management and general unreliability of Jenkins as the root cause for the issues the client was having. We could not really affect the general unreliability, so we focused on defining all Jenkins configurations as code - a formidable task at the time. Once Jenkins configuration was defined as code we helped the client implement a proper peer review for all Jenkins changes.

End result

The resulting Jenkins setup was much less prone to breakages than the earlier one. Configuration changes were exposed to peer review and even when a bug managed to slip in, it could be rolled back easily. Moreover, new Jenkins instances could be spun up much easier than before.

Main technologies

• Jenkins
• Puppet