Moving Jenkins from one physical server to another

This document explains how to transfer Jenkins and its projects from one server to another. This explains the process as it was done in 2014 to transfer Jenkins from buffalo.cs.washington.edu to tern.cs.washington.edu.

Contents:

Next time, use a VM?

In retrospect, it would have probably been faster to transfer Jenkins to a VM. A VM would have allowed for the following:

For these reasons, I recommend that a VM be considered as the target for Jenkins the next time it needs to be transferred.

Pre-install steps

Packages that need to be installed

This is not an exhaustive list. This is merely a list of packages that happened to be needed and were available in the RHEL 6.5 64-bit distribution but were not installed by default. Note that this does not include software we made local installations of (i.e. software that we made available to Jenkins from the local directory /www/secs-jenkins/tools ).

cups-devel
ant-jmf
ant-junit
dos2unix
dia
protobuf-c
protobuf-compiler
protobuf
hevea (May not be available in the RHEL repo and may need to be built from the SRPM for Fedora.)

To be able to run 32-bit processes on a 64-bit machine (some Jenkins projects use 32-bit binaries), also do:

yum groupinstall "Compatibility libraries"

Additionally install 32-bit versions (sometimes labeled as 'i686') of the following libraries:

zlib
yum install libXext-1.3.1-2.el6.i686
yum install libXrender-0.9.7-2.el6.i686
yum install fontconfig-2.8.0-3.el6.i686
yum install freetype-2.3.11-14.el6_3.1.i686
yum install libstdc++-4.4.7-4.el6.i686
yum install eclipse-rcp-3.6.1-6.13.el6.x86_64

Software that needed local installs

The following are currently found under /www/secs-jenkins/tools :

android-sdk-linux
cups
google-gson
hevea
jtreg
mercurial
Python
texinfo
toJunitXML

For the most part these should continue to work when they are transferred to a new machine. What was new during this transfer was that the versions of Python and Mercurial on the target machine were too old, and could not be easily upgraded, so we had to opt for the local installs of both and point each Jenkins project to use these.

Initial transfer of Jenkins

This step was handled by Scott Rose of tech support.

The /scratch/secs-jenkins directory on buffalo was transferred to an appropriate home on tern (/www/secs-jenkins). Make sure all subdirectories are transferred, including /www/secs-jenkins/java/ and tools/ The size of this folder is currently on the order of 100 GB. The transfer allows us to preserve the build histories. These are occasionally useful to recover an old build failure.

It is important to keep the secret keys in /scratch/secs-jenkins/jenkins-home/ since nobody knows the plain password for the zabihah@gmail.com email account (the one that sends build failure mails).

These were the steps followed by support to transfer Jenkins from one physical machine to another:

First one installs the Jenkins repo, then Jenkins itself. The repo configuration looks like:

% cat /etc/yum.repos.d/jenkins.repo 
[jenkins]
name=Jenkins
baseurl=http://pkg.jenkins-ci.org/redhat/
gpgcheck=1
% rpm -q jenkins
jenkins-1.563-1.1.noarch

These are packages that are built *for* Fedora but not *by* Fedora. Fedora doesn't itself package Jenkins.

The basic configuration for the service lives in /etc/sysconfig/jenkins. That's owned by and only writable by root, but it should have permissions so that members of pl_gang can read it.

The Jenkins service (a system service started at boot) runs as the user jenkins. Root apparently reads that file, then drops privileges.

By default, Jenkins would have its home - as configured by the value of JENKINS_HOME - in /var/lib/jenkins/, but /var tends to be too small. On buffalo, the machine we transferred Jenkins from, we had:

 # fgrep JENKINS_HOME /etc/sysconfig/jenkins
 # JENKINS_HOME="/var/lib/jenkins"
 JENKINS_HOME="/scratch/secs-jenkins/jenkins-home"

On tern, the machine we transferred it to, there is a gigantic (3.6T) ZFS pool (think "file system") currently mounted on /www. There is nothing special about that path, though, particularly on a machine that doesn't have apache httpd installed. /www/secs-jenkins/jenkins-home/ was set up as the JENKINS_HOME on tern.

The transfer steps are:
1. Stop jenkins on the source machine.
2. rsync the content to the target machine.
3. Start the service on the source machine.

If re-syncing is needed:

1. rsync
2. stop jenkins
3. rsync again to get changed content
4. restart the service

Also it's important to make sure that /etc/sysconfig/jenkins has the value of JENKINS_HOME set to /www/secs-jenkins/jenkins-home (or wherever the new Jenkins home is).

At this point, you should be able to see the Jenkins web dashboard (e.g. http://tern.cs.washington.edu:8080/) on the target machine and it should list the projects that were available on the source machine.

Getting Mercurial permissions

Make sure Mercurial, as used by Jenkins, is set up to trust you. Ask tech support to add your username to /www/secs-jenkins/jenkins-home/.hgrc

The [trusted] section of this file should now contain at least the following:

[trusted]
users = jenkins, apache, comma-separated-usernames
groups = pl_gang, jenkins

Miscellaneous fix-ups

I recommend turning off e-mail notifications of failed builds when performing the following steps. To do this, in the Jenkins dashboard under "Manage Jenkins", "Configure System", scroll down to "E-mail Notification". Simply remove the SMTP server and no e-mails can be sent. Be sure to write it down so you can put it back later when you are done configuring the new machine.

Post-commit web hooks

If all the projects are building correctly, you are ready to transfer the post-commit web hooks. These are the triggers that tell Jenkins to start a new build of a project whenever changes are pushed (checked in) to that project's repository.

For the Checker Framework project, for example, in Google Code, under Administer -> Source (https://code.google.com/p/checker-framework/adminSource) you will see that the post-commit web hook is set to http://tern.cs.washington.edu:8080/job/zzz-hook-checker-framework/build.

The process is similar for GitLab hosted projects. Go to Settings -> Webhooks & Services -> Add webhook. At this point you would enter the same URL information as above, take the rest of the defaults and select "Add webhook".

Change tern to the new machine name for all the post-commit web hooks for Google Code projects.

The post-commit hooks for projects hosted on our own network paths need to be set up on these paths. Specifically, modify the [hooks] section in the following two files to point to the the new Jenkins server instead of tern:

/projects/swlab1/darpa-apac/sparta-code/.hg/hgrc

The relevant line under [hooks] looks like:

changegroup = curl --silent -d "" http://tern.cs.washington.edu:8080/job/sparta-code/build?delay=0sec

/projects/swlab1/darpa-apac/sparta-sample-apps/.hg/hgrc

The relevant line under [hooks] looks like:

changegroup = curl --silent -d "" http://tern.cs.washington.edu:8080/job/sparta-sample-apps/build?delay=0sec

Finally, don't forget to turn automatic e-mails back on in the main configuration page under "Manage Jenkins", "Configure System", scrolling down to "E-mail Notification". If you didn't write down the old value for the SMTP server, try smtp.gmail.com

Ongoing administration tasks

These pages will occasionally list administration tasks to perform, such as updating plug-ins: http://tern.cs.washington.edu:8080/manage, http://tern.cs.washington.edu:8080/pluginManager/.

To update Jenkins itself, remember that it is updated through an RPM on the OS, so ask tech support to update that for you if the version you are running is very old.

Additionally, make sure that the local installs of the JDK under /www/secs-jenkins/java are kept up to date.

Note: If ever a software update and reboot makes your Jenkins projects go away, make sure that /etc/sysconfig/jenkins has the value of JENKINS_HOME set to /www/secs-jenkins/jenkins-home (or wherever the new Jenkins home is). It is possible that this value has been reset to /var/lib/jenkins.