This document explains how to transfer Jenkins and its projects from one server to another. This explains the process as it was done in 2014 to transfer Jenkins from buffalo.cs.washington.edu to tern.cs.washington.edu.
In retrospect, it would have probably been faster to transfer Jenkins to a VM. A VM would have allowed for the following:
For these reasons, I recommend that a VM be considered as the target for Jenkins the next time it needs to be transferred.
This is not an exhaustive list. This is merely a list of packages that happened to be needed and were available in the RHEL 6.5 64-bit distribution but were not installed by default. Note that this does not include software we made local installations of (i.e. software that we made available to Jenkins from the local directory /www/secs-jenkins/tools ).
cups-devel ant-jmf ant-junit dos2unix dia protobuf-c protobuf-compiler protobuf hevea (May not be available in the RHEL repo and may need to be built from the SRPM for Fedora.)
To be able to run 32-bit processes on a 64-bit machine (some Jenkins projects use 32-bit binaries), also do:
yum groupinstall "Compatibility libraries"
Additionally install 32-bit versions (sometimes labeled as 'i686') of the following libraries:
zlib yum install libXext-1.3.1-2.el6.i686 yum install libXrender-0.9.7-2.el6.i686 yum install fontconfig-2.8.0-3.el6.i686 yum install freetype-2.3.11-14.el6_3.1.i686 yum install libstdc++-4.4.7-4.el6.i686 yum install eclipse-rcp-3.6.1-6.13.el6.x86_64
The following are currently found under /www/secs-jenkins/tools :
android-sdk-linux cups google-gson hevea jtreg mercurial Python texinfo toJunitXML
For the most part these should continue to work when they are transferred to a new machine. What was new during this transfer was that the versions of Python and Mercurial on the target machine were too old, and could not be easily upgraded, so we had to opt for the local installs of both and point each Jenkins project to use these.
This step was handled by Scott Rose of tech support.
The /scratch/secs-jenkins directory on buffalo was transferred to an appropriate home on tern (/www/secs-jenkins). Make sure all subdirectories are transferred, including /www/secs-jenkins/java/ and tools/ The size of this folder is currently on the order of 100 GB. The transfer allows us to preserve the build histories. These are occasionally useful to recover an old build failure.
It is important to keep the secret keys in /scratch/secs-jenkins/jenkins-home/ since nobody knows the plain password for the email@example.com email account (the one that sends build failure mails).
These were the steps followed by support to transfer Jenkins from one physical machine to another:
First one installs the Jenkins repo, then Jenkins itself. The repo configuration looks like:
% cat /etc/yum.repos.d/jenkins.repo [jenkins] name=Jenkins baseurl=http://pkg.jenkins-ci.org/redhat/ gpgcheck=1
% rpm -q jenkins jenkins-1.563-1.1.noarch
These are packages that are built *for* Fedora but not *by* Fedora. Fedora doesn't itself package Jenkins.
The basic configuration for the service lives in /etc/sysconfig/jenkins. That's owned by and only writable by root, but it should have permissions so that members of pl_gang can read it.
The Jenkins service (a system service started at boot) runs as the user jenkins. Root apparently reads that file, then drops privileges.
By default, Jenkins would have its home - as configured by the value of JENKINS_HOME - in /var/lib/jenkins/, but /var tends to be too small. On buffalo, the machine we transferred Jenkins from, we had:
# fgrep JENKINS_HOME /etc/sysconfig/jenkins # JENKINS_HOME="/var/lib/jenkins" JENKINS_HOME="/scratch/secs-jenkins/jenkins-home"
On tern, the machine we transferred it to, there is a gigantic (3.6T) ZFS pool (think "file system") currently mounted on /www. There is nothing special about that path, though, particularly on a machine that doesn't have apache httpd installed. /www/secs-jenkins/jenkins-home/ was set up as the JENKINS_HOME on tern.
The transfer steps are: 1. Stop jenkins on the source machine. 2. rsync the content to the target machine. 3. Start the service on the source machine.
If re-syncing is needed:
1. rsync 2. stop jenkins 3. rsync again to get changed content 4. restart the service
Also it's important to make sure that /etc/sysconfig/jenkins has the value of JENKINS_HOME set to /www/secs-jenkins/jenkins-home (or wherever the new Jenkins home is).
At this point, you should be able to see the Jenkins web dashboard (e.g. http://tern.cs.washington.edu:8080/) on the target machine and it should list the projects that were available on the source machine.
Make sure Mercurial, as used by Jenkins, is set up to trust you. Ask tech support to add your username to /www/secs-jenkins/jenkins-home/.hgrc
The [trusted] section of this file should now contain at least the following:
[trusted] users = jenkins, apache, comma-separated-usernames groups = pl_gang, jenkins
I recommend turning off e-mail notifications of failed builds when performing the following steps. To do this, in the Jenkins dashboard under "Manage Jenkins", "Configure System", scroll down to "E-mail Notification". Simply remove the SMTP server and no e-mails can be sent. Be sure to write it down so you can put it back later when you are done configuring the new machine.
Get latest Python 2.7.x sources and build them
Put the sources in /www/secs-jenkins/tools/Python-2.7.8/ (or replace /www with the appropriate path, and use the appropriate Python version number)
Create a local copy of Makeinfo:
Under /www/secs-jenkins/tools/texinfo-5.2/tp, create a script named 'makeinfo' with these contents: perl /www/secs-jenkins/tools/texinfo-5.2/tp/texi2any.pl $@
Put texinfo in the path in the Jenkins config for Daikon, e.g.:
cd /www/secs-jenkins/tools/android-sdk-linux/tools ./android list sdk --all (find build-tools 19.0.1 in the resulting list - in my case it was #6 - and put that number below after -t ): ./android update sdk -u -a -t 6
If all the projects are building correctly, you are ready to transfer the post-commit web hooks. These are the triggers that tell Jenkins to start a new build of a project whenever changes are pushed (checked in) to that project's repository.
For the Checker Framework project, for example, in Google Code, under Administer -> Source (https://code.google.com/p/checker-framework/adminSource) you will see that the post-commit web hook is set to http://tern.cs.washington.edu:8080/job/zzz-hook-checker-framework/build.
The process is similar for GitLab hosted projects. Go to Settings -> Webhooks & Services -> Add webhook. At this point you would enter the same URL information as above, take the rest of the defaults and select "Add webhook".
Change tern to the new machine name for all the post-commit web hooks for Google Code projects.
The post-commit hooks for projects hosted on our own network paths need to be set up on these paths. Specifically, modify the [hooks] section in the following two files to point to the the new Jenkins server instead of tern:
The relevant line under [hooks] looks like:
changegroup = curl --silent -d "" http://tern.cs.washington.edu:8080/job/sparta-code/build?delay=0sec
The relevant line under [hooks] looks like:
changegroup = curl --silent -d "" http://tern.cs.washington.edu:8080/job/sparta-sample-apps/build?delay=0sec
Finally, don't forget to turn automatic e-mails back on in the main configuration page under "Manage Jenkins", "Configure System", scrolling down to "E-mail Notification". If you didn't write down the old value for the SMTP server, try smtp.gmail.com
These pages will occasionally list administration tasks to perform, such as updating plug-ins: http://tern.cs.washington.edu:8080/manage, http://tern.cs.washington.edu:8080/pluginManager/.
To update Jenkins itself, remember that it is updated through an RPM on the OS, so ask tech support to update that for you if the version you are running is very old.
Additionally, make sure that the local installs of the JDK under /www/secs-jenkins/java are kept up to date.
Note: If ever a software update and reboot makes your Jenkins projects go away, make sure that /etc/sysconfig/jenkins has the value of JENKINS_HOME set to /www/secs-jenkins/jenkins-home (or wherever the new Jenkins home is). It is possible that this value has been reset to /var/lib/jenkins.