Continuous Integration tools - How Skies built his CI Platform

Preface

I published this article on the technical blog Skies, a small tech startup (team of 7 people) working on Smart Storage, a software-defined storage solution targeting companies storing buttloads of data.

The objective of our product was:

• Reduce the energy consumption of servers.
• Understand the data to store them more efficiently, leading to space savings.

Introduction

Working on a large project with a team is not an easy task. A large project generally involves many lines of code in different languages with many libraries and sub projects dependencies. Developers work simultaneously on multiple parts of the software and it can become a nightmare to produce a stable and clean build.

That is when Continuous Integration (CI) comes into play. It is development practice requiring developers to push code on a daily basis in a common branch and execute a list of commands to test and build the project. A more exhaustive definition that we like can be found on Thoughtworks provides.

At Skies, it seems really important to us to spend some time at the beginning of the project to build a platform supporting this development practice in order to detect errors early and to reduce the technical debt. We want to increase our productivity and our product's reliability by ensuring that every aspect of the codebase is always tested (functional level, speed of execution, network performance, quality of the code). And for that, we have to use and set up some tools.

Our projects are mainly realized in C++ and are hosted on GitHub repositories. We have chosen to create unit tests with the library GoogleTest.

However, this article does not cover the creation of unit tests but describes the various tools to orchestrate and industrialize source code analysis. It goes from fetching and compiling the source code at each modification up to the generation of reports on a single interface with the publication of results on Slack or by email. We also go through:

• static code analysis
• complexity
• code duplication
• executing unit tests
• computing code coverage
• testing the application performance

We were looking for tools that meet the following criteria:

• open source license
• running on unix
• compatible with GIT
• supporting multiple programming languages (C++, Python, Javascript and more to come)
• maintained and documented
• used by established companies

Here is our candidates.

Jenkins

Of course, the first to be tested was Jenkins (MIT license). It is the most known CI tool and one of the most used (eBay, Google, Facebook, NetFlix, Yahoo and many others). Its installation was fairly simple and fast.

The configuration is done quite easily once you know the analysis tools you want to use. In our case, we used GoogleTest for unit tests, cppcheck for static code analysis and gcovr to calculate the coverage rate. The different compilations are done through a Makefile. This list of tasks is defined through an Ant configuration file. Then we just have to install Jenkins plugins to retrieve and format the results (Cobertura Plugin for the output of gcovr and Cppcheck Plug-in for cppcheck).

You can find the Ant file used for our test here:

 1 <?xml version="1.0" encoding="UTF-8"?>
 2 <project name="CppApp" default="jenkins" basedir=".">
 3     <description>
 4         Jenkins - Ant file - Cpp Project
 5     </description>
 6 
 7     <target name="init">
 8         <tstamp/>
 9     </target>
10 
11     <target name="clean" description="Clean up" >
12         <exec dir="./" executable="/usr/bin/make" failonerror="false">
13             <arg value="clean"/>
14         </exec>
15     </target>
16 
17     <target name="compile" description="Compile the source files">
18         <exec dir="./" executable="/usr/bin/make" failonerror="true">
19         </exec>
20     </target>
21 
22     <target name="compile-tests" description="Compile the tests files">
23         <exec dir="./" executable="/usr/bin/make" failonerror="true">
24             <arg value="unit_tests"/>
25         </exec>
26     </target>
27 
28     <target name="gtest" description="Run GTest" depends="compile-tests">
29         <exec dir="./" executable="./unit_tests" failonerror="true">
30             <arg line="--gtest_output='xml:./outputGTest.xml'"/>
31         </exec>
32     </target>
33 
34     <target name="cppcheck" description="Run cppcheck static code analysis" >
35         <exec dir="./" executable="/usr/bin/cppcheck" failonerror="true">
36             <arg line="--xml --xml-version=2 --enable=all --inconclusive --language=c++ ./src/"/>
37             <redirector error="outputCppCheck.xml"/>
38         </exec>
39     </target>
40 
41     <target name="gcovr-xml" description="Run gcovr and generate coverage XML output">
42         <exec dir="./" executable="/usr/local/bin/gcovr" failonerror="true">
43             <arg line="--branches --xml-pretty -r ./src/"/>
44             <redirector output="outputGcovr.xml"/>
45         </exec>
46     </target>
47 
48     <target name="gcovr-html" description="Run gcovr and generate coverage HTML output">
49         <exec dir="./" executable="/usr/local/bin/gcovr" failonerror="true">
50             <arg line="--branches -r ./src/ --html --html-details -o outputGcovrReport.html"/>
51         </exec>
52     </target>
53 </project>

The installation, the configuration, and test execution are not too difficult with Jenkins. However, analysis of results and reports for developers were clearly not pleasant. The Jenkins interface is old and not ergonomic at all although it works fine. We preferred to look for another CI system, more modern and ergonomic and keep this solution as a backup plan.

Strider-CD

Strider-CD is an open source CI and deployment platform. Its installation was really simple. I managed to clone my projects and to add all my commands from the web interface (those which were in our Ant file).

In a few minutes, I was able to successfully execute all my tests. However, the main issue with this CI is that there is nothing to analyze the reports; you can only set a list of commands, divided into 2 parts: tests and deploy.

Moreover, in spite of a quick installation, the interface is not perfect: some parts are not finished, buggy and there is a lack of features.

BuildBot

The next tool to be tested is BuildBot (GPL license). It is an Open Source CI written in Python. Compared to the previous systems, the installation and configuration of this one are a bit more complex.

Indeed, unlike other tools where most configurations were filled with a GUI, BuildBot must be configured through a Python file. It is inside that we define task lists, GitHub hook for calling a builder automatically, admin access, and more ... Even if it seems quite complex at first (the untidy documentation does not help) the configuration is more permissive; moreover, once you have understood each part to configure, it is fairly simple to adjust the tool to your problems.

However, there is no way to previewing reports. As Strider-CD, the tool only allows the execution of tasks and not the analysis of results.

Finally, the interface is not very modern like Jenkins, but it is possible to customize the HTML and CSS of the tool by overriding the template files.

PaaS products

Being in 2016, we could not overlook PaaS products offering Continuous Integration but ultimately we chose not to use them.

Why?

These tools are:

• offering a great UI
• easier and faster to set up

But:

• they offer less flexibility and features (for instance, many of them do not support pull requests or log compressions)
• we do not want to depend on 3rd party tools for such crucial parts and have our code accessible on remote servers
• some of them (like Hosted-CI) are just a hosted Jenkins server
• they are quite expensive (Travis-CI starts at $139/month) for non open source projects

Some might argue that hosting, installing, configuring and running a Jenkins server (or equivalent) is way more expensive but we really want to maintain control over our CI.

Although we have not tested extensively these online tools, they could still be useful to you, depending on your needs and criteria. Here are those which were able to hold our attention: TeamCity, Codeship, Bamboo, Drone.io, CircleCI, GitLab-CI and Travis-CI.

SonarQube

What we lacked was a tool that allowing us to analyse and format our results (unit tests, code coverage, static code analysis). We finally found SonarQube.

Sonar does not execute a bunch of commands like the previous tools. However, it can read the results (XML files for example) and format them by association with the source code of the project. The interface is really well made, clear, and allows us to see the problems directly on each file (with the aggregation of all results).

We also used Sonar-CXX, a plugin for Sonar to manage the C++. It can read the reports of Valgrind, cppcheck, RATS, gcovr, Vera++ ...

Sonar also made some further analysis of the code, such as its complexity or duplication rate. The advantage of using SonarQube is to have a global visibility over the source code; if the problems are put forward, it is easier to correct them.

Our stack

We have decided to use BuildBot and Sonar for our Continuous Integration platform.

BuildBot executes our commands (unit tests, cppcheck, gcovr, valgrind) after receiving an event from GitHub (with our hook) and pass the results to Sonar (through a script called sonar-runner).

Here are our configuration files for these tools:

 1 # -*- python -*-
 2 
 3 # BuildBot master settings
 4 
 5 from buildbot.plugins import *
 6 
 7 c = BuildmasterConfig = {}
 8 
 9 ####### BUILDSLAVES
10 
11 c['slaves'] = [buildslave.BuildSlave("slave", "S2n6zzCzKeuTChsmbpW7")]
12 
13 c['protocols'] = {'pb': {'port': 9989}}
14 
15 ####### CHANGESOURCES
16 
17 c['change_source'] = []
18 
19 ####### SCHEDULERS
20 
21 c['schedulers'] = []
22 c['schedulers'].append(schedulers.SingleBranchScheduler(name="all", change_filter=util.ChangeFilter(branch='master'), treeStableTimer=None, builderNames=["runtests"]))
23 c['schedulers'].append(schedulers.ForceScheduler(name="force", builderNames=["runtests"]))
24 
25 ####### BUILDERS
26 
27 factory = util.BuildFactory()
28 
29 factory.addStep(steps.Git(repourl='git@github.com:skies-io/repository.git', branch='master', mode='incremental'))
30 factory.addStep(steps.ShellCommand(command=["make", "clean"]))
31 factory.addStep(steps.ShellCommand(command=["make", "unit_tests"]))
32 factory.addStep(steps.ShellCommand(command="cppcheck -v --xml --xml-version=2 --enable=all --inconclusive --language=c++ -Iinclude/ src/ 2> report-cppcheck.xml"))
33 factory.addStep(steps.ShellCommand(command=["valgrind", "--xml=yes", "--xml-file=report-valgrind.xml", "./unit_tests", "--gtest_output=xml:report-xunit.xml"]))
34 factory.addStep(steps.ShellCommand(command="gcovr -x -r . > report-gcovr.xml"))
35 factory.addStep(steps.ShellCommand(command=["/opt/sonar-runner/bin/sonar-runner", "-X"]))
36 
37 c['builders'] = []
38 c['builders'].append(util.BuilderConfig(name="runtests", slavenames=["slave"], factory=factory))
39 
40 ####### STATUS TARGETS
41 
42 c['status'] = []
43 
44 from buildbot.status import html
45 from buildbot.status.web import authz, auth
46 
47 authz_cfg=authz.Authz(auth=auth.BasicAuth([("admin", "S2n6zzCzKeuTChsmbpW7")]), forceBuild = 'auth', forceAllBuilds = 'auth')
48 github = {'github': {'secret': "692cf6d5b41c53a7a62c7f323c91038a5ac34f784eb11e92670c425e", 'strict': True}}
49 c['status'].append(html.WebStatus(http_port=8010, authz=authz_cfg, change_hook_dialects=github))
50 
51 ####### PROJECT IDENTITY
52 
53 c['title'] = "Skies Smart Storage"
54 c['titleURL'] = "https://github.com/skies-io/repository/"
55 c['buildbotURL'] = "http://127.0.0.1:8010/"
56 
57 ####### DB URL
58 
59 c['db'] = {'db_url': "sqlite:///state.sqlite"}

 1 # https://github.com/SonarOpenCommunity/sonar-cxx/tree/master/sonar-cxx-plugin/src/samples/SampleProject2
 2 
 3 sonar.projectKey=CxxPlugin:SkiesSmartStorage
 4 sonar.projectName=SkiesSmartStorage
 5 sonar.projectVersion=1.0.0
 6 sonar.language=c++
 7 
 8 sonar.sources=src
 9 sonar.tests=tests
10 
11 sonar.cxx.cppcheck.reportPath=report-cppcheck.xml
12 sonar.cxx.coverage.reportPath=report-gcovr.xml
13 sonar.cxx.valgrind.reportPath=report-valgrind.xml
14 sonar.cxx.xunit.reportPath=report-xunit.xml
15 
16 sonar.cxx.includeDirectories=/usr/include,include,src

Although the configuration of BuildBot is not intuitive, it is one of the tool with the fewest constraints. Here is what the architecture looks like: