<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/continuous-integration-for-your-private-r-projects-with-circleci/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://appsilondatascience.com/blog/rstats/2018/02/07/circleci.html"> Appsilon Data Science Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>If you have ever developed or used an open-source R package, you’re likely familiar with continuous integration. By automating the process of testing each proposed change in the source code, you can reduce the risk of errors, avoid unnecessary overhead and increase the quality of developed solution. For data scientists, Hadley has a good description of <a rel="nofollow" href="http://r-pkgs.had.co.nz/check.html" target="_blank">why it’s worth using in R</a>.</p>
<p>The most popular CI solution in the R world is TravisCI. Overall it works great, has built-in community support for R and is free for any open source project. CircleCI offers a great alternative with a free plan that includes private repositories. This is a perfect solution if you’re building a package that can not be released publicly and you don’t have a paid Travis account. This post will quickly take you through setting up continuous integration for your private R project with CircleCI.</p>
<p>Even if you are not developing an R package, but simply working on a data science project for a client, you can still use the same approach to run your unit tests and linter each time you push a commit to repo.</p>
<p>At Appsilon Data Science we strive to have tests and continuous integration for every project to ensure we catch all errors as quickly as possible. Obviously, even source code for most commercial projects has to be private (not mentioning training data, which we don’t send to CI anyway). CircleCI was our first choice for continuous integration, and we were not disappointed. We even started using it for our <a rel="nofollow" href="https://github.com/Appsilon/shiny.router" target="_blank">open</a> <a rel="nofollow" href="https://github.com/Appsilon/shiny.i18n" target="_blank">source</a> <a rel="nofollow" href="https://github.com/Appsilon/shiny.semantic" target="_blank">packages</a>. If you’re interested in these packages, we’ve written posts about <a rel="nofollow" href="https://appsilondatascience.com/blog/rstats/2016/12/15/improve-Shiny-apps-user-interface.html" target="_blank">shiny.semantic</a> and <a rel="nofollow" href="https://appsilondatascience.com/blog/rstats/2016/12/08/shiny.router.html" target="_blank">shiny.router</a> in the past.</p>
<h1>Continuous integration for all your private R projects</h1>
<p><label for="gitlab-note"><i></i> sticky note</label><span>You can also use CircleCI with Bitbucket, but not yet with Gitlab. </span><br>
Our goal will be to set up continuous integration for a private R project hosted on GitHub. We’ll be assuming that it is a package and that we already have some unit tests. We want CircleCI to install all the necessary dependencies and run all tests in an isolated environment every time we push a commit to the repository. We’ll be using the free CircleCI plan for that. You can always graduate to a paid plan on CircleCI once your project needs that.</p>
<p>You can use virtually the same steps if the project is not an R package, you’ll just need to adjust how the project is tested.</p>
<h1>Configuring project for CircleCI</h1>
<p>For CircleCI to know how to handle our project, we need to add a configuration file to the project repo. There are two versions of CircleCI API you can use for that, and we’ll be using the newer Version 2.</p>
<p>CircleCI supports running tests in <a rel="nofollow" href="https://www.docker.com/" target="_blank">Docker</a> images, which is great for cleanly managing system-level dependencies. We’ll be using a Docker image that contains all system-level dependencies of our project plus everything that we need for building it. If you don’t need any additional system libraries, you can use the image built by us. Its name is <code>appsilon/ci-base:1.0</code>. You will need to modify the Docker image if your package or its dependencies require some libraries to be installed in the system. We will show you how to do that at the end of this article.</p>
<p>All R packages that your package needs will be installed automatically based on the package’s <code>DESCRIPTION</code> – no need to add them to the Docker image.</p>
<p>Let’s put the following contents in <code>.circleci/config.yml</code>:</p>
<div>
<pre><code>version: 2
jobs:
  build:
    working_directory: ~/main
    docker:
      - image: appsilon/ci-base:1.0
    steps:
      - checkout
      - run:
          command: |
            R -e 'devtools::install_deps(dependencies = TRUE)'
      - run:
          command: |
            R -e 'devtools::check()'
      - store_artifacts:
          path: man/
          destination: man
</code></pre>
</div>
<p>We start by specifying the CircleCI API version and working directory. Then we declare containers to run tests on by providing a Docker image. You can add additional containers here, e.g. with a database if you have tests that need to talk to external resources (of course these would not be unit tests, but higher levels of <a rel="nofollow" href="https://martinfowler.com/bliki/TestPyramid.html" target="_blank">test pyramid</a> are also valuable.)</p>
<p>Next, we define steps that should be run each time we’re running tests. We check out code from the repo, install package dependencies (as defined in <code>DESCRIPTION</code>) using <em>devtools</em>, and then perform a <code>devtools::check</code> of the package. This is all we need to do because check runs all of the tests.<br>
<label for="non-package-note"><i></i> sticky note</label><span>If your project is not a package, this will look similar, but you will need to change how dependencies are installed and how tests are run (perhaps this is a topic for another short article). </span></p>
<p>Finally, we store generated files in <code>man/</code> as artifacts to be able to download them later.</p>
<h2>Caching R packages library</h2>
<p>The above setup is all that we need. If package check or any other step fails, CircleCI will automatically report this as an error. However, to save our time (and computing power), it’s worth doing one more thing: caching installed R packages. This way we won’t waste time installing all packages from scratch each time tests run.</p>
<p>To cache the R packages library, let’s add a <code>restore_cache</code> and <code>save_cache</code> steps around dependencies installation:</p>
<div>
<pre><code>version: 2
jobs:
  build:
    working_directory: ~/main
    docker:
      - image: appsilon/ci-base:1.0
    steps:
      - checkout
      - restore_cache:
          keys:
            - deps1-{{ .Branch }}-{{ checksum &quot;DESCRIPTION&quot; }}
            - deps1-{{ .Branch }}
            - deps1-
      - run:
          command: |
            R -e 'devtools::install_deps(dependencies = TRUE)'
      - save_cache:
          key: deps1-{{ .Branch }}-{{ checksum &quot;DESCRIPTION&quot; }}
          paths:
            - &quot;/usr/local/lib/R/site-library&quot;
      - run:
          command: |
            R -e 'devtools::check()'
      - store_artifacts:
          path: man/
          destination: man
</code></pre>
</div>
<p>You can read more about caching in <a rel="nofollow" href="https://circleci.com/docs/2.0/caching/" target="_blank">CircleCI docs</a>. I recommend experimenting with cache keys to best fit your scenario.</p>
<h1>Create project on CircleCI</h1>
<p>Our project is ready! The final step is to go <a rel="nofollow" href="https://circleci.com/" target="_blank">circleci.com</a>, log in with your Github account, go to Projects and choose “Setup project” for your project’s repo. This will configure all needed keys on Github and start monitoring changes, so you need to be an admin of that repo to do that. By default, CircleCI will build all new commits, whether they are in a PR or not. You can customize that if you need to. One setting that I particularly recommend is to cancel running builds of a branch if new commits are pushed to this branch. This can save quite some time if you’re iterating quickly on a branch.</p>
<p><img alt="" src="https://i2.wp.com/appsilondatascience.com/blog/assets/article_images/2017-12-06-circleci/green-build.png?w=456" data-recalc-dims="1"></p>
<h1>Using custom Docker image for running tests</h1>
<p>Preparing a Docker image to run our tests is simple as we can use the popular rocker images as a base. Here’s the Dockerfile we used for building <code>appsilon/ci-base:1.0</code>:</p>
<div>
<pre><code>FROM r-base:3.4.1

RUN apt-get update  \
  &amp;&amp; apt-get install git libssl-dev ssh texlive-latex-base texlive-fonts-recommended libcurl4-openssl-dev libxml2-dev -y \
  &amp;&amp; rm -rf /var/lib/apt/lists/*

RUN R -e &quot;install.packages(c('devtools', 'roxygen2'), repos='http://cran.us.r-project.org')&quot;
</code></pre>
</div>
<p>As you can see, on top <code>r-base</code> image, we install several libraries that are required to build an R package. If you need additional libraries for your package, you can add them here as well. We also install two R packages – <code>devtools</code> and <code>roxygen2</code> that are not dependencies of our package, but we need them to build it.</p>
<p>To use your image on CircleCI, you’ll need to build it and push to Dockerhub, which is where CircleCI looks for images. Just run these commands in your <code>Dockerfile</code> directory, replacing <code>appsilon</code> with your Dockerhub account (you’ll need to <code>docker login</code> if you haven’t done this before):</p>
<div>
<pre><code>docker build -t appsilon/ci-base:1.0 .
docker push appsilon/ci-base:1.0
</code></pre>
</div>
<h1>Seeing the results</h1>
<p>Each time you push a new commit, CircleCI will check the package and report the results back to Github. They will be shown in commits and related PRs. You can also add this single line in project’s README.md to display a badge with build status of main branch:</p>
<div>
<pre><code>[![CircleCI](https://circleci.com/gh/Appsilon/ci.example.svg?style=svg)](https://circleci.com/gh/Appsilon/ci.example)
</code></pre>
</div>
<p>I hope this is helpful! Also, you can find a sample project with all configuration (including code linter) on <a rel="nofollow" href="https://github.com/Appsilon/ci.example" target="_blank">GitHub</a>.</p>
<p>Read the original post at<br>
<a rel="nofollow" href="http://appsilondatascience.com/blog/rstats/2018/02/07/circleci.html" target="_blank">Appsilon Data Science Blog</a>.</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://appsilondatascience.com/blog/rstats/2018/02/07/circleci.html"> Appsilon Data Science Blog</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/kzuq1q4xJJ8" height="1"></a></div>