<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/fragment-running-multiple-services-such-as-jupyter-notebooks-and-a-postgres-database-in-a-single-docker-container/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://blog.ouseful.info/2018/09/12/fragment-running-multiple-services-in-a-single-docker-container/"> Rstats – OUseful.Info, the blog…</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Over the last couple of days, I’ve been fettling the build scripts for the TM351 VM, which typically uses <code>vagrant</code> to build a <em>VirtualBox</em> VM from a set of shell scripts, so they can be used to build a single Docker container that runs all the TM351 services, specifically Jupyter notebooks, OpenRefine, PostgreSQL and MongoDB.</p>
<p>Docker containers are typically constructed to a run a single service, with compositions of containers wired together using Docker Compose to create applications that deliver, or rely on, more than one running service. For example, in a previous post (<a rel="nofollow" href="https://blog.ouseful.info/2017/09/06/ergast-local-api-server/" target="_blank">Setting up a Containerised Desktop API server (MySQL + Apache / PHP 5) for the ergast Motor Racing Data API</a>) I showed how to set up a couple of containers to work together, one running a MySQL database server, the other an http service that provided an API to the database.</p>
<p>So how to run multiple services in the same container? Docs on the Docker website suggest <a rel="nofollow" href="https://docs.docker.com/config/containers/multi-service_container/" target="_blank">using <code>supervisord</code> to run multiple services in a single container</a>, so here’s a fragment on how I’ve done that from my TM351 build.</p>
<p>To begin with, I’ve built the container up as a tiered set of containers, in a similar way to the way the stack of <a rel="nofollow" href="https://github.com/jupyter/docker-stacks" target="_blank">opinionated Jupyter notebook Docker containers</a> are constructed:</p>
<pre>#Define a stub to identify the images in this image stack
IMAGESTUB=psychemedia/tm361testm

# minimal
## Define a minimal container, eg a basic Linux container
## using whatever flavour of Linux we prefer
docker build --rm -t ${IMAGESTUB}-minimal-test ./minimal

# base
## The base container installs core packages
## The intention is to define a common build environment
## populated with packages likely to be common to many courses
docker build --rm --build-arg BASE=${IMAGESTUB}-minimal-test -t ${IMAGESTUB}-base-test ./base

#...</pre>
<p>One of the things I’ve done to try to generalise the build steps is allow the name a base container to be used to bootstrap a new one by passing the name of the base image in via an optional variable (in the above case, <code>--build-arg BASE=${IMAGESTUB}-minimal-test</code>). Each <code>Dockerfile</code> in a build step directory uses the following construction to work out which image to use as the <code>FROM</code> basis:</p>
<pre>#Set ARG values using --build-arg =
#Each ARG value can also have a default value
ARG BASE=psychemedia/ou-tm351-base-test
FROM ${BASE}

</pre>
<p>Using the same approach, I have used separate build tiers for the following components:</p>
<ul>
<li><code>jupyter base</code>: minimal Jupyter notebook install;</li>
<li><code>jupyter custom</code>: add some customisation onto a pre-existing Jupyter notebook install;</li>
<li><code>openrefine</code>: add the OpenRefine application; (note, we could just use <code>BASE=ubuntu</code> to create this a simple, standalone <em>OpenRefine</em> container);</li>
<li><code>postgres</code>: create a seeded PostgreSQL database; note, this could be split into two: a base postgres tier and then a customisation that adds users, creates and seed databases etc;</li>
<li><code>mongodb</code>: add in a seeded mongo database; again, the seeding could be added as an extra tier on a minimal database tier;</li>
<li><code>topup</code>: a tier to add in anything I’ve missed without having to go back to rebuild from an earlier step…</li>
</ul>
<p>The intention behind splitting out these tiers is that we might want to have a battle hardened OU postgres tier, for example, that could be shared between different courses. Alternatively, we might want to have tiers offering customisations for specific presentations of a course, whilst reusing several other fixed tiers intended to last out the life of the course.</p>
<p>By the by, it can be quite handy to poke inside an image once you’ve created it to check that everything is in the right place:</p>
<pre>#Explore inside animage by entering it with a shell command
docker run -it --entrypoint=/bin/bash psychemedia/ou-tm351-jupyter-base-test -i</pre>
<p>Once the services are in place, I add a final layer to the container that ensures <code>supervisord</code> is available and set up with an appropriate <code>supervisord.conf</code> configuration file:</p>
<pre>##Dockerfile
#Final tier Dockerfile
ARG BASE=psychemedia/testpieces
FROM ${BASE}

USER root
RUN apt-get update &amp;&amp; apt-get install -y supervisor

RUN mkdir -p /openrefine_projects  &amp;&amp; chown oustudent:100 /openrefine_projects
VOLUME /openrefine_projects

RUN mkdir -p /notebooks  &amp;&amp; chown oustudent:100 /notebooks
VOLUME /notebooks

RUN mkdir -p /var/log/supervisor
COPY monolithic_container_supervisord.conf /etc/supervisor/conf.d/supervisord.conf

EXPOSE 3334
EXPOSE 8888

CMD [&quot;/usr/bin/supervisord&quot;]
</pre>
<p>The <code>supervisord.conf</code> file is defined as follows:</p>
<pre>##supervisord.conf
##We can check running processes under supervisord with: supervisorctl

[supervisord]
nodaemon=true
logfile=/dev/stdout
loglevel=trace
logfile_maxbytes=0
#The HOME envt needs setting to the correct USER
#otherwise jupyter throws: [Errno 13] Permission denied: '/root/.local'
#https://github.com/jupyter/notebook/issues/1719
environment=HOME=/home/oustudent

[program:jupyternotebook]
#Note the auth is a bit ropey on this atm!
command=/usr/local/bin/jupyter notebook --port=8888 --ip=0.0.0.0 --y --log-level=WARN --no-browser --allow-root --NotebookApp.password= --NotebookApp.token=
#The directory we want to start in
#(replaces jupyter notebook parameter: --notebook-dir=/notebooks)
directory=/notebooks
autostart=true
autorestart=true
startsecs=5
user=oustudent
stdout_logfile=NONE
stderr_logfile=NONE

[program:postgresql]
command=/usr/lib/postgresql/9.5/bin/postgres -D /var/lib/postgresql/9.5/main -c config_file=/etc/postgresql/9.5/main/postgresql.conf
user=postgres
autostart=true
autorestart=true
startsecs=5

[program:mongodb]
command=/usr/bin/mongod --dbpath=/var/lib/mongodb --port=27351
user=mongodb
autostart=true
autorestart=true
startsecs=5

[program:openrefine]
command=/opt/openrefine-3.0-beta/refine -p 3334 -i 0.0.0.0 -d /vagrant/openrefine_projects
user=oustudent
autostart=true
autorestart=true
startsecs=5
stdout_logfile=NONE
stderr_logfile=NONE
</pre>
<p>One thing I need to do better is to find a way to stage the construction of the <code>supervisord.conf</code> file, bearing in mind that multiple tiers may relate to the same servicel for example, I have a <code>jupyter-base</code> tier to create a minimal Jupyter notebook server and then a <code>jupyter-base-custom</code> tier that adds in specific customisations, such as branding and course related notebook extensions.</p>
<p>When the final container is built, the <code>supervisord</code> command is run and the multiple services started.</p>
<p>One other thing to note: we’re hoping to run TM351 environments on an internal OpenStack cluster. The current cluster only allows students to expose a single port, and port 80 at that, from the VM (IP addresses are in scant supply, and network security lockdowns are in place all over the place). The current VM exposes at least two <code>http</code> services: <em>Jupyter notebooks</em> and <em>OpenRefine</em>, so we need a proxy in place if we are to expose them both via a single port. Helpfully, the <code>nbserverproxy</code> Jupyter extension (as described in <a rel="nofollow" href="https://blog.ouseful.info/2018/07/18/exposing-multiple-services-via-a-single-http-port-using-jupyter-nbserverproxy/" target="_blank">Exposing Multiple Services Via a Single http Port Using Jupyter nbserverproxy</a>), allows us to do just that. One thing to note, though – I had to enable it via the same user that launches the notebook server in the <code>suoervisord.conf</code> settings:</p>
<pre>##Dockerfile fragment

RUN $PIP install nbserverproxy

USER oustudent
RUN jupyter serverextension enable --py nbserverproxy
USER root
</pre>
<p>To run the VM, I can call something like:</p>
<p><code>docker run -p 8899:8888 -d psychemedia/tm351dockermonotest</code></p>
<p>and then to access the additional services, I can browse to e.g. <code>localhost:8899/proxy/3334/</code> to see the <em>OpenRefine</em> application.</p>
<p><em>PS in case you’re wondering why I syndicated this through RBloggers too, the same recipe will work if you’re using Jupyter notebooks with an R kernel, rather than the default IPython one.</em></p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://blog.ouseful.info/2018/09/12/fragment-running-multiple-services-in-a-single-docker-container/"> Rstats – OUseful.Info, the blog…</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/79umxbuf-JU" height="1"></a></div>