<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/singularity-as-a-software-distribution-deployment-tool/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://mlampros.github.io/mlampros.github.io/2018/07/26/singularity_containers/"> mlampros</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>In this blog post, I’ll explain how someone can take advantage of <a rel="nofollow" href="https://github.com/singularityware/singularity" target="_blank">Singularity</a> to make R or Python packages available as an image file to users. This is a necessity if the specific R or Python package is difficult to install across different operating systems making that way the installation process cumbersome. Lately, I’ve utilized the <a rel="nofollow" href="https://github.com/rstudio/reticulate" target="_blank">reticulate</a> package in R (it provides an interface between R and Python) and I realized from first hand how difficult it is, in some cases, to install R and Python packages and make them work nicely together in the same operating system. This blog post by no means presents the potential of Singularity or containerization tools, such as <a rel="nofollow" href="https://github.com/docker" target="_blank">docker</a>, but it’s mainly restricted to package distribution / deployment.</p>
<p></p>
<p>Singularity can be installed on all 3 operating systems (Linux, Macintosh, Windows), however the current status (as of July 2018) is that on Macintosh and Windows the user has to setup <a rel="nofollow" href="https://www.vagrantup.com/" target="_blank">Vagrant</a>, and <a rel="nofollow" href="https://www.sylabs.io/guides/2.5.1/user-guide.pdf#page=21" target="_blank">run Singularity</a> from there (<a rel="nofollow" href="https://groups.google.com/a/lbl.gov/forum/#!msg/singularity/7d9F16DCG8Y/iXo_wRxtCwAJ" target="_blank">this might change in the near future</a>).</p>

<h3>Singularity on Linux</h3>
<p></p>
<p>In the following lines I’ll make use of an Ubuntu cloud instance (the same steps can be accomplished on an Ubuntu Desktop with some exceptions) to explain how someone can download Singularity image files and run those images on Rstudio server (in case of R) or a Jupyter Notebook (in case of Python). I utilize Amazon Web Services (<a rel="nofollow" href="https://aws.amazon.com/" target="_blank">AWS</a>) and especially an <em>Ubuntu server 16.04</em> using a <em>t2.micro</em> instance (1GB memory, 1 core), however, someone can follow the same procedure on <a rel="nofollow" href="https://azure.microsoft.com/en-us/" target="_blank">Azure</a> or <a rel="nofollow" href="https://cloud.google.com/" target="_blank">Google Cloud</a> (at least of those two alternative cloud services I’m aware) as well. I’ll skip the steps on how someone can set-up an Ubuntu cloud instance, as it’s beyond the scope of this blog post (there are certainly many tutorials on the web for this purpose).</p>
<p></p>
<p>Assuming someone uses the command line console, the first thing to do is to install the <a rel="nofollow" href="https://github.com/mlampros/singularity_containers/blob/master/system_requirements.sh" target="_blank">system requirements</a> (in case of an Ubuntu Desktop upgrading the system should be skipped). Once the installation of the system requirements is finished the following folder should appear in the home directory,</p>
<p></p>
<div>
<div>
<pre><code><span>
</span><span>singularity</span><span>
</span></code></pre>
</div>
</div>
<h3>R language Singularity image files</h3>
<p></p>
<p>My <a rel="nofollow" href="https://github.com/mlampros/singularity_containers" target="_blank">singularity_containers</a> Github repository contains R and Python <a rel="nofollow" href="https://github.com/singularityhub/singularityhub.github.io/wiki/Build-A-Container" target="_blank">Singularity Recipes</a>, which are used to build the corresponding containers. My Github repository is connected to my <a rel="nofollow" href="https://singularity-hub.org/collections/1321" target="_blank">singularity-hub</a> account and once a change is triggered (for instance, a push to my repository) a new / updated container build will be created. An updated build – for instance for the <a rel="nofollow" href="https://github.com/mlampros/RGF" target="_blank">RGF package</a> – can be pulled from singularity-hub in the following way,</p>
<p></p>
<div>
<div>
<pre><code><span>
</span><span>singularity</span><span> </span><span>pull</span><span> </span><span>--</span><span>name</span><span> </span><span>RGF_r.simg</span><span> </span><span>shub</span><span>://</span><span>mlampros</span><span>/</span><span>singularity_containers</span><span>:</span><span>rgf_r</span><span>
</span></code></pre>
</div>
</div>
<p></p>
<p>This code line will create the <em>RGF_r.simg</em> image file in the home directory. One should now make sure that port 8787 is not used by another service / application by using,</p>
<p></p>
<div>
<div>
<pre><code><span>
</span><span>sudo</span><span> </span><span>netstat</span><span> </span><span>-</span><span>plnt</span><span> </span><span>|</span><span> </span><span>fgrep</span><span> </span><span>8787</span><span>
</span></code></pre>
</div>
</div>
<p></p>
<p>If this does not return something then one can proceed with,</p>
<p></p>
<div>
<div>
<pre><code><span>
</span><span>singularity</span><span> </span><span>run</span><span> </span><span>RGF_r.simg</span><span>
</span></code></pre>
</div>
</div>
<p></p>
<p>to run the image. If everything went ok and no errors occurred then by opening a second command line console and typing,</p>
<p></p>
<div>
<div>
<pre><code><span>
</span><span>sudo</span><span> </span><span>netstat</span><span> </span><span>-</span><span>plnt</span><span> </span><span>|</span><span> </span><span>fgrep</span><span> </span><span>8787</span><span>
</span></code></pre>
</div>
</div>
<p></p>
<p>one should observe that port 8787 is opened,</p>
<p></p>
<div>
<div>
<pre><code><span>
</span><span>tcp</span><span> </span><span>0</span><span> </span><span>0</span><span> </span><span>0.0.0.0</span><span>:</span><span>8787</span><span> </span><span>0.0.0.0</span><span>:*</span><span> </span><span>LISTEN</span><span> </span><span>23062</span><span>/</span><span>rserver</span><span>
</span></code></pre>
</div>
</div>
<p></p>
<p>The final step is to open a web-browser (chrome, firefox etc.) and give,</p>
<p></p>
<ul>
<li>http://Public DNS (IPv4):8787            ( <em>where “Public DNS (IPv4)” is specific to the Cloud instance you launched</em> )</li>
</ul>
<p>or</p>
<ul>
<li>http://0.0.0.0:8787            ( <em>in case that someone uses Singularity locally</em> )</li>
</ul>
<p></p>
<p>to launch the Rstudio-server and use the RGF package pre-installed with all requirements included (to stop the service use <em>CTRL + C</em> from the command line). I used RGF as an example here because for me personally, it was somehow cumbersome to install on my windows machine.</p>
<p>The same applies to the other two <em>R</em> singularity recipe files included in my singularity-hub account, i.e. <em>mlampros/singularity_containers:nmslib_r</em> and <em>mlampros/singularity_containers:fuzzywuzzy_r</em>.</p>
<p></p>
<h3>Python language Singularity image files</h3>
<p></p>
<p>The Python Singularity Recipe files which are also included in the same Github repository utilize port 8888 and follow a similar logic with the R files. The only difference is that when a user <strong>runs</strong> the image the <strong>sudo</strong> command is required (otherwise it will raise a permission error),</p>
<p></p>
<div>
<div>
<pre><code><span>
</span><span>singularity</span><span> </span><span>pull</span><span> </span><span>--</span><span>name</span><span> </span><span>RGF_py.simg</span><span> </span><span>shub</span><span>://</span><span>mlampros</span><span>/</span><span>singularity_containers</span><span>:</span><span>rgf_python</span><span>
</span><span>sudo</span><span> </span><span>singularity</span><span> </span><span>run</span><span> </span><span>RGF_py.simg</span><span>
</span></code></pre>
</div>
</div>
<p></p>
<p>The latter command will produce the following (example) output,</p>
<p></p>
<div>
<div>
<pre><code><span>
</span><span>The</span><span> </span><span>web</span><span>-</span><span>browser</span><span> </span><span>runs</span><span> </span><span>on</span><span> </span><span>localhost</span><span>:</span><span>8888</span><span>
</span><span>[</span><span>I</span><span> </span><span>09</span><span>:</span><span>56</span><span>:</span><span>03.427</span><span> </span><span>NotebookApp</span><span>]</span><span> </span><span>Writing</span><span> </span><span>notebook</span><span> </span><span>server</span><span> </span><span>cookie</span><span> </span><span>secret</span><span> </span><span>to</span><span> </span><span>/</span><span>root</span><span>/</span><span>.local</span><span>/</span><span>share</span><span>/</span><span>jupyter</span><span>/</span><span>runtime</span><span>/</span><span>notebook_cookie_secret</span><span>
</span><span>[</span><span>W</span><span> </span><span>09</span><span>:</span><span>56</span><span>:</span><span>03.779</span><span> </span><span>NotebookApp</span><span>]</span><span> </span><span>WARNING</span><span>:</span><span> </span><span>The</span><span> </span><span>notebook</span><span> </span><span>server</span><span> </span><span>is</span><span> </span><span>listening</span><span> </span><span>on</span><span> </span><span>all</span><span> </span><span>IP</span><span> </span><span>addresses</span><span> </span><span>and</span><span> </span><span>not</span><span> </span><span>using</span><span> </span><span>encryption.</span><span> </span><span>This</span><span> </span><span>is</span><span> </span><span>not</span><span> </span><span>recommended.</span><span>
</span><span>[</span><span>I</span><span> </span><span>09</span><span>:</span><span>56</span><span>:</span><span>03.789</span><span> </span><span>NotebookApp</span><span>]</span><span> </span><span>Serving</span><span> </span><span>notebooks</span><span> </span><span>from</span><span> </span><span>local</span><span> </span><span>directory</span><span>:</span><span> </span><span>/</span><span>root</span><span>
</span><span>[</span><span>I</span><span> </span><span>09</span><span>:</span><span>56</span><span>:</span><span>03.790</span><span> </span><span>NotebookApp</span><span>]</span><span> </span><span>The</span><span> </span><span>Jupyter</span><span> </span><span>Notebook</span><span> </span><span>is</span><span> </span><span>running</span><span> </span><span>at</span><span>:</span><span>
</span><span>[</span><span>I</span><span> </span><span>09</span><span>:</span><span>56</span><span>:</span><span>03.790</span><span> </span><span>NotebookApp</span><span>]</span><span> </span><span>http</span><span>://</span><span>(</span><span>ip</span><span>-172-31-21-76</span><span> </span><span>or</span><span> </span><span>127.0.0.1</span><span>)</span><span>:</span><span>8888</span><span>/?</span><span>token</span><span>=</span><span>1</span><span>fc90f01247498dac8d24ac918fe8da57fa46ee9e98eea4f</span><span>
</span><span>[</span><span>I</span><span> </span><span>09</span><span>:</span><span>56</span><span>:</span><span>03.790</span><span> </span><span>NotebookApp</span><span>]</span><span> </span><span>Use</span><span> </span><span>Control</span><span>-</span><span>C</span><span> </span><span>to</span><span> </span><span>stop</span><span> </span><span>this</span><span> </span><span>server</span><span> </span><span>and</span><span> </span><span>shut</span><span> </span><span>down</span><span> </span><span>all</span><span> </span><span>kernels</span><span> </span><span>(</span><span>twice</span><span> </span><span>to</span><span> </span><span>skip</span><span> </span><span>confirmation</span><span>)</span><span>.</span><span>
</span><span>[</span><span>C</span><span> </span><span>09</span><span>:</span><span>56</span><span>:</span><span>03.790</span><span> </span><span>NotebookApp</span><span>]</span><span> </span><span>Copy</span><span>/</span><span>paste</span><span> </span><span>this</span><span> </span><span>URL</span><span> </span><span>into</span><span> </span><span>your</span><span> </span><span>browser</span><span> </span><span>when</span><span> </span><span>you</span><span> </span><span>connect</span><span> </span><span>for</span><span> </span><span>the</span><span> </span><span>first</span><span> </span><span>time</span><span>,</span><span>
</span><span>to</span><span> </span><span>login</span><span> </span><span>with</span><span> </span><span>a</span><span> </span><span>token</span><span>:</span><span>
</span><span>http</span><span>://</span><span>(</span><span>ip</span><span>-172-31-21-76</span><span> </span><span>or</span><span> </span><span>127.0.0.1</span><span>)</span><span>:</span><span>8888</span><span>/?</span><span>token</span><span>=</span><span>1</span><span>fc90f01247498dac8d24ac918fe8da57fa46ee9e98eea4f</span><span>
</span><span>.......</span><span>
</span></code></pre>
</div>
</div>
<p></p>
<p>In the same way as before the user should open a web-browser and give either,</p>
<p></p>
<ul>
<li>http://Public DNS (IPv4):8888            ( <em>where “Public DNS (IPv4)” is specific to the Cloud instance you launched</em> )</li>
</ul>
<p>or</p>
<ul>
<li>http://127.0.0.1:8888            ( <em>in case that someone uses Singularity locally</em> )</li>
</ul>
<p></p>
<p>When someone connects for the first time to the Jupyter notebook then he / she has to give the output token as the password. For instance, based on the previous example output the token password would be <em>1fc90f01247498dac8d24ac918fe8da57fa46ee9e98eea4f</em>.</p>
<p>I also included an <a rel="nofollow" href="https://github.com/mlampros/singularity_containers/blob/master/Python/regression_RGF.ipynb" target="_blank">.ipynb</a> file which can be loaded to the Jupyter notebook to test the <a rel="nofollow" href="https://github.com/RGF-team/rgf/tree/master/python-package" target="_blank">rgf_python</a> package.</p>
<p>The same applies to the other two <em>Python</em> singularity recipe files included in my singularity-hub account, i.e. <em>mlampros/singularity_containers:nmslib_python</em> and <em>mlampros/singularity_containers:fuzzywuzzy_python</em>.</p>
<p></p>
<h3>Final words</h3>
<p></p>
<p>If someone intends to add authentication to the Singularity recipe files then valuable resources can be found in the <a rel="nofollow" href="https://github.com/nickjer/singularity-rstudio" target="_blank">https://github.com/nickjer/singularity-rstudio</a> Github repository, on which my Rstudio-server recipes heavily are based.</p>
<p></p>
<p>An updated version of <em>singularity_containers</em> can be found in my <a rel="nofollow" href="https://github.com/mlampros/singularity_containers" target="_blank">Github repository</a> and to report bugs   /   issues please use the following link, <a rel="nofollow" href="https://github.com/mlampros/singularity_containers/issues" target="_blank">https://github.com/mlampros/singularity_containers/issues</a>.</p>

<p><strong>References</strong> :</p>
<ul>
<li>https://github.com/singularityhub</li>
<li>https://vsoch.github.io/</li>
<li>https://github.com/nickjer/singularity-rstudio</li>
<li>https://github.com/nickjer/singularity-r</li>
<li>https://bwlewis.github.io/r-and-singularity/</li>
</ul>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://mlampros.github.io/mlampros.github.io/2018/07/26/singularity_containers/"> mlampros</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/JvK1L3lEbm4" height="1"></a></div>