<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/its-a-fake-%f0%9f%93%a6-revisiting-trust-in-foss-ecosystems/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://rud.is/b/2017/09/15/its-a-fake-%F0%9F%93%A6-revisiting-trust-in-foss-ecosystems/"> R – rud.is</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>I’ve blathered about trust before <sup><a rel="nofollow" href="https://rud.is/b/2017/02/23/on-watering-holes-trust-defensible-systems-and-data-science-community-security/" target="_blank">1</a></sup> <sup><a rel="nofollow" href="https://ropensci.org/blog/blog/2017/07/25/notary" target="_blank">2</a></sup>, but said blatherings were in a <em>“what if”</em> context. Unfortunately, the <em>if</em> has turned into a <em>when</em>, which begged for further blathering on a recent FOSS ecosystem cybersecurity incident.</p>
<p>The gg_spiffy <a rel="nofollow" href="http://twitter.com/thomasp85" target="_blank">@thomasp85</a> linked to a <a rel="nofollow" href="http://www.nbu.gov.sk/skcsirt-sa-20170909-pypi/" target="_blank">post by the SK-CSIRT</a> detailing the discovery and take-down of a series of malicious Python packages. Here’s their high-level incident summary:</p>
<div>
<p>SK-CSIRT identified malicious software libraries in the official Python package<br>
repository, PyPI, posing as well known libraries. A prominent example is a fake<br>
package urllib-1.21.1.tar.gz, based upon a well known package<br>
urllib3-1.21.1.tar.gz.</p>
<p>Such packages may have been downloaded by unwitting developer or administrator<br>
by various means, including the popular “pip” utility (pip install urllib).<br>
There is evidence that the fake packages have indeed been downloaded and<br>
incorporated into software multiple times between June 2017 and September 2017.</p>
</div>
<p>Words are great but, <i>unlike some other FOSS projects <span>(*cough* R *cough*)</span></i> the PyPI folks have <a rel="nofollow" href="https://bigquery.cloud.google.com/dataset/the-psf:pypi" target="_blank"><strong>authoritative</strong> log data</a> regarding package downloads from PyPI. This means we can begin to quantify the exposure. The Google BigQuery SQL was pretty straightforward:</p>
<pre><code>SELECT timestamp, file.project as package, country_code, file.version AS version
FROM (
  (TABLE_DATE_RANGE([the-psf:pypi.downloads], TIMESTAMP('2016-01-22'), TIMESTAMP('2017-09-15')))
)
WHERE file.project IN ('acqusition', apidev-coop', 'bzip', 'crypt', 'django-server',
                       'pwd', 'setup-tools', 'telnet', 'urlib3', 'urllib')</code></pre>
<p>Let’s see what the daily downloads of the malicious package look like:</p>
<p><a rel="nofollow" href="https://i0.wp.com/rud.is/b/wp-content/uploads/2017/09/download_counts_per_day-1.png?ssl=1" target="_blank"><img alt="" src="https://i0.wp.com/rud.is/b/wp-content/uploads/2017/09/download_counts_per_day-1.png?resize=780%2C532&ssl=1" class="wp-image-6265" data-recalc-dims="1"></a></p>
<p>But, we need counts of the mal-package dopplegangers (i.e. the good packages) to truly understand scope of exposure:</p>
<p><a rel="nofollow" href="https://i0.wp.com/rud.is/b/wp-content/uploads/2017/09/dopple_counts_per_day-1.png?ssl=1" target="_blank"><img alt="" src="https://i0.wp.com/rud.is/b/wp-content/uploads/2017/09/dopple_counts_per_day-1.png?resize=780%2C532&ssl=1" class="wp-image-6267" data-recalc-dims="1"></a></p>
<p>Thankfully, the CK-CSIRT folks caught this in time and the exposure was limited. But those are some popular tools that were targeted and it’s super-easy to sneak these into <code>requirements.txt</code> and scripts since the names are similar and the functionality is duplicated.</p>
<p>I’ll further note that the <code>crypto</code> package was “good” at some point in time then went away and was replaced with the nefarious one. That seems like a pretty big PyPI oversight (vis-a-vis package retirement &amp; name re-use), but I’m not casting stones. R’s <code>devtools::install_github()</code> and wanton <code>source()</code>ing are just as bad, and the non-CRAN ecosystem is an even more varmint-prone “wild west” environment.</p>
<p>Furthermore, this is a potential exposure issue in many FOSS package repository ecosystems. On the one hand, these are open environments with tons of room for experimentation, creativity and collaboration. On the other hand, they’re all-too-easy targets for malicious hackers to prey upon. </p>
<p>I, unfortunately, have no quick-fix solutions to offer. “Review your code and dependencies” is about the best I can suggest until individual ecosystems work on better integrity &amp; authenticity controls or there is a cross-ecosystem effort to establish “best practices” and perhaps even staffed, verified, audited, free services that work like a sheriff+notary to help ensure the safety of projects relying on open source components.</p>
<p>Python folks: double check that you weren’t a victim here (it’s super easy to type some of those package names wrong, and hopefully you’ve noticed builds failing if you had done so). </p>
<p>R folks: don’t be smug, watch your GitHub dependencies and double check your projects.</p>
<p>You can find the data and the scripts used to generate the charts (ironically enough) <a rel="nofollow" href="https://github.com/hrbrmstr/pypi-malicious-packages" target="_blank">on GitHub</a>.</p>
<p>Finally: I just want to close with a “thank you!” to PyPI’s Donald Stufft who (quickly!) pointed me to a <a rel="nofollow" href="https://langui.sh/2016/12/09/data-driven-decisions/" target="_blank">blog post</a> detailing the BigQuery setup.</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://rud.is/b/2017/09/15/its-a-fake-%F0%9F%93%A6-revisiting-trust-in-foss-ecosystems/"> R – rud.is</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/7vRnDhPZsbw" height="1"></a></div>