<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/software-engineering-data-sets/"></div></div><div><a data-url="https://www.r-bloggers.com/software-engineering-data-sets/" data-counturl="https://www.r-bloggers.com/software-engineering-data-sets/" data-count="horizontal" data-text="Software engineering data sets" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://shape-of-code.coding-guidelines.com/2016/09/05/software-engineering-data-sets/"> The Shape of Code » R</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>The pretty pictures from my <a target="_blank" href="http://shape-of-code.coding-guidelines.com/2012/06/22/background-to-my-book-project-empirical-software-engineering-with-r/">empirical software engineering book</a> are now <a target="_blank" href="http://www.knosof.co.uk/ESEUR/figures/index.html">online</a>, along with the <a target="_blank" href="https://github.com/Derek-Jones/ESEUR-code-data">210 data sets and R code</a> (330M).</p>
<p>Plotting the number of data sets in each year shows that empirical software engineering has really taken off in the last 10 years (<a target="_blank" href="http://www.coding-guidelines.com/R_code/datayear.tgz">code+data</a>). Around dozen or so confidential data sets are not included; I am only writing about data that can be made public.</p>
<p><img alt="Number of data sets per year" src="https://i2.wp.com/www.coding-guidelines.com/images/datayear.jpeg?w=456" data-recalc-dims="1"></p>
<p>It used to be rare to find the data associated with a paper on the author’s website. Of course, before around 1995 there was no web, but since around <a target="_blank" href="http://shape-of-code.coding-guidelines.com/2014/07/31/success-software-engineering-data-is-starting-to-become-very-dull/">2012 the idea has started to take off</a>.</p>
<p>Contact via email goes back to 1985 and before that people sent <a target="_blank" href="https://en.wikipedia.org/wiki/Magnetic_tape_data_storage">mag tapes</a> through the post and many years ago somebody sent me <a target="_blank" href="https://en.wikipedia.org/wiki/Punched_tape">punched tape</a> (there is nothing like seeing the bits with the naked eye).</p>
<p>I have sent several hundred emails asking for data and received 55 data sets. I’m hoping this release will spur those who have promised me data to invest some time to send it.</p>
<p>My experience is that research data often lives on laptops and dies when the laptop is replaced (a <a target="_blank" href="https://www.ncbi.nlm.nih.gov/pubmed/24361065">study of biologists</a>, who have been collecting data for hundreds of years, found a data ‘death rate’ of 17% a year). Had I started actively collecting data before 2010 the red line in the plot would be much higher for earlier years; I often received data from authors when writing my <a target="_blank" href="http://www.knosof.co.uk/cbook">C book</a> at the start of the century (Google went from nothing to being the best place to search, while I wrote).</p>
<p>In nine cases I extracted the data, either from the <a target="_blank" href="http://shape-of-code.coding-guidelines.com/2013/12/19/converting-graphs-in-pdf-files-to-csv-format/">pdf</a> or an <a target="_blank" href="http://shape-of-code.coding-guidelines.com/2015/03/04/extracting-the-original-data-from-a-heatmap-image/">image</a> and then <a target="_blank" href="http://shape-of-code.coding-guidelines.com/2015/06/17/extracting-absolute-values-from-percentage-data/">reverse engineered values</a>.</p>
<p>I have around 50 data sets waiting to be processed. Given that lots more are bound to arrive before the book is finished, I expect to easily reach the 300 mark. A tiny number given my aim of writing about all software engineering issues for which public data exists.</p>
<p>If you know of interesting software engineering data, that is not to be found in these plots, please let me know.</p>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="https://www.r-bloggers.com/software-engineering-data-sets/"></div></div><div><a data-url="https://www.r-bloggers.com/software-engineering-data-sets/" data-counturl="https://www.r-bloggers.com/software-engineering-data-sets/" data-count="vertical" data-text="Software engineering data sets" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://shape-of-code.coding-guidelines.com/2016/09/05/software-engineering-data-sets/"> The Shape of Code » R</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/YLKp3ZzTcpk"></a></div>