<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="http://www.r-bloggers.com/releasing-the-stacklite-dataset-of-stack-overflow-questions-and-tags/"></div></div><div><a data-url="http://www.r-bloggers.com/releasing-the-stacklite-dataset-of-stack-overflow-questions-and-tags/" data-counturl="http://www.r-bloggers.com/releasing-the-stacklite-dataset-of-stack-overflow-questions-and-tags/" data-count="horizontal" data-text="Releasing the StackLite dataset of Stack Overflow questions and tags" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://varianceexplained.org/r/stack-lite/"> Variance Explained</a></strong>, and kindly contributed to <a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>At Stack Overflow we’ve always been committed to sharing data: all content contributed to the site is <a target="_blank" href="https://creativecommons.org/licenses/by-sa/3.0/">CC-BY-SA</a> licensed, and we release regular <a target="_blank" href="https://archive.org/details/stackexchange">“data dumps”</a> of our entire history of questions and answers.</p>
<p>I’m excited to announce a new resource specially aimed at data scientists, analysts and other researchers, which we’re calling the <a target="_blank" href="https://github.com/dgrtwo/StackLite">StackLite dataset</a>.</p>
<h2>What’s in the StackLite dataset?</h2>
<p>For each Stack Overflow question asked since the beginning of the site, the dataset includes:</p>
<ul>
<li>Question ID</li>
<li>Creation date</li>
<li>Closed date, if applicable</li>
<li>Deletion date, if applicable</li>
<li>Score</li>
<li>Owner user ID (except for deleted questions)</li>
<li>Number of answers</li>
<li>Tags</li>
</ul>
<p>This is ideal for performing analyses such as:</p>
<ul>
<li>The increase or decrease in questions in each tag over time</li>
<li>Correlations among tags on questions</li>
<li>Which tags tend to get higher or lower scores</li>
<li>Which tags tend to be asked on weekends vs weekdays</li>
<li>Rates of question closure or deletion over time</li>
<li>The speed at which questions are closed or deleted</li>
</ul>
<h3>Examples in R</h3>
<p>The dataset is provided as csv.gz files, which means you can use almost any language or statistical tool to process it. But here I’ll share some examples of a simple analysis in R.</p>
<p>The question data and the question-tag pairings are stored separately. You can read in the dataset (once you’ve cloned or downloaded it <a target="_blank" href="https://github.com/dgrtwo/StackLite">from GitHub</a>) with:</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>readr</span><span>)</span><span>
</span><span>library</span><span>(</span><span>dplyr</span><span>)</span><span>
</span><span>questions</span><span> </span><span>&lt;-</span><span> </span><span>read_csv</span><span>(</span><span>&quot;stacklite/questions.csv.gz&quot;</span><span>)</span><span>
</span><span>question_tags</span><span> </span><span>&lt;-</span><span> </span><span>read_csv</span><span>(</span><span>&quot;stacklite/question_tags.csv.gz&quot;</span><span>)</span></code></pre>
</figure>
<p>The <code>questions</code> file has one row for each question:</p>
<figure>
<pre><code data-lang="r"><span>questions</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## Source: local data frame [15,497,156 x 7]
## 
##       Id        CreationDate          ClosedDate        DeletionDate Score
##    (int)              (time)              (time)              (time) (int)
## 1      1 2008-07-31 21:26:37                &lt;NA> 2011-03-28 00:53:47     1
## 2      4 2008-07-31 21:42:52                &lt;NA>                &lt;NA>   406
## 3      6 2008-07-31 22:08:08                &lt;NA>                &lt;NA>   181
## 4      8 2008-07-31 23:33:19 2013-06-03 04:00:25 2015-02-11 08:26:40    42
## 5      9 2008-07-31 23:40:59                &lt;NA>                &lt;NA>  1286
## 6     11 2008-07-31 23:55:37                &lt;NA>                &lt;NA>  1046
## 7     13 2008-08-01 00:42:38                &lt;NA>                &lt;NA>   415
## 8     14 2008-08-01 00:59:11                &lt;NA>                &lt;NA>   265
## 9     16 2008-08-01 04:59:33                &lt;NA>                &lt;NA>    65
## 10    17 2008-08-01 05:09:55                &lt;NA>                &lt;NA>    96
## ..   ...                 ...                 ...                 ...   ...
## Variables not shown: OwnerUserId (int), AnswerCount (int)</code></pre>
</figure>
<p>While the <code>question_tags</code> file has one row for each question-tag pair:</p>
<figure>
<pre><code data-lang="r"><span>question_tags</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## Source: local data frame [45,510,638 x 2]
## 
##       Id                 Tag
##    (int)               (chr)
## 1      1                data
## 2      4                  c#
## 3      4            winforms
## 4      4     type-conversion
## 5      4             decimal
## 6      4             opacity
## 7      6                html
## 8      6                 css
## 9      6                css3
## 10     6 internet-explorer-7
## ..   ...                 ...</code></pre>
</figure>
<p>As one example, you could find the most popular tags:</p>
<figure>
<pre><code data-lang="r"><span>question_tags</span><span> </span><span>%>%</span><span>
</span><span>count</span><span>(</span><span>Tag</span><span>,</span><span> </span><span>sort</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## Source: local data frame [55,661 x 2]
## 
##           Tag       n
##         (chr)   (int)
## 1  javascript 1454248
## 2        java 1409336
## 3         php 1241691
## 4          c# 1208953
## 5     android 1163810
## 6      jquery  931882
## 7      python  732188
## 8        html  690762
## 9         ios  573246
## 10        c++  571386
## ..        ...     ...</code></pre>
</figure>
<p>Or plot the number of questions asked per week:</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>ggplot2</span><span>)</span><span>
</span><span>library</span><span>(</span><span>lubridate</span><span>)</span><span>
</span><span>questions</span><span> </span><span>%>%</span><span>
</span><span>count</span><span>(</span><span>Week</span><span> </span><span>=</span><span> </span><span>round_date</span><span>(</span><span>CreationDate</span><span>,</span><span> </span><span>&quot;week&quot;</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>Week</span><span>,</span><span> </span><span>n</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_line</span><span>()</span></code></pre>
</figure>
<p><img alt="center" src="http://i0.wp.com/varianceexplained.org/figs/2016-07-18-stack-lite/questions_per_day-1.png?w=456" data-recalc-dims="1"></p>
<p>Many of the most interesting issues you can examine involve tags, which describe the programming language or technology used in a question. You could compare the growth or decline of particular tags over time:</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>lubridate</span><span>)</span><span>
</span><span>tags</span><span> </span><span>&lt;-</span><span> </span><span>c</span><span>(</span><span>&quot;c#&quot;</span><span>,</span><span> </span><span>&quot;javascript&quot;</span><span>,</span><span> </span><span>&quot;python&quot;</span><span>,</span><span> </span><span>&quot;r&quot;</span><span>)</span><span>
</span><span>q_per_year</span><span> </span><span>&lt;-</span><span> </span><span>questions</span><span> </span><span>%>%</span><span>
</span><span>count</span><span>(</span><span>Year</span><span> </span><span>=</span><span> </span><span>year</span><span>(</span><span>CreationDate</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>rename</span><span>(</span><span>YearTotal</span><span> </span><span>=</span><span> </span><span>n</span><span>)</span><span>
</span><span>tags_per_year</span><span> </span><span>&lt;-</span><span> </span><span>question_tags</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>Tag</span><span> </span><span>%in%</span><span> </span><span>tags</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>inner_join</span><span>(</span><span>questions</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>count</span><span>(</span><span>Year</span><span> </span><span>=</span><span> </span><span>year</span><span>(</span><span>CreationDate</span><span>),</span><span> </span><span>Tag</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>inner_join</span><span>(</span><span>q_per_year</span><span>)</span><span>
</span><span>ggplot</span><span>(</span><span>tags_per_year</span><span>,</span><span> </span><span>aes</span><span>(</span><span>Year</span><span>,</span><span> </span><span>n</span><span> </span><span>/</span><span> </span><span>YearTotal</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>Tag</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_line</span><span>()</span><span> </span><span>+</span><span>
</span><span>scale_y_continuous</span><span>(</span><span>labels</span><span> </span><span>=</span><span> </span><span>scales</span><span>::</span><span>percent_format</span><span>())</span><span> </span><span>+</span><span>
</span><span>ylab</span><span>(</span><span>&quot;% of Stack Overflow questions with this tag&quot;</span><span>)</span></code></pre>
</figure>
<p><img alt="center" src="http://i2.wp.com/varianceexplained.org/figs/2016-07-18-stack-lite/tags_per_year-1.png?w=456" data-recalc-dims="1"></p>
<h3>How this compares to other Stack Overflow resources</h3>
<p>Almost all of this data is already public within the <a target="_blank" href="https://archive.org/details/stackexchange">Stack Exchange Data Dump</a>. But the official data dump requires a lot of computational overhead to download and process (the Posts fit in a 27 GB XML file), even if the question you want to ask is very simple. The StackLite dataset, in contrast, is designed to be easy to read in and start analyzing. (For example, I was really impressed with Joshua Kunst’s <a target="_blank" href="http://jkunst.com/r/what-do-we-ask-in-stackoverflow/">analysis of tags over time</a>, and want to make it straightforward for others to write posts like that).</p>
<p>Similarly, this data can be examined within the <a target="_blank" href="https://data.stackexchange.com/">Stack Exchange Data Explorer</a> (SEDE), but it requires working with separate queries that each return at most 50,000 rows. The StackLite dataset offers analysts the chance to work with the data locally using their tool of choice.</p>
<h3>Enjoy!</h3>
<p>I’m hoping other analysts find this dataset interesting, and use it to perform meaningful and open research. (Be sure to comment below if you do!)</p>
<p>I’m especially happy to have this dataset public and easily accessible, since it gives me the chance to blog more analyses of Stack Overflow questions and tags while keeping my work reproducible and extendable by others. Keep an eye out for such posts in the future!</p>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="http://www.r-bloggers.com/releasing-the-stacklite-dataset-of-stack-overflow-questions-and-tags/"></div></div><div><a data-url="http://www.r-bloggers.com/releasing-the-stacklite-dataset-of-stack-overflow-questions-and-tags/" data-counturl="http://www.r-bloggers.com/releasing-the-stacklite-dataset-of-stack-overflow-questions-and-tags/" data-count="vertical" data-text="Releasing the StackLite dataset of Stack Overflow questions and tags" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://varianceexplained.org/r/stack-lite/"> Variance Explained</a></strong>.</div>
<hr>
<a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="http://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="http://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="http://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="http://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="http://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="http://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="http://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="http://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="http://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="http://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="http://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="http://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="http://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="http://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="http://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="http://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/6GoVm1xC4Sg"></a></div>