<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/exploring-handwritten-digit-classification-a-tidy-analysis-of-the-mnist-dataset/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://varianceexplained.org/r/digit-eda/"> Variance Explained</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>In a <a rel="nofollow" href="http://varianceexplained.org/r/ds-ml-ai/" target="_blank">recent post</a>, I offered a definition of the distinction between data science and machine learning: that data science is focused on extracting insights, while machine learning is interested in making predictions. I also noted that the two fields greatly overlap:</p>
<blockquote>
<p>I use both machine learning and data science in my work: I might fit a model on Stack Overflow traffic data to determine which users are likely to be looking for a job (machine learning), but then construct summaries and visualizations that examine why the model works (data science). This is an important way to discover flaws in your model, and to combat algorithmic bias. This is one reason that data scientists are often responsible for developing machine learning components of a product.</p>
</blockquote>
<p>I’d like to further explore how data science and machine learning complement each other, by demonstrating how I would use data science to approach a problem of image classification. We’ll work with a classic machine learning challenge: the <a rel="nofollow" href="https://en.wikipedia.org/wiki/MNIST_database" target="_blank">MNIST digit database</a>.</p>
<p><img alt="" src="https://i1.wp.com/varianceexplained.org/images/mnist.png?w=456" data-recalc-dims="1"></p>
<p>The challenge is to classify a handwritten digit based on a 28-by-28 black and white image. MNIST is often credited as one of the first datasets to prove the effectiveness of neural networks.</p>
<p>In a series of posts, I’ll be training classifiers to recognize digits from images, while using data exploration and visualization to build our intuitions about why each method works or doesn’t. Like most of my posts I’ll be analyzing the data through tidy principles, particularly using the dplyr, tidyr and ggplot2 packages. In this first post we’ll focus on <strong>exploratory data analysis</strong>, to show how you can better understand your data before you start training classification algorithms or measuring accuracy. This will help when we’re choosing a model or transforming our features.</p>
<h3>Preprocessing</h3>
<p>The default MNIST dataset is somewhat inconveniently formatted, but <a rel="nofollow" href="https://pjreddie.com/projects/mnist-in-csv/" target="_blank">Joseph Redmon has helpfully created a CSV-formatted version</a>. We can download it with the <code>readr</code> package.</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>readr</span><span>)</span><span><br data-jekyll-commonmark-ghpages=""></span><span>library</span><span>(</span><span>dplyr</span><span>)</span><span><br data-jekyll-commonmark-ghpages=""><br data-jekyll-commonmark-ghpages=""></span><span>mnist_raw</span><span> </span><span>&lt;-</span><span> </span><span>read_csv</span><span>(</span><span>&quot;https://pjreddie.com/media/files/mnist_train.csv&quot;</span><span>,</span><span> </span><span>col_names</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>)</span></code></pre>
</figure>
<figure>
<pre><code data-lang="r"><span>mnist_raw</span><span>[</span><span>1</span><span>:</span><span>10</span><span>,</span><span> </span><span>1</span><span>:</span><span>10</span><span>]</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## # A tibble: 10 x 10<br data-jekyll-commonmark-ghpages="">##       X1    X2    X3    X4    X5    X6    X7    X8    X9   X10<br data-jekyll-commonmark-ghpages="">##    </code></pre></figure>