<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/what-you-need-to-know-about-data-augmentation-for-machine-learning/"></div></div><div><a data-url="https://www.r-bloggers.com/what-you-need-to-know-about-data-augmentation-for-machine-learning/" data-counturl="https://www.r-bloggers.com/what-you-need-to-know-about-data-augmentation-for-machine-learning/" data-count="horizontal" data-text="What you need to know about data augmentation for machine learning" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://cartesianfaith.com/2016/10/06/what-you-need-to-know-about-data-augmentation-for-machine-learning/"> R – Cartesian Faith</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Plentiful high-quality data is the key to great machine learning models. But good data doesn’t grow on trees, and that scarcity can impede the development of a model. One way to get around a lack of data is to augment your dataset. Smart approaches to programmatic data augmentation can increase the size of your training set 10-fold or more. Even better, your model will often be more robust (and prevent overfitting) and can even be simpler due to a better training set.</p>
<p>There are many approaches to augmenting data. The simplest approaches include adding noise and applying transformations on existing data. Imputation and dimensional reduction can be used to add samples in sparse areas of the dataset. More advanced approaches include simulation of data based on <a target="_blank" href="https://arxiv.org/abs/1605.01534">dynamic systems</a> or <a target="_blank" href="http://bioinformatics.oxfordjournals.org/content/25/13/1602.long">evolutionary systems</a>. In this post we’ll focus on the two simplest approaches: adding noise and applying transformations.</p>
<h3>Regression Problems</h3>
<p>In many datasets we expect that there is unavoidable statistical noise due to sampling and other factors. For regression problems, we can explicitly add noise to our explanatory variables. Doing so can make the model more robust, although we need to take care when constructing the noise term. First, we want to avoid adding bias. We also need to ensure the noise is independent.</p>
<p>In my <a target="_blank" href="https://cartesianfaith.com/2016/09/23/a-primer-on-universal-function-approximation-with-deep-learning-in-torch-and-r/">function approximation example</a>, I demonstrated creating a simple neural network in Torch to approximate a function. Let’s use the same dataset and function but this time add noise as a preprocessing step. Before, I generated the training data directly in Torch. However, in my <a target="_blank" href="https://cartesianfaith.com/2016/09/29/a-simple-workflow-for-deep-learning/">simple workflow for deep learning</a>, I said I prefer using R for everything but training the model. Hence, I generate and add noise in R and then write out a headerless CSV file. This process expands the training set from 40k samples to ~160k samples.</p>
<p>The following function accomplishes this and uses an arbitrary function to add noise. The default function adds uniform noise.</p>
<pre>
perturb_unif &lt;- function(df, mult=4, fr=function(a) a + runif(length(a)) - .5) {
  fn &lt;- function(i) data.frame(x=fr(df[,1]), y=fr(df[,2]), z=df[,3])
  o &lt;- lapply(1:mult, fn)
  do.call(rbind, o)
}
</pre>
<p><em><strong>Exercise</strong>: What is the purpose of the -0.5 term?</em><br>
<em><strong>Exercise</strong>: What are some other valid noise functions? Why would you choose one over another?</em></p>
<p>I then read this file into Lua using a custom function <code>loadTrainingSet</code>, which is part of my <a target="_blank" href="https://github.com/zatonovo/deep_learning_ex">deep_learning_ex</a> guide. This function reads the CSV and creates a Torch-compatible table comprising the input and output tensors. The function simply assumes that the last column of the CSV is the output.</p>
<p>Using this approach, it’s possible to create a 20 hidden node network that performs as well as the 40 node network in the earlier post. Think about this: by adding noise (and increasing the size of the training set), we’ve managed to reduce the complexity of the network 2-fold.</p>
<div data-shortcode="caption" class="wp-caption"><img alt="Adding noise makes the model simpler and more robust" class="wp-image-2235" src="https://i0.wp.com/cartesianfaith.files.wordpress.com/2016/10/result_w_unif.png?resize=456%2C304&ssl=1" data-recalc-dims="1">
<p class="wp-caption-text">Adding noise simplifies the model and makes it more robust</p>
</div>
<p><em><strong>Kool-Aid Alert</strong>: Depending on the domain and hyper parameters chosen, this approach may not produce desirable results. As with most deep learning exercises, tuning of hyper parameters is mandatory. </em></p>
<h3>Classification Problems</h3>
<p>Noise can be used in classification problems as well. One particularly useful application is balancing a dataset. In a binary classification problem, suppose the data is split 80-20 between two classes. It is well known that such an unbalanced set is problematic for machine learning algorithms. Some will even just default to the class with the higher proportion of observations, since the naïve accuracy is reasonable. In small datasets balancing the dataset by trimming can be counterproductive. The alternative is to increase the samples in the smaller class. The same noise augmentation approach works well here.</p>
<p>Another group of classification problem is image classification. Following a similar approach, noise can be added to images, which can make the model more robust. Another popular technique is transforming the data. This makes sense for images since changes in perspective can change the apparent shape of an object. Transparent or reflective objects can also distort objects, but despite this distortion, we know the object to be the same. Affine transformations provide simple linear transforms that can expand a dataset. This includes shifting, scaling, rotating, flipping, etc. While a good starting point, andsome problems might benefit from more complex transformations.</p>
<div class="wp-caption"><img alt="Still a cat" src="https://i2.wp.com/s-media-cache-ak0.pinimg.com/564x/52/cb/44/52cb44e1856d890cf36c711159583f10.jpg?resize=456%2C342&ssl=1" data-recalc-dims="1">
<p class="wp-caption-text">Still a cat</p>
</div>
<p>Transformed images can be generated in a pre-processing step as above. The <a target="_blank" href="https://github.com/torch/image">Torch image toolkit</a> can be used this way. For example, here is how to flip an image horizontally.</p>
<pre>
img = image.load(&quot;cat.jpg&quot;,3,'byte')
img1 = image.hflip(img)
</pre>
<div data-shortcode="caption" class="wp-caption"><img alt="cat" class="wp-image-2243" src="https://i1.wp.com/cartesianfaith.files.wordpress.com/2016/10/cat.jpg?resize=240%2C179&ssl=1" data-recalc-dims="1">
<p class="wp-caption-text">Original cat</p>
</div>
<div data-shortcode="caption" class="wp-caption"><img alt="cat1" class="wp-image-2244" src="https://i2.wp.com/cartesianfaith.files.wordpress.com/2016/10/cat1.jpg?resize=240%2C179&ssl=1" data-recalc-dims="1">
<p class="wp-caption-text">Flipped cat</p>
</div>
<div></div>
<p>Alternatively, these variations can be generated inline during the training process. <a target="_blank" href="https://keras.io/preprocessing/image/">Keras uses this approach</a> with the <code>ImageDataGenerator</code> class. This means that images are transformed on the fly during training.</p>
<p><em><strong>Exercise</strong>: Which approach will produce better results? Why?</em></p>
<h3>Conclusion</h3>
<p>Just because you don’t have as much data as Google or Facebook doesn’t mean you should give up on machine learning. By augmenting your dataset, you can get excellent results with small data.</p>
<p><em>Use approaches not mentioned above to augment your data? Share in the comments below.</em></p>
<p> <a target="_blank" rel="nofollow" href="http://feeds.wordpress.com/1.0/gocomments/cartesianfaith.wordpress.com/2230/"><img alt="" src="http://feeds.wordpress.com/1.0/comments/cartesianfaith.wordpress.com/2230/" border="0"></a> <img alt="" src="https://i1.wp.com/pixel.wp.com/b.gif?resize=1%2C1&ssl=1" border="0" data-recalc-dims="1"></p>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="https://www.r-bloggers.com/what-you-need-to-know-about-data-augmentation-for-machine-learning/"></div></div><div><a data-url="https://www.r-bloggers.com/what-you-need-to-know-about-data-augmentation-for-machine-learning/" data-counturl="https://www.r-bloggers.com/what-you-need-to-know-about-data-augmentation-for-machine-learning/" data-count="vertical" data-text="What you need to know about data augmentation for machine learning" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://cartesianfaith.com/2016/10/06/what-you-need-to-know-about-data-augmentation-for-machine-learning/"> R – Cartesian Faith</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/nS4VXy9unUc"></a></div>