<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/a-simple-workflow-for-deep-learning/"></div></div><div><a data-url="https://www.r-bloggers.com/a-simple-workflow-for-deep-learning/" data-counturl="https://www.r-bloggers.com/a-simple-workflow-for-deep-learning/" data-count="horizontal" data-text="A simple workflow for deep learning" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://cartesianfaith.com/2016/09/29/a-simple-workflow-for-deep-learning/"> R – Cartesian Faith</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>As a follow-up to my <a target="_blank" href="https://cartesianfaith.com/2016/09/23/a-primer-on-universal-function-approximation-with-deep-learning-in-torch-and-r/">Primer On Universal Function Approximation with Deep Learning</a>, I’ve created a <a target="_blank" href="https://github.com/zatonovo/deep_learning_ex">project on Github</a> that provides a working example of building, training, and evaluating a neural network. Included are helper functions in Lua that I wrote to simplify creating the data and using some functional programming techniques.</p>
<p>The basic workflow for the example is this:</p>
<ol>
<li>Create/acquire a training set;</li>
<li>Analyze the data for traits, distributions, noise, etc.;</li>
<li>Design a deep learning architecture including the layers and activation functions. Also make sure you understand the type of problem you are trying to solve);</li>
<li>Choose hyper parameters, such as cost function, optimizer, and learning rate;</li>
<li>Train the model;</li>
<li>Evaluate in-sample and out-of-sample performance.</li>
</ol>
<p>My personal preference is to limit the use of a deep learning framework to building and training models. To construct the datasets and analyze performance, it’s easier to use R (YMMV of course). What’s nice about this approach is that if you primarily work in Python or R, then you can continue to use the tools you’re most familiar with. It also means that it’s easy to swap out one deep learning framework with another without having to start over. These frameworks are also a bit of a bear to setup (I’m looking at you, TensorFlow), particularly if you want to leverage GPUs. It’s also convenient to use a Docker image for this purpose to isolate the effects of a specialized configuration and make it repeatable if you want to work on *gasp* a second computer.</p>
<h3>Which Deep Learning Framework?</h3>
<p>Having some experience with TensorFlow, Theano, and Torch, I find Torch to have the friendliest high-level semantics. Theano and TensorFlow are much more low-level, which is not as well suited to practitioners or applied researchers. That means it’s a little harder to get started. The trade-off with Torch is that you have to learn Lua, which is a simple scripting language but also has some awkward paradigms (I’ve never been a fan of the Prototype object model). </p>
<p>On the other hand, Theano and TensorFlow are built on Python, so most people will be familiar with the language. However, my time could be spent better if I didn’t have to write my own mini-batch algorithm. As an alternative, Keras provides a semantically rich high-level interface that works with both Theano and TensorFlow. I will be adding a corresponding function approximation example in the <code>deep_learning_ex</code> project to make it easier to compare. At that point, it will be easier to compare compute performance, as well as how close the optimizers are to each other.</p>
<div class="wp-caption"><img height="360" alt="Deep learning can be painfully slow" width="480" src="https://i2.wp.com/www.adsbits.com/oc-content/uploads/3/18.jpg">
<p class="wp-caption-text">Deep learning can be painfully slow</p>
</div>
<p>In terms of TensorFlow, unless you plan on working for Google, I wouldn’t recommend using it. The fact that it requires Google’s proprietary Bazel build system means it’s DOA for me. When I want to work with deep learning, I really don’t have the patience to wait for a 1.1 GB download of <em>just the build system</em>. I mean, I only have Time Warner Cable for crissakes. Others have reported that even using pre-built models, like SyntaxNet are slow, so unless you have the compute power and storage capacity of Google’s data centers along with the bandwidth of Google Fiber, you’re better off watching YouTube.</p>
<h3>Learning Deep Learning Frameworks</h3>
<h4>Torch/Lua</h4>
<p>Learning <a target="_blank" href="http://torch.ch/">Torch</a> can be split into two tasks: learning Lua, and then understanding the Torch framework, specifically the <code>nn</code> package. Most people will find that learning Lua will take the majority of the time, as <code>nn</code> is nicely organized and easy to use.</p>
<p>If you already are comfortable with programming languages, then this <a target="_blank" href="https://coronalabs.com/learn-lua/">15 minute tutorial</a> is good. Alternatively, this other <a target="_blank" href="http://tylerneylon.com/a/learn-lua/">15 minute tutorial</a> is a bit more terse but rather comprehensive. This will cover the basics. Beyond that, you need to understand how to work with data, which is less well covered. The <a target="_blank" href="http://nocurve.com/2014/03/05/simple-csv-read-and-write-using-lua/">simplecsv module</a> can simplify I/O. </p>
<p>The actual data format that the optimzer needs is a table object with an attached <code>size</code> method. Each element of this table is itself a table with two elements: input and corresponding output. So this can be considered a row-major matrix representation of the data. To use the provided StochasticGradient optimizer, the data must be constructed this way, as shown in <code>ex_fun_approx.lua</code>. It is up to you to reserve some data for testing.</p>
<p>From a practical perspective, you don’t need to know much about Torch itself. It’s probably more efficient to familiarize yourself with the <a target="_blank" href="https://github.com/torch/nn">nn package</a> first. I spend most of my time in this documentation. At some later point, it might be worthwhile learning how Torch itself works, in which case their <a target="_blank" href="https://github.com/torch/torch7/blob/master/doc/index.md">github repo</a> is flush with documentation and examples. I haven’t needed to look elsewhere.</p>
<h4>Keras/Theano</h4>
<p>If Theano is like Torch, then Keras is like the nn package. Unless you need to descend into the bits, it’s probably best to stay high-level. Unlike <code>nn</code>, there are alternatives to Keras for Theano, which I won’t cover. Like Torch, Keras comes pre-installed in the Docker image provided in the <code>deep_learning_ex</code> repository. The best way to get started is to read the <a target="_blank" href="https://keras.io/">Keras documentation</a>, which includes a working example of a simple neural network.</p>
<p>As with <code>nn</code>, the trick is understanding the framework’s interface, particularly around what expectations it has for the data. Keras essentially expects a 4-tuple of (input training, input testing, output training, output testing). Their <a target="_blank" href="https://keras.io/datasets/">built-in datasets</a> all return data organized like this (actually two pairs representing input and output).</p>
<h3>Conclusion</h3>
<p>Deep learning doesn’t need to be hard to learn. By following the prescribed workflow, using the <a target="_blank" href="https://github.com/zatonovo/deep_learning_ex">provided Docker image</a>, and streamlining your learning of deep learning frameworks to the essentials, you can get up to speed quickly. </p>
<p>Have any resources you’d like to share? Add them in the comments!</p>
<p> <a target="_blank" rel="nofollow" href="http://feeds.wordpress.com/1.0/gocomments/cartesianfaith.wordpress.com/2219/"><img alt="" src="http://feeds.wordpress.com/1.0/comments/cartesianfaith.wordpress.com/2219/" border="0"></a> <img alt="" src="https://i1.wp.com/pixel.wp.com/b.gif?resize=1%2C1&ssl=1" border="0" data-recalc-dims="1"></p>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="https://www.r-bloggers.com/a-simple-workflow-for-deep-learning/"></div></div><div><a data-url="https://www.r-bloggers.com/a-simple-workflow-for-deep-learning/" data-counturl="https://www.r-bloggers.com/a-simple-workflow-for-deep-learning/" data-count="vertical" data-text="A simple workflow for deep learning" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://cartesianfaith.com/2016/09/29/a-simple-workflow-for-deep-learning/"> R – Cartesian Faith</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/BxT_Fpwb1Lk"></a></div>