<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/deep-learning-part-1-comparison-of-symbolic-deep-learning-frameworks/"></div></div><div><a data-url="https://www.r-bloggers.com/deep-learning-part-1-comparison-of-symbolic-deep-learning-frameworks/" data-counturl="https://www.r-bloggers.com/deep-learning-part-1-comparison-of-symbolic-deep-learning-frameworks/" data-count="horizontal" data-text="Deep Learning Part 1: Comparison of Symbolic Deep Learning Frameworks" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://blog.revolutionanalytics.com/2016/08/deep-learning-part-1.html"> Revolutions</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<div>
<p>by Anusua Trivedi, Microsoft Data Scientist</p>
<p><strong>Background and Approach</strong></p>
<p>This blog series is based on my upcoming talk on re-usability of Deep Learning Models at the <a target="_blank" href="http://conferences.oreilly.com/strata/hadoop-big-data-sg/public/schedule/detail/54278">Hadoop+Strata World Conference</a> in Singapore. This blog series will be in several parts – where I describe my experiences and go deep into the reasons behind my choices.</p>
<p>Deep learning is an emerging field of research, which has its application across multiple domains. I try to show how transfer learning and fine tuning strategy leads to re-usability of the same Convolution Neural Network model in different disjoint domains. Application of this model across various different domains brings value to using this fine-tuned model. </p>
<p>In this blog (Part1), I describe and compare the commonly used open-source deep learning frameworks. I dive deep into different pros and cons for each framework, and discuss why I chose <a target="_blank" href="https://github.com/Theano/Theano">Theano</a> for my work.</p>
<p>Please feel free to email me at <em><u>trivedianusua23@gmail.com</u></em> if you have questions.</p>
<p><strong>Symbolic Frameworks</strong></p>
<p>Symbolic computation frameworks (as in <a target="_blank" href="https://github.com/Microsoft/CNTK">CNTK</a>, <a target="_blank" href="https://github.com/dmlc/mxnet">MXNET</a>, <a target="_blank" href="https://www.tensorflow.org/">TensorFlow</a>, Theano) are specified as a symbolic graph of vector operations, such as matrix add/multiply or convolution. A layer is just a composition of those operations. The fine granularity of the building blocks (operations) allows users to invent new complex layer types without implementing them in a low-level language (as in <a target="_blank" href="http://caffe.berkeleyvision.org/">Caffe</a>).</p>
<p>I've used different symbolic computation frameworks in my work. However, I found each of them has their pros and cons in their design and current implementation, and none of them can perfectly satisfy all needs. For my problem needs , I decided to work with Theano.</p>
<p>Here we compare the following symbolic computation frameworks:</p>
<p><a target="_blank" href="https://github.com/Theano/Theano"><strong>Theano</strong></a></p>
<ul>
<li>Software: Theano</li>
<li>Creator: Université de Montréal</li>
<li>Software license: BSD license</li>
<li>Open source: Yes</li>
<li>Platform: Cross-platform</li>
<li>Written in: Python</li>
<li>Interface: Python</li>
<li>CUDA support: Yes</li>
<li>Automatic differentiation: Yes</li>
<li>Has pre-trained models: Through Lasagne's model zoo</li>
<li>Recurrent Nets: Yes</li>
<li>Convolutional Nets: Yes</li>
<li>RBM/DBNs: Yes</li>
</ul>
<p><a target="_blank" href="https://github.com/tensorflow/tensorflow/issues/654)%20(which%20is%20used%20frequently%20in%20sequence%20prediction%20tasks"><strong>TensorFlow</strong></a></p>
<ul>
<li>Software: TensorFlow</li>
<li>Creator: Google Brain Team</li>
<li>Software license: Apache 2.0</li>
<li>Open source: Yes</li>
<li>Platform: Linux, Mac OS X,</li>
<li>Windows support on roadmap</li>
<li>Written in: C++, Python</li>
<li>Interface: Python, C/C++</li>
<li>CUDA support: Yes</li>
<li>Automatic differentiation: Yes</li>
<li>Has pre-trained models: No</li>
<li>Recurrent Nets: Yes</li>
<li>Convolutional Nets: Yes</li>
<li>RBM/DBNs: Yes</li>
</ul>
<p><a target="_blank" href="https://github.com/dmlc/mxnet/"><strong>MXNET</strong></a></p>
<ul>
<li>Software: MXNET</li>
<li>Creator: Distributed (Deep) Machine Learning Community</li>
<li>Software license: Apache 2.0</li>
<li>Open source: Yes</li>
<li>Platform: Ubuntu, OS X, Windows, AWS, Android, iOS, JavaScript</li>
<li>Written in: C++, Python, Julia, Matlab, R, Scala</li>
<li>Interface: C++, Python, Julia, Matlab, JavaScript, R, Scala</li>
<li>CUDA support: Yes</li>
<li>Automatic differentiation: Yes</li>
<li>Has pre-trained models: Yes</li>
<li>Recurrent Nets: Yes</li>
<li>Convolutional Nets: Yes</li>
<li>RBM/DBNs: Yes</li>
</ul>
<p><strong>Non-symbolic frameworks</strong></p>
<p><strong>PROS</strong>:</p>
<ul>
<li>Non-symbolic (imperative) neural network frameworks like <a target="_blank" href="https://github.com/torch/"><strong><em>torch</em></strong></a>, <a target="_blank" href="https://github.com/BVLC/caffe/"><strong><em>caffe</em></strong></a> etc. tend to have  very similar design in their computation part.</li>
<li>In terms of expressiveness, imperative frameworks with a good design can also expose graph-like interface (e.g. <a target="_blank" href="https://github.com/torch/nngraph"><strong><em>torch/nngraph</em></strong></a>).</li>
</ul>
<p><strong>CONS</strong>:</p>
<ul>
<li>The main drawbacks of imperative frameworks actually lie in manual optimization. For example, in-place operation has to be manually implemented.</li>
<li>Most imperative frameworks are not designed well enough to have comparable expressiveness as symbolic frameworks.</li>
</ul>
<p><strong>Symbolic frameworks</strong></p>
<p><strong>PROS</strong>:</p>
<ul>
<li>Symbolic frameworks can possibly infer optimization automatically from the dependency graph.</li>
<li>A symbolic framework can exploit much more memory reuse opportunities, as is well done in <strong><em>MXNET</em></strong>.</li>
<li>Symbolic frameworks can automatically compute an optimal schedule. This is explained in <a target="_blank" href="http://download.tensorflow.org/paper/whitepaper2015.pdf"><strong><em>TensorFlow whitepaper</em></strong></a>.</li>
</ul>
<p><strong>CONS:</strong></p>
<ul>
<li>Available open source symbolic frameworks currently are still not good enough to beat imperative frameworks in performance.</li>
</ul>
<p><strong>Adding New Operations</strong></p>
<p>In all of these frameworks, adding an Operation with reasonable performance is not easy.</p>
<table width="519">
<thead>
<tr>
<td>
<p><strong>Theano / MXNET</strong></p>
</td>
<td>
<p><strong>TensorFlow</strong></p>
</td>
</tr>
</thead>
<tbody>
<tr>
<td>
<p>Can add Operation in Python with inline C support.</p>
</td>
<td>
<p>Forward in C++, symbolic gradient in Python.</p>
</td>
</tr>
</tbody>
</table>
<p><strong>Code Re-usability</strong></p>
<p>Training deep networks are time-consuming. So, Caffe has released some pre-trained model/weights (model zoo) which could be used as initial weights while transfer learning or fine tuning deep networks on domain specific or custom images.</p>
<ul>
<li><strong>Theano<br></strong>Lasagne is a high-level framework built on top of Theano. It’s very easy to use Caffe pre-tained model weights in Lasagne.</li>
<li><strong>TensorFlow<br></strong>No support for pre-trained model.<strong><br></strong></li>
<li><strong>MXNET<br></strong>MXNET has a <a target="_blank" href="https://github.com/dmlc/mxnet/tree/master/tools/caffe_converter"><em>caffe_converter tool</em></a> which allows to convert pre-trained caffe model weights to fit MXNET.</li>
</ul>
<p><strong>Low-level Tensor Operators</strong></p>
<p>A reasonably efficient implementation of low-level operators can serve as ingredients in writing new models, saving the effort to write new Operations.</p>
<table width="459">
<thead>
<tr>
<td width="333">
<p><strong>Theano</strong></p>
</td>
<td width="333">
<p><strong>TensorFlow</strong></p>
</td>
<td>
<p><strong>MXNET</strong></p>
</td>
</tr>
</thead>
<tbody>
<tr>
<td width="333">
<p>A lot of basic Operations</p>
</td>
<td width="332">
<p>Fairly good</p>
</td>
<td>
<p>Very few</p>
</td>
</tr>
</tbody>
</table>
<p><strong>C</strong><strong>ontrol Flow Operator</strong></p>
<p>Control flow operators make the symbolic engine more expressive and generic.</p>
<table width="600">
<thead>
<tr>
<td width="333">
<p><strong>Theano</strong></p>
</td>
<td width="333">
<p><strong>TensorFlow</strong></p>
</td>
<td width="333">
<p><strong>MXNET</strong></p>
</td>
</tr>
</thead>
<tbody>
<tr>
<td width="333">
<p>   Supported</p>
</td>
<td width="333">
<p>Experimental</p>
</td>
<td width="333">
<p>Not Supported</p>
</td>
</tr>
</tbody>
</table>
<p><strong>H</strong><strong>igh</strong>–<strong>level</strong> <strong>Support</strong></p>
<ul>
<li><strong>Theano<br></strong>Pure symbolic computation framework. High-level frameworks can be built to fit desired means of use. Successful examples include <a target="_blank" href="http://keras.io/"><em>Keras</em></a>, <a target="_blank" href="http://lasagne.readthedocs.org/en/latest/"><em>Lasagne</em></a>, <a target="_blank" href="http://blocks.readthedocs.org/en/latest/"><em>blocks</em></a>.<strong><br></strong></li>
<li><strong>TensorFlow<br></strong>Has good design considerations for neural network training, and at the same time avoid being totally a neural network framework, which is a wonderful job. The <strong><em>graph collection</em></strong>, <strong><em>queues</em></strong>, <strong><em>image augmenters</em></strong> etc. can be useful building blocks for a higher-level wrapper.</li>
<li><strong>MXNET<br></strong>Apart from the symbolic part, MXNET also comes with all necessary <a target="_blank" href="https://github.com/dmlc/mxnet/tree/master/example/image-classification"><em>components</em></a> for image classification, going all the way through data loading to building a model that has a method to start training.</li>
</ul>
<p><strong>Performance</strong></p>
<p><strong>Benchmarking Using Single-GPU</strong></p>
<p>I benchmark LeNet model on MNIST Dataset using a Single-GPU (NVIDIA Quadro K1200 GPU).</p>
<table width="516">
<thead>
<tr>
<td width="333">
<p><strong>Theano</strong></p>
</td>
<td width="333">
<p><strong>TensorFlow</strong></p>
</td>
<td width="333">
<p><strong>MXNET</strong></p>
</td>
</tr>
</thead>
<tbody>
<tr>
<td width="333">
<p>Great</p>
</td>
<td width="333">
<p>Not so good</p>
</td>
<td width="333">
<p>Excellent</p>
</td>
</tr>
</tbody>
</table>
<p><strong>Memory</strong></p>
<p>GPU memory is limited and may usually be a problem for large models.</p>
<table width="518">
<thead>
<tr>
<td width="333">
<p><strong>Theano</strong></p>
</td>
<td width="333">
<p><strong>TensorFlow</strong></p>
</td>
<td width="333">
<p><strong>MXNET</strong></p>
</td>
</tr>
</thead>
<tbody>
<tr>
<td width="333">
<p>Great</p>
</td>
<td width="333">
<p>Not so good</p>
</td>
<td width="333">
<p>Excellent</p>
</td>
</tr>
</tbody>
</table>
<p><strong>Single</strong>–<strong>GPU</strong> <strong>Speed</strong></p>
<p><strong>Theano</strong> takes a long time to compile a graph, especially with complex models. <strong>TensorFlow</strong> is a bit slower.</p>
<table width="518">
<thead>
<tr>
<td>
<p><strong>Theano / MXNET</strong></p>
</td>
<td>
<p><strong>TensorFlow</strong></p>
</td>
</tr>
</thead>
<tbody>
<tr>
<td>
<p>comparable to CuDNNv4</p>
</td>
<td>
<p>about 0.5x slower</p>
</td>
</tr>
</tbody>
</table>
<p><strong>Parallel</strong>/<strong>Distributed</strong> <strong>Support</strong></p>
<table width="518">
<thead>
<tr>
<td width="398">
<p><strong>Theano</strong></p>
</td>
<td width="353">
<p><strong>TensorFlow</strong></p>
</td>
<td>
<p><strong>MXNET</strong></p>
</td>
</tr>
</thead>
<tbody>
<tr>
<td width="398">
<p>experimental multi-GPU</p>
</td>
<td width="353">
<p>multi-GPU</p>
</td>
<td>
<p>distributed</p>
</td>
</tr>
</tbody>
</table>
<p><strong>Conclusion</strong></p>
<p>Theano (with higher-level Lasagne &amp; Keras) is a great choice for deep learning models. It’s very easy to implement new networks &amp; modify existing networks using Lasagne/Keras. I prefer python, and thus prefer using Lasagne/Keras due to their very mature python interface. However, they do not support R. I have tried using transfer learning and fine tuning in Lasagne/Keras, and it’s very easy to modify an existing network and customize it with domain-specific custom data.</p>
<p>Comparisons of different frameworks show that MXNET is the best choice (better performance/memory). Moreover, it has a great R support. In fact, it is the only framework that <a target="_blank" href="http://dmlc.ml/rstats/2015/11/03/training-deep-net-with-R.html">supports all functions in R</a>. In MXNET, transfer learning and fine tuning networks are possible, but not as easy (as compared to Lasagne/Keras). This makes modifying existing trained networks more difficult, and thus a bit difficult to use domain-specific custom data.</p>
<p> </p>
</div>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="https://www.r-bloggers.com/deep-learning-part-1-comparison-of-symbolic-deep-learning-frameworks/"></div></div><div><a data-url="https://www.r-bloggers.com/deep-learning-part-1-comparison-of-symbolic-deep-learning-frameworks/" data-counturl="https://www.r-bloggers.com/deep-learning-part-1-comparison-of-symbolic-deep-learning-frameworks/" data-count="vertical" data-text="Deep Learning Part 1: Comparison of Symbolic Deep Learning Frameworks" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://blog.revolutionanalytics.com/2016/08/deep-learning-part-1.html"> Revolutions</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/VGfkUxZYYtI"></a></div>