<p>With proper visualization, we could have a better understanding of the mechanism of Deep Learning. </p>
<ul><li>Monitoring training/testing metrics through the learning curve. You know how well your model learning from the data.</li>
<li>Visualizing the dynamics under the gradients of a layer with histogram, you know your network is live or dead(gradient).</li>
<li>By interpreting the embedding features from a layer, using t-SNE for high-dimensional data visualization, you get intuitions from its representation power.</li>
</ul><p>There are way more techniques in visualizing neural networks than the above. So that’s why we want to build a handy tool for our MXNet users.</p>
<p>Thanks to the community, we already have <a href="https://www.tensorflow.org/versions/master/how_tos/graph_viz/index.html">TensorBoard</a> and it’s easy-to-use and meets most daily use cases. However, TensorBoard is built together with TensorFlow and we have to come up a way to make a stand-alone version for general visualization purpose.</p>
<h2>Before we start</h2>
<p>It's my first time to get involved in an open-sourced project like MXNet, and it has some visualization solutions there. I found there’re several similar issues requests for TensorBoard-liked tool, some people want to build the tool from scratch, while <a href="https://github.com/piiswrong">@piiswrong</a> asked whether is possible to strip TensorBoard from TensorFlow. I like the latter one, so I created an issue <a href="https://github.com/dmlc/mxnet/issues/4003">dmlc/mxnet#4003</a> for discussion, proposed my solution and roadmap towards this direction. </p>
<h2>The Logging Part</h2>
<p>Technically, TensorBoard contains two parts: logging and rendering. In TensorBoard, it supports these types of data:</p>
<ul><li>Scalar. </li>
<li>Image.</li>
<li>Video.</li>
<li>Histogram.</li>
<li>Graph. The TensorFlow computational graph.</li>
<li>Embedding.</li>
</ul><h3>Get summary without running TensorFlow</h3>
<p>In TensorFlow, <code>summary</code> object could be generated by running a <code>session</code> or by running an operation, here's an example in <a href="https://www.tensorflow.org/how_tos/summaries_and_tensorboard/">TensorBoard Document</a></p>
<div><pre><code><span>with</span> <span>tf</span><span>.</span><span>name_scope</span><span>(</span><span>'cross_entropy'</span><span>):</span>
  <span># The raw formulation of cross-entropy,</span>
  <span>#</span>
  <span># tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(tf.softmax(y)),</span>
  <span># reduction_indices=[1]))</span>
  <span>#</span>
  <span># can be numerically unstable.</span>
  <span>#</span>
  <span># So here we use tf.nn.softmax_cross_entropy_with_logits on the</span>
  <span># raw outputs of the nn_layer above, and then average across</span>
  <span># the batch.</span>
  <span>diff</span> <span>=</span> <span>tf</span><span>.</span><span>nn</span><span>.</span><span>softmax_cross_entropy_with_logits</span><span>(</span><span>y</span><span>,</span> <span>y_</span><span>)</span>
  <span>with</span> <span>tf</span><span>.</span><span>name_scope</span><span>(</span><span>'total'</span><span>):</span>
    <span>cross_entropy</span> <span>=</span> <span>tf</span><span>.</span><span>reduce_mean</span><span>(</span><span>diff</span><span>)</span>
<span>tf</span><span>.</span><span>summary</span><span>.</span><span>scalar</span><span>(</span><span>'cross_entropy'</span><span>,</span> <span>cross_entropy</span><span>)</span>
</code></pre></div>
<p>Luckily, those summaries are <a href="https://developers.google.com/protocol-buffers/">Protocol Buffers</a> and it makes things easy, which means it could be changed into pure Python. Thanks to the community, I found <a href="https://github.com/mufeili">@mufeili</a> has been working on this feature and TensorBoard has provided a relatively clean API for this purpose, which allows us to get these without running TensorFlow’s operation. </p>
<p>The logging part code is placed in <a href="https://github.com/dmlc/tensorboard/tree/master/python">tensorboard/python</a>.</p>
<h3>Logging events in pure Python</h3>
<p>TensorBoard relies on the <code>Event</code> file, in which <code>summary</code> information is included in this file.</p>
<div><pre><code><span>$ </span>tensorboard --logdir<span>=</span>path-to-event-files
</code></pre></div>
<p>See <a href="https://github.com/dmlc/tensorboard/blob/master/python/tensorboard/record_writer.py">tensorboard/record_writer.py</a> for details.</p>
<h2>The Rendering Part</h2>
<p>The next step is to build TensorBoard's rendering part. The goal is to provide an easy-to-maintain solution, so we didn’t try to strip the relevant codes and build it from scratch. Rather, we pull the TensorFlow codebase and use bazel to build TensorBoard:</p>
<div><pre><code><span>$ </span>git clone https://github.com/tensorflow/tensorflow
<span>$ </span><span>cd </span>tensorflow
<span>$ </span>./configure
<span>$ </span>bazel build tensorflow/tensorboard:tensorboard
</code></pre></div>
<p>Then all dependencies could be found in <code>/bazel-bin/tensorflow/tensorboard</code>, a Python binary <code>tensorboard</code> to launch the app and its dependencies <code>tensorboard.runfiles</code>.</p>
<p>Here’s the file structure before we build TensorBoard(and move files):</p>
<div><pre><code>├── LICENSE
├── Makefile
├── README.md
├── installer.sh
├── python
│   ├── README.md
│   ├── setup.py
│   └── tensorboard
│       ├── __init__.py
│       ├── crc32c.py
│       ├── event_file_writer.py
│       ├── record_writer.py
│       ├── src
│       │   └── __init__.py
│       ├── summary.py
│       └── writer.py
├── tensorboard
│   └── src
│       ├── event.proto
│       ├── resource_handle.proto
│       ├── summary.proto
│       ├── tensor.proto
│       ├── tensor_shape.proto
│       └── types.proto
└── tools
    └── pip_package
        ├── MANIFEST.in
        ├── README
        └── build_pip_package.sh
</code></pre></div>
<p>Then we get <code>tensorboard</code> and <code>tensorboard.runfiles/</code>:</p>
<div><pre><code>├── LICENSE
├── Makefile
├── README.md
├── installer.sh
├── python
│   ├── MANIFEST.in
│   ├── README
│   ├── README.md
│   ├── setup.py
│   └── tensorboard
│       ├── __init__.py
│       ├── crc32c.py
│       ├── event_file_writer.py
│       ├── record_writer.py
│       ├── src
│       ├── summary.py
│       ├── tensorboard           &lt;--- python binary
│       ├── tensorboard.runfiles  &lt;--- directory/dependencies
│       └── writer.py
├── tensorboard
│   └── src
│       ├── event.proto
│       ├── resource_handle.proto
│       ├── summary.proto
│       ├── tensor.proto
│       ├── tensor_shape.proto
│       └── types.proto
└── tools
    └── pip_package
        ├── MANIFEST.in
        ├── README
        └── build_pip_package.sh
</code></pre></div>
<p>The most important, the <code>tensorboard</code> binary is directly usable, as it searches and imports relevant packages:</p>
<div><pre><code><span># Find the runfiles tree</span>
<span>def</span> <span>FindModuleSpace</span><span>():</span>
  <span># Follow symlinks, looking for my module space</span>
  <span>stub_filename</span> <span>=</span> <span>os</span><span>.</span><span>path</span><span>.</span><span>abspath</span><span>(</span><span>sys</span><span>.</span><span>argv</span><span>[</span><span>0</span><span>])</span>
  <span>while</span> <span>True</span><span>:</span>
    <span># Found it?</span>
    <span>module_space</span> <span>=</span> <span>stub_filename</span> <span>+</span> <span>'.runfiles'</span>
    <span>if</span> <span>os</span><span>.</span><span>path</span><span>.</span><span>isdir</span><span>(</span><span>module_space</span><span>):</span>
      <span>break</span>

    <span>runfiles_pattern</span> <span>=</span> <span>&quot;(.*</span><span>\</span><span>.runfiles)/.*&quot;</span>
    <span>if</span> <span>IsWindows</span><span>():</span>
      <span>runfiles_pattern</span> <span>=</span> <span>&quot;(.*</span><span>\</span><span>.runfiles)</span><span>\\</span><span>.*&quot;</span>
    <span>matchobj</span> <span>=</span> <span>re</span><span>.</span><span>match</span><span>(</span><span>runfiles_pattern</span><span>,</span> <span>os</span><span>.</span><span>path</span><span>.</span><span>abspath</span><span>(</span><span>sys</span><span>.</span><span>argv</span><span>[</span><span>0</span><span>]))</span>
    <span>if</span> <span>matchobj</span><span>:</span>
      <span>module_space</span> <span>=</span> <span>matchobj</span><span>.</span><span>group</span><span>(</span><span>1</span><span>)</span>
      <span>break</span>

    <span>raise</span> <span>AssertionError</span><span>(</span><span>'Cannot find .runfiles directory for </span><span>%</span><span>s'</span> <span>%</span>
                         <span>sys</span><span>.</span><span>argv</span><span>[</span><span>0</span><span>])</span>
  <span>return</span> <span>module_space</span>
</code></pre></div>
<p>However, this package search logic may have some problems if we install from the Python wheel. And we would discuss the solution later.</p>
<h2>Build pip package for both logging and rendering</h2>
<p>Now we’ve finished all preparations for building a Python wheel. As we have a binary file and want to launch TensorBoard in the following way:</p>
<div><pre><code><span>$ </span>tensorboard --logdir<span>=</span>path-to-event-file/
</code></pre></div>
<p>It could be done by assigning binary script entry in <code>setup.py</code>, see <a href="https://github.com/dmlc/tensorboard/blob/master/python/setup.py">tensorboard/setup.py</a> for more. The basic idea of a setup function, in this scenario, is to pack all necessary dependencies and setup the binary file correctly.</p>
<p>Ops, back to the <code>FindModuleSpace</code> issue mentioned above. The <code>tensorboard</code> binary file would be copied to <code>/Users/zihao.zzh/anaconda2/envs/tensorboard/bin/tensorboard</code>, for example. But when you launch TensorBoard, it searches the <code>.runfiles</code> in the same directory, so we make a <a href="https://github.com/dmlc/tensorboard/blob/master/tensorboard-binary.patch">tensorboard/tensorboard-binary.patch</a> and apply this patch in the building process.</p>
<p>It provides an <code>installer.sh</code> to automate this process for you, and you might want to check it out here: <a href="https://github.com/dmlc/tensorboard/blob/master/installer.sh">tensorboard/installer.sh</a></p>
<h2>How to use?</h2>
<p>We create a tutorial to understand the vanish gradient problem through visualization under <code>docs/tutorial/</code></p>
<h2>Future works</h2>
<ul><li>Add this component as submodule in MXNet.</li>
<li>Supports <code>image</code>, <code>video</code>, <code>embedding</code> and even <code>graph</code> for MXNet.</li>
<li>Make package installation more easier by providing a pre-built wheel, then users could install this package in one line. </li>
</ul><p>Feel free to join us and contribute!</p>
<h2>About the author</h2>
<p><a href="https://github.com/zihaolucky">Zihao Zheng</a> is an algorithm engineer at AI Lab, Alibaba Group. Before joined the industry, he studied Mathematics at South China Normal University and learned Machine Learning and Computer Science from MOOCs.</p>
<p>Many thanks to the community, TensorBoard authors and contributors!!</p>