<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/tfruns-tools-for-tensorflow-training-runs/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://tensorflow.rstudio.com/blog/tfruns.html"> TensorFlow for R</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>The <a rel="nofollow" href="https://tensorflow.rstudio.com/tools/tfruns/articles/overview.html" target="_blank">tfruns package</a> provides a suite of tools for tracking, visualizing, and managing TensorFlow training runs and experiments from R. Use the tfruns package to:</p>
<ul>
<li>
<p>Track the hyperparameters, metrics, output, and source code of every training run.</p>
</li>
<li>
<p>Compare hyperparmaeters and metrics across runs to find the best performing model.</p>
</li>
<li>
<p>Automatically generate reports to visualize individual training runs or comparisons between runs. </p>
</li>
</ul>
<p>You can install the <strong>tfruns</strong> package from GitHub as follows:</p>
<div>
<pre><code>devtools<span>::</span><span><a rel="nofollow" href="http://www.rdocumentation.org/packages/devtools/topics/install_github?tap_a=5644-dce66f&tap_s=10907-287229" target="_blank">install_github</a></span>(<span>&quot;rstudio/tfruns&quot;</span>)</code></pre>
</div>
<p>Complete documentation for tfruns is available on the <a rel="nofollow" href="https://tensorflow.rstudio.com/tools/tfruns/articles/overview.html" target="_blank">TensorFlow for R website</a>.</p>
<p><strong>tfruns</strong> is intended to be used with the <a rel="nofollow" href="https://tensorflow.rstudio.com/keras" target="_blank">keras</a> and/or the <a rel="nofollow" href="https://tensorflow.rstudio.com/keras" target="_blank">tfestimators</a> packages, both of which provide higher level interfaces to TensorFlow from R. These packages can be installed with:</p>
<div>
<pre><code><span># keras</span>
<span>install.packages</span>(<span>&quot;keras&quot;</span>)

<span># tfestimators</span>
devtools<span>::</span><span><a rel="nofollow" href="http://www.rdocumentation.org/packages/devtools/topics/install_github?tap_a=5644-dce66f&tap_s=10907-287229" target="_blank">install_github</a></span>(<span>&quot;rstudio/tfestimators&quot;</span>)</code></pre>
</div>
<div>
<h2>Training</h2>
<p>In the following sections we’ll describe the various capabilities of <strong>tfruns</strong>. Our example training script (<a rel="nofollow" href="https://github.com/rstudio/tfruns/blob/master/inst/examples/mnist_mlp/mnist_mlp.R" target="_blank">mnist_mlp.R</a>) trains a Keras model to recognize MNIST digits.</p>
<p>To train a model with <strong>tfruns</strong>, just use the <code><a rel="nofollow" href="https://tensorflow.rstudio.com/tools/truns/reference/training_run.html" target="_blank">training_run()</a></code> function in place of the <code>source()</code> function to execute your R script. For example:</p>
<div>
<pre><code><span>library</span>(tfruns)
<span><a rel="nofollow" href="https://tensorflow.rstudio.com/tools/truns/reference/training_run.html" target="_blank">training_run</a></span>(<span>&quot;mnist_mlp.R&quot;</span>)</code></pre>
</div>
<p>When training is completed, a summary of the run will automatically be displayed if you are within an interactive R session:</p>
<p><kbd><img src="https://i0.wp.com/tensorflow.rstudio.com/tools/tfruns/articles/images/view_run.png?w=456&ssl=1" data-recalc-dims="1"></kbd></p>
<p>The metrics and output of each run are automatically captured within a <em>run directory</em> which is unique for each run that you initiate. Note that for Keras and TF Estimator models this data is captured automatically (no changes to your source code are required).</p>
<p>You can call the <code><a rel="nofollow" href="https://tensorflow.rstudio.com/tools/truns/reference/latest_run.html" target="_blank">latest_run()</a></code> function to view the results of the last run (including the path to the run directory which stores all of the run’s output):</p>
<div>
<pre><code><span><a rel="nofollow" href="https://tensorflow.rstudio.com/tools/truns/reference/latest_run.html" target="_blank">latest_run</a></span>()</code></pre>
</div>
<pre><code>$ run_dir           : chr &quot;runs/2017-10-02T14-23-38Z&quot;
$ eval_loss         : num 0.0956
$ eval_acc          : num 0.98
$ metric_loss       : num 0.0624
$ metric_acc        : num 0.984
$ metric_val_loss   : num 0.0962
$ metric_val_acc    : num 0.98
$ flag_dropout1     : num 0.4
$ flag_dropout2     : num 0.3
$ samples           : int 48000
$ validation_samples: int 12000
$ batch_size        : int 128
$ epochs            : int 20
$ epochs_completed  : int 20
$ metrics           : chr &quot;(metrics data frame)&quot;
$ model             : chr &quot;(model summary)&quot;
$ loss_function     : chr &quot;categorical_crossentropy&quot;
$ optimizer         : chr &quot;RMSprop&quot;
$ learning_rate     : num 0.001
$ script            : chr &quot;mnist_mlp.R&quot;
$ start             : POSIXct[1:1], format: &quot;2017-10-02 14:23:38&quot;
$ end               : POSIXct[1:1], format: &quot;2017-10-02 14:24:24&quot;
$ completed         : logi TRUE
$ output            : chr &quot;(script ouptut)&quot;
$ source_code       : chr &quot;(source archive)&quot;
$ context           : chr &quot;local&quot;
$ type              : chr &quot;training&quot;</code></pre>
<p>The run directory used in the example above is “runs/2017-10-02T14-23-38Z”. Run directories are by default generated within the “runs” subdirectory of the current working directory, and use a timestamp as the name of the run directory. You can view the report for any given run using the <code><a rel="nofollow" href="https://tensorflow.rstudio.com/tools/truns/reference/view_run.html" target="_blank">view_run()</a></code> function:</p>
<div>
<pre><code><span><a rel="nofollow" href="https://tensorflow.rstudio.com/tools/truns/reference/view_run.html" target="_blank">view_run</a></span>(<span>&quot;runs/2017-10-02T14-23-38Z&quot;</span>)</code></pre>
</div>
</div>
<div>
<h2>Comparing Runs</h2>
<p>Let’s make a couple of changes to our training script to see if we can improve model performance. We’ll change the number of units in our first dense layer to 128, change the <code>learning_rate</code> from 0.001 to 0.003 and run 30 rather than 20 <code>epochs</code>. After making these changes to the source code we re-run the script using <code><a rel="nofollow" href="https://tensorflow.rstudio.com/tools/truns/reference/training_run.html" target="_blank">training_run()</a></code> as before:</p>
<div>
<pre><code><span><a rel="nofollow" href="https://tensorflow.rstudio.com/tools/truns/reference/training_run.html" target="_blank">training_run</a></span>(<span>&quot;mnist_mlp.R&quot;</span>)</code></pre>
</div>
<p>This will also show us a report summarizing the results of the run, but what we are really interested in is a comparison between this run and the previous one. We can view a comparison via the <code><a rel="nofollow" href="https://tensorflow.rstudio.com/tools/truns/reference/compare_runs.html" target="_blank">compare_runs()</a></code> function:</p>
<div>
<pre><code><span><a rel="nofollow" href="https://tensorflow.rstudio.com/tools/truns/reference/compare_runs.html" target="_blank">compare_runs</a></span>()</code></pre>
</div>
<p><kbd><img src="https://i0.wp.com/tensorflow.rstudio.com/tools/tfruns/articles/images/compare_runs.png?w=456&ssl=1" data-recalc-dims="1"></kbd></p>
<p>The comparison report shows the model attributes and metrics side-by-side, as well as differences in the source code and output of the training script.</p>
<p>Note that <code><a rel="nofollow" href="https://tensorflow.rstudio.com/tools/truns/reference/compare_runs.html" target="_blank">compare_runs()</a></code> will by default compare the last two runs, however you can pass any two run directories you like to be compared.</p>
</div>
<div>
<h2>Using Flags</h2>
<p>Tuning a model often requires exploring the impact of changes to many hyperparameters. The best way to approach this is generally not by changing the source code of the training script as we did above, but instead by defining flags for key parameters you may want to vary. In the example script you can see that we have done this for the <code>dropout</code> layers:</p>
<div>
<pre><code>FLAGS &lt;-<span> </span><span><a rel="nofollow" href="https://tensorflow.rstudio.com/tools/truns/reference/flags.html" target="_blank">flags</a></span>(
  <span><a rel="nofollow" href="https://tensorflow.rstudio.com/tools/truns/reference/flags.html" target="_blank">flag_numeric</a></span>(<span>&quot;dropout1&quot;</span>, <span>0.4</span>),
  <span><a rel="nofollow" href="https://tensorflow.rstudio.com/tools/truns/reference/flags.html" target="_blank">flag_numeric</a></span>(<span>&quot;dropout2&quot;</span>, <span>0.3</span>)
)</code></pre>
</div>
<p>These flags are then used in the definition of our model here:</p>
<div>
<pre><code>model &lt;-<span> </span><span>keras_model_sequential</span>()
model <span>%>%</span>
<span> </span><span>layer_dense</span>(<span>units =</span> <span>128</span>, <span>activation =</span> <span>'relu'</span>, <span>input_shape =</span> <span>c</span>(<span>784</span>)) <span>%>%</span>
<span> </span><span>layer_dropout</span>(<span>rate =</span> FLAGS<span>$</span>dropout1) <span>%>%</span>
<span> </span><span>layer_dense</span>(<span>units =</span> <span>128</span>, <span>activation =</span> <span>'relu'</span>) <span>%>%</span>
<span> </span><span>layer_dropout</span>(<span>rate =</span> FLAGS<span>$</span>dropout2) <span>%>%</span>
<span> </span><span>layer_dense</span>(<span>units =</span> <span>10</span>, <span>activation =</span> <span>'softmax'</span>)</code></pre>
</div>
<p>Once we’ve defined flags, we can pass alternate flag values to <code><a rel="nofollow" href="https://tensorflow.rstudio.com/tools/truns/reference/training_run.html" target="_blank">training_run()</a></code> as follows:</p>
<div>
<pre><code><span><a rel="nofollow" href="https://tensorflow.rstudio.com/tools/truns/reference/training_run.html" target="_blank">training_run</a></span>(<span>'mnist_mlp.R'</span>, <span>flags =</span> <span>c</span>(<span>dropout1 =</span> <span>0.2</span>, <span>dropout2 =</span> <span>0.2</span>))</code></pre>
</div>
<p>You aren’t required to specify all of the flags (any flags excluded will simply use their default value).</p>
<p>Flags make it very straightforward to systematically explore the impact of changes to hyperparameters on model performance, for example:</p>
<div>
<pre><code><span>for</span> (dropout1 <span>in</span> <span>c</span>(<span>0.1</span>, <span>0.2</span>, <span>0.3</span>))
  <span><a rel="nofollow" href="https://tensorflow.rstudio.com/tools/truns/reference/training_run.html" target="_blank">training_run</a></span>(<span>'mnist_mlp.R'</span>, <span>flags =</span> <span>c</span>(<span>dropout1 =</span> dropout1))</code></pre>
</div>
<p>Flag values are automatically included in run data with a “flag_” prefix (e.g. <code>flag_dropout1</code>, <code>flag_dropout2</code>).</p>
<p>See the article on <a rel="nofollow" href="https://tensorflow.rstudio.com/tools/training_flags.html" target="_blank">training flags</a> for additional documentation on using flags.</p>
</div>
<div>
<h2>Analyzing Runs</h2>
<p>We’ve demonstrated visualizing and comparing one or two runs, however as you accumulate more runs you’ll generally want to analyze and compare runs many runs. You can use the <code><a rel="nofollow" href="https://tensorflow.rstudio.com/tools/truns/reference/ls_runs.html" target="_blank">ls_runs()</a></code> function to yield a data frame with summary information on all of the runs you’ve conducted within a given directory:</p>
<div>
<pre><code><span><a rel="nofollow" href="https://tensorflow.rstudio.com/tools/truns/reference/ls_runs.html" target="_blank">ls_runs</a></span>()</code></pre>
</div>
<pre><code># A tibble: 6 x 27
                    run_dir eval_loss eval_acc metric_loss metric_acc metric_val_loss
                      </code></pre></div>