<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/hyper-parameter-tuning-with-grid-search-for-deep-learning/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://shiring.github.io/machine_learning/2017/03/07/grid_search"> Shirin's playgRound</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><a target="_blank" rel="nofollow" href="https://shiring.github.io/machine_learning/2017/02/27/h2o">Last week I showed how to build a deep neural network with <strong>h2o</strong> and <strong>rsparkling</strong></a>. As we could see there, it is not trivial to optimize the hyper-parameters for modeling. Hyper-parameter tuning with grid search allows us to test different combinations of hyper-parameters and find one with improved accuracy.</p>
<p>Keep in mind though, that hyper-parameter tuning can only improve the model so much without overfitting. If you can’t achieve sufficient accuracy, the input features might simply not be adequate for the predictions you are trying to model. It might be necessary to go back to the original features and try e.g. feature engineering methods.</p>
<p></p>
<h3>Preparing Spark instance and plotting theme</h3>
<p>Check out <a target="_blank" rel="nofollow" href="https://shiring.github.io/machine_learning/2017/03/07/grid_search">last week’s post</a> for details on how I prepared the data.</p>
<div>
<pre><code><span>library</span><span>(</span><span>rsparkling</span><span>)</span><span>
</span><span>options</span><span>(</span><span>rsparkling.sparklingwater.version</span><span> </span><span>=</span><span> </span><span>&quot;2.0.3&quot;</span><span>)</span><span>
</span><span>library</span><span>(</span><span>h</span><span>2</span><span>o</span><span>)</span><span>
</span><span>library</span><span>(</span><span>dplyr</span><span>)</span><span>
</span><span>library</span><span>(</span><span>sparklyr</span><span>)</span><span>
</span><span>sc</span><span> </span><span>&lt;-</span><span> </span><span>spark_connect</span><span>(</span><span>master</span><span> </span><span>=</span><span> </span><span>&quot;local&quot;</span><span>,</span><span> </span><span>version</span><span> </span><span>=</span><span> </span><span>&quot;2.0.0&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>library</span><span>(</span><span>ggplot2</span><span>)</span><span>
</span><span>library</span><span>(</span><span>ggrepel</span><span>)</span><span>
</span><span>my_theme</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>base_size</span><span> </span><span>=</span><span> </span><span>12</span><span>,</span><span> </span><span>base_family</span><span> </span><span>=</span><span> </span><span>&quot;sans&quot;</span><span>){</span><span>
</span><span>theme_minimal</span><span>(</span><span>base_size</span><span> </span><span>=</span><span> </span><span>base_size</span><span>,</span><span> </span><span>base_family</span><span> </span><span>=</span><span> </span><span>base_family</span><span>)</span><span> </span><span>+</span><span>
</span><span>theme</span><span>(</span><span>
</span><span>axis.text</span><span> </span><span>=</span><span> </span><span>element_text</span><span>(</span><span>size</span><span> </span><span>=</span><span> </span><span>12</span><span>),</span><span>
</span><span>axis.title</span><span> </span><span>=</span><span> </span><span>element_text</span><span>(</span><span>size</span><span> </span><span>=</span><span> </span><span>14</span><span>),</span><span>
</span><span>panel.grid.major</span><span> </span><span>=</span><span> </span><span>element_line</span><span>(</span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;grey&quot;</span><span>),</span><span>
</span><span>panel.grid.minor</span><span> </span><span>=</span><span> </span><span>element_blank</span><span>(),</span><span>
</span><span>panel.background</span><span> </span><span>=</span><span> </span><span>element_rect</span><span>(</span><span>fill</span><span> </span><span>=</span><span> </span><span>&quot;aliceblue&quot;</span><span>),</span><span>
</span><span>strip.background</span><span> </span><span>=</span><span> </span><span>element_rect</span><span>(</span><span>fill</span><span> </span><span>=</span><span> </span><span>&quot;darkgrey&quot;</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;grey&quot;</span><span>,</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>1</span><span>),</span><span>
</span><span>strip.text</span><span> </span><span>=</span><span> </span><span>element_text</span><span>(</span><span>face</span><span> </span><span>=</span><span> </span><span>&quot;bold&quot;</span><span>,</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>12</span><span>,</span><span> </span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;white&quot;</span><span>),</span><span>
</span><span>legend.position</span><span> </span><span>=</span><span> </span><span>&quot;right&quot;</span><span>,</span><span>
</span><span>legend.justification</span><span> </span><span>=</span><span> </span><span>&quot;top&quot;</span><span>,</span><span> </span><span>panel.border</span><span> </span><span>=</span><span> </span><span>element_rect</span><span>(</span><span>color</span><span> </span><span>=</span><span> </span><span>&quot;grey&quot;</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>NA</span><span>,</span><span> </span><span>size</span><span> </span><span>=</span><span> </span><span>0.5</span><span>)</span><span>
</span><span>)</span><span>
</span><span>}</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>arrhythmia_sc</span><span> </span><span>&lt;-</span><span> </span><span>copy_to</span><span>(</span><span>sc</span><span>,</span><span> </span><span>arrhythmia_subset</span><span>)</span><span>
</span><span>arrhythmia_hf</span><span> </span><span>&lt;-</span><span> </span><span>as_h2o_frame</span><span>(</span><span>sc</span><span>,</span><span> </span><span>arrhythmia_sc</span><span>,</span><span> </span><span>strict_version_check</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>arrhythmia_hf</span><span>[,</span><span> </span><span>2</span><span>]</span><span> </span><span>&lt;-</span><span> </span><span>h</span><span>2</span><span>o.asfactor</span><span>(</span><span>arrhythmia_hf</span><span>[,</span><span> </span><span>2</span><span>])</span><span>
</span><span>arrhythmia_hf</span><span>[,</span><span> </span><span>3</span><span>]</span><span> </span><span>&lt;-</span><span> </span><span>h</span><span>2</span><span>o.asfactor</span><span>(</span><span>arrhythmia_hf</span><span>[,</span><span> </span><span>3</span><span>])</span><span>
</span><span>splits</span><span> </span><span>&lt;-</span><span> </span><span>h</span><span>2</span><span>o.splitFrame</span><span>(</span><span>arrhythmia_hf</span><span>,</span><span> </span><span>ratios</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>0.7</span><span>,</span><span> </span><span>0.15</span><span>),</span><span> </span><span>seed</span><span> </span><span>=</span><span> </span><span>1</span><span>)</span><span>
</span><span>train</span><span> </span><span>&lt;-</span><span> </span><span>splits</span><span>[[</span><span>1</span><span>]]</span><span>
</span><span>valid</span><span> </span><span>&lt;-</span><span> </span><span>splits</span><span>[[</span><span>2</span><span>]]</span><span>
</span><span>test</span><span> </span><span>&lt;-</span><span> </span><span>splits</span><span>[[</span><span>3</span><span>]]</span><span>
</span><span>response</span><span> </span><span>&lt;-</span><span> </span><span>&quot;diagnosis&quot;</span><span>
</span><span>weights</span><span> </span><span>&lt;-</span><span> </span><span>&quot;weights&quot;</span><span>
</span><span>features</span><span> </span><span>&lt;-</span><span> </span><span>setdiff</span><span>(</span><span>colnames</span><span>(</span><span>train</span><span>),</span><span> </span><span>c</span><span>(</span><span>response</span><span>,</span><span> </span><span>weights</span><span>,</span><span> </span><span>&quot;class&quot;</span><span>))</span><span>
</span></code></pre>
</div>
<p></p>
<h3>Grid Search</h3>
<p>We can use the <code>h2o.grid()</code> function to perform a Random Grid Search (RGS). We could also test all possible combinations of parameters with Cartesian Grid or exhaustive search, but RGS is much faster when we have a large number of possible combinations and usually finds sufficiently accurate models.</p>
<p>For RGS, we first define a set of hyper-parameters and search criteria to fine-tune our models. Because there are many hyper-parameters, each with a range of possible values, we want to find an (ideally) optimal combination to maximize our model’s accuracy.<br>
We can also specify how long we want to run the grid search for. Based on the results of each model tested in the grid, we can choose the one with the highest accuracy or best performance for the question on hand.</p>
<h4>Activation Functions</h4>
<ul>
<li><code>Rectifier</code>: is the default activation function. It is the fastest and most versatile option. It can lead to instability though and tends to be lower in accuracy.</li>
<li><code>Tanh</code>: The hyperbolic tangent is a scaled and shifted variant of the sigmoid activation function. It can take on values from -1 to 1 and centers around 0. Tanh needs more computational power than e.g. the Rectifier function.</li>
<li>
<p><code>Maxout</code>: is an activation function that is the max of the inputs. It is computationally quite demanding but can produce high accuracy models.</p>
</li>
<li><code>...WithDropout</code>: When we specify <code>with dropout</code>, a random subset of the network is trained and the weights of all sub-networks are averaged. It works together with the parameter <code>hidden_dropout_ratios</code>, which controls the amount of layer neurons that are randomly dropped for each hidden layer. Hidden dropout ratios are useful for preventing overfitting on learned features.</li>
</ul>
<h4>Hidden layers</h4>
<ul>
<li>are the most important hyper-parameter to set for deep neural networks, as they specify how many hidden layers and how many nodes per hidden layer the model should learn</li>
</ul>
<h4>L1 and L2 penalties</h4>
<ul>
<li><code>L1</code>: lets only strong weights survive</li>
<li><code>L2</code>: prevents any single weight from getting too big.</li>
</ul>
<h4>Rho and Epsilon</h4>
<ul>
<li><code>rho</code>: similar to prior weight updates</li>
<li><code>epsilon</code>: prevents getting stuck in local optima</li>
</ul>
<div>
<pre><code><span>hyper_params</span><span> </span><span>&lt;-</span><span> </span><span>list</span><span>(</span><span>
</span><span>activation</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;Rectifier&quot;</span><span>,</span><span> </span><span>&quot;Maxout&quot;</span><span>,</span><span> </span><span>&quot;Tanh&quot;</span><span>,</span><span> </span><span>&quot;RectifierWithDropout&quot;</span><span>,</span><span> </span><span>&quot;MaxoutWithDropout&quot;</span><span>,</span><span> </span><span>&quot;TanhWithDropout&quot;</span><span>),</span><span> </span><span>hidden</span><span> </span><span>=</span><span> </span><span>list</span><span>(</span><span>c</span><span>(</span><span>5</span><span>,</span><span> </span><span>5</span><span>,</span><span> </span><span>5</span><span>,</span><span> </span><span>5</span><span>,</span><span> </span><span>5</span><span>),</span><span> </span><span>c</span><span>(</span><span>10</span><span>,</span><span> </span><span>10</span><span>,</span><span> </span><span>10</span><span>,</span><span> </span><span>10</span><span>),</span><span> </span><span>c</span><span>(</span><span>50</span><span>,</span><span> </span><span>50</span><span>,</span><span> </span><span>50</span><span>),</span><span> </span><span>c</span><span>(</span><span>100</span><span>,</span><span> </span><span>100</span><span>,</span><span> </span><span>100</span><span>)),</span><span>
</span><span>epochs</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>50</span><span>,</span><span> </span><span>100</span><span>,</span><span> </span><span>200</span><span>),</span><span>
</span><span>l</span><span>1</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>0</span><span>,</span><span> </span><span>0.00001</span><span>,</span><span> </span><span>0.0001</span><span>),</span><span> </span><span>l</span><span>2</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>0</span><span>,</span><span> </span><span>0.00001</span><span>,</span><span> </span><span>0.0001</span><span>),</span><span>
</span><span>rate</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>0</span><span>,</span><span> </span><span>01</span><span>,</span><span> </span><span>0.005</span><span>,</span><span> </span><span>0.001</span><span>),</span><span>
</span><span>rate_annealing</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>1e-8</span><span>,</span><span> </span><span>1e-7</span><span>,</span><span> </span><span>1e-6</span><span>),</span><span>
</span><span>rho</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>0.9</span><span>,</span><span> </span><span>0.95</span><span>,</span><span> </span><span>0.99</span><span>,</span><span> </span><span>0.999</span><span>),</span><span>
</span><span>epsilon</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>1e-10</span><span>,</span><span> </span><span>1e-8</span><span>,</span><span> </span><span>1e-6</span><span>,</span><span> </span><span>1e-4</span><span>),</span><span>
</span><span>momentum_start</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>0</span><span>,</span><span> </span><span>0.5</span><span>),</span><span>
</span><span>momentum_stable</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>0.99</span><span>,</span><span> </span><span>0.5</span><span>,</span><span> </span><span>0</span><span>),</span><span>
</span><span>input_dropout_ratio</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>0</span><span>,</span><span> </span><span>0.1</span><span>,</span><span> </span><span>0.2</span><span>),</span><span>
</span><span>max_w2</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>10</span><span>,</span><span> </span><span>100</span><span>,</span><span> </span><span>1000</span><span>,</span><span> </span><span>3.4028235e+38</span><span>)</span><span>
</span><span>)</span><span>
</span></code></pre>
</div>
<p></p>
<h4>Early stopping criteria</h4>
<ul>
<li><code>stopping_metric</code>: metric that we want to use as stopping criterion</li>
<li><code>stopping_tolerance</code> and <code>stopping_rounds</code>: training stops when the the stopping metric does not improve by the stopping tolerance proportion any more (e.g. by 0.05 or 5%) for the number of consecutive rounds defined by stopping rounds.</li>
</ul>
<div>
<pre><code><span>search_criteria</span><span> </span><span>&lt;-</span><span> </span><span>list</span><span>(</span><span>strategy</span><span> </span><span>=</span><span> </span><span>&quot;RandomDiscrete&quot;</span><span>,</span><span> </span><span>max_models</span><span> </span><span>=</span><span> </span><span>100</span><span>,</span><span>
</span><span>max_runtime_secs</span><span> </span><span>=</span><span> </span><span>900</span><span>,</span><span>
</span><span>stopping_tolerance</span><span> </span><span>=</span><span> </span><span>0.001</span><span>,</span><span>
</span><span>stopping_rounds</span><span> </span><span>=</span><span> </span><span>15</span><span>,</span><span>
</span><span>seed</span><span> </span><span>=</span><span> </span><span>42</span><span>)</span><span>
</span></code></pre>
</div>
<p>Now, we can train the model with combinations of hyper-parameters from our specified stopping criteria and hyper-parameter grid.</p>
<div>
<pre><code><span>dl_grid</span><span> </span><span>&lt;-</span><span> </span><span>h</span><span>2</span><span>o.grid</span><span>(</span><span>algorithm</span><span> </span><span>=</span><span> </span><span>&quot;deeplearning&quot;</span><span>,</span><span> </span><span>x</span><span> </span><span>=</span><span> </span><span>features</span><span>,</span><span>
</span><span>y</span><span> </span><span>=</span><span> </span><span>response</span><span>,</span><span>
</span><span>weights_column</span><span> </span><span>=</span><span> </span><span>weights</span><span>,</span><span>
</span><span>grid_id</span><span> </span><span>=</span><span> </span><span>&quot;dl_grid&quot;</span><span>,</span><span>
</span><span>training_frame</span><span> </span><span>=</span><span> </span><span>train</span><span>,</span><span>
</span><span>validation_frame</span><span> </span><span>=</span><span> </span><span>valid</span><span>,</span><span>
</span><span>nfolds</span><span> </span><span>=</span><span> </span><span>25</span><span>,</span><span> </span><span>fold_assignment</span><span> </span><span>=</span><span> </span><span>&quot;Stratified&quot;</span><span>,</span><span>
</span><span>hyper_params</span><span> </span><span>=</span><span> </span><span>hyper_params</span><span>,</span><span>
</span><span>search_criteria</span><span> </span><span>=</span><span> </span><span>search_criteria</span><span>,</span><span>
</span><span>seed</span><span> </span><span>=</span><span> </span><span>42</span><span>
</span><span>)</span><span>
</span></code></pre>
</div>
<p>We now want to extract the best model from the grid model list. What makes a model <em>the best</em> depends on the question you want to address with it: in some cases, the model with highest AUC is the most suitable, or the one with the lowest mean squared error, etc. See <a target="_blank" rel="nofollow" href="https://shiring.github.io/machine_learning/2017/02/27/h2o">last week’s post</a> again for a more detailed discussion of performance metrics.</p>
<p>For demonstration purposes, I am choosing the best models from a range of possible quality criteria. We first use the <code>h2o.getGrid()</code> function to sort all models by the quality metric we choose (depending on the metric, you want it ordered by descending or ascending values). We can then get the model that’s the first in the list to work with further. This model’s hyper-parameters can be found with <code>best_model@allparameters</code>. You can now work with your best model as with any regular model in <strong>h2o</strong> (for an example see <a target="_blank" rel="nofollow" href="https://shiring.github.io/machine_learning/2017/02/27/h2o">last week’s post</a>).</p>
<div>
<pre><code><span># performance metrics where smaller is better -> order with decreasing = FALSE
</span><span>sort_options_1</span><span> </span><span>&lt;-</span><span> </span><span>c</span><span>(</span><span>&quot;mean_per_class_error&quot;</span><span>,</span><span> </span><span>&quot;mse&quot;</span><span>,</span><span> </span><span>&quot;err&quot;</span><span>)</span><span>
</span><span>for</span><span> </span><span>(</span><span>sort_by_1</span><span> </span><span>in</span><span> </span><span>sort_options_1</span><span>)</span><span> </span><span>{</span><span>
</span><span>grid</span><span> </span><span>&lt;-</span><span> </span><span>h</span><span>2</span><span>o.getGrid</span><span>(</span><span>&quot;dl_grid&quot;</span><span>,</span><span> </span><span>sort_by</span><span> </span><span>=</span><span> </span><span>sort_by_1</span><span>,</span><span> </span><span>decreasing</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>)</span><span>
</span><span>model_ids</span><span> </span><span>&lt;-</span><span> </span><span>grid</span><span>@</span><span>model_ids</span><span>
</span><span>best_model</span><span> </span><span>&lt;-</span><span> </span><span>h</span><span>2</span><span>o.getModel</span><span>(</span><span>model_ids</span><span>[[</span><span>1</span><span>]])</span><span>
</span><span>assign</span><span>(</span><span>paste0</span><span>(</span><span>&quot;best_model_&quot;</span><span>,</span><span> </span><span>sort_by_1</span><span>),</span><span> </span><span>best_model</span><span>)</span><span>
</span><span>}</span><span>
</span><span># performance metrics where bigger is better -> order with decreasing = TRUE
</span><span>sort_options_2</span><span> </span><span>&lt;-</span><span> </span><span>c</span><span>(</span><span>&quot;auc&quot;</span><span>,</span><span> </span><span>&quot;precision&quot;</span><span>,</span><span> </span><span>&quot;accuracy&quot;</span><span>,</span><span> </span><span>&quot;recall&quot;</span><span>,</span><span> </span><span>&quot;specificity&quot;</span><span>)</span><span>
</span><span>for</span><span> </span><span>(</span><span>sort_by_2</span><span> </span><span>in</span><span> </span><span>sort_options_2</span><span>)</span><span> </span><span>{</span><span>
</span><span>grid</span><span> </span><span>&lt;-</span><span> </span><span>h</span><span>2</span><span>o.getGrid</span><span>(</span><span>&quot;dl_grid&quot;</span><span>,</span><span> </span><span>sort_by</span><span> </span><span>=</span><span> </span><span>sort_by_2</span><span>,</span><span> </span><span>decreasing</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span>
</span><span>model_ids</span><span> </span><span>&lt;-</span><span> </span><span>grid</span><span>@</span><span>model_ids</span><span>
</span><span>best_model</span><span> </span><span>&lt;-</span><span> </span><span>h</span><span>2</span><span>o.getModel</span><span>(</span><span>model_ids</span><span>[[</span><span>1</span><span>]])</span><span>
</span><span>assign</span><span>(</span><span>paste0</span><span>(</span><span>&quot;best_model_&quot;</span><span>,</span><span> </span><span>sort_by_2</span><span>),</span><span> </span><span>best_model</span><span>)</span><span>
</span><span>}</span><span>
</span></code></pre>
</div>
<p>Let’s plot the mean per class error for each best model:</p>
<div>
<pre><code><span>library</span><span>(</span><span>tibble</span><span>)</span><span>
</span><span>sort_options</span><span> </span><span>&lt;-</span><span> </span><span>c</span><span>(</span><span>&quot;mean_per_class_error&quot;</span><span>,</span><span> </span><span>&quot;mse&quot;</span><span>,</span><span> </span><span>&quot;err&quot;</span><span>,</span><span> </span><span>&quot;auc&quot;</span><span>,</span><span> </span><span>&quot;precision&quot;</span><span>,</span><span> </span><span>&quot;accuracy&quot;</span><span>,</span><span> </span><span>&quot;recall&quot;</span><span>,</span><span> </span><span>&quot;specificity&quot;</span><span>)</span><span>
</span><span>for</span><span> </span><span>(</span><span>sort_by</span><span> </span><span>in</span><span> </span><span>sort_options</span><span>)</span><span> </span><span>{</span><span>
</span><span>best_model</span><span> </span><span>&lt;-</span><span> </span><span>get</span><span>(</span><span>paste0</span><span>(</span><span>&quot;best_model_&quot;</span><span>,</span><span> </span><span>sort_by</span><span>))</span><span>
</span><span>errors</span><span> </span><span>&lt;-</span><span> </span><span>h</span><span>2</span><span>o.mean_per_class_error</span><span>(</span><span>best_model</span><span>,</span><span> </span><span>train</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span>valid</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span>xval</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span>
</span><span>errors_df</span><span> </span><span>&lt;-</span><span> </span><span>data.frame</span><span>(</span><span>model_id</span><span> </span><span>=</span><span> </span><span>best_model</span><span>@</span><span>model_id</span><span>,</span><span> </span><span>sort</span><span> </span><span>=</span><span> </span><span>sort_by</span><span>,</span><span> </span><span>errors</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>rownames_to_column</span><span>(</span><span>var</span><span> </span><span>=</span><span> </span><span>&quot;rowname&quot;</span><span>)</span><span>
</span><span>if</span><span> </span><span>(</span><span>sort_by</span><span> </span><span>==</span><span> </span><span>&quot;mean_per_class_error&quot;</span><span>)</span><span> </span><span>{</span><span>
</span><span>errors_df_comb</span><span> </span><span>&lt;-</span><span> </span><span>errors_df</span><span>
</span><span>}</span><span> </span><span>else</span><span> </span><span>{</span><span>
</span><span>errors_df_comb</span><span> </span><span>&lt;-</span><span> </span><span>rbind</span><span>(</span><span>errors_df_comb</span><span>,</span><span> </span><span>errors_df</span><span>)</span><span>
</span><span>}</span><span>
</span><span>}</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>order</span><span> </span><span>&lt;-</span><span> </span><span>subset</span><span>(</span><span>errors_df_comb</span><span>,</span><span> </span><span>rowname</span><span> </span><span>==</span><span> </span><span>&quot;xval&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>arrange</span><span>(</span><span>errors</span><span>)</span><span>
</span><span>errors_df_comb</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>sort</span><span> </span><span>=</span><span> </span><span>factor</span><span>(</span><span>sort</span><span>,</span><span> </span><span>levels</span><span> </span><span>=</span><span> </span><span>order</span><span>$</span><span>sort</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>sort</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>errors</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>model_id</span><span>))</span><span> </span><span>+</span><span>
</span><span>facet_grid</span><span>(</span><span>rowname</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span> </span><span>scales</span><span> </span><span>=</span><span> </span><span>&quot;free&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>geom_bar</span><span>(</span><span>stat</span><span> </span><span>=</span><span> </span><span>&quot;identity&quot;</span><span>,</span><span> </span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.8</span><span>)</span><span> </span><span>+</span><span>
</span><span>scale_fill_brewer</span><span>(</span><span>palette</span><span> </span><span>=</span><span> </span><span>&quot;Set1&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>my_theme</span><span>()</span><span> </span><span>+</span><span>
</span><span>theme</span><span>(</span><span>axis.text.x</span><span> </span><span>=</span><span> </span><span>element_text</span><span>(</span><span>angle</span><span> </span><span>=</span><span> </span><span>45</span><span>,</span><span> </span><span>vjust</span><span> </span><span>=</span><span> </span><span>1</span><span>,</span><span> </span><span>hjust</span><span>=</span><span>1</span><span>),</span><span>
</span><span>plot.margin</span><span> </span><span>=</span><span> </span><span>unit</span><span>(</span><span>c</span><span>(</span><span>0.5</span><span>,</span><span> </span><span>0</span><span>,</span><span> </span><span>0</span><span>,</span><span> </span><span>1</span><span>),</span><span> </span><span>&quot;cm&quot;</span><span>))</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img alt="" src="https://i2.wp.com/shiring.github.io/machine_learning/2017/03/07/grid_search_files/figure-markdown_github/unnamed-chunk-11-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p></p>
<h3>Model performance</h3>
<p>The ultimate performance test for our model will be it’s prediction accuracy on the test set it hasn’t seen before. Here, I will compare the AUC and mean squared error for each best model from before. You could of course look at any other quality metric that is most appropriate for your model.</p>
<div>
<pre><code><span>for</span><span> </span><span>(</span><span>sort_by</span><span> </span><span>in</span><span> </span><span>sort_options</span><span>)</span><span> </span><span>{</span><span>
</span><span>best_model</span><span> </span><span>&lt;-</span><span> </span><span>get</span><span>(</span><span>paste0</span><span>(</span><span>&quot;best_model_&quot;</span><span>,</span><span> </span><span>sort_by</span><span>))</span><span>
</span><span>mse_auc_test</span><span> </span><span>&lt;-</span><span> </span><span>data.frame</span><span>(</span><span>model_id</span><span> </span><span>=</span><span> </span><span>best_model</span><span>@</span><span>model_id</span><span>,</span><span>
</span><span>sort</span><span> </span><span>=</span><span> </span><span>sort_by</span><span>,</span><span> </span><span>mse</span><span> </span><span>=</span><span> </span><span>h</span><span>2</span><span>o.mse</span><span>(</span><span>h</span><span>2</span><span>o.performance</span><span>(</span><span>best_model</span><span>,</span><span> </span><span>test</span><span>)),</span><span>
</span><span>auc</span><span> </span><span>=</span><span> </span><span>h</span><span>2</span><span>o.auc</span><span>(</span><span>h</span><span>2</span><span>o.performance</span><span>(</span><span>best_model</span><span>,</span><span> </span><span>test</span><span>)))</span><span>
</span><span>if</span><span> </span><span>(</span><span>sort_by</span><span> </span><span>==</span><span> </span><span>&quot;mean_per_class_error&quot;</span><span>)</span><span> </span><span>{</span><span>
</span><span>mse_auc_test_comb</span><span> </span><span>&lt;-</span><span> </span><span>mse_auc_test</span><span>
</span><span>}</span><span> </span><span>else</span><span> </span><span>{</span><span>
</span><span>mse_auc_test_comb</span><span> </span><span>&lt;-</span><span> </span><span>rbind</span><span>(</span><span>mse_auc_test_comb</span><span>,</span><span> </span><span>mse_auc_test</span><span>)</span><span>
</span><span>}</span><span>
</span><span>}</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>library</span><span>(</span><span>tidyr</span><span>)</span><span>
</span><span>mse_auc_test_comb</span><span> </span><span>%>%</span><span>
</span><span>gather</span><span>(</span><span>x</span><span>,</span><span> </span><span>y</span><span>,</span><span> </span><span>mse</span><span>:</span><span>auc</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>sort</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>y</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>model_id</span><span>))</span><span> </span><span>+</span><span>
</span><span>facet_grid</span><span>(</span><span>x</span><span> </span><span>~</span><span> </span><span>.</span><span>,</span><span> </span><span>scales</span><span> </span><span>=</span><span> </span><span>&quot;free&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>geom_bar</span><span>(</span><span>stat</span><span> </span><span>=</span><span> </span><span>&quot;identity&quot;</span><span>,</span><span> </span><span>alpha</span><span> </span><span>=</span><span> </span><span>0.8</span><span>,</span><span> </span><span>position</span><span> </span><span>=</span><span> </span><span>&quot;dodge&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>scale_fill_brewer</span><span>(</span><span>palette</span><span> </span><span>=</span><span> </span><span>&quot;Set1&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>my_theme</span><span>()</span><span> </span><span>+</span><span>
</span><span>theme</span><span>(</span><span>axis.text.x</span><span> </span><span>=</span><span> </span><span>element_text</span><span>(</span><span>angle</span><span> </span><span>=</span><span> </span><span>45</span><span>,</span><span> </span><span>vjust</span><span> </span><span>=</span><span> </span><span>1</span><span>,</span><span> </span><span>hjust</span><span>=</span><span>1</span><span>),</span><span>
</span><span>plot.margin</span><span> </span><span>=</span><span> </span><span>unit</span><span>(</span><span>c</span><span>(</span><span>0.5</span><span>,</span><span> </span><span>0</span><span>,</span><span> </span><span>0</span><span>,</span><span> </span><span>1.5</span><span>),</span><span> </span><span>&quot;cm&quot;</span><span>))</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>&quot;value&quot;</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img alt="" src="https://i1.wp.com/shiring.github.io/machine_learning/2017/03/07/grid_search_files/figure-markdown_github/unnamed-chunk-13-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>I will then create a dataframe with predictions for each test sample with all best models. As in last week’s post, I want to compare the default predictions with stringent predictions.</p>
<div>
<pre><code><span>for</span><span> </span><span>(</span><span>sort_by</span><span> </span><span>in</span><span> </span><span>sort_options</span><span>)</span><span> </span><span>{</span><span>
</span><span>best_model</span><span> </span><span>&lt;-</span><span> </span><span>get</span><span>(</span><span>paste0</span><span>(</span><span>&quot;best_model_&quot;</span><span>,</span><span> </span><span>sort_by</span><span>))</span><span>
</span><span>finalRf_predictions</span><span> </span><span>&lt;-</span><span> </span><span>data.frame</span><span>(</span><span>model_id</span><span> </span><span>=</span><span> </span><span>rep</span><span>(</span><span>best_model</span><span>@</span><span>model_id</span><span>,</span><span> </span><span>nrow</span><span>(</span><span>test</span><span>)),</span><span>
</span><span>sort</span><span> </span><span>=</span><span> </span><span>rep</span><span>(</span><span>sort_by</span><span>,</span><span> </span><span>nrow</span><span>(</span><span>test</span><span>)),</span><span>
</span><span>class</span><span> </span><span>=</span><span> </span><span>as.vector</span><span>(</span><span>test</span><span>$</span><span>class</span><span>),</span><span> </span><span>actual</span><span> </span><span>=</span><span> </span><span>as.vector</span><span>(</span><span>test</span><span>$</span><span>diagnosis</span><span>),</span><span> </span><span>as.data.frame</span><span>(</span><span>h</span><span>2</span><span>o.predict</span><span>(</span><span>object</span><span> </span><span>=</span><span> </span><span>best_model</span><span>,</span><span> </span><span>newdata</span><span> </span><span>=</span><span> </span><span>test</span><span>)))</span><span>
</span><span>finalRf_predictions</span><span>$</span><span>accurate</span><span> </span><span>&lt;-</span><span> </span><span>ifelse</span><span>(</span><span>finalRf_predictions</span><span>$</span><span>actual</span><span> </span><span>==</span><span> </span><span>finalRf_predictions</span><span>$</span><span>predict</span><span>,</span><span> </span><span>&quot;yes&quot;</span><span>,</span><span> </span><span>&quot;no&quot;</span><span>)</span><span>
</span><span>finalRf_predictions</span><span>$</span><span>predict_stringent</span><span> </span><span>&lt;-</span><span> </span><span>ifelse</span><span>(</span><span>finalRf_predictions</span><span>$</span><span>arrhythmia</span><span> </span><span>></span><span> </span><span>0.8</span><span>,</span><span> </span><span>&quot;arrhythmia&quot;</span><span>,</span><span> </span><span>ifelse</span><span>(</span><span>finalRf_predictions</span><span>$</span><span>healthy</span><span> </span><span>></span><span> </span><span>0.8</span><span>,</span><span> </span><span>&quot;healthy&quot;</span><span>,</span><span> </span><span>&quot;uncertain&quot;</span><span>))</span><span>
</span><span>finalRf_predictions</span><span>$</span><span>accurate_stringent</span><span> </span><span>&lt;-</span><span> </span><span>ifelse</span><span>(</span><span>finalRf_predictions</span><span>$</span><span>actual</span><span> </span><span>==</span><span> </span><span>finalRf_predictions</span><span>$</span><span>predict_stringent</span><span>,</span><span> </span><span>&quot;yes&quot;</span><span>,</span><span> </span><span>ifelse</span><span>(</span><span>finalRf_predictions</span><span>$</span><span>predict_stringent</span><span> </span><span>==</span><span> </span><span>&quot;uncertain&quot;</span><span>,</span><span> </span><span>&quot;na&quot;</span><span>,</span><span> </span><span>&quot;no&quot;</span><span>))</span><span>
</span><span>if</span><span> </span><span>(</span><span>sort_by</span><span> </span><span>==</span><span> </span><span>&quot;mean_per_class_error&quot;</span><span>)</span><span> </span><span>{</span><span>
</span><span>finalRf_predictions_comb</span><span> </span><span>&lt;-</span><span> </span><span>finalRf_predictions</span><span>
</span><span>}</span><span> </span><span>else</span><span> </span><span>{</span><span>
</span><span>finalRf_predictions_comb</span><span> </span><span>&lt;-</span><span> </span><span>rbind</span><span>(</span><span>finalRf_predictions_comb</span><span>,</span><span> </span><span>finalRf_predictions</span><span>)</span><span>
</span><span>}</span><span>
</span><span>}</span><span>
</span></code></pre>
</div>
<p>To get a better overview, I am going to plot the predictions (default and stringent):</p>
<div>
<pre><code><span>finalRf_predictions_comb</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>actual</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>accurate</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_bar</span><span>(</span><span>position</span><span> </span><span>=</span><span> </span><span>&quot;dodge&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>scale_fill_brewer</span><span>(</span><span>palette</span><span> </span><span>=</span><span> </span><span>&quot;Set1&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>my_theme</span><span>()</span><span> </span><span>+</span><span>
</span><span>facet_wrap</span><span>(</span><span>~</span><span> </span><span>sort</span><span>,</span><span> </span><span>ncol</span><span> </span><span>=</span><span> </span><span>4</span><span>)</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>fill</span><span> </span><span>=</span><span> </span><span>&quot;Were\npredictions\naccurate?&quot;</span><span>,</span><span>
</span><span>title</span><span> </span><span>=</span><span> </span><span>&quot;Default predictions&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img alt="" src="https://i0.wp.com/shiring.github.io/machine_learning/2017/03/07/grid_search_files/figure-markdown_github/unnamed-chunk-15-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<div>
<pre><code><span>finalRf_predictions_comb</span><span> </span><span>%>%</span><span>
</span><span>subset</span><span>(</span><span>accurate_stringent</span><span> </span><span>!=</span><span> </span><span>&quot;na&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>actual</span><span>,</span><span> </span><span>fill</span><span> </span><span>=</span><span> </span><span>accurate_stringent</span><span>))</span><span> </span><span>+</span><span>
</span><span>geom_bar</span><span>(</span><span>position</span><span> </span><span>=</span><span> </span><span>&quot;dodge&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>scale_fill_brewer</span><span>(</span><span>palette</span><span> </span><span>=</span><span> </span><span>&quot;Set1&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>my_theme</span><span>()</span><span> </span><span>+</span><span>
</span><span>facet_wrap</span><span>(</span><span>~</span><span> </span><span>sort</span><span>,</span><span> </span><span>ncol</span><span> </span><span>=</span><span> </span><span>4</span><span>)</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>fill</span><span> </span><span>=</span><span> </span><span>&quot;Were\npredictions\naccurate?&quot;</span><span>,</span><span>
</span><span>title</span><span> </span><span>=</span><span> </span><span>&quot;Stringent predictions&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p><img alt="" src="https://i1.wp.com/shiring.github.io/machine_learning/2017/03/07/grid_search_files/figure-markdown_github/unnamed-chunk-16-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>With predictions made by different models, we can see where each model performs best. This obviously corresponds with the quality metric we chose to define the best model. Stringent prediction thresholds reduced the number of false predictions but of course also the number of predictions made at all.</p>
<p>We can now decide which model is most relevant. Let’s say, we want this model to give recommendations for further validating a diagnosis of arrhythmia, we might want to detect as many arrhythmia cases as possible, while being okay with also getting some erroneously diagnosed healthy subjects. In this case, the people could flag patients with likely arrhythmia for further testing. Here, this would mean that we wanted to minimize the number of wrong predictions of arrhythmia cases, so we would prefer the mse, auc and specificity model (which is the same model, chosen by all three qualitry metrics). The worst model for this purpose would be the recall model, which also did not make any predictions that passed the stringency threshold. The recall model would have been best if we wanted to be confident that healthy people get flagged correctly. However, in a real-life setting, you can imagine that it would be much worse if a sick patient got sent home because he was wrongly diagnosed as healthy than if a healthy person got submitted to further testing for possible arrhythmia.</p>
<hr>
<p>Other machine learning topics I have covered include</p>
<ul>
<li>how to run machine learning with Apache Spark and H2O: <a target="_blank" rel="nofollow" href="https://shiring.github.io/machine_learning/2017/02/27/h2o">Building deep neural nets with h2o and rsparkling that predict arrhythmia of the heart</a></li>
<li>how to run machine learning with Apache Spark in R: <a target="_blank" rel="nofollow" href="https://shiring.github.io/machine_learning/2017/02/19/food_spark">Predicting food preferences with sparklyr (machine learning)</a></li>
<li>a basic machine learning workflow: <a target="_blank" rel="nofollow" href="https://shiring.github.io/machine_learning/2016/11/27/flu_outcome_ML_post">Can we predict flu deaths with Machine Learning and R?</a></li>
<li>extreme gradient boosting: <a target="_blank" rel="nofollow" href="https://shiring.github.io/machine_learning/2016/12/02/flu_outcome_ML_2_post">Extreme Gradient Boosting and Preprocessing in Machine Learning – Addendum to predicting flu outcome with R</a></li>
<li>feature selection: <a target="_blank" rel="nofollow" href="https://shiring.github.io/machine_learning/2017/01/15/rfe_ga_post">Feature Selection in Machine Learning (Breast Cancer Datasets)</a></li>
</ul>
<hr>
<p></p>
<div>
<pre><code>## R version 3.3.2 (2016-10-31)
## Platform: x86_64-w64-mingw32/x64 (64-bit)
## Running under: Windows 7 x64 (build 7601) Service Pack 1
## 
## locale:
## [1] LC_COLLATE=English_United States.1252 
## [2] LC_CTYPE=English_United States.1252   
## [3] LC_MONETARY=English_United States.1252
## [4] LC_NUMERIC=C                          
## [5] LC_TIME=English_United States.1252    
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] tidyr_0.6.1         tibble_1.2          ggrepel_0.6.5      
## [4] ggplot2_2.2.1.9000  sparklyr_0.5.3-9002 dplyr_0.5.0        
## [7] h2o_3.10.3.6        rsparkling_0.1.0   
## 
## loaded via a namespace (and not attached):
##  [1] Rcpp_0.12.9        RColorBrewer_1.1-2 plyr_1.8.4        
##  [4] bitops_1.0-6       base64enc_0.1-3    tools_3.3.2       
##  [7] digest_0.6.12      jsonlite_1.2       evaluate_0.10     
## [10] gtable_0.2.0       shiny_1.0.0        DBI_0.5-1         
## [13] rstudioapi_0.6     yaml_2.1.14        parallel_3.3.2    
## [16] withr_1.0.2        httr_1.2.1         stringr_1.2.0     
## [19] knitr_1.15.1       rappdirs_0.3.1     rprojroot_1.2     
## [22] grid_3.3.2         R6_2.2.0           rmarkdown_1.3     
## [25] reshape2_1.4.2     magrittr_1.5       backports_1.0.5   
## [28] scales_0.4.1       htmltools_0.3.5    assertthat_0.1    
## [31] mime_0.5           xtable_1.8-2       colorspace_1.3-2  
## [34] httpuv_1.3.3       labeling_0.3       config_0.2        
## [37] stringi_1.1.2      RCurl_1.95-4.8     lazyeval_0.2.0    
## [40] munsell_0.4.3
</code></pre>
</div>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://shiring.github.io/machine_learning/2017/03/07/grid_search"> Shirin's playgRound</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/itAZBf5mQ3U"></a></div>