<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/deep-learning-from-first-principles-in-python-r-and-octave-part-6/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://gigadom.wordpress.com/2018/04/16/deep-learning-from-first-principles-in-python-r-and-octave-part-6/"> R – Giga thoughts …</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><em>“Today you are You, that is truer than true. There is no one alive who is Youer than You.”</em><br>
Dr. Seuss</p>
<p><em>“Explanations exist; they have existed for all time; there is always a well-known solution to every human problem — neat, plausible, and wrong.”</em><br>
H L Mencken</p>
<h1>Introduction</h1>
<p>In this 6th instalment of ‘Deep Learning from first principles in Python, R and Octave-Part6’, I look at a couple of different initialization techniques used in Deep Learning, L2 regularization and the ‘dropout’ method. Specifically, I implement “He initialization” &amp; “Xavier Initialization”. My earlier posts in this series of Deep Learning included</p>
<p>1. <a rel="nofollow" href="https://gigadom.wordpress.com/2018/01/04/deep-learning-from-basic-principles-in-python-r-and-octave-part-1/" target="_blank">Part 1</a> – In the 1st part, I implemented logistic regression as a simple 2 layer Neural Network<br>
2. <a rel="nofollow" href="https://gigadom.wordpress.com/2018/01/11/deep-learning-from-first-principles-in-python-r-and-octave-part-2/" target="_blank">Part 2</a> – In part 2, implemented the most basic of Neural Networks, with just 1 hidden layer, and any number of activation units in that hidden layer. The implementation was in vectorized Python, R and Octave<br>
3. <a rel="nofollow" href="https://gigadom.wordpress.com/2018/01/30/deep-learning-from-first-principles-in-python-r-and-octave-part-3/" target="_blank">Part 3</a> -In part 3, I derive the equations and also implement a L-Layer Deep Learning network with either the relu, tanh or sigmoid activation function in Python, R and Octave. The output activation unit was a sigmoid function for logistic classification<br>
4. <a rel="nofollow" href="https://gigadom.wordpress.com/2018/02/26/deep-learning-from-first-principles-in-python-r-and-octave-part-4/" target="_blank">Part 4</a> – This part looks at multi-class classification, and I derive the Jacobian of a Softmax function and implement a simple problem to perform multi-class classification.<br>
5. <a rel="nofollow" href="https://gigadom.wordpress.com/2018/03/23/deep-learning-from-first-principles-in-python-r-and-octave-part-5/" target="_blank">Part 5</a> – In the 5th part, I extend the L-Layer Deep Learning network implemented in Part 3, to include the Softmax classification. I also use this L-layer implementation to classify MNIST handwritten digits with Python, R and Octave.</p>
<p>The code in Python, R and Octave are identical, and just take into account some of the minor idiosyncrasies of the individual language. In this post, I implement different initialization techniques (random, He, Xavier), L2 regularization and finally dropout. Hence my generic L-Layer Deep Learning network includes these additional enhancements for enabling/disabling initialization methods, regularization or dropout in the algorithm. It already included sigmoid &amp; softmax output activation for binary and multi-class classification, besides allowing relu, tanh and sigmoid activation for hidden units.</p>
<p>This R Markdown file and the code for Python, R and Octave can be cloned/downloaded from <strong>Github</strong> at <a rel="nofollow" href="https://github.com/tvganesh/DeepLearning-Part6" target="_blank">DeepLearning-Part6</a></p>
<div>
<h2>1. Initialization techniques</h2>
<p>The usual initialization technique is to generate Gaussian or uniform random numbers and multiply it by a small value like 0.01. Two techniques which are used to speed up convergence is the <a rel="nofollow" href="https://www.cv-foundation.org/openaccess/content_iccv_2015/papers/He_Delving_Deep_into_ICCV_2015_paper.pdf" target="_blank">He</a> initialization or <a rel="nofollow" href="http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf" target="_blank">Xavier</a>. These initialization techniques enable gradient descent to converge faster.</p>
<h2>1.1 a Default initialization – Python</h2>
<p>This technique just initializes the weights to small random values based on Gaussian or uniform distribution</p>
<pre><code><span>import</span> numpy <span>as</span> np
<span>import</span> matplotlib
<span>import</span> matplotlib.pyplot <span>as</span> plt
<span>import</span> sklearn.linear_model
<span>import</span> pandas <span>as</span> pd
<span>import</span> sklearn
<span>import</span> sklearn.datasets
exec(open(<span>&quot;DLfunctions61.py&quot;</span>).read())
<span>#Load the data</span>
train_X, train_Y, test_X, test_Y = load_dataset()
<span># Set the layers dimensions</span>
layersDimensions = [<span>2</span>,<span>7</span>,<span>1</span>]

<span># Train a deep learning network with random initialization</span>
parameters = L_Layer_DeepModel(train_X, train_Y, layersDimensions, hiddenActivationFunc=<span>'relu'</span>, outputActivationFunc=<span>&quot;sigmoid&quot;</span>,learningRate = <span>0.6</span>, num_iterations = <span>9000</span>, initType=<span>&quot;default&quot;</span>, print_cost = <span>True</span>,figure=<span>&quot;fig1.png&quot;</span>)

<span># Clear the plot</span>
plt.clf()
plt.close()

<span># Plot the decision boundary</span>
plot_decision_boundary(<span>lambda</span> x: predict(parameters, x.T), train_X, train_Y,str(<span>0.6</span>),figure1=<span>&quot;fig2.png&quot;</span>)</code></pre>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/fig1.png?w=456" data-orig-size="389,278" data-medium-file="https://gigadom.files.wordpress.com/2018/04/fig1.png?w=456?w=300" data-attachment-id="6506" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/fig1.png?w=456" data-permalink="https://gigadom.wordpress.com/2018/04/16/deep-learning-from-first-principles-in-python-r-and-octave-part-6/fig1-17/" data-large-file="https://gigadom.files.wordpress.com/2018/04/fig1.png?w=456?w=389" alt="" data-image-title="fig1" sizes="(max-width: 389px) 100vw, 389px" data-comments-opened="1" class="wp-image-6506" data-recalc-dims="1"></p>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/fig2.png?w=456" data-orig-size="398,278" data-medium-file="https://gigadom.files.wordpress.com/2018/04/fig2.png?w=456?w=300" data-attachment-id="6507" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/fig2.png?w=456" data-permalink="https://gigadom.wordpress.com/2018/04/16/deep-learning-from-first-principles-in-python-r-and-octave-part-6/fig2-14/" data-large-file="https://gigadom.files.wordpress.com/2018/04/fig2.png?w=456?w=398" alt="" data-image-title="fig2" sizes="(max-width: 398px) 100vw, 398px" data-comments-opened="1" class="wp-image-6507" data-recalc-dims="1"></p>
<h2>1.1 b He initialization – Python</h2>
<p>‘He’ initialization attributed to <a rel="nofollow" href="https://www.cv-foundation.org/openaccess/content_iccv_2015/papers/He_Delving_Deep_into_ICCV_2015_paper.pdf" target="_blank">He et al</a>, multiplies the random weights by<br>
<img alt="sqrt{frac{2}{dimension of previous layer}}" title="sqrt{frac{2}{dimension of previous layer}}" src="https://s0.wp.com/latex.php?latex=%5Csqrt%7B%5Cfrac%7B2%7D%7Bdimension%5C+of%5C+previous%5C+layer%7D%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"></p>
<pre><code><span>import</span> numpy <span>as</span> np
<span>import</span> matplotlib
<span>import</span> matplotlib.pyplot <span>as</span> plt
<span>import</span> sklearn.linear_model
<span>import</span> pandas <span>as</span> pd
<span>import</span> sklearn
<span>import</span> sklearn.datasets
exec(open(<span>&quot;DLfunctions61.py&quot;</span>).read())

<span>#Load the data</span>
train_X, train_Y, test_X, test_Y = load_dataset()
<span># Set the layers dimensions</span>
layersDimensions = [<span>2</span>,<span>7</span>,<span>1</span>]

<span># Train a deep learning network with He initialization</span>
parameters = L_Layer_DeepModel(train_X, train_Y, layersDimensions, hiddenActivationFunc=<span>'relu'</span>, outputActivationFunc=<span>&quot;sigmoid&quot;</span>, learningRate =<span>0.6</span>,    num_iterations = <span>10000</span>,initType=<span>&quot;He&quot;</span>,print_cost = <span>True</span>,                           figure=<span>&quot;fig3.png&quot;</span>)

plt.clf()
plt.close()
<span># Plot the decision boundary</span>
plot_decision_boundary(<span>lambda</span> x: predict(parameters, x.T), train_X, train_Y,str(<span>0.6</span>),figure1=<span>&quot;fig4.png&quot;</span>)</code></pre>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/fig3.png?w=456" data-orig-size="570,453" data-medium-file="https://gigadom.files.wordpress.com/2018/04/fig3.png?w=456?w=300" data-attachment-id="6509" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/fig3.png?w=456" data-permalink="https://gigadom.wordpress.com/2018/04/16/deep-learning-from-first-principles-in-python-r-and-octave-part-6/fig3-13/" data-large-file="https://gigadom.files.wordpress.com/2018/04/fig3.png?w=456?w=570" alt="" data-image-title="fig3" sizes="(max-width: 570px) 100vw, 570px" data-comments-opened="1" class="wp-image-6509" data-recalc-dims="1"><br>
<img data-orig-file="https://gigadom.files.wordpress.com/2018/04/fig2.jpg?w=456" data-orig-size="582,453" data-medium-file="https://gigadom.files.wordpress.com/2018/04/fig2.jpg?w=456?w=300" data-attachment-id="6590" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/fig2.jpg?w=456" data-permalink="https://gigadom.wordpress.com/2018/04/16/deep-learning-from-first-principles-in-python-r-and-octave-part-6/fig2-15/" data-large-file="https://gigadom.files.wordpress.com/2018/04/fig2.jpg?w=456?w=582" alt="" data-image-title="fig2" sizes="(max-width: 582px) 100vw, 582px" data-comments-opened="1" class="wp-image-6590" data-recalc-dims="1"></p>
<h2>1.1 c Xavier initialization – Python</h2>
<p><a rel="nofollow" href="http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf" target="_blank">Xavier </a> initialization multiply the random weights by<br>
<img alt="sqrt{frac{1}{dimension of previous layer}}" title="sqrt{frac{1}{dimension of previous layer}}" src="https://s0.wp.com/latex.php?latex=%5Csqrt%7B%5Cfrac%7B1%7D%7Bdimension%5C+of%5C+previous%5C+layer%7D%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"></p>
<pre><code><span>import</span> numpy <span>as</span> np
<span>import</span> matplotlib
<span>import</span> matplotlib.pyplot <span>as</span> plt
<span>import</span> sklearn.linear_model
<span>import</span> pandas <span>as</span> pd
<span>import</span> sklearn
<span>import</span> sklearn.datasets
exec(open(<span>&quot;DLfunctions61.py&quot;</span>).read())

<span>#Load the data</span>
train_X, train_Y, test_X, test_Y = load_dataset()
<span># Set the layers dimensions</span>
layersDimensions = [<span>2</span>,<span>7</span>,<span>1</span>]
 
# Train a L layer Deep Learning network
parameters = L_Layer_DeepModel(train_X, train_Y, layersDimensions, hiddenActivationFunc=<span>'relu'</span>, outputActivationFunc=<span>&quot;sigmoid&quot;</span>,
                            learningRate = <span>0.6</span>,num_iterations = <span>10000</span>, initType=<span>&quot;Xavier&quot;</span>,print_cost = <span>True</span>,
                            figure=<span>&quot;fig5.png&quot;</span>)

<span># Plot the decision boundary</span>
plot_decision_boundary(<span>lambda</span> x: predict(parameters, x.T), train_X, train_Y,str(<span>0.6</span>),figure1=<span>&quot;fig6.png&quot;</span>)</code></pre>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/fig5.png?w=456" data-orig-size="570,453" data-medium-file="https://gigadom.files.wordpress.com/2018/04/fig5.png?w=456?w=300" data-attachment-id="6512" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/fig5.png?w=456" data-permalink="https://gigadom.wordpress.com/2018/04/16/deep-learning-from-first-principles-in-python-r-and-octave-part-6/fig5-7/" data-large-file="https://gigadom.files.wordpress.com/2018/04/fig5.png?w=456?w=570" alt="" data-image-title="fig5" sizes="(max-width: 570px) 100vw, 570px" data-comments-opened="1" class="wp-image-6512" data-recalc-dims="1"><br>
<img data-orig-file="https://gigadom.files.wordpress.com/2018/04/fig6.png?w=456" data-orig-size="582,453" data-medium-file="https://gigadom.files.wordpress.com/2018/04/fig6.png?w=456?w=300" data-attachment-id="6513" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/fig6.png?w=456" data-permalink="https://gigadom.wordpress.com/2018/04/16/deep-learning-from-first-principles-in-python-r-and-octave-part-6/fig6-6/" data-large-file="https://gigadom.files.wordpress.com/2018/04/fig6.png?w=456?w=582" alt="" data-image-title="fig6" sizes="(max-width: 582px) 100vw, 582px" data-comments-opened="1" class="wp-image-6513" data-recalc-dims="1"></p>
</div>
<div>
<h2>1.2a Default initialization – R</h2>
<pre><code><span>source</span>(<span>&quot;DLfunctions61.R&quot;</span>)
#Load the data
z &lt;- as.matrix(read.csv(<span>&quot;circles.csv&quot;</span>,header=<span>FALSE</span>)) 
x &lt;- z[,<span>1</span>:<span>2</span>]
y &lt;- z[,<span>3</span>]
X &lt;- t(x)
Y &lt;- t(y)
#Set the layer dimensions
layersDimensions = c(<span>2</span>,<span>11</span>,<span>1</span>)
# Train a deep learning network
retvals = L_Layer_DeepModel(X, Y, layersDimensions,
                            hiddenActivationFunc=<span>'relu'</span>,
                            outputActivationFunc=<span>&quot;sigmoid&quot;</span>,
                            learningRate = <span>0.5</span>,
                            numIterations = <span>9000</span>, 
                            initType=<span>&quot;default&quot;</span>,
                            print_cost = True)</code></pre>
<pre><code><span>#Plot the cost vs iterations</span>
iterations &lt;- seq(<span>0</span>,<span>9000</span>,<span>1000</span>)
costs=retvals$costs
df=data.frame(iterations,costs)
ggplot(df,aes(x=iterations,y=costs)) + geom_point() + geom_line(color=<span>&quot;blue&quot;</span>) +
 ggtitle(<span>&quot;Costs vs iterations&quot;</span>) + xlab(<span>&quot;No of iterations&quot;</span>) + ylab(<span>&quot;Cost&quot;</span>)</code></pre>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/fig1-1.png" data-orig-size="1344,960" data-medium-file="https://gigadom.files.wordpress.com/2018/04/fig1-1.png?w=300" data-attachment-id="6520" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/fig1-1.png?w=450&h=483&fit=676%2C483" data-permalink="https://gigadom.wordpress.com/2018/04/16/deep-learning-from-first-principles-in-python-r-and-octave-part-6/fig1-1-9/" data-large-file="https://gigadom.files.wordpress.com/2018/04/fig1-1.png?w=450&amp;h=483&amp;fit=676%2C483" alt="" data-image-title="fig1-1" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6520" data-recalc-dims="1"></p>
<pre><code><span># Plot the decision boundary</span>
plotDecisionBoundary(z,retvals,hiddenActivationFunc=<span>&quot;relu&quot;</span>,lr=<span>0.5</span>)</code></pre>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/fig1-2.png" data-orig-size="1344,960" data-medium-file="https://gigadom.files.wordpress.com/2018/04/fig1-2.png?w=300" data-attachment-id="6521" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/fig1-2.png?w=450&h=483&fit=676%2C483" data-permalink="https://gigadom.wordpress.com/2018/04/16/deep-learning-from-first-principles-in-python-r-and-octave-part-6/fig1-2-6/" data-large-file="https://gigadom.files.wordpress.com/2018/04/fig1-2.png?w=450&amp;h=483&amp;fit=676%2C483" alt="" data-image-title="fig1-2" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6521" data-recalc-dims="1"></p>
</div>
<div>
<h2>1.2b He initialization – R</h2>
<pre><code><span>source</span>(<span>&quot;DLfunctions61.R&quot;</span>)
# Load the data
z &lt;- as.matrix(read.csv(<span>&quot;circles.csv&quot;</span>,header=<span>FALSE</span>)) 
x &lt;- z[,<span>1</span>:<span>2</span>]
y &lt;- z[,<span>3</span>]
X &lt;- t(x)
Y &lt;- t(y)
# Set the layer dimensions
layersDimensions = c(<span>2</span>,<span>11</span>,<span>1</span>)
# Train a deep learning network
retvals = L_Layer_DeepModel(X, Y, layersDimensions,
                            hiddenActivationFunc=<span>'relu'</span>,
                            outputActivationFunc=<span>&quot;sigmoid&quot;</span>,
                            learningRate = <span>0.5</span>,
                            numIterations = <span>9000</span>, 
                            initType=<span>&quot;He&quot;</span>,
                            print_cost = True)</code></pre>
<pre><code><span>#Plot the cost vs iterations</span>
iterations &lt;- seq(<span>0</span>,<span>9000</span>,<span>1000</span>)
costs=retvals$costs
df=data.frame(iterations,costs)
ggplot(df,aes(x=iterations,y=costs)) + geom_point() + geom_line(color=<span>&quot;blue&quot;</span>) +
    ggtitle(<span>&quot;Costs vs iterations&quot;</span>) + xlab(<span>&quot;No of iterations&quot;</span>) + ylab(<span>&quot;Cost&quot;</span>)</code></pre>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/fig-2-11.png" data-orig-size="1344,960" data-medium-file="https://gigadom.files.wordpress.com/2018/04/fig-2-11.png?w=300" data-attachment-id="6518" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/fig-2-11.png?w=450&h=483&fit=676%2C483" data-permalink="https://gigadom.wordpress.com/2018/04/16/deep-learning-from-first-principles-in-python-r-and-octave-part-6/fig-2-1-2/" data-large-file="https://gigadom.files.wordpress.com/2018/04/fig-2-11.png?w=450&amp;h=483&amp;fit=676%2C483" alt="" data-image-title="fig 2-1" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6518" data-recalc-dims="1"></p>
<pre><code><span># Plot the decision boundary</span>
plotDecisionBoundary(z,retvals,hiddenActivationFunc=<span>&quot;relu&quot;</span>,<span>0.5</span>,lr=<span>0.5</span>)</code></pre>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/fig-2-21.png" data-orig-size="1344,960" data-medium-file="https://gigadom.files.wordpress.com/2018/04/fig-2-21.png?w=300" data-attachment-id="6519" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/fig-2-21.png?w=450&h=483&fit=676%2C483" data-permalink="https://gigadom.wordpress.com/2018/04/16/deep-learning-from-first-principles-in-python-r-and-octave-part-6/fig-2-2-2/" data-large-file="https://gigadom.files.wordpress.com/2018/04/fig-2-21.png?w=450&amp;h=483&amp;fit=676%2C483" alt="" data-image-title="fig 2-2" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6519" data-recalc-dims="1"></p>
</div>
<div>
<h2>1.2c Xavier initialization – R</h2>
<pre><code><span>## Xav initialization </span># Set the layer dimensions
layersDimensions = c(<span>2</span>,<span>11</span>,<span>1</span>)
# Train a deep learning network
retvals = L_Layer_DeepModel(X, Y, layersDimensions,
                            hiddenActivationFunc=<span>'relu'</span>,
                            outputActivationFunc=<span>&quot;sigmoid&quot;</span>,
                            learningRate = <span>0.5</span>,
                            numIterations = <span>9000</span>, 
                            initType=<span>&quot;Xav&quot;</span>,
                            print_cost = True)</code></pre>
<pre><code><span>#Plot the cost vs iterations</span>
iterations &lt;- seq(<span>0</span>,<span>9000</span>,<span>1000</span>)
costs=retvals$costs
df=data.frame(iterations,costs)
ggplot(df,aes(x=iterations,y=costs)) + geom_point() + geom_line(color=<span>&quot;blue&quot;</span>) +
    ggtitle(<span>&quot;Costs vs iterations&quot;</span>) + xlab(<span>&quot;No of iterations&quot;</span>) + ylab(<span>&quot;Cost&quot;</span>)</code></pre>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/fig3-1.png" data-orig-size="1344,960" data-medium-file="https://gigadom.files.wordpress.com/2018/04/fig3-1.png?w=300" data-attachment-id="6522" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/fig3-1.png?w=450&h=483&fit=676%2C483" data-permalink="https://gigadom.wordpress.com/2018/04/16/deep-learning-from-first-principles-in-python-r-and-octave-part-6/fig3-1-7/" data-large-file="https://gigadom.files.wordpress.com/2018/04/fig3-1.png?w=450&amp;h=483&amp;fit=676%2C483" alt="" data-image-title="fig3-1" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6522" data-recalc-dims="1"></p>
<pre><code><span># Plot the decision boundary</span>
plotDecisionBoundary(z,retvals,hiddenActivationFunc=<span>&quot;relu&quot;</span>,<span>0.5</span>)</code></pre>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/fig3-2.png" data-orig-size="1344,960" data-medium-file="https://gigadom.files.wordpress.com/2018/04/fig3-2.png?w=300" data-attachment-id="6523" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/fig3-2.png?w=450&h=483&fit=676%2C483" data-permalink="https://gigadom.wordpress.com/2018/04/16/deep-learning-from-first-principles-in-python-r-and-octave-part-6/fig3-2-3/" data-large-file="https://gigadom.files.wordpress.com/2018/04/fig3-2.png?w=450&amp;h=483&amp;fit=676%2C483" alt="" data-image-title="fig3-2" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6523" data-recalc-dims="1"></p>
</div>
<div>
<h2>1.3a Default initialization – Octave</h2>
<pre>source(&quot;DL61functions.m&quot;)
# Read the data
data=csvread(&quot;circles.csv&quot;);

X=data(:,1:2);
Y=data(:,3);
# Set the layer dimensions
layersDimensions = [2 11  1]; #tanh=-0.5(ok), #relu=0.1 best!

# Train a deep learning network
[weights biases costs]=L_Layer_DeepModel(X', Y', layersDimensions,
                               hiddenActivationFunc='relu', 
                               outputActivationFunc=&quot;sigmoid&quot;,
                               learningRate = 0.5,
                               lambd=0, 
                               keep_prob=1,
                               numIterations = 10000,
                               initType=&quot;default&quot;);
# Plot cost vs iterations
plotCostVsIterations(10000,costs)  
#Plot decision boundary                            
plotDecisionBoundary(data,weights, biases,keep_prob=1, hiddenActivationFunc=&quot;relu&quot;)
</pre>
</div>
<h2> <img data-orig-file="https://gigadom.files.wordpress.com/2018/04/fig1-o.png" data-orig-size="1200,900" data-medium-file="https://gigadom.files.wordpress.com/2018/04/fig1-o.png?w=300" data-attachment-id="6545" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/fig1-o.png?w=450&h=507&fit=676%2C507" data-permalink="https://gigadom.wordpress.com/2018/04/16/deep-learning-from-first-principles-in-python-r-and-octave-part-6/fig1-o-2/" data-large-file="https://gigadom.files.wordpress.com/2018/04/fig1-o.png?w=450&amp;h=507&amp;fit=676%2C507" alt="" data-image-title="fig1-o" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6545" data-recalc-dims="1"><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/fig2-o.png" data-orig-size="1200,900" data-medium-file="https://gigadom.files.wordpress.com/2018/04/fig2-o.png?w=300" data-attachment-id="6546" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/fig2-o.png?w=450&h=507&fit=676%2C507" data-permalink="https://gigadom.wordpress.com/2018/04/16/deep-learning-from-first-principles-in-python-r-and-octave-part-6/fig2-o-2/" data-large-file="https://gigadom.files.wordpress.com/2018/04/fig2-o.png?w=450&amp;h=507&amp;fit=676%2C507" alt="" data-image-title="fig2-o" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6546" data-recalc-dims="1"></h2>
<h2>1.3b He initialization – Octave</h2>
<pre>source(&quot;DL61functions.m&quot;)
#Load data
data=csvread(&quot;circles.csv&quot;);
X=data(:,1:2);
Y=data(:,3);
# Set the layer dimensions
layersDimensions = [2 11  1]; #tanh=-0.5(ok), #relu=0.1 best!

# Train a deep learning network
[weights biases costs]=L_Layer_DeepModel(X', Y', layersDimensions,
                               hiddenActivationFunc='relu', 
                               outputActivationFunc=&quot;sigmoid&quot;,
                               learningRate = 0.5,
                               lambd=0, 
                               keep_prob=1,
                               numIterations = 8000,
                               initType=&quot;He&quot;);
plotCostVsIterations(8000,costs)   
#Plot decision boundary                              
plotDecisionBoundary(data,weights, biases,keep_prob=1,hiddenActivationFunc=&quot;relu&quot;)
</pre>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/fighe-o.png" data-orig-size="1200,900" data-medium-file="https://gigadom.files.wordpress.com/2018/04/fighe-o.png?w=300" data-attachment-id="6548" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/fighe-o.png?w=450&h=507&fit=676%2C507" data-permalink="https://gigadom.wordpress.com/2018/04/16/deep-learning-from-first-principles-in-python-r-and-octave-part-6/fighe-o/" data-large-file="https://gigadom.files.wordpress.com/2018/04/fighe-o.png?w=450&amp;h=507&amp;fit=676%2C507" alt="" data-image-title="figHe-o" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6548" data-recalc-dims="1"></p>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/fighe1-o.png" data-orig-size="1200,900" data-medium-file="https://gigadom.files.wordpress.com/2018/04/fighe1-o.png?w=300" data-attachment-id="6549" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/fighe1-o.png?w=450&h=507&fit=676%2C507" data-permalink="https://gigadom.wordpress.com/2018/04/16/deep-learning-from-first-principles-in-python-r-and-octave-part-6/fighe1-o/" data-large-file="https://gigadom.files.wordpress.com/2018/04/fighe1-o.png?w=450&amp;h=507&amp;fit=676%2C507" alt="" data-image-title="figHe1-o" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6549" data-recalc-dims="1"></p>
<h2>1.3c Xavier initialization – Octave</h2>
<pre>source(&quot;DL61functions.m&quot;)
#Load data
data=csvread(&quot;circles.csv&quot;);
X=data(:,1:2);
Y=data(:,3);
#Set layer dimensions
layersDimensions = [2 11  1]; #tanh=-0.5(ok), #relu=0.1 best!

# Train a deep learning network
[weights biases costs]=L_Layer_DeepModel(X', Y', layersDimensions,
                               hiddenActivationFunc='relu', 
                               outputActivationFunc=&quot;sigmoid&quot;,
                               learningRate = 0.5,
                               lambd=0, 
                               keep_prob=1,
                               numIterations = 8000,
                               initType=&quot;Xav&quot;);

plotCostVsIterations(8000,costs)                               
plotDecisionBoundary(data,weights, biases,keep_prob=1,hiddenActivationFunc=&quot;relu&quot;)
</pre>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/figxav-o.png" data-orig-size="1200,900" data-medium-file="https://gigadom.files.wordpress.com/2018/04/figxav-o.png?w=300" data-attachment-id="6550" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/figxav-o.png?w=450&h=507&fit=676%2C507" data-permalink="https://gigadom.wordpress.com/2018/04/16/deep-learning-from-first-principles-in-python-r-and-octave-part-6/figxav-o/" data-large-file="https://gigadom.files.wordpress.com/2018/04/figxav-o.png?w=450&amp;h=507&amp;fit=676%2C507" alt="" data-image-title="figXav-o" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6550" data-recalc-dims="1"></p>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/figxav1-o.png" data-orig-size="1200,900" data-medium-file="https://gigadom.files.wordpress.com/2018/04/figxav1-o.png?w=300" data-attachment-id="6551" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/figxav1-o.png?w=450&h=507&fit=676%2C507" data-permalink="https://gigadom.wordpress.com/2018/04/16/deep-learning-from-first-principles-in-python-r-and-octave-part-6/figxav1-o/" data-large-file="https://gigadom.files.wordpress.com/2018/04/figxav1-o.png?w=450&amp;h=507&amp;fit=676%2C507" alt="" data-image-title="figXav1-o" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6551" data-recalc-dims="1"></p>
<div>
<h2>2.1a Regularization : Circles data – Python</h2>
<p>The cross entropy cost for Logistic classification is given as<br>
<img alt="J = frac{1}{m}sum_{i=1}^{m}y^{i}log((a^{L})^{(i)}) - (1-y^{i})log((a^{L})^{(i)})" title="J = frac{1}{m}sum_{i=1}^{m}y^{i}log((a^{L})^{(i)}) - (1-y^{i})log((a^{L})^{(i)})" src="https://s0.wp.com/latex.php?latex=J+%3D+%5Cfrac%7B1%7D%7Bm%7D%5Csum_%7Bi%3D1%7D%5E%7Bm%7Dy%5E%7Bi%7Dlog%28%28a%5E%7BL%7D%29%5E%7B%28i%29%7D%29+-+%281-y%5E%7Bi%7D%29log%28%28a%5E%7BL%7D%29%5E%7B%28i%29%7D%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"></p>
<p>The regularized L2 cost is given by<br>
<img alt="J = frac{1}{m}sum_{i=1}^{m}y^{i}log((a^{L})^{(i)}) - (1-y^{i})log((a^{L})^{(i)}) + frac{lambda}{2m}sum sum sum W_{kj}^{l}" title="J = frac{1}{m}sum_{i=1}^{m}y^{i}log((a^{L})^{(i)}) - (1-y^{i})log((a^{L})^{(i)}) + frac{lambda}{2m}sum sum sum W_{kj}^{l}" src="https://s0.wp.com/latex.php?latex=J+%3D+%5Cfrac%7B1%7D%7Bm%7D%5Csum_%7Bi%3D1%7D%5E%7Bm%7Dy%5E%7Bi%7Dlog%28%28a%5E%7BL%7D%29%5E%7B%28i%29%7D%29+-+%281-y%5E%7Bi%7D%29log%28%28a%5E%7BL%7D%29%5E%7B%28i%29%7D%29+%2B+%5Cfrac%7B%5Clambda%7D%7B2m%7D%5Csum+%5Csum+%5Csum+W_%7Bkj%7D%5E%7Bl%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"></p>
</div>
<p>Incidentally, a detailed implementation of L1 and L2 regularization in Machine Learning, besides other feature selection methods can be found in my book ‘<a rel="nofollow" href="https://gigadom.wordpress.com/2017/12/05/my-book-practical-machine-learning-with-r-and-python-on-amazon/" target="_blank">Practical Machine Learning with R and Python: Machine Learning in stereo</a>‘ available in Amazon. My book is compact and can be used as a quick reference for ML algorithms and associated metrics</p>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/untitled1.png?w=450&amp;h=214&amp;fit=676%2C214" data-orig-size="676,214" data-medium-file="https://gigadom.files.wordpress.com/2018/04/untitled1.png?w=450&amp;h=214&amp;fit=676%2C214?w=300" data-attachment-id="6564" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/untitled1.png?w=450&h=214&fit=676%2C214" data-permalink="https://gigadom.wordpress.com/2018/04/16/deep-learning-from-first-principles-in-python-r-and-octave-part-6/untitled1-9/" data-large-file="https://gigadom.files.wordpress.com/2018/04/untitled1.png?w=450&amp;h=214&amp;fit=676%2C214?w=676" alt="" data-image-title="untitled1" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6564" data-recalc-dims="1"></p>
<div>
<pre><code><span>import</span> numpy <span>as</span> np
<span>import</span> matplotlib
<span>import</span> matplotlib.pyplot <span>as</span> plt
<span>import</span> sklearn.linear_model
<span>import</span> pandas <span>as</span> pd
<span>import</span> sklearn
<span>import</span> sklearn.datasets
exec(open(<span>&quot;DLfunctions61.py&quot;</span>).read())

<span>#Load the data</span>
train_X, train_Y, test_X, test_Y = load_dataset()
<span># Set the layers dimensions</span>
layersDimensions = [<span>2</span>,<span>7</span>,<span>1</span>]

<span># Train a deep learning network</span>
parameters = L_Layer_DeepModel(train_X, train_Y, layersDimensions, hiddenActivationFunc=<span>'relu'</span>,  
                               outputActivationFunc=<span>&quot;sigmoid&quot;</span>,learningRate = <span>0.6</span>, lambd=<span>0.1</span>, num_iterations = <span>9000</span>, 
                               initType=<span>&quot;default&quot;</span>, print_cost = <span>True</span>,figure=<span>&quot;fig7.png&quot;</span>)

<span># Clear the plot</span>
plt.clf()
plt.close()

<span># Plot the decision boundary</span>
plot_decision_boundary(<span>lambda</span> x: predict(parameters, x.T), train_X, train_Y,str(<span>0.6</span>),figure1=<span>&quot;fig8.png&quot;</span>)


plt.clf()
plt.close()
#Plot the decision boundary
plot_decision_boundary(<span>lambda</span> x: predict(parameters, x.T,keep_prob=<span>0.9</span>), train_X, train_Y,str(<span>2.2</span>),<span>&quot;fig8.png&quot;</span>,)</code></pre>
</div>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/fig71.png?w=456" data-orig-size="389,278" data-medium-file="https://gigadom.files.wordpress.com/2018/04/fig71.png?w=456?w=300" data-attachment-id="6527" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/fig71.png?w=456" data-permalink="https://gigadom.wordpress.com/2018/04/16/deep-learning-from-first-principles-in-python-r-and-octave-part-6/fig7-11/" data-large-file="https://gigadom.files.wordpress.com/2018/04/fig71.png?w=456?w=389" alt="" data-image-title="fig7" sizes="(max-width: 389px) 100vw, 389px" data-comments-opened="1" class="wp-image-6527" data-recalc-dims="1"></p>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/fig81.png?w=456" data-orig-size="398,278" data-medium-file="https://gigadom.files.wordpress.com/2018/04/fig81.png?w=456?w=300" data-attachment-id="6528" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/fig81.png?w=456" data-permalink="https://gigadom.wordpress.com/2018/04/16/deep-learning-from-first-principles-in-python-r-and-octave-part-6/fig8-8/" data-large-file="https://gigadom.files.wordpress.com/2018/04/fig81.png?w=456?w=398" alt="" data-image-title="fig8" sizes="(max-width: 398px) 100vw, 398px" data-comments-opened="1" class="wp-image-6528" data-recalc-dims="1"></p>
<div>
<h2>2.1 b Regularization: Spiral data  – Python</h2>
<pre><code><span>import</span> numpy <span>as</span> np
<span>import</span> matplotlib
<span>import</span> matplotlib.pyplot <span>as</span> plt
<span>import</span> sklearn.linear_model
<span>import</span> pandas <span>as</span> pd
<span>import</span> sklearn
<span>import</span> sklearn.datasets
exec(open(<span>&quot;DLfunctions61.py&quot;</span>).read())
N = <span>100</span> <span># number of points per class</span>
D = <span>2</span> <span># dimensionality</span>
K = <span>3</span> <span># number of classes</span>
X = np.zeros((N*K,D)) <span># data matrix (each row = single example)</span>
y = np.zeros(N*K, dtype=<span>'uint8'</span>) <span># class labels</span>
<span>for</span> j <span>in</span> range(K):
  ix = range(N*j,N*(j+<span>1</span>))
  r = np.linspace(<span>0.0</span>,<span>1</span>,N) <span># radius</span>
  t = np.linspace(j*<span>4</span>,(j+<span>1</span>)*<span>4</span>,N) + np.random.randn(N)*<span>0.2</span> <span># theta</span>
  X[ix] = np.c_[r*np.sin(t), r*np.cos(t)]
  y[ix] = j


<span># Plot the data</span>
plt.scatter(X[:, <span>0</span>], X[:, <span>1</span>], c=y, s=<span>40</span>, cmap=plt.cm.Spectral)
plt.clf()
plt.close() 
#Set layer dimensions 
layersDimensions = [<span>2</span>,<span>100</span>,<span>3</span>]
y1=y.reshape(<span>-1</span>,<span>1</span>).T
# Train a deep learning network
parameters = L_Layer_DeepModel(X.T, y1, layersDimensions, hiddenActivationFunc=<span>'relu'</span>, outputActivationFunc=<span>&quot;softmax&quot;</span>,
                           learningRate = <span>0.6</span>,lambd=<span>0.2</span>, num_iterations = <span>5000</span>, print_cost = <span>True</span>,figure=<span>&quot;fig9.png&quot;</span>)

plt.clf()
plt.close()  
W1=parameters[<span>'W1'</span>]
b1=parameters[<span>'b1'</span>]
W2=parameters[<span>'W2'</span>]
b2=parameters[<span>'b2'</span>]
plot_decision_boundary1(X, y1,W1,b1,W2,b2,figure2=<span>&quot;fig10.png&quot;</span>)</code></pre>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/fig9.png?w=456" data-orig-size="570,453" data-medium-file="https://gigadom.files.wordpress.com/2018/04/fig9.png?w=456?w=300" data-attachment-id="6529" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/fig9.png?w=456" data-permalink="https://gigadom.wordpress.com/2018/04/16/deep-learning-from-first-principles-in-python-r-and-octave-part-6/fig9/" data-large-file="https://gigadom.files.wordpress.com/2018/04/fig9.png?w=456?w=570" alt="" data-image-title="fig9" sizes="(max-width: 570px) 100vw, 570px" data-comments-opened="1" class="wp-image-6529" data-recalc-dims="1"></p>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/fig10.png?w=456" data-orig-size="562,416" data-medium-file="https://gigadom.files.wordpress.com/2018/04/fig10.png?w=456?w=300" data-attachment-id="6530" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/fig10.png?w=456" data-permalink="https://gigadom.wordpress.com/2018/04/16/deep-learning-from-first-principles-in-python-r-and-octave-part-6/fig10/" data-large-file="https://gigadom.files.wordpress.com/2018/04/fig10.png?w=456?w=562" alt="" data-image-title="fig10" sizes="(max-width: 562px) 100vw, 562px" data-comments-opened="1" class="wp-image-6530" data-recalc-dims="1"></p>
<h2>2.2a Regularization: Circles data  – R</h2>
<pre><code><span>source</span>(<span>&quot;DLfunctions61.R&quot;</span>)
#Load data
df=read.csv(<span>&quot;circles.csv&quot;</span>,header=<span>FALSE</span>)
z &lt;- as.matrix(read.csv(<span>&quot;circles.csv&quot;</span>,header=<span>FALSE</span>)) 
x &lt;- z[,<span>1</span>:<span>2</span>]
y &lt;- z[,<span>3</span>]
X &lt;- t(x)
Y &lt;- t(y)
#Set layer dimensions
layersDimensions = c(<span>2</span>,<span>11</span>,<span>1</span>)
# Train a deep learning network
retvals = L_Layer_DeepModel(X, Y, layersDimensions,
                            hiddenActivationFunc=<span>'relu'</span>,
                            outputActivationFunc=<span>&quot;sigmoid&quot;</span>,
                            learningRate = <span>0.5</span>,
                            lambd=<span>0.1</span>,
                            numIterations = <span>9000</span>, 
                            initType=<span>&quot;default&quot;</span>,
                            print_cost = True)</code><code>
</code></pre>
<pre><code><span>#Plot the cost vs iterations</span>
iterations &lt;- seq(<span>0</span>,<span>9000</span>,<span>1000</span>)
costs=retvals$costs
df=data.frame(iterations,costs)
ggplot(df,aes(x=iterations,y=costs)) + geom_point() + geom_line(color=<span>&quot;blue&quot;</span>) +
    ggtitle(<span>&quot;Costs vs iterations&quot;</span>) + xlab(<span>&quot;No of iterations&quot;</span>) + ylab(<span>&quot;Cost&quot;</span>)</code></pre>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/fig4-1.png" data-orig-size="1344,960" data-medium-file="https://gigadom.files.wordpress.com/2018/04/fig4-1.png?w=300" data-attachment-id="6532" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/fig4-1.png?w=450&h=483&fit=676%2C483" data-permalink="https://gigadom.wordpress.com/2018/04/16/deep-learning-from-first-principles-in-python-r-and-octave-part-6/fig4-1-8/" data-large-file="https://gigadom.files.wordpress.com/2018/04/fig4-1.png?w=450&amp;h=483&amp;fit=676%2C483" alt="" data-image-title="fig4-1" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6532" data-recalc-dims="1"></p>
<pre><code><span># Plot the decision boundary</span>
plotDecisionBoundary(z,retvals,hiddenActivationFunc=<span>&quot;relu&quot;</span>,<span>0.5</span>)</code></pre>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/fig4-2.png" data-orig-size="1344,960" data-medium-file="https://gigadom.files.wordpress.com/2018/04/fig4-2.png?w=300" data-attachment-id="6533" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/fig4-2.png?w=450&h=483&fit=676%2C483" data-permalink="https://gigadom.wordpress.com/2018/04/16/deep-learning-from-first-principles-in-python-r-and-octave-part-6/fig4-2-3/" data-large-file="https://gigadom.files.wordpress.com/2018/04/fig4-2.png?w=450&amp;h=483&amp;fit=676%2C483" alt="" data-image-title="fig4-2" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6533" data-recalc-dims="1"></p>
</div>
<div>
<h2>2.2b Regularization:Spiral data – R</h2>
<pre><code><span># Read the spiral dataset
</span>#Load the data
<span>source</span>(<span>&quot;DLfunctions61.R&quot;</span>)
Z &lt;- as.matrix(read.csv(<span>&quot;spiral.csv&quot;</span>,header=<span>FALSE</span>)) 

<span># Setup the data</span>
X &lt;- Z[,<span>1</span>:<span>2</span>]
y &lt;- Z[,<span>3</span>]
X &lt;- t(X)
Y &lt;- t(y)

layersDimensions = c(<span>2</span>, <span>15</span>, <span>6</span>, <span>3</span>) 
# Train a deep learning network
retvals = L_Layer_DeepModel(X, Y, layersDimensions,
                            hiddenActivationFunc=<span>'relu'</span>,
                            outputActivationFunc=<span>&quot;softmax&quot;</span>,
                            learningRate = <span>5.1</span>,
                            lambd=<span>0.01</span>,
                            numIterations = <span>9000</span>, 
                            print_cost = True)</code></pre>
<pre><code>parameters&lt;-retvals$parameters
plotDecisionBoundary1(Z,parameters)</code></pre>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/r-reg.png" data-orig-size="1200,586" data-medium-file="https://gigadom.files.wordpress.com/2018/04/r-reg.png?w=300" data-attachment-id="6584" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/r-reg.png?w=450&h=330&fit=676%2C330" data-permalink="https://gigadom.wordpress.com/2018/04/16/deep-learning-from-first-principles-in-python-r-and-octave-part-6/r-reg/" data-large-file="https://gigadom.files.wordpress.com/2018/04/r-reg.png?w=450&amp;h=330&amp;fit=676%2C330" alt="" data-image-title="r-reg" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6584" data-recalc-dims="1"><br>
<span>2.3a Regularization: Circles data – Octave</span></p>
</div>
<pre>source(&quot;DL61functions.m&quot;)
#Load data
data=csvread(&quot;circles.csv&quot;);
X=data(:,1:2);
Y=data(:,3);
layersDimensions = [2 11  1]; #tanh=-0.5(ok), #relu=0.1 best!

# Train a deep learning network
[weights biases costs]=L_Layer_DeepModel(X', Y', layersDimensions,
                               hiddenActivationFunc='relu', 
                               outputActivationFunc=&quot;sigmoid&quot;,
                               learningRate = 0.5,
                               lambd=0.2,
                               keep_prob=1,
                               numIterations = 8000,
                               initType=&quot;default&quot;);

plotCostVsIterations(8000,costs)  
#Plot decision boundary                              
plotDecisionBoundary(data,weights, biases,keep_prob=1,hiddenActivationFunc=&quot;relu&quot;)
</pre>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/figreg-o.png" data-orig-size="1200,900" data-medium-file="https://gigadom.files.wordpress.com/2018/04/figreg-o.png?w=300" data-attachment-id="6553" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/figreg-o.png?w=450&h=507&fit=676%2C507" data-permalink="https://gigadom.wordpress.com/2018/04/16/deep-learning-from-first-principles-in-python-r-and-octave-part-6/figreg-o/" data-large-file="https://gigadom.files.wordpress.com/2018/04/figreg-o.png?w=450&amp;h=507&amp;fit=676%2C507" alt="" data-image-title="figReg-o" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6553" data-recalc-dims="1"></p>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/figreg1-o.png" data-orig-size="1200,900" data-medium-file="https://gigadom.files.wordpress.com/2018/04/figreg1-o.png?w=300" data-attachment-id="6554" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/figreg1-o.png?w=450&h=507&fit=676%2C507" data-permalink="https://gigadom.wordpress.com/2018/04/16/deep-learning-from-first-principles-in-python-r-and-octave-part-6/figreg1-o/" data-large-file="https://gigadom.files.wordpress.com/2018/04/figreg1-o.png?w=450&amp;h=507&amp;fit=676%2C507" alt="" data-image-title="figReg1-o" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6554" data-recalc-dims="1"></p>
<h2>2.3b Regularization:Spiral data  2 – Octave</h2>
<pre>source(&quot;DL61functions.m&quot;)
data=csvread(&quot;spiral.csv&quot;);

# Setup the data
X=data(:,1:2);
Y=data(:,3);
layersDimensions = [2 100 3]
# Train a deep learning network
[weights biases costs]=L_Layer_DeepModel(X', Y', layersDimensions,
                               hiddenActivationFunc='relu', 
                               outputActivationFunc=&quot;softmax&quot;,
                               learningRate = 0.6,
                               lambd=0.2,
                               keep_prob=1,
                               numIterations = 10000);
                              
plotCostVsIterations(10000,costs)
#Plot decision boundary
plotDecisionBoundary1(data,weights, biases,keep_prob=1,hiddenActivationFunc=&quot;relu&quot;)  
</pre>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/figreg2-o.png" data-orig-size="1200,900" data-medium-file="https://gigadom.files.wordpress.com/2018/04/figreg2-o.png?w=300" data-attachment-id="6555" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/figreg2-o.png?w=450&h=507&fit=676%2C507" data-permalink="https://gigadom.wordpress.com/2018/04/16/deep-learning-from-first-principles-in-python-r-and-octave-part-6/figreg2-o/" data-large-file="https://gigadom.files.wordpress.com/2018/04/figreg2-o.png?w=450&amp;h=507&amp;fit=676%2C507" alt="" data-image-title="figReg2-o" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6555" data-recalc-dims="1"></p>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/reg-oct.jpg" data-orig-size="879,659" data-medium-file="https://gigadom.files.wordpress.com/2018/04/reg-oct.jpg?w=300" data-attachment-id="6588" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/reg-oct.jpg?w=450&h=507&fit=676%2C507" data-permalink="https://gigadom.wordpress.com/2018/04/16/deep-learning-from-first-principles-in-python-r-and-octave-part-6/reg-oct/" data-large-file="https://gigadom.files.wordpress.com/2018/04/reg-oct.jpg?w=450&amp;h=507&amp;fit=676%2C507" alt="" data-image-title="reg-oct" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6588" data-recalc-dims="1"></p>
<div>
<h2>3.1 a Dropout: Circles data – Python</h2>
<p>The ‘dropout’ regularization technique was used with great effectiveness, to prevent overfitting  by Alex Krizhevsky, Ilya Sutskever and Prof Geoffrey E. Hinton in the <a rel="nofollow" href="https://www.nvidia.cn/content/tesla/pdf/machine-learning/imagenet-classification-with-deep-convolutional-nn.pdf" target="_blank">Imagenet classification with Deep Convolutional Neural Networks</a></p>
<p>The technique of dropout works by dropping a random set of activation units in each hidden layer, based on a ‘keep_prob’ criteria in the forward propagation cycle. Here is the code for Octave. A ‘dropoutMat’ is created for each layer which specifies which units to drop <strong>Note</strong>: The same ‘dropoutMat has to be used which computing the gradients in the backward propagation cycle. Hence the dropout matrices are stored in a cell array.</p>
<pre> for l =1:L-1  
    ...      
    D=rand(size(A)(1),size(A)(2));
    D = (D &lt; keep_prob) ;
    # Zero out some hidden units
    A= A .* D;    
    # Divide by keep_prob to keep the expected value of A the same                                  
    A = A ./ keep_prob; 
    # Store D in a dropoutMat cell array
    dropoutMat{l}=D;
    ...
 endfor</pre>
<p>In the backward propagation cycle we have</p>
<pre>    for l =(L-1):-1:1
          ...
          D = dropoutMat{l};  
          # Zero out the dAl based on same dropout matrix       
          dAl= dAl .* D;   
          # Divide by keep_prob to maintain the expected value                                       
          dAl = dAl ./ keep_prob;
          ...
    endfor 
</pre>
<pre><code><span>import</span> numpy <span>as</span> np
<span>import</span> matplotlib
<span>import</span> matplotlib.pyplot <span>as</span> plt
<span>import</span> sklearn.linear_model
<span>import</span> pandas <span>as</span> pd
<span>import</span> sklearn
<span>import</span> sklearn.datasets
exec(open(<span>&quot;DLfunctions61.py&quot;</span>).read())
<span>#Load the data</span>
train_X, train_Y, test_X, test_Y = load_dataset()
<span># Set the layers dimensions</span>
layersDimensions = [<span>2</span>,<span>7</span>,<span>1</span>]

<span># Train a deep learning network</span>
parameters = L_Layer_DeepModel(train_X, train_Y, layersDimensions, hiddenActivationFunc=<span>'relu'</span>,  
                               outputActivationFunc=<span>&quot;sigmoid&quot;</span>,learningRate = <span>0.6</span>, keep_prob=<span>0.7</span>, num_iterations = <span>9000</span>, 
                               initType=<span>&quot;default&quot;</span>, print_cost = <span>True</span>,figure=<span>&quot;fig11.png&quot;</span>)

<span># Clear the plot</span>
plt.clf()
plt.close()

<span># Plot the decision boundary</span>
plot_decision_boundary(<span>lambda</span> x: predict(parameters, x.T,keep_prob=<span>0.7</span>), train_X, train_Y,str(<span>0.6</span>),figure1=<span>&quot;fig12.png&quot;</span>)
</code></pre>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/fig11.png?w=456" data-orig-size="579,453" data-medium-file="https://gigadom.files.wordpress.com/2018/04/fig11.png?w=456?w=300" data-attachment-id="6537" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/fig11.png?w=456" data-permalink="https://gigadom.wordpress.com/2018/04/16/deep-learning-from-first-principles-in-python-r-and-octave-part-6/fig11-2/" data-large-file="https://gigadom.files.wordpress.com/2018/04/fig11.png?w=456?w=579" alt="" data-image-title="fig11" sizes="(max-width: 579px) 100vw, 579px" data-comments-opened="1" class="wp-image-6537" data-recalc-dims="1"></p>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/fig12.png?w=456" data-orig-size="582,453" data-medium-file="https://gigadom.files.wordpress.com/2018/04/fig12.png?w=456?w=300" data-attachment-id="6538" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/fig12.png?w=456" data-permalink="https://gigadom.wordpress.com/2018/04/16/deep-learning-from-first-principles-in-python-r-and-octave-part-6/fig12-2/" data-large-file="https://gigadom.files.wordpress.com/2018/04/fig12.png?w=456?w=582" alt="" data-image-title="fig12" sizes="(max-width: 582px) 100vw, 582px" data-comments-opened="1" class="wp-image-6538" data-recalc-dims="1"></p>
<div>
<h3>3.1b Dropout: Spiral data – Python</h3>
<pre><code><span>import</span> numpy <span>as</span> np
<span>import</span> matplotlib
<span>import</span> matplotlib.pyplot <span>as</span> plt
<span>import</span> sklearn.linear_model
<span>import</span> pandas <span>as</span> pd
<span>import</span> sklearn
<span>import</span> sklearn.datasets
exec(open(<span>&quot;DLfunctions61.py&quot;</span>).read())
<span># Create an input data set - Taken from CS231n Convolutional Neural networks,</span>
<span># http://cs231n.github.io/neural-networks-case-study/</span>
               

N = <span>100</span> <span># number of points per class</span>
D = <span>2</span> <span># dimensionality</span>
K = <span>3</span> <span># number of classes</span>
X = np.zeros((N*K,D)) <span># data matrix (each row = single example)</span>
y = np.zeros(N*K, dtype=<span>'uint8'</span>) <span># class labels</span>
<span>for</span> j <span>in</span> range(K):
  ix = range(N*j,N*(j+<span>1</span>))
  r = np.linspace(<span>0.0</span>,<span>1</span>,N) <span># radius</span>
  t = np.linspace(j*<span>4</span>,(j+<span>1</span>)*<span>4</span>,N) + np.random.randn(N)*<span>0.2</span> <span># theta</span>
  X[ix] = np.c_[r*np.sin(t), r*np.cos(t)]
  y[ix] = j


<span># Plot the data</span>
plt.scatter(X[:, <span>0</span>], X[:, <span>1</span>], c=y, s=<span>40</span>, cmap=plt.cm.Spectral)
plt.clf()
plt.close()  
layersDimensions = [<span>2</span>,<span>100</span>,<span>3</span>]
y1=y.reshape(<span>-1</span>,<span>1</span>).T
# Train a deep learning network
parameters = L_Layer_DeepModel(X.T, y1, layersDimensions, hiddenActivationFunc=<span>'relu'</span>, outputActivationFunc=<span>&quot;softmax&quot;</span>,
                           learningRate = <span>0.6</span>,keep_prob=<span>0.5</span>, num_iterations = <span>5000</span>, print_cost = <span>True</span>,figure=<span>&quot;fig13.png&quot;</span>)

plt.clf()
plt.close()  
W1=parameters[<span>'W1'</span>]
b1=parameters[<span>'b1'</span>]
W2=parameters[<span>'W2'</span>]
b2=parameters[<span>'b2'</span>]
#Plot decision boundary
plot_decision_boundary1(X, y1,W1,b1,W2,b2,figure2=<span>&quot;fig14.png&quot;</span>)</code></pre>
</div>
</div>
<div><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/fig13.png?w=456" data-orig-size="570,453" data-medium-file="https://gigadom.files.wordpress.com/2018/04/fig13.png?w=456?w=300" data-attachment-id="6539" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/fig13.png?w=456" data-permalink="https://gigadom.wordpress.com/2018/04/16/deep-learning-from-first-principles-in-python-r-and-octave-part-6/fig13/" data-large-file="https://gigadom.files.wordpress.com/2018/04/fig13.png?w=456?w=570" alt="" data-image-title="fig13" sizes="(max-width: 570px) 100vw, 570px" data-comments-opened="1" class="wp-image-6539" data-recalc-dims="1"></div>
<div>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/fig101.png?w=456" data-orig-size="562,416" data-medium-file="https://gigadom.files.wordpress.com/2018/04/fig101.png?w=456?w=300" data-attachment-id="6540" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/fig101.png?w=456" data-permalink="https://gigadom.wordpress.com/2018/04/16/deep-learning-from-first-principles-in-python-r-and-octave-part-6/fig10-4/" data-large-file="https://gigadom.files.wordpress.com/2018/04/fig101.png?w=456?w=562" alt="" data-image-title="fig10" sizes="(max-width: 562px) 100vw, 562px" data-comments-opened="1" class="wp-image-6540" data-recalc-dims="1"></p>
<h2>3.2a Dropout: Circles data – R</h2>
<pre><code><span>source</span>(<span>&quot;DLfunctions61.R&quot;</span>)
#Load data
df=read.csv(<span>&quot;circles.csv&quot;</span>,header=<span>FALSE</span>)
z &lt;- as.matrix(read.csv(<span>&quot;circles.csv&quot;</span>,header=<span>FALSE</span>)) 

x &lt;- z[,<span>1</span>:<span>2</span>]
y &lt;- z[,<span>3</span>]
X &lt;- t(x)
Y &lt;- t(y)
layersDimensions = c(<span>2</span>,<span>11</span>,<span>1</span>)
# Train a deep learning network
retvals = L_Layer_DeepModel(X, Y, layersDimensions,
                            hiddenActivationFunc=<span>'relu'</span>,
                            outputActivationFunc=<span>&quot;sigmoid&quot;</span>,
                            learningRate = <span>0.5</span>,
                            keep_prob=<span>0.8</span>,
                            numIterations = <span>9000</span>, 
                            initType=<span>&quot;default&quot;</span>,
                            print_cost = True)</code></pre>
<pre><code><span># Plot the decision boundary</span>
plotDecisionBoundary(z,retvals,keep_prob=<span>0.6</span>, hiddenActivationFunc=<span>&quot;relu&quot;</span>,<span>0.5</span>)</code></pre>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/fig7-1.png" data-orig-size="1344,960" data-medium-file="https://gigadom.files.wordpress.com/2018/04/fig7-1.png?w=300" data-attachment-id="6541" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/fig7-1.png?w=450&h=483&fit=676%2C483" data-permalink="https://gigadom.wordpress.com/2018/04/16/deep-learning-from-first-principles-in-python-r-and-octave-part-6/fig7-1-2/" data-large-file="https://gigadom.files.wordpress.com/2018/04/fig7-1.png?w=450&amp;h=483&amp;fit=676%2C483" alt="" data-image-title="fig7-1" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6541" data-recalc-dims="1"></p>
<h2>3.2b Dropout: Spiral data – R</h2>
<pre><code><span># Read the spiral dataset</span>
<span>source</span>(<span>&quot;DLfunctions61.R&quot;</span>)
# Load data
Z &lt;- as.matrix(read.csv(<span>&quot;spiral.csv&quot;</span>,header=<span>FALSE</span>)) 

<span># Setup the data</span>
X &lt;- Z[,<span>1</span>:<span>2</span>]
y &lt;- Z[,<span>3</span>]
X &lt;- t(X)
Y &lt;- t(y)

layersDimensions = c(<span>2</span>, <span>15</span>, <span>6</span>, <span>3</span>)
# Train a deep learning network
retvals = L_Layer_DeepModel(X, Y, layersDimensions,
                            hiddenActivationFunc=<span>'relu'</span>,
                            outputActivationFunc=<span>&quot;softmax&quot;</span>,
                            learningRate = <span>5.1</span>,
                            keep_prob=<span>0.9</span>,
                            numIterations = <span>9000</span>, 
                            print_cost = True)</code></pre>
<pre><code>parameters&lt;-retvals$parameters
#Plot decision boundary
plotDecisionBoundary1(Z,parameters)</code></pre>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/dropout-r.png" data-orig-size="1200,586" data-medium-file="https://gigadom.files.wordpress.com/2018/04/dropout-r.png?w=300" data-attachment-id="6586" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/dropout-r.png?w=450&h=330&fit=676%2C330" data-permalink="https://gigadom.wordpress.com/2018/04/16/deep-learning-from-first-principles-in-python-r-and-octave-part-6/dropout-r/" data-large-file="https://gigadom.files.wordpress.com/2018/04/dropout-r.png?w=450&amp;h=330&amp;fit=676%2C330" alt="" data-image-title="dropout-r" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6586" data-recalc-dims="1"></p>
<h2>3.3a Dropout: Circles data – Octave</h2>
<pre>data=csvread(&quot;circles.csv&quot;);

X=data(:,1:2);
Y=data(:,3);
layersDimensions = [2 11  1]; #tanh=-0.5(ok), #relu=0.1 best!

# Train a deep learning network
[weights biases costs]=L_Layer_DeepModel(X', Y', layersDimensions,
                               hiddenActivationFunc='relu', 
                               outputActivationFunc=&quot;sigmoid&quot;,
                               learningRate = 0.5,
                               lambd=0,
                               keep_prob=0.8,
                               numIterations = 10000,
                               initType=&quot;default&quot;);
plotCostVsIterations(10000,costs) 
#Plot decision boundary
plotDecisionBoundary1(data,weights, biases,keep_prob=1, hiddenActivationFunc=&quot;relu&quot;) 
</pre>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/dropout11-0.png" data-orig-size="1200,900" data-medium-file="https://gigadom.files.wordpress.com/2018/04/dropout11-0.png?w=300" data-attachment-id="6558" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/dropout11-0.png?w=450&h=507&fit=676%2C507" data-permalink="https://gigadom.wordpress.com/2018/04/16/deep-learning-from-first-principles-in-python-r-and-octave-part-6/dropout11-0/" data-large-file="https://gigadom.files.wordpress.com/2018/04/dropout11-0.png?w=450&amp;h=507&amp;fit=676%2C507" alt="" data-image-title="dropout11-0" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6558" data-recalc-dims="1"></p>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/dropout1.png" data-orig-size="1200,900" data-medium-file="https://gigadom.files.wordpress.com/2018/04/dropout1.png?w=300" data-attachment-id="6559" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/dropout1.png?w=450&h=507&fit=676%2C507" data-permalink="https://gigadom.wordpress.com/2018/04/16/deep-learning-from-first-principles-in-python-r-and-octave-part-6/dropout1/" data-large-file="https://gigadom.files.wordpress.com/2018/04/dropout1.png?w=450&amp;h=507&amp;fit=676%2C507" alt="" data-image-title="dropout1" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6559" data-recalc-dims="1"></p>
<h2>3.3b Dropout  Spiral data – Octave</h2>
<pre>source(&quot;DL61functions.m&quot;)
data=csvread(&quot;spiral.csv&quot;);

# Setup the data
X=data(:,1:2);
Y=data(:,3);

layersDimensions = [numFeats numHidden  numOutput];  
# Train a deep learning network
[weights biases costs]=L_Layer_DeepModel(X', Y', layersDimensions,
                               hiddenActivationFunc='relu', 
                               outputActivationFunc=&quot;softmax&quot;,
                               learningRate = 0.1,
                               lambd=0,
                               keep_prob=0.8,
                               numIterations = 10000); 

plotCostVsIterations(10000,costs)    
#Plot decision boundary                            
plotDecisionBoundary1(data,weights, biases,keep_prob=1, hiddenActivationFunc=&quot;relu&quot;)  
</pre>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/dropout2-0.png" data-orig-size="1200,900" data-medium-file="https://gigadom.files.wordpress.com/2018/04/dropout2-0.png?w=300" data-attachment-id="6560" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/dropout2-0.png?w=450&h=507&fit=676%2C507" data-permalink="https://gigadom.wordpress.com/2018/04/16/deep-learning-from-first-principles-in-python-r-and-octave-part-6/dropout2-0/" data-large-file="https://gigadom.files.wordpress.com/2018/04/dropout2-0.png?w=450&amp;h=507&amp;fit=676%2C507" alt="" data-image-title="dropout2-0" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6560" data-recalc-dims="1"></p>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/dropout2.png" data-orig-size="1200,900" data-medium-file="https://gigadom.files.wordpress.com/2018/04/dropout2.png?w=300" data-attachment-id="6561" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/dropout2.png?w=450&h=507&fit=676%2C507" data-permalink="https://gigadom.wordpress.com/2018/04/16/deep-learning-from-first-principles-in-python-r-and-octave-part-6/dropout2/" data-large-file="https://gigadom.files.wordpress.com/2018/04/dropout2.png?w=450&amp;h=507&amp;fit=676%2C507" alt="" data-image-title="dropout2" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6561" data-recalc-dims="1"></p>
<p><strong>Note</strong>: The Python, R and Octave code can be cloned/downloaded from Github at <a rel="nofollow" href="https://github.com/tvganesh/DeepLearning-Part6" target="_blank">DeepLearning-Part6</a><br>
<strong>Conclusion</strong><br>
This post further enhances my earlier L-Layer generic implementation of a Deep Learning network to include options for initialization techniques, L2 regularization or dropout regularization</p>
<p><strong>References</strong><br>
1. <a rel="nofollow" href="https://www.coursera.org/specializations/deep-learning" target="_blank">Deep Learning Specialization</a><br>
2. <a rel="nofollow" href="https://www.coursera.org/learn/neural-networks" target="_blank">Neural Networks for Machine Learning</a></p>
<p>Also see<br>
1. <a rel="nofollow" href="https://gigadom.wordpress.com/2013/02/21/architecting-a-cloud-based-ip-multimedia-system-ims/" target="_blank">Architecting a cloud based IP Multimedia System (IMS)</a><br>
2. <a rel="nofollow" href="https://gigadom.wordpress.com/2017/09/29/using-linear-programming-lp-for-optimizing-bowling-change-or-batting-lineup/" target="_blank">Using Linear Programming (LP) for optimizing bowling change or batting lineup in T20 cricket</a><br>
3. <a rel="nofollow" href="https://gigadom.wordpress.com/2017/12/05/my-book-practical-machine-learning-with-r-and-python-on-amazon/" target="_blank">My book ‘Practical Machine Learning with R and Python’ on Amazon</a><br>
4.<a rel="nofollow" href="https://gigadom.wordpress.com/2013/06/11/simulating-a-web-joint-in-android/" target="_blank"> Simulating a Web Joint in Android</a><br>
5. <a rel="nofollow" href="https://gigadom.wordpress.com/2017/02/14/inswinger-yorkr-swings-into-international-t20s/" target="_blank">Inswinger: yorkr swings into International T20s</a><br>
6. <a rel="nofollow" href="https://gigadom.wordpress.com/2016/06/23/introducing-qcsimulator-a-5-qubit-quantum-computing-simulator-in-r/" target="_blank">Introducing QCSimulator: A 5-qubit quantum computing simulator in R</a><br>
7. <a rel="nofollow" href="https://gigadom.wordpress.com/2011/10/31/opencv-ramblings-on-derivatives-histograms-and-contours/" target="_blank">Computer Vision: Ramblings on derivatives, histograms and contours</a><br>
8. <a rel="nofollow" href="https://gigadom.wordpress.com/2014/11/05/bend-it-like-bluemix-mongodb-using-auto-scale-part-1/" target="_blank">Bend it like Bluemix, MongoDB using Auto-scale – Part 1!</a><br>
9. <a rel="nofollow" href="https://gigadom.wordpress.com/2017/12/16/the-3rd-paperback-edition-of-my-books-on-cricket-on-amazon/" target="_blank">The 3rd paperback &amp; kindle editions of my books on Cricket, now on Amazon</a></p>
<p>To see all posts click <a rel="nofollow" href="https://gigadom.wordpress.com/aa-2/" target="_blank">Index of posts</a></p>
</div>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://gigadom.wordpress.com/2018/04/16/deep-learning-from-first-principles-in-python-r-and-octave-part-6/"> R – Giga thoughts …</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/UqZXsnEpGS0" height="1"></a></div>