<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/deep-learning-from-first-principles-in-python-r-and-octave-part-7/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://gigadom.wordpress.com/2018/04/29/deep-learning-from-first-principles-in-python-r-and-octave-part-7/"> R – Giga thoughts …</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><em>Artificial Intelligence is the new electricity. – </em>Prof Andrew Ng</p>
<p><em>Most of human and animal learning is unsupervised learning. If intelligence was a cake, unsupervised learning would be the cake, supervised learning would be the icing on the cake, and reinforcement learning would be the cherry on the cake. We know how to make the icing and the cherry, but we don’t know how to make the cake. We need to solve the unsupervised learning problem before we can even think of getting to true AI.  – </em>Yann LeCun, March 14, 2016 (Facebook)</p>
<div>
<h1>Introduction</h1>
<p>In this post ‘Deep Learning from first principles with Python, R and Octave-Part 7’, I implement optimization methods used in Stochastic Gradient Descent (SGD) to speed up the convergence. Specifically I discuss and implement the following gradient descent optimization techniques</p>
<p>a.Vanilla Stochastic Gradient Descent<br>
b.Learning rate decay<br>
c. Momentum method<br>
d. RMSProp<br>
e. Adaptive Moment Estimation (Adam)</p>
<p>This post, further enhances my generic  L-Layer Deep Learning Network implementations in  vectorized Python, R and Octave to also include the Stochastic Gradient Descent optimization techniques. You can clone/download the code from Github at <a rel="nofollow" href="https://github.com/tvganesh/DeepLearning-Part7" target="_blank">DeepLearning-Part7</a></p>
<p>Incidentally, a good discussion of the various optimizations methods used in Stochastic Gradient Optimization techniques can be seen at <a rel="nofollow" href="http://ruder.io/optimizing-gradient-descent/" target="_blank">Sebastian Ruder’s blog</a></p>
<p><strong>Note</strong>: In the vectorized Python, R and Octave implementations below only a  1024 random training samples were used. This was to reduce the computation time. You are free to use the entire data set (60000 training data) for the computation.</p>
<p>This post is largely based of on Prof Andrew Ng’s <a rel="nofollow" href="https://www.coursera.org/specializations/deep-learning" target="_blank">Deep Learning Specialization</a>.  All the above optimization techniques for Stochastic Gradient Descent are based on the technique of exponentially weighted average method. So for example if we had some time series data <img alt="theta_{1},theta_{2},theta_{3}... theta_{t}" title="theta_{1},theta_{2},theta_{3}... theta_{t}" src="https://s0.wp.com/latex.php?latex=%5Ctheta_%7B1%7D%2C%5Ctheta_%7B2%7D%2C%5Ctheta_%7B3%7D...+%5Ctheta_%7Bt%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> then we we can represent the exponentially average value at time ‘t’ as a sequence of the the previous value <img alt="v_{t-1}" title="v_{t-1}" src="https://s0.wp.com/latex.php?latex=v_%7Bt-1%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> and <img alt="theta_{t}" title="theta_{t}" src="https://s0.wp.com/latex.php?latex=%5Ctheta_%7Bt%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> as shown below<br>
<img alt="v_{t} = beta v_{t-1} + (1-beta)theta_{t}" title="v_{t} = beta v_{t-1} + (1-beta)theta_{t}" src="https://s0.wp.com/latex.php?latex=v_%7Bt%7D+%3D+%5Cbeta+v_%7Bt-1%7D+%2B+%281-%5Cbeta%29%5Ctheta_%7Bt%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"></p>
<p>Here <img alt="v_{t}" title="v_{t}" src="https://s0.wp.com/latex.php?latex=v_%7Bt%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> represent the average of the data set over <img alt="frac {1}{1-beta}" title="frac {1}{1-beta}" src="https://s0.wp.com/latex.php?latex=%5Cfrac+%7B1%7D%7B1-%5Cbeta%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex">  By choosing different values of <img alt="beta" title="beta" src="https://s0.wp.com/latex.php?latex=%5Cbeta&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex">, we can average over a larger or smaller number of the data points.<br>
We can write the equations as follows<br>
<img alt="v_{t} = beta v_{t-1} + (1-beta)theta_{t}" title="v_{t} = beta v_{t-1} + (1-beta)theta_{t}" src="https://s0.wp.com/latex.php?latex=v_%7Bt%7D+%3D+%5Cbeta+v_%7Bt-1%7D+%2B+%281-%5Cbeta%29%5Ctheta_%7Bt%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="v_{t-1} = beta v_{t-2} + (1-beta)theta_{t-1}" title="v_{t-1} = beta v_{t-2} + (1-beta)theta_{t-1}" src="https://s0.wp.com/latex.php?latex=v_%7Bt-1%7D+%3D+%5Cbeta+v_%7Bt-2%7D+%2B+%281-%5Cbeta%29%5Ctheta_%7Bt-1%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="v_{t-2} = beta v_{t-3} + (1-beta)theta_{t-2}" title="v_{t-2} = beta v_{t-3} + (1-beta)theta_{t-2}" src="https://s0.wp.com/latex.php?latex=v_%7Bt-2%7D+%3D+%5Cbeta+v_%7Bt-3%7D+%2B+%281-%5Cbeta%29%5Ctheta_%7Bt-2%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
and<br>
<img alt="v_{t-k} = beta v_{t-(k+1))} + (1-beta)theta_{t-k}" title="v_{t-k} = beta v_{t-(k+1))} + (1-beta)theta_{t-k}" src="https://s0.wp.com/latex.php?latex=v_%7Bt-k%7D+%3D+%5Cbeta+v_%7Bt-%28k%2B1%29%29%7D+%2B+%281-%5Cbeta%29%5Ctheta_%7Bt-k%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
By substitution we have<br>
<img alt="v_{t} = (1-beta)theta_{t} + beta v_{t-1}" title="v_{t} = (1-beta)theta_{t} + beta v_{t-1}" src="https://s0.wp.com/latex.php?latex=v_%7Bt%7D+%3D+%281-%5Cbeta%29%5Ctheta_%7Bt%7D+%2B+%5Cbeta+v_%7Bt-1%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="v_{t} = (1-beta)theta_{t} + beta ((1-beta)theta_{t-1}) + beta v_{t-2}" title="v_{t} = (1-beta)theta_{t} + beta ((1-beta)theta_{t-1}) + beta v_{t-2}" src="https://s0.wp.com/latex.php?latex=v_%7Bt%7D+%3D+%281-%5Cbeta%29%5Ctheta_%7Bt%7D+%2B+%5Cbeta+%28%281-%5Cbeta%29%5Ctheta_%7Bt-1%7D%29+%2B+%5Cbeta+v_%7Bt-2%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="v_{t} = (1-beta)theta_{t} + beta ((1-beta)theta_{t-1}) + beta ((1-beta)theta_{t-2}+ beta v_{t-3} )" title="v_{t} = (1-beta)theta_{t} + beta ((1-beta)theta_{t-1}) + beta ((1-beta)theta_{t-2}+ beta v_{t-3} )" src="https://s0.wp.com/latex.php?latex=v_%7Bt%7D+%3D+%281-%5Cbeta%29%5Ctheta_%7Bt%7D+%2B+%5Cbeta+%28%281-%5Cbeta%29%5Ctheta_%7Bt-1%7D%29+%2B+%5Cbeta+%28%281-%5Cbeta%29%5Ctheta_%7Bt-2%7D%2B+%5Cbeta+v_%7Bt-3%7D+%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"></p>
<p>Hence it can be seen that the <img alt="v_{t}" title="v_{t}" src="https://s0.wp.com/latex.php?latex=v_%7Bt%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> is the weighted sum over the previous values <img alt="theta_{k}" title="theta_{k}" src="https://s0.wp.com/latex.php?latex=%5Ctheta_%7Bk%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex">, which is an exponentially decaying function.</p>
<p>By the way, also take a look at my compact, minimal book “<a rel="nofollow" href="https://www.amazon.com/dp/B077WFS87Z" target="_blank">Practical Machine Learning with R and Python- Machine Learning in stereo</a>” available in Amazon in paperback($9.99) and Kindle($6.99) versions. This book is ideal for a quick reference of the various ML functions and associated measurements in both R and Python which are essential to delve deep into Deep Learning.<br>
<img data-orig-file="https://gigadom.files.wordpress.com/2018/04/untitled2.png" data-orig-size="1187,399" data-medium-file="https://gigadom.files.wordpress.com/2018/04/untitled2.png?w=300" data-attachment-id="6682" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/untitled2.png?w=450&h=227&fit=676%2C227" data-permalink="https://gigadom.wordpress.com/2018/04/29/deep-learning-from-first-principles-in-python-r-and-octave-part-7/untitled-121/" data-large-file="https://gigadom.files.wordpress.com/2018/04/untitled2.png?w=450&amp;h=227&amp;fit=676%2C227" alt="" data-image-title="Untitled" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6682" data-recalc-dims="1"></p>
<div>
<h2>1.1a. Stochastic Gradient Descent (Vanilla) – Python</h2>
<pre><code><span>import</span> numpy <span>as</span> np
<span>import</span> matplotlib
<span>import</span> matplotlib.pyplot <span>as</span> plt
<span>import</span> sklearn.linear_model
<span>import</span> pandas <span>as</span> pd
<span>import</span> sklearn
<span>import</span> sklearn.datasets
exec(open(<span>&quot;DLfunctions7.py&quot;</span>).read())
exec(open(<span>&quot;load_mnist.py&quot;</span>).read())

# Read the training data
training=list(read(dataset=<span>'training'</span>,path=<span>&quot;.\\mnist&quot;</span>))
test=list(read(dataset=<span>'testing'</span>,path=<span>&quot;.\\mnist&quot;</span>))
lbls=[]
pxls=[]
<span>for</span> i <span>in</span> range(<span>60000</span>):
       l,p=training[i]
       lbls.append(l)
       pxls.append(p)
labels= np.array(lbls)
pixels=np.array(pxls)       
y=labels.reshape(<span>-1</span>,<span>1</span>)
X=pixels.reshape(pixels.shape[<span>0</span>],<span>-1</span>)
X1=X.T
Y1=y.T

<span># Create a list of 1024 random numbers.</span>
permutation = list(np.random.permutation(<span>2</span>**<span>10</span>))
<span># Subset 16384 from the data</span>
X2 = X1[:, permutation]
Y2 = Y1[:, permutation].reshape((<span>1</span>,<span>2</span>**<span>10</span>))
# Set the layer dimensions  
layersDimensions=[<span>784</span>, <span>15</span>,<span>9</span>,<span>10</span>] 
# Perform SGD with regular gradient descent
parameters = L_Layer_DeepModel_SGD(X2, Y2, layersDimensions, hiddenActivationFunc=<span>'relu'</span>, 
                                   outputActivationFunc=<span>&quot;softmax&quot;</span>,learningRate = <span>0.01</span> ,
                                   optimizer=<span>&quot;gd&quot;</span>,
                                   mini_batch_size =<span>512</span>, num_epochs = 1<span>000</span>, print_cost = <span>True</span>,figure=<span>&quot;fig1.png&quot;</span>)
</code></pre>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/fig111.png?w=456" data-orig-size="389,278" data-medium-file="https://gigadom.files.wordpress.com/2018/04/fig111.png?w=456?w=300" data-attachment-id="6683" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/fig111.png?w=456" data-permalink="https://gigadom.wordpress.com/2018/04/29/deep-learning-from-first-principles-in-python-r-and-octave-part-7/fig11-3/" data-large-file="https://gigadom.files.wordpress.com/2018/04/fig111.png?w=456?w=389" alt="" data-image-title="fig11" sizes="(max-width: 389px) 100vw, 389px" data-comments-opened="1" class="wp-image-6683" data-recalc-dims="1"></p>
</div>
<div>
<h2>1.1b. Stochastic Gradient Descent (Vanilla) – R</h2>
<pre><code><span>source</span>(<span>&quot;mnist.R&quot;</span>)
<span>source</span>(<span>&quot;DLfunctions7.R&quot;</span>)
#Load and read MNIST data
load_mnist() 
x &lt;- t(train$x)
X &lt;- x[,<span>1</span>:<span>60000</span>]
y &lt;-train$y
y1 &lt;- y[<span>1</span>:<span>60000</span>]
y2 &lt;- as.matrix(y1)
Y=t(y2)

<span># Subset 1024 random samples from MNIST </span>
permutation = c(sample(<span>2</span>^<span>10</span>))
<span># Randomly shuffle the training data</span>
X1 = X[, permutation]
y1 = Y[<span>1</span>, permutation]
y2 &lt;- as.matrix(y1)
Y1=t(y2)
# Set layer dimensions
layersDimensions=c(<span>784</span>, <span>15</span>,<span>9</span>, <span>10</span>) 
<span># Perform SGD with regular gradient descent</span>
retvalsSGD= L_Layer_DeepModel_SGD(X1, Y1, layersDimensions,
                            hiddenActivationFunc=<span>'tanh'</span>,
                            outputActivationFunc=<span>&quot;softmax&quot;</span>,
                            learningRate = <span>0.05</span>,
                            optimizer=<span>&quot;gd&quot;</span>,
                            mini_batch_size = <span>512</span>, 
                            num_epochs = <span>5000</span>, 
                            print_cost = True)</code></pre>
<pre><code><span>#Plot the cost vs iterations</span>
iterations &lt;- seq(<span>0</span>,<span>5000</span>,<span>1000</span>)
costs=retvalsSGD$costs
df=data.frame(iterations,costs)
ggplot(df,aes(x=iterations,y=costs)) + geom_point() + geom_line(color=<span>&quot;blue&quot;</span>) +
 ggtitle(<span>&quot;Costs vs no of epochs&quot;</span>) + xlab(<span>&quot;No of epochss&quot;</span>) + ylab(<span>&quot;Cost&quot;</span>)</code></pre>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/fig1-21.png" data-orig-size="1344,960" data-medium-file="https://gigadom.files.wordpress.com/2018/04/fig1-21.png?w=300" data-attachment-id="6659" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/fig1-21.png?w=450&h=483&fit=676%2C483" data-permalink="https://gigadom.wordpress.com/2018/04/29/deep-learning-from-first-principles-in-python-r-and-octave-part-7/fig1-2-7/" data-large-file="https://gigadom.files.wordpress.com/2018/04/fig1-21.png?w=450&amp;h=483&amp;fit=676%2C483" alt="" data-image-title="fig1-2" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6659" data-recalc-dims="1"></p>
</div>
<div>
<h2>1.1c. Stochastic Gradient Descent (Vanilla) – Octave</h2>
<pre>source(&quot;DL7functions.m&quot;)
#Load and read MNIST
load('./mnist/mnist.txt.gz'); 
#Create a random permutatation from 1024
permutation = randperm(1024);
disp(length(permutation));

# Use this 1024 as the batch
X=trainX(permutation,:);
Y=trainY(permutation,:);

# Set layer dimensions
layersDimensions=[784, 15, 9, 10];
# Perform SGD with regular gradient descent
[weights biases costs]=L_Layer_DeepModel_SGD(X', Y', layersDimensions,
 hiddenActivationFunc='relu', 
 outputActivationFunc=&quot;softmax&quot;,
 learningRate = 0.005,
 lrDecay=true, 
 decayRate=1,
 lambd=0,
 keep_prob=1,
 optimizer=&quot;gd&quot;,
 beta=0.9,
 beta1=0.9,
 beta2=0.999,
 epsilon=10^-8,
 mini_batch_size = 512, 
 num_epochs = 5000);

plotCostVsEpochs(5000,costs);
</pre>
</div>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/figreg2-o1.png" data-orig-size="1200,900" data-medium-file="https://gigadom.files.wordpress.com/2018/04/figreg2-o1.png?w=300" data-attachment-id="6673" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/figreg2-o1.png?w=450&h=507&fit=676%2C507" data-permalink="https://gigadom.wordpress.com/2018/04/29/deep-learning-from-first-principles-in-python-r-and-octave-part-7/figreg2-o-2/" data-large-file="https://gigadom.files.wordpress.com/2018/04/figreg2-o1.png?w=450&amp;h=507&amp;fit=676%2C507" alt="" data-image-title="figReg2-o" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6673" data-recalc-dims="1"></p>
<div>
<h2>2.1. Stochastic Gradient Descent with Learning rate decay</h2>
<p>Since in Stochastic Gradient Descent,with  each epoch, we use slight different samples, the gradient descent algorithm, oscillates across the ravines and wanders around the minima, when a fixed learning rate is used. In this technique of ‘learning rate decay’ the learning rate is slowly decreased with the number of epochs and becomes smaller and smaller, so that gradient descent can take smaller steps towards the minima.</p>
<p>There are several techniques employed in learning rate decay</p>
<p>a) Exponential decay: <img alt="alpha = decayRate^{epochNum} *alpha_{0}" title="alpha = decayRate^{epochNum} *alpha_{0}" src="https://s0.wp.com/latex.php?latex=%5Calpha+%3D+decayRate%5E%7BepochNum%7D+%2A%5Calpha_%7B0%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
b) 1/t decay : <img alt="alpha = frac{alpha_{0}}{1 + decayRate*epochNum}" title="alpha = frac{alpha_{0}}{1 + decayRate*epochNum}" src="https://s0.wp.com/latex.php?latex=%5Calpha+%3D+%5Cfrac%7B%5Calpha_%7B0%7D%7D%7B1+%2B+decayRate%2AepochNum%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
c) <img alt="alpha = frac {decayRate}{sqrt(epochNum)}*alpha_{0}" title="alpha = frac {decayRate}{sqrt(epochNum)}*alpha_{0}" src="https://s0.wp.com/latex.php?latex=%5Calpha+%3D+%5Cfrac+%7BdecayRate%7D%7B%5Csqrt%28epochNum%29%7D%2A%5Calpha_%7B0%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"></p>
<p>In my implementation I have used the ‘exponential decay’. The code snippet for Python is shown below</p>
<pre>if lrDecay == True:
   learningRate = np.power(decayRate,(num_epochs/1000)) * learningRate
<code></code></pre>
</div>
<div>
<h2>2.1a. Stochastic Gradient Descent with Learning rate decay – Python</h2>
<pre><code><span>import</span> numpy <span>as</span> np
<span>import</span> matplotlib
<span>import</span> matplotlib.pyplot <span>as</span> plt
<span>import</span> sklearn.linear_model
<span>import</span> pandas <span>as</span> pd
<span>import</span> sklearn
<span>import</span> sklearn.datasets
exec(open(<span>&quot;DLfunctions7.py&quot;</span>).read())
exec(open(<span>&quot;load_mnist.py&quot;</span>).read())

# Read the MNIST data
training=list(read(dataset=<span>'training'</span>,path=<span>&quot;.\\mnist&quot;</span>))
test=list(read(dataset=<span>'testing'</span>,path=<span>&quot;.\\mnist&quot;</span>))
lbls=[]
pxls=[]
<span>for</span> i <span>in</span> range(<span>60000</span>):
       l,p=training[i]
       lbls.append(l)
       pxls.append(p)
labels= np.array(lbls)
pixels=np.array(pxls)       
y=labels.reshape(<span>-1</span>,<span>1</span>)
X=pixels.reshape(pixels.shape[<span>0</span>],<span>-1</span>)
X1=X.T
Y1=y.T

<span># Create a list of random numbers of 1024</span>
permutation = list(np.random.permutation(<span>2</span>**<span>10</span>))
<span># Subset 16384 from the data</span>
X2 = X1[:, permutation]
Y2 = Y1[:, permutation].reshape((<span>1</span>,<span>2</span>**<span>10</span>))
# Set layer dimensions
layersDimensions=[<span>784</span>, <span>15</span>,<span>9</span>,<span>10</span>] 
# Perform SGD with learning rate decay
parameters = L_Layer_DeepModel_SGD(X2, Y2, layersDimensions, hiddenActivationFunc=<span>'relu'</span>, 
                                   outputActivationFunc=<span>&quot;softmax&quot;</span>,
                                   learningRate = <span>0.01</span> , lrDecay=<span>True</span>, decayRate=<span>0.9999</span>,
                                   optimizer=<span>&quot;gd&quot;</span>,
                                   mini_batch_size =<span>512</span>, num_epochs = 1<span>000</span>, print_cost = <span>True</span>,figure=<span>&quot;fig2.png&quot;</span>)</code></pre>
</div>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/fig211.png?w=456" data-orig-size="389,278" data-medium-file="https://gigadom.files.wordpress.com/2018/04/fig211.png?w=456?w=300" data-attachment-id="6685" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/fig211.png?w=456" data-permalink="https://gigadom.wordpress.com/2018/04/29/deep-learning-from-first-principles-in-python-r-and-octave-part-7/fig21/" data-large-file="https://gigadom.files.wordpress.com/2018/04/fig211.png?w=456?w=389" alt="" data-image-title="fig21" sizes="(max-width: 389px) 100vw, 389px" data-comments-opened="1" class="wp-image-6685" data-recalc-dims="1"></p>
<div>
<h2>2.1b. Stochastic Gradient Descent with Learning rate decay – R</h2>
<pre><code><span>source</span>(<span>&quot;mnist.R&quot;</span>)
<span>source</span>(<span>&quot;DLfunctions7.R&quot;</span>)
# Read and load MNIST
load_mnist()
x &lt;- t(train$x)
X &lt;- x[,<span>1</span>:<span>60000</span>]
y &lt;-train$y
y1 &lt;- y[<span>1</span>:<span>60000</span>]
y2 &lt;- as.matrix(y1)
Y=t(y2)

<span># Subset 1024 random samples from MNIST </span>
permutation = c(sample(<span>2</span>^<span>10</span>))
<span># Randomly shuffle the training data</span>
X1 = X[, permutation]
y1 = Y[<span>1</span>, permutation]
y2 &lt;- as.matrix(y1)
Y1=t(y2)
# Set layer dimensions
layersDimensions=c(<span>784</span>, <span>15</span>,<span>9</span>, <span>10</span>) 
<span># Perform SGD with Learning rate decay</span>
retvalsSGD= L_Layer_DeepModel_SGD(X1, Y1, layersDimensions,
                                  hiddenActivationFunc=<span>'tanh'</span>,
                                  outputActivationFunc=<span>&quot;softmax&quot;</span>,
                                  learningRate = <span>0.05</span>,
                                  lrDecay=<span>TRUE</span>,
                                  decayRate=<span>0.9999</span>,
                                  optimizer=<span>&quot;gd&quot;</span>,
                                  mini_batch_size = <span>512</span>, 
                                  num_epochs = <span>5000</span>, 
                                  print_cost = True)</code></pre>
<pre><code><span>#Plot the cost vs iterations</span>
iterations &lt;- seq(<span>0</span>,<span>5000</span>,<span>1000</span>)
costs=retvalsSGD$costs
df=data.frame(iterations,costs)
ggplot(df,aes(x=iterations,y=costs)) + geom_point() + geom_line(color=<span>&quot;blue&quot;</span>) +
 ggtitle(<span>&quot;Costs vs number of epochs&quot;</span>) + xlab(<span>&quot;No of epochs&quot;</span>) + ylab(<span>&quot;Cost&quot;</span>)</code></pre>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/fig2-1.png" data-orig-size="1344,960" data-medium-file="https://gigadom.files.wordpress.com/2018/04/fig2-1.png?w=300" data-attachment-id="6662" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/fig2-1.png?w=450&h=483&fit=676%2C483" data-permalink="https://gigadom.wordpress.com/2018/04/29/deep-learning-from-first-principles-in-python-r-and-octave-part-7/fig2-1-8/" data-large-file="https://gigadom.files.wordpress.com/2018/04/fig2-1.png?w=450&amp;h=483&amp;fit=676%2C483" alt="" data-image-title="fig2-1" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6662" data-recalc-dims="1"></p>
</div>
<div>
<h2>2.1c. Stochastic Gradient Descent with Learning rate decay – Octave</h2>
<pre>source(&quot;DL7functions.m&quot;)
#Load and read MNIST
load('./mnist/mnist.txt.gz'); 
#Create a random permutatation from 1024
permutation = randperm(1024);
disp(length(permutation));

# Use this 1024 as the batch
X=trainX(permutation,:);
Y=trainY(permutation,:);

# Set layer dimensions
layersDimensions=[784, 15, 9, 10];
# Perform SGD with regular Learning rate decay
[weights biases costs]=L_Layer_DeepModel_SGD(X', Y', layersDimensions,
 hiddenActivationFunc='relu', 
 outputActivationFunc=&quot;softmax&quot;,
 learningRate = 0.01,
 lrDecay=true, 
 decayRate=0.999,
 lambd=0,
 keep_prob=1,
 optimizer=&quot;gd&quot;,
 beta=0.9,
 beta1=0.9,
 beta2=0.999,
 epsilon=10^-8,
 mini_batch_size = 512, 
 num_epochs = 5000);
plotCostVsEpochs(5000,costs)
</pre>
</div>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/fig2-o1.png" data-orig-size="1200,900" data-medium-file="https://gigadom.files.wordpress.com/2018/04/fig2-o1.png?w=300" data-attachment-id="6675" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/fig2-o1.png?w=450&h=507&fit=676%2C507" data-permalink="https://gigadom.wordpress.com/2018/04/29/deep-learning-from-first-principles-in-python-r-and-octave-part-7/fig2-o-3/" data-large-file="https://gigadom.files.wordpress.com/2018/04/fig2-o1.png?w=450&amp;h=507&amp;fit=676%2C507" alt="" data-image-title="fig2-o" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6675" data-recalc-dims="1"></p>
<div>
<h2>3.1. Stochastic Gradient Descent with Momentum</h2>
<p>Stochastic Gradient Descent with Momentum uses the exponentially weighted average method discusses above and more generally moves faster into the ravine than across it. The equations are<br>
<img alt="v_{dW}^l = beta v_{dW}^l + (1-beta)dW^{l}" title="v_{dW}^l = beta v_{dW}^l + (1-beta)dW^{l}" src="https://s0.wp.com/latex.php?latex=v_%7BdW%7D%5El+%3D+%5Cbeta+v_%7BdW%7D%5El+%2B+%281-%5Cbeta%29dW%5E%7Bl%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="v_{db}^l = beta v_{db}^l + (1-beta)db^{l}" title="v_{db}^l = beta v_{db}^l + (1-beta)db^{l}" src="https://s0.wp.com/latex.php?latex=v_%7Bdb%7D%5El+%3D+%5Cbeta+v_%7Bdb%7D%5El+%2B+%281-%5Cbeta%29db%5E%7Bl%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="W^{l} = W^{l} - alpha v_{dW}^l" title="W^{l} = W^{l} - alpha v_{dW}^l" src="https://s0.wp.com/latex.php?latex=W%5E%7Bl%7D+%3D+W%5E%7Bl%7D+-+%5Calpha+v_%7BdW%7D%5El&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="b^{l} = b^{l} - alpha v_{db}^l" title="b^{l} = b^{l} - alpha v_{db}^l" src="https://s0.wp.com/latex.php?latex=b%5E%7Bl%7D+%3D+b%5E%7Bl%7D+-+%5Calpha+v_%7Bdb%7D%5El&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> where<br>
<img alt="v_{dW}" title="v_{dW}" src="https://s0.wp.com/latex.php?latex=v_%7BdW%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> and <img alt="v_{db}" title="v_{db}" src="https://s0.wp.com/latex.php?latex=v_%7Bdb%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> are the momentum terms which are exponentially weighted with the corresponding gradients ‘dW’ and ‘db’ at the corresponding layer ‘l’ The code snippet for Stochastic Gradient Descent with momentum in R is shown below</p>
<pre># Perform Gradient Descent with momentum
# Input : Weights and biases
#       : beta
#       : gradients
#       : learning rate
#       : outputActivationFunc - Activation function at hidden layer sigmoid/softmax
#output : Updated weights after 1 iteration
gradientDescentWithMomentum  &lt;- function(parameters, gradients,v, beta, learningRate,outputActivationFunc=&quot;sigmoid&quot;){

    L = length(parameters)/2 # number of layers in the neural network    
    # Update rule for each parameter. Use a for loop.
    for(l in 1:(L-1)){
        # Compute velocities
        # v['dWk'] = beta *v['dWk'] + (1-beta)*dWk
        v[[paste(&quot;dW&quot;,l, sep=&quot;&quot;)]] = beta*v[[paste(&quot;dW&quot;,l, sep=&quot;&quot;)]] + 
                   (1-beta) * gradients[[paste('dW',l,sep=&quot;&quot;)]]
        v[[paste(&quot;db&quot;,l, sep=&quot;&quot;)]] = beta*v[[paste(&quot;db&quot;,l, sep=&quot;&quot;)]] + 
            (1-beta) * gradients[[paste('db',l,sep=&quot;&quot;)]]
        
        parameters[[paste(&quot;W&quot;,l,sep=&quot;&quot;)]] = parameters[[paste(&quot;W&quot;,l,sep=&quot;&quot;)]] -
            learningRate* v[[paste(&quot;dW&quot;,l, sep=&quot;&quot;)]] 
        parameters[[paste(&quot;b&quot;,l,sep=&quot;&quot;)]] = parameters[[paste(&quot;b&quot;,l,sep=&quot;&quot;)]] -
            learningRate* v[[paste(&quot;db&quot;,l, sep=&quot;&quot;)]] 
    }    
    # Compute for the Lth layer
    if(outputActivationFunc==&quot;sigmoid&quot;){
        v[[paste(&quot;dW&quot;,L, sep=&quot;&quot;)]] = beta*v[[paste(&quot;dW&quot;,L, sep=&quot;&quot;)]] + 
            (1-beta) * gradients[[paste('dW',L,sep=&quot;&quot;)]]
        v[[paste(&quot;db&quot;,L, sep=&quot;&quot;)]] = beta*v[[paste(&quot;db&quot;,L, sep=&quot;&quot;)]] + 
            (1-beta) * gradients[[paste('db',L,sep=&quot;&quot;)]]
        
        parameters[[paste(&quot;W&quot;,L,sep=&quot;&quot;)]] = parameters[[paste(&quot;W&quot;,L,sep=&quot;&quot;)]] -
            learningRate* v[[paste(&quot;dW&quot;,l, sep=&quot;&quot;)]]  
        parameters[[paste(&quot;b&quot;,L,sep=&quot;&quot;)]] = parameters[[paste(&quot;b&quot;,L,sep=&quot;&quot;)]] -
            learningRate* v[[paste(&quot;db&quot;,l, sep=&quot;&quot;)]]
        
    }else if (outputActivationFunc==&quot;softmax&quot;){
        v[[paste(&quot;dW&quot;,L, sep=&quot;&quot;)]] = beta*v[[paste(&quot;dW&quot;,L, sep=&quot;&quot;)]] + 
            (1-beta) * t(gradients[[paste('dW',L,sep=&quot;&quot;)]])
        v[[paste(&quot;db&quot;,L, sep=&quot;&quot;)]] = beta*v[[paste(&quot;db&quot;,L, sep=&quot;&quot;)]] + 
            (1-beta) * t(gradients[[paste('db',L,sep=&quot;&quot;)]])       
        parameters[[paste(&quot;W&quot;,L,sep=&quot;&quot;)]] = parameters[[paste(&quot;W&quot;,L,sep=&quot;&quot;)]] -
            learningRate* t(gradients[[paste(&quot;dW&quot;,L,sep=&quot;&quot;)]])
        parameters[[paste(&quot;b&quot;,L,sep=&quot;&quot;)]] = parameters[[paste(&quot;b&quot;,L,sep=&quot;&quot;)]] -
            learningRate* t(gradients[[paste(&quot;db&quot;,L,sep=&quot;&quot;)]])
    }
    return(parameters)
}</pre>
</div>
<div>
<h2>3.1a. Stochastic Gradient Descent with Momentum- Python</h2>
<pre><code><span>import</span> numpy <span>as</span> np
<span>import</span> matplotlib
<span>import</span> matplotlib.pyplot <span>as</span> plt
<span>import</span> sklearn.linear_model
<span>import</span> pandas <span>as</span> pd
<span>import</span> sklearn
<span>import</span> sklearn.datasets
# Read and load data
exec(open(<span>&quot;DLfunctions7.py&quot;</span>).read())
exec(open(<span>&quot;load_mnist.py&quot;</span>).read())
training=list(read(dataset=<span>'training'</span>,path=<span>&quot;.\\mnist&quot;</span>))
test=list(read(dataset=<span>'testing'</span>,path=<span>&quot;.\\mnist&quot;</span>))
lbls=[]
pxls=[]
<span>for</span> i <span>in</span> range(<span>60000</span>):
       l,p=training[i]
       lbls.append(l)
       pxls.append(p)
labels= np.array(lbls)
pixels=np.array(pxls)       
y=labels.reshape(<span>-1</span>,<span>1</span>)
X=pixels.reshape(pixels.shape[<span>0</span>],<span>-1</span>)
X1=X.T
Y1=y.T

<span># Create a list of random numbers of 1024</span>
permutation = list(np.random.permutation(<span>2</span>**<span>10</span>))
<span># Subset 16384 from the data</span>
X2 = X1[:, permutation]
Y2 = Y1[:, permutation].reshape((<span>1</span>,<span>2</span>**<span>10</span>))
layersDimensions=[<span>784</span>, <span>15</span>,<span>9</span>,<span>10</span>] 
# Perform SGD with momentum
parameters = L_Layer_DeepModel_SGD(X2, Y2, layersDimensions, hiddenActivationFunc=<span>'relu'</span>, 
                                   outputActivationFunc=<span>&quot;softmax&quot;</span>,learningRate = <span>0.01</span> ,
                                   optimizer=<span>&quot;momentum&quot;</span>, beta=<span>0.9</span>,
                                   mini_batch_size =<span>512</span>, num_epochs = 1<span>000</span>, print_cost = <span>True</span>,figure=<span>&quot;fig3.png&quot;</span>)</code></pre>
</div>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/fig311.png?w=456" data-orig-size="389,278" data-medium-file="https://gigadom.files.wordpress.com/2018/04/fig311.png?w=456?w=300" data-attachment-id="6687" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/fig311.png?w=456" data-permalink="https://gigadom.wordpress.com/2018/04/29/deep-learning-from-first-principles-in-python-r-and-octave-part-7/fig31/" data-large-file="https://gigadom.files.wordpress.com/2018/04/fig311.png?w=456?w=389" alt="" data-image-title="fig31" sizes="(max-width: 389px) 100vw, 389px" data-comments-opened="1" class="wp-image-6687" data-recalc-dims="1"></p>
<div>
<h2>3.1b. Stochastic Gradient Descent with Momentum- R</h2>
<pre><code><span>source</span>(<span>&quot;mnist.R&quot;</span>)
<span>source</span>(<span>&quot;DLfunctions7.R&quot;</span>)
load_mnist()
x &lt;- t(train$x)
X &lt;- x[,<span>1</span>:<span>60000</span>]
y &lt;-train$y
y1 &lt;- y[<span>1</span>:<span>60000</span>]
y2 &lt;- as.matrix(y1)
Y=t(y2)

<span># Subset 1024 random samples from MNIST </span>
permutation = c(sample(<span>2</span>^<span>10</span>))
<span># Randomly shuffle the training data</span>
X1 = X[, permutation]
y1 = Y[<span>1</span>, permutation]
y2 &lt;- as.matrix(y1)
Y1=t(y2)
layersDimensions=c(<span>784</span>, <span>15</span>,<span>9</span>, <span>10</span>) 
# Perform SGD with momentum
retvalsSGD= L_Layer_DeepModel_SGD(X1, Y1, layersDimensions,
                                  hiddenActivationFunc=<span>'tanh'</span>,
                                  outputActivationFunc=<span>&quot;softmax&quot;</span>,
                                  learningRate = <span>0.05</span>,
                                  optimizer=<span>&quot;momentum&quot;</span>,
                                  beta=<span>0.9</span>,
                                  mini_batch_size = <span>512</span>, 
                                  num_epochs = <span>5000</span>, 
                                  print_cost = True)</code><code>

</code></pre>
<pre><code><span>#Plot the cost vs iterations</span>
iterations &lt;- seq(<span>0</span>,<span>5000</span>,<span>1000</span>)
costs=retvalsSGD$costs
df=data.frame(iterations,costs)
ggplot(df,aes(x=iterations,y=costs)) + geom_point() + geom_line(color=<span>&quot;blue&quot;</span>) +
 ggtitle(<span>&quot;Costs vs number of epochs&quot;</span>) + xlab(<span>&quot;No of epochs&quot;</span>) + ylab(<span>&quot;Cost&quot;</span>)</code></pre>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/fig4-11.png" data-orig-size="1344,960" data-medium-file="https://gigadom.files.wordpress.com/2018/04/fig4-11.png?w=300" data-attachment-id="6666" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/fig4-11.png?w=450&h=483&fit=676%2C483" data-permalink="https://gigadom.wordpress.com/2018/04/29/deep-learning-from-first-principles-in-python-r-and-octave-part-7/fig4-1-9/" data-large-file="https://gigadom.files.wordpress.com/2018/04/fig4-11.png?w=450&amp;h=483&amp;fit=676%2C483" alt="" data-image-title="fig4-1" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6666" data-recalc-dims="1"></p>
</div>
<div>
<h2>3.1c. Stochastic Gradient Descent with Momentum- Octave</h2>
<pre>source(&quot;DL7functions.m&quot;)
#Load and read MNIST
load('./mnist/mnist.txt.gz'); 
#Create a random permutatation from 60K
permutation = randperm(1024);
disp(length(permutation));

# Use this 1024 as the batch
X=trainX(permutation,:);
Y=trainY(permutation,:);

# Set layer dimensions
layersDimensions=[784, 15, 9, 10];
# Perform SGD with Momentum
[weights biases costs]=L_Layer_DeepModel_SGD(X', Y', layersDimensions,
 hiddenActivationFunc='relu', 
 outputActivationFunc=&quot;softmax&quot;,
 learningRate = 0.01,
 lrDecay=false, 
 decayRate=1,
 lambd=0,
 keep_prob=1,
 optimizer=&quot;momentum&quot;,
 beta=0.9,
 beta1=0.9,
 beta2=0.999,
 epsilon=10^-8,
 mini_batch_size = 512, 
 num_epochs = 5000);

plotCostVsEpochs(5000,costs)
</pre>
</div>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/fig3-o.png" data-orig-size="1200,900" data-medium-file="https://gigadom.files.wordpress.com/2018/04/fig3-o.png?w=300" data-attachment-id="6676" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/fig3-o.png?w=450&h=507&fit=676%2C507" data-permalink="https://gigadom.wordpress.com/2018/04/29/deep-learning-from-first-principles-in-python-r-and-octave-part-7/fig3-o/" data-large-file="https://gigadom.files.wordpress.com/2018/04/fig3-o.png?w=450&amp;h=507&amp;fit=676%2C507" alt="" data-image-title="fig3-o" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6676" data-recalc-dims="1"></p>
<div>
<h2>4.1. Stochastic Gradient Descent with RMSProp</h2>
<p>Stochastic Gradient Descent with RMSProp tries to move faster towards the minima while dampening the oscillations across the ravine.<br>
The equations are</p>
<p><img alt="s_{dW}^l = beta_{1} s_{dW}^l + (1-beta_{1})(dW^{l})^{2}" title="s_{dW}^l = beta_{1} s_{dW}^l + (1-beta_{1})(dW^{l})^{2}" src="https://s0.wp.com/latex.php?latex=s_%7BdW%7D%5El+%3D+%5Cbeta_%7B1%7D+s_%7BdW%7D%5El+%2B+%281-%5Cbeta_%7B1%7D%29%28dW%5E%7Bl%7D%29%5E%7B2%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="s_{db}^l = beta_{1} s_{db}^l + (1-beta_{1})(db^{l})^2" title="s_{db}^l = beta_{1} s_{db}^l + (1-beta_{1})(db^{l})^2" src="https://s0.wp.com/latex.php?latex=s_%7Bdb%7D%5El+%3D+%5Cbeta_%7B1%7D+s_%7Bdb%7D%5El+%2B+%281-%5Cbeta_%7B1%7D%29%28db%5E%7Bl%7D%29%5E2&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="W^{l} = W^{l} - frac {alpha s_{dW}^l}{sqrt (s_{dW}^l + epsilon) }" title="W^{l} = W^{l} - frac {alpha s_{dW}^l}{sqrt (s_{dW}^l + epsilon) }" src="https://s0.wp.com/latex.php?latex=W%5E%7Bl%7D+%3D+W%5E%7Bl%7D+-+%5Cfrac+%7B%5Calpha+s_%7BdW%7D%5El%7D%7B%5Csqrt+%28s_%7BdW%7D%5El+%2B+%5Cepsilon%29+%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="b^{l} = b^{l} - frac {alpha s_{db}^l}{sqrt (s_{db}^l + epsilon) }" title="b^{l} = b^{l} - frac {alpha s_{db}^l}{sqrt (s_{db}^l + epsilon) }" src="https://s0.wp.com/latex.php?latex=b%5E%7Bl%7D+%3D+b%5E%7Bl%7D+-+%5Cfrac+%7B%5Calpha+s_%7Bdb%7D%5El%7D%7B%5Csqrt+%28s_%7Bdb%7D%5El+%2B+%5Cepsilon%29+%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
where <img alt="s_{dW}" title="s_{dW}" src="https://s0.wp.com/latex.php?latex=s_%7BdW%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> and <img alt="s_{db}" title="s_{db}" src="https://s0.wp.com/latex.php?latex=s_%7Bdb%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> are the RMSProp terms which are exponentially weighted with the corresponding gradients ‘dW’ and ‘db’ at the corresponding layer ‘l’</p>
<p>The code snippet in Octave is shown below</p>
<pre># Update parameters with RMSProp
# Input : parameters
#       : gradients
#       : s
#       : beta
#       : learningRate
#       : 
#output : Updated parameters RMSProp
function [weights biases] = gradientDescentWithRMSProp(weights, biases,gradsDW,gradsDB, sdW, sdB, beta1, epsilon, learningRate,outputActivationFunc=&quot;sigmoid&quot;)
    L = size(weights)(2); # number of layers in the neural network
    # Update rule for each parameter. 
    for l=1:(L-1)
        sdW{l} =  beta1*sdW{l} + (1 -beta1) * gradsDW{l} .* gradsDW{l};
        sdB{l} =  beta1*sdB{l} + (1 -beta1) * gradsDB{l} .* gradsDB{l};
        weights{l} = weights{l} - learningRate* gradsDW{l} ./ sqrt(sdW{l} + epsilon); 
        biases{l} = biases{l} -  learningRate* gradsDB{l} ./ sqrt(sdB{l} + epsilon);
    endfor
  
    if (strcmp(outputActivationFunc,&quot;sigmoid&quot;))
        sdW{L} =  beta1*sdW{L} + (1 -beta1) * gradsDW{L} .* gradsDW{L};
        sdB{L} =  beta1*sdB{L} + (1 -beta1) * gradsDB{L} .* gradsDB{L};
        weights{L} = weights{L} -learningRate* gradsDW{L} ./ sqrt(sdW{L} +epsilon); 
        biases{L} = biases{L} -learningRate* gradsDB{L} ./ sqrt(sdB{L} + epsilon);
     elseif (strcmp(outputActivationFunc,&quot;softmax&quot;))
        sdW{L} =  beta1*sdW{L} + (1 -beta1) * gradsDW{L}' .* gradsDW{L}';
        sdB{L} =  beta1*sdB{L} + (1 -beta1) * gradsDB{L}' .* gradsDB{L}';
        weights{L} = weights{L} -learningRate* gradsDW{L}' ./ sqrt(sdW{L} +epsilon); 
        biases{L} = biases{L} -learningRate* gradsDB{L}' ./ sqrt(sdB{L} + epsilon);
     endif   
end
</pre>
</div>
<div>
<h2>4.1a. Stochastic Gradient Descent with RMSProp – Python</h2>
<pre><code><span>import</span> numpy <span>as</span> np
<span>import</span> matplotlib
<span>import</span> matplotlib.pyplot <span>as</span> plt
<span>import</span> sklearn.linear_model
<span>import</span> pandas <span>as</span> pd
<span>import</span> sklearn
<span>import</span> sklearn.datasets
exec(open(<span>&quot;DLfunctions7.py&quot;</span>).read())
exec(open(<span>&quot;load_mnist.py&quot;</span>).read())

# Read and load MNIST
training=list(read(dataset=<span>'training'</span>,path=<span>&quot;.\\mnist&quot;</span>))
test=list(read(dataset=<span>'testing'</span>,path=<span>&quot;.\\mnist&quot;</span>))
lbls=[]
pxls=[]
<span>for</span> i <span>in</span> range(<span>60000</span>):
       l,p=training[i]
       lbls.append(l)
       pxls.append(p)
labels= np.array(lbls)
pixels=np.array(pxls)       
y=labels.reshape(<span>-1</span>,<span>1</span>)
X=pixels.reshape(pixels.shape[<span>0</span>],<span>-1</span>)
X1=X.T
Y1=y.T

print(<span>&quot;X1=&quot;</span>,X1.shape)
print(<span>&quot;y1=&quot;</span>,Y1.shape)

<span># Create a list of random numbers of 1024</span>
permutation = list(np.random.permutation(<span>2</span>**<span>10</span>))
<span># Subset 16384 from the data</span>
X2 = X1[:, permutation]
Y2 = Y1[:, permutation].reshape((<span>1</span>,<span>2</span>**<span>10</span>))
  
layersDimensions=[<span>784</span>, <span>15</span>,<span>9</span>,<span>10</span>] 
# Use SGD with RMSProp
parameters = L_Layer_DeepModel_SGD(X2, Y2, layersDimensions, hiddenActivationFunc=<span>'relu'</span>, 
                                   outputActivationFunc=<span>&quot;softmax&quot;</span>,learningRate = <span>0.01</span> ,
                                   optimizer=<span>&quot;rmsprop&quot;</span>, beta1=<span>0.7</span>, epsilon=<span>1e-8</span>,
                                   mini_batch_size =<span>512</span>, num_epochs = 1<span>000</span>, print_cost = <span>True</span>,figure=<span>&quot;fig4.png&quot;</span>)</code></pre>
</div>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/fig411.png?w=456" data-orig-size="389,278" data-medium-file="https://gigadom.files.wordpress.com/2018/04/fig411.png?w=456?w=300" data-attachment-id="6690" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/fig411.png?w=456" data-permalink="https://gigadom.wordpress.com/2018/04/29/deep-learning-from-first-principles-in-python-r-and-octave-part-7/fig41/" data-large-file="https://gigadom.files.wordpress.com/2018/04/fig411.png?w=456?w=389" alt="" data-image-title="fig41" sizes="(max-width: 389px) 100vw, 389px" data-comments-opened="1" class="wp-image-6690" data-recalc-dims="1"></p>
<div>
<h2>4.1b. Stochastic Gradient Descent with RMSProp – R</h2>
<pre><code><span>source</span>(<span>&quot;mnist.R&quot;</span>)
<span>source</span>(<span>&quot;DLfunctions7.R&quot;</span>)
load_mnist()
x &lt;- t(train$x)
X &lt;- x[,<span>1</span>:<span>60000</span>]
y &lt;-train$y
y1 &lt;- y[<span>1</span>:<span>60000</span>]
y2 &lt;- as.matrix(y1)
Y=t(y2)

<span># Subset 1024 random samples from MNIST </span>
permutation = c(sample(<span>2</span>^<span>10</span>))
<span># Randomly shuffle the training data</span>
X1 = X[, permutation]
y1 = Y[<span>1</span>, permutation]
y2 &lt;- as.matrix(y1)
Y1=t(y2)
layersDimensions=c(<span>784</span>, <span>15</span>,<span>9</span>, <span>10</span>) 
#Perform SGD with RMSProp
retvalsSGD= L_Layer_DeepModel_SGD(X1, Y1, layersDimensions,
                                  hiddenActivationFunc=<span>'tanh'</span>,
                                  outputActivationFunc=<span>&quot;softmax&quot;</span>,
                                  learningRate = <span>0.001</span>,
                                  optimizer=<span>&quot;rmsprop&quot;</span>,
                                  beta1=<span>0.9</span>,
                                  epsilon=<span>10</span>^-<span>8</span>,
                                  mini_batch_size = <span>512</span>, 
                                  num_epochs = <span>5000</span> , 
                                  print_cost = True)</code></pre>
<pre><code><span>#Plot the cost vs iterations</span>
iterations &lt;- seq(<span>0</span>,<span>5000</span>,<span>1000</span>)
costs=retvalsSGD$costs
df=data.frame(iterations,costs)
ggplot(df,aes(x=iterations,y=costs)) + geom_point() + geom_line(color=<span>&quot;blue&quot;</span>) +
 ggtitle(<span>&quot;Costs vs number of epochs&quot;</span>) + xlab(<span>&quot;No of epochs&quot;</span>) + ylab(<span>&quot;Cost&quot;</span>)
</code></pre>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/fig5-11.png" data-orig-size="1344,960" data-medium-file="https://gigadom.files.wordpress.com/2018/04/fig5-11.png?w=300" data-attachment-id="6669" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/fig5-11.png?w=450&h=483&fit=676%2C483" data-permalink="https://gigadom.wordpress.com/2018/04/29/deep-learning-from-first-principles-in-python-r-and-octave-part-7/fig5-1-6/" data-large-file="https://gigadom.files.wordpress.com/2018/04/fig5-11.png?w=450&amp;h=483&amp;fit=676%2C483" alt="" data-image-title="fig5-1" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6669" data-recalc-dims="1"></p>
<h2>4.1c. Stochastic Gradient Descent with RMSProp – Octave</h2>
<pre>source(&quot;DL7functions.m&quot;)
load('./mnist/mnist.txt.gz'); 
#Create a random permutatation from 1024
permutation = randperm(1024);

# Use this 1024 as the batch
X=trainX(permutation,:);
Y=trainY(permutation,:);

# Set layer dimensions
layersDimensions=[784, 15, 9, 10];
#Perform SGD with RMSProp
[weights biases costs]=L_Layer_DeepModel_SGD(X', Y', layersDimensions,
 hiddenActivationFunc='relu', 
 outputActivationFunc=&quot;softmax&quot;,
 learningRate = 0.005,
 lrDecay=false, 
 decayRate=1,
 lambd=0,
 keep_prob=1,
 optimizer=&quot;rmsprop&quot;,
 beta=0.9,
 beta1=0.9,
 beta2=0.999,
 epsilon=1,
 mini_batch_size = 512, 
 num_epochs = 5000);
plotCostVsEpochs(5000,costs)
</pre>
</div>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/fig4-o.png" data-orig-size="1200,900" data-medium-file="https://gigadom.files.wordpress.com/2018/04/fig4-o.png?w=300" data-attachment-id="6677" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/fig4-o.png?w=450&h=507&fit=676%2C507" data-permalink="https://gigadom.wordpress.com/2018/04/29/deep-learning-from-first-principles-in-python-r-and-octave-part-7/fig4-o/" data-large-file="https://gigadom.files.wordpress.com/2018/04/fig4-o.png?w=450&amp;h=507&amp;fit=676%2C507" alt="" data-image-title="fig4-o" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6677" data-recalc-dims="1"></p>
<div>
<h2>5.1. Stochastic Gradient Descent with Adam</h2>
<p>Adaptive Moment Estimate is a combination of the momentum (1st moment) and RMSProp(2nd moment). The equations for Adam are below<br>
<img alt="v_{dW}^l = beta_{1} v_{dW}^l + (1-beta_{1})dW^{l}" title="v_{dW}^l = beta_{1} v_{dW}^l + (1-beta_{1})dW^{l}" src="https://s0.wp.com/latex.php?latex=v_%7BdW%7D%5El+%3D+%5Cbeta_%7B1%7D+v_%7BdW%7D%5El+%2B+%281-%5Cbeta_%7B1%7D%29dW%5E%7Bl%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="v_{db}^l = beta_{1} v_{db}^l + (1-beta_{1})db^{l}" title="v_{db}^l = beta_{1} v_{db}^l + (1-beta_{1})db^{l}" src="https://s0.wp.com/latex.php?latex=v_%7Bdb%7D%5El+%3D+%5Cbeta_%7B1%7D+v_%7Bdb%7D%5El+%2B+%281-%5Cbeta_%7B1%7D%29db%5E%7Bl%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
The bias corrections for the 1st moment<br>
<img alt="vCorrected_{dW}^l= frac {v_{dW}^l}{1 - beta_{1}^{t}}" title="vCorrected_{dW}^l= frac {v_{dW}^l}{1 - beta_{1}^{t}}" src="https://s0.wp.com/latex.php?latex=vCorrected_%7BdW%7D%5El%3D+%5Cfrac+%7Bv_%7BdW%7D%5El%7D%7B1+-+%5Cbeta_%7B1%7D%5E%7Bt%7D%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="vCorrected_{db}^l= frac {v_{db}^l}{1 - beta_{1}^{t}}" title="vCorrected_{db}^l= frac {v_{db}^l}{1 - beta_{1}^{t}}" src="https://s0.wp.com/latex.php?latex=vCorrected_%7Bdb%7D%5El%3D+%5Cfrac+%7Bv_%7Bdb%7D%5El%7D%7B1+-+%5Cbeta_%7B1%7D%5E%7Bt%7D%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"></p>
<p>Similarly the moving average for the 2nd moment- RMSProp<br>
<img alt="s_{dW}^l = beta_{2} s_{dW}^l + (1-beta_{2})(dW^{l})^2" title="s_{dW}^l = beta_{2} s_{dW}^l + (1-beta_{2})(dW^{l})^2" src="https://s0.wp.com/latex.php?latex=s_%7BdW%7D%5El+%3D+%5Cbeta_%7B2%7D+s_%7BdW%7D%5El+%2B+%281-%5Cbeta_%7B2%7D%29%28dW%5E%7Bl%7D%29%5E2&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="s_{db}^l = beta_{2} s_{db}^l + (1-beta_{2})(db^{l})^2" title="s_{db}^l = beta_{2} s_{db}^l + (1-beta_{2})(db^{l})^2" src="https://s0.wp.com/latex.php?latex=s_%7Bdb%7D%5El+%3D+%5Cbeta_%7B2%7D+s_%7Bdb%7D%5El+%2B+%281-%5Cbeta_%7B2%7D%29%28db%5E%7Bl%7D%29%5E2&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
The bias corrections for the 2nd moment<br>
<img alt="sCorrected_{dW}^l= frac {s_{dW}^l}{1 - beta_{2}^{t}}" title="sCorrected_{dW}^l= frac {s_{dW}^l}{1 - beta_{2}^{t}}" src="https://s0.wp.com/latex.php?latex=sCorrected_%7BdW%7D%5El%3D+%5Cfrac+%7Bs_%7BdW%7D%5El%7D%7B1+-+%5Cbeta_%7B2%7D%5E%7Bt%7D%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="sCorrected_{db}^l= frac {s_{db}^l}{1 - beta_{2}^{t}}" title="sCorrected_{db}^l= frac {s_{db}^l}{1 - beta_{2}^{t}}" src="https://s0.wp.com/latex.php?latex=sCorrected_%7Bdb%7D%5El%3D+%5Cfrac+%7Bs_%7Bdb%7D%5El%7D%7B1+-+%5Cbeta_%7B2%7D%5E%7Bt%7D%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"></p>
<p>The Adam Gradient Descent is given by<br>
<img alt="W^{l} = W^{l} - frac {alpha vCorrected_{dW}^l}{sqrt (s_{dW}^l + epsilon) }" title="W^{l} = W^{l} - frac {alpha vCorrected_{dW}^l}{sqrt (s_{dW}^l + epsilon) }" src="https://s0.wp.com/latex.php?latex=W%5E%7Bl%7D+%3D+W%5E%7Bl%7D+-+%5Cfrac+%7B%5Calpha+vCorrected_%7BdW%7D%5El%7D%7B%5Csqrt+%28s_%7BdW%7D%5El+%2B+%5Cepsilon%29+%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="b^{l} = b^{l} - frac {alpha vCorrected_{db}^l}{sqrt (s_{db}^l + epsilon) }" title="b^{l} = b^{l} - frac {alpha vCorrected_{db}^l}{sqrt (s_{db}^l + epsilon) }" src="https://s0.wp.com/latex.php?latex=b%5E%7Bl%7D+%3D+b%5E%7Bl%7D+-+%5Cfrac+%7B%5Calpha+vCorrected_%7Bdb%7D%5El%7D%7B%5Csqrt+%28s_%7Bdb%7D%5El+%2B+%5Cepsilon%29+%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
The code snippet of Adam in R is included below</p>
<pre># Perform Gradient Descent with Adam
# Input : Weights and biases
#       : beta1
#       : epsilon
#       : gradients
#       : learning rate
#       : outputActivationFunc - Activation function at hidden layer sigmoid/softmax
#output : Updated weights after 1 iteration
gradientDescentWithAdam  &lt;- function(parameters, gradients,v, s, t, 
                        beta1=0.9, beta2=0.999, epsilon=10^-8, learningRate=0.1,outputActivationFunc=&quot;sigmoid&quot;){
    
    L = length(parameters)/2 # number of layers in the neural network
    v_corrected &lt;- list()
    s_corrected &lt;- list()
    # Update rule for each parameter. Use a for loop.
    for(l in 1:(L-1)){
        # v['dWk'] = beta *v['dWk'] + (1-beta)*dWk
        v[[paste(&quot;dW&quot;,l, sep=&quot;&quot;)]] = beta1*v[[paste(&quot;dW&quot;,l, sep=&quot;&quot;)]] + 
            (1-beta1) * gradients[[paste('dW',l,sep=&quot;&quot;)]]
        v[[paste(&quot;db&quot;,l, sep=&quot;&quot;)]] = beta1*v[[paste(&quot;db&quot;,l, sep=&quot;&quot;)]] + 
            (1-beta1) * gradients[[paste('db',l,sep=&quot;&quot;)]]
        
        
        # Compute bias-corrected first moment estimate. 
        v_corrected[[paste(&quot;dW&quot;,l, sep=&quot;&quot;)]] = v[[paste(&quot;dW&quot;,l, sep=&quot;&quot;)]]/(1-beta1^t)
        v_corrected[[paste(&quot;db&quot;,l, sep=&quot;&quot;)]] = v[[paste(&quot;db&quot;,l, sep=&quot;&quot;)]]/(1-beta1^t)
               
        # Element wise multiply of gradients
        s[[paste(&quot;dW&quot;,l, sep=&quot;&quot;)]] = beta2*s[[paste(&quot;dW&quot;,l, sep=&quot;&quot;)]] + 
            (1-beta2) * gradients[[paste('dW',l,sep=&quot;&quot;)]] * gradients[[paste('dW',l,sep=&quot;&quot;)]]
        s[[paste(&quot;db&quot;,l, sep=&quot;&quot;)]] = beta2*s[[paste(&quot;db&quot;,l, sep=&quot;&quot;)]] + 
            (1-beta2) * gradients[[paste('db',l,sep=&quot;&quot;)]] * gradients[[paste('db',l,sep=&quot;&quot;)]]
        
        # Compute bias-corrected second moment estimate. 
        s_corrected[[paste(&quot;dW&quot;,l, sep=&quot;&quot;)]] = s[[paste(&quot;dW&quot;,l, sep=&quot;&quot;)]]/(1-beta2^t)
        s_corrected[[paste(&quot;db&quot;,l, sep=&quot;&quot;)]] = s[[paste(&quot;db&quot;,l, sep=&quot;&quot;)]]/(1-beta2^t)
        
        # Update parameters. 
        d1=sqrt(s_corrected[[paste(&quot;dW&quot;,l, sep=&quot;&quot;)]]+epsilon)
        d2=sqrt(s_corrected[[paste(&quot;db&quot;,l, sep=&quot;&quot;)]]+epsilon)        
                
        parameters[[paste(&quot;W&quot;,l,sep=&quot;&quot;)]] = parameters[[paste(&quot;W&quot;,l,sep=&quot;&quot;)]] -
            learningRate * v_corrected[[paste(&quot;dW&quot;,l, sep=&quot;&quot;)]]/d1
        parameters[[paste(&quot;b&quot;,l,sep=&quot;&quot;)]] = parameters[[paste(&quot;b&quot;,l,sep=&quot;&quot;)]] -
            learningRate*v_corrected[[paste(&quot;db&quot;,l, sep=&quot;&quot;)]]/d2
    }    
    # Compute for the Lth layer
    if(outputActivationFunc==&quot;sigmoid&quot;){
        v[[paste(&quot;dW&quot;,L, sep=&quot;&quot;)]] = beta1*v[[paste(&quot;dW&quot;,L, sep=&quot;&quot;)]] + 
            (1-beta1) * gradients[[paste('dW',L,sep=&quot;&quot;)]]
        v[[paste(&quot;db&quot;,L, sep=&quot;&quot;)]] = beta1*v[[paste(&quot;db&quot;,L, sep=&quot;&quot;)]] + 
            (1-beta1) * gradients[[paste('db',L,sep=&quot;&quot;)]]
                
        # Compute bias-corrected first moment estimate. 
        v_corrected[[paste(&quot;dW&quot;,L, sep=&quot;&quot;)]] = v[[paste(&quot;dW&quot;,L, sep=&quot;&quot;)]]/(1-beta1^t)
        v_corrected[[paste(&quot;db&quot;,L, sep=&quot;&quot;)]] = v[[paste(&quot;db&quot;,L, sep=&quot;&quot;)]]/(1-beta1^t)
                
        # Element wise multiply of gradients
        s[[paste(&quot;dW&quot;,L, sep=&quot;&quot;)]] = beta2*s[[paste(&quot;dW&quot;,L, sep=&quot;&quot;)]] + 
            (1-beta2) * gradients[[paste('dW',L,sep=&quot;&quot;)]] * gradients[[paste('dW',L,sep=&quot;&quot;)]]
        s[[paste(&quot;db&quot;,L, sep=&quot;&quot;)]] = beta2*s[[paste(&quot;db&quot;,L, sep=&quot;&quot;)]] + 
            (1-beta2) * gradients[[paste('db',L,sep=&quot;&quot;)]] * gradients[[paste('db',L,sep=&quot;&quot;)]]
        
        # Compute bias-corrected second moment estimate. 
        s_corrected[[paste(&quot;dW&quot;,L, sep=&quot;&quot;)]] = s[[paste(&quot;dW&quot;,L, sep=&quot;&quot;)]]/(1-beta2^t)
        s_corrected[[paste(&quot;db&quot;,L, sep=&quot;&quot;)]] = s[[paste(&quot;db&quot;,L, sep=&quot;&quot;)]]/(1-beta2^t)
        
        # Update parameters. 
        d1=sqrt(s_corrected[[paste(&quot;dW&quot;,L, sep=&quot;&quot;)]]+epsilon)
        d2=sqrt(s_corrected[[paste(&quot;db&quot;,L, sep=&quot;&quot;)]]+epsilon)  
        
        parameters[[paste(&quot;W&quot;,L,sep=&quot;&quot;)]] = parameters[[paste(&quot;W&quot;,L,sep=&quot;&quot;)]] -
            learningRate * v_corrected[[paste(&quot;dW&quot;,L, sep=&quot;&quot;)]]/d1
        parameters[[paste(&quot;b&quot;,L,sep=&quot;&quot;)]] = parameters[[paste(&quot;b&quot;,L,sep=&quot;&quot;)]] -
            learningRate*v_corrected[[paste(&quot;db&quot;,L, sep=&quot;&quot;)]]/d2
        
    }else if (outputActivationFunc==&quot;softmax&quot;){
        v[[paste(&quot;dW&quot;,L, sep=&quot;&quot;)]] = beta1*v[[paste(&quot;dW&quot;,L, sep=&quot;&quot;)]] + 
            (1-beta1) * t(gradients[[paste('dW',L,sep=&quot;&quot;)]])
        v[[paste(&quot;db&quot;,L, sep=&quot;&quot;)]] = beta1*v[[paste(&quot;db&quot;,L, sep=&quot;&quot;)]] + 
            (1-beta1) * t(gradients[[paste('db',L,sep=&quot;&quot;)]])
                
        # Compute bias-corrected first moment estimate. 
        v_corrected[[paste(&quot;dW&quot;,L, sep=&quot;&quot;)]] = v[[paste(&quot;dW&quot;,L, sep=&quot;&quot;)]]/(1-beta1^t)
        v_corrected[[paste(&quot;db&quot;,L, sep=&quot;&quot;)]] = v[[paste(&quot;db&quot;,L, sep=&quot;&quot;)]]/(1-beta1^t)        
        
        # Element wise multiply of gradients
        s[[paste(&quot;dW&quot;,L, sep=&quot;&quot;)]] = beta2*s[[paste(&quot;dW&quot;,L, sep=&quot;&quot;)]] + 
            (1-beta2) * t(gradients[[paste('dW',L,sep=&quot;&quot;)]]) * t(gradients[[paste('dW',L,sep=&quot;&quot;)]])
        s[[paste(&quot;db&quot;,L, sep=&quot;&quot;)]] = beta2*s[[paste(&quot;db&quot;,L, sep=&quot;&quot;)]] + 
            (1-beta2) * t(gradients[[paste('db',L,sep=&quot;&quot;)]]) * t(gradients[[paste('db',L,sep=&quot;&quot;)]])
        
        # Compute bias-corrected second moment estimate. 
        s_corrected[[paste(&quot;dW&quot;,L, sep=&quot;&quot;)]] = s[[paste(&quot;dW&quot;,L, sep=&quot;&quot;)]]/(1-beta2^t)
        s_corrected[[paste(&quot;db&quot;,L, sep=&quot;&quot;)]] = s[[paste(&quot;db&quot;,L, sep=&quot;&quot;)]]/(1-beta2^t)
        
        # Update parameters. 
        d1=sqrt(s_corrected[[paste(&quot;dW&quot;,L, sep=&quot;&quot;)]]+epsilon)
        d2=sqrt(s_corrected[[paste(&quot;db&quot;,L, sep=&quot;&quot;)]]+epsilon) 
        
        parameters[[paste(&quot;W&quot;,L,sep=&quot;&quot;)]] = parameters[[paste(&quot;W&quot;,L,sep=&quot;&quot;)]] -
            learningRate * v_corrected[[paste(&quot;dW&quot;,L, sep=&quot;&quot;)]]/d1
        parameters[[paste(&quot;b&quot;,L,sep=&quot;&quot;)]] = parameters[[paste(&quot;b&quot;,L,sep=&quot;&quot;)]] -
            learningRate*v_corrected[[paste(&quot;db&quot;,L, sep=&quot;&quot;)]]/d2
    }
    return(parameters)
}
</pre>
</div>
<div>
<h2>5.1a. Stochastic Gradient Descent with Adam – Python</h2>
<pre><code><span>import</span> numpy <span>as</span> np
<span>import</span> matplotlib
<span>import</span> matplotlib.pyplot <span>as</span> plt
<span>import</span> sklearn.linear_model
<span>import</span> pandas <span>as</span> pd
<span>import</span> sklearn
<span>import</span> sklearn.datasets
exec(open(<span>&quot;DLfunctions7.py&quot;</span>).read())
exec(open(<span>&quot;load_mnist.py&quot;</span>).read())
training=list(read(dataset=<span>'training'</span>,path=<span>&quot;.\\mnist&quot;</span>))
test=list(read(dataset=<span>'testing'</span>,path=<span>&quot;.\\mnist&quot;</span>))
lbls=[]
pxls=[]
print(len(training))
<span>#for i in range(len(training)):</span>
<span>for</span> i <span>in</span> range(<span>60000</span>):
       l,p=training[i]
       lbls.append(l)
       pxls.append(p)
labels= np.array(lbls)
pixels=np.array(pxls)       
y=labels.reshape(<span>-1</span>,<span>1</span>)
X=pixels.reshape(pixels.shape[<span>0</span>],<span>-1</span>)
X1=X.T
Y1=y.T


<span># Create a list of random numbers of 1024</span>
permutation = list(np.random.permutation(<span>2</span>**<span>10</span>))
<span># Subset 16384 from the data</span>
X2 = X1[:, permutation]
Y2 = Y1[:, permutation].reshape((<span>1</span>,<span>2</span>**<span>10</span>))
layersDimensions=[<span>784</span>, <span>15</span>,<span>9</span>,<span>10</span>] 
#Perform SGD with Adam optimization
parameters = L_Layer_DeepModel_SGD(X2, Y2, layersDimensions, hiddenActivationFunc=<span>'relu'</span>, 
                                   outputActivationFunc=<span>&quot;softmax&quot;</span>,learningRate = <span>0.01</span> ,
                                   optimizer=<span>&quot;adam&quot;</span>, beta1=<span>0.9</span>, beta2=<span>0.9</span>, epsilon = <span>1e-8</span>,
                                   mini_batch_size =<span>512</span>, num_epochs = 1<span>000</span>, print_cost = <span>True</span>, figure=<span>&quot;fig5.png&quot;</span>)</code></pre>
</div>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/figa.png?w=456" data-orig-size="389,278" data-medium-file="https://gigadom.files.wordpress.com/2018/04/figa.png?w=456?w=300" data-attachment-id="6694" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/figa.png?w=456" data-permalink="https://gigadom.wordpress.com/2018/04/29/deep-learning-from-first-principles-in-python-r-and-octave-part-7/figa-2/" data-large-file="https://gigadom.files.wordpress.com/2018/04/figa.png?w=456?w=389" alt="" data-image-title="figa" sizes="(max-width: 389px) 100vw, 389px" data-comments-opened="1" class="wp-image-6694" data-recalc-dims="1"></p>
<div>
<h2>5.1b. Stochastic Gradient Descent with Adam – R</h2>
<pre><code><span>source</span>(<span>&quot;mnist.R&quot;</span>)
<span>source</span>(<span>&quot;DLfunctions7.R&quot;</span>)
load_mnist()
x &lt;- t(train$x)
X &lt;- x[,<span>1</span>:<span>60000</span>]
y &lt;-train$y
y1 &lt;- y[<span>1</span>:<span>60000</span>]
y2 &lt;- as.matrix(y1)
Y=t(y2)

<span># Subset 1024 random samples from MNIST </span>
permutation = c(sample(<span>2</span>^<span>10</span>))
<span># Randomly shuffle the training data</span>
X1 = X[, permutation]
y1 = Y[<span>1</span>, permutation]
y2 &lt;- as.matrix(y1)
Y1=t(y2)
layersDimensions=c(<span>784</span>, <span>15</span>,<span>9</span>, <span>10</span>) 
#Perform SGD with Adam
retvalsSGD= L_Layer_DeepModel_SGD(X1, Y1, layersDimensions,
                                  hiddenActivationFunc=<span>'tanh'</span>,
                                  outputActivationFunc=<span>&quot;softmax&quot;</span>,
                                  learningRate = <span>0.005</span>,
                                  optimizer=<span>&quot;adam&quot;</span>,
                                  beta1=<span>0.7</span>,
                                  beta2=<span>0.9</span>,
                                  epsilon=<span>10</span>^-<span>8</span>,
                                  mini_batch_size = <span>512</span>, 
                                  num_epochs = <span>5000</span> , 
                                  print_cost = True)</code></pre>
<pre><code><span>#Plot the cost vs iterations</span>
iterations &lt;- seq(<span>0</span>,<span>5000</span>,<span>1000</span>)
costs=retvalsSGD$costs
df=data.frame(iterations,costs)
ggplot(df,aes(x=iterations,y=costs)) + geom_point() + geom_line(color=<span>&quot;blue&quot;</span>) +
 ggtitle(<span>&quot;Costs vs number of epochs&quot;</span>) + xlab(<span>&quot;No of epochs&quot;</span>) + ylab(<span>&quot;Cost&quot;</span>)</code></pre>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/fig6-1.png" data-orig-size="1344,960" data-medium-file="https://gigadom.files.wordpress.com/2018/04/fig6-1.png?w=300" data-attachment-id="6671" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/fig6-1.png?w=450&h=483&fit=676%2C483" data-permalink="https://gigadom.wordpress.com/2018/04/29/deep-learning-from-first-principles-in-python-r-and-octave-part-7/fig6-1-2/" data-large-file="https://gigadom.files.wordpress.com/2018/04/fig6-1.png?w=450&amp;h=483&amp;fit=676%2C483" alt="" data-image-title="fig6-1" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6671" data-recalc-dims="1"></p>
<h2>5.1c. Stochastic Gradient Descent with Adam – Octave</h2>
<pre>source(&quot;DL7functions.m&quot;)
load('./mnist/mnist.txt.gz'); 
#Create a random permutatation from 1024
permutation = randperm(1024);
disp(length(permutation));

# Use this 1024 as the batch
X=trainX(permutation,:);
Y=trainY(permutation,:);
# Set layer dimensions
layersDimensions=[784, 15, 9, 10];

# Note the high value for epsilon. 
#Otherwise GD with Adam does not seem to converge   
# Perform SGD with Adam         
[weights biases costs]=L_Layer_DeepModel_SGD(X', Y', layersDimensions,
                       hiddenActivationFunc='relu', 
                       outputActivationFunc=&quot;softmax&quot;,
                       learningRate = 0.1,
                       lrDecay=false, 
                       decayRate=1,
                       lambd=0,
                       keep_prob=1,
                       optimizer=&quot;adam&quot;,
                       beta=0.9,
                       beta1=0.9,
                       beta2=0.9,
                       epsilon=100,
                       mini_batch_size = 512, 
                       num_epochs = 5000);
plotCostVsEpochs(5000,costs)
</pre>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/04/fig5-o.png" data-orig-size="1200,900" data-medium-file="https://gigadom.files.wordpress.com/2018/04/fig5-o.png?w=300" data-attachment-id="6678" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/fig5-o.png?w=450&h=507&fit=676%2C507" data-permalink="https://gigadom.wordpress.com/2018/04/29/deep-learning-from-first-principles-in-python-r-and-octave-part-7/fig5-o/" data-large-file="https://gigadom.files.wordpress.com/2018/04/fig5-o.png?w=450&amp;h=507&amp;fit=676%2C507" alt="" data-image-title="fig5-o" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6678" data-recalc-dims="1"></p>
<p><strong>Conclusion</strong>: In this post I discuss and implement several Stochastic Gradient Descent optimization methods. The implementation of these methods enhance my already existing generic L-Layer Deep Learning Network implementation in vectorized Python, R and Octave, which I had discussed in the previous post in this series on Deep Learning from first principles in Python, R and Octave. Check it out, if you haven’t already. As already mentioned the code for this post can be cloned/forked from Github at <a rel="nofollow" href="https://github.com/tvganesh/DeepLearning-Part7" target="_blank">DeepLearning-Part7</a></p>
</div>
<p>Watch this space! I’ll be back!</p>
<p>Also see<br>
1.<a rel="nofollow" href="https://gigadom.wordpress.com/2017/12/05/my-book-practical-machine-learning-with-r-and-python-on-amazon/" target="_blank">My book ‘Practical Machine Learning with R and Python’ on Amazon</a><br>
2. <a rel="nofollow" href="https://gigadom.wordpress.com/2018/01/30/deep-learning-from-first-principles-in-python-r-and-octave-part-3/" target="_blank">Deep Learning from first principles in Python, R and Octave – Part 3</a><br>
3. <a rel="nofollow" href="https://gigadom.wordpress.com/2011/11/09/experiments-with-deblurring-using-opencv/" target="_blank">Experiments with deblurring using OpenCV</a><br>
3.<a rel="nofollow" href="https://gigadom.wordpress.com/2011/05/13/design-principles-of-scalable-distributed-systems/" target="_blank"> Design Principles of Scalable, Distributed Systems</a><br>
4. <a rel="nofollow" href="https://gigadom.wordpress.com/2015/10/02/natural-language-processing-what-would-shakespeare-say/" target="_blank">Natural language processing: What would Shakespeare say?</a><br>
5. <a rel="nofollow" href="https://gigadom.wordpress.com/2016/04/21/yorkr-crashes-the-ipl-party-part-3/" target="_blank">yorkr crashes the IPL party! – Part 3!</a><br>
6. <a rel="nofollow" href="https://gigadom.wordpress.com/2017/03/11/cricketr-flexes-new-muscles-the-final-analysis/" target="_blank">cricketr flexes new muscles: The final analysis</a></p>
<p>To see all post click <a rel="nofollow" href="https://gigadom.wordpress.com/aa-2/" target="_blank">Index of posts</a></p>
</div>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://gigadom.wordpress.com/2018/04/29/deep-learning-from-first-principles-in-python-r-and-octave-part-7/"> R – Giga thoughts …</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/KWROKFku2Vw" height="1"></a></div>