<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/deep-learning-from-first-principles-in-python-r-and-octave-part-8/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://gigadom.wordpress.com/2018/05/06/deep-learning-from-first-principles-in-python-r-and-octave-part-8/"> R – Giga thoughts …</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<div>
<h2>1. Introduction</h2>
<p><em>You don’t understand anything until you learn it more than one way.</em> Marvin Minsky<br>
<em>No computer has ever been designed that is ever aware of what it’s doing; but most of the time, we aren’t either. </em>Marvin Minsky<br>
<em>A wealth of information creates a poverty of attention.</em> Herbert Simon</p>
<p>This post, Deep Learning from first Principles in Python, R and Octave-Part8, is my final post in my Deep Learning from first principles series. In this post, I discuss and implement a key functionality needed while building Deep Learning networks viz. ‘Gradient Checking’. Gradient Checking is an important method to check the correctness of your implementation, specifically the forward propagation and the backward propagation cycles of an implementation. In addition I also discuss some tips for tuning hyper-parameters of a Deep Learning network based on my experience.</p>
<p>My post in this  ‘Deep Learning Series’ so far were<br>
1. <a rel="nofollow" href="https://gigadom.wordpress.com/2018/01/04/deep-learning-from-basic-principles-in-python-r-and-octave-part-1/" target="_blank">Deep Learning from first principles in Python, R and Octave – Part 1</a> In part 1, I implement logistic regression as a neural network in vectorized Python, R and Octave<br>
2. <a rel="nofollow" href="https://gigadom.wordpress.com/2018/01/11/deep-learning-from-first-principles-in-python-r-and-octave-part-2/" target="_blank">Deep Learning from first principles in Python, R and Octave – Part 2</a> In the second part I implement a simple Neural network with just 1 hidden layer and a sigmoid activation output function<br>
3. <a rel="nofollow" href="https://gigadom.wordpress.com/2018/01/30/deep-learning-from-first-principles-in-python-r-and-octave-part-3/" target="_blank">Deep Learning from first principles in Python, R and Octave – Part 3</a> The 3rd part implemented a multi-layer Deep Learning Network with sigmoid activation output in vectorized Python, R and Octave<br>
4. <a rel="nofollow" href="https://gigadom.wordpress.com/2018/02/26/deep-learning-from-first-principles-in-python-r-and-octave-part-4/" target="_blank">Deep Learning from first principles in Python, R and Octave – Part 4</a> The 4th part deals with multi-class classification. Specifically, I derive the Jacobian of the Softmax function and enhance my L-Layer DL network to include Softmax output function in addition to Sigmoid activation<br>
5. <a rel="nofollow" href="https://gigadom.wordpress.com/2018/03/23/deep-learning-from-first-principles-in-python-r-and-octave-part-5/" target="_blank">Deep Learning from first principles in Python, R and Octave – Part 5</a> This post uses the Softmax classifier implemented to classify MNIST digits using a L-layer Deep Learning network<br>
6. <a rel="nofollow" href="https://gigadom.wordpress.com/2018/04/16/deep-learning-from-first-principles-in-python-r-and-octave-part-6/" target="_blank">Deep Learning from first principles in Python, R and Octave – Part 6</a> The 6th part adds more bells and whistles to my L-Layer DL network, by including different initialization types namely He and Xavier. Besides L2 Regularization and random dropout is added.<br>
7. <a rel="nofollow" href="https://gigadom.wordpress.com/2018/04/29/deep-learning-from-first-principles-in-python-r-and-octave-part-7/" target="_blank">Deep Learning from first principles in Python, R and Octave – Part 7</a> The 7th part deals with Stochastic Gradient Descent Optimization methods including momentum, RMSProp and Adam<br>
8. Deep Learning from first principles in Python, R and Octave – Part 8 – This post implements a critical function for ensuring the correctness of a L-Layer Deep Learning network implementation using Gradient Checking</p>
<p>By the way , also take a look at my compact, minimal book “<a rel="nofollow" href="https://www.amazon.com/dp/B077WFS87Z" target="_blank">Practical Machine Learning with R and Python- Machine Learning in stereo</a>” available in Amazon in paperback($9.99) and Kindle($6.99) versions. This book is ideal for a quick reference of the various ML functions and associated measurements in both R and Python which are essential to delve deep into Deep Learning.<br>
<img data-orig-file="https://gigadom.files.wordpress.com/2018/04/untitled2.png" data-orig-size="1187,399" data-medium-file="https://gigadom.files.wordpress.com/2018/04/untitled2.png?w=300" data-attachment-id="6682" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/04/untitled2.png?w=450&h=227&fit=676%2C227" data-permalink="https://gigadom.wordpress.com/2018/04/29/deep-learning-from-first-principles-in-python-r-and-octave-part-7/untitled-121/" data-large-file="https://gigadom.files.wordpress.com/2018/04/untitled2.png?w=676" alt="" data-image-title="Untitled" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6682" data-recalc-dims="1"></p>
<p>Gradient Checking is based on the following approach. One iteration of Gradient Descent computes and updates the parameters <img alt="theta" title="theta" src="https://s0.wp.com/latex.php?latex=%5Ctheta&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> by doing<br>
<img alt="theta := theta - frac{d}{dtheta}J(theta)" title="theta := theta - frac{d}{dtheta}J(theta)" src="https://s0.wp.com/latex.php?latex=%5Ctheta+%3A%3D+%5Ctheta+-+%5Cfrac%7Bd%7D%7Bd%5Ctheta%7DJ%28%5Ctheta%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex">.<br>
To minimize the cost we will need to minimize <img alt="J(theta)" title="J(theta)" src="https://s0.wp.com/latex.php?latex=J%28%5Ctheta%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
Let <img alt="g(theta)" title="g(theta)" src="https://s0.wp.com/latex.php?latex=g%28%5Ctheta%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> be a function that computes the derivative <img alt="frac {d}{dtheta}J(theta)" title="frac {d}{dtheta}J(theta)" src="https://s0.wp.com/latex.php?latex=%5Cfrac+%7Bd%7D%7Bd%5Ctheta%7DJ%28%5Ctheta%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex">. Gradient Checking allows us to numerically evaluate the implementation of the function <img alt="g(theta)" title="g(theta)" src="https://s0.wp.com/latex.php?latex=g%28%5Ctheta%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> and verify its correctness.<br>
We know the derivative of a function is given by<br>
<img alt="frac {d}{dtheta}J(theta) = lim-&gt;0 frac {J(theta +epsilon) - J(theta -epsilon)} {2*epsilon}" title="frac {d}{dtheta}J(theta) = lim-&gt;0 frac {J(theta +epsilon) - J(theta -epsilon)} {2*epsilon}" src="https://s0.wp.com/latex.php?latex=%5Cfrac+%7Bd%7D%7Bd%5Ctheta%7DJ%28%5Ctheta%29+%3D+lim-%3E0+%5Cfrac+%7BJ%28%5Ctheta+%2B%5Cepsilon%29+-+J%28%5Ctheta+-%5Cepsilon%29%7D+%7B2%2A%5Cepsilon%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<strong>Note</strong>: The above derivative is based on the 2 sided derivative. The 1-sided derivative  is given by <img alt="frac {d}{dtheta}J(theta) = lim-&gt;0 frac {J(theta +epsilon) - J(theta)} {epsilon}" title="frac {d}{dtheta}J(theta) = lim-&gt;0 frac {J(theta +epsilon) - J(theta)} {epsilon}" src="https://s0.wp.com/latex.php?latex=%5Cfrac+%7Bd%7D%7Bd%5Ctheta%7DJ%28%5Ctheta%29+%3D+lim-%3E0+%5Cfrac+%7BJ%28%5Ctheta+%2B%5Cepsilon%29+-+J%28%5Ctheta%29%7D+%7B%5Cepsilon%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
Gradient Checking is based on the 2-sided derivative because the error is of the order <img alt="O(epsilon^{2})" title="O(epsilon^{2})" src="https://s0.wp.com/latex.php?latex=O%28%5Cepsilon%5E%7B2%7D%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> as opposed <img alt="O(epsilon)" title="O(epsilon)" src="https://s0.wp.com/latex.php?latex=O%28%5Cepsilon%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> for the 1-sided derivative.<br>
Hence Gradient Check uses the 2 sided derivative as follows.<br>
<img alt="g(theta) = lim-&gt;0 frac {J(theta +epsilon) - J(theta -epsilon)} {2*epsilon}" title="g(theta) = lim-&gt;0 frac {J(theta +epsilon) - J(theta -epsilon)} {2*epsilon}" src="https://s0.wp.com/latex.php?latex=g%28%5Ctheta%29+%3D+lim-%3E0+%5Cfrac+%7BJ%28%5Ctheta+%2B%5Cepsilon%29+-+J%28%5Ctheta+-%5Cepsilon%29%7D+%7B2%2A%5Cepsilon%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"></p>
<p>In Gradient Check the following is done<br>
A) Run one normal cycle of your implementation by doing the following<br>
a) Compute the output activation by running 1 cycle of forward propagation<br>
b) Compute the cost using the output activation<br>
c) Compute the gradients using backpropation (grad)</p>
<p>B) Perform gradient check steps as below<br>
a) Set <img alt="theta" title="theta" src="https://s0.wp.com/latex.php?latex=%5Ctheta&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> . Flatten all ‘weights’ and ‘bias’ matrices and vectors to a column vector.<br>
b) Initialize <img alt="theta+" title="theta+" src="https://s0.wp.com/latex.php?latex=%5Ctheta%2B&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> by bumping up <img alt="theta" title="theta" src="https://s0.wp.com/latex.php?latex=%5Ctheta&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> by adding <img alt="epsilon" title="epsilon" src="https://s0.wp.com/latex.php?latex=%5Cepsilon&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> (<img alt="theta + epsilon" title="theta + epsilon" src="https://s0.wp.com/latex.php?latex=%5Ctheta+%2B+%5Cepsilon&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex">)<br>
c) Perform forward propagation with <img alt="theta+" title="theta+" src="https://s0.wp.com/latex.php?latex=%5Ctheta%2B&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
d) Compute cost with <img alt="theta+" title="theta+" src="https://s0.wp.com/latex.php?latex=%5Ctheta%2B&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> i.e. <img alt="J(theta+)" title="J(theta+)" src="https://s0.wp.com/latex.php?latex=J%28%5Ctheta%2B%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
e) Initialize  <img alt="theta-" title="theta-" src="https://s0.wp.com/latex.php?latex=%5Ctheta-&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> by bumping down <img alt="theta" title="theta" src="https://s0.wp.com/latex.php?latex=%5Ctheta&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> by subtracting <img alt="epsilon" title="epsilon" src="https://s0.wp.com/latex.php?latex=%5Cepsilon&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> <img alt="(theta - epsilon)" title="(theta - epsilon)" src="https://s0.wp.com/latex.php?latex=%28%5Ctheta+-+%5Cepsilon%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
f) Perform forward propagation with <img alt="theta-" title="theta-" src="https://s0.wp.com/latex.php?latex=%5Ctheta-&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
g) Compute cost with <img alt="theta-" title="theta-" src="https://s0.wp.com/latex.php?latex=%5Ctheta-&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> i.e.  <img alt="J(theta-)" title="J(theta-)" src="https://s0.wp.com/latex.php?latex=J%28%5Ctheta-%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
h) Compute <img alt="frac {d} {dtheta} J(theta)" title="frac {d} {dtheta} J(theta)" src="https://s0.wp.com/latex.php?latex=%5Cfrac+%7Bd%7D+%7Bd%5Ctheta%7D+J%28%5Ctheta%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> or ‘gradapprox’ as<img alt="frac {J(theta+) - J(theta-) } {2epsilon} " title="frac {J(theta+) - J(theta-) } {2epsilon} " src="https://s0.wp.com/latex.php?latex=%5Cfrac+%7BJ%28%5Ctheta%2B%29+-+J%28%5Ctheta-%29+%7D+%7B2%5Cepsilon%7D+&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex">using the 2 sided derivative.<br>
i) Compute L2norm or the Euclidean distance between ‘grad’ and ‘gradapprox’. If the<br>
diference is of the order of <img alt="10^{-5}" title="10^{-5}" src="https://s0.wp.com/latex.php?latex=10%5E%7B-5%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> or <img alt="10^{-7}" title="10^{-7}" src="https://s0.wp.com/latex.php?latex=10%5E%7B-7%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> the implementation is correct. In the <a rel="nofollow" href="https://www.coursera.org/specializations/deep-learning" target="_blank">Deep Learning Specialization</a> Prof Andrew Ng mentions that if the difference is of the order of <img alt="10^{-7}" title="10^{-7}" src="https://s0.wp.com/latex.php?latex=10%5E%7B-7%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> then the implementation is correct. A difference of <img alt="10^{-5}" title="10^{-5}" src="https://s0.wp.com/latex.php?latex=10%5E%7B-5%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> is also ok. Anything more than that is a cause of worry and you should look at your code more closely. To see more details click <a rel="nofollow" href="http://ufldl.stanford.edu/wiki/index.php/Gradient_checking_and_advanced_optimization" target="_blank">Gradient checking and advanced optimization</a></p>
<p>You can clone/download the code from Github at <a rel="nofollow" href="https://github.com/tvganesh/DeepLearning-Part8" target="_blank">DeepLearning-Part8</a></p>
<p>After spending a better part of 3 days, I now realize how critical Gradient Check is for ensuring the correctness of you implementation. Initially I was getting very high difference and did not know how to understand the results or debug my implementation. After many hours of staring at the results, I  was able to finally arrive at a way, to localize issues in the implementation. In fact, I did catch a small bug in my Python code, which did not exist in the R and Octave implementations. I will demonstrate this below</p>
</div>
<div>
<h2>1.1a Gradient Check – Sigmoid Activation – Python</h2>
<pre><code><span>import</span> numpy <span>as</span> np
<span>import</span> matplotlib

exec(open(<span>&quot;DLfunctions8.py&quot;</span>).read())
exec(open(<span>&quot;testcases.py&quot;</span>).read())
#Load the data
train_X, train_Y, test_X, test_Y = load_dataset()
#Set layer dimensions
layersDimensions = [<span>2</span>,<span>4</span>,<span>1</span>]  
parameters = initializeDeepModel(layersDimensions)
#Perform forward prop
AL, caches, dropoutMat = forwardPropagationDeep(train_X, parameters, keep_prob=<span>1</span>, hiddenActivationFunc=<span>&quot;relu&quot;</span>,outputActivationFunc=<span>&quot;sigmoid&quot;</span>)
#Compute cost
cost = computeCost(AL, train_Y, outputActivationFunc=<span>&quot;sigmoid&quot;</span>) 
print(<span>&quot;cost=&quot;</span>,cost)
#Perform backprop and get gradients
gradients = backwardPropagationDeep(AL, train_Y, caches, dropoutMat, lambd=<span>0</span>, keep_prob=<span>1</span>,                                   hiddenActivationFunc=<span>&quot;relu&quot;</span>,outputActivationFunc=<span>&quot;sigmoid&quot;</span>)

epsilon = <span>1e-7</span>
outputActivationFunc=<span>&quot;sigmoid&quot;</span>

<span># Set-up variables
</span># Flatten parameters to a vector
parameters_values, _ = dictionary_to_vector(parameters)
#Flatten gradients to a vector
grad = gradients_to_vector(parameters,gradients)
num_parameters = parameters_values.shape[<span>0</span>]
#Initialize
J_plus = np.zeros((num_parameters, <span>1</span>))
J_minus = np.zeros((num_parameters, <span>1</span>))
gradapprox = np.zeros((num_parameters, <span>1</span>))

<span># Compute gradapprox using 2 sided derivative</span>
<span>for</span> i <span>in</span> range(num_parameters):
    <span># Compute J_plus[i]. </span>
    thetaplus = np.copy(parameters_values)                                   
    thetaplus[i][<span>0</span>] = thetaplus[i][<span>0</span>] + epsilon                                 
    AL, caches, dropoutMat = forwardPropagationDeep(train_X, vector_to_dictionary(parameters,thetaplus), keep_prob=<span>1</span>, 
                                              hiddenActivationFunc=<span>&quot;relu&quot;</span>,outputActivationFunc=outputActivationFunc)
    J_plus[i] = computeCost(AL, train_Y, outputActivationFunc=outputActivationFunc) 
    
    
    <span># Compute J_minus[i]. </span>
    thetaminus = np.copy(parameters_values)                                     
    thetaminus[i][<span>0</span>] = thetaminus[i][<span>0</span>] - epsilon                                     
    AL, caches, dropoutMat  = forwardPropagationDeep(train_X, vector_to_dictionary(parameters,thetaminus), keep_prob=<span>1</span>, 
                                              hiddenActivationFunc=<span>&quot;relu&quot;</span>,outputActivationFunc=outputActivationFunc)     
    J_minus[i] = computeCost(AL, train_Y, outputActivationFunc=outputActivationFunc)                            
       
    <span># Compute gradapprox[i] </span>
    gradapprox[i] = (J_plus[i] - J_minus[i])/(<span>2</span>*epsilon)

<span># Compare gradapprox to backward propagation gradients by computing difference. </span>
numerator = np.linalg.norm(grad-gradapprox)                                           
denominator = np.linalg.norm(grad) +  np.linalg.norm(gradapprox)                                         
difference =  numerator/denominator                                        

#Check the difference
<span>if</span> difference > <span>1e-5</span>:
    <span>print</span> (<span>&quot;\033[93m&quot;</span> + <span>&quot;There is a mistake in the backward propagation! difference = &quot;</span> + str(difference) + <span>&quot;\033[0m&quot;</span>)
<span>else</span>:
    <span>print</span> (<span>&quot;\033[92m&quot;</span> + <span>&quot;Your backward propagation works perfectly fine! difference = &quot;</span> + str(difference) + <span>&quot;\033[0m&quot;</span>)
print(difference)
print(<span>&quot;\n&quot;</span>)    
# The technique below can be used to identify 
# which of the parameters are in error
<span># Covert grad to dictionary</span>
m=vector_to_dictionary2(parameters,grad)
print(<span>&quot;Gradients from backprop&quot;</span>)
print(m)
print(<span>&quot;\n&quot;</span>)
<span># Convert gradapprox to dictionary</span>
n=vector_to_dictionary2(parameters,gradapprox)
print(<span>&quot;Gradapprox from gradient check&quot;</span>)
print(n)
</code></pre>
<pre><code>## (300, 2)
## (300,)
## cost= 0.6931455556341791
## [92mYour backward propagation works perfectly fine! difference = 1.1604150683743381e-06[0m
## 1.1604150683743381e-06
## 
## 
## Gradients from backprop
## {'dW1': array([[-6.19439955e-06, -2.06438046e-06],
##        [-1.50165447e-05,  7.50401672e-05],
##        [ 1.33435433e-04,  1.74112143e-04],
##        [-3.40909024e-05, -1.38363681e-04]]), 'db1': array([[ 7.31333221e-07],
##        [ 7.98425950e-06],
##        [ 8.15002817e-08],
##        [-5.69821155e-08]]), 'dW2': array([[2.73416304e-04, 2.96061451e-04, 7.51837363e-05, 1.01257729e-04]]), 'db2': array([[-7.22232235e-06]])}
## 
## 
## Gradapprox from gradient check
## {'dW1': array([[-6.19448937e-06, -2.06501483e-06],
##        [-1.50168766e-05,  7.50399742e-05],
##        [ 1.33435485e-04,  1.74112391e-04],
##        [-3.40910633e-05, -1.38363765e-04]]), 'db1': array([[ 7.31081862e-07],
##        [ 7.98472399e-06],
##        [ 8.16013923e-08],
##        [-5.71764858e-08]]), 'dW2': array([[2.73416290e-04, 2.96061509e-04, 7.51831930e-05, 1.01257891e-04]]), 'db2': array([[-7.22255589e-06]])}</code></pre>
</div>
<div>
<h2>1.1b Gradient Check – Softmax Activation – Python (Error!!)</h2>
<p>In the code below I show, how I managed to spot a bug in your implementation</p>
<pre><code><span>import</span> numpy <span>as</span> np
exec(open(<span>&quot;DLfunctions8.py&quot;</span>).read())
N = <span>100</span> <span># number of points per class</span>
D = <span>2</span> <span># dimensionality</span>
K = <span>3</span> <span># number of classes</span>
X = np.zeros((N*K,D)) <span># data matrix (each row = single example)</span>
y = np.zeros(N*K, dtype=<span>'uint8'</span>) <span># class labels</span>
<span>for</span> j <span>in</span> range(K):
  ix = range(N*j,N*(j+<span>1</span>))
  r = np.linspace(<span>0.0</span>,<span>1</span>,N) <span># radius</span>
  t = np.linspace(j*<span>4</span>,(j+<span>1</span>)*<span>4</span>,N) + np.random.randn(N)*<span>0.2</span> <span># theta</span>
  X[ix] = np.c_[r*np.sin(t), r*np.cos(t)]
  y[ix] = j


<span># Plot the data</span>
<span>#plt.scatter(X[:, 0], X[:, 1], c=y, s=40, cmap=plt.cm.Spectral)</span>
layersDimensions = [<span>2</span>,<span>3</span>,<span>3</span>]
y1=y.reshape(<span>-1</span>,<span>1</span>).T
train_X=X.T
train_Y=y1

parameters = initializeDeepModel(layersDimensions)
#Compute forward prop
AL, caches, dropoutMat = forwardPropagationDeep(train_X, parameters, keep_prob=<span>1</span>, 
                                                hiddenActivationFunc=<span>&quot;relu&quot;</span>,outputActivationFunc=<span>&quot;softmax&quot;</span>)
#Compute cost
cost = computeCost(AL, train_Y, outputActivationFunc=<span>&quot;softmax&quot;</span>) 
print(<span>&quot;cost=&quot;</span>,cost)
#Compute gradients from backprop
gradients = backwardPropagationDeep(AL, train_Y, caches, dropoutMat, lambd=<span>0</span>, keep_prob=<span>1</span>, 
                                    hiddenActivationFunc=<span>&quot;relu&quot;</span>,outputActivationFunc=<span>&quot;softmax&quot;</span>)
<span># Note the transpose of the gradients for Softmax has to be taken</span>
L= len(parameters)//<span>2</span>
print(L)
gradients[<span>'dW'</span>+str(L)]=gradients[<span>'dW'</span>+str(L)].T
gradients[<span>'db'</span>+str(L)]=gradients[<span>'db'</span>+str(L)].T
# Perform gradient check
gradient_check_n(parameters, gradients, train_X, train_Y, epsilon = <span>1e-7</span>,outputActivationFunc=<span>&quot;softmax&quot;</span>)</code></pre>
<pre><code>
cost= 1.0986187818144022
2
There is a mistake in the backward propagation! difference = 0.7100295155692544
0.7100295155692544


Gradients from backprop
{'dW1': array([[ 0.00050125,  0.00045194],
       [ 0.00096392,  0.00039641],
       [-0.00014276, -0.00045639]]), 'db1': array([[ 0.00070082],
       [-0.00224399],
       [ 0.00052305]]), 'dW2': array([[-8.40953794e-05, -9.52657769e-04, -1.10269379e-04],
       [-7.45469382e-04,  9.49795606e-04,  2.29045434e-04],
       [ 8.29564761e-04,  2.86216305e-06, -1.18776055e-04]]), 
     'db2': array([[<strong>-0.00253808</strong>],
       [<strong>-0.00505508</strong>],
       [ <strong>0.00759315</strong>]])}


Gradapprox from gradient check
{'dW1': array([[ 0.00050125,  0.00045194],
       [ 0.00096392,  0.00039641],
       [-0.00014276, -0.00045639]]), 'db1': array([[ 0.00070082],
       [-0.00224399],
       [ 0.00052305]]), 'dW2': array([[-8.40960634e-05, -9.52657953e-04, -1.10268461e-04],
       [-7.45469242e-04,  9.49796908e-04,  2.29045671e-04],
       [ 8.29565305e-04,  2.86104473e-06, -1.18776100e-04]]), 
     'db2': array([[-8.46211989e-06],
       [-1.68487446e-05],
       [ 2.53108645e-05]])}</code></pre>
<p>Gradient Check gives a high value of the difference of 0.7100295. Inspecting the Gradients and Gradapprox we can see there is a very big discrepancy in db2. After I went over my code I discovered that I my computation in the function layerActivationBackward for Softmax was</p>
<pre> 
   # Erroneous code
   if activationFunc == 'softmax':
        dW = 1/numtraining * np.dot(A_prev,dZ)
        db = np.sum(dZ, axis=0, keepdims=True)
        dA_prev = np.dot(dZ,W)
instead of
   # Fixed code
   if activationFunc == 'softmax':
        dW = 1/numtraining * np.dot(A_prev,dZ)
        db = <strong>1/numtraining</strong> *  np.sum(dZ, axis=0, keepdims=True)
        dA_prev = np.dot(dZ,W)
</pre>
<p>After fixing this error when I ran Gradient Check I get</p>
</div>
<div>
<h2>1.1c Gradient Check – Softmax Activation – Python (Corrected!!)</h2>
<pre><code><span>import</span> numpy <span>as</span> np
exec(open(<span>&quot;DLfunctions8.py&quot;</span>).read())
N = <span>100</span> <span># number of points per class</span>
D = <span>2</span> <span># dimensionality</span>
K = <span>3</span> <span># number of classes</span>
X = np.zeros((N*K,D)) <span># data matrix (each row = single example)</span>
y = np.zeros(N*K, dtype=<span>'uint8'</span>) <span># class labels</span>
<span>for</span> j <span>in</span> range(K):
  ix = range(N*j,N*(j+<span>1</span>))
  r = np.linspace(<span>0.0</span>,<span>1</span>,N) <span># radius</span>
  t = np.linspace(j*<span>4</span>,(j+<span>1</span>)*<span>4</span>,N) + np.random.randn(N)*<span>0.2</span> <span># theta</span>
  X[ix] = np.c_[r*np.sin(t), r*np.cos(t)]
  y[ix] = j


<span># Plot the data</span>
<span>#plt.scatter(X[:, 0], X[:, 1], c=y, s=40, cmap=plt.cm.Spectral)</span>
layersDimensions = [<span>2</span>,<span>3</span>,<span>3</span>]
y1=y.reshape(<span>-1</span>,<span>1</span>).T
train_X=X.T
train_Y=y1
#Set layer dimensions
parameters = initializeDeepModel(layersDimensions)
#Perform forward prop
AL, caches, dropoutMat = forwardPropagationDeep(train_X, parameters, keep_prob=<span>1</span>, 
                                                hiddenActivationFunc=<span>&quot;relu&quot;</span>,outputActivationFunc=<span>&quot;softmax&quot;</span>)
#Compute cost
cost = computeCost(AL, train_Y, outputActivationFunc=<span>&quot;softmax&quot;</span>) 
print(<span>&quot;cost=&quot;</span>,cost)
#Compute gradients from backprop
gradients = backwardPropagationDeep(AL, train_Y, caches, dropoutMat, lambd=<span>0</span>, keep_prob=<span>1</span>, 
                                    hiddenActivationFunc=<span>&quot;relu&quot;</span>,outputActivationFunc=<span>&quot;softmax&quot;</span>)
<span># Note the transpose of the gradients for Softmax has to be taken</span>
L= len(parameters)//<span>2</span>
print(L)
gradients[<span>'dW'</span>+str(L)]=gradients[<span>'dW'</span>+str(L)].T
gradients[<span>'db'</span>+str(L)]=gradients[<span>'db'</span>+str(L)].T
#Perform gradient check
gradient_check_n(parameters, gradients, train_X, train_Y, epsilon = <span>1e-7</span>,outputActivationFunc=<span>&quot;softmax&quot;</span>)</code></pre>
<pre><code>## cost= 1.0986193170234435
## 2
## [92mYour backward propagation works perfectly fine! difference = 5.268804859613151e-07[0m
## 5.268804859613151e-07
## 
## 
## Gradients from backprop
## {'dW1': array([[ 0.00053206,  0.00038987],
##        [ 0.00093941,  0.00038077],
##        [-0.00012177, -0.0004692 ]]), 'db1': array([[ 0.00072662],
##        [-0.00210198],
##        [ 0.00046741]]), 'dW2': array([[-7.83441270e-05, -9.70179498e-04, -1.08715815e-04],
##        [-7.70175008e-04,  9.54478237e-04,  2.27690198e-04],
##        [ 8.48519135e-04,  1.57012608e-05, -1.18974383e-04]]), 'db2': array([[-8.52190476e-06],
##        [-1.69954294e-05],
##        [ 2.55173342e-05]])}
## 
## 
## Gradapprox from gradient check
## {'dW1': array([[ 0.00053206,  0.00038987],
##        [ 0.00093941,  0.00038077],
##        [-0.00012177, -0.0004692 ]]), 'db1': array([[ 0.00072662],
##        [-0.00210198],
##        [ 0.00046741]]), 'dW2': array([[-7.83439980e-05, -9.70180603e-04, -1.08716369e-04],
##        [-7.70173925e-04,  9.54478718e-04,  2.27690089e-04],
##        [ 8.48520143e-04,  1.57018842e-05, -1.18973720e-04]]), 'db2': array([[-8.52096171e-06],
##        [-1.69964043e-05],
##        [ 2.55162558e-05]])}</code></pre>
</div>
<div>
<h2>1.2a Gradient Check – Sigmoid Activation – R</h2>
<pre><code><span>source</span>(<span>&quot;DLfunctions8.R&quot;</span>)</code><code>
z &lt;- as.matrix(read.csv(<span>&quot;circles.csv&quot;</span>,header=<span>FALSE</span>)) 

x &lt;- z[,<span>1</span>:<span>2</span>]
y &lt;- z[,<span>3</span>]
X &lt;- t(x)
Y &lt;- t(y)
#Set layer dimensions
layersDimensions = c(<span>2</span>,<span>5</span>,<span>1</span>)
parameters = initializeDeepModel(layersDimensions)
#Perform forward prop
retvals = forwardPropagationDeep(X, parameters,keep_prob=<span>1</span>, hiddenActivationFunc=<span>&quot;relu&quot;</span>,
                                 outputActivationFunc=<span>&quot;sigmoid&quot;</span>)
AL &lt;- retvals[[<span>'AL'</span>]]
caches &lt;- retvals[[<span>'caches'</span>]]
dropoutMat &lt;- retvals[[<span>'dropoutMat'</span>]]
#Compute cost
cost &lt;- computeCost(AL, Y,outputActivationFunc=<span>&quot;sigmoid&quot;</span>,
                    numClasses=layersDimensions[length(layersDimensions)])
print(cost)</code></pre>
<pre><code>## [1] 0.6931447</code></pre>
<pre><code><span># Backward propagation.</span>
gradients = backwardPropagationDeep(AL, Y, caches, dropoutMat, lambd=<span>0</span>, keep_prob=<span>1</span>, hiddenActivationFunc=<span>&quot;relu&quot;</span>,
                                    outputActivationFunc=<span>&quot;sigmoid&quot;</span>,numClasses=layersDimensions[length(layersDimensions)])</code></pre>
<pre><code>epsilon = <span>1e-07</span>
outputActivationFunc=<span>&quot;sigmoid&quot;
</span>#Convert parameter list to vector
parameters_values = list_to_vector(parameters)
#Convert gradient list to vector
grad = gradients_to_vector(parameters,gradients)
num_parameters = dim(parameters_values)[<span>1</span>]
#Initialize
J_plus = matrix(rep(<span>0</span>,num_parameters),
                nrow=num_parameters,ncol=<span>1</span>)
J_minus = matrix(rep(<span>0</span>,num_parameters),
                 nrow=num_parameters,ncol=<span>1</span>)
gradapprox = matrix(rep(<span>0</span>,num_parameters),
                    nrow=num_parameters,ncol=<span>1</span>)

<span># Compute gradapprox</span>
<span>for</span>(i <span>in</span> <span>1</span>:num_parameters){
    <span># Compute J_plus[i]. </span>
    thetaplus = parameters_values                                   
    thetaplus[i][<span>1</span>] = thetaplus[i][<span>1</span>] + epsilon                                 
    retvals = forwardPropagationDeep(X, vector_to_list(parameters,thetaplus), keep_prob=<span>1</span>, 
                                                hiddenActivationFunc=<span>&quot;relu&quot;</span>,outputActivationFunc=outputActivationFunc)
    
    AL &lt;- retvals[[<span>'AL'</span>]]
    J_plus[i] = computeCost(AL, Y, outputActivationFunc=outputActivationFunc) 


   <span># Compute J_minus[i]. </span>
    thetaminus = parameters_values                                     
    thetaminus[i][<span>1</span>] = thetaminus[i][<span>1</span>] - epsilon                                     
    retvals  = forwardPropagationDeep(X, vector_to_list(parameters,thetaminus), keep_prob=<span>1</span>, 
                                                 hiddenActivationFunc=<span>&quot;relu&quot;</span>,outputActivationFunc=outputActivationFunc)     
    AL &lt;- retvals[[<span>'AL'</span>]]
    J_minus[i] = computeCost(AL, Y, outputActivationFunc=outputActivationFunc)                            

    <span># Compute gradapprox[i] </span>
    gradapprox[i] = (J_plus[i] - J_minus[i])/(<span>2</span>*epsilon)
}
<span># Compare gradapprox to backward propagation gradients by computing difference.</span>
#Compute L2Norm
numerator = L2NormVec(grad-gradapprox)                                           
denominator = L2NormVec(grad) +  L2NormVec(gradapprox)                                         
difference =  numerator/denominator 
<span>if</span>(difference > <span>1e-5</span>){
    cat(<span>&quot;There is a mistake, the difference is too high&quot;</span>,difference)
} <span>else</span>{
    cat(<span>&quot;The implementations works perfectly&quot;</span>, difference)
}</code></pre>
<pre><code>## The implementations works perfectly 1.279911e-06</code></pre>
<pre><code><span># This can be used to check</span>
print(<span>&quot;Gradients from backprop&quot;</span>)</code></pre>
<pre><code>## [1] &quot;Gradients from backprop&quot;</code></pre>
<pre><code>vector_to_list2(parameters,grad)</code></pre>
<pre><code>## $dW1
##               [,1]          [,2]
## [1,] -7.641588e-05 -3.427989e-07
## [2,] -9.049683e-06  6.906304e-05
## [3,]  3.401039e-06 -1.503914e-04
## [4,]  1.535226e-04 -1.686402e-04
## [5,] -6.029292e-05 -2.715648e-04
## 
## $db1
##               [,1]
## [1,]  6.930318e-06
## [2,] -3.283117e-05
## [3,]  1.310647e-05
## [4,] -3.454308e-05
## [5,] -2.331729e-08
## 
## $dW2
##              [,1]         [,2]         [,3]        [,4]         [,5]
## [1,] 0.0001612356 0.0001113475 0.0002435824 0.000362149 2.874116e-05
## 
## $db2
##              [,1]
## [1,] -1.16364e-05</code></pre>
<pre><code>print(<span>&quot;Grad approx from gradient check&quot;</span>)</code></pre>
<pre><code>## [1] &quot;Grad approx from gradient check&quot;</code></pre>
<pre><code>vector_to_list2(parameters,gradapprox)</code></pre>
<pre><code>## $dW1
##               [,1]          [,2]
## [1,] -7.641554e-05 -3.430589e-07
## [2,] -9.049428e-06  6.906253e-05
## [3,]  3.401168e-06 -1.503919e-04
## [4,]  1.535228e-04 -1.686401e-04
## [5,] -6.029288e-05 -2.715650e-04
## 
## $db1
##               [,1]
## [1,]  6.930012e-06
## [2,] -3.283096e-05
## [3,]  1.310618e-05
## [4,] -3.454237e-05
## [5,] -2.275957e-08
## 
## $dW2
##              [,1]         [,2]         [,3]         [,4]        [,5]
## [1,] 0.0001612355 0.0001113476 0.0002435829 0.0003621486 2.87409e-05
## 
## $db2
##              [,1]
## [1,] -1.16368e-05</code></pre>
</div>
<div>
<h2>1.2b Gradient Check – Softmax Activation – R</h2>
<pre><code><span>source</span>(<span>&quot;DLfunctions8.R&quot;</span>)
Z &lt;- as.matrix(read.csv(<span>&quot;spiral.csv&quot;</span>,header=<span>FALSE</span>)) 

<span># Setup the data</span>
X &lt;- Z[,<span>1</span>:<span>2</span>]
y &lt;- Z[,<span>3</span>]
X &lt;- t(X)
Y &lt;- t(y)
layersDimensions = c(<span>2</span>, <span>3</span>, <span>3</span>)
parameters = initializeDeepModel(layersDimensions)
#Perform forward prop
retvals = forwardPropagationDeep(X, parameters,keep_prob=<span>1</span>, hiddenActivationFunc=<span>&quot;relu&quot;</span>,
                                 outputActivationFunc=<span>&quot;softmax&quot;</span>)
AL &lt;- retvals[[<span>'AL'</span>]]
caches &lt;- retvals[[<span>'caches'</span>]]
dropoutMat &lt;- retvals[[<span>'dropoutMat'</span>]]
#Compute cost
cost &lt;- computeCost(AL, Y,outputActivationFunc=<span>&quot;softmax&quot;</span>,
                    numClasses=layersDimensions[length(layersDimensions)])
print(cost)</code></pre>
<pre><code>## [1] 1.098618</code></pre>
<pre><code><span># Backward propagation.</span>
gradients = backwardPropagationDeep(AL, Y, caches, dropoutMat, lambd=<span>0</span>, keep_prob=<span>1</span>, hiddenActivationFunc=<span>&quot;relu&quot;</span>,
                                    outputActivationFunc=<span>&quot;softmax&quot;</span>,numClasses=layersDimensions[length(layersDimensions)])</code></pre>
<pre><code><span># Need to take transpose of the last layer for Softmax</span>
L=length(parameters)/<span>2</span>
gradients[[paste(<span>'dW'</span>,L,sep=<span>&quot;&quot;</span>)]]=t(gradients[[paste(<span>'dW'</span>,L,sep=<span>&quot;&quot;</span>)]])
gradients[[paste(<span>'db'</span>,L,sep=<span>&quot;&quot;</span>)]]=t(gradients[[paste(<span>'db'</span>,L,sep=<span>&quot;&quot;</span>)]])
#Perform gradient check
gradient_check_n(parameters, gradients, X, Y, 
                 epsilon = <span>1e-7</span>,outputActivationFunc=<span>&quot;softmax&quot;</span>)</code></pre>
<pre><code>## The implementations works perfectly 3.903011e-07[1] &quot;Gradients from backprop&quot;
## $dW1
##              [,1]          [,2]
## [1,] 0.0007962367 -0.0001907606
## [2,] 0.0004444254  0.0010354412
## [3,] 0.0003078611  0.0007591255
## 
## $db1
##               [,1]
## [1,] -0.0017305136
## [2,]  0.0005393734
## [3,]  0.0012484550
## 
## $dW2
##               [,1]          [,2]          [,3]
## [1,] -3.515627e-04  7.487283e-04 -3.971656e-04
## [2,] -6.381521e-05 -1.257328e-06  6.507254e-05
## [3,] -1.719479e-04 -4.857264e-04  6.576743e-04
## 
## $db2
##               [,1]
## [1,] -5.536383e-06
## [2,] -1.824656e-05
## [3,]  2.378295e-05
## 
## [1] &quot;Grad approx from gradient check&quot;
## $dW1
##              [,1]          [,2]
## [1,] 0.0007962364 -0.0001907607
## [2,] 0.0004444256  0.0010354406
## [3,] 0.0003078615  0.0007591250
## 
## $db1
##               [,1]
## [1,] -0.0017305135
## [2,]  0.0005393741
## [3,]  0.0012484547
## 
## $dW2
##               [,1]          [,2]          [,3]
## [1,] -3.515632e-04  7.487277e-04 -3.971656e-04
## [2,] -6.381451e-05 -1.257883e-06  6.507239e-05
## [3,] -1.719469e-04 -4.857270e-04  6.576739e-04
## 
## $db2
##               [,1]
## [1,] -5.536682e-06
## [2,] -1.824652e-05
## [3,]  2.378209e-05</code></pre>
</div>
<div>
<h2>1.3a Gradient Check – Sigmoid Activation – Octave</h2>
<pre>source(&quot;DL8functions.m&quot;)
################## Circles
data=csvread(&quot;circles.csv&quot;);

X=data(:,1:2);
Y=data(:,3);
#Set layer dimensions
layersDimensions = [2 5  1]; #tanh=-0.5(ok), #relu=0.1 best!
[weights biases] = initializeDeepModel(layersDimensions);
#Perform forward prop
[AL forward_caches activation_caches droputMat] = forwardPropagationDeep(X', weights, biases,keep_prob=1, 
                 hiddenActivationFunc=&quot;relu&quot;, outputActivationFunc=&quot;sigmoid&quot;);
#Compute cost
cost = computeCost(AL, Y',outputActivationFunc=outputActivationFunc,numClasses=layersDimensions(size(layersDimensions)(2)));
disp(cost);
#Compute gradients from cost
[gradsDA gradsDW gradsDB] = backwardPropagationDeep(AL, Y', activation_caches,forward_caches, droputMat, lambd=0, keep_prob=1, 
                                 hiddenActivationFunc=&quot;relu&quot;, outputActivationFunc=&quot;sigmoid&quot;,
                                 numClasses=layersDimensions(size(layersDimensions)(2)));
epsilon = 1e-07;
outputActivationFunc=&quot;sigmoid&quot;;
# Convert paramter cell array to vector
parameters_values = cellArray_to_vector(weights, biases);
#Convert gradient cell array to vector
grad = gradients_to_vector(gradsDW,gradsDB);
num_parameters = size(parameters_values)(1);
#Initialize
J_plus = zeros(num_parameters, 1);
J_minus = zeros(num_parameters, 1);
gradapprox = zeros(num_parameters, 1);
# Compute gradapprox
for i = 1:num_parameters
    # Compute J_plus[i]. 
    thetaplus = parameters_values;                                   
    thetaplus(i,1) = thetaplus(i,1) + epsilon;      
    [weights1 biases1] =vector_to_cellArray(weights, biases,thetaplus);    
    [AL forward_caches activation_caches droputMat] = forwardPropagationDeep(X', weights1, biases1, keep_prob=1, 
                                              hiddenActivationFunc=&quot;relu&quot;,outputActivationFunc=outputActivationFunc);
    J_plus(i) = computeCost(AL, Y', outputActivationFunc=outputActivationFunc); 
        
    # Compute J_minus[i]. 
    thetaminus = parameters_values;                                  
    thetaminus(i,1) = thetaminus(i,1) - epsilon ;    
    [weights1 biases1] = vector_to_cellArray(weights, biases,thetaminus);    
    [AL forward_caches activation_caches droputMat]  = forwardPropagationDeep(X',weights1, biases1, keep_prob=1, 
                                              hiddenActivationFunc=&quot;relu&quot;,outputActivationFunc=outputActivationFunc);     
    J_minus(i) = computeCost(AL, Y', outputActivationFunc=outputActivationFunc);                            
      
    # Compute gradapprox[i]   
    gradapprox(i) = (J_plus(i) - J_minus(i))/(2*epsilon);

endfor

#Compute L2Norm
numerator = L2NormVec(grad-gradapprox);                                           
denominator = L2NormVec(grad) +  L2NormVec(gradapprox);                                         
difference =  numerator/denominator;
disp(difference);
#Check difference
if difference > 1e-04
   printf(&quot;There is a mistake in the implementation &quot;);
   disp(difference);
else
   printf(&quot;The implementation works perfectly&quot;);
      disp(difference);
endif
[weights1 biases1] = vector_to_cellArray(weights, biases,grad);  
printf(&quot;Gradients from back propagation&quot;); 
disp(weights1);
disp(biases1); 
[weights2 biases2] = vector_to_cellArray(weights, biases,gradapprox); 
printf(&quot;Gradients from gradient check&quot;);
disp(weights2);
disp(biases2); 
</pre>
<pre>0.69315
1.4893e-005
The implementation works perfectly 1.4893e-005
Gradients from back propagation
{
[1,1] =
5.0349e-005 2.1323e-005
8.8632e-007 1.8231e-006
9.3784e-005 1.0057e-004
1.0875e-004 -1.9529e-007
5.4502e-005 3.2721e-005
[1,2] =
1.0567e-005 6.0615e-005 4.6004e-005 1.3977e-004 1.0405e-004
}
{
[1,1] =
-1.8716e-005
1.1309e-009
4.7686e-005
1.2051e-005
-1.4612e-005
[1,2] = 9.5808e-006
}
Gradients from gradient check
{
[1,1] =
5.0348e-005 2.1320e-005
8.8485e-007 1.8219e-006
9.3784e-005 1.0057e-004
1.0875e-004 -1.9762e-007
5.4502e-005 3.2723e-005
[1,2] =
[1,2] =
1.0565e-005 6.0614e-005 4.6007e-005 1.3977e-004 1.0405e-004
}
{
[1,1] =
-1.8713e-005
1.1102e-009
4.7687e-005
1.2048e-005
-1.4609e-005
[1,2] = 9.5790e-006
}
</pre>
</div>
<div>
<h2>1.3b Gradient Check – Softmax Activation – Octave</h2>
<pre>source(&quot;DL8functions.m&quot;)
data=csvread(&quot;spiral.csv&quot;);

# Setup the data
X=data(:,1:2);
Y=data(:,3);
# Set the layer dimensions
layersDimensions = [2 3  3]; 
[weights biases] = initializeDeepModel(layersDimensions);
# Run forward prop
[AL forward_caches activation_caches droputMat] = forwardPropagationDeep(X', weights, biases,keep_prob=1, 
                 hiddenActivationFunc=&quot;relu&quot;, outputActivationFunc=&quot;softmax&quot;);
# Compute cost
cost = computeCost(AL, Y',outputActivationFunc=outputActivationFunc,numClasses=layersDimensions(size(layersDimensions)(2)));
disp(cost);
# Perform backward prop
[gradsDA gradsDW gradsDB] = backwardPropagationDeep(AL, Y', activation_caches,forward_caches, droputMat, lambd=0, keep_prob=1, 
                                 hiddenActivationFunc=&quot;relu&quot;, outputActivationFunc=&quot;softmax&quot;,
                                 numClasses=layersDimensions(size(layersDimensions)(2)));

#Take transpose of last layer for Softmax                                
L=size(weights)(2);
gradsDW{L}= gradsDW{L}';
gradsDB{L}= gradsDB{L}';   
#Perform gradient check
difference= gradient_check_n(weights, biases, gradsDW,gradsDB, X, Y, epsilon = 1e-7,
                  outputActivationFunc=&quot;softmax&quot;,numClasses=layersDimensions(size(layersDimensions)(2)));
</pre>
<pre> 1.0986
The implementation works perfectly  2.0021e-005
Gradients from back propagation
{
  [1,1] =
    -7.1590e-005  4.1375e-005
    -1.9494e-004  -5.2014e-005
    -1.4554e-004  5.1699e-005
  [1,2] =
    3.3129e-004  1.9806e-004  -1.5662e-005
    -4.9692e-004  -3.7756e-004  -8.2318e-005
    1.6562e-004  1.7950e-004  9.7980e-005
}
{
  [1,1] =
    -3.0856e-005
    -3.3321e-004
    -3.8197e-004
  [1,2] =
    1.2046e-006
    2.9259e-007
    -1.4972e-006
}
Gradients from gradient check
{
  [1,1] =
    -7.1586e-005  4.1377e-005
    -1.9494e-004  -5.2013e-005
    -1.4554e-004  5.1695e-005
    3.3129e-004  1.9806e-004  -1.5664e-005
    -4.9692e-004  -3.7756e-004  -8.2316e-005
    1.6562e-004  1.7950e-004  9.7979e-005
}
{
  [1,1] =
    -3.0852e-005
    -3.3321e-004
    -3.8197e-004
  [1,2] =
    1.1902e-006
    2.8200e-007
    -1.4644e-006
}
</pre>
</div>
<div>
<h2>2.1 Tip for tuning hyperparameters</h2>
<p>Deep Learning Networks come with a large number of hyper parameters which require tuning. The hyper parameters are</p>
<p>1. <img alt="alpha" title="alpha" src="https://s0.wp.com/latex.php?latex=%5Calpha&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> -learning rate<br>
2. Number of layers<br>
3. Number of hidden units<br>
4. Number of iterations<br>
5. Momentum – <img alt="beta" title="beta" src="https://s0.wp.com/latex.php?latex=%5Cbeta&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> – 0.9<br>
6. RMSProp – <img alt="beta_{1}" title="beta_{1}" src="https://s0.wp.com/latex.php?latex=%5Cbeta_%7B1%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> – 0.9<br>
7. Adam – <img alt="beta_{1}" title="beta_{1}" src="https://s0.wp.com/latex.php?latex=%5Cbeta_%7B1%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex">,<img alt="beta_{2}" title="beta_{2}" src="https://s0.wp.com/latex.php?latex=%5Cbeta_%7B2%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> and <img alt="epsilon" title="epsilon" src="https://s0.wp.com/latex.php?latex=%5Cepsilon&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
8. learning rate decay<br>
9. mini batch size<br>
10. Initialization method – He, Xavier<br>
11. Regularization</p>
<p>– Among the above the most critical is learning rate decay. Rather than just trying out random values, it may help to try out values on a logarithmic scale. So we could try out<br>
values -0.01,0.1,1.0,10 etc. If we find that the cost is between 0.01 and 0.1 we could use a technique similar to binary search, so we can try 0.01, 0.05. If we need to be bigger than 0.01 and 0.05 we could try 0.25  and then keep halving the distance etc.<br>
– The performance of Momentum and RMSProp are very good and work well with values 0.9. Even with this, it is better to try out values of 1-<img alt="beta" title="beta" src="https://s0.wp.com/latex.php?latex=%5Cbeta&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> in the logarithmic range. So 1-<img alt="beta" title="beta" src="https://s0.wp.com/latex.php?latex=%5Cbeta&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> could 0.001,0.01,0.1 and hence <img alt="beta" title="beta" src="https://s0.wp.com/latex.php?latex=%5Cbeta&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> would be 0.999,0.99 or 0.9<br>
– Increasing the number of hidden units or number of hidden layers need to be done gradually. I have noticed that increasing number of hidden layers heavily does not improve performance and sometimes degrades it.<br>
– Sometimes, I tend to increase the number of iterations if I think I see a steady decrease in the cost for a certain learning rate<br>
– It may also help to add learning rate decay if you see there is an oscillation while it decreases.<br>
– Xavier and He initializations also help in a fast convergence and are worth trying out.</p>
</div>
<div>
<h2>3.1 Final thoughts</h2>
<p>As I come to a close in this Deep Learning Series from first principles in Python, R and Octave, I must admit that I learnt a lot in the process.</p>
<p>* Building a L-layer, vectorized Deep Learning Network in Python, R and Octave was extremely challenging but very rewarding<br>
* One benefit of building vectorized versions in Python, R and Octave was that I was looking at each function that I was implementing thrice, and hence I was able to fix any bugs in any of the languages<br>
* In addition since I built the generic L-Layer DL network with all the bells and whistles, layer by layer I further had an opportunity to look at all the functions in each successive post.<br>
* Each language has its advantages and disadvantages. From the performance perspective I think Python is the best, followed by Octave and then R<br>
* Interesting, I noticed that even if small bugs creep into your implementation, the DL network does learn and does generate a valid set of weights and biases, however this may not be an optimum solution. In one case of an inadvertent bug, I was not updating the weights in the final layer of the DL network. Yet, using all the other layers, the DL network was able to come with a reasonable solution (maybe like random dropout, remaining units can still learn the data!)<br>
* Having said that, the Gradient Check method discussed and implemented in this post can be very useful in ironing out bugs.<br>
Feel free to clone/download the code from Github at <a rel="nofollow" href="https://github.com/tvganesh/DeepLearning-Part8" target="_blank">DeepLearning-Part8</a></p>
</div>
<div>
<h2>Conclusion</h2>
<p>These last couple of months when I was writing the posts and the also churning up the code in Python, R and Octave were  very hectic. There have been times when I found that implementations of some function to be extremely demanding and I almost felt like giving up. Other times, I have spent quite some time on an intractable DL network which would not respond to changes in hyper-parameters. All in all, it was a great learning experience. I would suggest that you start from my first post <a>Deep Learning from first principles in Python, R and Octave-Part 1</a> and work your way up. Feel free to take the code apart and try out things. That is the only way you will learn.</p>
<p>Hope you had as much fun as I had. Stay tuned. I will be back!!!</p>
<p>Also see<br>
1. <a rel="nofollow" href="https://gigadom.wordpress.com/2017/12/05/my-book-practical-machine-learning-with-r-and-python-on-amazon/" target="_blank">My book ‘Practical Machine Learning with R and Python’ on Amazon</a><br>
2. <a rel="nofollow" href="https://gigadom.wordpress.com/2015/10/15/revisiting-crimes-against-women-in-india/" target="_blank">Revisiting crimes against women in India</a><br>
3. <a rel="nofollow" href="https://gigadom.wordpress.com/2015/11/06/literacy-in-india-a-deepr-dive-2/" target="_blank">Literacy in India – A deepR dive</a><br>
4. <a rel="nofollow" href="https://gigadom.wordpress.com/2015/11/29/sixer-r-package-cricketrs-new-avatar/" target="_blank">Sixer – R package cricketr’s new Shiny avatar</a><br>
5. <a rel="nofollow" href="https://gigadom.wordpress.com/2014/11/05/bend-it-like-bluemix-mongodb-using-auto-scale-part-1/" target="_blank">Bend it like Bluemix, MongoDB using Auto-scale – Part 1!</a><br>
6. <a rel="nofollow" href="https://gigadom.wordpress.com/2011/10/31/opencv-ramblings-on-derivatives-histograms-and-contours/" target="_blank">Computer Vision: Ramblings on derivatives, histograms and contours</a><br>
7. <a rel="nofollow" href="https://gigadom.wordpress.com/2016/06/23/introducing-qcsimulator-a-5-qubit-quantum-computing-simulator-in-r/" target="_blank">Introducing QCSimulator: A 5-qubit quantum computing simulator in R</a><br>
8. <a rel="nofollow" href="https://gigadom.wordpress.com/2013/07/08/a-closer-look-at-robot-horse-on-a-canter-in-android/" target="_blank">A closer look at “Robot Horse on a Trot” in Android</a></p>
<p>To see all post click <a rel="nofollow" href="https://gigadom.wordpress.com/aa-2/" target="_blank">Index of Posts</a></p>
</div>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://gigadom.wordpress.com/2018/05/06/deep-learning-from-first-principles-in-python-r-and-octave-part-8/"> R – Giga thoughts …</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/9bMQYLkDKFo" height="1"></a></div>