<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/deep-learning-from-first-principles-in-python-r-and-octave-part-3/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://gigadom.wordpress.com/2018/01/30/deep-learning-from-first-principles-in-python-r-and-octave-part-3/"> R – Giga thoughts …</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><em>“Once upon a time, I, Chuang Tzu, dreamt I was a butterfly, fluttering hither and thither, to all intents and purposes a butterfly. I was conscious only of following my fancies as a butterfly, and was unconscious of my individuality as a man. Suddenly, I awoke, and there I lay, myself again. Now I do not know whether I was then a man dreaming I was a butterfly, or whether I am now a butterfly dreaming that I am a man.”</em><br>
from The Brain: The Story of you – David Eagleman</p>
<p><em>“Thought is a great big vector of neural activity”</em><br>
Prof Geoffrey Hinton</p>
<h1>Introduction</h1>
<p>This is the third part in my series on Deep Learning from first principles in Python, R and Octave. In the first part <a rel="nofollow" href="https://gigadom.wordpress.com/2017/10/06/practical-machine-learning-with-r-and-python-part-1/" target="_blank">Deep Learning from first principles in Python, R and Octave-Part 1</a>, I implemented logistic regression as a 2 layer neural network. The 2nd part <a rel="nofollow" href="https://gigadom.wordpress.com/2018/01/11/deep-learning-from-first-principles-in-python-r-and-octave-part-2/" target="_blank">Deep Learning from first principles in Python, R and Octave-Part 2</a>, dealt with the implementation of 3 layer Neural Networks with 1 hidden layer to perform classification tasks, where the 2 classes cannot be separated by a linear boundary. In this third part, I implement a multi-layer, Deep Learning (DL) network of arbitrary depth (any number of hidden layers) and arbitrary height (any number of activation units in each hidden layer). The implementations of these Deep Learning networks, in all the 3 parts, are based on vectorized versions in Python, R and Octave. The implementation in the 3rd part is for a L-layer Deep Netwwork, but without any regularization, early stopping, momentum or learning rate adaptation techniques. However even the barebones multi-layer DL, is a handful and has enough hyperparameters to fine-tune and adjust.</p>
<p>The implementation of the vectorized L-layer Deep Learning network in Python, R and Octave were both exhausting, and exacting!! Keeping track of the indices, layer number and matrix dimensions required quite bit of focus. While the implementation was demanding, it was also very exciting to get the code to work. The trick was to be able to shift gears between the slight quirkiness between the languages. Here are some of challenges I faced</p>
<p>1. Python and Octave allow multiple return values to be unpacked in a single statement. With R, unpacking multiple return values from a list, requires the list returned, to be unpacked separately. I did see that there is a package <a rel="nofollow" href="https://stackoverflow.com/questions/1826519/how-to-assign-from-a-function-which-returns-more-than-one-value" target="_blank">gsubfn</a>, which does this.  I hope this feature becomes a base R feature.<br>
2. Python and R allow dissimilar elements to be saved and returned from functions using dictionaries or lists respectively. However there is no real equivalent in Octave. The closest I got to this functionality in Octave, was the ‘cell array’. But the cell array can be accessed only by the index, and not with the key as in a Python dictionary or R list. This makes things just a bit more difficult in Octave.<br>
3. Python and Octave include implicit broadcasting. In R, broadcasting is not implicit, but R has a nifty function, the sweep(), with which we can broadcast either by columns or by rows<br>
4. The closest equivalent of Python’s dictionary, or R’s list, in Octave is the cell array. However I had to manage separate cell arrays for weights and biases and during gradient descent and separate gradients dW and dB<br>
5. In Python the rank-1 numpy arrays can be annoying at times. This issue is not present in R and Octave.</p>
<p>Though the number of lines of code for Deep Learning functions in Python, R and Octave are about ~350 apiece, they have been some of the most difficult code I have implemented. The current vectorized implementation supports the relu, sigmoid and tanh activation functions as of now. I will be adding other activation functions like the ‘leaky relu’, ‘softmax’ and others, to the implementation in the weeks to come.</p>
<p>While testing with different hyper-parameters namely i) the number of hidden layers, ii) the number of activation units in each layer, iii) the activation function and iv) the number iterations, I found the L-layer Deep Learning Network to be very sensitive to these hyper-parameters. It is not easy to tune the parameters. Adding more hidden layers, or more units per layer, does not help and mostly results in gradient descent getting stuck in some local minima. It does take a fair amount of trial and error and very close observation on how the DL network performs for logical changes. We then can zero in on the most the optimal solution. Feel free to download/fork my code from Github <a rel="nofollow" href="https://github.com/tvganesh/DeepLearning-Part3" target="_blank">DeepLearning-Part 3</a> and play around with the hyper-parameters for your own problems.</p>
<h4>Derivation of a Multi Layer Deep Learning Network</h4>
<p>Lets take a simple 3 layer Neural network with 3 hidden layers and an output layer<br>
<img data-orig-file="https://gigadom.files.wordpress.com/2018/01/1.png" data-orig-size="1184,343" data-medium-file="https://gigadom.files.wordpress.com/2018/01/1.png?w=300" data-attachment-id="6287" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/1.png?w=450&h=196&fit=676%2C196" data-permalink="https://gigadom.wordpress.com/2018/01/30/deep-learning-from-first-principles-in-python-r-and-octave-part-3/1-54/" data-large-file="https://gigadom.files.wordpress.com/2018/01/1.png?w=450&amp;h=196&amp;fit=676%2C196" alt="" data-image-title="1" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6287" data-recalc-dims="1"><br>
In the forward propagation cycle the equations are</p>
<p><img alt="Z_{1} = W_{1}A_{0} +b_{1}" title="Z_{1} = W_{1}A_{0} +b_{1}" src="https://s0.wp.com/latex.php?latex=Z_%7B1%7D+%3D+W_%7B1%7DA_%7B0%7D+%2Bb_%7B1%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex">  and  <img alt="A_{1} = g(Z_{1})" title="A_{1} = g(Z_{1})" src="https://s0.wp.com/latex.php?latex=A_%7B1%7D+%3D+g%28Z_%7B1%7D%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="Z_{2} = W_{2}A_{1} +b_{2}" title="Z_{2} = W_{2}A_{1} +b_{2}" src="https://s0.wp.com/latex.php?latex=Z_%7B2%7D+%3D+W_%7B2%7DA_%7B1%7D+%2Bb_%7B2%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex">  and  <img alt="A_{2} = g(Z_{2})" title="A_{2} = g(Z_{2})" src="https://s0.wp.com/latex.php?latex=A_%7B2%7D+%3D+g%28Z_%7B2%7D%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="Z_{3} = W_{3}A_{2} +b_{3}" title="Z_{3} = W_{3}A_{2} +b_{3}" src="https://s0.wp.com/latex.php?latex=Z_%7B3%7D+%3D+W_%7B3%7DA_%7B2%7D+%2Bb_%7B3%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex">  and <img alt="A_{3} = g(Z_{3})" title="A_{3} = g(Z_{3})" src="https://s0.wp.com/latex.php?latex=A_%7B3%7D+%3D+g%28Z_%7B3%7D%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"></p>
<p>The loss function is given by<br>
<img alt="L = -(ylogA3 + (1-y)log(1-A3))" title="L = -(ylogA3 + (1-y)log(1-A3))" src="https://s0.wp.com/latex.php?latex=L+%3D+-%28ylogA3+%2B+%281-y%29log%281-A3%29%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
and <img alt="dL/dA3 = -(Y/A_{3} + (1-Y)/(1-A_{3}))" title="dL/dA3 = -(Y/A_{3} + (1-Y)/(1-A_{3}))" src="https://s0.wp.com/latex.php?latex=dL%2FdA3+%3D+-%28Y%2FA_%7B3%7D+%2B+%281-Y%29%2F%281-A_%7B3%7D%29%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"></p>
<p>For a binary classification the output activation function is the sigmoid function given by<br>
<img alt="A_{3} = 1/(1+ e^{-Z3})" title="A_{3} = 1/(1+ e^{-Z3})" src="https://s0.wp.com/latex.php?latex=A_%7B3%7D+%3D+1%2F%281%2B+e%5E%7B-Z3%7D%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex">. It can be shown that<br>
<img alt="dA_{3}/dZ_{3} = A_{3}(1-A_3)" title="dA_{3}/dZ_{3} = A_{3}(1-A_3)" src="https://s0.wp.com/latex.php?latex=dA_%7B3%7D%2FdZ_%7B3%7D+%3D+A_%7B3%7D%281-A_3%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> see equation 2 in <a rel="nofollow" href="https://gigadom.wordpress.com/2018/01/04/deep-learning-from-basic-principles-in-python-r-and-octave-part-1/" target="_blank">Part 1</a></p>
<p><img alt="partial L/partial Z_{3} = partial L/partial A_{3}* partial A_{3}/partial Z_{3} = A3-Y" title="partial L/partial Z_{3} = partial L/partial A_{3}* partial A_{3}/partial Z_{3} = A3-Y" src="https://s0.wp.com/latex.php?latex=%5Cpartial+L%2F%5Cpartial+Z_%7B3%7D+%3D+%5Cpartial+L%2F%5Cpartial+A_%7B3%7D%2A+%5Cpartial+A_%7B3%7D%2F%5Cpartial+Z_%7B3%7D+%3D+A3-Y&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> see equation (f) in  <a rel="nofollow" href="https://gigadom.wordpress.com/2018/01/04/deep-learning-from-basic-principles-in-python-r-and-octave-part-1/" target="_blank">Part 1</a><br>
and since<br>
<img alt="partial L/partial A_{2} = partial L/partial Z_{3} * partial Z_{3}/partial A_{2} = (A_{3} -Y) * W_{3}" title="partial L/partial A_{2} = partial L/partial Z_{3} * partial Z_{3}/partial A_{2} = (A_{3} -Y) * W_{3}" src="https://s0.wp.com/latex.php?latex=%5Cpartial+L%2F%5Cpartial+A_%7B2%7D+%3D+%5Cpartial+L%2F%5Cpartial+Z_%7B3%7D+%2A+%5Cpartial+Z_%7B3%7D%2F%5Cpartial+A_%7B2%7D+%3D+%28A_%7B3%7D+-Y%29+%2A+W_%7B3%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> because <img alt="partial Z_{3}/partial A_{2} = W_{3}" title="partial Z_{3}/partial A_{2} = W_{3}" src="https://s0.wp.com/latex.php?latex=%5Cpartial+Z_%7B3%7D%2F%5Cpartial+A_%7B2%7D+%3D+W_%7B3%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> -(1a)<br>
and <img alt="partial L/partial Z_{2} =partial L/partial A_{2} * partial A_{2}/partial Z_{2} = (A_{3} -Y) * W_{3} *g'(Z_{2})" title="partial L/partial Z_{2} =partial L/partial A_{2} * partial A_{2}/partial Z_{2} = (A_{3} -Y) * W_{3} *g'(Z_{2})" src="https://s0.wp.com/latex.php?latex=%5Cpartial+L%2F%5Cpartial+Z_%7B2%7D+%3D%5Cpartial+L%2F%5Cpartial+A_%7B2%7D+%2A+%5Cpartial+A_%7B2%7D%2F%5Cpartial+Z_%7B2%7D+%3D+%28A_%7B3%7D+-Y%29+%2A+W_%7B3%7D+%2Ag%27%28Z_%7B2%7D%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> -(1b)<br>
<img alt="partial L/partial W_{2} = partial L/partial Z_{2} * A_{1}" title="partial L/partial W_{2} = partial L/partial Z_{2} * A_{1}" src="https://s0.wp.com/latex.php?latex=%5Cpartial+L%2F%5Cpartial+W_%7B2%7D+%3D+%5Cpartial+L%2F%5Cpartial+Z_%7B2%7D+%2A+A_%7B1%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> -(1c)<br>
since <img alt="partial Z_{2}/partial W_{2} = A_{1}" title="partial Z_{2}/partial W_{2} = A_{1}" src="https://s0.wp.com/latex.php?latex=%5Cpartial+Z_%7B2%7D%2F%5Cpartial+W_%7B2%7D+%3D+A_%7B1%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
and<br>
<img alt="partial L/partial b_{2} = partial L/partial Z_{2}" title="partial L/partial b_{2} = partial L/partial Z_{2}" src="https://s0.wp.com/latex.php?latex=%5Cpartial+L%2F%5Cpartial+b_%7B2%7D+%3D+%5Cpartial+L%2F%5Cpartial+Z_%7B2%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> -(1d)<br>
because<br>
<img alt="partial Z_{2}/partial b_{2} =1" title="partial Z_{2}/partial b_{2} =1" src="https://s0.wp.com/latex.php?latex=%5Cpartial+Z_%7B2%7D%2F%5Cpartial+b_%7B2%7D+%3D1&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"></p>
<p>Also</p>
<p><img alt="partial L/partial A_{1} =partial L/partial Z_{2} * partial Z_{2}/partial A_{1} = partial L/partial Z_{2} * W_{2}" title="partial L/partial A_{1} =partial L/partial Z_{2} * partial Z_{2}/partial A_{1} = partial L/partial Z_{2} * W_{2}" src="https://s0.wp.com/latex.php?latex=%5Cpartial+L%2F%5Cpartial+A_%7B1%7D+%3D%5Cpartial+L%2F%5Cpartial+Z_%7B2%7D+%2A+%5Cpartial+Z_%7B2%7D%2F%5Cpartial+A_%7B1%7D+%3D%C2%A0%5Cpartial+L%2F%5Cpartial+Z_%7B2%7D+%2A+W_%7B2%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex">     – (2a)<br>
<img alt="partial L/partial Z_{1} =partial L/partial A_{1} * partial A_{1}/partial Z_{1} = partial L/partial A_{1} * W_{3} *g'(Z_{2})" title="partial L/partial Z_{1} =partial L/partial A_{1} * partial A_{1}/partial Z_{1} = partial L/partial A_{1} * W_{3} *g'(Z_{2})" src="https://s0.wp.com/latex.php?latex=%5Cpartial+L%2F%5Cpartial+Z_%7B1%7D+%3D%5Cpartial+L%2F%5Cpartial+A_%7B1%7D+%2A+%5Cpartial+A_%7B1%7D%2F%5Cpartial+Z_%7B1%7D+%3D+%5Cpartial+L%2F%5Cpartial+A_%7B1%7D+%2A+W_%7B3%7D+%2Ag%27%28Z_%7B2%7D%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> – (2b)<br>
<img alt="partial L/partial W_{1} = partial L/partial Z_{1} * A_{0}" title="partial L/partial W_{1} = partial L/partial Z_{1} * A_{0}" src="https://s0.wp.com/latex.php?latex=%5Cpartial+L%2F%5Cpartial+W_%7B1%7D+%3D+%5Cpartial+L%2F%5Cpartial+Z_%7B1%7D+%2A+A_%7B0%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> – (2c)<br>
<img alt="partial L/partial b_{1} = partial L/partial Z_{1}" title="partial L/partial b_{1} = partial L/partial Z_{1}" src="https://s0.wp.com/latex.php?latex=%5Cpartial+L%2F%5Cpartial+b_%7B1%7D+%3D+%5Cpartial+L%2F%5Cpartial+Z_%7B1%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> – (2d)</p>
<p>Inspecting the above equations (1a – 1d &amp; 2a-2d), our ‘Uber Deep (bottomless)’ brain  can easily discern the pattern in these equations. The equation for any layer ‘l’ is of the form<br>
<img alt="Z_{l} = W_{l}A_{l-1} +b_{l}" title="Z_{l} = W_{l}A_{l-1} +b_{l}" src="https://s0.wp.com/latex.php?latex=Z_%7Bl%7D+%3D+W_%7Bl%7DA_%7Bl-1%7D+%2Bb_%7Bl%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex">     and  <img alt="A_{l} = g(Z_{l})" title="A_{l} = g(Z_{l})" src="https://s0.wp.com/latex.php?latex=A_%7Bl%7D+%3D+g%28Z_%7Bl%7D%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
The equation for the backward propagation have the general form<br>
<img alt="partial L/partial A_{l} = partial L/partial Z_{l+1} * W^{l+1}" title="partial L/partial A_{l} = partial L/partial Z_{l+1} * W^{l+1}" src="https://s0.wp.com/latex.php?latex=%5Cpartial+L%2F%5Cpartial+A_%7Bl%7D+%3D+%5Cpartial+L%2F%5Cpartial+Z_%7Bl%2B1%7D+%2A+W%5E%7Bl%2B1%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="partial L/partial Z_{l}=partial L/partial A_{l} *g'(Z_{l})" title="partial L/partial Z_{l}=partial L/partial A_{l} *g'(Z_{l})" src="https://s0.wp.com/latex.php?latex=%5Cpartial+L%2F%5Cpartial+Z_%7Bl%7D%3D%5Cpartial+L%2F%5Cpartial+A_%7Bl%7D+%2Ag%27%28Z_%7Bl%7D%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="partial L/partial W_{l} =partial L/partial Z_{l} *A^{l-1}" title="partial L/partial W_{l} =partial L/partial Z_{l} *A^{l-1}" src="https://s0.wp.com/latex.php?latex=%5Cpartial+L%2F%5Cpartial+W_%7Bl%7D+%3D%5Cpartial+L%2F%5Cpartial+Z_%7Bl%7D+%2AA%5E%7Bl-1%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="partial L/partial b_{l} =partial L/partial Z_{l}" title="partial L/partial b_{l} =partial L/partial Z_{l}" src="https://s0.wp.com/latex.php?latex=%5Cpartial+L%2F%5Cpartial+b_%7Bl%7D+%3D%5Cpartial+L%2F%5Cpartial+Z_%7Bl%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"></p>
<p>Some other important results The derivatives of the activation functions in the implemented Deep Learning network<br>
g(z) = sigmoid(z) = <img alt="1/(1+e^{-z})" title="1/(1+e^{-z})" src="https://s0.wp.com/latex.php?latex=1%2F%281%2Be%5E%7B-z%7D%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> = a g’(z) = a(1-a) – See <a rel="nofollow" href="https://gigadom.wordpress.com/2018/01/04/deep-learning-from-basic-principles-in-python-r-and-octave-part-1/" target="_blank">Part 1</a><br>
g(z) = tanh(z) = a g’(z) = <img alt="1 - a^{2}" title="1 - a^{2}" src="https://s0.wp.com/latex.php?latex=1+-+a%5E%7B2%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
g(z) = relu(z) = z  when z>0 and 0 when z 0 and 0 when z &lt;=0<br>
While it appears that there is a discontinuity for the derivative at 0 the small value at the discontinuity does not present a problem</p>
<p>The implementation of the multi layer vectorized Deep Learning Network for Python, R and Octave is included below. For all these implementations, initially I create the size and configuration of the the Deep Learning network with the layer dimennsions So for example layersDimension Vector ‘V’ of length L indicating ‘L’ layers where</p>
<p>V (in Python)= <img alt="[v_{0}, v_{1}, v_{2}" title="[v_{0}, v_{1}, v_{2}" src="https://s0.wp.com/latex.php?latex=%5Bv_%7B0%7D%2C+v_%7B1%7D%2C+v_%7B2%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex">, … <img alt="v_{L-1}]" title="v_{L-1}]" src="https://s0.wp.com/latex.php?latex=v_%7BL-1%7D%5D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
V (in R)= <img alt="c(v_{1}, v_{2}, v_{3}" title="c(v_{1}, v_{2}, v_{3}" src="https://s0.wp.com/latex.php?latex=c%28v_%7B1%7D%2C+v_%7B2%7D%2C+v_%7B3%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> , … <img alt="v_{L}" title="v_{L}" src="https://s0.wp.com/latex.php?latex=v_%7BL%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
V (in Octave)= [ <img alt="v_{1} v_{2} v_{3}" title="v_{1} v_{2} v_{3}" src="https://s0.wp.com/latex.php?latex=v_%7B1%7D+v_%7B2%7D+v_%7B3%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> … <img alt="v_{L}]" title="v_{L}]" src="https://s0.wp.com/latex.php?latex=v_%7BL%7D%5D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"></p>
<p>In all of these implementations the first element is the number of input features to the Deep Learning network and the last element is always a ‘sigmoid’ activation function since all the problems deal with binary classification.</p>
<p>The number of elements between the first and the last element are the number of hidden layers and the magnitude of each <img alt="v_{i}" title="v_{i}" src="https://s0.wp.com/latex.php?latex=v_%7Bi%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> is the number of activation units in each hidden layer, which is specified while actually executing the Deep Learning network using the function L_Layer_DeepModel(), in all the implementations Python, R and Octave</p>
<div>
<h2>1a. Classification with Multi layer Deep Learning Network – Relu activation(Python)</h2>
<p>In the code below a 4 layer Neural Network is trained to generate a non-linear boundary between the classes. In the code below the ‘Relu’ Activation function is used. The number of activation units in each layer is 9. The cost vs iterations is plotted in addition to the decision boundary. Further the accuracy, precision, recall and F1 score are also computed</p>
<pre><code><span>import</span> os
<span>import</span> numpy <span>as</span> np
<span>import</span> matplotlib.pyplot <span>as</span> plt
<span>import</span> matplotlib.colors
<span>import</span> sklearn.linear_model

<span>from</span> sklearn.model_selection <span>import</span> train_test_split
<span>from</span> sklearn.datasets <span>import</span> make_classification, make_blobs
<span>from</span> matplotlib.colors <span>import</span> ListedColormap
<span>import</span> sklearn
<span>import</span> sklearn.datasets

<span>#from DLfunctions import plot_decision_boundary</span>
execfile(<span>&quot;./DLfunctions34.py&quot;</span>) <span># </span>
os.chdir(<span>&quot;C:\\software\\DeepLearning-Posts\\part3&quot;</span>)

<span># Create clusters of 2 classes</span>
X1, Y1 = make_blobs(n_samples = <span>400</span>, n_features = <span>2</span>, centers = <span>9</span>,
                       cluster_std = <span>1.3</span>, random_state = <span>4</span>)
<span>#Create 2 classes</span>
Y1=Y1.reshape(<span>400</span>,<span>1</span>)
Y1 = Y1 % <span>2</span>
X2=X1.T
Y2=Y1.T
<span># Set the dimensions of DL Network </span>
<span># Below we have </span>
<span># 2 - 2 input features</span>
<span># 9,9 - 2 hidden layers with 9 activation units per layer and</span>
<span># 1 - 1 sigmoid activation unit in the output layer as this is a binary classification</span>
<span># The activation in the hidden layer is the 'relu' specified in L_Layer_DeepModel</span>

layersDimensions = [<span>2</span>, <span>9</span>, <span>9</span>,<span>1</span>] <span># 4-layer model</span>
parameters = L_Layer_DeepModel(X2, Y2, layersDimensions,hiddenActivationFunc=<span>'relu'</span>, learning_rate = <span>0.3</span>,num_iterations = <span>2500</span>, fig=<span>&quot;fig1.png&quot;</span>)
<span>#Plot the decision boundary</span>
plot_decision_boundary(<span>lambda</span> x: predict(parameters, x.T), X2,Y2,str(<span>0.3</span>),<span>&quot;fig2.png&quot;</span>)

<span># Compute the confusion matrix</span>
yhat = predict(parameters,X2)
<span>from</span> sklearn.metrics <span>import</span> confusion_matrix
a=confusion_matrix(Y2.T,yhat.T)
<span>from</span> sklearn.metrics <span>import</span> accuracy_score, precision_score, recall_score, f1_score
<span>print</span>(<span>'Accuracy: {:.2f}'</span>.format(accuracy_score(Y2.T, yhat.T)))
<span>print</span>(<span>'Precision: {:.2f}'</span>.format(precision_score(Y2.T, yhat.T)))
<span>print</span>(<span>'Recall: {:.2f}'</span>.format(recall_score(Y2.T, yhat.T)))
<span>print</span>(<span>'F1: {:.2f}'</span>.format(f1_score(Y2.T, yhat.T)))</code><code>
## Accuracy: 0.90
## Precision: 0.91
## Recall: 0.87
## F1: 0.89</code></pre>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/01/fig12.png?w=456" data-orig-size="572,455" data-medium-file="https://gigadom.files.wordpress.com/2018/01/fig12.png?w=456?w=300" data-attachment-id="6246" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/fig12.png?w=456" data-permalink="https://gigadom.wordpress.com/2018/01/30/deep-learning-from-first-principles-in-python-r-and-octave-part-3/fig1-14/" data-large-file="https://gigadom.files.wordpress.com/2018/01/fig12.png?w=456?w=572" alt="" data-image-title="fig1" sizes="(max-width: 572px) 100vw, 572px" data-comments-opened="1" class="wp-image-6246" data-recalc-dims="1"></p>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/01/fig21.png?w=456" data-orig-size="578,455" data-medium-file="https://gigadom.files.wordpress.com/2018/01/fig21.png?w=456?w=300" data-attachment-id="6247" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/fig21.png?w=456" data-permalink="https://gigadom.wordpress.com/2018/01/30/deep-learning-from-first-principles-in-python-r-and-octave-part-3/fig2-12/" data-large-file="https://gigadom.files.wordpress.com/2018/01/fig21.png?w=456?w=578" alt="" data-image-title="fig2" sizes="(max-width: 578px) 100vw, 578px" data-comments-opened="1" class="wp-image-6247" data-recalc-dims="1"></p>
<p>For more details on metrics like Accuracy, Recall, Precision etc. used in classification take a look at my post <a rel="nofollow" href="https://gigadom.wordpress.com/2017/10/13/practical-machine-learning-with-r-and-python-part-2/" target="_blank">Practical Machine Learning with R and Python – Part 2</a>. More details about these and other metrics besides implementation of the most common machine learning algorithms are available in my book <a rel="nofollow" href="https://gigadom.wordpress.com/2017/12/05/my-book-practical-machine-learning-with-r-and-python-on-amazon/" target="_blank">My book ‘Practical Machine Learning with R and Python’ on Amazon</a></p>
</div>
<div>
<h2>1b. Classification with Multi layer Deep Learning Network – Relu activation(R)</h2>
<p>In the code below, binary classification is performed on the same data set as above using the Relu activation function. The DL network is same as above<code></code><code></code></p>
<pre><code><span>library</span><span>(</span><span>ggplot2</span><span>)</span>
<span># Read the data</span>
<span>z</span> <span>&lt;-</span> <span>as.matrix</span><span>(</span><span>read.csv</span><span>(</span><span>&quot;data.csv&quot;</span>,<span>header</span><span>=</span><span>FALSE</span><span>)</span><span>)</span> 
<span>x</span> <span>&lt;-</span> <span>z</span><span>[</span>,<span>1</span><span>:</span><span>2</span><span>]</span>
<span>y</span> <span>&lt;-</span> <span>z</span><span>[</span>,<span>3</span><span>]</span>
<span>X1</span> <span>&lt;-</span> <span>t</span><span>(</span><span>x</span><span>)</span>
<span>Y1</span> <span>&lt;-</span> <span>t</span><span>(</span><span>y</span><span>)</span>

<span># Set the dimensions of the Deep Learning network</span>
<span># No of input features =2, 2 hidden layers with 9 activation units and 1 output layer</span>
<span>layersDimensions</span> <span>=</span> <span>c</span><span>(</span><span>2</span>, <span>9</span>, <span>9</span>,<span>1</span><span>)</span>
<span># Execute the Deep Learning Neural Network</span>
<span>retvals</span> <span>=</span> <span>L_Layer_DeepModel</span><span>(</span><span>X1</span>, <span>Y1</span>, <span>layersDimensions</span>,
                               <span>hiddenActivationFunc</span><span>=</span><span>'relu'</span>, 
                               <span>learningRate</span> <span>=</span> <span>0.3</span>,
                               <span>numIterations</span> <span>=</span> <span>5000</span>, 
                               <span>print_cost</span> <span>=</span> <span>True</span><span>)</span></code></pre>
<pre><code><span>library</span><span>(</span><span>ggplot2</span><span>)</span>
<span>source</span><span>(</span><span>&quot;DLfunctions33.R&quot;</span><span>)</span>
<span># Get the computed costs</span>
<span>costs</span> <span>&lt;-</span> <span>retvals</span><span>[</span><span>[</span><span>'costs'</span><span>]</span><span>]</span>
<span># Create a sequence of iterations</span>
<span>numIterations</span><span>=</span><span>5000</span>
<span>iterations</span> <span>&lt;-</span> <span>seq</span><span>(</span><span>0</span>,<span>numIterations</span>,<span>by</span><span>=</span><span>1000</span><span>)</span>
<span>df</span> <span>&lt;-</span><span>data.frame</span><span>(</span><span>iterations</span>,<span>costs</span><span>)</span>
<span># Plot the Costs vs number of iterations</span>
<span>ggplot</span><span>(</span><span>df</span>,<span>aes</span><span>(</span><span>x</span><span>=</span><span>iterations</span>,<span>y</span><span>=</span><span>costs</span><span>)</span><span>)</span> <span>+</span> <span>geom_point</span><span>(</span><span>)</span> <span>+</span><span>geom_line</span><span>(</span><span>color</span><span>=</span><span>&quot;blue&quot;</span><span>)</span> <span>+</span>
    <span>xlab</span><span>(</span><span>'No of iterations'</span><span>)</span> <span>+</span> <span>ylab</span><span>(</span><span>'Cost'</span><span>)</span> <span>+</span> <span>ggtitle</span><span>(</span><span>&quot;Cost vs No of iterations&quot;</span><span>)</span></code></pre>
<p><img><img data-orig-file="https://gigadom.files.wordpress.com/2018/01/fig1-11.png" data-orig-size="1344,960" data-medium-file="https://gigadom.files.wordpress.com/2018/01/fig1-11.png?w=300" data-attachment-id="6248" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/fig1-11.png?w=450&h=483&fit=676%2C483" data-permalink="https://gigadom.wordpress.com/2018/01/30/deep-learning-from-first-principles-in-python-r-and-octave-part-3/fig1-1-7/" data-large-file="https://gigadom.files.wordpress.com/2018/01/fig1-11.png?w=450&amp;h=483&amp;fit=676%2C483" alt="" data-image-title="fig1-1" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6248" data-recalc-dims="1"></p>
<pre><code><span># Plot the decision boundary</span>
<span>plotDecisionBoundary</span><span>(</span><span>z</span>,<span>retvals</span>,<span>hiddenActivationFunc</span><span>=</span><span>&quot;relu&quot;</span>,<span>0.3</span><span>)</span></code></pre>
<p><img><img data-orig-file="https://gigadom.files.wordpress.com/2018/01/fig1-2.png" data-orig-size="1344,960" data-medium-file="https://gigadom.files.wordpress.com/2018/01/fig1-2.png?w=300" data-attachment-id="6249" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/fig1-2.png?w=450&h=483&fit=676%2C483" data-permalink="https://gigadom.wordpress.com/2018/01/30/deep-learning-from-first-principles-in-python-r-and-octave-part-3/fig1-2-3/" data-large-file="https://gigadom.files.wordpress.com/2018/01/fig1-2.png?w=450&amp;h=483&amp;fit=676%2C483" alt="" data-image-title="fig1-2" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6249" data-recalc-dims="1"></p>
<pre><code><span>library</span><span>(</span><span>caret</span><span>)</span></code><code></code><code></code></pre>
<pre><code><span># Predict the output for the data values</span>
<span>yhat</span> <span>&lt;-</span><span>predict</span><span>(</span><span>retvals</span><span>$</span><span>parameters</span>,<span>X1</span>,<span>hiddenActivationFunc</span><span>=</span><span>&quot;relu&quot;</span><span>)</span>
<span>yhat</span><span>[</span><span>yhat</span><span>==</span><span>FALSE</span><span>]</span><span>=</span><span>0</span>
<span>yhat</span><span>[</span><span>yhat</span><span>==</span><span>TRUE</span><span>]</span><span>=</span><span>1</span>
<span># Compute the confusion matrix</span>
<span>confusionMatrix</span><span>(</span><span>yhat</span>,<span>Y1</span><span>)</span></code></pre>
<pre><code>## Confusion Matrix and Statistics
## 
##           Reference
## Prediction   0   1
##          0 201  10
##          1  21 168
##                                           
##                Accuracy : 0.9225          
##                  95% CI : (0.8918, 0.9467)
##     No Information Rate : 0.555           
##     P-Value [Acc > NIR] : &lt; 2e-16         
##                                           
##                   Kappa : 0.8441          
##  Mcnemar's Test P-Value : 0.07249         
##                                           
##             Sensitivity : 0.9054          
##             Specificity : 0.9438          
##          Pos Pred Value : 0.9526          
##          Neg Pred Value : 0.8889          
##              Prevalence : 0.5550          
##          Detection Rate : 0.5025          
##    Detection Prevalence : 0.5275          
##       Balanced Accuracy : 0.9246          
##                                           
##        'Positive' Class : 0               
## </code></pre>
</div>
<div>
<h2>1c. Classification with Multi layer Deep Learning Network – Relu activation(Octave)</h2>
<p>Included below is the code for performing classification. Incidentally Octave does not seem to have implemented the confusion matrix,  but confusionmat is available in Matlab.<code><br>
# Read the data<br>
data=csvread(&quot;data.csv&quot;);<br>
X=data(:,1:2);<br>
Y=data(:,3);<br>
# Set layer dimensions<br>
layersDimensions = [2 9 7  1] #tanh=-0.5(ok), #relu=0.1 best!<br>
# Execute Deep Network<br>
[weights biases costs]=L_Layer_DeepModel(X', Y', layersDimensions,<br>
hiddenActivationFunc='relu',<br>
learningRate = 0.1,<br>
numIterations = 10000);<br>
plotCostVsIterations(10000,costs);<br>
plotDecisionBoundary(data,weights, biases,hiddenActivationFunc=&quot;tanh&quot;)<br>
</code><br>
<img data-orig-file="https://gigadom.files.wordpress.com/2018/01/figure2.png" data-orig-size="1200,900" data-medium-file="https://gigadom.files.wordpress.com/2018/01/figure2.png?w=300" data-attachment-id="6268" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/figure2.png?w=450&h=507&fit=676%2C507" data-permalink="https://gigadom.wordpress.com/2018/01/30/deep-learning-from-first-principles-in-python-r-and-octave-part-3/figure2/" data-large-file="https://gigadom.files.wordpress.com/2018/01/figure2.png?w=450&amp;h=507&amp;fit=676%2C507" alt="" data-image-title="figure2" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6268" data-recalc-dims="1"><br>
<img data-orig-file="https://gigadom.files.wordpress.com/2018/01/figure3.png" data-orig-size="1200,900" data-medium-file="https://gigadom.files.wordpress.com/2018/01/figure3.png?w=300" data-attachment-id="6269" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/figure3.png?w=450&h=507&fit=676%2C507" data-permalink="https://gigadom.wordpress.com/2018/01/30/deep-learning-from-first-principles-in-python-r-and-octave-part-3/figure3/" data-large-file="https://gigadom.files.wordpress.com/2018/01/figure3.png?w=450&amp;h=507&amp;fit=676%2C507" alt="" data-image-title="figure3" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6269" data-recalc-dims="1"></p>
<h2>2a. Classification with Multi layer Deep Learning Network – Tanh activation(Python)</h2>
<p>Below the Tanh activation function is used to perform the same classification. I found the Tanh activation required a simpler Neural Network of 3 layers.</p>
<pre><code><span># Tanh activation</span>
<span>import</span> os
<span>import</span> numpy <span>as</span> np
<span>import</span> matplotlib.pyplot <span>as</span> plt
<span>import</span> matplotlib.colors
<span>import</span> sklearn.linear_model

<span>from</span> sklearn.model_selection <span>import</span> train_test_split
<span>from</span> sklearn.datasets <span>import</span> make_classification, make_blobs
<span>from</span> matplotlib.colors <span>import</span> ListedColormap
<span>import</span> sklearn
<span>import</span> sklearn.datasets

<span>#from DLfunctions import plot_decision_boundary</span>
os.chdir(<span>&quot;C:\\software\\DeepLearning-Posts\\part3&quot;</span>)
execfile(<span>&quot;./DLfunctions34.py&quot;</span>) 
<span># Create the dataset</span>
X1, Y1 = make_blobs(n_samples = <span>400</span>, n_features = <span>2</span>, centers = <span>9</span>,
                       cluster_std = <span>1.3</span>, random_state = <span>4</span>)
<span>#Create 2 classes</span>
Y1=Y1.reshape(<span>400</span>,<span>1</span>)
Y1 = Y1 % <span>2</span>
X2=X1.T
Y2=Y1.T
<span># Set the dimensions of the Neural Network</span>
layersDimensions = [<span>2</span>, <span>4</span>, <span>1</span>] <span># 3-layer model</span>
<span># Compute the DL network</span>
parameters = L_Layer_DeepModel(X2, Y2, layersDimensions, hiddenActivationFunc=<span>'tanh'</span>, learning_rate = .<span>5</span>,num_iterations = <span>2500</span>,fig=<span>&quot;fig3.png&quot;</span>)
<span>#Plot the decision boundary</span>
plot_decision_boundary(<span>lambda</span> x: predict(parameters, x.T), X2,Y2,str(<span>0.5</span>),<span>&quot;fig4.png&quot;</span>)
</code></pre>
</div>
<div>
<h2><img data-orig-file="https://gigadom.files.wordpress.com/2018/01/fig32.png?w=456" data-orig-size="581,455" data-medium-file="https://gigadom.files.wordpress.com/2018/01/fig32.png?w=456?w=300" data-attachment-id="6251" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/fig32.png?w=456" data-permalink="https://gigadom.wordpress.com/2018/01/30/deep-learning-from-first-principles-in-python-r-and-octave-part-3/fig3-11/" data-large-file="https://gigadom.files.wordpress.com/2018/01/fig32.png?w=456?w=581" alt="" data-image-title="fig3" sizes="(max-width: 581px) 100vw, 581px" data-comments-opened="1" class="wp-image-6251" data-recalc-dims="1"></h2>
<h2><img data-orig-file="https://gigadom.files.wordpress.com/2018/01/fig41.png?w=456" data-orig-size="578,455" data-medium-file="https://gigadom.files.wordpress.com/2018/01/fig41.png?w=456?w=300" data-attachment-id="6252" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/fig41.png?w=456" data-permalink="https://gigadom.wordpress.com/2018/01/30/deep-learning-from-first-principles-in-python-r-and-octave-part-3/fig4-7/" data-large-file="https://gigadom.files.wordpress.com/2018/01/fig41.png?w=456?w=578" alt="" data-image-title="fig4" sizes="(max-width: 578px) 100vw, 578px" data-comments-opened="1" class="wp-image-6252" data-recalc-dims="1"></h2>
<h2>2b. Classification with Multi layer Deep Learning Network – Tanh activation(R)</h2>
<p>R performs better with a Tanh activation than the Relu as can be seen below</p>
<pre><code> <span>#Set the dimensions of the Neural Network</span>
<span>layersDimensions</span> <span>=</span> <span>c</span><span>(</span><span>2</span>, <span>9</span>, <span>9</span>,<span>1</span><span>)</span>
<span>library</span><span>(</span><span>ggplot2</span><span>)</span>
<span># Read the data</span>
<span>z</span> <span>&lt;-</span> <span>as.matrix</span><span>(</span><span>read.csv</span><span>(</span><span>&quot;data.csv&quot;</span>,<span>header</span><span>=</span><span>FALSE</span><span>)</span><span>)</span> 
<span>x</span> <span>&lt;-</span> <span>z</span><span>[</span>,<span>1</span><span>:</span><span>2</span><span>]</span>
<span>y</span> <span>&lt;-</span> <span>z</span><span>[</span>,<span>3</span><span>]</span>
<span>X1</span> <span>&lt;-</span> <span>t</span><span>(</span><span>x</span><span>)</span>
<span>Y1</span> <span>&lt;-</span> <span>t</span><span>(</span><span>y</span><span>)</span>
<span># Execute the Deep Model</span>
<span>retvals</span> <span>=</span> <span>L_Layer_DeepModel</span><span>(</span><span>X1</span>, <span>Y1</span>, <span>layersDimensions</span>,
                            <span>hiddenActivationFunc</span><span>=</span><span>'tanh'</span>, 
                            <span>learningRate</span> <span>=</span> <span>0.3</span>,
                            <span>numIterations</span> <span>=</span> <span>5000</span>, 
                            <span>print_cost</span> <span>=</span> <span>True</span><span>)</span></code></pre>
<pre><code><span># Get the costs</span>
<span>costs</span> <span>&lt;-</span> <span>retvals</span><span>[</span><span>[</span><span>'costs'</span><span>]</span><span>]</span>
<span>iterations</span> <span>&lt;-</span> <span>seq</span><span>(</span><span>0</span>,<span>numIterations</span>,<span>by</span><span>=</span><span>1000</span><span>)</span>
<span>df</span> <span>&lt;-</span><span>data.frame</span><span>(</span><span>iterations</span>,<span>costs</span><span>)</span>
<span># Plot Cost vs number of iterations</span>
<span>ggplot</span><span>(</span><span>df</span>,<span>aes</span><span>(</span><span>x</span><span>=</span><span>iterations</span>,<span>y</span><span>=</span><span>costs</span><span>)</span><span>)</span> <span>+</span> <span>geom_point</span><span>(</span><span>)</span> <span>+</span><span>geom_line</span><span>(</span><span>color</span><span>=</span><span>&quot;blue&quot;</span><span>)</span> <span>+</span>
    <span>xlab</span><span>(</span><span>'No of iterations'</span><span>)</span> <span>+</span> <span>ylab</span><span>(</span><span>'Cost'</span><span>)</span> <span>+</span> <span>ggtitle</span><span>(</span><span>&quot;Cost vs No of iterations&quot;</span><span>)</span></code></pre>
<p><img><img data-orig-file="https://gigadom.files.wordpress.com/2018/01/fig2-11.png" data-orig-size="1344,960" data-medium-file="https://gigadom.files.wordpress.com/2018/01/fig2-11.png?w=300" data-attachment-id="6254" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/fig2-11.png?w=450&h=483&fit=676%2C483" data-permalink="https://gigadom.wordpress.com/2018/01/30/deep-learning-from-first-principles-in-python-r-and-octave-part-3/fig2-1-7/" data-large-file="https://gigadom.files.wordpress.com/2018/01/fig2-11.png?w=450&amp;h=483&amp;fit=676%2C483" alt="" data-image-title="fig2-1" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6254" data-recalc-dims="1"></p>
<pre><code><span>#Plot the decision boundary</span>
<span>plotDecisionBoundary</span><span>(</span><span>z</span>,<span>retvals</span>,<span>hiddenActivationFunc</span><span>=</span><span>&quot;tanh&quot;</span>,<span>0.3</span><span>)</span></code></pre>
<p><img><img data-orig-file="https://gigadom.files.wordpress.com/2018/01/fig2-2.png" data-orig-size="1344,960" data-medium-file="https://gigadom.files.wordpress.com/2018/01/fig2-2.png?w=300" data-attachment-id="6255" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/fig2-2.png?w=450&h=483&fit=676%2C483" data-permalink="https://gigadom.wordpress.com/2018/01/30/deep-learning-from-first-principles-in-python-r-and-octave-part-3/fig2-2-2/" data-large-file="https://gigadom.files.wordpress.com/2018/01/fig2-2.png?w=450&amp;h=483&amp;fit=676%2C483" alt="" data-image-title="fig2-2" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6255" data-recalc-dims="1"></p>
</div>
<div>
<h2>2c. Classification with Multi layer Deep Learning Network – Tanh activation(Octave)</h2>
<p>The code below uses the   Tanh activation in the hidden layers for Octave<code><br>
# Read the data<br>
data=csvread(&quot;data.csv&quot;);<br>
X=data(:,1:2);<br>
Y=data(:,3);<br>
# Set layer dimensions<br>
layersDimensions = [2 9 7  1] #tanh=-0.5(ok), #relu=0.1 best!<br>
# Execute Deep Network<br>
[weights biases costs]=L_Layer_DeepModel(X', Y', layersDimensions,<br>
hiddenActivationFunc='tanh',<br>
learningRate = 0.1,<br>
numIterations = 10000);<br>
plotCostVsIterations(10000,costs);<br>
plotDecisionBoundary(data,weights, biases,hiddenActivationFunc=&quot;tanh&quot;)<br>
</code><br>
<img data-orig-file="https://gigadom.files.wordpress.com/2018/01/figure21.png" data-orig-size="1200,900" data-medium-file="https://gigadom.files.wordpress.com/2018/01/figure21.png?w=300" data-attachment-id="6271" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/figure21.png?w=450&h=507&fit=676%2C507" data-permalink="https://gigadom.wordpress.com/2018/01/30/deep-learning-from-first-principles-in-python-r-and-octave-part-3/figure21/" data-large-file="https://gigadom.files.wordpress.com/2018/01/figure21.png?w=450&amp;h=507&amp;fit=676%2C507" alt="" data-image-title="figure21" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6271" data-recalc-dims="1"><br>
<img data-orig-file="https://gigadom.files.wordpress.com/2018/01/figure31.png" data-orig-size="1200,900" data-medium-file="https://gigadom.files.wordpress.com/2018/01/figure31.png?w=300" data-attachment-id="6272" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/figure31.png?w=450&h=507&fit=676%2C507" data-permalink="https://gigadom.wordpress.com/2018/01/30/deep-learning-from-first-principles-in-python-r-and-octave-part-3/figure31/" data-large-file="https://gigadom.files.wordpress.com/2018/01/figure31.png?w=450&amp;h=507&amp;fit=676%2C507" alt="" data-image-title="figure31" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6272" data-recalc-dims="1"></p>
<h2>3. Bernoulli’s Lemniscate</h2>
<p>To make things  more interesting, I create a 2D figure of the Bernoulli’s lemniscate to perform non-linear classification. The Lemniscate is given by the equation<br>
<img alt="(x^{2} + y^{2})^{2}" title="(x^{2} + y^{2})^{2}" src="https://s0.wp.com/latex.php?latex=%28x%5E%7B2%7D+%2B+y%5E%7B2%7D%29%5E%7B2%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> = <img alt="2a^{2}*(x^{2}-y^{2})" title="2a^{2}*(x^{2}-y^{2})" src="https://s0.wp.com/latex.php?latex=2a%5E%7B2%7D%2A%28x%5E%7B2%7D-y%5E%7B2%7D%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"></p>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/01/fig50.png?w=456" data-orig-size="546,416" data-medium-file="https://gigadom.files.wordpress.com/2018/01/fig50.png?w=456?w=300" data-attachment-id="6259" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/fig50.png?w=456" data-permalink="https://gigadom.wordpress.com/2018/01/30/deep-learning-from-first-principles-in-python-r-and-octave-part-3/fig50/" data-large-file="https://gigadom.files.wordpress.com/2018/01/fig50.png?w=456?w=546" alt="" data-image-title="fig50" sizes="(max-width: 546px) 100vw, 546px" data-comments-opened="1" class="wp-image-6259" data-recalc-dims="1"></p>
</div>
<div>
<h2>3a. Classifying a lemniscate with Deep Learning Network – Relu activation(Python)</h2>
<pre><code><span>import</span> os
<span>import</span> numpy <span>as</span> np 
<span>import</span> matplotlib.pyplot <span>as</span> plt
os.chdir(<span>&quot;C:\\software\\DeepLearning-Posts\\part3&quot;</span>)
execfile(<span>&quot;./DLfunctions33.py&quot;</span>) 
x1=np.random.uniform(<span>0</span>,<span>10</span>,<span>2000</span>).reshape(<span>2000</span>,<span>1</span>)
x2=np.random.uniform(<span>0</span>,<span>10</span>,<span>2000</span>).reshape(<span>2000</span>,<span>1</span>)

X=np.append(x1,x2,axis=<span>1</span>)
X.shape

<span># Create a subset of values where squared is &lt;0,4. Perform ravel() to flatten this vector</span>
<span># Create the equation</span>
<span># (x^{2} + y^{2})^2 - 2a^2*(x^{2}-y^{2}) &lt;=0</span>
a=np.power(np.power(X[:,<span>0</span>]-<span>5</span>,<span>2</span>) + np.power(X[:,<span>1</span>]-<span>5</span>,<span>2</span>),<span>2</span>)
b=np.power(X[:,<span>0</span>]-<span>5</span>,<span>2</span>) - np.power(X[:,<span>1</span>]-<span>5</span>,<span>2</span>)
c= a - (b*np.power(<span>4</span>,<span>2</span>)) &lt;=<span>0</span>
Y=c.reshape(<span>2000</span>,<span>1</span>)
<span># Create a scatter plot of the lemniscate</span>
plt.scatter(X[:,<span>0</span>], X[:,<span>1</span>], c=Y, marker= <span>'o'</span>, s=<span>15</span>,cmap=<span>&quot;viridis&quot;</span>)
Z=np.append(X,Y,axis=<span>1</span>)
plt.savefig(<span>&quot;fig50.png&quot;</span>,bbox_inches=<span>'tight'</span>)
plt.clf()

<span># Set the data for classification</span>
X2=X.T
Y2=Y.T
<span># These settings work the best</span>
<span># Set the Deep Learning layer dimensions for a Relu activation</span>
layersDimensions = [<span>2</span>,<span>7</span>,<span>4</span>,<span>1</span>]
<span>#Execute the DL network</span>
parameters = L_Layer_DeepModel(X2, Y2, layersDimensions, hiddenActivationFunc=<span>'relu'</span>, learning_rate = <span>0.5</span>,num_iterations = <span>10000</span>, fig=<span>&quot;fig5.png&quot;</span>)
<span>#Plot the decision boundary</span>
plot_decision_boundary(<span>lambda</span> x: predict(parameters, x.T), X2, Y2,str(<span>2.2</span>),<span>&quot;fig6.png&quot;</span>)

<span># Compute the Confusion matrix</span>
yhat = predict(parameters,X2)
<span>from</span> sklearn.metrics <span>import</span> confusion_matrix
a=confusion_matrix(Y2.T,yhat.T)
<span>from</span> sklearn.metrics <span>import</span> accuracy_score, precision_score, recall_score, f1_score
<span>print</span>(<span>'Accuracy: {:.2f}'</span>.format(accuracy_score(Y2.T, yhat.T)))
<span>print</span>(<span>'Precision: {:.2f}'</span>.format(precision_score(Y2.T, yhat.T)))
<span>print</span>(<span>'Recall: {:.2f}'</span>.format(recall_score(Y2.T, yhat.T)))
<span>print</span>(<span>'F1: {:.2f}'</span>.format(f1_score(Y2.T, yhat.T)))</code><code>
## Accuracy: 0.93
## Precision: 0.77
## Recall: 0.76
## F1: 0.76</code></pre>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/01/fig51.png?w=456" data-orig-size="570,455" data-medium-file="https://gigadom.files.wordpress.com/2018/01/fig51.png?w=456?w=300" data-attachment-id="6260" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/fig51.png?w=456" data-permalink="https://gigadom.wordpress.com/2018/01/30/deep-learning-from-first-principles-in-python-r-and-octave-part-3/fig5-6/" data-large-file="https://gigadom.files.wordpress.com/2018/01/fig51.png?w=456?w=570" alt="" data-image-title="fig5" sizes="(max-width: 570px) 100vw, 570px" data-comments-opened="1" class="wp-image-6260" data-recalc-dims="1"></p>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/01/fig61.png?w=456" data-orig-size="566,455" data-medium-file="https://gigadom.files.wordpress.com/2018/01/fig61.png?w=456?w=300" data-attachment-id="6261" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/fig61.png?w=456" data-permalink="https://gigadom.wordpress.com/2018/01/30/deep-learning-from-first-principles-in-python-r-and-octave-part-3/fig61/" data-large-file="https://gigadom.files.wordpress.com/2018/01/fig61.png?w=456?w=566" alt="" data-image-title="fig61" sizes="(max-width: 566px) 100vw, 566px" data-comments-opened="1" class="wp-image-6261" data-recalc-dims="1"></p>
<p>We could get better performance by tuning further. Do play around if you fork the code.<br>
<strong>Note:</strong>: The lemniscate data is saved as a CSV and then read in R and also in Octave. I do this instead of recreating the lemniscate shape</p>
</div>
<div>
<h2>3b. Classifying a lemniscate with Deep Learning Network – Relu activation(R code)</h2>
<p>The R decision boundary for the Bernoulli’s lemniscate is shown below</p>
<pre><code><span>Z</span> <span>&lt;-</span> <span>as.matrix</span><span>(</span><span>read.csv</span><span>(</span><span>&quot;lemniscate.csv&quot;</span>,<span>header</span><span>=</span><span>FALSE</span><span>)</span><span>)</span>
<span>Z1</span><span>=</span><span>data.frame</span><span>(</span><span>Z</span><span>)</span>
<span># Create a scatter plot of the lemniscate</span>
<span>ggplot</span><span>(</span><span>Z1</span>,<span>aes</span><span>(</span><span>x</span><span>=</span><span>V1</span>,<span>y</span><span>=</span><span>V2</span>,<span>col</span><span>=</span><span>V3</span><span>)</span><span>)</span> <span>+</span><span>geom_point</span><span>(</span><span>)</span></code></pre>
<pre><code><span>#Set the data for the DL network</span>
<span>X</span><span>=</span><span>Z</span><span>[</span>,<span>1</span><span>:</span><span>2</span><span>]</span>
<span>Y</span><span>=</span><span>Z</span><span>[</span>,<span>3</span><span>]</span>

<span>X1</span><span>=</span><span>t</span><span>(</span><span>X</span><span>)</span>
<span>Y1</span><span>=</span><span>t</span><span>(</span><span>Y</span><span>)</span>

<span># Set the layer dimensions for the tanh activation function</span>
<span>layersDimensions</span> <span>=</span> <span>c</span><span>(</span><span>2</span>,<span>5</span>,<span>4</span>,<span>1</span><span>)</span>
<span># Execute the Deep Learning network with Tanh activation</span>
<span>retvals</span> <span>=</span> <span>L_Layer_DeepModel</span><span>(</span><span>X1</span>, <span>Y1</span>, <span>layersDimensions</span>, 
                               <span>hiddenActivationFunc</span><span>=</span><span>'tanh'</span>, 
                               <span>learningRate</span> <span>=</span> <span>0.3</span>,
                               <span>numIterations</span> <span>=</span> <span>20000</span>, <span>print_cost</span> <span>=</span> <span>True</span><span>)</span></code></pre>
<pre><code><span># Plot cost vs iteration</span>
<span>costs</span> <span>&lt;-</span> <span>retvals</span><span>[</span><span>[</span><span>'costs'</span><span>]</span><span>]</span>
<span>numIterations</span> <span>=</span> <span>20000</span>
<span>iterations</span> <span>&lt;-</span> <span>seq</span><span>(</span><span>0</span>,<span>numIterations</span>,<span>by</span><span>=</span><span>1000</span><span>)</span>
<span>df</span> <span>&lt;-</span><span>data.frame</span><span>(</span><span>iterations</span>,<span>costs</span><span>)</span>
<span>ggplot</span><span>(</span><span>df</span>,<span>aes</span><span>(</span><span>x</span><span>=</span><span>iterations</span>,<span>y</span><span>=</span><span>costs</span><span>)</span><span>)</span> <span>+</span> <span>geom_point</span><span>(</span><span>)</span> <span>+</span><span>geom_line</span><span>(</span><span>color</span><span>=</span><span>&quot;blue&quot;</span><span>)</span> <span>+</span>
    <span>xlab</span><span>(</span><span>'No of iterations'</span><span>)</span> <span>+</span> <span>ylab</span><span>(</span><span>'Cost'</span><span>)</span> <span>+</span> <span>ggtitle</span><span>(</span><span>&quot;Cost vs No of iterations&quot;</span><span>)</span></code></pre>
<p><img><img data-orig-file="https://gigadom.files.wordpress.com/2018/01/fig3-2.png" data-orig-size="1344,960" data-medium-file="https://gigadom.files.wordpress.com/2018/01/fig3-2.png?w=300" data-attachment-id="6262" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/fig3-2.png?w=450&h=483&fit=676%2C483" data-permalink="https://gigadom.wordpress.com/2018/01/30/deep-learning-from-first-principles-in-python-r-and-octave-part-3/fig3-2-2/" data-large-file="https://gigadom.files.wordpress.com/2018/01/fig3-2.png?w=450&amp;h=483&amp;fit=676%2C483" alt="" data-image-title="fig3-2" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6262" data-recalc-dims="1"></p>
<pre><code><span>#Plot the decision boundary</span>
<span>plotDecisionBoundary</span><span>(</span><span>Z</span>,<span>retvals</span>,<span>hiddenActivationFunc</span><span>=</span><span>&quot;tanh&quot;</span>,<span>0.3</span><span>)</span></code></pre>
<p><img><img data-orig-file="https://gigadom.files.wordpress.com/2018/01/fig3-3.png" data-orig-size="1344,960" data-medium-file="https://gigadom.files.wordpress.com/2018/01/fig3-3.png?w=300" data-attachment-id="6263" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/fig3-3.png?w=450&h=483&fit=676%2C483" data-permalink="https://gigadom.wordpress.com/2018/01/30/deep-learning-from-first-principles-in-python-r-and-octave-part-3/fig3-3-2/" data-large-file="https://gigadom.files.wordpress.com/2018/01/fig3-3.png?w=450&amp;h=483&amp;fit=676%2C483" alt="" data-image-title="fig3-3" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6263" data-recalc-dims="1"></p>
</div>
<div>
<h2>3c. Classifying a lemniscate with Deep Learning Network – Relu activation(Octave code)</h2>
<p>Octave is used to generate the non-linear lemniscate boundary.<br>
<code><br>
# Read the data<br>
data=csvread(&quot;lemniscate.csv&quot;);<br>
X=data(:,1:2);<br>
Y=data(:,3);<br>
# Set the dimensions of the layers<br>
layersDimensions = [2 9 7  1]<br>
# Compute the DL network<br>
[weights biases costs]=L_Layer_DeepModel(X', Y', layersDimensions,<br>
hiddenActivationFunc='relu',<br>
learningRate = 0.20,<br>
numIterations = 10000);<br>
plotCostVsIterations(10000,costs);<br>
plotDecisionBoundary(data,weights, biases,hiddenActivationFunc=&quot;relu&quot;)<br>
</code></p>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/01/figure22.png" data-orig-size="1200,900" data-medium-file="https://gigadom.files.wordpress.com/2018/01/figure22.png?w=300" data-attachment-id="6273" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/figure22.png?w=450&h=507&fit=676%2C507" data-permalink="https://gigadom.wordpress.com/2018/01/30/deep-learning-from-first-principles-in-python-r-and-octave-part-3/figure22/" data-large-file="https://gigadom.files.wordpress.com/2018/01/figure22.png?w=450&amp;h=507&amp;fit=676%2C507" alt="" data-image-title="figure22" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6273" data-recalc-dims="1"><br>
<img data-orig-file="https://gigadom.files.wordpress.com/2018/01/figure32.png" data-orig-size="1200,900" data-medium-file="https://gigadom.files.wordpress.com/2018/01/figure32.png?w=300" data-attachment-id="6274" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/figure32.png?w=450&h=507&fit=676%2C507" data-permalink="https://gigadom.wordpress.com/2018/01/30/deep-learning-from-first-principles-in-python-r-and-octave-part-3/figure32/" data-large-file="https://gigadom.files.wordpress.com/2018/01/figure32.png?w=450&amp;h=507&amp;fit=676%2C507" alt="" data-image-title="figure32" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6274" data-recalc-dims="1"></p>
<h2>4a. Binary Classification using MNIST – Python code</h2>
<p>Finally I perform a simple classification using the MNIST handwritten digits, which according to Prof Geoffrey Hinton is “the Drosophila of Deep Learning”.</p>
<p>The Python code for reading the MNIST data is taken from Alex Kesling’s github link <a rel="nofollow" href="https://gist.github.com/akesling/5358964" target="_blank">MNIST</a>.</p>
<p>In the Python code below, I perform a simple binary classification between the handwritten digit ‘5’ and ‘not 5’ which is all other digits. I will perform the proper classification of all digits using the  Softmax classifier some time later.</p>
<pre><code><span>import</span> os
<span>import</span> numpy <span>as</span> np 
<span>import</span> matplotlib.pyplot <span>as</span> plt
os.chdir(<span>&quot;C:\\software\\DeepLearning-Posts\\part3&quot;</span>)
execfile(<span>&quot;./DLfunctions34.py&quot;</span>) 
execfile(<span>&quot;./load_mnist.py&quot;</span>)
training=list(read(dataset=<span>'training'</span>,path=<span>&quot;./mnist&quot;</span>))
test=list(read(dataset=<span>'testing'</span>,path=<span>&quot;./mnist&quot;</span>))
lbls=[]
pxls=[]
<span>print</span>(len(training))

<span># Select the first 10000 training data and the labels</span>
<span>for</span> i <span>in</span> range(<span>10000</span>):
       l,p=training[i]
       lbls.append(l)
       pxls.append(p)
labels= np.array(lbls)
pixels=np.array(pxls)   

<span># Sey y=1 when labels == 5 and 0 otherwise</span>
y=(labels==<span>5</span>).reshape(-<span>1</span>,<span>1</span>)
X=pixels.reshape(pixels.shape[<span>0</span>],-<span>1</span>)

<span># Create the necessary feature and target variable</span>
X1=X.T
Y1=y.T

<span># Create the layer dimensions. The number of features are 28 x 28 = 784 since the 28 x 28</span>
<span># pixels is flattened to single vector of length 784.</span>
layersDimensions=[<span>784</span>, <span>15</span>,<span>9</span>,<span>7</span>,<span>1</span>] <span># Works very well</span>
parameters = L_Layer_DeepModel(X1, Y1, layersDimensions, hiddenActivationFunc=<span>'relu'</span>, learning_rate = <span>0.1</span>,num_iterations = <span>1000</span>, fig=<span>&quot;fig7.png&quot;</span>)

<span># Test data</span>
lbls1=[]
pxls1=[]
<span>for</span> i <span>in</span> range(<span>800</span>):
       l,p=test[i]
       lbls1.append(l)
       pxls1.append(p)
 
testLabels=np.array(lbls1)
testData=np.array(pxls1)

ytest=(testLabels==<span>5</span>).reshape(-<span>1</span>,<span>1</span>)
Xtest=testData.reshape(testData.shape[<span>0</span>],-<span>1</span>)
Xtest1=Xtest.T
Ytest1=ytest.T

yhat = predict(parameters,Xtest1)
<span>from</span> sklearn.metrics <span>import</span> confusion_matrix
a=confusion_matrix(Ytest1.T,yhat.T)
<span>from</span> sklearn.metrics <span>import</span> accuracy_score, precision_score, recall_score, f1_score
<span>print</span>(<span>'Accuracy: {:.2f}'</span>.format(accuracy_score(Ytest1.T, yhat.T)))
<span>print</span>(<span>'Precision: {:.2f}'</span>.format(precision_score(Ytest1.T, yhat.T)))
<span>print</span>(<span>'Recall: {:.2f}'</span>.format(recall_score(Ytest1.T, yhat.T)))
<span>print</span>(<span>'F1: {:.2f}'</span>.format(f1_score(Ytest1.T, yhat.T)))

probs=predict_proba(parameters,Xtest1)
<span>from</span> sklearn.metrics <span>import</span> precision_recall_curve

precision, recall, thresholds = precision_recall_curve(Ytest1.T, probs.T)
closest_zero = np.argmin(np.abs(thresholds))
closest_zero_p = precision[closest_zero]
closest_zero_r = recall[closest_zero]
plt.xlim([<span>0.0</span>, <span>1.01</span>])
plt.ylim([<span>0.0</span>, <span>1.01</span>])
plt.plot(precision, recall, label=<span>'Precision-Recall Curve'</span>)
plt.plot(closest_zero_p, closest_zero_r, <span>'o'</span>, markersize = <span>12</span>, fillstyle = <span>'none'</span>, c=<span>'r'</span>, mew=<span>3</span>)
plt.xlabel(<span>'Precision'</span>, fontsize=<span>16</span>)
plt.ylabel(<span>'Recall'</span>, fontsize=<span>16</span>)
plt.savefig(<span>&quot;fig8.png&quot;</span>,bbox_inches=<span>'tight'</span>)
</code></pre>
<pre><code>
## Accuracy: 0.99
## Precision: 0.96
## Recall: 0.89
## F1: 0.92</code></pre>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/01/fig71.png?w=456" data-orig-size="570,455" data-medium-file="https://gigadom.files.wordpress.com/2018/01/fig71.png?w=456?w=300" data-attachment-id="6264" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/fig71.png?w=456" data-permalink="https://gigadom.wordpress.com/2018/01/30/deep-learning-from-first-principles-in-python-r-and-octave-part-3/fig7-9/" data-large-file="https://gigadom.files.wordpress.com/2018/01/fig71.png?w=456?w=570" alt="" data-image-title="fig7" sizes="(max-width: 570px) 100vw, 570px" data-comments-opened="1" class="wp-image-6264" data-recalc-dims="1"></p>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/01/fig82.png?w=456" data-orig-size="579,442" data-medium-file="https://gigadom.files.wordpress.com/2018/01/fig82.png?w=456?w=300" data-attachment-id="6265" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/fig82.png?w=456" data-permalink="https://gigadom.wordpress.com/2018/01/30/deep-learning-from-first-principles-in-python-r-and-octave-part-3/fig8-6/" data-large-file="https://gigadom.files.wordpress.com/2018/01/fig82.png?w=456?w=579" alt="" data-image-title="fig8" sizes="(max-width: 579px) 100vw, 579px" data-comments-opened="1" class="wp-image-6265" data-recalc-dims="1"></p>
<p>In addition to plotting the Cost vs Iterations, I also plot the Precision-Recall curve to show how the Precision and Recall, which are complementary to each other vary with respect to the other. To know more about Precision-Recall, please check my post <a rel="nofollow" href="https://gigadom.wordpress.com/2017/10/29/practical-machine-learning-with-r-and-python-part-4/" target="_blank">Practical Machine Learning with R and Python – Part 4</a>. You could also check out my book <a rel="nofollow" href="https://gigadom.wordpress.com/2017/12/05/my-book-practical-machine-learning-with-r-and-python-on-amazon/" target="_blank">My book ‘Practical Machine Learning with R and Python’ on Amazon</a> for details on the key metrics and algorithms for classification and regression problems. A physical copy of the book is much better than scrolling down a webpage. Personally, I tend to use my own book quite frequently to refer to R, Python constructs,  subsetting, machine Learning function calls and the necessary parameters etc. It is useless to commit any of this to memory, and a physical copy of a book is much easier to thumb through for the relevant code snippet. Pick up your copy today!</p>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/01/untitled1.png" data-orig-size="780,247" data-medium-file="https://gigadom.files.wordpress.com/2018/01/untitled1.png?w=300" data-attachment-id="6284" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/untitled1.png?w=450&h=214&fit=676%2C214" data-permalink="https://gigadom.wordpress.com/2018/01/30/deep-learning-from-first-principles-in-python-r-and-octave-part-3/untitled-118/" data-large-file="https://gigadom.files.wordpress.com/2018/01/untitled1.png?w=450&amp;h=214&amp;fit=676%2C214" alt="" data-image-title="Untitled" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6284" data-recalc-dims="1"></p>
</div>
<div>
<h2>4b. Binary Classification using MNIST – R code</h2>
<p>In the R code below the same binary classification of the digit ‘5’ and the ‘not 5’ is performed. The code to read and display the MNIST data is taken from Brendan O’ Connor’s github link at <a rel="nofollow" href="https://gist.github.com/brendano/39760" target="_blank">MNIST</a></p>
<pre><code><span>source</span><span>(</span><span>&quot;mnist.R&quot;</span><span>)</span>
<span>load_mnist</span><span>(</span><span>)</span>
<span>#show_digit(train$x[2,]</span>
<span>layersDimensions</span><span>=</span><span>c</span><span>(</span><span>784</span>, <span>7</span>,<span>7</span>,<span>3</span>,<span>1</span><span>)</span> <span># Works at 1500</span>
<span>x</span> <span>&lt;-</span> <span>t</span><span>(</span><span>train</span><span>$</span><span>x</span><span>)</span>
<span># Choose only 5000 training data</span>
<span>x2</span> <span>&lt;-</span> <span>x</span><span>[</span>,<span>1</span><span>:</span><span>5000</span><span>]</span>
<span>y</span> <span>&lt;-</span><span>train</span><span>$</span><span>y</span>
<span># Set labels for all digits that are 'not 5' to 0</span>
<span>y</span><span>[</span><span>y</span><span>!=</span><span>5</span><span>]</span> <span>&lt;-</span> <span>0</span>
<span># Set labels of digit 5 as 1</span>
<span>y</span><span>[</span><span>y</span><span>==</span><span>5</span><span>]</span> <span>&lt;-</span> <span>1</span>
<span># Set the data</span>
<span>y1</span> <span>&lt;-</span> <span>as.matrix</span><span>(</span><span>y</span><span>)</span>
<span>y2</span> <span>&lt;-</span> <span>t</span><span>(</span><span>y1</span><span>)</span>
<span># Choose the 1st 5000 data</span>
<span>y3</span> <span>&lt;-</span> <span>y2</span><span>[</span>,<span>1</span><span>:</span><span>5000</span><span>]</span>

<span>#Execute the Deep Learning Model</span>
<span>retvals</span> <span>=</span> <span>L_Layer_DeepModel</span><span>(</span><span>x2</span>, <span>y3</span>, <span>layersDimensions</span>, 
                               <span>hiddenActivationFunc</span><span>=</span><span>'tanh'</span>, 
                               <span>learningRate</span> <span>=</span> <span>0.3</span>,
                               <span>numIterations</span> <span>=</span> <span>3000</span>, <span>print_cost</span> <span>=</span> <span>True</span><span>)</span></code></pre>
<pre><code><span># Plot cost vs iteration</span>
<span>costs</span> <span>&lt;-</span> <span>retvals</span><span>[</span><span>[</span><span>'costs'</span><span>]</span><span>]</span>
<span>numIterations</span> <span>=</span> <span>3000</span>
<span>iterations</span> <span>&lt;-</span> <span>seq</span><span>(</span><span>0</span>,<span>numIterations</span>,<span>by</span><span>=</span><span>1000</span><span>)</span>
<span>df</span> <span>&lt;-</span><span>data.frame</span><span>(</span><span>iterations</span>,<span>costs</span><span>)</span>
<span>ggplot</span><span>(</span><span>df</span>,<span>aes</span><span>(</span><span>x</span><span>=</span><span>iterations</span>,<span>y</span><span>=</span><span>costs</span><span>)</span><span>)</span> <span>+</span> <span>geom_point</span><span>(</span><span>)</span> <span>+</span><span>geom_line</span><span>(</span><span>color</span><span>=</span><span>&quot;blue&quot;</span><span>)</span> <span>+</span>
    <span>xlab</span><span>(</span><span>'No of iterations'</span><span>)</span> <span>+</span> <span>ylab</span><span>(</span><span>'Cost'</span><span>)</span> <span>+</span> <span>ggtitle</span><span>(</span><span>&quot;Cost vs No of iterations&quot;</span><span>)</span></code></pre>
</div>
<p><img><img data-orig-file="https://gigadom.files.wordpress.com/2018/01/fig4-11.png" data-orig-size="1344,960" data-medium-file="https://gigadom.files.wordpress.com/2018/01/fig4-11.png?w=300" data-attachment-id="6276" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/fig4-11.png?w=450&h=483&fit=676%2C483" data-permalink="https://gigadom.wordpress.com/2018/01/30/deep-learning-from-first-principles-in-python-r-and-octave-part-3/fig4-1-7/" data-large-file="https://gigadom.files.wordpress.com/2018/01/fig4-11.png?w=450&amp;h=483&amp;fit=676%2C483" alt="" data-image-title="fig4-1" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6276" data-recalc-dims="1"></p>
<pre><code><span># Compute probability scores</span>
<span>scores</span> <span>&lt;-</span> <span>computeScores</span><span>(</span><span>retvals</span><span>$</span><span>parameters</span>, <span>x2</span>,<span>hiddenActivationFunc</span><span>=</span><span>'relu'</span><span>)</span>
<span>a</span><span>=</span><span>y3</span><span>==</span><span>1</span>
<span>b</span><span>=</span><span>y3</span><span>==</span><span>0</span>

<span># Compute probabilities of class 0 and class 1</span>
<span>class1</span><span>=</span><span>scores</span><span>[</span><span>a</span><span>]</span>
<span>class0</span><span>=</span><span>scores</span><span>[</span><span>b</span><span>]</span>

<span># Plot ROC curve</span>
<span>pr</span> <span>&lt;-</span><span>pr.curve</span><span>(</span><span>scores.class0</span><span>=</span><span>class1</span>,
        <span>scores.class1</span><span>=</span><span>class0</span>,
       <span>curve</span><span>=</span><span>T</span><span>)</span>

<span>plot</span><span>(</span><span>pr</span><span>)</span></code></pre>
<div>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/01/fig4-2.png" data-orig-size="1344,960" data-medium-file="https://gigadom.files.wordpress.com/2018/01/fig4-2.png?w=300" data-attachment-id="6277" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/fig4-2.png?w=450&h=483&fit=676%2C483" data-permalink="https://gigadom.wordpress.com/2018/01/30/deep-learning-from-first-principles-in-python-r-and-octave-part-3/fig4-2-2/" data-large-file="https://gigadom.files.wordpress.com/2018/01/fig4-2.png?w=450&amp;h=483&amp;fit=676%2C483" alt="" data-image-title="fig4-2" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6277" data-recalc-dims="1"></p>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/01/fig4-3.png" data-orig-size="1344,960" data-medium-file="https://gigadom.files.wordpress.com/2018/01/fig4-3.png?w=300" data-attachment-id="6278" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/fig4-3.png?w=450&h=483&fit=676%2C483" data-permalink="https://gigadom.wordpress.com/2018/01/30/deep-learning-from-first-principles-in-python-r-and-octave-part-3/fig4-3-2/" data-large-file="https://gigadom.files.wordpress.com/2018/01/fig4-3.png?w=450&amp;h=483&amp;fit=676%2C483" alt="" data-image-title="fig4-3" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6278" data-recalc-dims="1"></p>
<p>The AUC curve hugs the top left corner and hence the performance of the classifier is quite good.</p>
</div>
<div>
<h2>4c. Binary Classification using MNIST – Octave code</h2>
<p>This code to load MNIST data was taken from <a rel="nofollow" href="http://daniel-e.github.io/2017-10-20-loading-mnist-handwritten-digits-with-octave-or-matlab/" target="_blank">Daniel E blog. </a><br>
Precision recall curves are available in Matlab but are yet to be implemented in Octave’s statistics package.<br>
<code><br>
load('./mnist/mnist.txt.gz');  % load the dataset<br>
# Subset the 'not 5' digits<br>
a=(trainY != 5);<br>
# Subset '5'<br>
b=(trainY == 5);<br>
#make a copy of trainY<br>
#Set 'not 5' as 0 and '5' as 1<br>
y=trainY;<br>
y(a)=0;<br>
y(b)=1;<br>
X=trainX(1:5000,:);<br>
Y=y(1:5000);<br>
# Set the dimensions of layer<br>
layersDimensions=[784, 7,7,3,1];<br>
# Compute the DL network<br>
[weights biases costs]=L_Layer_DeepModel(X', Y', layersDimensions,<br>
hiddenActivationFunc='relu',<br>
learningRate = 0.1,<br>
numIterations = 5000);<br>
</code></p>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/01/figure23.png" data-orig-size="1200,900" data-medium-file="https://gigadom.files.wordpress.com/2018/01/figure23.png?w=300" data-attachment-id="6281" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/figure23.png?w=450&h=507&fit=676%2C507" data-permalink="https://gigadom.wordpress.com/2018/01/30/deep-learning-from-first-principles-in-python-r-and-octave-part-3/figure23/" data-large-file="https://gigadom.files.wordpress.com/2018/01/figure23.png?w=450&amp;h=507&amp;fit=676%2C507" alt="" data-image-title="figure23" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6281" data-recalc-dims="1"></p>
<h1>Conclusion</h1>
<p>It was quite a challenge coding a Deep Learning Network in Python, R and Octave. The Deep Learning network implementation, in this post,is the base Deep Learning network, without any of the regularization methods included. Here are some key learning that I got while playing with different multi-layer networks on different problems</p>
<p>a. Deep Learning Networks come with many levers, the hyper-parameters,<br>
– learning rate<br>
– activation unit<br>
– number of hidden layers<br>
– number of units per hidden layer<br>
– number of iterations while performing gradient descent<br>
b. Deep Networks are very sensitive. A change in any of the hyper-parameter makes it perform very differently<br>
c. Initially I thought adding more hidden layers, or more units per hidden layer will make the DL network better at learning. On the contrary, there is a performance degradation after the optimal DL configuration<br>
d. At a sub-optimal number of hidden layers or number of hidden units, gradient descent seems to get stuck at a local minima<br>
e. There were occasions when the cost came down, only to increase slowly as the number of iterations were increased. Probably early stopping would have helped.<br>
f. I also did come across situations of ‘exploding/vanishing gradient’, cost went to Inf/-Inf. Here I would think inclusion of ‘momentum method’ would have helped</p>
<p>I intend to add the additional hyper-parameters of L1, L2 regularization, momentum method, early stopping etc. into the code in my future posts.<br>
Feel free to fork/clone the code from Github <a rel="nofollow" href="https://github.com/tvganesh/DeepLearning-Part3" target="_blank">Deep Learning – Part 3</a>, and take the DL network apart and play around with it.</p>
<p>I will be continuing this series with more hyper-parameters to handle vanishing and exploding gradients, early stopping and regularization in the weeks to come. I also intend to add some more activation functions to this basic Multi-Layer Network.<br>
Hang around, there are more exciting things to come.</p>
<p>Watch this space!</p>
</div>
<p>References<br>
1. <a rel="nofollow" href="https://www.coursera.org/specializations/deep-learning" target="_blank">Deep Learning Specialization</a><br>
2. <a rel="nofollow" href="https://www.coursera.org/learn/neural-networks" target="_blank">Neural Networks for Machine Learning</a><br>
3. <a rel="nofollow" href="http://www.deeplearningbook.org/" target="_blank">Deep Learning, Ian Goodfellow, Yoshua Bengio and Aaron Courville</a><br>
4. <a rel="nofollow" href="https://gigadom.wordpress.com/2017/01/21/neural-networks-the-mechanics-of-backpropagation/" target="_blank">Neural Networks: The mechanics of backpropagation</a><br>
5. <a rel="nofollow" href="https://www.coursera.org/learn/machine-learning" target="_blank">Machine Learning</a></p>
<p>Also see<br>
1.<a rel="nofollow" href="https://gigadom.wordpress.com/2017/12/05/my-book-practical-machine-learning-with-r-and-python-on-amazon/" target="_blank">My book ‘Practical Machine Learning with R and Python’ on Amazon</a><br>
2. <a rel="nofollow" href="https://gigadom.wordpress.com/2017/09/11/my-travels-through-the-realms-of-data-science-machine-learning-deep-learning-and-ai/" target="_blank">My travels through the realms of Data Science, Machine Learning, Deep Learning and (AI)</a><br>
3. <a rel="nofollow" href="https://gigadom.wordpress.com/2013/02/14/designing-a-social-web-portal/" target="_blank">Designing a Social Web Portal</a><br>
4. <a rel="nofollow" href="https://gigadom.wordpress.com/2017/01/05/googlyplus-yorkr-analyzes-ipl-players-teams-matches-with-plots-and-tables/" target="_blank">GooglyPlus: yorkr analyzes IPL players, teams, matches with plots and tables</a><br>
4. <a rel="nofollow" href="https://gigadom.wordpress.com/2016/06/23/introducing-qcsimulator-a-5-qubit-quantum-computing-simulator-in-r/" target="_blank">Introducing QCSimulator: A 5-qubit quantum computing simulator in R</a><br>
6. <a rel="nofollow" href="https://gigadom.wordpress.com/2013/07/23/presentation-on-intelligent-networks-camel-protocol-services-applications/" target="_blank">Presentation on “Intelligent Networks, CAMEL protocol, services &amp; applications</a><br>
7. <a rel="nofollow" href="https://gigadom.wordpress.com/2011/05/13/design-principles-of-scalable-distributed-systems/" target="_blank">Design Principles of Scalable, Distributed Systems</a></p>
<p>To see all posts see <a rel="nofollow" href="https://gigadom.wordpress.com/aa-2/" target="_blank">Index of posts</a></p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://gigadom.wordpress.com/2018/01/30/deep-learning-from-first-principles-in-python-r-and-octave-part-3/"> R – Giga thoughts …</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/rhrt6fd89AY" height="1"></a></div>