<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/deep-learning-from-first-principles-in-python-r-and-octave-part-1/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://gigadom.wordpress.com/2018/01/04/deep-learning-from-basic-principles-in-python-r-and-octave-part-1/"> R – Giga thoughts …</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><em>“You don’t perceive objects as they are. You perceive them as you are.”</em><br>
<em>“Your interpretation of physical objects has everything to do with the historical trajectory of your brain – and little to do with the objects themselves.”</em><br>
<em>“The brain generates its own reality, even before it receives information coming in from the eyes and the other senses. This is known as the internal model”</em></p>
<pre><code>                          David Eagleman - The Brain: The Story of You</code></pre>
<p>This is the first in the series of posts, I intend to write on Deep Learning. This post is inspired by the <a rel="nofollow" href="https://www.coursera.org/specializations/deep-learning" target="_blank">Deep Learning Specialization</a> by Prof Andrew Ng on Coursera and <a rel="nofollow" href="https://www.coursera.org/learn/neural-networks" target="_blank">Neural Networks for Machine Learning</a> by Prof Geoffrey Hinton also on Coursera. In this post I implement Logistic regression with a 2 layer Neural Network i.e. a Neural Network that just has an input layer and an output layer and with no hidden layer.<em>I am certain that any self-respecting Deep Learning/Neural Network would consider a Neural Network without hidden layers as no Neural Network at all!</em></p>
<p>This 2 layer network is implemented in Python, R and Octave languages. I have included Octave, into the mix, as Octave is a close cousin of Matlab. These implementations in Python, R and Octave are equivalent vectorized implementations. So, if you are familiar in any one of the languages, you should be able to look at the corresponding code in the other two. You can download this R Markdown file and Octave code from <a rel="nofollow" href="https://github.com/tvganesh/DeepLearning-Part1" target="_blank">DeepLearning -Part 1</a></p>
<p>To start with, Logistic Regression is performed using sklearn’s logistic regression package for the cancer data set also from sklearn. This is shown below</p>
<h2>1. Logistic Regression</h2>
<pre><code><span>import</span> numpy <span>as</span> np
<span>import</span> pandas <span>as</span> pd
<span>import</span> os
<span>import</span> matplotlib.pyplot <span>as</span> plt
<span>from</span> sklearn.model_selection <span>import</span> train_test_split
<span>from</span> sklearn.linear_model <span>import</span> LogisticRegression
<span>from</span> sklearn.datasets <span>import</span> make_classification, make_blobs

<span>from</span> sklearn.metrics <span>import</span> confusion_matrix
<span>from</span> matplotlib.colors <span>import</span> ListedColormap
<span>from</span> sklearn.datasets <span>import</span> load_breast_cancer
<span># Load the cancer data</span>
(X_cancer, y_cancer) = load_breast_cancer(return_X_y = <span>True</span>)
X_train, X_test, y_train, y_test = train_test_split(X_cancer, y_cancer,
                                                   random_state = <span>0</span>)
<span># Call the Logisitic Regression function</span>
clf = LogisticRegression().fit(X_train, y_train)
<span>print</span>(<span>'Accuracy of Logistic regression classifier on training set: {:.2f}'</span>
     .format(clf.score(X_train, y_train)))
<span>print</span>(<span>'Accuracy of Logistic regression classifier on test set: {:.2f}'</span>
     .format(clf.score(X_test, y_test)))</code></pre>
<pre><code>## Accuracy of Logistic regression classifier on training set: 0.96
## Accuracy of Logistic regression classifier on test set: 0.96</code></pre>
<p>To check on other classification algorithms, check my post <a rel="nofollow" href="https://gigadom.wordpress.com/2017/10/13/practical-machine-learning-with-r-and-python-part-2/" target="_blank">Practical Machine Learning with R and Python – Part 2</a>. You could also take a look at my book, available on Amazon , in which I implement the most popular Machine Learning algorithms in both R and Python – <a rel="nofollow" href="https://gigadom.wordpress.com/2017/12/05/my-book-practical-machine-learning-with-r-and-python-on-amazon/" target="_blank">My book ‘Practical Machine Learning with R and Python’ on Amazon</a></p>
<h2>2. Logistic Regression as a 2 layer Neural Network</h2>
<p>In the following section Logistic Regression is implemented as a 2 layer Neural Network in Python, R and Octave. The same cancer data set from sklearn will be used to train and test the Neural Network in Python, R and Octave. This can be represented diagrammatically as below</p>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/01/lr-11.png" data-orig-size="1280,720" data-medium-file="https://gigadom.files.wordpress.com/2018/01/lr-11.png?w=300" data-attachment-id="6154" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/lr-11.png?w=450&h=380&fit=676%2C380" data-permalink="https://gigadom.wordpress.com/2018/01/04/deep-learning-from-basic-principles-in-python-r-and-octave-part-1/lr-1-2/" data-large-file="https://gigadom.files.wordpress.com/2018/01/lr-11.png?w=450&amp;h=380&amp;fit=676%2C380" alt="" data-image-title="LR-1" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6154" data-recalc-dims="1"></p>
<p> </p>
<p>The cancer data set has 30 input features, and the target variable ‘output’ is either 0 or 1. Hence the sigmoid activation function will be used in the output layer for classification.</p>
<p>This simple 2 layer Neural Network is shown below<br>
At the input layer there are 30 features and the corresponding weights of these inputs which are initialized to small random values.<br>
<img alt="Z= w_{1}x_{1} +w_{2}x_{2} +..+ w_{30}x_{30} + b" title="Z= w_{1}x_{1} +w_{2}x_{2} +..+ w_{30}x_{30} + b" src="https://s0.wp.com/latex.php?latex=Z%3D+w_%7B1%7Dx_%7B1%7D+%2Bw_%7B2%7Dx_%7B2%7D+%2B..%2B+w_%7B30%7Dx_%7B30%7D+%2B+b&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
where ‘b’ is the bias term</p>
<p>The Activation function is the sigmoid function which is <img alt="a= 1/(1+e^{-z})" title="a= 1/(1+e^{-z})" src="https://s0.wp.com/latex.php?latex=a%3D+1%2F%281%2Be%5E%7B-z%7D%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
The Loss, when the sigmoid function is used in the output layer, is given by<br>
<img alt="L=-(ylog(a) + (1-y)log(1-a))" title="L=-(ylog(a) + (1-y)log(1-a))" src="https://s0.wp.com/latex.php?latex=L%3D-%28ylog%28a%29+%2B+%281-y%29log%281-a%29%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> (1)</p>
<h2>Gradient Descent</h2>
<h3>Forward propagation</h3>
<p>In forward propagation cycle of the Neural Network the output Z and the output of activation function, the sigmoid function, is first computed. Then using the output ‘y’ for the given features, the ‘Loss’ is computed using equation (1) above.</p>
<h3>Backward propagation</h3>
<p>The backward propagation cycle determines how the ‘Loss’ is impacted for small variations from the previous layers upto the input layer. In other words, backward propagation computes the changes in the weights at the input layer, which will minimize the loss. Several cycles of gradient descent are performed in the path of steepest descent to find the local minima. In other words the set of weights and biases, at the input layer, which will result in the lowest loss is computed by gradient descent. The weights at the input layer are decreased by a parameter known as the ‘learning rate’. Too big a ‘learning rate’ can overshoot the local minima, and too small a ‘learning rate’ can take a long time to reach the local minima. This is done for ‘m’ training examples.</p>
<p><strong>Chain rule of differentiation</strong><br>
Let y=f(u)<br>
and u=g(x) then<br>
<img alt="partial y/partial x = partial y/partial u * partial u/partial x" title="partial y/partial x = partial y/partial u * partial u/partial x" src="https://s0.wp.com/latex.php?latex=%5Cpartial+y%2F%5Cpartial+x+%3D+%5Cpartial+y%2F%5Cpartial+u+%2A+%5Cpartial+u%2F%5Cpartial+x&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"></p>
<p><strong>Derivative of sigmoid</strong><br>
<img alt="sigma=1/(1+e^{-z})" title="sigma=1/(1+e^{-z})" src="https://s0.wp.com/latex.php?latex=%5Csigma%3D1%2F%281%2Be%5E%7B-z%7D%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
Let <img alt="x= 1 + e^{-z}" title="x= 1 + e^{-z}" src="https://s0.wp.com/latex.php?latex=x%3D+1+%2B+e%5E%7B-z%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex">  then<br>
<img alt="sigma = 1/x" title="sigma = 1/x" src="https://s0.wp.com/latex.php?latex=%5Csigma+%3D+1%2Fx&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="partial sigma/partial x = -1/x^{2}" title="partial sigma/partial x = -1/x^{2}" src="https://s0.wp.com/latex.php?latex=%5Cpartial+%5Csigma%2F%5Cpartial+x+%3D+-1%2Fx%5E%7B2%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="partial x/partial z = -e^{-z}" title="partial x/partial z = -e^{-z}" src="https://s0.wp.com/latex.php?latex=%5Cpartial+x%2F%5Cpartial+z+%3D+-e%5E%7B-z%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
Using the chain rule of differentiation we get<br>
<img alt="partial sigma/partial z = partial sigma/partial x * partial x/partial z" title="partial sigma/partial z = partial sigma/partial x * partial x/partial z" src="https://s0.wp.com/latex.php?latex=%5Cpartial+%5Csigma%2F%5Cpartial+z+%3D+%5Cpartial+%5Csigma%2F%5Cpartial+x+%2A+%5Cpartial+x%2F%5Cpartial+z&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="=-1/(1+e^{-z})^{2}* -e^{-z} = e^{-z}/(1+e^{-z})^{2}" title="=-1/(1+e^{-z})^{2}* -e^{-z} = e^{-z}/(1+e^{-z})^{2}" src="https://s0.wp.com/latex.php?latex=%3D-1%2F%281%2Be%5E%7B-z%7D%29%5E%7B2%7D%2A+-e%5E%7B-z%7D+%3D+e%5E%7B-z%7D%2F%281%2Be%5E%7B-z%7D%29%5E%7B2%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
Therefore <img alt="partial sigma/partial z = sigma(1-sigma)" title="partial sigma/partial z = sigma(1-sigma)" src="https://s0.wp.com/latex.php?latex=%5Cpartial+%5Csigma%2F%5Cpartial+z+%3D+%5Csigma%281-%5Csigma%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex">        -(2)</p>
<p>The 3 equations for the 2 layer Neural Network representation of Logistic Regression are<br>
<img alt="L=-(y*log(a) + (1-y)*log(1-a))" title="L=-(y*log(a) + (1-y)*log(1-a))" src="https://s0.wp.com/latex.php?latex=L%3D-%28y%2Alog%28a%29+%2B+%281-y%29%2Alog%281-a%29%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex">      -(a)<br>
<img alt="a=1/(1+e^{-z})" title="a=1/(1+e^{-z})" src="https://s0.wp.com/latex.php?latex=a%3D1%2F%281%2Be%5E%7B-z%7D%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex">      -(b)<br>
<img alt="Z= w_{1}x_{1} +w_{2}x_{2} +...+ w_{30}x_{30} +b" title="Z= w_{1}x_{1} +w_{2}x_{2} +...+ w_{30}x_{30} +b" src="https://s0.wp.com/latex.php?latex=Z%3D+w_%7B1%7Dx_%7B1%7D+%2Bw_%7B2%7Dx_%7B2%7D+%2B...%2B+w_%7B30%7Dx_%7B30%7D+%2Bb&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex">   -(c)</p>
<p>The back propagation step requires the computation of <img alt="dL/dw_{i}" title="dL/dw_{i}" src="https://s0.wp.com/latex.php?latex=dL%2Fdw_%7Bi%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> and <img alt="dL/db_{i}" title="dL/db_{i}" src="https://s0.wp.com/latex.php?latex=dL%2Fdb_%7Bi%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex">. In the case of regression it would be <img alt="dE/dw_{i}" title="dE/dw_{i}" src="https://s0.wp.com/latex.php?latex=dE%2Fdw_%7Bi%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> and <img alt="dE/db_{i}" title="dE/db_{i}" src="https://s0.wp.com/latex.php?latex=dE%2Fdb_%7Bi%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> where dE is the Mean Squared Error function.<br>
Computing the derivatives for back propagation we have<br>
<img alt="dL/da = -(y/a + (1-y)/(1-a))" title="dL/da = -(y/a + (1-y)/(1-a))" src="https://s0.wp.com/latex.php?latex=dL%2Fda+%3D+-%28y%2Fa+%2B+%281-y%29%2F%281-a%29%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex">          -(d)<br>
because <img alt="d/dx(logx) = 1/x" title="d/dx(logx) = 1/x" src="https://s0.wp.com/latex.php?latex=d%2Fdx%28logx%29+%3D+1%2Fx&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
Also from equation (2) we get<br>
<img alt="da/dZ = a (1-a)" title="da/dZ = a (1-a)" src="https://s0.wp.com/latex.php?latex=da%2FdZ+%3D+a+%281-a%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex">                                  – (e)<br>
By chain rule<br>
<img alt="partial L/partial Z = partial L/partial a * partial a/partial Z" title="partial L/partial Z = partial L/partial a * partial a/partial Z" src="https://s0.wp.com/latex.php?latex=%5Cpartial+L%2F%5Cpartial+Z+%3D+%5Cpartial+L%2F%5Cpartial+a+%2A+%5Cpartial+a%2F%5Cpartial+Z&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
therefore substituting the results of (d) &amp; (e) we get<br>
<img alt="partial L/partial Z = -(y/a + (1-y)/(1-a)) * a(1-a) = a-y" title="partial L/partial Z = -(y/a + (1-y)/(1-a)) * a(1-a) = a-y" src="https://s0.wp.com/latex.php?latex=%5Cpartial+L%2F%5Cpartial+Z+%3D+-%28y%2Fa+%2B+%281-y%29%2F%281-a%29%29+%2A+a%281-a%29+%3D+a-y&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex">         (f)<br>
Finally<br>
<img alt="partial L/partial w_{i}= partial L/partial a * partial a/partial Z * partial Z/partial w_{i}" title="partial L/partial w_{i}= partial L/partial a * partial a/partial Z * partial Z/partial w_{i}" src="https://s0.wp.com/latex.php?latex=%5Cpartial+L%2F%5Cpartial+w_%7Bi%7D%3D+%5Cpartial+L%2F%5Cpartial+a+%2A+%5Cpartial+a%2F%5Cpartial+Z+%2A+%5Cpartial+Z%2F%5Cpartial+w_%7Bi%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex">                                                           -(g)<br>
<img alt="partial Z/partial w_{i} = x_{i}" title="partial Z/partial w_{i} = x_{i}" src="https://s0.wp.com/latex.php?latex=%5Cpartial+Z%2F%5Cpartial+w_%7Bi%7D+%3D+x_%7Bi%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex">            – (h)<br>
and from (f) we have  <img alt="partial L/partial Z =a-y" title="partial L/partial Z =a-y" src="https://s0.wp.com/latex.php?latex=%5Cpartial+L%2F%5Cpartial+Z+%3Da-y&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
Therefore  (g) reduces to<br>
<img alt="partial L/partial w_{i} = x_{i}* (a-y)" title="partial L/partial w_{i} = x_{i}* (a-y)" src="https://s0.wp.com/latex.php?latex=%5Cpartial+L%2F%5Cpartial+w_%7Bi%7D+%3D+x_%7Bi%7D%2A+%28a-y%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> -(i)<br>
Also<br>
<img alt="partial L/partial b = partial L/partial a * partial a/partial Z * partial Z/partial b" title="partial L/partial b = partial L/partial a * partial a/partial Z * partial Z/partial b" src="https://s0.wp.com/latex.php?latex=%5Cpartial+L%2F%5Cpartial+b+%3D+%5Cpartial+L%2F%5Cpartial+a+%2A+%5Cpartial+a%2F%5Cpartial+Z+%2A+%5Cpartial+Z%2F%5Cpartial+b&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> -(j)<br>
Since<br>
<img alt="partial Z/partial b = 1" title="partial Z/partial b = 1" src="https://s0.wp.com/latex.php?latex=%5Cpartial+Z%2F%5Cpartial+b+%3D+1&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="partial L/partial b = a-y" title="partial L/partial b = a-y" src="https://s0.wp.com/latex.php?latex=%5Cpartial+L%2F%5Cpartial+b+%3D+a-y&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"></p>
<p>The gradient computes the weights at the input layer and the corresponding bias by using the values<br>
of <img alt="dw_{i}" title="dw_{i}" src="https://s0.wp.com/latex.php?latex=dw_%7Bi%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> and <img alt="db" title="db" src="https://s0.wp.com/latex.php?latex=db&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="w_{i} := w_{i} -alpha * dw_{i}" title="w_{i} := w_{i} -alpha * dw_{i}" src="https://s0.wp.com/latex.php?latex=w_%7Bi%7D+%3A%3D+w_%7Bi%7D+-%5Calpha+%2A+dw_%7Bi%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="b := b -alpha * db" title="b := b -alpha * db" src="https://s0.wp.com/latex.php?latex=b+%3A%3D+b+-%5Calpha+%2A+db&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
I found the computation graph representation in the book Deep Learning: Ian Goodfellow, Yoshua Bengio, Aaron Courville, very useful to visualize and also compute the backward propagation. For the 2 layer Neural Network of Logistic Regression the computation graph is shown below</p>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/01/lr-1.png" data-orig-size="1280,720" data-medium-file="https://gigadom.files.wordpress.com/2018/01/lr-1.png?w=300" data-attachment-id="6141" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/lr-1.png?w=450&h=380&fit=676%2C380" data-permalink="https://gigadom.wordpress.com/2018/01/04/deep-learning-from-basic-principles-in-python-r-and-octave-part-1/lr-1/" data-large-file="https://gigadom.files.wordpress.com/2018/01/lr-1.png?w=450&amp;h=380&amp;fit=676%2C380" alt="" data-image-title="LR-1" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6141" data-recalc-dims="1"></p>
<h3>3. Neural Network for Logistic Regression -Python code (vectorized)</h3>
<pre><code><span>import</span> numpy <span>as</span> np
<span>import</span> pandas <span>as</span> pd
<span>import</span> os
<span>import</span> matplotlib.pyplot <span>as</span> plt
<span>from</span> sklearn.model_selection <span>import</span> train_test_split

<span># Define the sigmoid function</span>
<span><span>def</span> <span>sigmoid</span><span>(z)</span>:</span>  
    a=<span>1</span>/(<span>1</span>+np.exp(-z))    
    <span>return</span> a

<span># Initialize</span>
<span><span>def</span> <span>initialize</span><span>(dim)</span>:</span>
    w = np.zeros(dim).reshape(dim,<span>1</span>)
    b = <span>0</span>   
    <span>return</span> w

<span># Compute the loss</span>
<span><span>def</span> <span>computeLoss</span><span>(numTraining,Y,A)</span>:</span>
    loss=-<span>1</span>/numTraining *np.sum(Y*np.log(A) + (<span>1</span>-Y)*(np.log(<span>1</span>-A)))
    <span>return</span>(loss)

<span># Execute the forward propagation</span>
<span><span>def</span> <span>forwardPropagation</span><span>(w,b,X,Y)</span>:</span>
    <span># Compute Z</span>
    Z=np.dot(w.T,X)+b
    <span># Determine the number of training samples</span>
    numTraining=float(len(X))
    <span># Compute the output of the sigmoid activation function </span>
    A=sigmoid(Z)
    <span>#Compute the loss</span>
    loss = computeLoss(numTraining,Y,A)
    <span># Compute the gradients dZ, dw and db</span>
    dZ=A-Y
    dw=<span>1</span>/numTraining*np.dot(X,dZ.T)
    db=<span>1</span>/numTraining*np.sum(dZ)
    
    <span># Return the results as a dictionary</span>
    gradients = {<span>&quot;dw&quot;</span>: dw,
             <span>&quot;db&quot;</span>: db}
    loss = np.squeeze(loss)
    <span>return</span> gradients,loss

<span># Compute Gradient Descent </span>
<span><span>def</span> <span>gradientDescent</span><span>(w, b, X, Y, numIerations, learningRate)</span>:</span>
    losses=[]
    idx =[]
    <span># Iterate </span>
    <span>for</span> i <span>in</span> range(numIerations):
        gradients,loss=forwardPropagation(w,b,X,Y)
        <span>#Get the derivates</span>
        dw = gradients[<span>&quot;dw&quot;</span>]
        db = gradients[<span>&quot;db&quot;</span>]
        w = w-learningRate*dw
        b = b-learningRate*db

        <span># Store the loss</span>
        <span>if</span> i % <span>100</span> == <span>0</span>:
            idx.append(i)
            losses.append(loss)      
        <span># Set params and grads</span>
        params = {<span>&quot;w&quot;</span>: w,
                  <span>&quot;b&quot;</span>: b}  
        grads = {<span>&quot;dw&quot;</span>: dw,
                 <span>&quot;db&quot;</span>: db}
    
    <span>return</span> params, grads, losses,idx

<span># Predict the output for a training set </span>
<span><span>def</span> <span>predict</span><span>(w,b,X)</span>:</span>
    size=X.shape[<span>1</span>]
    yPredicted=np.zeros((<span>1</span>,size))
    Z=np.dot(w.T,X)
    <span># Compute the sigmoid</span>
    A=sigmoid(Z)
    <span>for</span> i <span>in</span> range(A.shape[<span>1</span>]):
        <span>#If the value is > 0.5 then set as 1</span>
        <span>if</span>(A[<span>0</span>][i] > <span>0.5</span>):
            yPredicted[<span>0</span>][i]=<span>1</span>
        <span>else</span>:
        <span># Else set as 0</span>
            yPredicted[<span>0</span>][i]=<span>0</span>

    <span>return</span> yPredicted

<span>#Normalize the data </span>
<span><span>def</span> <span>normalize</span><span>(x)</span>:</span>
    x_norm = <span>None</span>
    x_norm = np.linalg.norm(x,axis=<span>1</span>,keepdims=<span>True</span>)
    x= x/x_norm
    <span>return</span> x

   
<span># Run the 2 layer Neural Network on the cancer data set</span>

<span>from</span> sklearn.datasets <span>import</span> load_breast_cancer
<span># Load the cancer data</span>
(X_cancer, y_cancer) = load_breast_cancer(return_X_y = <span>True</span>)
<span># Create train and test sets</span>
X_train, X_test, y_train, y_test = train_test_split(X_cancer, y_cancer,
                                                   random_state = <span>0</span>)
<span># Normalize the data for better performance</span>
X_train1=normalize(X_train)


<span># Create weight vectors of zeros. The size is the number of features in the data set=30</span>
w=np.zeros((X_train.shape[<span>1</span>],<span>1</span>))
<span>#w=np.zeros((30,1))</span>
b=<span>0</span>

<span>#Normalize the training data so that gradient descent performs better</span>
X_train1=normalize(X_train)
<span>#Transpose X_train so that we have a matrix as (features, numSamples)</span>
X_train2=X_train1.T

<span># Reshape to remove the rank 1 array and then transpose</span>
y_train1=y_train.reshape(len(y_train),<span>1</span>)
y_train2=y_train1.T

<span># Run gradient descent for 4000 times and compute the weights</span>
parameters, grads, costs,idx = gradientDescent(w, b, X_train2, y_train2, numIerations=<span>4000</span>, learningRate=<span>0.75</span>)
w = parameters[<span>&quot;w&quot;</span>]
b = parameters[<span>&quot;b&quot;</span>]
   

<span># Normalize X_test</span>
X_test1=normalize(X_test)
<span>#Transpose X_train so that we have a matrix as (features, numSamples)</span>
X_test2=X_test1.T

<span>#Reshape y_test</span>
y_test1=y_test.reshape(len(y_test),<span>1</span>)
y_test2=y_test1.T

<span># Predict the values for </span>
yPredictionTest = predict(w, b, X_test2)
yPredictionTrain = predict(w, b, X_train2)

<span># Print the accuracy</span>
<span>print</span>(<span>&quot;train accuracy: {} %&quot;</span>.format(<span>100</span> - np.mean(np.abs(yPredictionTrain - y_train2)) * <span>100</span>))
<span>print</span>(<span>&quot;test accuracy: {} %&quot;</span>.format(<span>100</span> - np.mean(np.abs(yPredictionTest - y_test)) * <span>100</span>))

<span># Plot the Costs vs the number of iterations</span>
fig1=plt.plot(idx,costs)
fig1=plt.title(<span>&quot;Gradient descent-Cost vs No of iterations&quot;</span>)
fig1=plt.xlabel(<span>&quot;No of iterations&quot;</span>)
fig1=plt.ylabel(<span>&quot;Cost&quot;</span>)
fig1.figure.savefig(<span>&quot;fig1&quot;</span>, bbox_inches=<span>'tight'</span>)</code></pre>
<pre><code>## train accuracy: 90.3755868545 %
## test accuracy: 89.5104895105 %</code></pre>
<p><strong>Note</strong>: It can be seen that the Accuracy on the training and test set is 90.37% and 89.51%. This is comparatively poorer than the 96% which the logistic regression of sklearn achieves! But this is mainly because of the absence of hidden layers which is the real power of neural networks.</p>
<h3><img data-orig-file="https://gigadom.files.wordpress.com/2018/01/fig1.png?w=456" data-orig-size="569,455" data-medium-file="https://gigadom.files.wordpress.com/2018/01/fig1.png?w=456?w=300" data-attachment-id="6133" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/fig1.png?w=456" data-permalink="https://gigadom.wordpress.com/2018/01/04/deep-learning-from-basic-principles-in-python-r-and-octave-part-1/fig1-12/" data-large-file="https://gigadom.files.wordpress.com/2018/01/fig1.png?w=456?w=569" alt="" data-image-title="fig1" sizes="(max-width: 569px) 100vw, 569px" data-comments-opened="1" class="wp-image-6133" data-recalc-dims="1"></h3>
<h3>4. Neural Network for Logistic Regression -R code (vectorized)</h3>
<pre><code><span>source</span><span>(</span><span>&quot;RFunctions-1.R&quot;</span><span>)</span></code><code></code><code></code></pre>
<pre><code><span># Define the sigmoid function</span>
<span>sigmoid</span> <span>&lt;-</span> <span>function</span><span>(</span><span>z</span><span>)</span><span>{</span>
    <span>a</span> <span>&lt;-</span> <span>1</span><span>/</span><span>(</span><span>1</span><span>+</span> <span>exp</span><span>(</span><span>-</span><span>z</span><span>)</span><span>)</span>
    <span>a</span>
<span>}</span>

<span># Compute the loss</span>
<span>computeLoss</span> <span>&lt;-</span> <span>function</span><span>(</span><span>numTraining</span>,<span>Y</span>,<span>A</span><span>)</span><span>{</span>
    <span>loss</span> <span>&lt;-</span> <span>-</span><span>1</span><span>/</span><span>numTraining</span><span>*</span> <span>sum</span><span>(</span><span>Y</span><span>*</span><span>log</span><span>(</span><span>A</span><span>)</span> <span>+</span> <span>(</span><span>1</span><span>-</span><span>Y</span><span>)</span><span>*</span><span>log</span><span>(</span><span>1</span><span>-</span><span>A</span><span>)</span><span>)</span>
    <span>return</span><span>(</span><span>loss</span><span>)</span>
<span>}</span>

<span># Compute forward propagation</span>
<span>forwardPropagation</span> <span>&lt;-</span> <span>function</span><span>(</span><span>w</span>,<span>b</span>,<span>X</span>,<span>Y</span><span>)</span><span>{</span>
    <span># Compute Z</span>
    <span>Z</span> <span>&lt;-</span> <span>t</span><span>(</span><span>w</span><span>)</span> <span>%*%</span> <span>X</span> <span>+</span><span>b</span>
    <span>#Set the number of samples</span>
    <span>numTraining</span> <span>&lt;-</span> <span>ncol</span><span>(</span><span>X</span><span>)</span>
    <span># Compute the activation function</span>
    <span>A</span><span>=</span><span>sigmoid</span><span>(</span><span>Z</span><span>)</span> 
    
    <span>#Compute the loss</span>
    <span>loss</span> <span>&lt;-</span> <span>computeLoss</span><span>(</span><span>numTraining</span>,<span>Y</span>,<span>A</span><span>)</span>
    
    <span># Compute the gradients dZ, dw and db</span>
    <span>dZ</span><span>&lt;-</span><span>A</span><span>-</span><span>Y</span>
    <span>dw</span><span>&lt;-</span><span>1</span><span>/</span><span>numTraining</span> <span>*</span> <span>X</span> <span>%*%</span> <span>t</span><span>(</span><span>dZ</span><span>)</span>
    <span>db</span><span>&lt;-</span><span>1</span><span>/</span><span>numTraining</span><span>*</span><span>sum</span><span>(</span><span>dZ</span><span>)</span>
    
    <span>fwdProp</span> <span>&lt;-</span> <span>list</span><span>(</span><span>&quot;loss&quot;</span> <span>=</span> <span>loss</span>, <span>&quot;dw&quot;</span> <span>=</span> <span>dw</span>, <span>&quot;db&quot;</span> <span>=</span> <span>db</span><span>)</span>
    <span>return</span><span>(</span><span>fwdProp</span><span>)</span>
<span>}</span>

<span># Perform one cycle of Gradient descent</span>
<span>gradientDescent</span> <span>&lt;-</span> <span>function</span><span>(</span><span>w</span>, <span>b</span>, <span>X</span>, <span>Y</span>, <span>numIerations</span>, <span>learningRate</span><span>)</span><span>{</span>
    <span>losses</span> <span>&lt;-</span> <span>NULL</span>
    <span>idx</span> <span>&lt;-</span> <span>NULL</span>
    <span># Loop through the number of iterations</span>
    <span>for</span><span>(</span><span>i</span> <span>in</span> <span>1</span><span>:</span><span>numIerations</span><span>)</span><span>{</span>
        <span>fwdProp</span> <span>&lt;-</span><span>forwardPropagation</span><span>(</span><span>w</span>,<span>b</span>,<span>X</span>,<span>Y</span><span>)</span>
        <span>#Get the derivatives</span>
        <span>dw</span> <span>&lt;-</span> <span>fwdProp</span><span>$</span><span>dw</span>
        <span>db</span> <span>&lt;-</span> <span>fwdProp</span><span>$</span><span>db</span>
        <span>#Perform gradient descent</span>
        <span>w</span> <span>=</span> <span>w</span><span>-</span><span>learningRate</span><span>*</span><span>dw</span>
        <span>b</span> <span>=</span> <span>b</span><span>-</span><span>learningRate</span><span>*</span><span>db</span>
        <span>l</span> <span>&lt;-</span> <span>fwdProp</span><span>$</span><span>loss</span>
        <span># Stoe the loss</span>
        <span>if</span><span>(</span><span>i</span> <span>%%</span> <span>100</span> <span>==</span> <span>0</span><span>)</span><span>{</span>
            <span>idx</span> <span>&lt;-</span> <span>c</span><span>(</span><span>idx</span>,<span>i</span><span>)</span>
            <span>losses</span> <span>&lt;-</span> <span>c</span><span>(</span><span>losses</span>,<span>l</span><span>)</span>  
        <span>}</span>
    <span>}</span>
    
    <span># Return the weights and losses</span>
    <span>gradDescnt</span> <span>&lt;-</span> <span>list</span><span>(</span><span>&quot;w&quot;</span><span>=</span><span>w</span>,<span>&quot;b&quot;</span><span>=</span><span>b</span>,<span>&quot;dw&quot;</span><span>=</span><span>dw</span>,<span>&quot;db&quot;</span><span>=</span><span>db</span>,<span>&quot;losses&quot;</span><span>=</span><span>losses</span>,<span>&quot;idx&quot;</span><span>=</span><span>idx</span><span>)</span>
   
    <span>return</span><span>(</span><span>gradDescnt</span><span>)</span>
<span>}</span>

<span># Compute the predicted value for input</span>
<span>predict</span> <span>&lt;-</span> <span>function</span><span>(</span><span>w</span>,<span>b</span>,<span>X</span><span>)</span><span>{</span>
    <span>m</span><span>=</span><span>dim</span><span>(</span><span>X</span><span>)</span><span>[</span><span>2</span><span>]</span>
    <span># Create a ector of 0's</span>
    <span>yPredicted</span><span>=</span><span>matrix</span><span>(</span><span>rep</span><span>(</span><span>0</span>,<span>m</span><span>)</span>,<span>nrow</span><span>=</span><span>1</span>,<span>ncol</span><span>=</span><span>m</span><span>)</span>
    <span>Z</span> <span>&lt;-</span> <span>t</span><span>(</span><span>w</span><span>)</span> <span>%*%</span> <span>X</span> <span>+</span><span>b</span>
    <span># Compute sigmoid</span>
    <span>A</span><span>=</span><span>sigmoid</span><span>(</span><span>Z</span><span>)</span>
    <span>for</span><span>(</span><span>i</span> <span>in</span> <span>1</span><span>:</span><span>dim</span><span>(</span><span>A</span><span>)</span><span>[</span><span>2</span><span>]</span><span>)</span><span>{</span>
        <span># If A > 0.5 set value as 1</span>
        <span>if</span><span>(</span><span>A</span><span>[</span><span>1</span>,<span>i</span><span>]</span> <span>></span> <span>0.5</span><span>)</span>
        <span>yPredicted</span><span>[</span><span>1</span>,<span>i</span><span>]</span><span>=</span><span>1</span>
       <span>else</span>
        <span># Else set as 0</span>
        <span>yPredicted</span><span>[</span><span>1</span>,<span>i</span><span>]</span><span>=</span><span>0</span>
    <span>}</span>

    <span>return</span><span>(</span><span>yPredicted</span><span>)</span>
<span>}</span>

<span># Normalize the matrix</span>
<span>normalize</span> <span>&lt;-</span> <span>function</span><span>(</span><span>x</span><span>)</span><span>{</span>
    <span>#Create the norm of the matrix.Perform the Frobenius norm of the matrix </span>
    <span>n</span><span>&lt;-</span><span>as.matrix</span><span>(</span><span>sqrt</span><span>(</span><span>rowSums</span><span>(</span><span>x</span><span>^</span><span>2</span><span>)</span><span>)</span><span>)</span>
    <span>#Sweep by rows by norm. Note '1' in the function which performing on every row</span>
    <span>normalized</span><span>&lt;-</span><span>sweep</span><span>(</span><span>x</span>, <span>1</span>, <span>n</span>, <span>FUN</span><span>=</span><span>&quot;/&quot;</span><span>)</span>
    <span>return</span><span>(</span><span>normalized</span><span>)</span>
<span>}</span>

<span># Run the 2 layer Neural Network on the cancer data set</span>
<span># Read the data (from sklearn)</span>
<span>cancer</span> <span>&lt;-</span> <span>read.csv</span><span>(</span><span>&quot;cancer.csv&quot;</span><span>)</span>
<span># Rename the target variable</span>
<span>names</span><span>(</span><span>cancer</span><span>)</span> <span>&lt;-</span> <span>c</span><span>(</span><span>seq</span><span>(</span><span>1</span>,<span>30</span><span>)</span>,<span>&quot;output&quot;</span><span>)</span>
<span># Split as training and test sets</span>
<span>train_idx</span> <span>&lt;-</span> <span>trainTestSplit</span><span>(</span><span>cancer</span>,<span>trainPercent</span><span>=</span><span>75</span>,<span>seed</span><span>=</span><span>5</span><span>)</span>
<span>train</span> <span>&lt;-</span> <span>cancer</span><span>[</span><span>train_idx</span>, <span>]</span>
<span>test</span> <span>&lt;-</span> <span>cancer</span><span>[</span><span>-</span><span>train_idx</span>, <span>]</span>

<span># Set the features</span>
<span>X_train</span> <span>&lt;-</span><span>train</span><span>[</span>,<span>1</span><span>:</span><span>30</span><span>]</span>
<span>y_train</span> <span>&lt;-</span> <span>train</span><span>[</span>,<span>31</span><span>]</span>
<span>X_test</span> <span>&lt;-</span> <span>test</span><span>[</span>,<span>1</span><span>:</span><span>30</span><span>]</span>
<span>y_test</span> <span>&lt;-</span> <span>test</span><span>[</span>,<span>31</span><span>]</span>
<span># Create a matrix of 0's with the number of features</span>
<span>w</span> <span>&lt;-</span><span>matrix</span><span>(</span><span>rep</span><span>(</span><span>0</span>,<span>dim</span><span>(</span><span>X_train</span><span>)</span><span>[</span><span>2</span><span>]</span><span>)</span><span>)</span>
<span>b</span> <span>&lt;-</span><span>0</span>
<span>X_train1</span> <span>&lt;-</span> <span>normalize</span><span>(</span><span>X_train</span><span>)</span>
<span>X_train2</span><span>=</span><span>t</span><span>(</span><span>X_train1</span><span>)</span>

<span># Reshape then transpose</span>
<span>y_train1</span><span>=</span><span>as.matrix</span><span>(</span><span>y_train</span><span>)</span>
<span>y_train2</span><span>=</span><span>t</span><span>(</span><span>y_train1</span><span>)</span>

<span># Perform gradient descent</span>
<span>gradDescent</span><span>=</span> <span>gradientDescent</span><span>(</span><span>w</span>, <span>b</span>, <span>X_train2</span>, <span>y_train2</span>, <span>numIerations</span><span>=</span><span>3000</span>, <span>learningRate</span><span>=</span><span>0.77</span><span>)</span>


<span># Normalize X_test</span>
<span>X_test1</span><span>=</span><span>normalize</span><span>(</span><span>X_test</span><span>)</span>
<span>#Transpose X_train so that we have a matrix as (features, numSamples)</span>
<span>X_test2</span><span>=</span><span>t</span><span>(</span><span>X_test1</span><span>)</span>

<span>#Reshape y_test and take transpose</span>
<span>y_test1</span><span>=</span><span>as.matrix</span><span>(</span><span>y_test</span><span>)</span>
<span>y_test2</span><span>=</span><span>t</span><span>(</span><span>y_test1</span><span>)</span>

<span># Use the values of the weights generated from Gradient Descent</span>
<span>yPredictionTest</span> <span>=</span> <span>predict</span><span>(</span><span>gradDescent</span><span>$</span><span>w</span>, <span>gradDescent</span><span>$</span><span>b</span>, <span>X_test2</span><span>)</span>
<span>yPredictionTrain</span> <span>=</span> <span>predict</span><span>(</span><span>gradDescent</span><span>$</span><span>w</span>, <span>gradDescent</span><span>$</span><span>b</span>, <span>X_train2</span><span>)</span>

<span>sprintf</span><span>(</span><span>&quot;Train accuracy: %f&quot;</span>,<span>(</span><span>100</span> <span>-</span> <span>mean</span><span>(</span><span>abs</span><span>(</span><span>yPredictionTrain</span> <span>-</span> <span>y_train2</span><span>)</span><span>)</span> <span>*</span> <span>100</span><span>)</span><span>)</span></code></pre>
<pre><code>## [1] &quot;Train accuracy: 90.845070&quot;</code></pre>
<pre><code><span>sprintf</span><span>(</span><span>&quot;test accuracy: %f&quot;</span>,<span>(</span><span>100</span> <span>-</span> <span>mean</span><span>(</span><span>abs</span><span>(</span><span>yPredictionTest</span> <span>-</span> <span>y_test</span><span>)</span><span>)</span> <span>*</span> <span>100</span><span>)</span><span>)</span></code></pre>
<pre><code>## [1] &quot;test accuracy: 87.323944&quot;</code></pre>
<pre><code><span>df</span> <span>&lt;-</span><span>data.frame</span><span>(</span><span>gradDescent</span><span>$</span><span>idx</span>, <span>gradDescent</span><span>$</span><span>losses</span><span>)</span>
<span>names</span><span>(</span><span>df</span><span>)</span> <span>&lt;-</span> <span>c</span><span>(</span><span>&quot;iterations&quot;</span>,<span>&quot;losses&quot;</span><span>)</span>
<span>ggplot</span><span>(</span><span>df</span>,<span>aes</span><span>(</span><span>x</span><span>=</span><span>iterations</span>,<span>y</span><span>=</span><span>losses</span><span>)</span><span>)</span> <span>+</span> <span>geom_point</span><span>(</span><span>)</span> <span>+</span> <span>geom_line</span><span>(</span><span>col</span><span>=</span><span>&quot;blue&quot;</span><span>)</span> <span>+</span>
    <span>ggtitle</span><span>(</span><span>&quot;Gradient Descent - Losses vs No of Iterations&quot;</span><span>)</span> <span>+</span>
    <span>xlab</span><span>(</span><span>&quot;No of iterations&quot;</span><span>)</span> <span>+</span> <span>ylab</span><span>(</span><span>&quot;Losses&quot;</span><span>)</span></code></pre>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/01/fig-1-1.png" data-orig-size="1344,960" data-medium-file="https://gigadom.files.wordpress.com/2018/01/fig-1-1.png?w=300" data-attachment-id="6124" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/fig-1-1.png?w=450&h=483&fit=676%2C483" data-permalink="https://gigadom.wordpress.com/2018/01/04/deep-learning-from-basic-principles-in-python-r-and-octave-part-1/fig-1-1/" data-large-file="https://gigadom.files.wordpress.com/2018/01/fig-1-1.png?w=450&amp;h=483&amp;fit=676%2C483" alt="" data-image-title="fig 1-1" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6124" data-recalc-dims="1"></p>
<h3>4. Neural Network for Logistic Regression -Octave code (vectorized)</h3>
<p><code><br>
1;<br>
# Define sigmoid function<br>
function a = sigmoid(z)<br>
a = 1 ./ (1+ exp(-z));<br>
end<br>
# Compute the loss<br>
function loss=computeLoss(numtraining,Y,A)<br>
loss = -1/numtraining * sum((Y .* log(A)) + (1-Y) .* log(1-A));<br>
end</code><br>
<code><br>
# Perform forward propagation<br>
function [loss,dw,db,dZ] = forwardPropagation(w,b,X,Y)<br>
% Compute Z<br>
Z = w' * X + b;<br>
numtraining = size(X)(1,2);<br>
# Compute sigmoid<br>
A = sigmoid(Z);</code><br>
<code><br>
#Compute loss. Note this is element wise product<br>
loss =computeLoss(numtraining,Y,A);<br>
# Compute the gradients dZ, dw and db<br>
dZ = A-Y;<br>
dw = 1/numtraining* X * dZ';<br>
db =1/numtraining*sum(dZ);</code></p>
<p>end<br>
<code><br>
# Compute Gradient Descent<br>
function [w,b,dw,db,losses,index]=gradientDescent(w, b, X, Y, numIerations, learningRate)<br>
#Initialize losses and idx<br>
losses=[];<br>
index=[];<br>
# Loop through the number of iterations<br>
for i=1:numIerations,<br>
[loss,dw,db,dZ] = forwardPropagation(w,b,X,Y);<br>
# Perform Gradient descent<br>
w = w - learningRate*dw;<br>
b = b - learningRate*db;<br>
if(mod(i,100) ==0)<br>
# Append index and loss<br>
index = [index i];<br>
losses = [losses loss];<br>
endif</code></p>
<p>end<br>
end<br>
<code><br>
# Determine the predicted value for dataset<br>
function yPredicted = predict(w,b,X)<br>
m = size(X)(1,2);<br>
yPredicted=zeros(1,m);<br>
# Compute Z<br>
Z = w' * X + b;<br>
# Compute sigmoid<br>
A = sigmoid(Z);<br>
for i=1:size(X)(1,2),<br>
# Set predicted as 1 if A > 0,5<br>
if(A(1,i) >= 0.5)<br>
yPredicted(1,i)=1;<br>
else<br>
yPredicted(1,i)=0;<br>
endif<br>
end<br>
end</code><br>
<code><br>
# Normalize by dividing each value by the sum of squares<br>
function normalized = normalize(x)<br>
# Compute Frobenius norm. Square the elements, sum rows and then find square root<br>
a = sqrt(sum(x .^ 2,2));<br>
# Perform element wise division<br>
normalized = x ./ a;<br>
end</code><br>
<code><br>
# Split into train and test sets<br>
function [X_train,y_train,X_test,y_test] = trainTestSplit(dataset,trainPercent)<br>
# Create a random index<br>
ix = randperm(length(dataset));<br>
# Split into training<br>
trainSize = floor(trainPercent/100 * length(dataset));<br>
train=dataset(ix(1:trainSize),:);<br>
# And test<br>
test=dataset(ix(trainSize+1:length(dataset)),:);<br>
X_train = train(:,1:30);<br>
y_train = train(:,31);<br>
X_test = test(:,1:30);<br>
y_test = test(:,31);<br>
end</code></p>
<p><code><br>
cancer=csvread(&quot;cancer.csv&quot;);<br>
[X_train,y_train,X_test,y_test] = trainTestSplit(cancer,75);<br>
w=zeros(size(X_train)(1,2),1);<br>
b=0;<br>
X_train1=normalize(X_train);<br>
X_train2=X_train1';<br>
y_train1=y_train';<br>
[w1,b1,dw,db,losses,idx]=gradientDescent(w, b, X_train2, y_train1, numIerations=3000, learningRate=0.75);<br>
# Normalize X_test<br>
X_test1=normalize(X_test);<br>
#Transpose X_train so that we have a matrix as (features, numSamples)<br>
X_test2=X_test1';<br>
y_test1=y_test';<br>
# Use the values of the weights generated from Gradient Descent<br>
yPredictionTest = predict(w1, b1, X_test2);<br>
yPredictionTrain = predict(w1, b1, X_train2);<br>
</code></p>
<p><code><br>
trainAccuracy=100-mean(abs(yPredictionTrain - y_train1))*100<br>
testAccuracy=100- mean(abs(yPredictionTest - y_test1))*100<br>
trainAccuracy =  90.845<br>
testAccuracy =  89.510<br>
graphics_toolkit('gnuplot')<br>
plot(idx,losses);<br>
title ('Gradient descent- Cost vs No of iterations');<br>
xlabel (&quot;No of iterations&quot;);<br>
ylabel (&quot;Cost&quot;);</code></p>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/01/lr.jpg" data-orig-size="1200,900" data-medium-file="https://gigadom.files.wordpress.com/2018/01/lr.jpg?w=300" data-attachment-id="6134" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/01/lr.jpg?w=450&h=507&fit=676%2C507" data-permalink="https://gigadom.wordpress.com/2018/01/04/deep-learning-from-basic-principles-in-python-r-and-octave-part-1/lr/" data-large-file="https://gigadom.files.wordpress.com/2018/01/lr.jpg?w=450&amp;h=507&amp;fit=676%2C507" alt="" data-image-title="lr" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6134" data-recalc-dims="1"></p>
<p><strong>Conclusion</strong><br>
This post starts with a simple 2 layer Neural Network implementation of Logistic Regression. Clearly the performance of this simple Neural Network is comparatively poor to the highly optimized sklearn’s Logistic Regression. This is because the above neural network did not have any hidden layers. Deep Learning &amp; Neural Networks achieve extraordinary performance because of the presence of deep hidden layers</p>
<p>The Deep Learning journey has begun… Don’t miss the bus!<br>
Stay tuned for more interesting posts in Deep Learning!!</p>
<p>References<br>
1. <a rel="nofollow" href="https://www.coursera.org/specializations/deep-learning" target="_blank">Deep Learning Specialization</a><br>
2. <a rel="nofollow" href="https://www.coursera.org/learn/neural-networks" target="_blank">Neural Networks for Machine Learning</a><br>
3. <a rel="nofollow" href="http://www.deeplearningbook.org/" target="_blank">Deep Learning, Ian Goodfellow, Yoshua Bengio and Aaron Courville</a><br>
4. <a rel="nofollow" href="https://gigadom.wordpress.com/2017/01/21/neural-networks-the-mechanics-of-backpropagation/" target="_blank">Neural Networks: The mechanics of backpropagation</a><br>
5. <a rel="nofollow" href="https://www.coursera.org/learn/machine-learning" target="_blank">Machine Learning</a></p>
<p>Also see<br>
1. <a rel="nofollow" href="https://gigadom.wordpress.com/2017/12/05/my-book-practical-machine-learning-with-r-and-python-on-amazon/" target="_blank">My book ‘Practical Machine Learning with R and Python’ on Amazon</a><br>
2. <a rel="nofollow" href="https://gigadom.wordpress.com/2014/01/03/simplifying-machine-learning-bias-variance-regularization-and-odd-facts-part-4/" target="_blank">Simplifying Machine Learning: Bias, Variance, regularization and odd facts – Part 4</a><br>
3. <a rel="nofollow" href="https://gigadom.wordpress.com/2017/12/16/the-3rd-paperback-edition-of-my-books-on-cricket-on-amazon/" target="_blank">The 3rd paperback &amp; kindle editions of my books on Cricket, now on Amazon</a><br>
4. <a rel="nofollow" href="https://gigadom.wordpress.com/2017/10/29/practical-machine-learning-with-r-and-python-part-4/" target="_blank">Practical Machine Learning with R and Python – Part 4</a><br>
5. <a rel="nofollow" href="https://gigadom.wordpress.com/2016/06/23/introducing-qcsimulator-a-5-qubit-quantum-computing-simulator-in-r/" target="_blank">Introducing QCSimulator: A 5-qubit quantum computing simulator in R</a><br>
6. <a rel="nofollow" href="https://gigadom.wordpress.com/2014/07/27/a-bluemix-recipe-with-mongodb-and-node-js/" target="_blank">A Bluemix recipe with MongoDB and Node.js</a><br>
7. <a rel="nofollow" href="https://gigadom.wordpress.com/2017/09/11/my-travels-through-the-realms-of-data-science-machine-learning-deep-learning-and-ai/" target="_blank">My travels through the realms of Data Science, Machine Learning, Deep Learning and (AI)</a></p>
<p>To see all posts check <a rel="nofollow" href="https://gigadom.wordpress.com/aa-2/" target="_blank">Index of posts</a></p>
<p> <a rel="nofollow" href="https://feeds.wordpress.com/1.0/gocomments/gigadom.wordpress.com/6104/" target="_blank"><img border="0" alt="" src="https://feeds.wordpress.com/1.0/comments/gigadom.wordpress.com/6104/"></a> <img border="0" src="https://i1.wp.com/pixel.wp.com/b.gif?resize=1%2C1&ssl=1" alt="" width="1" data-recalc-dims="1" height="1"></p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://gigadom.wordpress.com/2018/01/04/deep-learning-from-basic-principles-in-python-r-and-octave-part-1/"> R – Giga thoughts …</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/_vTKFMIVRws" height="1"></a></div>