<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/deep-learning-from-first-principles-in-python-r-and-octave-part-4/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://gigadom.wordpress.com/2018/02/26/deep-learning-from-first-principles-in-python-r-and-octave-part-4/"> R – Giga thoughts …</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>In this 4th post of my series on Deep Learning from first principles in Python, R and Octave – Part 4, I explore the details of creating a multi-class classifier using the Softmax activation unit in a neural network. The earlier posts in this series were</p>
<p>1. <a rel="nofollow" href="https://gigadom.wordpress.com/2018/01/04/deep-learning-from-basic-principles-in-python-r-and-octave-part-1/" target="_blank">Deep Learning from first principles in Python, R and Octave – Part 1</a>. In this post I implemented logistic regression as a simple Neural Network in vectorized Python, R and Octave<br>
2. <a rel="nofollow" href="https://gigadom.wordpress.com/2018/01/11/deep-learning-from-first-principles-in-python-r-and-octave-part-2/" target="_blank">Deep Learning from first principles in Python, R and Octave – Part 2</a>. This 2nd part implemented the most elementary neural network with 1 hidden layer and any number of activation units in the hidden layer with sigmoid activation at the output layer<br>
3. <a rel="nofollow" href="https://gigadom.wordpress.com/2018/01/30/deep-learning-from-first-principles-in-python-r-and-octave-part-3/" target="_blank">Deep Learning from first principles in Python, R and Octave – Part 3</a>. The 3rd implemented a multi-layer Deep Learning network with an arbitrary number if hidden layers and activation units per hidden layer. The output layer was for binary classification which was based on the sigmoid unit. This multi-layer deep network was implemented in vectorized Python, R and Octave</p>
<p>This 4th part takes a swing at multi-class classification and uses the Softmax as the activation unit in the output layer. Inclusion of the Softmax activation unit in the activation layer requires us to compute the derivative of Softmax, or rather the “Jacobian” of the Softmax function, besides also computing the log loss for this Softmax activation during back propagation. Since the derivation of the Jacobian of a Softmax and the computation of the Cross Entropy/log loss is very involved, I have implemented a basic neural network with just 1 hidden layer with the Softmax activation at the output layer. I also perform multi-class classification based on the ‘spiral’ data set from <a rel="nofollow" href="http://cs231n.github.io/neural-networks-case-study/" target="_blank">CS231n Convolutional Neural Networks </a>Stanford course, to test the performance and correctness of the implementations in Python, R and Octave. You can clone download the code for the Python, R and Octave implementations from Github at <a rel="nofollow" href="https://github.com/tvganesh/DeepLearning-Part4" target="_blank">Deep Learning – Part 4</a></p>
<p>The Softmax function takes an N dimensional vector as input and generates a N dimensional vector as output.<br>
The Softmax function is given by<br>
<img alt="S_{j}= frac{e_{j}}{sum_{i}^{N}e_{k}}" title="S_{j}= frac{e_{j}}{sum_{i}^{N}e_{k}}" src="https://s0.wp.com/latex.php?latex=S_%7Bj%7D%3D+%5Cfrac%7Be_%7Bj%7D%7D%7B%5Csum_%7Bi%7D%5E%7BN%7De_%7Bk%7D%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
There is a probabilistic interpretation of the Softmax, since the sum of the Softmax values of a set of vectors will always add up to 1, given that each Softmax value is divided by the total of all values.</p>
<p>As mentioned earlier, the Softmax takes a vector input and returns a vector of outputs.  For e.g. the Softmax of a vector a=[1, 3, 6]  is another vector S=[0.0063,0.0471,0.9464]. Notice that vector output is proportional to the input vector.  Also, taking the derivative of a vector by another vector, is known as the Jacobian. By the way, <a rel="nofollow" href="https://arxiv.org/pdf/1802.01528.pdf" target="_blank">The Matrix Calculus You Need For Deep Learning </a>by Terence Parr and Jeremy Howard, is very good paper that distills all the main mathematical concepts for Deep Learning in one place.</p>
<p>Let us take a simple 2 layered neural network with just 2 activation units in the hidden layer is shown below<br>
<img data-orig-file="https://gigadom.files.wordpress.com/2018/02/dl-12.png" data-orig-size="865,316" data-medium-file="https://gigadom.files.wordpress.com/2018/02/dl-12.png?w=300" data-attachment-id="6413" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/02/dl-12.png?w=450&h=247&fit=676%2C247" data-permalink="https://gigadom.wordpress.com/2018/02/26/deep-learning-from-first-principles-in-python-r-and-octave-part-4/dl-1-4/" data-large-file="https://gigadom.files.wordpress.com/2018/02/dl-12.png?w=450&amp;h=247&amp;fit=676%2C247" alt="" data-image-title="dl-1" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6413" data-recalc-dims="1"><br>
<img alt="Z_{1}^{1} =W_{11}^{1}x_{1} + W_{21}^{1}x_{2} + b_{1}^{1}" title="Z_{1}^{1} =W_{11}^{1}x_{1} + W_{21}^{1}x_{2} + b_{1}^{1}" src="https://s0.wp.com/latex.php?latex=Z_%7B1%7D%5E%7B1%7D+%3DW_%7B11%7D%5E%7B1%7Dx_%7B1%7D+%2B+W_%7B21%7D%5E%7B1%7Dx_%7B2%7D+%2B+b_%7B1%7D%5E%7B1%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="Z_{2}^{1} =W_{12}^{1}x_{1} + W_{22}^{1}x_{2} + b_{2}^{1}" title="Z_{2}^{1} =W_{12}^{1}x_{1} + W_{22}^{1}x_{2} + b_{2}^{1}" src="https://s0.wp.com/latex.php?latex=Z_%7B2%7D%5E%7B1%7D+%3DW_%7B12%7D%5E%7B1%7Dx_%7B1%7D+%2B+W_%7B22%7D%5E%7B1%7Dx_%7B2%7D+%2B+b_%7B2%7D%5E%7B1%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
and<br>
<img alt="A_{1}^{1} = g'(Z_{1}^{1})" title="A_{1}^{1} = g'(Z_{1}^{1})" src="https://s0.wp.com/latex.php?latex=A_%7B1%7D%5E%7B1%7D+%3D+g%27%28Z_%7B1%7D%5E%7B1%7D%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="A_{2}^{1} = g'(Z_{2}^{1})" title="A_{2}^{1} = g'(Z_{2}^{1})" src="https://s0.wp.com/latex.php?latex=A_%7B2%7D%5E%7B1%7D+%3D+g%27%28Z_%7B2%7D%5E%7B1%7D%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
where g'() is the activation unit in the hidden layer which can be a relu, sigmoid or a<br>
tanh function</p>
<p><strong>Note</strong>: The superscript denotes the layer. The above denotes the equation for layer 1<br>
of the neural network. For layer 2 with the Softmax activation, the equations are<br>
<img alt="Z_{1}^{2} =W_{11}^{2}x_{1} + W_{21}^{2}x_{2} + b_{1}^{2}" title="Z_{1}^{2} =W_{11}^{2}x_{1} + W_{21}^{2}x_{2} + b_{1}^{2}" src="https://s0.wp.com/latex.php?latex=Z_%7B1%7D%5E%7B2%7D+%3DW_%7B11%7D%5E%7B2%7Dx_%7B1%7D+%2B+W_%7B21%7D%5E%7B2%7Dx_%7B2%7D+%2B+b_%7B1%7D%5E%7B2%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="Z_{2}^{2} =W_{12}^{2}x_{1} + W_{22}^{2}x_{2} + b_{2}^{2}" title="Z_{2}^{2} =W_{12}^{2}x_{1} + W_{22}^{2}x_{2} + b_{2}^{2}" src="https://s0.wp.com/latex.php?latex=Z_%7B2%7D%5E%7B2%7D+%3DW_%7B12%7D%5E%7B2%7Dx_%7B1%7D+%2B+W_%7B22%7D%5E%7B2%7Dx_%7B2%7D+%2B+b_%7B2%7D%5E%7B2%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
and<br>
<img alt="A_{1}^{2} = S(A_{1}^{1})" title="A_{1}^{2} = S(A_{1}^{1})" src="https://s0.wp.com/latex.php?latex=A_%7B1%7D%5E%7B2%7D+%3D+S%28A_%7B1%7D%5E%7B1%7D%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="A_{2}^{2} = S(A_{2}^{1})" title="A_{2}^{2} = S(A_{2}^{1})" src="https://s0.wp.com/latex.php?latex=A_%7B2%7D%5E%7B2%7D+%3D+S%28A_%7B2%7D%5E%7B1%7D%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
where S() is the Softmax activation function<br>
<img alt="S=begin{pmatrix}  S(Z_{1}^{2})\ S(Z_{2}^{2})  end{pmatrix}" title="S=begin{pmatrix}  S(Z_{1}^{2})\ S(Z_{2}^{2})  end{pmatrix}" src="https://s0.wp.com/latex.php?latex=S%3D%5Cbegin%7Bpmatrix%7D++S%28Z_%7B1%7D%5E%7B2%7D%29%5C%5C+S%28Z_%7B2%7D%5E%7B2%7D%29++%5Cend%7Bpmatrix%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="S=begin{pmatrix}  frac{e^{Z1}}{e^{Z1}+e^{Z2}}\ frac{e^{Z2}}{e^{Z1}+e^{Z2}}  end{pmatrix}" title="S=begin{pmatrix}  frac{e^{Z1}}{e^{Z1}+e^{Z2}}\ frac{e^{Z2}}{e^{Z1}+e^{Z2}}  end{pmatrix}" src="https://s0.wp.com/latex.php?latex=S%3D%5Cbegin%7Bpmatrix%7D++%5Cfrac%7Be%5E%7BZ1%7D%7D%7Be%5E%7BZ1%7D%2Be%5E%7BZ2%7D%7D%5C%5C+%5Cfrac%7Be%5E%7BZ2%7D%7D%7Be%5E%7BZ1%7D%2Be%5E%7BZ2%7D%7D++%5Cend%7Bpmatrix%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"></p>
<p>The Jacobian of the softmax ‘S’ is given by<br>
<img alt="begin{pmatrix}  frac {partial S_{1}}{partial Z_{1}} &amp; frac {partial S_{1}}{partial Z_{2}}\  frac {partial S_{2}}{partial Z_{1}} &amp; frac {partial S_{2}}{partial Z_{2}}  end{pmatrix}" title="begin{pmatrix}  frac {partial S_{1}}{partial Z_{1}} &amp; frac {partial S_{1}}{partial Z_{2}}\  frac {partial S_{2}}{partial Z_{1}} &amp; frac {partial S_{2}}{partial Z_{2}}  end{pmatrix}" src="https://s0.wp.com/latex.php?latex=%5Cbegin%7Bpmatrix%7D++%5Cfrac+%7B%5Cpartial+S_%7B1%7D%7D%7B%5Cpartial+Z_%7B1%7D%7D+%26+%5Cfrac+%7B%5Cpartial+S_%7B1%7D%7D%7B%5Cpartial+Z_%7B2%7D%7D%5C%5C++%5Cfrac+%7B%5Cpartial+S_%7B2%7D%7D%7B%5Cpartial+Z_%7B1%7D%7D+%26+%5Cfrac+%7B%5Cpartial+S_%7B2%7D%7D%7B%5Cpartial+Z_%7B2%7D%7D++%5Cend%7Bpmatrix%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="begin{pmatrix}  frac{partial}{partial Z_{1}} frac {e^{Z1}}{e^{Z1}+ e^{Z2}} &amp; frac{partial}{partial Z_{2}} frac {e^{Z1}}{e^{Z1}+ e^{Z2}}\  frac{partial}{partial Z_{1}} frac {e^{Z2}}{e^{Z1}+ e^{Z2}} &amp; frac{partial}{partial Z_{2}} frac {e^{Z2}}{e^{Z1}+ e^{Z2}}  end{pmatrix}" title="begin{pmatrix}  frac{partial}{partial Z_{1}} frac {e^{Z1}}{e^{Z1}+ e^{Z2}} &amp; frac{partial}{partial Z_{2}} frac {e^{Z1}}{e^{Z1}+ e^{Z2}}\  frac{partial}{partial Z_{1}} frac {e^{Z2}}{e^{Z1}+ e^{Z2}} &amp; frac{partial}{partial Z_{2}} frac {e^{Z2}}{e^{Z1}+ e^{Z2}}  end{pmatrix}" src="https://s0.wp.com/latex.php?latex=%5Cbegin%7Bpmatrix%7D++%5Cfrac%7B%5Cpartial%7D%7B%5Cpartial+Z_%7B1%7D%7D+%5Cfrac+%7Be%5E%7BZ1%7D%7D%7Be%5E%7BZ1%7D%2B+e%5E%7BZ2%7D%7D+%26+%5Cfrac%7B%5Cpartial%7D%7B%5Cpartial+Z_%7B2%7D%7D+%5Cfrac+%7Be%5E%7BZ1%7D%7D%7Be%5E%7BZ1%7D%2B+e%5E%7BZ2%7D%7D%5C%5C++%5Cfrac%7B%5Cpartial%7D%7B%5Cpartial+Z_%7B1%7D%7D+%5Cfrac+%7Be%5E%7BZ2%7D%7D%7Be%5E%7BZ1%7D%2B+e%5E%7BZ2%7D%7D+%26+%5Cfrac%7B%5Cpartial%7D%7B%5Cpartial+Z_%7B2%7D%7D+%5Cfrac+%7Be%5E%7BZ2%7D%7D%7Be%5E%7BZ1%7D%2B+e%5E%7BZ2%7D%7D++%5Cend%7Bpmatrix%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex">     – (A)</p>
<p>Now the ‘division-rule’  of derivatives is as follows. If u and v are functions of x, then<br>
<img alt="frac{d}{dx} frac {u}{v} =frac {vdu -udv}{v^{2}}" title="frac{d}{dx} frac {u}{v} =frac {vdu -udv}{v^{2}}" src="https://s0.wp.com/latex.php?latex=%5Cfrac%7Bd%7D%7Bdx%7D+%5Cfrac+%7Bu%7D%7Bv%7D+%3D%5Cfrac+%7Bvdu+-udv%7D%7Bv%5E%7B2%7D%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
Using this to compute each element of the above Jacobian matrix, we see that<br>
when i=j we have<br>
<img alt="frac {partial}{partial Z1}frac{e^{Z1}}{e^{Z1}+e^{Z2}} =  frac {sum e^{Z1} - e^{Z1^{2}}}{sum ^{2}}" title="frac {partial}{partial Z1}frac{e^{Z1}}{e^{Z1}+e^{Z2}} =  frac {sum e^{Z1} - e^{Z1^{2}}}{sum ^{2}}" src="https://s0.wp.com/latex.php?latex=%5Cfrac+%7B%5Cpartial%7D%7B%5Cpartial+Z1%7D%5Cfrac%7Be%5E%7BZ1%7D%7D%7Be%5E%7BZ1%7D%2Be%5E%7BZ2%7D%7D+%3D++%5Cfrac+%7B%5Csum+e%5E%7BZ1%7D+-+e%5E%7BZ1%5E%7B2%7D%7D%7D%7B%5Csum+%5E%7B2%7D%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
and when <img alt="i neq j" title="i neq j" src="https://s0.wp.com/latex.php?latex=i+%5Cneq+j&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="frac {partial}{partial Z1}frac{e^{Z2}}{e^{Z1}+e^{Z2}} =  frac {0 - e^{z1}e^{Z2}}{sum ^{2}}" title="frac {partial}{partial Z1}frac{e^{Z2}}{e^{Z1}+e^{Z2}} =  frac {0 - e^{z1}e^{Z2}}{sum ^{2}}" src="https://s0.wp.com/latex.php?latex=%5Cfrac+%7B%5Cpartial%7D%7B%5Cpartial+Z1%7D%5Cfrac%7Be%5E%7BZ2%7D%7D%7Be%5E%7BZ1%7D%2Be%5E%7BZ2%7D%7D+%3D++%5Cfrac+%7B0+-+e%5E%7Bz1%7De%5E%7BZ2%7D%7D%7B%5Csum+%5E%7B2%7D%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
This is of the general form<br>
<img alt="frac {partial S_{j}}{partial z_{i}} = S_{i}( 1-S_{j})" title="frac {partial S_{j}}{partial z_{i}} = S_{i}( 1-S_{j})" src="https://s0.wp.com/latex.php?latex=%5Cfrac+%7B%5Cpartial+S_%7Bj%7D%7D%7B%5Cpartial+z_%7Bi%7D%7D+%3D+S_%7Bi%7D%28+1-S_%7Bj%7D%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex">  when i=j<br>
and<br>
<img alt="frac {partial S_{j}}{partial z_{i}} = -S_{i}S_{j}" title="frac {partial S_{j}}{partial z_{i}} = -S_{i}S_{j}" src="https://s0.wp.com/latex.php?latex=%5Cfrac+%7B%5Cpartial+S_%7Bj%7D%7D%7B%5Cpartial+z_%7Bi%7D%7D+%3D+-S_%7Bi%7DS_%7Bj%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex">  when <img alt="i neq j" title="i neq j" src="https://s0.wp.com/latex.php?latex=i+%5Cneq+j&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<strong>Note</strong>: Since the Softmax essentially gives the probability the following<br>
notation is also used<br>
<img alt="frac {partial p_{j}}{partial z_{i}} = p_{i}( 1-p_{j}) " title="frac {partial p_{j}}{partial z_{i}} = p_{i}( 1-p_{j}) " src="https://s0.wp.com/latex.php?latex=%5Cfrac+%7B%5Cpartial+p_%7Bj%7D%7D%7B%5Cpartial+z_%7Bi%7D%7D+%3D+p_%7Bi%7D%28+1-p_%7Bj%7D%29+&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> when i=j<br>
and<br>
<img alt="frac {partial p_{j}}{partial z_{i}} = -p_{i}p_{j} when i neq j" title="frac {partial p_{j}}{partial z_{i}} = -p_{i}p_{j} when i neq j" src="https://s0.wp.com/latex.php?latex=%5Cfrac+%7B%5Cpartial+p_%7Bj%7D%7D%7B%5Cpartial+z_%7Bi%7D%7D+%3D+-p_%7Bi%7Dp_%7Bj%7D+when+i+%5Cneq+j&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
If you throw the “Kronecker delta” into the equation, then the above equations can be expressed even more concisely as<br>
<img alt="frac {partial p_{j}}{partial z_{i}} = p_{i} (delta_{ij} - p_{j})" title="frac {partial p_{j}}{partial z_{i}} = p_{i} (delta_{ij} - p_{j})" src="https://s0.wp.com/latex.php?latex=%5Cfrac+%7B%5Cpartial+p_%7Bj%7D%7D%7B%5Cpartial+z_%7Bi%7D%7D+%3D+p_%7Bi%7D+%28%5Cdelta_%7Bij%7D+-+p_%7Bj%7D%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
where <img alt="delta_{ij} = 1" title="delta_{ij} = 1" src="https://s0.wp.com/latex.php?latex=%5Cdelta_%7Bij%7D+%3D+1&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> when i=j and 0 when <img alt="i neq j" title="i neq j" src="https://s0.wp.com/latex.php?latex=i+%5Cneq+j&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"></p>
<p>This reduces the Jacobian of the simple 2 output softmax vectors  equation (A) as<br>
<img alt="begin{pmatrix}  p_{1}(1-p_{1}) &amp; -p_{1}p_{2} \  -p_{2}p_{1} &amp; p_{2}(1-p_{2})  end{pmatrix}" title="begin{pmatrix}  p_{1}(1-p_{1}) &amp; -p_{1}p_{2} \  -p_{2}p_{1} &amp; p_{2}(1-p_{2})  end{pmatrix}" src="https://s0.wp.com/latex.php?latex=%5Cbegin%7Bpmatrix%7D++p_%7B1%7D%281-p_%7B1%7D%29+%26+-p_%7B1%7Dp_%7B2%7D+%5C%5C++-p_%7B2%7Dp_%7B1%7D+%26+p_%7B2%7D%281-p_%7B2%7D%29++%5Cend%7Bpmatrix%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
The loss of Softmax is given by<br>
<img alt="L = -sum y_{i} log(p_{i})" title="L = -sum y_{i} log(p_{i})" src="https://s0.wp.com/latex.php?latex=L+%3D+-%5Csum+y_%7Bi%7D+log%28p_%7Bi%7D%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
For the 2 valued Softmax output this is<br>
<img alt="frac {dL}{dp1} = -frac {y_{1}}{p_{1}}" title="frac {dL}{dp1} = -frac {y_{1}}{p_{1}}" src="https://s0.wp.com/latex.php?latex=%5Cfrac+%7BdL%7D%7Bdp1%7D+%3D+-%5Cfrac+%7By_%7B1%7D%7D%7Bp_%7B1%7D%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="frac {dL}{dp2} = -frac {y_{2}}{p_{2}}" title="frac {dL}{dp2} = -frac {y_{2}}{p_{2}}" src="https://s0.wp.com/latex.php?latex=%5Cfrac+%7BdL%7D%7Bdp2%7D+%3D+-%5Cfrac+%7By_%7B2%7D%7D%7Bp_%7B2%7D%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
Using the chain rule we can write<br>
<img alt="frac {partial L}{partial w_{pq}} = sum _{i}frac {partial L}{partial p_{i}} frac {partial p_{i}}{partial w_{pq}}" title="frac {partial L}{partial w_{pq}} = sum _{i}frac {partial L}{partial p_{i}} frac {partial p_{i}}{partial w_{pq}}" src="https://s0.wp.com/latex.php?latex=%5Cfrac+%7B%5Cpartial+L%7D%7B%5Cpartial+w_%7Bpq%7D%7D+%3D+%5Csum+_%7Bi%7D%5Cfrac+%7B%5Cpartial+L%7D%7B%5Cpartial+p_%7Bi%7D%7D+%5Cfrac+%7B%5Cpartial+p_%7Bi%7D%7D%7B%5Cpartial+w_%7Bpq%7D%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> (1)<br>
and<br>
<img alt="frac {partial p_{i}}{partial w_{pq}} = sum _{k}frac {partial p_{i}}{partial z_{k}} frac {partial z_{k}}{partial w_{pq}}" title="frac {partial p_{i}}{partial w_{pq}} = sum _{k}frac {partial p_{i}}{partial z_{k}} frac {partial z_{k}}{partial w_{pq}}" src="https://s0.wp.com/latex.php?latex=%5Cfrac+%7B%5Cpartial+p_%7Bi%7D%7D%7B%5Cpartial+w_%7Bpq%7D%7D+%3D+%5Csum+_%7Bk%7D%5Cfrac+%7B%5Cpartial+p_%7Bi%7D%7D%7B%5Cpartial+z_%7Bk%7D%7D+%5Cfrac+%7B%5Cpartial+z_%7Bk%7D%7D%7B%5Cpartial+w_%7Bpq%7D%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> (2)<br>
In expanded form this is<br>
<img alt="frac {partial L}{partial w_{pq}} = sum _{i}frac {partial L}{partial p_{i}} sum _{k}frac {partial p_{i}}{partial z_{k}} frac {partial z_{k}}{partial w_{pq}}" title="frac {partial L}{partial w_{pq}} = sum _{i}frac {partial L}{partial p_{i}} sum _{k}frac {partial p_{i}}{partial z_{k}} frac {partial z_{k}}{partial w_{pq}}" src="https://s0.wp.com/latex.php?latex=%5Cfrac+%7B%5Cpartial+L%7D%7B%5Cpartial+w_%7Bpq%7D%7D+%3D+%5Csum+_%7Bi%7D%5Cfrac+%7B%5Cpartial+L%7D%7B%5Cpartial+p_%7Bi%7D%7D+%5Csum+_%7Bk%7D%5Cfrac+%7B%5Cpartial+p_%7Bi%7D%7D%7B%5Cpartial+z_%7Bk%7D%7D+%5Cfrac+%7B%5Cpartial+z_%7Bk%7D%7D%7B%5Cpartial+w_%7Bpq%7D%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
Also<br>
<img alt="frac {partial L}{partial Z_{i}} =sum _{i} frac {partial L}{partial p} frac {partial p}{partial Z_{i}}" title="frac {partial L}{partial Z_{i}} =sum _{i} frac {partial L}{partial p} frac {partial p}{partial Z_{i}}" src="https://s0.wp.com/latex.php?latex=%5Cfrac+%7B%5Cpartial+L%7D%7B%5Cpartial+Z_%7Bi%7D%7D+%3D%5Csum+_%7Bi%7D+%5Cfrac+%7B%5Cpartial+L%7D%7B%5Cpartial+p%7D+%5Cfrac+%7B%5Cpartial+p%7D%7B%5Cpartial+Z_%7Bi%7D%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
Therefore<br>
<img alt="frac {partial L}{partial Z_{1}} =frac {partial L}{partial p_{1}} frac {partial p_{1}}{partial Z_{1}} +frac {partial L}{partial p_{2}} frac {partial p_{2}}{partial Z_{1}}" title="frac {partial L}{partial Z_{1}} =frac {partial L}{partial p_{1}} frac {partial p_{1}}{partial Z_{1}} +frac {partial L}{partial p_{2}} frac {partial p_{2}}{partial Z_{1}}" src="https://s0.wp.com/latex.php?latex=%5Cfrac+%7B%5Cpartial+L%7D%7B%5Cpartial+Z_%7B1%7D%7D+%3D%5Cfrac+%7B%5Cpartial+L%7D%7B%5Cpartial+p_%7B1%7D%7D+%5Cfrac+%7B%5Cpartial+p_%7B1%7D%7D%7B%5Cpartial+Z_%7B1%7D%7D+%2B%5Cfrac+%7B%5Cpartial+L%7D%7B%5Cpartial+p_%7B2%7D%7D+%5Cfrac+%7B%5Cpartial+p_%7B2%7D%7D%7B%5Cpartial+Z_%7B1%7D%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="frac {partial L}{partial z_{1}}=-frac {y1}{p1} p1(1-p1) - frac {y2}{p2}*(-p_{2}p_{1})" title="frac {partial L}{partial z_{1}}=-frac {y1}{p1} p1(1-p1) - frac {y2}{p2}*(-p_{2}p_{1})" src="https://s0.wp.com/latex.php?latex=%5Cfrac+%7B%5Cpartial+L%7D%7B%5Cpartial+z_%7B1%7D%7D%3D-%5Cfrac+%7By1%7D%7Bp1%7D+p1%281-p1%29+-+%5Cfrac+%7By2%7D%7Bp2%7D%2A%28-p_%7B2%7Dp_%7B1%7D%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
Since<br>
<img alt="frac {partial p_{j}}{partial z_{i}} = p_{i}( 1-p_{j})" title="frac {partial p_{j}}{partial z_{i}} = p_{i}( 1-p_{j})" src="https://s0.wp.com/latex.php?latex=%5Cfrac+%7B%5Cpartial+p_%7Bj%7D%7D%7B%5Cpartial+z_%7Bi%7D%7D+%3D+p_%7Bi%7D%28+1-p_%7Bj%7D%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> when i=j<br>
and<br>
<img alt="frac {partial p_{j}}{partial z_{i}} = -p_{i}p_{j}" title="frac {partial p_{j}}{partial z_{i}} = -p_{i}p_{j}" src="https://s0.wp.com/latex.php?latex=%5Cfrac+%7B%5Cpartial+p_%7Bj%7D%7D%7B%5Cpartial+z_%7Bi%7D%7D+%3D+-p_%7Bi%7Dp_%7Bj%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> when <img alt="i neq j" title="i neq j" src="https://s0.wp.com/latex.php?latex=i+%5Cneq+j&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
which simplifies to<br>
<img alt="frac {partial L}{partial Z_{1}} = -y_{1} + y_{1}p_{1} + y_{2}p_{1} =" title="frac {partial L}{partial Z_{1}} = -y_{1} + y_{1}p_{1} + y_{2}p_{1} =" src="https://s0.wp.com/latex.php?latex=%5Cfrac+%7B%5Cpartial+L%7D%7B%5Cpartial+Z_%7B1%7D%7D+%3D+-y_%7B1%7D+%2B+y_%7B1%7Dp_%7B1%7D+%2B+y_%7B2%7Dp_%7B1%7D+%3D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="p_{1}sum (y_{1} + y_2) - y_{1}" title="p_{1}sum (y_{1} + y_2) - y_{1}" src="https://s0.wp.com/latex.php?latex=p_%7B1%7D%5Csum+%28y_%7B1%7D+%2B+y_2%29+-+y_%7B1%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="frac {partial L}{partial Z_{1}}= p_{1} - y_{1}" title="frac {partial L}{partial Z_{1}}= p_{1} - y_{1}" src="https://s0.wp.com/latex.php?latex=%5Cfrac+%7B%5Cpartial+L%7D%7B%5Cpartial+Z_%7B1%7D%7D%3D+p_%7B1%7D+-+y_%7B1%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
Since<br>
<img alt="sum_{i} y_{i} =1" title="sum_{i} y_{i} =1" src="https://s0.wp.com/latex.php?latex=%5Csum_%7Bi%7D+y_%7Bi%7D+%3D1&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
Similarly<br>
<img alt="frac {partial L}{partial Z_{2}} =frac {partial L}{partial p_{1}} frac {partial p_{1}}{partial Z_{2}} +frac {partial L}{partial p_{2}} frac {partial p_{2}}{partial Z_{2}} " title="frac {partial L}{partial Z_{2}} =frac {partial L}{partial p_{1}} frac {partial p_{1}}{partial Z_{2}} +frac {partial L}{partial p_{2}} frac {partial p_{2}}{partial Z_{2}} " src="https://s0.wp.com/latex.php?latex=%5Cfrac+%7B%5Cpartial+L%7D%7B%5Cpartial+Z_%7B2%7D%7D+%3D%5Cfrac+%7B%5Cpartial+L%7D%7B%5Cpartial+p_%7B1%7D%7D+%5Cfrac+%7B%5Cpartial+p_%7B1%7D%7D%7B%5Cpartial+Z_%7B2%7D%7D+%2B%5Cfrac+%7B%5Cpartial+L%7D%7B%5Cpartial+p_%7B2%7D%7D+%5Cfrac+%7B%5Cpartial+p_%7B2%7D%7D%7B%5Cpartial+Z_%7B2%7D%7D+&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="frac {partial L}{partial z_{2}}=-frac {y1}{p1}*(p_{1}p_{2}) - frac {y2}{p2}*p_{2}(1-p_{2})" title="frac {partial L}{partial z_{2}}=-frac {y1}{p1}*(p_{1}p_{2}) - frac {y2}{p2}*p_{2}(1-p_{2})" src="https://s0.wp.com/latex.php?latex=%5Cfrac+%7B%5Cpartial+L%7D%7B%5Cpartial+z_%7B2%7D%7D%3D-%5Cfrac+%7By1%7D%7Bp1%7D%2A%28p_%7B1%7Dp_%7B2%7D%29+-+%5Cfrac+%7By2%7D%7Bp2%7D%2Ap_%7B2%7D%281-p_%7B2%7D%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="y_{1}p_{2} + y_{2}p_{2} - y_{2}" title="y_{1}p_{2} + y_{2}p_{2} - y_{2}" src="https://s0.wp.com/latex.php?latex=y_%7B1%7Dp_%7B2%7D+%2B+y_%7B2%7Dp_%7B2%7D+-+y_%7B2%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="frac {partial L}{partial Z_{2}} =p_{2}sum (y_{1} + y_2) - y_{2}\ = p_{2} - y_{2}" title="frac {partial L}{partial Z_{2}} =p_{2}sum (y_{1} + y_2) - y_{2}\ = p_{2} - y_{2}" src="https://s0.wp.com/latex.php?latex=%5Cfrac+%7B%5Cpartial+L%7D%7B%5Cpartial+Z_%7B2%7D%7D+%3Dp_%7B2%7D%5Csum+%28y_%7B1%7D+%2B+y_2%29+-+y_%7B2%7D%5C%5C+%3D+p_%7B2%7D+-+y_%7B2%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
In general this is of the form<br>
<img alt="frac {partial L}{partial z_{i}} = p_{i} -y_{i}" title="frac {partial L}{partial z_{i}} = p_{i} -y_{i}" src="https://s0.wp.com/latex.php?latex=%5Cfrac+%7B%5Cpartial+L%7D%7B%5Cpartial+z_%7Bi%7D%7D+%3D+p_%7Bi%7D+-y_%7Bi%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
For e.g if the probabilities computed were p=[0.1, 0.7, 0.2] then this implies that the class with probability 0.7 is the likely class. This would imply that the ‘One hot encoding’ for  yi  would be yi=[0,1,0] therefore the gradient pi-yi = [0.1,-0.3,0.2]</p>
<p><strong>Note: Further, we could extend this derivation for a Softmax activation output that outputs 3 classes<br>
<img alt="S=begin{pmatrix}  frac{e^{z1}}{e^{z1}+e^{z2}+e^{z3}}\ frac{e^{z2}}{e^{z1}+e^{z2}+e^{z3}} \ frac{e^{z3}}{e^{z1}+e^{z2}+e^{z3}}  end{pmatrix}" title="S=begin{pmatrix}  frac{e^{z1}}{e^{z1}+e^{z2}+e^{z3}}\ frac{e^{z2}}{e^{z1}+e^{z2}+e^{z3}} \ frac{e^{z3}}{e^{z1}+e^{z2}+e^{z3}}  end{pmatrix}" src="https://s0.wp.com/latex.php?latex=S%3D%5Cbegin%7Bpmatrix%7D++%5Cfrac%7Be%5E%7Bz1%7D%7D%7Be%5E%7Bz1%7D%2Be%5E%7Bz2%7D%2Be%5E%7Bz3%7D%7D%5C%5C+%5Cfrac%7Be%5E%7Bz2%7D%7D%7Be%5E%7Bz1%7D%2Be%5E%7Bz2%7D%2Be%5E%7Bz3%7D%7D+%5C%5C+%5Cfrac%7Be%5E%7Bz3%7D%7D%7Be%5E%7Bz1%7D%2Be%5E%7Bz2%7D%2Be%5E%7Bz3%7D%7D++%5Cend%7Bpmatrix%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex">
<p>We could derive<br>
<img alt="frac {partial L}{partial z1}= frac {partial L}{partial p_{1}} frac {partial p_{1}}{partial z_{1}} +frac {partial L}{partial p_{2}} frac {partial p_{2}}{partial z_{1}} +frac {partial L}{partial p_{3}} frac {partial p_{3}}{partial z_{1}}" title="frac {partial L}{partial z1}= frac {partial L}{partial p_{1}} frac {partial p_{1}}{partial z_{1}} +frac {partial L}{partial p_{2}} frac {partial p_{2}}{partial z_{1}} +frac {partial L}{partial p_{3}} frac {partial p_{3}}{partial z_{1}}" src="https://s0.wp.com/latex.php?latex=%5Cfrac+%7B%5Cpartial+L%7D%7B%5Cpartial+z1%7D%3D+%5Cfrac+%7B%5Cpartial+L%7D%7B%5Cpartial+p_%7B1%7D%7D+%5Cfrac+%7B%5Cpartial+p_%7B1%7D%7D%7B%5Cpartial+z_%7B1%7D%7D+%2B%5Cfrac+%7B%5Cpartial+L%7D%7B%5Cpartial+p_%7B2%7D%7D+%5Cfrac+%7B%5Cpartial+p_%7B2%7D%7D%7B%5Cpartial+z_%7B1%7D%7D+%2B%5Cfrac+%7B%5Cpartial+L%7D%7B%5Cpartial+p_%7B3%7D%7D+%5Cfrac+%7B%5Cpartial+p_%7B3%7D%7D%7B%5Cpartial+z_%7B1%7D%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> which similarly reduces to<br>
<img alt="frac {partial L}{partial z_{1}}=-frac {y1}{p1} p1(1-p1) - frac {y2}{p2}*(-p_{2}p_{1}) - frac {y3}{p3}*(-p_{3}p_{1})" title="frac {partial L}{partial z_{1}}=-frac {y1}{p1} p1(1-p1) - frac {y2}{p2}*(-p_{2}p_{1}) - frac {y3}{p3}*(-p_{3}p_{1})" src="https://s0.wp.com/latex.php?latex=%5Cfrac+%7B%5Cpartial+L%7D%7B%5Cpartial+z_%7B1%7D%7D%3D-%5Cfrac+%7By1%7D%7Bp1%7D+p1%281-p1%29+-+%5Cfrac+%7By2%7D%7Bp2%7D%2A%28-p_%7B2%7Dp_%7B1%7D%29+-+%5Cfrac+%7By3%7D%7Bp3%7D%2A%28-p_%7B3%7Dp_%7B1%7D%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="-y_{1}+ y_{1}p_{1} + y_{2}p_{1} + y_{3}p1 =  p_{1}sum (y_{1} + y_2 + y_3) - y_{1}  = p_{1} - y_{1}" title="-y_{1}+ y_{1}p_{1} + y_{2}p_{1} + y_{3}p1 =  p_{1}sum (y_{1} + y_2 + y_3) - y_{1}  = p_{1} - y_{1}" src="https://s0.wp.com/latex.php?latex=-y_%7B1%7D%2B+y_%7B1%7Dp_%7B1%7D+%2B+y_%7B2%7Dp_%7B1%7D+%2B+y_%7B3%7Dp1+%3D++p_%7B1%7D%5Csum+%28y_%7B1%7D+%2B+y_2+%2B+y_3%29+-+y_%7B1%7D++%3D+p_%7B1%7D+-+y_%7B1%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
Interestingly, despite the lengthy derivations the final result is simple and intuitive!</p>
<p>As seen in my post ‘<a rel="nofollow" href="https://gigadom.wordpress.com/2018/01/30/deep-learning-from-first-principles-in-python-r-and-octave-part-3/" target="_blank">Deep Learning from first principles with Python, R and Octave – Part 3</a> the key equations for forward and backward propagation are<br>
<img data-orig-file="https://gigadom.files.wordpress.com/2018/02/dl-21.png" data-orig-size="915,283" data-medium-file="https://gigadom.files.wordpress.com/2018/02/dl-21.png?w=300" data-attachment-id="6401" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/02/dl-21.png?w=450&h=209&fit=676%2C209" data-permalink="https://gigadom.wordpress.com/2018/02/26/deep-learning-from-first-principles-in-python-r-and-octave-part-4/dl-2-2/" data-large-file="https://gigadom.files.wordpress.com/2018/02/dl-21.png?w=450&amp;h=209&amp;fit=676%2C209" alt="" data-image-title="dl-2" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6401" data-recalc-dims="1"><br>
<strong>Forward propagation equations layer 1</strong><br>
<img alt="Z_{1} = W_{1}X +b_{1}" title="Z_{1} = W_{1}X +b_{1}" src="https://s0.wp.com/latex.php?latex=Z_%7B1%7D+%3D+W_%7B1%7DX+%2Bb_%7B1%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex">     and  <img alt="A_{1} = g(Z_{1})" title="A_{1} = g(Z_{1})" src="https://s0.wp.com/latex.php?latex=A_%7B1%7D+%3D+g%28Z_%7B1%7D%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<strong>Forward propagation equations layer 1</strong><br>
<img alt="Z_{2} = W_{2}A_{1} +b_{2}" title="Z_{2} = W_{2}A_{1} +b_{2}" src="https://s0.wp.com/latex.php?latex=Z_%7B2%7D+%3D+W_%7B2%7DA_%7B1%7D+%2Bb_%7B2%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex">  and  <img alt="A_{2} = S(Z_{2})" title="A_{2} = S(Z_{2})" src="https://s0.wp.com/latex.php?latex=A_%7B2%7D+%3D+S%28Z_%7B2%7D%29&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"></p>
<p><strong>Backward propagation equations layer 2</strong><br>
<img alt="partial L/partial W_{2} =partial L/partial Z_{2} *A_{1}" title="partial L/partial W_{2} =partial L/partial Z_{2} *A_{1}" src="https://s0.wp.com/latex.php?latex=%5Cpartial+L%2F%5Cpartial+W_%7B2%7D+%3D%5Cpartial+L%2F%5Cpartial+Z_%7B2%7D+%2AA_%7B1%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="partial L/partial b_{2} =partial L/partial Z_{2}" title="partial L/partial b_{2} =partial L/partial Z_{2}" src="https://s0.wp.com/latex.php?latex=%5Cpartial+L%2F%5Cpartial+b_%7B2%7D+%3D%5Cpartial+L%2F%5Cpartial+Z_%7B2%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="partial L/partial A_{1} = partial L/partial Z_{2} * W_{2}" title="partial L/partial A_{1} = partial L/partial Z_{2} * W_{2}" src="https://s0.wp.com/latex.php?latex=%5Cpartial+L%2F%5Cpartial+A_%7B1%7D+%3D+%5Cpartial+L%2F%5Cpartial+Z_%7B2%7D+%2A+W_%7B2%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<strong>Backward propagation equations layer 1</strong><br>
<img alt="partial L/partial W_{1} =partial L/partial Z_{1} *A_{0}" title="partial L/partial W_{1} =partial L/partial Z_{1} *A_{0}" src="https://s0.wp.com/latex.php?latex=%5Cpartial+L%2F%5Cpartial+W_%7B1%7D+%3D%5Cpartial+L%2F%5Cpartial+Z_%7B1%7D+%2AA_%7B0%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
<img alt="partial L/partial b_{1} =partial L/partial Z_{2}" title="partial L/partial b_{1} =partial L/partial Z_{2}" src="https://s0.wp.com/latex.php?latex=%5Cpartial+L%2F%5Cpartial+b_%7B1%7D+%3D%5Cpartial+L%2F%5Cpartial+Z_%7B2%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"></p>
<h4>2.0 Spiral data set</h4>
<p>As I mentioned earlier, I will be using the ‘spiral’ data from <a rel="nofollow" href="http://cs231n.github.io/neural-networks-case-study/" target="_blank">CS231n Convolutional Neural Networks</a> to ensure that my vectorized implementations in Python, R and Octave are correct. Here is the ‘spiral’ data set.</p>
<pre><code><span>import</span> numpy <span>as</span> np
<span>import</span> matplotlib.pyplot <span>as</span> plt
<span>import</span> os
os.chdir(<span>&quot;C:/junk/dl-4/dl-4&quot;</span>)
<span>exec</span>(open(<span>&quot;././DLfunctions41.py&quot;</span>).read())

<span># Create an input data set - Taken from CS231n Convolutional Neural networks</span>
<span># http://cs231n.github.io/neural-networks-case-study/</span>
N = <span>100</span> <span># number of points per class</span>
D = <span>2</span> <span># dimensionality</span>
K = <span>3</span> <span># number of classes</span>
X = np.zeros((N*K,D)) <span># data matrix (each row = single example)</span>
y = np.zeros(N*K, dtype=<span>'uint8'</span>) <span># class labels</span>
<span>for</span> j <span>in</span> range(K):
  ix = range(N*j,N*(j+<span>1</span>))
  r = np.linspace(<span>0.0</span>,<span>1</span>,N) <span># radius</span>
  t = np.linspace(j*<span>4</span>,(j+<span>1</span>)*<span>4</span>,N) + np.random.randn(N)*<span>0.2</span> <span># theta</span>
  X[ix] = np.c_[r*np.sin(t), r*np.cos(t)]
  y[ix] = j
<span># Plot the data</span>
plt.scatter(X[:, <span>0</span>], X[:, <span>1</span>], c=y, s=<span>40</span>, cmap=plt.cm.Spectral)
plt.savefig(<span>&quot;fig1.png&quot;</span>, bbox_inches=<span>'tight'</span>)</code></pre>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/02/fig1.png?w=456" data-orig-size="571,418" data-medium-file="https://gigadom.files.wordpress.com/2018/02/fig1.png?w=456?w=300" data-attachment-id="6366" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/02/fig1.png?w=456" data-permalink="https://gigadom.wordpress.com/2018/02/26/deep-learning-from-first-principles-in-python-r-and-octave-part-4/fig1-15/" data-large-file="https://gigadom.files.wordpress.com/2018/02/fig1.png?w=456?w=571" alt="" data-image-title="fig1" sizes="(max-width: 571px) 100vw, 571px" data-comments-opened="1" class="wp-image-6366" data-recalc-dims="1"><br>
The implementations of the vectorized Python, R and Octave code are shown diagrammatically below<br>
<img data-orig-file="https://gigadom.files.wordpress.com/2018/02/dl-3.png" data-orig-size="822,662" data-medium-file="https://gigadom.files.wordpress.com/2018/02/dl-3.png?w=300" data-attachment-id="6416" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/02/dl-3.png?w=450&h=544&fit=676%2C544" data-permalink="https://gigadom.wordpress.com/2018/02/26/deep-learning-from-first-principles-in-python-r-and-octave-part-4/dl-3/" data-large-file="https://gigadom.files.wordpress.com/2018/02/dl-3.png?w=450&amp;h=544&amp;fit=676%2C544" alt="" data-image-title="dl-3" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6416" data-recalc-dims="1"></p>
<h4>2.1 Multi-class classification with Softmax – Python code</h4>
<p>A simple 2 layer Neural network with a single hidden layer , with 100 Relu activation units in the hidden layer and the Softmax activation unit in the output layer is used for multi-class classification. This Deep Learning Network, plots the non-linear boundary of the 3 classes as shown below</p>
<pre><code><span>import</span> numpy <span>as</span> np
<span>import</span> matplotlib.pyplot <span>as</span> plt
<span>import</span> os
os.chdir(<span>&quot;C:/junk/dl-4/dl-4&quot;</span>)
<span>exec</span>(open(<span>&quot;././DLfunctions41.py&quot;</span>).read())

<span># Read the input data</span>
N = <span>100</span> <span># number of points per class</span>
D = <span>2</span> <span># dimensionality</span>
K = <span>3</span> <span># number of classes</span>
X = np.zeros((N*K,D)) <span># data matrix (each row = single example)</span>
y = np.zeros(N*K, dtype=<span>'uint8'</span>) <span># class labels</span>
<span>for</span> j <span>in</span> range(K):
  ix = range(N*j,N*(j+<span>1</span>))
  r = np.linspace(<span>0.0</span>,<span>1</span>,N) <span># radius</span>
  t = np.linspace(j*<span>4</span>,(j+<span>1</span>)*<span>4</span>,N) + np.random.randn(N)*<span>0.2</span> <span># theta</span>
  X[ix] = np.c_[r*np.sin(t), r*np.cos(t)]
  y[ix] = j
  
<span># Set the number of features, hidden units in hidden layer and number of classess</span>
numHidden=<span>100</span> <span># No of hidden units in hidden layer</span>
numFeats= <span>2</span> <span># dimensionality</span>
numOutput = <span>3</span> <span># number of classes</span>

<span># Initialize the model</span>
parameters=initializeModel(numFeats,numHidden,numOutput)
W1= parameters[<span>'W1'</span>]
b1= parameters[<span>'b1'</span>]
W2= parameters[<span>'W2'</span>]
b2= parameters[<span>'b2'</span>]

<span># Set the learning rate</span>
learningRate=<span>0.6</span> 

<span># Initialize losses</span>
losses=[]
<span># Perform Gradient descent</span>
<span>for</span> i <span>in</span> range(<span>10000</span>):
    <span># Forward propagation through hidden layer with Relu units</span>
    A1,cache1= layerActivationForward(X.T,W1,b1,<span>'relu'</span>)
    
    <span># Forward propagation through output layer with Softmax</span>
    A2,cache2 = layerActivationForward(A1,W2,b2,<span>'softmax'</span>)
    
    <span># No of training examples</span>
    numTraining = X.shape[<span>0</span>]
    <span># Compute log probs. Take the log prob of correct class based on output y</span>
    correct_logprobs = -np.log(A2[range(numTraining),y])
    <span># Conpute loss</span>
    loss = np.sum(correct_logprobs)/numTraining
    
    <span># Print the loss</span>
    <span>if</span> i % <span>1000</span> == <span>0</span>:
        <span>print</span>(<span>&quot;iteration %d: loss %f&quot;</span> % (i, loss))
        losses.append(loss)

    dA=<span>0</span>

    <span># Backward propagation through output layer with Softmax</span>
    dA1,dW2,db2 = layerActivationBackward(dA, cache2, y, activationFunc=<span>'softmax'</span>)
    <span># Backward propagation through hidden layer with Relu unit</span>
    dA0,dW1,db1 = layerActivationBackward(dA1.T, cache1, y, activationFunc=<span>'relu'</span>)
    
    <span>#Update paramaters with the learning rate</span>
    W1 += -learningRate * dW1
    b1 += -learningRate * db1
    W2 += -learningRate * dW2.T
    b2 += -learningRate * db2.T

<span>#Plot losses vs iterations </span>
i=np.arange(<span>0</span>,<span>10000</span>,<span>1000</span>)
plt.plot(i,losses)

plt.xlabel(<span>'Iterations'</span>)
plt.ylabel(<span>'Loss'</span>)
plt.title(<span>'Losses vs Iterations'</span>)
plt.savefig(<span>&quot;fig2.png&quot;</span>, bbox=<span>&quot;tight&quot;</span>)

<span>#Compute the multi-class Confusion Matrix</span>
<span>from</span> sklearn.metrics <span>import</span> confusion_matrix
<span>from</span> sklearn.metrics <span>import</span> accuracy_score, precision_score, recall_score, f1_score

<span># We need to determine the predicted values from the learnt data</span>
<span># Forward propagation through hidden layer with Relu units</span>
A1,cache1= layerActivationForward(X.T,W1,b1,<span>'relu'</span>)
    
<span># Forward propagation through output layer with Softmax</span>
A2,cache2 = layerActivationForward(A1,W2,b2,<span>'softmax'</span>)
<span>#Compute predicted values from weights and biases</span>
yhat=np.argmax(A2, axis=<span>1</span>)

a=confusion_matrix(y.T,yhat.T)
<span>print</span>(<span>&quot;Multi-class Confusion Matrix&quot;</span>)
<span>print</span>(a)</code></pre>
<pre><code>## iteration 0: loss 1.098507
## iteration 1000: loss 0.214611
## iteration 2000: loss 0.043622
## iteration 3000: loss 0.032525
## iteration 4000: loss 0.025108
## iteration 5000: loss 0.021365
## iteration 6000: loss 0.019046
## iteration 7000: loss 0.017475
## iteration 8000: loss 0.016359
## iteration 9000: loss 0.015703
## Multi-class Confusion Matrix
## [[ 99   1   0]
##  [  0 100   0]
##  [  0   1  99]]</code></pre>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/02/fig2.png?w=456" data-orig-size="640,480" data-medium-file="https://gigadom.files.wordpress.com/2018/02/fig2.png?w=456?w=300" data-attachment-id="6367" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/02/fig2.png?w=456" data-permalink="https://gigadom.wordpress.com/2018/02/26/deep-learning-from-first-principles-in-python-r-and-octave-part-4/fig2-13/" data-large-file="https://gigadom.files.wordpress.com/2018/02/fig2.png?w=456?w=640" alt="" data-image-title="fig2" sizes="(max-width: 640px) 100vw, 640px" data-comments-opened="1" class="wp-image-6367" data-recalc-dims="1"></p>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/02/fig3.png?w=456" data-orig-size="432,288" data-medium-file="https://gigadom.files.wordpress.com/2018/02/fig3.png?w=456?w=300" data-attachment-id="6368" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/02/fig3.png?w=456" data-permalink="https://gigadom.wordpress.com/2018/02/26/deep-learning-from-first-principles-in-python-r-and-octave-part-4/fig3-12/" data-large-file="https://gigadom.files.wordpress.com/2018/02/fig3.png?w=456?w=432" alt="" data-image-title="fig3" sizes="(max-width: 432px) 100vw, 432px" data-comments-opened="1" class="wp-image-6368" data-recalc-dims="1"></p>
<p>To know more details about Confusion Matrix and other related measures check out my book  <a rel="nofollow" href="https://gigadom.wordpress.com/2017/12/05/my-book-practical-machine-learning-with-r-and-python-on-amazon/" target="_blank">My book ‘Practical Machine Learning with R and Python’ on Amazon</a><br>
<img data-orig-file="https://gigadom.files.wordpress.com/2018/02/untitled1.png?w=450&amp;h=214&amp;fit=676%2C214" data-orig-size="676,214" data-medium-file="https://gigadom.files.wordpress.com/2018/02/untitled1.png?w=450&amp;h=214&amp;fit=676%2C214?w=300" data-attachment-id="6406" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/02/untitled1.png?w=450&h=214&fit=676%2C214" data-permalink="https://gigadom.wordpress.com/2018/02/26/deep-learning-from-first-principles-in-python-r-and-octave-part-4/untitled1-7/" data-large-file="https://gigadom.files.wordpress.com/2018/02/untitled1.png?w=450&amp;h=214&amp;fit=676%2C214?w=676" alt="" data-image-title="untitled1" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6406" data-recalc-dims="1"></p>
<h4>2.2 Multi-class classification with Softmax – R code</h4>
<p>The spiral data set created with Python was saved, and is used as the input with R code. The R Neural Network seems to perform much,much slower than both Python and Octave. Not sure why! Incidentally the computation of loss and the softmax derivative are identical for both R and Octave. yet R is much slower. To compute the softmax derivative I create matrices for the One Hot Encoded yi and then stack them before subtracting pi-yi. I am sure there is a more elegant and more efficient way to do this, much like Python. Any suggestions?</p>
<pre><code><span>library</span>(ggplot2)
<span>library</span>(dplyr)
<span>library</span>(RColorBrewer)</code><code></code><code></code></pre>
<pre><code></code><code><span>source</span>(<span>&quot;DLfunctions41.R&quot;</span>)
<span># Read the spiral dataset</span>
Z &lt;- as.matrix(read.csv(<span>&quot;spiral.csv&quot;</span>,header=<span>FALSE</span>)) 
Z1=data.frame(Z)
<span>#Plot the dataset</span>
ggplot(Z1,aes(x=V1,y=V2,col=V3)) +geom_point() + 
  scale_colour_gradientn(colours = brewer.pal(<span>10</span>, <span>&quot;Spectral&quot;</span>))</code></pre>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/02/fig1-1.png" data-orig-size="1344,960" data-medium-file="https://gigadom.files.wordpress.com/2018/02/fig1-1.png?w=300" data-attachment-id="6388" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/02/fig1-1.png?w=450&h=483&fit=676%2C483" data-permalink="https://gigadom.wordpress.com/2018/02/26/deep-learning-from-first-principles-in-python-r-and-octave-part-4/fig1-1-8/" data-large-file="https://gigadom.files.wordpress.com/2018/02/fig1-1.png?w=450&amp;h=483&amp;fit=676%2C483" alt="" data-image-title="fig1-1" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6388" data-recalc-dims="1"></p>
<pre><code></code><code><span># Setup the data
X &lt;-Z[,<span>1</span>:<span>2</span>]
y &lt;-Z[,<span>3</span>]
X1 &lt;-t(X)
Y1 &lt;-t(y)
<span># Initialize number of features, number of hidden units in hidden layer and</span>
<span># number of classes</span>
numFeats&lt;-<span>2</span> <span># No features</span>
numHidden&lt;-<span>100</span> <span># No of hidden units</span>
numOutput&lt;-<span>3</span> <span># No of classes</span>
<span># Initialize model</span>
parameters &lt;-initializeModel(numFeats, numHidden,numOutput)
W1 &lt;-parameters[[<span>'W1'</span>]]
b1 &lt;-parameters[[<span>'b1'</span>]]
W2 &lt;-parameters[[<span>'W2'</span>]]
b2 &lt;-parameters[[<span>'b2'</span>]]
<span># Set the learning rate</span>
learningRate &lt;-<span>0.5</span>
<span># Initialize losses</span>
losses &lt;-<span>NULL</span>
<span># Perform gradient descent</span>
<span>for</span>(i <span>in</span> <span>0</span>:<span>9000</span>){
<span># Forward propagation through hidden layer with Relu units</span>
retvals &lt;-layerActivationForward(X1,W1,b1,<span>'relu'</span>)
A1 &lt;-retvals[[<span>'A'</span>]]
cache1 &lt;-retvals[[<span>'cache'</span>]]
forward_cache1 &lt;-cache1[[<span>'forward_cache1'</span>]]
activation_cache &lt;-cache1[[<span>'activation_cache'</span>]]
<span># Forward propagation through output layer with Softmax units</span>
retvals = layerActivationForward(A1,W2,b2,<span>'softmax'</span>)
A2 &lt;-retvals[[<span>'A'</span>]]
cache2 &lt;-retvals[[<span>'cache'</span>]]
forward_cache2 &lt;-cache2[[<span>'forward_cache1'</span>]]
activation_cache2 &lt;-cache2[[<span>'activation_cache'</span>]]
<span># No oftraining examples</span>
numTraining &lt;-dim(X)[<span>1</span>]
dA &lt;-<span>0</span>
<span># Select the elements where the y values are 0, 1 or 2 and make a vector</span>
a=c(A2[y==<span>0</span>,<span>1</span>],A2[y==<span>1</span>,<span>2</span>],A2[y==<span>2</span>,<span>3</span>])
<span># Take log</span>
correct_probs = -log(a)
<span># Compute loss</span>
loss= sum(correct_probs)/numTraining
<span>if</span>(i %% <span>1000</span> == <span>0</span>){
sprintf(<span>&quot;iteration %d: loss %f&quot;</span>,i, loss)
print(loss)
}
<span># Backward propagation through output layer with Softmax units</span>
retvals = layerActivationBackward(dA, cache2, y, activationFunc=<span>'softmax'</span>)
dA1 = retvals[[<span>'dA_prev'</span>]]
dW2= retvals[[<span>'dW'</span>]]
db2= retvals[[<span>'db'</span>]]
<span># Backward propagation through hidden layer with Relu units</span>
retvals = layerActivationBackward(t(dA1), cache1, y, activationFunc=<span>'relu'</span>)
dA0 = retvals[[<span>'dA_prev'</span>]]
dW1= retvals[[<span>'dW'</span>]]
db1= retvals[[<span>'db'</span>]]
<span># Update parameters</span>
W1 &lt;-W1 - learningRate * dW1
b1 &lt;-b1 - learningRate * db1
W2 &lt;-W2 - learningRate * t(dW2)
b2 &lt;-b2 - learningRate * t(db2)
}</span></code></pre>
<pre><code>## [1] 1.212487
## [1] 0.5740867
## [1] 0.4048824
## [1] 0.3561941
## [1] 0.2509576
## [1] 0.7351063
## [1] 0.2066114
## [1] 0.2065875
## [1] 0.2151943
## [1] 0.1318807</code></pre>
<p> </p>
<pre><code><span>#Create iterations</span>
iterations &lt;- seq(<span>0</span>,<span>10</span>)
<span>#df=data.frame(iterations,losses)</span>
<span>ggplot(df,aes(x=iterations,y=losses)) + geom_point() + geom_line(color=&quot;blue&quot;) +</span>
 <span> ggtitle(&quot;Losses vs iterations&quot;) + xlab(&quot;Iterations&quot;) + ylab(&quot;Loss&quot;)</span>

plotDecisionBoundary(Z,W1,b1,W2,b2)</code></pre>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/02/untitled.png" data-orig-size="1201,617" data-medium-file="https://gigadom.files.wordpress.com/2018/02/untitled.png?w=300" data-attachment-id="6393" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/02/untitled.png?w=450&h=347&fit=676%2C347" data-permalink="https://gigadom.wordpress.com/2018/02/26/deep-learning-from-first-principles-in-python-r-and-octave-part-4/untitled-119/" data-large-file="https://gigadom.files.wordpress.com/2018/02/untitled.png?w=450&amp;h=347&amp;fit=676%2C347" alt="" data-image-title="Untitled" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6393" data-recalc-dims="1"><br>
<img><img data-orig-file="https://gigadom.files.wordpress.com/2018/02/fig1-21.png" data-orig-size="1344,960" data-medium-file="https://gigadom.files.wordpress.com/2018/02/fig1-21.png?w=300" data-attachment-id="6389" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/02/fig1-21.png?w=450&h=483&fit=676%2C483" data-permalink="https://gigadom.wordpress.com/2018/02/26/deep-learning-from-first-principles-in-python-r-and-octave-part-4/fig1-2-5/" data-large-file="https://gigadom.files.wordpress.com/2018/02/fig1-21.png?w=450&amp;h=483&amp;fit=676%2C483" alt="" data-image-title="fig1-2" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6389" data-recalc-dims="1"><br>
<strong>Multi-class Confusion Matrix</strong></p>
<pre><code>library(caret)
library(e1071)

# Forward propagation through hidden layer with Relu units
retvals &lt;- layerActivationForward(X1,W1,b1,'relu')
A1 &lt;- retvals[['A']]

# Forward propagation through output layer with Softmax units
retvals = layerActivationForward(A1,W2,b2,'softmax')
A2 &lt;- retvals[['A']]
yhat &lt;- apply(A2, 1,which.max) -1
Confusion Matrix and Statistics
          Reference
Prediction  0  1  2
         0 97  0  1
         1  2 96  4
         2  1  4 95

Overall Statistics                                        
               Accuracy : 0.96            
                 95% CI : (0.9312, 0.9792)
    No Information Rate : 0.3333          
    P-Value [Acc > NIR] : &lt;2e-16          
                                          
                  Kappa : 0.94            
 Mcnemar's Test P-Value : 0.5724          
Statistics by Class:

                     Class: 0 Class: 1 Class: 2
Sensitivity            0.9700   0.9600   0.9500
Specificity            0.9950   0.9700   0.9750
Pos Pred Value         0.9898   0.9412   0.9500
Neg Pred Value         0.9851   0.9798   0.9750
Prevalence             0.3333   0.3333   0.3333
Detection Rate         0.3233   0.3200   0.3167
Detection Prevalence   0.3267   0.3400   0.3333
Balanced Accuracy      0.9825   0.9650   0.9625
</code></pre>
<p>My book “<a rel="nofollow" href="https://www.amazon.com/dp/1973443503" target="_blank">Practical Machine Learning with R and Python</a>” includes the implementation for many Machine Learning algorithms and associated metrics. Pick up your copy today!</p>
<h4>2.3 Multi-class classification with Softmax – Octave code</h4>
<p>A 2 layer Neural network with the Softmax activation unit in the output layer is constructed in Octave. The same spiral data set is used for Octave also<br>
<code><br>
source(&quot;DL41functions.m&quot;)<br>
# Read the spiral data<br>
data=csvread(&quot;spiral.csv&quot;);<br>
# Setup the data<br>
X=data(:,1:2);<br>
Y=data(:,3);<br>
# Set the number of features, number of hidden units in hidden layer and number of classes<br>
numFeats=2; #No features<br>
numHidden=100; # No of hidden units<br>
numOutput=3; # No of classes<br>
# Initialize model<br>
[W1 b1 W2 b2] = initializeModel(numFeats,numHidden,numOutput);<br>
# Initialize losses<br>
losses=[]<br>
#Initialize learningRate<br>
learningRate=0.5;<br>
for k =1:10000<br>
# Forward propagation through hidden layer with Relu units<br>
[A1,cache1 activation_cache1]= layerActivationForward(X',W1,b1,activationFunc ='relu');<br>
# Forward propagation through output layer with Softmax units<br>
[A2,cache2 activation_cache2] =<br>
layerActivationForward(A1,W2,b2,activationFunc='softmax');<br>
# No of training examples<br>
numTraining = size(X)(1);<br>
# Select rows where Y=0,1,and 2 and concatenate to a long vector<br>
a=[A2(Y==0,1) ;A2(Y==1,2) ;A2(Y==2,3)];<br>
#Select the correct column for log prob<br>
correct_probs = -log(a);<br>
#Compute log loss<br>
loss= sum(correct_probs)/numTraining;<br>
if(mod(k,1000) == 0)<br>
disp(loss);<br>
losses=[losses loss];<br>
endif<br>
dA=0;<br>
# Backward propagation through output layer with Softmax units<br>
[dA1 dW2 db2] = layerActivationBackward(dA, cache2, activation_cache2,Y,activationFunc='softmax');<br>
# Backward propagation through hidden layer with Relu units<br>
[dA0,dW1,db1] = layerActivationBackward(dA1', cache1, activation_cache1, Y, activationFunc='relu');<br>
#Update parameters<br>
W1 += -learningRate * dW1;<br>
b1 += -learningRate * db1;<br>
W2 += -learningRate * dW2';<br>
b2 += -learningRate * db2';<br>
endfor<br>
# Plot Losses vs Iterations<br>
iterations=0:1000:9000<br>
plotCostVsIterations(iterations,losses)<br>
# Plot the decision boundary<br>
plotDecisionBoundary( X,Y,W1,b1,W2,b2)</code></p>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2018/02/figo2.png" data-orig-size="1200,900" data-medium-file="https://gigadom.files.wordpress.com/2018/02/figo2.png?w=300" data-attachment-id="6383" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/02/figo2.png?w=450&h=507&fit=676%2C507" data-permalink="https://gigadom.wordpress.com/2018/02/26/deep-learning-from-first-principles-in-python-r-and-octave-part-4/figo2/" data-large-file="https://gigadom.files.wordpress.com/2018/02/figo2.png?w=450&amp;h=507&amp;fit=676%2C507" alt="" data-image-title="figo2" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6383" data-recalc-dims="1"></p>
<h4><img data-orig-file="https://gigadom.files.wordpress.com/2018/02/fig-o.png" data-orig-size="784,678" data-medium-file="https://gigadom.files.wordpress.com/2018/02/fig-o.png?w=300" data-attachment-id="6384" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2018/02/fig-o.png?w=450&h=585&fit=676%2C585" data-permalink="https://gigadom.wordpress.com/2018/02/26/deep-learning-from-first-principles-in-python-r-and-octave-part-4/fig-o/" data-large-file="https://gigadom.files.wordpress.com/2018/02/fig-o.png?w=450&amp;h=585&amp;fit=676%2C585" alt="" data-image-title="fig-o" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-6384" data-recalc-dims="1"></h4>
<p>The code for the Python, R and Octave implementations can be downloaded from Github at <a rel="nofollow" href="https://github.com/tvganesh/DeepLearning-Part4" target="_blank">Deep Learning – Part 4</a></p>
<h4>Conclusion</h4>
<p>In this post I have implemented a 2 layer Neural Network with the Softmax classifier. In <a rel="nofollow" href="https://gigadom.wordpress.com/2018/01/30/deep-learning-from-first-principles-in-python-r-and-octave-part-3/" target="_blank">Part 3</a>, I implemented a multi-layer Deep Learning Network. I intend to include the Softmax activation unit into the generalized multi-layer Deep Network along with the other activation units of sigmoid,tanh and relu.</p>
<p>Stick around, I’ll be back!!<br>
Watch this space!</p>
<p><strong>References</strong><br>
1. <a rel="nofollow" href="https://www.coursera.org/specializations/deep-learning" target="_blank">Deep Learning Specialization</a><br>
2. <a rel="nofollow" href="https://www.coursera.org/learn/neural-networks" target="_blank">Neural Networks for Machine Learning</a><br>
3. <a rel="nofollow" href="http://cs231n.github.io/neural-networks-case-study/" target="_blank">CS231 Convolutional Neural Networks for Visual Recognition</a><br>
4. <a rel="nofollow" href="https://eli.thegreenplace.net/2016/the-softmax-function-and-its-derivative/" target="_blank">Eli Bendersky’s Website – The Softmax function and its derivative</a><br>
5. <a rel="nofollow" href="https://stats.stackexchange.com/questions/235528/backpropagation-with-softmax-cross-entropy" target="_blank">Cross Validated – Backpropagation with Softmax / Cross Entropy</a><br>
6. <a rel="nofollow" href="https://stackoverflow.com/questions/41663874/cs231n-how-to-calculate-gradient-for-softmax-loss-function" target="_blank">Stackoverflow – CS231n: How to calculate gradient for Softmax loss function?</a><br>
7. <a rel="nofollow" href="https://math.stackexchange.com/questions/945871/derivative-of-softmax-loss-function" target="_blank">Math Stack Exchange – Derivative of Softmax</a><br>
8. <a rel="nofollow" href="https://arxiv.org/abs/1802.01528" target="_blank">The Matrix Calculus for Deep Learning</a></p>
<p><b>You may like</b><br>
1.<a rel="nofollow" href="https://gigadom.wordpress.com/2017/12/05/my-book-practical-machine-learning-with-r-and-python-on-amazon/" target="_blank">My book ‘Practical Machine Learning with R and Python’ on Amazon</a><br>
2. <a rel="nofollow" href="https://gigadom.wordpress.com/2017/09/11/my-travels-through-the-realms-of-data-science-machine-learning-deep-learning-and-ai/" target="_blank">My travels through the realms of Data Science, Machine Learning, Deep Learning and (AI)</a><br>
3. <a rel="nofollow" href="https://gigadom.wordpress.com/2012/05/11/deblurring-with-opencv-weiner-filter-reloaded/" target="_blank">Deblurring with OpenCV: Weiner filter reloaded</a><br>
4. <a rel="nofollow" href="https://gigadom.wordpress.com/2013/12/11/a-method-to-crowd-source-pothole-marking-on-indian-roads/" target="_blank">A method to crowd source pothole marking on (Indian) roads</a><br>
5. <a rel="nofollow" href="https://gigadom.wordpress.com/2014/08/19/rock-n-roll-with-bluemix-cloudant-nodeexpress/" target="_blank">Rock N’ Roll with Bluemix, Cloudant &amp; NodeExpress</a><br>
6. <a rel="nofollow" href="https://gigadom.wordpress.com/2012/01/03/sea-shells-on-the-seashore/" target="_blank">Sea shells on the seashore</a><br>
7. <a rel="nofollow" href="https://gigadom.wordpress.com/2011/05/13/design-principles-of-scalable-distributed-systems/" target="_blank">Design Principles of Scalable, Distributed Systems</a></p>
<p>To see all post click <a rel="nofollow" href="https://gigadom.wordpress.com/aa-2/" target="_blank">Index of posts</a></p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://gigadom.wordpress.com/2018/02/26/deep-learning-from-first-principles-in-python-r-and-octave-part-4/"> R – Giga thoughts …</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/n0Q25cFb-K4" height="1"></a></div></strong>