<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/classification-from-scratch-neural-nets-6-8/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://freakonometrics.hypotheses.org/52774"> R-english – Freakonometrics</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Sixth post of our series on <a rel="nofollow" href="https://freakonometrics.hypotheses.org/52731" target="_blank">classification from scratch</a>. The latest one was on the <a rel="nofollow" href="https://freakonometrics.hypotheses.org/52894" target="_blank">lasso regression</a>, which was still based on a <a rel="nofollow" href="https://freakonometrics.hypotheses.org/52747" target="_blank">logistic regression</a> model, assuming that the variable of interest <img alt="Y" title="Y" src="https://s0.wp.com/latex.php?latex=Y&bg=ffffff&fg=000&s=0" class="latex"> has a Bernoulli distribution. From now on, we will discuss technique that did not originate from those probabilistic models, even if they might still have a probabilistic interpretation. Somehow. Today, we will start with neural nets.</p>
<p>Maybe I should start with a disclaimer. The goal is not to replicate well designed R functions, used for predictive modeling. It is simply to get a basic understanding of what’s going on.</p>
<h2>Networs, nodes and edges</h2>
<p>First of all, neurals nets are nets, or <a rel="nofollow" href="https://freakonometrics.hypotheses.org/category/courses/networks" target="_blank">networks</a>. I will skip the parallel with “neural” <a rel="nofollow" href="https://en.wikipedia.org/wiki/Neuron" target="_blank">stuff</a> because it does not help me understanding what is happening (all apologies for my poor knowledge on biology, and cells)<br>
<a rel="nofollow" href="https://en.wikipedia.org/wiki/Neuron" target="_blank"><img src="https://i1.wp.com/freakonometrics.hypotheses.org/files/2018/06/http://freakonometrics.hypotheses.org/files/2018/06/Blausen_0657_MultipolarNeuron.png?resize=331%2C213&ssl=1" alt="" width="331" data-recalc-dims="1" height="213"></a><br>
So, it’s about some network. Networks have nodes, and edges (possibly connected) that connect nodes,<br>
<img src="https://i2.wp.com/freakonometrics.hypotheses.org/files/2018/06/net2.png?resize=383%2C164&ssl=1" alt="" width="383" data-recalc-dims="1" height="164"><br>
or maybe, to more specific (at least it helped me understanding what’s going on), some sort of <a rel="nofollow" href="https://en.wikipedia.org/wiki/Flow_network" target="_blank">flow network</a>,<br>
<img src="https://i0.wp.com/freakonometrics.hypotheses.org/files/2018/06/net1.png?resize=366%2C162&ssl=1" alt="" width="366" data-recalc-dims="1" height="162"><br>
In such a network, we usually have sources (here multiple) sources (here <img alt="\color{red}\{s_1,s_2,s_3\}" title="\color{red}\{s_1,s_2,s_3\}" src="https://s0.wp.com/latex.php?latex=%5Ccolor%7Bred%7D%5C%7Bs_1%2Cs_2%2Cs_3%5C%7D&bg=ffffff&fg=000&s=0" class="latex">), on the left, on a sink (here <img alt="\{\color{blue}t\}" title="\{\color{blue}t\}" src="https://s0.wp.com/latex.php?latex=%5C%7B%5Ccolor%7Bblue%7Dt%5C%7D&bg=ffffff&fg=000&s=0" class="latex">), on the right. To continue with this metaphorical introduction, information from the sources should reach the sink. An usually, sources are explanatory variables, <img alt="\{\mathbf{x}_1,\cdots,\mathbf{x}_p\}" title="\{\mathbf{x}_1,\cdots,\mathbf{x}_p\}" src="https://s0.wp.com/latex.php?latex=%5C%7B%5Cmathbf%7Bx%7D_1%2C%5Ccdots%2C%5Cmathbf%7Bx%7D_p%5C%7D&bg=ffffff&fg=000&s=0" class="latex">, and the sink is our variable of interest <img alt="\mathbf{y}" title="\mathbf{y}" src="https://s0.wp.com/latex.php?latex=%5Cmathbf%7By%7D&bg=ffffff&fg=000&s=0" class="latex">. And we want to create a graph, from the sources to the sink. We will have directed edges, with only one (unique) direction, where we will put weights. It is not a flow, the parallel with flow will stop here. For instance, the most simple network will be the following one, with no layer (i.e no node between the source and the sink)<br>
<img alt="" src="https://i2.wp.com/freakonometrics.hypotheses.org/files/2018/05/NN-ub1.png?w=456&ssl=1" data-recalc-dims="1"><br>
The output here is a binary variable <img alt="y\in\{0,1\}" title="y\in\{0,1\}" src="https://s0.wp.com/latex.php?latex=y%5Cin%5C%7B0%2C1%5C%7D&bg=ffffff&fg=000&s=0" class="latex"> (it can also be <img alt="y\in\{-1,+1\}" title="y\in\{-1,+1\}" src="https://s0.wp.com/latex.php?latex=y%5Cin%5C%7B-1%2C%2B1%5C%7D&bg=ffffff&fg=000&s=0" class="latex"> but here, it’s not a big deal). In our network, our output will be <img alt="y\in(0,1)" title="y\in(0,1)" src="https://s0.wp.com/latex.php?latex=y%5Cin%280%2C1%29&bg=ffffff&fg=000&s=0" class="latex">, because it is more easy to handly. For instance, consider <img alt="y=f(" title="y=f(" src="https://s0.wp.com/latex.php?latex=y%3Df%28&bg=ffffff&fg=000&s=0" class="latex">something<img alt=")" title=")" src="https://s0.wp.com/latex.php?latex=%29&bg=ffffff&fg=000&s=0" class="latex">, for some function <img alt="f" title="f" src="https://s0.wp.com/latex.php?latex=f&bg=ffffff&fg=000&s=0" class="latex"> taking values in <img alt="(0,1)" title="(0,1)" src="https://s0.wp.com/latex.php?latex=%280%2C1%29&bg=ffffff&fg=000&s=0" class="latex">. One can consider the <a rel="nofollow" href="https://en.wikipedia.org/wiki/Sigmoid_function" target="_blank">sigmoid</a> function<img alt="f(x)=\frac{1}{1+e^{-x}}=\frac{e^{x}}{e^{x}+1}" title="f(x)=\frac{1}{1+e^{-x}}=\frac{e^{x}}{e^{x}+1}" src="https://s0.wp.com/latex.php?latex=f%28x%29%3D%5Cfrac%7B1%7D%7B1%2Be%5E%7B-x%7D%7D%3D%5Cfrac%7Be%5E%7Bx%7D%7D%7Be%5E%7Bx%7D%2B1%7D&bg=ffffff&fg=000&s=0" class="latex">which is actually the <a rel="nofollow" href="https://en.wikipedia.org/wiki/Logistic_function" target="_blank">logistic</a> function (so we should not be surprised to have results somehow close the logistic regression…). This function <img alt="f" title="f" src="https://s0.wp.com/latex.php?latex=f&bg=ffffff&fg=000&s=0" class="latex"> is called the <a rel="nofollow" href="https://en.wikipedia.org/wiki/Activation_function" target="_blank">activation function</a>, and there are thousands of such functions. If <img alt="y\in\{-1,+1\}" title="y\in\{-1,+1\}" src="https://s0.wp.com/latex.php?latex=y%5Cin%5C%7B-1%2C%2B1%5C%7D&bg=ffffff&fg=000&s=0" class="latex">, people consider the hyperbolic tangent<img alt="f(x)=\tanh(x)={\frac {(e^{x}-e^{-x})}{(e^{x}+e^{-x})}}" title="f(x)=\tanh(x)={\frac {(e^{x}-e^{-x})}{(e^{x}+e^{-x})}}" src="https://s0.wp.com/latex.php?latex=f%28x%29%3D%5Ctanh%28x%29%3D%7B%5Cfrac+%7B%28e%5E%7Bx%7D-e%5E%7B-x%7D%29%7D%7B%28e%5E%7Bx%7D%2Be%5E%7B-x%7D%29%7D%7D&bg=ffffff&fg=000&s=0" class="latex">or the inverse tangent function<br>
<img alt="f(x)=\tan ^{-1}(x)" title="f(x)=\tan ^{-1}(x)" src="https://s0.wp.com/latex.php?latex=f%28x%29%3D%5Ctan+%5E%7B-1%7D%28x%29&bg=ffffff&fg=000&s=0" class="latex">And as input for such function, we consider a weighted sum of incoming nodes. So here<img alt="y_i=f\left(\sum_{j=1}^p\omega_j x_{j,i}\right)" title="y_i=f\left(\sum_{j=1}^p\omega_j x_{j,i}\right)" src="https://s0.wp.com/latex.php?latex=y_i%3Df%5Cleft%28%5Csum_%7Bj%3D1%7D%5Ep%5Comega_j+x_%7Bj%2Ci%7D%5Cright%29&bg=ffffff&fg=000&s=0" class="latex">We can also add a constant actually<img alt="y_i=f\left(\omega_0+\sum_{j=1}^p\omega_j x_{j,i}\right)" title="y_i=f\left(\omega_0+\sum_{j=1}^p\omega_j x_{j,i}\right)" src="https://s0.wp.com/latex.php?latex=y_i%3Df%5Cleft%28%5Comega_0%2B%5Csum_%7Bj%3D1%7D%5Ep%5Comega_j+x_%7Bj%2Ci%7D%5Cright%29&bg=ffffff&fg=000&s=0" class="latex">So far, we are not far away from the logistic regression. Except that our starting point was a probabilistic model, in the sense that the later was interpreted as a probability (the probability that <img alt="Y=1" title="Y=1" src="https://s0.wp.com/latex.php?latex=Y%3D1&bg=ffffff&fg=000&s=0" class="latex">) and we wanted the model with the highest likelihood. But we’ll talk about selection of weights later one. First, let us construct our first (very simple) neural network. First, we have the sigmoid function</p>
<div><table><tr><td><pre>1
</pre></td><td><pre>sigmoid <span>=</span> <span>function</span><span>(</span>x<span>)</span> <span>1</span> <span>/</span> <span>(</span><span>1</span> <span>+</span> <span>exp</span><span>(</span><span>-</span>x<span>)</span><span>)</span></pre></td></tr></table></div>
<p>The consider some weights. In our model with seven explanatory variables, with need 7 weights. Or 8 if we include the constant term. Let us consider <img alt="\mathbf{\omega}=\mathbf{1}" title="\mathbf{\omega}=\mathbf{1}" src="https://s0.wp.com/latex.php?latex=%5Cmathbf%7B%5Comega%7D%3D%5Cmathbf%7B1%7D&bg=ffffff&fg=000&s=0" class="latex">,</p>
<div><table><tr><td><pre>1
2
3
</pre></td><td><pre>weights_0 <span>=</span> <span>rep</span><span>(</span><span>1</span>,<span>8</span><span>)</span>
X <span>=</span> <span>as.<span>matrix</span></span><span>(</span><span>cbind</span><span>(</span><span>1</span>,myocarde<span>[</span>,<span>1</span><span>:</span><span>7</span><span>]</span><span>)</span><span>)</span>
y_5_1 <span>=</span> sigmoid<span>(</span>X <span>%*%</span> weights_0<span>)</span></pre></td></tr></table></div>
<p>that’s kind of stupid because all our predictions are 1, here. Let us try something else. Like <img alt="\mathbf{\omega}=\widehat{\mathbf{\beta}}^{ols}" title="\mathbf{\omega}=\widehat{\mathbf{\beta}}^{ols}" src="https://s0.wp.com/latex.php?latex=%5Cmathbf%7B%5Comega%7D%3D%5Cwidehat%7B%5Cmathbf%7B%5Cbeta%7D%7D%5E%7Bols%7D&bg=ffffff&fg=000&s=0" class="latex">. It is optimized, somehow, but we needed something to visualize what’s going on</p>
<div><table><tr><td><pre>1
</pre></td><td><pre>weights_0 <span>=</span> <span>lm</span><span>(</span>PRONO~.,<span>data</span><span>=</span>myocarde<span>)</span>$coefficients</pre></td></tr></table></div>
<p>then use</p>
<div><table><tr><td><pre>1
</pre></td><td><pre>y_5_1 <span>=</span> sigmoid<span>(</span>X <span>%*%</span> weights_0<span>)</span></pre></td></tr></table></div>
<p>In order to see if we get a “good” prediction, let use plot the <a rel="nofollow" href="https://en.wikipedia.org/wiki/Receiver_operating_characteristic" target="_blank">ROC curve</a>, and compare it with the one we got with a (simple) logistic regression</p>
<div><table><tr><td><pre>1
2
3
4
5
6
7
8
9
</pre></td><td><pre><span>library</span><span>(</span>ROCR<span>)</span>
pred <span>=</span> ROCR<span>::</span><span>prediction</span><span>(</span>y_5_1,myocarde$PRONO<span>)</span>
perf <span>=</span> ROCR<span>::</span><span>performance</span><span>(</span>pred,<span>&quot;tpr&quot;</span>, <span>&quot;fpr&quot;</span><span>)</span>
<span>plot</span><span>(</span>perf,<span>col</span><span>=</span><span>&quot;blue&quot;</span>,lwd<span>=</span><span>2</span><span>)</span>
reg <span>=</span> <span>glm</span><span>(</span>PRONO~.,<span>data</span><span>=</span>myocarde,<span>family</span><span>=</span><span>binomial</span><span>(</span>link <span>=</span> <span>&quot;logit&quot;</span><span>)</span><span>)</span>
y_0 <span>=</span> <span>predict</span><span>(</span>reg,type<span>=</span><span>&quot;response&quot;</span><span>)</span>
pred0 <span>=</span> ROCR<span>::</span><span>prediction</span><span>(</span>y_0,myocarde$PRONO<span>)</span>
perf0 <span>=</span> ROCR<span>::</span><span>performance</span><span>(</span>pred0,<span>&quot;tpr&quot;</span>, <span>&quot;fpr&quot;</span><span>)</span>
<span>plot</span><span>(</span>perf0,add<span>=</span>TRUE,<span>col</span><span>=</span><span>&quot;red&quot;</span><span>)</span></pre></td></tr></table></div>
<p><img alt="" src="https://i0.wp.com/freakonometrics.hypotheses.org/files/2018/06/nn-1.png?w=456&ssl=1" data-recalc-dims="1"><br>
That’s not bad for a very first attempt. Except that we’ve been cheating here, since we did use <img alt="\mathbf{\omega}=\widehat{\mathbf{\beta}}^{ols}" title="\mathbf{\omega}=\widehat{\mathbf{\beta}}^{ols}" src="https://s0.wp.com/latex.php?latex=%5Cmathbf%7B%5Comega%7D%3D%5Cwidehat%7B%5Cmathbf%7B%5Cbeta%7D%7D%5E%7Bols%7D&bg=ffffff&fg=000&s=0" class="latex">. How, for real, should we choose those weights?</p>
<h2>Using a loss function</h2>
<p>Well, if we want an “optimal” set of weights, we need to “optimize” an objective function. So we need to quantify the loss of a mistake, between the prediction, and the observation. Consider here a quadratic loss function</p>
<div><table><tr><td><pre>1
2
</pre></td><td><pre>loss <span>=</span> <span>function</span><span>(</span><span>weights</span><span>)</span><span>{</span>
  <span>mean</span><span>(</span> <span>(</span>myocarde$PRONO<span>-</span>sigmoid<span>(</span>X <span>%*%</span> <span>weights</span><span>)</span><span>)</span><span>^</span><span>2</span><span>)</span> <span>}</span></pre></td></tr></table></div>
<p>It might be stupid to use a quadratic loss function for a classification, but here, it’s not the point. We just want to understand what is the algorithm we use, and the loss function <img alt="\ell" title="\ell" src="https://s0.wp.com/latex.php?latex=%5Cell&bg=ffffff&fg=000&s=0" class="latex"> is just one parameter. Then we want to solve<img alt="\mathbf{\omega}^\star=\text{argmin}\left\lbrace\frac{1}{n}\sum_{i=1}^n\ell\left(y_i,f(\omega_0+\mathbf{x}_i^T\mathbf{\omega})\right)\right\rbrace" title="\mathbf{\omega}^\star=\text{argmin}\left\lbrace\frac{1}{n}\sum_{i=1}^n\ell\left(y_i,f(\omega_0+\mathbf{x}_i^T\mathbf{\omega})\right)\right\rbrace" src="https://s0.wp.com/latex.php?latex=%5Cmathbf%7B%5Comega%7D%5E%5Cstar%3D%5Ctext%7Bargmin%7D%5Cleft%5Clbrace%5Cfrac%7B1%7D%7Bn%7D%5Csum_%7Bi%3D1%7D%5En%5Cell%5Cleft%28y_i%2Cf%28%5Comega_0%2B%5Cmathbf%7Bx%7D_i%5ET%5Cmathbf%7B%5Comega%7D%29%5Cright%29%5Cright%5Crbrace&bg=ffffff&fg=000&s=0" class="latex">Thus, consider</p>
<div><table><tr><td><pre>1
</pre></td><td><pre>weights_1 <span>=</span> <span>optim</span><span>(</span>weights_0,loss<span>)</span>$par</pre></td></tr></table></div>
<p>(where the starting point is the OLS estimate). Again, to see what’s going on, let us visualize the ROC curve</p>
<div><table><tr><td><pre>1
2
3
4
5
</pre></td><td><pre>y_5_2 <span>=</span> sigmoid<span>(</span>X <span>%*%</span> weights_1<span>)</span>
pred <span>=</span> ROCR<span>::</span><span>prediction</span><span>(</span>y_5_2,myocarde$PRONO<span>)</span>
perf <span>=</span> ROCR<span>::</span><span>performance</span><span>(</span>pred,<span>&quot;tpr&quot;</span>, <span>&quot;fpr&quot;</span><span>)</span>
<span>plot</span><span>(</span>perf,<span>col</span><span>=</span><span>&quot;blue&quot;</span>,lwd<span>=</span><span>2</span><span>)</span>
<span>plot</span><span>(</span>perf0,add<span>=</span>TRUE,<span>col</span><span>=</span><span>&quot;red&quot;</span><span>)</span></pre></td></tr></table></div>
<p><img alt="" src="https://i2.wp.com/freakonometrics.hypotheses.org/files/2018/06/nn-2.png?w=456&ssl=1" data-recalc-dims="1"><br>
That’s not amazing, but again, that’s only a first step.</p>
<h2>A single layer</h2>
<p>Let us add a single layer in our network.<br>
<img alt="" src="https://i1.wp.com/freakonometrics.hypotheses.org/files/2018/05/NN-ub2.png?w=456&ssl=1" data-recalc-dims="1"><br>
Those nodes are connected to the sources (incoming from sources) from the left, and then connected to the sink, on the right. Those nodes are not inter-connected. And again, for that network, we need edges (i.e series of weights). For instance, on the network above, we did add one single layer, with (only) three nodes.</p>
<p>For such a network, the prediction formula is <img alt="\mathbf{y}=f\left( \omega_0+ \sum_{h=1}^3\omega_h f_h\left(\omega_{h,0}+ \sum_{j=1}^p \omega_{h,j} x_j\right)\right)" title="\mathbf{y}=f\left( \omega_0+ \sum_{h=1}^3\omega_h f_h\left(\omega_{h,0}+ \sum_{j=1}^p \omega_{h,j} x_j\right)\right)" src="https://s0.wp.com/latex.php?latex=%5Cmathbf%7By%7D%3Df%5Cleft%28+%5Comega_0%2B+%5Csum_%7Bh%3D1%7D%5E3%5Comega_h+f_h%5Cleft%28%5Comega_%7Bh%2C0%7D%2B+%5Csum_%7Bj%3D1%7D%5Ep+%5Comega_%7Bh%2Cj%7D+x_j%5Cright%29%5Cright%29&bg=ffffff&fg=000&s=0" class="latex">or more synthetically<img alt="\mathbf{y}=f\left( \omega_0+ \sum_{h=1}^3 \omega_hf_h\left(\omega_{h,0}+ \mathbf{x}^T\mathbf{\omega}_h\right)\right)" title="\mathbf{y}=f\left( \omega_0+ \sum_{h=1}^3 \omega_hf_h\left(\omega_{h,0}+ \mathbf{x}^T\mathbf{\omega}_h\right)\right)" src="https://s0.wp.com/latex.php?latex=%5Cmathbf%7By%7D%3Df%5Cleft%28+%5Comega_0%2B+%5Csum_%7Bh%3D1%7D%5E3+%5Comega_hf_h%5Cleft%28%5Comega_%7Bh%2C0%7D%2B+%5Cmathbf%7Bx%7D%5ET%5Cmathbf%7B%5Comega%7D_h%5Cright%29%5Cright%29&bg=ffffff&fg=000&s=0" class="latex">Usually, we consider the same activation function everywhere. Don’t ask me why, I find that weird.</p>
<p>Now, we have a lot of weights to choose. Let us use again OLS estimates</p>
<div><table><tr><td><pre>1
2
3
4
5
6
</pre></td><td><pre>weights_1 <span>&amp;</span>lt<span>;-</span> <span>lm</span><span>(</span>PRONO~<span>1</span><span>+</span>FRCAR<span>+</span>INCAR<span>+</span>INSYS<span>+</span>PAPUL<span>+</span>PVENT,<span>data</span><span>=</span>myocarde<span>)</span>$coefficients
X1 <span>=</span> <span>as.<span>matrix</span></span><span>(</span><span>cbind</span><span>(</span><span>1</span>,myocarde<span>[</span>,<span>c</span><span>(</span><span>&quot;FRCAR&quot;</span>,<span>&quot;INCAR&quot;</span>,<span>&quot;INSYS&quot;</span>,<span>&quot;PAPUL&quot;</span>,<span>&quot;PVENT&quot;</span><span>)</span><span>]</span><span>)</span><span>)</span>
weights_2 <span>&amp;</span>lt<span>;-</span> <span>lm</span><span>(</span>PRONO~<span>1</span><span>+</span>INSYS<span>+</span>PRDIA,<span>data</span><span>=</span>myocarde<span>)</span>$coefficients
X2<span>=</span><span>as.<span>matrix</span></span><span>(</span><span>cbind</span><span>(</span><span>1</span>,myocarde<span>[</span>,<span>c</span><span>(</span><span>&quot;INSYS&quot;</span>,<span>&quot;PRDIA&quot;</span><span>)</span><span>]</span><span>)</span><span>)</span>
weights_3 <span>&amp;</span>lt<span>;-</span> <span>lm</span><span>(</span>PRONO~<span>1</span><span>+</span>PAPUL<span>+</span>PVENT<span>+</span>REPUL,<span>data</span><span>=</span>myocarde<span>)</span>$coefficients
X3<span>=</span><span>as.<span>matrix</span></span><span>(</span><span>cbind</span><span>(</span><span>1</span>,myocarde<span>[</span>,<span>c</span><span>(</span><span>&quot;PAPUL&quot;</span>,<span>&quot;PVENT&quot;</span>,<span>&quot;REPUL&quot;</span><span>)</span><span>]</span><span>)</span><span>)</span></pre></td></tr></table></div>
<p>In that case, we did specify edges, and which sources (explanatory variables) should be used for each additional node. Actually, here, other techniques could be have been used, like using a <a rel="nofollow" href="https://en.wikipedia.org/wiki/Principal_component_analysis" target="_blank">PCA</a>. Each node will then be one of the components. But we’ll use that idea later one…</p>
<div><table><tr><td><pre>1
</pre></td><td><pre>X <span>=</span> <span>cbind</span><span>(</span>sigmoid<span>(</span>X1 <span>%*%</span> weights_1<span>)</span>, sigmoid<span>(</span>X2 <span>%*%</span> weights_2<span>)</span>, sigmoid<span>(</span>X3 <span>%*%</span> weights_3<span>)</span><span>)</span></pre></td></tr></table></div>
<p>But we’re not done here. Those were weights from the source to the know nodes, in the layer. We still need the weights from the nodes to the sink. Here, let use use a simple average</p>
<div><table><tr><td><pre>1
2
</pre></td><td><pre><span>weights</span> <span>=</span> <span>c</span><span>(</span><span>1</span><span>/</span><span>3</span>,<span>1</span><span>/</span><span>3</span>,<span>1</span><span>/</span><span>3</span><span>)</span>
y_5_3 <span>&amp;</span>lt<span>;-</span> sigmoid<span>(</span>X <span>%*%</span> <span>weights</span><span>)</span></pre></td></tr></table></div>
<p>Again, we can plot the ROC curve to see what we’ve done…</p>
<div><table><tr><td><pre>1
2
3
4
</pre></td><td><pre>pred <span>=</span> ROCR<span>::</span><span>prediction</span><span>(</span>y_5_3,myocarde$PRONO<span>)</span>
perf <span>=</span> ROCR<span>::</span><span>performance</span><span>(</span>pred,<span>&quot;tpr&quot;</span>, <span>&quot;fpr&quot;</span><span>)</span>
<span>plot</span><span>(</span>perf,<span>col</span><span>=</span><span>&quot;blue&quot;</span>,lwd<span>=</span><span>2</span><span>)</span>
<span>plot</span><span>(</span>perf0,add<span>=</span>TRUE,<span>col</span><span>=</span><span>&quot;red&quot;</span><span>)</span></pre></td></tr></table></div>
<p><img alt="" src="https://i1.wp.com/freakonometrics.hypotheses.org/files/2018/06/nn-3.png?w=456&ssl=1" data-recalc-dims="1"></p>
<h2>On back propagation</h2>
<p>Now, we need some optimal selection of those weights. Observe that with only 3 nodes, there are already <img alt="(7+1)\times3+3=27" title="(7+1)\times3+3=27" src="https://s0.wp.com/latex.php?latex=%287%2B1%29%5Ctimes3%2B3%3D27&bg=ffffff&fg=000&s=0" class="latex"> parameters in that model! Clearly, parcimony is not the major issue when you start using neural nets! If <img alt="p(\mathbf{x})=f\left( \omega_0+ \sum_{h=1}^3 \omega_hf_h\left(\omega_{h,0}+ \mathbf{x}^T\mathbf{\omega}_h\right)\right)" title="p(\mathbf{x})=f\left( \omega_0+ \sum_{h=1}^3 \omega_hf_h\left(\omega_{h,0}+ \mathbf{x}^T\mathbf{\omega}_h\right)\right)" src="https://s0.wp.com/latex.php?latex=p%28%5Cmathbf%7Bx%7D%29%3Df%5Cleft%28+%5Comega_0%2B+%5Csum_%7Bh%3D1%7D%5E3+%5Comega_hf_h%5Cleft%28%5Comega_%7Bh%2C0%7D%2B+%5Cmathbf%7Bx%7D%5ET%5Cmathbf%7B%5Comega%7D_h%5Cright%29%5Cright%29&bg=ffffff&fg=000&s=0" class="latex">we want to solve<img alt="\mathbf{\omega}^\star=\text{argmin}\left\lbrace\frac{1}{n}\sum_{i=1}^n\ell\left(y_i,p(\mathbf{x}_i)\right)\right\rbrace" title="\mathbf{\omega}^\star=\text{argmin}\left\lbrace\frac{1}{n}\sum_{i=1}^n\ell\left(y_i,p(\mathbf{x}_i)\right)\right\rbrace" src="https://s0.wp.com/latex.php?latex=%5Cmathbf%7B%5Comega%7D%5E%5Cstar%3D%5Ctext%7Bargmin%7D%5Cleft%5Clbrace%5Cfrac%7B1%7D%7Bn%7D%5Csum_%7Bi%3D1%7D%5En%5Cell%5Cleft%28y_i%2Cp%28%5Cmathbf%7Bx%7D_i%29%5Cright%29%5Cright%5Crbrace&bg=ffffff&fg=000&s=0" class="latex">for some loss function, which is<img alt="\mathbf{\omega}^\star=\text{argmin}\left\lbrace\frac{1}{n}\sum_{i=1}^n (y_i-p(\mathbf{x}_i))^2 \right\rbrace" title="\mathbf{\omega}^\star=\text{argmin}\left\lbrace\frac{1}{n}\sum_{i=1}^n (y_i-p(\mathbf{x}_i))^2 \right\rbrace" src="https://s0.wp.com/latex.php?latex=%5Cmathbf%7B%5Comega%7D%5E%5Cstar%3D%5Ctext%7Bargmin%7D%5Cleft%5Clbrace%5Cfrac%7B1%7D%7Bn%7D%5Csum_%7Bi%3D1%7D%5En+%28y_i-p%28%5Cmathbf%7Bx%7D_i%29%29%5E2+%5Cright%5Crbrace&bg=ffffff&fg=000&s=0" class="latex">for the quadratic norm, or<img alt="\mathbf{\omega}^\star=\text{argmin}\left\lbrace\frac{1}{n}\sum_{i=1}^n (y_i\log p(\mathbf{x}_i)+[1-y_i]\log [1-p(\mathbf{x}_i)]) \right\rbrace" title="\mathbf{\omega}^\star=\text{argmin}\left\lbrace\frac{1}{n}\sum_{i=1}^n (y_i\log p(\mathbf{x}_i)+[1-y_i]\log [1-p(\mathbf{x}_i)]) \right\rbrace" src="https://s0.wp.com/latex.php?latex=%5Cmathbf%7B%5Comega%7D%5E%5Cstar%3D%5Ctext%7Bargmin%7D%5Cleft%5Clbrace%5Cfrac%7B1%7D%7Bn%7D%5Csum_%7Bi%3D1%7D%5En+%28y_i%5Clog+p%28%5Cmathbf%7Bx%7D_i%29%2B%5B1-y_i%5D%5Clog+%5B1-p%28%5Cmathbf%7Bx%7D_i%29%5D%29+%5Cright%5Crbrace&bg=ffffff&fg=000&s=0" class="latex">if we want to use <a rel="nofollow" href="https://en.wikipedia.org/wiki/Cross_entropy" target="_blank">cross-entropy</a>.</p>
<p>For convenience, let us center all the variable we create, otherwise, we get numerical problems.</p>
<div><table><tr><td><pre>1
2
3
4
5
6
7
8
9
10
11
12
</pre></td><td><pre>center <span>=</span> <span>function</span><span>(</span>z<span>)</span> <span>(</span>z<span>-</span><span>mean</span><span>(</span>z<span>)</span><span>)</span><span>/</span><span>sd</span><span>(</span>z<span>)</span>
loss <span>=</span> <span>function</span><span>(</span><span>weights</span><span>)</span><span>{</span>
weights_1 <span>=</span> <span>weights</span><span>[</span><span>0</span><span>+</span><span>(</span><span>1</span><span>:</span><span>7</span><span>)</span><span>]</span>
weights_2 <span>=</span> <span>weights</span><span>[</span><span>7</span><span>+</span><span>(</span><span>1</span><span>:</span><span>7</span><span>)</span><span>]</span>
weights_3 <span>=</span> <span>weights</span><span>[</span><span>14</span><span>+</span><span>(</span><span>1</span><span>:</span><span>7</span><span>)</span><span>]</span>
weights_  <span>=</span> <span>weights</span><span>[</span><span>21</span><span>+</span><span>1</span><span>:</span><span>4</span><span>]</span>
X1<span>=</span>X2<span>=</span>X3<span>=</span><span>as.<span>matrix</span></span><span>(</span>myocarde<span>[</span>,<span>1</span><span>:</span><span>7</span><span>]</span><span>)</span>
Z1 <span>=</span> center<span>(</span>X1 <span>%*%</span> weights_1<span>)</span>
Z2 <span>=</span> center<span>(</span>X2 <span>%*%</span> weights_2<span>)</span>
Z3 <span>=</span> center<span>(</span>X3 <span>%*%</span> weights_3<span>)</span>
X <span>=</span> <span>cbind</span><span>(</span><span>1</span>,sigmoid<span>(</span>Z1<span>)</span>, sigmoid<span>(</span>Z2<span>)</span>, sigmoid<span>(</span>Z3<span>)</span><span>)</span>
<span>mean</span><span>(</span> <span>(</span>myocarde$PRONO<span>-</span>sigmoid<span>(</span>X <span>%*%</span> weights_<span>)</span><span>)</span><span>^</span><span>2</span><span>)</span><span>}</span></pre></td></tr></table></div>
<p>Now that we have our objective function, consider some starting points. We can consider weights from a PCA, and then use a gradient descent algorithm,</p>
<div><table><tr><td><pre>1
2
3
4
</pre></td><td><pre>pca <span>=</span> <span>princomp</span><span>(</span>myocarde<span>[</span>,<span>1</span><span>:</span><span>7</span><span>]</span><span>)</span>
W <span>=</span> get_pca_var<span>(</span>pca<span>)</span>$contrib
weights_0 <span>=</span> <span>c</span><span>(</span>W<span>[</span>,<span>1</span><span>]</span>,W<span>[</span>,<span>2</span><span>]</span>,W<span>[</span>,<span>3</span><span>]</span>,<span>c</span><span>(</span><span>-</span><span>1</span>,<span>rep</span><span>(</span><span>1</span>,<span>3</span><span>)</span><span>/</span><span>3</span><span>)</span><span>)</span>
weights_opt <span>=</span> <span>optim</span><span>(</span>weights_0,loss<span>)</span>$par</pre></td></tr></table></div>
<p>The prediction is then obtained using</p>
<div><table><tr><td><pre>1
2
3
4
5
6
7
8
9
10
</pre></td><td><pre>weights_1 <span>=</span> weights_opt<span>[</span><span>0</span><span>+</span><span>(</span><span>1</span><span>:</span><span>7</span><span>)</span><span>]</span>
weights_2 <span>=</span> weights_opt<span>[</span><span>7</span><span>+</span><span>(</span><span>1</span><span>:</span><span>7</span><span>)</span><span>]</span>
weights_3 <span>=</span> weights_opt<span>[</span><span>14</span><span>+</span><span>(</span><span>1</span><span>:</span><span>7</span><span>)</span><span>]</span>
weights_  <span>=</span> weights_opt<span>[</span><span>21</span><span>+</span><span>1</span><span>:</span><span>4</span><span>]</span>
X1<span>=</span>X2<span>=</span>X3<span>=</span><span>as.<span>matrix</span></span><span>(</span>myocarde<span>[</span>,<span>1</span><span>:</span><span>7</span><span>]</span><span>)</span>
Z1 <span>=</span> center<span>(</span>X1 <span>%*%</span> weights_1<span>)</span>
Z2 <span>=</span> center<span>(</span>X2 <span>%*%</span> weights_2<span>)</span>
Z3 <span>=</span> center<span>(</span>X3 <span>%*%</span> weights_3<span>)</span>
X <span>=</span> <span>cbind</span><span>(</span><span>1</span>,sigmoid<span>(</span>Z1<span>)</span>, sigmoid<span>(</span>Z2<span>)</span>, sigmoid<span>(</span>Z3<span>)</span><span>)</span>
y_5_4 <span>=</span> sigmoid<span>(</span>X <span>%*%</span> weights_<span>)</span></pre></td></tr></table></div>
<p>And as previously, why not plot the ROC curve of that model</p>
<div><table><tr><td><pre>1
2
3
4
</pre></td><td><pre>pred <span>=</span> ROCR<span>::</span><span>prediction</span><span>(</span>y_5_4,myocarde$PRONO<span>)</span>
perf <span>=</span> ROCR<span>::</span><span>performance</span><span>(</span>pred,<span>&quot;tpr&quot;</span>, <span>&quot;fpr&quot;</span><span>)</span>
<span>plot</span><span>(</span>perf,<span>col</span><span>=</span><span>&quot;blue&quot;</span>,lwd<span>=</span><span>2</span><span>)</span>
<span>plot</span><span>(</span>perf,add<span>=</span>TRUE,<span>col</span><span>=</span><span>&quot;red&quot;</span><span>)</span></pre></td></tr></table></div>
<p><img alt="" src="https://i1.wp.com/freakonometrics.hypotheses.org/files/2018/06/nn-4.png?w=456&ssl=1" data-recalc-dims="1"><br>
That’s not too bad. But with 27 coefficients, that’s what we would expect, no?</p>
<h2>Using nnet() function</h2>
<p>That’s more or less what is done in neural nets functions. Let us now have a look at some dedicated R functions.</p>
<div><table><tr><td><pre>1
2
3
4
</pre></td><td><pre><span>library</span><span>(</span>nnet<span>)</span>
myocarde_minmax <span>=</span> myocarde
minmax <span>=</span> <span>function</span><span>(</span>z<span>)</span> <span>(</span>z<span>-</span><span>min</span><span>(</span>z<span>)</span><span>)</span><span>/</span><span>(</span><span>max</span><span>(</span>z<span>)</span><span>-</span><span>min</span><span>(</span>z<span>)</span><span>)</span>
<span>for</span><span>(</span>j <span>in</span> <span>1</span><span>:</span><span>7</span><span>)</span> myocarde_minmax<span>[</span>,j<span>]</span> <span>=</span> minmax<span>(</span>myocarde_minmax<span>[</span>,j<span>]</span><span>)</span></pre></td></tr></table></div>
<p>Here, variables are linearly transformed, to take values in <img alt="(0,1)" title="(0,1)" src="https://s0.wp.com/latex.php?latex=%280%2C1%29&bg=ffffff&fg=000&s=0" class="latex">. Then we can construct a neural network with one single layer, and three nodes,</p>
<div><table><tr><td><pre>1
2
3
4
5
6
7
8
9
10
11
12
</pre></td><td><pre>model_nnet <span>=</span> nnet<span>(</span>PRONO~.,<span>data</span><span>=</span>myocarde_minmax,size<span>=</span><span>3</span><span>)</span>
<span>summary</span><span>(</span>model_nnet<span>)</span>
a <span>7</span><span>-</span><span>3</span><span>-</span><span>1</span> network <span>with</span> <span>28</span> <span>weights</span>
<span>options</span> were <span>-</span>
 b<span>-&amp;</span>gt<span>;</span>h1 i1<span>-&amp;</span>gt<span>;</span>h1 i2<span>-&amp;</span>gt<span>;</span>h1 i3<span>-&amp;</span>gt<span>;</span>h1 i4<span>-&amp;</span>gt<span>;</span>h1 i5<span>-&amp;</span>gt<span>;</span>h1 i6<span>-&amp;</span>gt<span>;</span>h1 i7<span>-&amp;</span>gt<span>;</span>h1 
 <span>-</span><span>9.60</span>  <span>-</span><span>1.79</span>  <span>21.00</span>  <span>14.72</span> <span>-</span><span>20.45</span>  <span>-</span><span>5.05</span>  <span>14.37</span> <span>-</span><span>17.37</span> 
 b<span>-&amp;</span>gt<span>;</span>h2 i1<span>-&amp;</span>gt<span>;</span>h2 i2<span>-&amp;</span>gt<span>;</span>h2 i3<span>-&amp;</span>gt<span>;</span>h2 i4<span>-&amp;</span>gt<span>;</span>h2 i5<span>-&amp;</span>gt<span>;</span>h2 i6<span>-&amp;</span>gt<span>;</span>h2 i7<span>-&amp;</span>gt<span>;</span>h2 
  <span>4.72</span>   <span>2.83</span>  <span>-</span><span>3.37</span>  <span>-</span><span>1.64</span>   <span>1.49</span>   <span>2.12</span>   <span>2.31</span>   <span>4.00</span> 
 b<span>-&amp;</span>gt<span>;</span>h3 i1<span>-&amp;</span>gt<span>;</span>h3 i2<span>-&amp;</span>gt<span>;</span>h3 i3<span>-&amp;</span>gt<span>;</span>h3 i4<span>-&amp;</span>gt<span>;</span>h3 i5<span>-&amp;</span>gt<span>;</span>h3 i6<span>-&amp;</span>gt<span>;</span>h3 i7<span>-&amp;</span>gt<span>;</span>h3 
 <span>-</span><span>0.58</span>  <span>-</span><span>6.03</span>  <span>25.14</span>  <span>18.03</span>  <span>-</span><span>1.19</span>   <span>7.52</span> <span>-</span><span>19.47</span> <span>-</span><span>12.95</span> 
  b<span>-&amp;</span>gt<span>;</span>o  h1<span>-&amp;</span>gt<span>;</span>o  h2<span>-&amp;</span>gt<span>;</span>o  h3<span>-&amp;</span>gt<span>;</span>o 
 <span>-</span><span>1.32</span>  <span>29.00</span> <span>-</span><span>10.32</span>  <span>26.27</span></pre></td></tr></table></div>
<p>Here, it is the complete full network. And actually, there are (online) some functions that can he used to visualize that network</p>
<div><table><tr><td><pre>1
2
3
</pre></td><td><pre><span>library</span><span>(</span>devtools<span>)</span>
source_url<span>(</span><span>'https://gist.githubusercontent.com/fawda123/7471137/raw/466c1474d0a505ff044412703516c34f1a4684a5/nnet_plot_update.r'</span><span>)</span>
plot.<span>nnet</span><span>(</span>model_nnet<span>)</span></pre></td></tr></table></div>
<p><img alt="" src="https://i2.wp.com/freakonometrics.hypotheses.org/files/2018/06/nn-5.png?w=456&ssl=1" data-recalc-dims="1"><br>
Nice, isn’t it? We clearly see the intermediary layer, with three nodes, and on top the constants. Edges are the plain lines, the darker, the heavier (in terms of weights).</p>
<h2>Using neuralnet()</h2>
<p>Other R functions can actually be considered.</p>
<div><table><tr><td><pre>1
2
3
4
</pre></td><td><pre><span>library</span><span>(</span>neuralnet<span>)</span>
model_nnet <span>=</span> neuralnet<span>(</span><span>formula</span><span>(</span><span>glm</span><span>(</span>PRONO~.,<span>data</span><span>=</span>myocarde_minmax<span>)</span><span>)</span>,
myocarde_minmax,hidden<span>=</span><span>3</span>, act.<span>fct</span> <span>=</span> sigmoid<span>)</span>
<span>plot</span><span>(</span>model_nnet<span>)</span></pre></td></tr></table></div>
<p><img alt="" src="https://i2.wp.com/freakonometrics.hypotheses.org/files/2018/06/nn-7.png?w=456&ssl=1" data-recalc-dims="1"><br>
Again, for the same network structure, with one (hidden) layer, and three nodes in it.</p>
<h2>Network with multiple layers</h2>
<p>The good thing is that it’s not possible to add more layers. Like two layers. Nodes from the first layer are no longuer connected with the sink, but with nodes in the second layer. And those nodes will then be connected to the sink. We now have something like<br>
<img alt="p(\mathbf{x})=f\left( \omega_0+ \sum_{h=1}^3 \omega_h f_h\left(\omega_{h,0}+ \mathbf{z}_h^T\mathbf{\omega}_h\right)\right)" title="p(\mathbf{x})=f\left( \omega_0+ \sum_{h=1}^3 \omega_h f_h\left(\omega_{h,0}+ \mathbf{z}_h^T\mathbf{\omega}_h\right)\right)" src="https://s0.wp.com/latex.php?latex=p%28%5Cmathbf%7Bx%7D%29%3Df%5Cleft%28+%5Comega_0%2B+%5Csum_%7Bh%3D1%7D%5E3+%5Comega_h+f_h%5Cleft%28%5Comega_%7Bh%2C0%7D%2B+%5Cmathbf%7Bz%7D_h%5ET%5Cmathbf%7B%5Comega%7D_h%5Cright%29%5Cright%29&bg=ffffff&fg=000&s=0" class="latex">where<img alt="\mathbf{z}_h=f\left( \omega_{h,0}+ \sum_{j=1}^{k_h} \omega_{h,j} f_{h,j}\left(\omega_{h,j,0}+ \mathbf{x}^T\mathbf{\omega}_{h,j}\right)\right)" title="\mathbf{z}_h=f\left( \omega_{h,0}+ \sum_{j=1}^{k_h} \omega_{h,j} f_{h,j}\left(\omega_{h,j,0}+ \mathbf{x}^T\mathbf{\omega}_{h,j}\right)\right)" src="https://s0.wp.com/latex.php?latex=%5Cmathbf%7Bz%7D_h%3Df%5Cleft%28+%5Comega_%7Bh%2C0%7D%2B+%5Csum_%7Bj%3D1%7D%5E%7Bk_h%7D+%5Comega_%7Bh%2Cj%7D+f_%7Bh%2Cj%7D%5Cleft%28%5Comega_%7Bh%2Cj%2C0%7D%2B+%5Cmathbf%7Bx%7D%5ET%5Cmathbf%7B%5Comega%7D_%7Bh%2Cj%7D%5Cright%29%5Cright%29&bg=ffffff&fg=000&s=0" class="latex">I may be rambling here (a little bit) but that’s a lot of parameters. Here is the visualization of such a network,</p>
<div><table><tr><td><pre>1
2
3
4
</pre></td><td><pre><span>library</span><span>(</span>neuralnet<span>)</span>
model_nnet <span>=</span> neuralnet<span>(</span><span>formula</span><span>(</span><span>glm</span><span>(</span>PRONO~.,<span>data</span><span>=</span>myocarde_minmax<span>)</span><span>)</span>,
myocarde_minmax,hidden<span>=</span><span>3</span>, act.<span>fct</span> <span>=</span> sigmoid<span>)</span>
<span>plot</span><span>(</span>model_nnet<span>)</span></pre></td></tr></table></div>
<p><img alt="" src="https://i2.wp.com/f-origin.hypotheses.org/wp-content/blogs.dir/253/files/2018/06/nn-8.png?w=456&ssl=1" data-recalc-dims="1"></p>
<h2>Application</h2>
<p>Let us get back on our simple dataset, with only two covariates.</p>
<div><table><tr><td><pre>1
2
3
4
5
6
</pre></td><td><pre><span>library</span><span>(</span>neuralnet<span>)</span>
df_minmax <span>=</span><span>df</span>
df_minmax$y<span>=</span><span>(</span>df_minmax$y<span>==</span><span>&quot;1&quot;</span><span>)</span><span>*</span><span>1</span>
minmax <span>=</span> <span>function</span><span>(</span>z<span>)</span> <span>(</span>z<span>-</span><span>min</span><span>(</span>z<span>)</span><span>)</span><span>/</span><span>(</span><span>max</span><span>(</span>z<span>)</span><span>-</span><span>min</span><span>(</span>z<span>)</span><span>)</span>
<span>for</span><span>(</span>j <span>in</span> <span>1</span><span>:</span><span>2</span><span>)</span> df_minmax<span>[</span>,j<span>]</span> <span>=</span> minmax<span>(</span><span>df</span><span>[</span>,j<span>]</span><span>)</span>
X <span>=</span> <span>as.<span>matrix</span></span><span>(</span><span>cbind</span><span>(</span><span>1</span>,df_minmax<span>[</span>,<span>1</span><span>:</span><span>2</span><span>]</span><span>)</span><span>)</span></pre></td></tr></table></div>
<p>Consider only one layer, with two nodes</p>
<div><table><tr><td><pre>1
2
3
</pre></td><td><pre>model_nnet <span>=</span> neuralnet<span>(</span><span>formula</span><span>(</span><span>lm</span><span>(</span>y~.,<span>data</span><span>=</span>df_minmax<span>)</span><span>)</span>,
df_minmax,hidden<span>=</span><span>c</span><span>(</span><span>2</span><span>)</span><span>)</span>
<span>plot</span><span>(</span>model_nnet<span>)</span></pre></td></tr></table></div>
<p><img src="https://i2.wp.com/freakonometrics.hypotheses.org/files/2018/06/Capture-d%E2%80%99e%CC%81cran-2018-06-05-a%CC%80-10.46.20.png?resize=240%2C264&ssl=1" alt="" width="240" data-recalc-dims="1" height="264"><br>
Here, we did not specify it, but the activation function is the sigmoid (actually, it is called logistic here)</p>
<div><table><tr><td><pre>1
2
3
4
5
6
7
8
9
</pre></td><td><pre>model_nnet$act.<span>fct</span>
<span>function</span> <span>(</span>x<span>)</span> 
<span>{</span>
    <span>1</span><span>/</span><span>(</span><span>1</span> <span>+</span> <span>exp</span><span>(</span><span>-</span>x<span>)</span><span>)</span>
<span>}</span>
 
<span>attr</span><span>(</span>,<span>&quot;type&quot;</span><span>)</span>
<span>[</span><span>1</span><span>]</span> <span>&quot;logistic&quot;</span>
f<span>=</span>model_nnet$act.<span>fct</span></pre></td></tr></table></div>
<p>The weights (on the figure) can be obtained using</p>
<div><table><tr><td><pre>1
2
3
</pre></td><td><pre>w0 <span>=</span> model_nnet$weights<span>[</span><span>[</span><span>1</span><span>]</span><span>]</span><span>[</span><span>[</span><span>2</span><span>]</span><span>]</span><span>[</span>,<span>1</span><span>]</span>
w1 <span>=</span> model_nnet$weights<span>[</span><span>[</span><span>1</span><span>]</span><span>]</span><span>[</span><span>[</span><span>1</span><span>]</span><span>]</span><span>[</span>,<span>1</span><span>]</span>
w2 <span>=</span> model_nnet$weights<span>[</span><span>[</span><span>1</span><span>]</span><span>]</span><span>[</span><span>[</span><span>1</span><span>]</span><span>]</span><span>[</span>,<span>2</span><span>]</span></pre></td></tr></table></div>
<p>Now, to get our prediction,<br>
we should use<img alt="p(\mathbf{x})=f\left( \omega_0+ \omega_1 f(\omega_{1,0}+ \mathbf{x}_h^T\mathbf{\omega}_{1,1:2})+\omega_1 f(\omega_{2,0}+ \mathbf{x}_h^T\mathbf{\omega}_{2,1:2})\right)" title="p(\mathbf{x})=f\left( \omega_0+ \omega_1 f(\omega_{1,0}+ \mathbf{x}_h^T\mathbf{\omega}_{1,1:2})+\omega_1 f(\omega_{2,0}+ \mathbf{x}_h^T\mathbf{\omega}_{2,1:2})\right)" src="https://s0.wp.com/latex.php?latex=p%28%5Cmathbf%7Bx%7D%29%3Df%5Cleft%28+%5Comega_0%2B+%5Comega_1+f%28%5Comega_%7B1%2C0%7D%2B+%5Cmathbf%7Bx%7D_h%5ET%5Cmathbf%7B%5Comega%7D_%7B1%2C1%3A2%7D%29%2B%5Comega_1+f%28%5Comega_%7B2%2C0%7D%2B+%5Cmathbf%7Bx%7D_h%5ET%5Cmathbf%7B%5Comega%7D_%7B2%2C1%3A2%7D%29%5Cright%29&bg=ffffff&fg=000&s=0" class="latex">which can be obtained using</p>
<div><table><tr><td><pre>1
2
3
4
5
6
7
8
9
10
11
12
</pre></td><td><pre>f<span>(</span><span>cbind</span><span>(</span><span>1</span>,f<span>(</span>X<span>%*%</span>w1<span>)</span>,f<span>(</span>X<span>%*%</span>w2<span>)</span><span>)</span><span>%*%</span>w0<span>)</span>
              <span>[</span>,<span>1</span><span>]</span>
 <span>[</span><span>1</span>,<span>]</span> <span>0.7336477343</span>
 <span>[</span><span>2</span>,<span>]</span> <span>0.7317999050</span>
 <span>[</span><span>3</span>,<span>]</span> <span>0.7185803540</span>
 <span>[</span><span>4</span>,<span>]</span> <span>0.7404005280</span>
 <span>[</span><span>5</span>,<span>]</span> <span>0.7518482779</span>
 <span>[</span><span>6</span>,<span>]</span> <span>0.4939774149</span>
 <span>[</span><span>7</span>,<span>]</span> <span>0.4965876378</span>
 <span>[</span><span>8</span>,<span>]</span> <span>0.7101714888</span>
 <span>[</span><span>9</span>,<span>]</span> <span>0.5050760026</span>
<span>[</span><span>10</span>,<span>]</span> <span>0.5049877644</span></pre></td></tr></table></div>
<p>Unfortunately, it is not the output of the model here,</p>
<div><table><tr><td><pre>1
2
3
4
5
6
7
8
9
10
11
12
13
14
</pre></td><td><pre>neuralnet<span>::</span><span>prediction</span><span>(</span>model_nnet<span>)</span>
Data Error<span>:</span>	<span>0</span><span>;</span>
$rep1
       x1           x2              y
<span>1</span>  <span>0.1250</span> <span>0.0000000000</span>  <span>0.02030470787</span>
<span>2</span>  <span>0.0625</span> <span>0.1176470588</span>  <span>0.89621706711</span>
<span>3</span>  <span>0.9375</span> <span>0.2352941176</span>  <span>0.01995171956</span>
<span>4</span>  <span>0.0000</span> <span>0.4705882353</span>  <span>1.10849420363</span>
<span>5</span>  <span>0.5000</span> <span>0.4705882353</span> <span>-</span><span>0.01364966058</span>
<span>6</span>  <span>0.3125</span> <span>0.5294117647</span> <span>-</span><span>0.02409150561</span>
<span>7</span>  <span>0.6875</span> <span>0.8235294118</span>  <span>0.93743057765</span>
<span>8</span>  <span>0.3750</span> <span>0.8823529412</span>  <span>1.01320924782</span>
<span>9</span>  <span>1.0000</span> <span>0.9058823529</span>  <span>1.04805134309</span>
<span>10</span> <span>0.5625</span> <span>1.0000000000</span>  <span>1.00377379767</span></pre></td></tr></table></div>
<p>If anyone has a clue, I’d be glad to know what went wrong here… I find that odd to have outputs outside the <img alt="(0,1)" title="(0,1)" src="https://s0.wp.com/latex.php?latex=%280%2C1%29&bg=ffffff&fg=000&s=0" class="latex"> interval, but the output is neither<img alt="p(\mathbf{x})=\omega_{0,0}+ \omega_{0,1} f(\omega_{1,0}+ \mathbf{x}_h^T\mathbf{\omega}_{1,1:2})+\omega_{0,2} f(\omega_{2,0}+ \mathbf{x}_h^T\mathbf{\omega}_{2,1:2})" title="p(\mathbf{x})=\omega_{0,0}+ \omega_{0,1} f(\omega_{1,0}+ \mathbf{x}_h^T\mathbf{\omega}_{1,1:2})+\omega_{0,2} f(\omega_{2,0}+ \mathbf{x}_h^T\mathbf{\omega}_{2,1:2})" src="https://s0.wp.com/latex.php?latex=p%28%5Cmathbf%7Bx%7D%29%3D%5Comega_%7B0%2C0%7D%2B+%5Comega_%7B0%2C1%7D+f%28%5Comega_%7B1%2C0%7D%2B+%5Cmathbf%7Bx%7D_h%5ET%5Cmathbf%7B%5Comega%7D_%7B1%2C1%3A2%7D%29%2B%5Comega_%7B0%2C2%7D+f%28%5Comega_%7B2%2C0%7D%2B+%5Cmathbf%7Bx%7D_h%5ET%5Cmathbf%7B%5Comega%7D_%7B2%2C1%3A2%7D%29&bg=ffffff&fg=000&s=0" class="latex"></p>
<div><table><tr><td><pre>1
2
3
4
5
6
7
8
9
10
11
12
</pre></td><td><pre><span>cbind</span><span>(</span><span>1</span>,f<span>(</span>X<span>%*%</span>w1<span>)</span>,f<span>(</span>X<span>%*%</span>w2<span>)</span><span>)</span><span>%*%</span>w0
                <span>[</span>,<span>1</span><span>]</span>
 <span>[</span><span>1</span>,<span>]</span>  <span>1.01320924782</span>
 <span>[</span><span>2</span>,<span>]</span>  <span>1.00377379767</span>
 <span>[</span><span>3</span>,<span>]</span>  <span>0.93743057765</span>
 <span>[</span><span>4</span>,<span>]</span>  <span>1.04805134309</span>
 <span>[</span><span>5</span>,<span>]</span>  <span>1.10849420363</span>
 <span>[</span><span>6</span>,<span>]</span> <span>-</span><span>0.02409150561</span>
 <span>[</span><span>7</span>,<span>]</span> <span>-</span><span>0.01364966058</span>
 <span>[</span><span>8</span>,<span>]</span>  <span>0.89621706711</span>
 <span>[</span><span>9</span>,<span>]</span>  <span>0.02030470787</span>
<span>[</span><span>10</span>,<span>]</span>  <span>0.01995171956</span></pre></td></tr></table></div>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://freakonometrics.hypotheses.org/52774"> R-english – Freakonometrics</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/l3hWjHtivng" height="1"></a></div>