<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/practical-machine-learning-with-r-and-python-part-2/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://gigadom.wordpress.com/2017/10/13/practical-machine-learning-with-r-and-python-part-2/"> R – Giga thoughts …</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>In this 2nd part of the series “Practical Machine Learning with R and Python – Part 2”, I continue where I left off in my first post <a rel="nofollow" href="https://gigadom.wordpress.com/2017/10/06/practical-machine-learning-with-r-and-python-part-1/" target="_blank">Practical Machine Learning with R and Python – Part 2</a>. In this post I cover the some classification algorithmns and cross validation. Specifically I touch<br>
-Logistic Regression<br>
-K Nearest Neighbors (KNN) classification<br>
-Leave out one Cross Validation (LOOCV)<br>
-K Fold Cross Validation<br>
in both R and Python.</p>
<p>As in my initial post the algorithms are based on the following courses.</p>
<ul>
<li><a rel="nofollow" href="https://lagunita.stanford.edu/courses/HumanitiesandScience/StatLearning/Winter2015/about" target="_blank">Statistical Learning</a>, Prof Trevor Hastie &amp; Prof Robert Tibesherani, Online Stanford</li>
<li><a rel="nofollow" href="https://www.coursera.org/learn/python-machine-learning" target="_blank">Applied Machine Learning in Python</a> Prof Kevyn-Collin Thomson, University Of Michigan, Coursera</li>
</ul>
<p>You can download this R Markdown file along with the data from <a rel="nofollow" href="https://github.com/tvganesh/MachineLearning-RandPython-Part2" target="_blank">Github</a>. I hope these posts can be used as a quick reference in R and Python and Machine Learning.I have tried to include the coolest part of either course in this post.</p>
<p>The following classification problem is based on Logistic Regression. The data is an included data set in Scikit-Learn, which I have saved as csv and use it also for R. The fit of a classification Machine Learning Model depends on how correctly classifies the data. There are several measures of testing a model’s classification performance. They are</p>
<p>–<strong>Accuracy </strong>= TP + TN / (TP + TN + FP + FN) – Fraction of all classes correctly classified<br>
–<strong>Precision </strong>= TP / (TP + FP) – Fraction of correctly classified positives among those classified as positive<br>
– <strong>Recall </strong>= TP / (TP + FN) Also known as sensitivity, or True Positive Rate (True positive) – Fraction of correctly classified as positive among all positives in the data<br>
– <strong>F1 </strong>= 2 * Precision * Recall / (Precision + Recall)</p>
<div>
<h2>1a. Logistic Regression – R code</h2>
<p>The caret and e1071 package is required for using the confusionMatrix call</p>
<pre><code><span>source</span><span>(</span><span>&quot;RFunctions.R&quot;</span><span>)
<span>library</span>(<span>dplyr</span>)
<span>library</span>(<span>caret</span>)
<span>library</span>(<span>e1071</span>)</span></code><code></code><code></code><code></code><code></code></pre>
<pre><code><span># Read the data (from sklearn)</span>
<span>cancer</span> <span>&lt;-</span> <span>read.csv</span><span>(</span><span>&quot;cancer.csv&quot;</span><span>)</span>
<span># Rename the target variable</span>
<span>names</span><span>(</span><span>cancer</span><span>)</span> <span>&lt;-</span> <span>c</span><span>(</span><span>seq</span><span>(</span><span>1</span>,<span>30</span><span>)</span>,<span>&quot;output&quot;</span><span>)</span>
<span># Split as training and test sets</span>
<span>train_idx</span> <span>&lt;-</span> <span>trainTestSplit</span><span>(</span><span>cancer</span>,<span>trainPercent</span><span>=</span><span>75</span>,<span>seed</span><span>=</span><span>5</span><span>)</span>
<span>train</span> <span>&lt;-</span> <span>cancer</span><span>[</span><span>train_idx</span>, <span>]</span>
<span>test</span> <span>&lt;-</span> <span>cancer</span><span>[</span><span>-</span><span>train_idx</span>, <span>]</span>

<span># Fit a generalized linear logistic model, </span>
<span>fit</span><span>=</span><span>glm</span><span>(</span><span>output</span><span>~</span>.,<span>family</span><span>=</span><span>binomial</span>,<span>data</span><span>=</span><span>train</span>,<span>control</span> <span>=</span> <span>list</span><span>(</span><span>maxit</span> <span>=</span> <span>50</span><span>)</span><span>)</span></code></pre>
<pre><code><span># Predict the output from the model</span>
<span>a</span><span>=</span><span>predict</span><span>(</span><span>fit</span>,<span>newdata</span><span>=</span><span>train</span>,<span>type</span><span>=</span><span>&quot;response&quot;</span><span>)</span>
<span># Set response >0.5 as 1 and &lt;=0.5 as 0</span>
<span>b</span><span>=</span><span>ifelse</span><span>(</span><span>a</span><span>></span><span>0.5</span>,<span>1</span>,<span>0</span><span>)</span>
<span># Compute the confusion matrix for training data</span>
<span>confusionMatrix</span><span>(</span><span>b</span>,<span>train</span><span>$</span><span>output</span><span>)</span></code></pre>
<pre><code>## Confusion Matrix and Statistics
## 
##           Reference
## Prediction   0   1
##          0 154   0
##          1   0 272
##                                      
##                Accuracy : 1          
##                  95% CI : (0.9914, 1)
##     No Information Rate : 0.6385     
##     P-Value [Acc > NIR] : &lt; 2.2e-16  
##                                      
##                   Kappa : 1          
##  Mcnemar's Test P-Value : NA         
##                                      
##             Sensitivity : 1.0000     
##             Specificity : 1.0000     
##          Pos Pred Value : 1.0000     
##          Neg Pred Value : 1.0000     
##              Prevalence : 0.3615     
##          Detection Rate : 0.3615     
##    Detection Prevalence : 0.3615     
##       Balanced Accuracy : 1.0000     
##                                      
##        'Positive' Class : 0          
## </code></pre>
<pre><code><span>m</span><span>=</span><span>predict</span><span>(</span><span>fit</span>,<span>newdata</span><span>=</span><span>test</span>,<span>type</span><span>=</span><span>&quot;response&quot;</span><span>)</span>
<span>n</span><span>=</span><span>ifelse</span><span>(</span><span>m</span><span>></span><span>0.5</span>,<span>1</span>,<span>0</span><span>)</span>
<span># Compute the confusion matrix for test output</span>
<span>confusionMatrix</span><span>(</span><span>n</span>,<span>test</span><span>$</span><span>output</span><span>)</span></code></pre>
<pre><code>## Confusion Matrix and Statistics
## 
##           Reference
## Prediction  0  1
##          0 52  4
##          1  5 81
##                                           
##                Accuracy : 0.9366          
##                  95% CI : (0.8831, 0.9706)
##     No Information Rate : 0.5986          
##     P-Value [Acc > NIR] : &lt;2e-16          
##                                           
##                   Kappa : 0.8677          
##  Mcnemar's Test P-Value : 1               
##                                           
##             Sensitivity : 0.9123          
##             Specificity : 0.9529          
##          Pos Pred Value : 0.9286          
##          Neg Pred Value : 0.9419          
##              Prevalence : 0.4014          
##          Detection Rate : 0.3662          
##    Detection Prevalence : 0.3944          
##       Balanced Accuracy : 0.9326          
##                                           
##        'Positive' Class : 0               
## </code></pre>
</div>
<div>
<h2>1b. Logistic Regression – Python code</h2>
<pre><code><span>import</span> numpy <span>as</span> np
<span>import</span> pandas <span>as</span> pd
<span>import</span> os
<span>import</span> matplotlib.pyplot <span>as</span> plt
<span>from</span> sklearn.model_selection <span>import</span> train_test_split
<span>from</span> sklearn.linear_model <span>import</span> LogisticRegression
os.chdir(<span>&quot;C:\\Users\\Ganesh\\RandPython&quot;</span>)
<span>from</span> sklearn.datasets <span>import</span> make_classification, make_blobs

<span>from</span> sklearn.metrics <span>import</span> confusion_matrix
<span>from</span> matplotlib.colors <span>import</span> ListedColormap
<span>from</span> sklearn.datasets <span>import</span> load_breast_cancer
<span># Load the cancer data</span>
(X_cancer, y_cancer) = load_breast_cancer(return_X_y = <span>True</span>)
X_train, X_test, y_train, y_test = train_test_split(X_cancer, y_cancer,
                                                   random_state = <span>0</span>)
<span># Call the Logisitic Regression function</span>
clf = LogisticRegression().fit(X_train, y_train)
fig, subaxes = plt.subplots(<span>1</span>, <span>1</span>, figsize=(<span>7</span>, <span>5</span>))
<span># Fit a model</span>
clf = LogisticRegression().fit(X_train, y_train)

<span># Compute and print the Accuray scores</span>
<span>print</span>(<span>'Accuracy of Logistic regression classifier on training set: {:.2f}'</span>
     .format(clf.score(X_train, y_train)))
<span>print</span>(<span>'Accuracy of Logistic regression classifier on test set: {:.2f}'</span>
     .format(clf.score(X_test, y_test)))
y_predicted=clf.predict(X_test)
<span># Compute and print confusion matrix</span>
confusion = confusion_matrix(y_test, y_predicted)
<span>from</span> sklearn.metrics <span>import</span> accuracy_score, precision_score, recall_score, f1_score
<span>print</span>(<span>'Accuracy: {:.2f}'</span>.format(accuracy_score(y_test, y_predicted)))
<span>print</span>(<span>'Precision: {:.2f}'</span>.format(precision_score(y_test, y_predicted)))
<span>print</span>(<span>'Recall: {:.2f}'</span>.format(recall_score(y_test, y_predicted)))
<span>print</span>(<span>'F1: {:.2f}'</span>.format(f1_score(y_test, y_predicted)))</code></pre>
<pre><code>## Accuracy of Logistic regression classifier on training set: 0.96
## Accuracy of Logistic regression classifier on test set: 0.96
## Accuracy: 0.96
## Precision: 0.99
## Recall: 0.94
## F1: 0.97</code></pre>
</div>
<div>
<h2>2. Dummy variables</h2>
<p>The following R and Python code show how dummy variables are handled in R and Python. Dummy variables are categorival variables which have to be converted into appropriate values before using them in Machine Learning Model For e.g. if we had currency as ‘dollar’, ‘rupee’ and ‘yen’ then the dummy variable will convert this as<br>
dollar 0 0 0<br>
rupee 0 0 1<br>
yen 0 1 0</p>
<h2>2a. Logistic Regression with dummy variables- R code</h2>
<pre><code><span># Load the dummies library</span>
<span>library</span><span>(</span><span>dummies</span><span>)</span> </code></pre>
<pre><code><span>df</span> <span>&lt;-</span> <span>read.csv</span><span>(</span><span>&quot;adult1.csv&quot;</span>,<span>stringsAsFactors</span> <span>=</span> <span>FALSE</span>,<span>na.strings</span> <span>=</span> <span>c</span><span>(</span><span>&quot;&quot;</span>,<span>&quot; &quot;</span>,<span>&quot; ?&quot;</span><span>)</span><span>)</span>

<span># Remove rows which have NA</span>
<span>df1</span> <span>&lt;-</span> <span>df</span><span>[</span><span>complete.cases</span><span>(</span><span>df</span><span>)</span>,<span>]</span>
<span>dim</span><span>(</span><span>df1</span><span>)</span></code></pre>
<pre><code>## [1] 30161    16</code></pre>
<pre><code><span># Select specific columns</span>
<span>adult</span> <span>&lt;-</span> <span>df1</span> <span>%>%</span> <span>dplyr</span><span>:</span><span>:</span><span>select</span><span>(</span><span>age</span>,<span>occupation</span>,<span>education</span>,<span>educationNum</span>,<span>capitalGain</span>,
                               <span>capital.loss</span>,<span>hours.per.week</span>,<span>native.country</span>,<span>salary</span><span>)</span>
<span># Set the dummy data with appropriate values</span>
<span>adult1</span> <span>&lt;-</span> <span>dummy.data.frame</span><span>(</span><span>adult</span>, <span>sep</span> <span>=</span> <span>&quot;.&quot;</span><span>)</span>

<span>#Split as training and test</span>
<span>train_idx</span> <span>&lt;-</span> <span>trainTestSplit</span><span>(</span><span>adult1</span>,<span>trainPercent</span><span>=</span><span>75</span>,<span>seed</span><span>=</span><span>1111</span><span>)</span>
<span>train</span> <span>&lt;-</span> <span>adult1</span><span>[</span><span>train_idx</span>, <span>]</span>
<span>test</span> <span>&lt;-</span> <span>adult1</span><span>[</span><span>-</span><span>train_idx</span>, <span>]</span>

<span># Fit a binomial logistic regression</span>
<span>fit</span><span>=</span><span>glm</span><span>(</span><span>salary</span><span>~</span>.,<span>family</span><span>=</span><span>binomial</span>,<span>data</span><span>=</span><span>train</span><span>)</span></code></pre>
<pre><code><span># Predict response</span>
<span>a</span><span>=</span><span>predict</span><span>(</span><span>fit</span>,<span>newdata</span><span>=</span><span>train</span>,<span>type</span><span>=</span><span>&quot;response&quot;</span><span>)</span></code></pre>
<pre><code><span># If response >0.5 then it is a 1 and 0 otherwise</span>
<span>b</span><span>=</span><span>ifelse</span><span>(</span><span>a</span><span>></span><span>0.5</span>,<span>1</span>,<span>0</span><span>)</span>
<span>confusionMatrix</span><span>(</span><span>b</span>,<span>train</span><span>$</span><span>salary</span><span>)</span></code></pre>
<pre><code>## Confusion Matrix and Statistics
## 
##           Reference
## Prediction     0     1
##          0 16065  3145
##          1   968  2442
##                                           
##                Accuracy : 0.8182          
##                  95% CI : (0.8131, 0.8232)
##     No Information Rate : 0.753           
##     P-Value [Acc > NIR] : &lt; 2.2e-16       
##                                           
##                   Kappa : 0.4375          
##  Mcnemar's Test P-Value : &lt; 2.2e-16       
##                                           
##             Sensitivity : 0.9432          
##             Specificity : 0.4371          
##          Pos Pred Value : 0.8363          
##          Neg Pred Value : 0.7161          
##              Prevalence : 0.7530          
##          Detection Rate : 0.7102          
##    Detection Prevalence : 0.8492          
##       Balanced Accuracy : 0.6901          
##                                           
##        'Positive' Class : 0               
## </code></pre>
<pre><code><span># Compute and display confusion matrix</span>
<span>m</span><span>=</span><span>predict</span><span>(</span><span>fit</span>,<span>newdata</span><span>=</span><span>test</span>,<span>type</span><span>=</span><span>&quot;response&quot;</span><span>)</span></code></pre>
<pre><code>## Warning in predict.lm(object, newdata, se.fit, scale = 1, type =
## ifelse(type == : prediction from a rank-deficient fit may be misleading</code></pre>
<pre><code><span>n</span><span>=</span><span>ifelse</span><span>(</span><span>m</span><span>></span><span>0.5</span>,<span>1</span>,<span>0</span><span>)</span>
<span>confusionMatrix</span><span>(</span><span>n</span>,<span>test</span><span>$</span><span>salary</span><span>)</span></code></pre>
<pre><code>## Confusion Matrix and Statistics
## 
##           Reference
## Prediction    0    1
##          0 5263 1099
##          1  357  822
##                                           
##                Accuracy : 0.8069          
##                  95% CI : (0.7978, 0.8158)
##     No Information Rate : 0.7453          
##     P-Value [Acc > NIR] : &lt; 2.2e-16       
##                                           
##                   Kappa : 0.4174          
##  Mcnemar's Test P-Value : &lt; 2.2e-16       
##                                           
##             Sensitivity : 0.9365          
##             Specificity : 0.4279          
##          Pos Pred Value : 0.8273          
##          Neg Pred Value : 0.6972          
##              Prevalence : 0.7453          
##          Detection Rate : 0.6979          
##    Detection Prevalence : 0.8437          
##       Balanced Accuracy : 0.6822          
##                                           
##        'Positive' Class : 0               
## </code></pre>
</div>
<div>
<h2>2b. Logistic Regression with dummy variables- Python code</h2>
<p>Pandas has a get_dummies function for handling dummies</p>
<pre><code><span>import</span> numpy <span>as</span> np
<span>import</span> pandas <span>as</span> pd
<span>import</span> os
<span>import</span> matplotlib.pyplot <span>as</span> plt
<span>from</span> sklearn.model_selection <span>import</span> train_test_split
<span>from</span> sklearn.linear_model <span>import</span> LogisticRegression
<span>from</span> sklearn.metrics <span>import</span> confusion_matrix
<span>from</span> sklearn.metrics <span>import</span> accuracy_score, precision_score, recall_score, f1_score
<span># Read data</span>
df =pd.read_csv(<span>&quot;adult1.csv&quot;</span>,encoding=<span>&quot;ISO-8859-1&quot;</span>,na_values=[<span>&quot;&quot;</span>,<span>&quot; &quot;</span>,<span>&quot; ?&quot;</span>])
<span># Drop rows with NA</span>
df1=df.dropna()
<span>print</span>(df1.shape)
<span># Select specific columns</span>
adult = df1[[<span>'age'</span>,<span>'occupation'</span>,<span>'education'</span>,<span>'educationNum'</span>,<span>'capitalGain'</span>,<span>'capital-loss'</span>, 
             <span>'hours-per-week'</span>,<span>'native-country'</span>,<span>'salary'</span>]]

X=adult[[<span>'age'</span>,<span>'occupation'</span>,<span>'education'</span>,<span>'educationNum'</span>,<span>'capitalGain'</span>,<span>'capital-loss'</span>, 
             <span>'hours-per-week'</span>,<span>'native-country'</span>]]
<span># Set approporiate values for dummy variables</span>
X_adult=pd.get_dummies(X,columns=[<span>'occupation'</span>,<span>'education'</span>,<span>'native-country'</span>])
y=adult[<span>'salary'</span>]

X_adult_train, X_adult_test, y_train, y_test = train_test_split(X_adult, y,
                                                   random_state = <span>0</span>)
clf = LogisticRegression().fit(X_adult_train, y_train)

<span># Compute and display Accuracy and Confusion matrix</span>
<span>print</span>(<span>'Accuracy of Logistic regression classifier on training set: {:.2f}'</span>
     .format(clf.score(X_adult_train, y_train)))
<span>print</span>(<span>'Accuracy of Logistic regression classifier on test set: {:.2f}'</span>
     .format(clf.score(X_adult_test, y_test)))
y_predicted=clf.predict(X_adult_test)
confusion = confusion_matrix(y_test, y_predicted)
<span>print</span>(<span>'Accuracy: {:.2f}'</span>.format(accuracy_score(y_test, y_predicted)))
<span>print</span>(<span>'Precision: {:.2f}'</span>.format(precision_score(y_test, y_predicted)))
<span>print</span>(<span>'Recall: {:.2f}'</span>.format(recall_score(y_test, y_predicted)))
<span>print</span>(<span>'F1: {:.2f}'</span>.format(f1_score(y_test, y_predicted)))</code></pre>
<pre><code>## (30161, 16)
## Accuracy of Logistic regression classifier on training set: 0.82
## Accuracy of Logistic regression classifier on test set: 0.81
## Accuracy: 0.81
## Precision: 0.68
## Recall: 0.41
## F1: 0.51</code></pre>
</div>
<div>
<h2>3a – K Nearest Neighbors Classification – R code</h2>
<p>The <a rel="nofollow" href="https://archive.ics.uci.edu/ml/datasets/Adult" target="_blank">Adult</a> data set is taken from <a rel="nofollow" href="https://archive.ics.uci.edu/ml/datasets.html" target="_blank">UCI Machine Learning Repository</a></p>
<pre><code><span>source</span><span>(</span><span>&quot;RFunctions.R&quot;</span><span>)</span>
<span>df</span> <span>&lt;-</span> <span>read.csv</span><span>(</span><span>&quot;adult1.csv&quot;</span>,<span>stringsAsFactors</span> <span>=</span> <span>FALSE</span>,<span>na.strings</span> <span>=</span> <span>c</span><span>(</span><span>&quot;&quot;</span>,<span>&quot; &quot;</span>,<span>&quot; ?&quot;</span><span>)</span><span>)</span>
<span># Remove rows which have NA</span>
<span>df1</span> <span>&lt;-</span> <span>df</span><span>[</span><span>complete.cases</span><span>(</span><span>df</span><span>)</span>,<span>]</span>
<span>dim</span><span>(</span><span>df1</span><span>)</span></code></pre>
<pre><code>## [1] 30161    16</code></pre>
<pre><code><span># Select specific columns</span>
<span>adult</span> <span>&lt;-</span> <span>df1</span> <span>%>%</span> <span>dplyr</span><span>:</span><span>:</span><span>select</span><span>(</span><span>age</span>,<span>occupation</span>,<span>education</span>,<span>educationNum</span>,<span>capitalGain</span>,
                               <span>capital.loss</span>,<span>hours.per.week</span>,<span>native.country</span>,<span>salary</span><span>)</span>
<span># Set dummy variables</span>
<span>adult1</span> <span>&lt;-</span> <span>dummy.data.frame</span><span>(</span><span>adult</span>, <span>sep</span> <span>=</span> <span>&quot;.&quot;</span><span>)</span>

<span>#Split train and test as required by KNN classsification model</span>
<span>train_idx</span> <span>&lt;-</span> <span>trainTestSplit</span><span>(</span><span>adult1</span>,<span>trainPercent</span><span>=</span><span>75</span>,<span>seed</span><span>=</span><span>1111</span><span>)</span>
<span>train</span> <span>&lt;-</span> <span>adult1</span><span>[</span><span>train_idx</span>, <span>]</span>
<span>test</span> <span>&lt;-</span> <span>adult1</span><span>[</span><span>-</span><span>train_idx</span>, <span>]</span>
<span>train.X</span> <span>&lt;-</span> <span>train</span><span>[</span>,<span>1</span><span>:</span><span>76</span><span>]</span>
<span>train.y</span> <span>&lt;-</span> <span>train</span><span>[</span>,<span>77</span><span>]</span>
<span>test.X</span> <span>&lt;-</span> <span>test</span><span>[</span>,<span>1</span><span>:</span><span>76</span><span>]</span>
<span>test.y</span> <span>&lt;-</span> <span>test</span><span>[</span>,<span>77</span><span>]</span>

<span># Fit a model for 1,3,5,10 and 15 neighbors</span>
<span>cMat</span> <span>&lt;-</span> <span>NULL</span>
<span>neighbors</span> <span>&lt;-</span><span>c</span><span>(</span><span>1</span>,<span>3</span>,<span>5</span>,<span>10</span>,<span>15</span><span>)</span>
<span>for</span><span>(</span><span>i</span> <span>in</span> <span>seq_along</span><span>(</span><span>neighbors</span><span>)</span><span>)</span><span>{</span>
    <span>fit</span> <span>=</span><span>knn</span><span>(</span><span>train.X</span>,<span>test.X</span>,<span>train.y</span>,<span>k</span><span>=</span><span>i</span><span>)</span>
    <span>table</span><span>(</span><span>fit</span>,<span>test.y</span><span>)</span>
    <span>a</span><span>&lt;-</span><span>confusionMatrix</span><span>(</span><span>fit</span>,<span>test.y</span><span>)</span>
    <span>cMat</span><span>[</span><span>i</span><span>]</span> <span>&lt;-</span> <span>a</span><span>$</span><span>overall</span><span>[</span><span>1</span><span>]</span>
    <span>print</span><span>(</span><span>a</span><span>$</span><span>overall</span><span>[</span><span>1</span><span>]</span><span>)</span>
<span>}</span></code></pre>
<pre><code>##  Accuracy 
## 0.7835831 
##  Accuracy 
## 0.8162047 
##  Accuracy 
## 0.8089113 
##  Accuracy 
## 0.8209787 
##  Accuracy 
## 0.8184591</code></pre>
<pre><code><span>#Plot the Accuracy for each of the KNN models</span>
<span>df</span> <span>&lt;-</span> <span>data.frame</span><span>(</span><span>neighbors</span>,<span>Accuracy</span><span>=</span><span>cMat</span><span>)</span>
<span>ggplot</span><span>(</span><span>df</span>,<span>aes</span><span>(</span><span>x</span><span>=</span><span>neighbors</span>,<span>y</span><span>=</span><span>Accuracy</span><span>)</span><span>)</span> <span>+</span> <span>geom_point</span><span>(</span><span>)</span> <span>+</span><span>geom_line</span><span>(</span><span>color</span><span>=</span><span>&quot;blue&quot;</span><span>)</span> <span>+</span>
    <span>xlab</span><span>(</span><span>&quot;Number of neighbors&quot;</span><span>)</span> <span>+</span> <span>ylab</span><span>(</span><span>&quot;Accuracy&quot;</span><span>)</span> <span>+</span>
    <span>ggtitle</span><span>(</span><span>&quot;KNN regression - Accuracy vs Number of Neighors (Unnormalized)&quot;</span><span>)</span></code></pre>
<p><img><img data-orig-file="https://gigadom.files.wordpress.com/2017/10/unnamed-chunk-5-1.png" data-orig-size="1344,960" data-medium-file="https://gigadom.files.wordpress.com/2017/10/unnamed-chunk-5-1.png?w=300" data-attachment-id="5641" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2017/10/unnamed-chunk-5-1.png?w=450&h=483&fit=676%2C483" data-permalink="https://gigadom.wordpress.com/2017/10/13/practical-machine-learning-with-r-and-python-part-2/unnamed-chunk-5-1-2/" data-large-file="https://gigadom.files.wordpress.com/2017/10/unnamed-chunk-5-1.png?w=450&amp;h=483&amp;fit=676%2C483" alt="" data-image-title="unnamed-chunk-5-1" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-5641" data-recalc-dims="1"></p>
</div>
<div>
<h2>3b – K Nearest Neighbors Classification – Python code</h2>
<pre><code><span>import</span> numpy <span>as</span> np
<span>import</span> pandas <span>as</span> pd
<span>import</span> os
<span>import</span> matplotlib.pyplot <span>as</span> plt
<span>from</span> sklearn.model_selection <span>import</span> train_test_split
<span>from</span> sklearn.metrics <span>import</span> confusion_matrix
<span>from</span> sklearn.metrics <span>import</span> accuracy_score, precision_score, recall_score, f1_score
<span>from</span> sklearn.neighbors <span>import</span> KNeighborsClassifier
<span>from</span> sklearn.preprocessing <span>import</span> MinMaxScaler

<span># Read data</span>
df =pd.read_csv(<span>&quot;adult1.csv&quot;</span>,encoding=<span>&quot;ISO-8859-1&quot;</span>,na_values=[<span>&quot;&quot;</span>,<span>&quot; &quot;</span>,<span>&quot; ?&quot;</span>])
df1=df.dropna()
<span>print</span>(df1.shape)
<span># Select specific columns</span>
adult = df1[[<span>'age'</span>,<span>'occupation'</span>,<span>'education'</span>,<span>'educationNum'</span>,<span>'capitalGain'</span>,<span>'capital-loss'</span>, 
             <span>'hours-per-week'</span>,<span>'native-country'</span>,<span>'salary'</span>]]

X=adult[[<span>'age'</span>,<span>'occupation'</span>,<span>'education'</span>,<span>'educationNum'</span>,<span>'capitalGain'</span>,<span>'capital-loss'</span>, 
             <span>'hours-per-week'</span>,<span>'native-country'</span>]]
             
<span>#Set values for dummy variables</span>
X_adult=pd.get_dummies(X,columns=[<span>'occupation'</span>,<span>'education'</span>,<span>'native-country'</span>])
y=adult[<span>'salary'</span>]

X_adult_train, X_adult_test, y_train, y_test = train_test_split(X_adult, y,
                                                   random_state = <span>0</span>)
                                                   
<span># KNN classification in Python requires the data to be scaled. </span>
<span># Scale the data</span>
scaler = MinMaxScaler()
X_train_scaled = scaler.fit_transform(X_adult_train)
<span># Apply scaling to test set also</span>
X_test_scaled = scaler.transform(X_adult_test)
<span># Compute the KNN model for 1,3,5,10 &amp; 15 neighbors</span>
accuracy=[]
neighbors=[<span>1</span>,<span>3</span>,<span>5</span>,<span>10</span>,<span>15</span>]
<span>for</span> i <span>in</span> neighbors:
    knn = KNeighborsClassifier(n_neighbors = i)
    knn.fit(X_train_scaled, y_train)
    accuracy.append(knn.score(X_test_scaled, y_test))
    <span>print</span>(<span>'Accuracy test score: {:.3f}'</span>
        .format(knn.score(X_test_scaled, y_test)))

<span># Plot the models with the Accuracy attained for each of these models </span>
fig1=plt.plot(neighbors,accuracy)
fig1=plt.title(<span>&quot;KNN regression - Accuracy vs Number of neighbors&quot;</span>)
fig1=plt.xlabel(<span>&quot;Neighbors&quot;</span>)
fig1=plt.ylabel(<span>&quot;Accuracy&quot;</span>)
fig1.figure.savefig(<span>'foo1.png'</span>, bbox_inches=<span>'tight'</span>)</code></pre>
<pre><code>## (30161, 16)
## Accuracy test score: 0.749
## Accuracy test score: 0.779
## Accuracy test score: 0.793
## Accuracy test score: 0.804
## Accuracy test score: 0.803</code></pre>
<p>Output image:</p>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2017/10/foo12.png?w=456" data-orig-size="579,455" data-medium-file="https://gigadom.files.wordpress.com/2017/10/foo12.png?w=456?w=300" data-attachment-id="5637" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2017/10/foo12.png?w=456" data-permalink="https://gigadom.wordpress.com/2017/10/13/practical-machine-learning-with-r-and-python-part-2/foo1-3/" data-large-file="https://gigadom.files.wordpress.com/2017/10/foo12.png?w=456?w=579" alt="" data-image-title="foo1" sizes="(max-width: 579px) 100vw, 579px" data-comments-opened="1" class="wp-image-5637" data-recalc-dims="1"></p>
<div></div>
</div>
<div>
<h2>4 MPG vs Horsepower</h2>
<p>The following scatter plot shows the non-linear relation between mpg and horsepower. This will be used as the data input for computing K Fold Cross Validation Error</p>
<h2>4a MPG vs Horsepower scatter plot – R Code</h2>
<pre><code><span>df</span><span>=</span><span>read.csv</span><span>(</span><span>&quot;auto_mpg.csv&quot;</span>,<span>stringsAsFactors</span> <span>=</span> <span>FALSE</span><span>)</span> <span># Data from UCI</span>
<span>df1</span> <span>&lt;-</span> <span>as.data.frame</span><span>(</span><span>sapply</span><span>(</span><span>df</span>,<span>as.numeric</span><span>)</span><span>)</span></code></pre>
<pre><code><span>df2</span> <span>&lt;-</span> <span>df1</span> <span>%>%</span> <span>dplyr</span><span>:</span><span>:</span><span>select</span><span>(</span><span>cylinder</span>,<span>displacement</span>, <span>horsepower</span>,<span>weight</span>, <span>acceleration</span>, <span>year</span>,<span>mpg</span><span>)</span>
<span>df3</span> <span>&lt;-</span> <span>df2</span><span>[</span><span>complete.cases</span><span>(</span><span>df2</span><span>)</span>,<span>]</span>
<span>ggplot</span><span>(</span><span>df3</span>,<span>aes</span><span>(</span><span>x</span><span>=</span><span>horsepower</span>,<span>y</span><span>=</span><span>mpg</span><span>)</span><span>)</span> <span>+</span> <span>geom_point</span><span>(</span><span>)</span> <span>+</span> <span>xlab</span><span>(</span><span>&quot;Horsepower&quot;</span><span>)</span> <span>+</span> 
    <span>ylab</span><span>(</span><span>&quot;Miles Per gallon&quot;</span><span>)</span> <span>+</span> <span>ggtitle</span><span>(</span><span>&quot;Miles per Gallon vs Hosrsepower&quot;</span><span>)</span></code></pre>
<p><img><img data-orig-file="https://gigadom.files.wordpress.com/2017/10/unnamed-chunk-7-1.png" data-orig-size="1344,960" data-medium-file="https://gigadom.files.wordpress.com/2017/10/unnamed-chunk-7-1.png?w=300" data-attachment-id="5642" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2017/10/unnamed-chunk-7-1.png?w=450&h=483&fit=676%2C483" data-permalink="https://gigadom.wordpress.com/2017/10/13/practical-machine-learning-with-r-and-python-part-2/unnamed-chunk-7-1/" data-large-file="https://gigadom.files.wordpress.com/2017/10/unnamed-chunk-7-1.png?w=450&amp;h=483&amp;fit=676%2C483" alt="" data-image-title="unnamed-chunk-7-1" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-5642" data-recalc-dims="1"></p>
</div>
<div>
<h2>4b MPG vs Horsepower scatter plot – Python Code</h2>
<pre><code><span>import</span> numpy <span>as</span> np
<span>import</span> pandas <span>as</span> pd
<span>import</span> os
<span>import</span> matplotlib.pyplot <span>as</span> plt
autoDF =pd.read_csv(<span>&quot;auto_mpg.csv&quot;</span>,encoding=<span>&quot;ISO-8859-1&quot;</span>)
autoDF.shape
autoDF.columns
autoDF1=autoDF[[<span>'mpg'</span>,<span>'cylinder'</span>,<span>'displacement'</span>,<span>'horsepower'</span>,<span>'weight'</span>,<span>'acceleration'</span>,<span>'year'</span>]]
autoDF2 = autoDF1.apply(pd.to_numeric, errors=<span>'coerce'</span>)
autoDF3=autoDF2.dropna()
autoDF3.shape
<span>#X=autoDF3[['cylinder','displacement','horsepower','weight']]</span>
X=autoDF3[[<span>'horsepower'</span>]]
y=autoDF3[<span>'mpg'</span>]

fig11=plt.scatter(X,y)
fig11=plt.title(<span>&quot;KNN regression - Accuracy vs Number of neighbors&quot;</span>)
fig11=plt.xlabel(<span>&quot;Neighbors&quot;</span>)
fig11=plt.ylabel(<span>&quot;Accuracy&quot;</span>)
fig11.figure.savefig(<span>'foo11.png'</span>, bbox_inches=<span>'tight'</span>)
</code></pre>
</div>
<div>
<h2><img data-orig-file="https://gigadom.files.wordpress.com/2017/10/foo111.png?w=456" data-orig-size="566,455" data-medium-file="https://gigadom.files.wordpress.com/2017/10/foo111.png?w=456?w=300" data-attachment-id="5638" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2017/10/foo111.png?w=456" data-permalink="https://gigadom.wordpress.com/2017/10/13/practical-machine-learning-with-r-and-python-part-2/foo11/" data-large-file="https://gigadom.files.wordpress.com/2017/10/foo111.png?w=456?w=566" alt="" data-image-title="foo11" sizes="(max-width: 566px) 100vw, 566px" data-comments-opened="1" class="wp-image-5638" data-recalc-dims="1"></h2>
<h2>5 K Fold Cross Validation</h2>
<p>K Fold Cross Validation is a technique in which the data set is divided into K Folds or K partitions. The Machine Learning model is trained on K-1 folds and tested on the Kth fold i.e.<br>
we will have K-1 folds for training data and 1 for testing the ML model. Since we can partition this as <img alt="C_{1}^{K}" title="C_{1}^{K}" src="https://s0.wp.com/latex.php?latex=C_%7B1%7D%5E%7BK%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> or K choose 1, there will be K such partitions. The K Fold Cross<br>
Validation estimates the average validation error that we can expect on a new unseen test data.</p>
<p>The formula for K Fold Cross validation is as follows</p>
<p><img alt="MSE_{K} = frac{sum (y-yhat)^{2}}{n_{K}}" title="MSE_{K} = frac{sum (y-yhat)^{2}}{n_{K}}" src="https://s0.wp.com/latex.php?latex=MSE_%7BK%7D+%3D+%5Cfrac%7B%5Csum+%28y-yhat%29%5E%7B2%7D%7D%7Bn_%7BK%7D%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
and<br>
<img alt="n_{K} = frac{N}{K}" title="n_{K} = frac{N}{K}" src="https://s0.wp.com/latex.php?latex=n_%7BK%7D+%3D+%5Cfrac%7BN%7D%7BK%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
and<br>
<img alt="CV_{K} = sum_{K=1}^{K} (frac{n_{K}}{N}) MSE_{K}" title="CV_{K} = sum_{K=1}^{K} (frac{n_{K}}{N}) MSE_{K}" src="https://s0.wp.com/latex.php?latex=CV_%7BK%7D+%3D+%5Csum_%7BK%3D1%7D%5E%7BK%7D+%28%5Cfrac%7Bn_%7BK%7D%7D%7BN%7D%29+MSE_%7BK%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"></p>
<p>where <img alt="n_{K}" title="n_{K}" src="https://s0.wp.com/latex.php?latex=n_%7BK%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> is the number of elements in partition ‘K’ and N is the total number of elements<br>
<img alt="CV_{K} =sum_{K=1}^{K} MSE_{K}" title="CV_{K} =sum_{K=1}^{K} MSE_{K}" src="https://s0.wp.com/latex.php?latex=CV_%7BK%7D+%3D%5Csum_%7BK%3D1%7D%5E%7BK%7D+MSE_%7BK%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"></p>
<p><img alt="CV_{K} =frac{sum_{K=1}^{K} MSE_{K}}{K}" title="CV_{K} =frac{sum_{K=1}^{K} MSE_{K}}{K}" src="https://s0.wp.com/latex.php?latex=CV_%7BK%7D+%3D%5Cfrac%7B%5Csum_%7BK%3D1%7D%5E%7BK%7D+MSE_%7BK%7D%7D%7BK%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
Leave Out one Cross Validation (LOOCV) is a special case of K Fold Cross Validation where N-1 data points are used to train the model and 1 data point is used to test the model. There are N such paritions of N-1 &amp; 1 that are possible. The mean error is measured The Cross Valifation Error for LOOCV is</p>
<p><img alt="CV_{N} = frac{1}{n} *frac{sum_{1}^{n}(y-yhat)^{2}}{1-h_{i}}" title="CV_{N} = frac{1}{n} *frac{sum_{1}^{n}(y-yhat)^{2}}{1-h_{i}}" src="https://s0.wp.com/latex.php?latex=CV_%7BN%7D+%3D+%5Cfrac%7B1%7D%7Bn%7D+%2A%5Cfrac%7B%5Csum_%7B1%7D%5E%7Bn%7D%28y-yhat%29%5E%7B2%7D%7D%7B1-h_%7Bi%7D%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"><br>
where <img alt="h_{i}" title="h_{i}" src="https://s0.wp.com/latex.php?latex=h_%7Bi%7D&bg=ffffff&%23038;fg=444444&%23038;s=0" class="latex"> is the diagonal hat matrix</p>
<p>see [<a rel="nofollow" href="https://lagunita.stanford.edu/courses/HumanitiesandScience/StatLearning/Winter2015/about" target="_blank">Statistical Learning</a>]</p>
<p>The above formula is also included in this <a rel="nofollow" href="https://gerardnico.com/wiki/data_mining/cross_validation" target="_blank">blog post</a></p>
<p>It took me a day and a half to implement the K Fold Cross Validation formula. I think it is correct. In any case do let me know if you think it is off</p>
<h2>5a. Leave out one cross validation (LOOCV) – R Code</h2>
<p>R uses the package ‘boot’ for performing Cross Validation error computation</p>
<pre><code><span>library</span><span>(</span><span>boot</span><span>)</span>
<span>library</span><span>(</span><span>reshape2</span><span>)</span>
<span># Read data</span>
<span>df</span><span>=</span><span>read.csv</span><span>(</span><span>&quot;auto_mpg.csv&quot;</span>,<span>stringsAsFactors</span> <span>=</span> <span>FALSE</span><span>)</span> <span># Data from UCI</span>
<span>df1</span> <span>&lt;-</span> <span>as.data.frame</span><span>(</span><span>sapply</span><span>(</span><span>df</span>,<span>as.numeric</span><span>)</span><span>)</span></code></pre>
<pre><code><span># Select complete cases</span>
<span>df2</span> <span>&lt;-</span> <span>df1</span> <span>%>%</span> <span>dplyr</span><span>:</span><span>:</span><span>select</span><span>(</span><span>cylinder</span>,<span>displacement</span>, <span>horsepower</span>,<span>weight</span>, <span>acceleration</span>, <span>year</span>,<span>mpg</span><span>)</span>
<span>df3</span> <span>&lt;-</span> <span>df2</span><span>[</span><span>complete.cases</span><span>(</span><span>df2</span><span>)</span>,<span>]</span>
<span>set.seed</span><span>(</span><span>17</span><span>)</span>
<span>cv.error</span><span>=</span><span>rep</span><span>(</span><span>0</span>,<span>10</span><span>)</span>
<span># For polynomials 1,2,3... 10 fit a LOOCV model</span>
<span>for</span> <span>(</span><span>i</span> <span>in</span> <span>1</span><span>:</span><span>10</span><span>)</span><span>{</span>
    <span>glm.fit</span><span>=</span><span>glm</span><span>(</span><span>mpg</span><span>~</span><span>poly</span><span>(</span><span>horsepower</span>,<span>i</span><span>)</span>,<span>data</span><span>=</span><span>df3</span><span>)</span>
    <span>cv.error</span><span>[</span><span>i</span><span>]</span><span>=</span><span>cv.glm</span><span>(</span><span>df3</span>,<span>glm.fit</span><span>)</span><span>$</span><span>delta</span><span>[</span><span>1</span><span>]</span>
    
<span>}</span>
<span>cv.error</span></code></pre>
<pre><code>##  [1] 24.23151 19.24821 19.33498 19.42443 19.03321 18.97864 18.83305
##  [8] 18.96115 19.06863 19.49093</code></pre>
<pre><code><span># Create and display a plot</span>
<span>folds</span> <span>&lt;-</span> <span>seq</span><span>(</span><span>1</span>,<span>10</span><span>)</span>
<span>df</span> <span>&lt;-</span> <span>data.frame</span><span>(</span><span>folds</span>,<span>cvError</span><span>=</span><span>cv.error</span><span>)</span>
<span>ggplot</span><span>(</span><span>df</span>,<span>aes</span><span>(</span><span>x</span><span>=</span><span>folds</span>,<span>y</span><span>=</span><span>cvError</span><span>)</span><span>)</span> <span>+</span> <span>geom_point</span><span>(</span><span>)</span> <span>+</span><span>geom_line</span><span>(</span><span>color</span><span>=</span><span>&quot;blue&quot;</span><span>)</span> <span>+</span>
    <span>xlab</span><span>(</span><span>&quot;Degree of Polynomial&quot;</span><span>)</span> <span>+</span> <span>ylab</span><span>(</span><span>&quot;Cross Validation Error&quot;</span><span>)</span> <span>+</span>
    <span>ggtitle</span><span>(</span><span>&quot;Leave one out Cross Validation - Cross Validation Error vs Degree of Polynomial&quot;</span><span>)</span></code></pre>
<p><img><img data-orig-file="https://gigadom.files.wordpress.com/2017/10/unnamed-chunk-9-1.png" data-orig-size="1344,960" data-medium-file="https://gigadom.files.wordpress.com/2017/10/unnamed-chunk-9-1.png?w=300" data-attachment-id="5643" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2017/10/unnamed-chunk-9-1.png?w=450&h=483&fit=676%2C483" data-permalink="https://gigadom.wordpress.com/2017/10/13/practical-machine-learning-with-r-and-python-part-2/unnamed-chunk-9-1/" data-large-file="https://gigadom.files.wordpress.com/2017/10/unnamed-chunk-9-1.png?w=450&amp;h=483&amp;fit=676%2C483" alt="" data-image-title="unnamed-chunk-9-1" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-5643" data-recalc-dims="1"></p>
</div>
<div>
<h2>5b. Leave out one cross validation (LOOCV) – Python Code</h2>
<p>In Python there is no available function to compute Cross Validation error and we have to compute the above formula. I have done this after several hours. I think it is now in reasonable shape. Do let me know if you think otherwise. For LOOCV I use the K Fold Cross Validation with K=N</p>
<pre><code><span>import</span> numpy <span>as</span> np
<span>import</span> pandas <span>as</span> pd
<span>import</span> os
<span>import</span> matplotlib.pyplot <span>as</span> plt
<span>from</span> sklearn.linear_model <span>import</span> LinearRegression
<span>from</span> sklearn.cross_validation <span>import</span> train_test_split, KFold
<span>from</span> sklearn.preprocessing <span>import</span> PolynomialFeatures
<span>from</span> sklearn.metrics <span>import</span> mean_squared_error
<span># Read data</span>
autoDF =pd.read_csv(<span>&quot;auto_mpg.csv&quot;</span>,encoding=<span>&quot;ISO-8859-1&quot;</span>)
autoDF.shape
autoDF.columns
autoDF1=autoDF[[<span>'mpg'</span>,<span>'cylinder'</span>,<span>'displacement'</span>,<span>'horsepower'</span>,<span>'weight'</span>,<span>'acceleration'</span>,<span>'year'</span>]]
autoDF2 = autoDF1.apply(pd.to_numeric, errors=<span>'coerce'</span>)
<span># Remove rows with NAs</span>
autoDF3=autoDF2.dropna()
autoDF3.shape
X=autoDF3[[<span>'horsepower'</span>]]
y=autoDF3[<span>'mpg'</span>]

<span># For polynomial degree 1,2,3... 10</span>
<span><span>def</span> <span>computeCVError</span><span>(X,y,folds)</span>:</span>
    deg=[]
    mse=[]
    degree1=[<span>1</span>,<span>2</span>,<span>3</span>,<span>4</span>,<span>5</span>,<span>6</span>,<span>7</span>,<span>8</span>,<span>9</span>,<span>10</span>]
    
    nK=len(X)/float(folds)
    xval_err=<span>0</span>
    <span># For degree 'j'</span>
    <span>for</span> j <span>in</span> degree1: 
        <span># Split as 'folds'</span>
        kf = KFold(len(X),n_folds=folds)
        <span>for</span> train_index, test_index <span>in</span> kf:
            <span># Create the appropriate train and test partitions from the fold index</span>
            X_train, X_test = X.iloc[train_index], X.iloc[test_index]
            y_train, y_test = y.iloc[train_index], y.iloc[test_index]  

            <span># For the polynomial degree 'j'</span>
            poly = PolynomialFeatures(degree=j)        
            <span># Transform the X_train and X_test</span>
            X_train_poly = poly.fit_transform(X_train)
            X_test_poly = poly.fit_transform(X_test)
            <span># Fit a model on the transformed data</span>
            linreg = LinearRegression().fit(X_train_poly, y_train)
            <span># Compute yhat or ypred</span>
            y_pred = linreg.predict(X_test_poly)   
            <span># Compute MSE * n_K/N</span>
            test_mse = mean_squared_error(y_test, y_pred)*float(len(X_train))/float(len(X))     
            <span># Add the test_mse for this partition of the data</span>
            mse.append(test_mse)
        <span># Compute the mean of all folds for degree 'j' </span>
        deg.append(np.mean(mse))
        
    <span>return</span>(deg)


df=pd.DataFrame()
<span>print</span>(len(X))
<span># Call the function once. For LOOCV K=N. hence len(X) is passed as number of folds</span>
cvError=computeCVError(X,y,len(X))

<span># Create and plot LOOCV</span>
df=pd.DataFrame(cvError)
fig3=df.plot()
fig3=plt.title(<span>&quot;Leave one out Cross Validation - Cross Validation Error vs Degree of Polynomial&quot;</span>)
fig3=plt.xlabel(<span>&quot;Degree of Polynomial&quot;</span>)
fig3=plt.ylabel(<span>&quot;Cross validation Error&quot;</span>)
fig3.figure.savefig(<span>'foo3.png'</span>, bbox_inches=<span>'tight'</span>)</code></pre>
<p><img> <img data-orig-file="https://gigadom.files.wordpress.com/2017/10/foo31.png" data-orig-size="682,455" data-medium-file="https://gigadom.files.wordpress.com/2017/10/foo31.png?w=300" data-attachment-id="5639" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2017/10/foo31.png?w=450&h=451&fit=676%2C451" data-permalink="https://gigadom.wordpress.com/2017/10/13/practical-machine-learning-with-r-and-python-part-2/foo3-2/" data-large-file="https://gigadom.files.wordpress.com/2017/10/foo31.png?w=450&amp;h=451&amp;fit=676%2C451" alt="" data-image-title="foo3" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-5639" data-recalc-dims="1"></p>
<h2>6a K Fold Cross Validation – R code</h2>
<p>Here K Fold Cross Validation is done for 4, 5 and 10 folds using the R package boot and the glm package</p>
<pre><code><span>library</span><span>(</span><span>boot</span><span>)</span>
<span>library</span><span>(</span><span>reshape2</span><span>)</span>
<span>set.seed</span><span>(</span><span>17</span><span>)</span>
<span>#Read data</span>
<span>df</span><span>=</span><span>read.csv</span><span>(</span><span>&quot;auto_mpg.csv&quot;</span>,<span>stringsAsFactors</span> <span>=</span> <span>FALSE</span><span>)</span> <span># Data from UCI</span>
<span>df1</span> <span>&lt;-</span> <span>as.data.frame</span><span>(</span><span>sapply</span><span>(</span><span>df</span>,<span>as.numeric</span><span>)</span><span>)</span></code></pre>
<pre><code><span>df2</span> <span>&lt;-</span> <span>df1</span> <span>%>%</span> <span>dplyr</span><span>:</span><span>:</span><span>select</span><span>(</span><span>cylinder</span>,<span>displacement</span>, <span>horsepower</span>,<span>weight</span>, <span>acceleration</span>, <span>year</span>,<span>mpg</span><span>)</span>
<span>df3</span> <span>&lt;-</span> <span>df2</span><span>[</span><span>complete.cases</span><span>(</span><span>df2</span><span>)</span>,<span>]</span>
<span>a</span><span>=</span><span>matrix</span><span>(</span><span>rep</span><span>(</span><span>0</span>,<span>30</span><span>)</span>,<span>nrow</span><span>=</span><span>3</span>,<span>ncol</span><span>=</span><span>10</span><span>)</span>
<span>set.seed</span><span>(</span><span>17</span><span>)</span>
<span># Set the folds as 4,5 and 10</span>
<span>folds</span><span>&lt;-</span><span>c</span><span>(</span><span>4</span>,<span>5</span>,<span>10</span><span>)</span>
<span>for</span><span>(</span><span>i</span> <span>in</span> <span>seq_along</span><span>(</span><span>folds</span><span>)</span><span>)</span><span>{</span>
    <span>cv.error.10</span><span>=</span><span>rep</span><span>(</span><span>0</span>,<span>10</span><span>)</span>
    <span>for</span> <span>(</span><span>j</span> <span>in</span> <span>1</span><span>:</span><span>10</span><span>)</span><span>{</span>
        <span># Fit a generalized linear model</span>
        <span>glm.fit</span><span>=</span><span>glm</span><span>(</span><span>mpg</span><span>~</span><span>poly</span><span>(</span><span>horsepower</span>,<span>j</span><span>)</span>,<span>data</span><span>=</span><span>df3</span><span>)</span>
        <span># Compute K Fold Validation error</span>
        <span>a</span><span>[</span><span>i</span>,<span>j</span><span>]</span><span>=</span><span>cv.glm</span><span>(</span><span>df3</span>,<span>glm.fit</span>,<span>K</span><span>=</span><span>folds</span><span>[</span><span>i</span><span>]</span><span>)</span><span>$</span><span>delta</span><span>[</span><span>1</span><span>]</span>
        
    <span>}</span>
    
<span>}</span>

<span># Create and display the K Fold Cross Validation Error</span>
<span>b</span> <span>&lt;-</span> <span>t</span><span>(</span><span>a</span><span>)</span>
<span>df</span> <span>&lt;-</span> <span>data.frame</span><span>(</span><span>b</span><span>)</span>
<span>df1</span> <span>&lt;-</span> <span>cbind</span><span>(</span><span>seq</span><span>(</span><span>1</span>,<span>10</span><span>)</span>,<span>df</span><span>)</span>
<span>names</span><span>(</span><span>df1</span><span>)</span> <span>&lt;-</span> <span>c</span><span>(</span><span>&quot;PolynomialDegree&quot;</span>,<span>&quot;4-fold&quot;</span>,<span>&quot;5-fold&quot;</span>,<span>&quot;10-fold&quot;</span><span>)</span>

<span>df2</span> <span>&lt;-</span> <span>melt</span><span>(</span><span>df1</span>,<span>id</span><span>=</span><span>&quot;PolynomialDegree&quot;</span><span>)</span>
<span>ggplot</span><span>(</span><span>df2</span><span>)</span> <span>+</span> <span>geom_line</span><span>(</span><span>aes</span><span>(</span><span>x</span><span>=</span><span>PolynomialDegree</span>, <span>y</span><span>=</span><span>value</span>, <span>colour</span><span>=</span><span>variable</span><span>)</span>,<span>size</span><span>=</span><span>2</span><span>)</span> <span>+</span>
    <span>xlab</span><span>(</span><span>&quot;Degree of Polynomial&quot;</span><span>)</span> <span>+</span> <span>ylab</span><span>(</span><span>&quot;Cross Validation Error&quot;</span><span>)</span> <span>+</span>
    <span>ggtitle</span><span>(</span><span>&quot;K Fold Cross Validation - Cross Validation Error vs Degree of Polynomial&quot;</span><span>)</span></code></pre>
<p><img><img data-orig-file="https://gigadom.files.wordpress.com/2017/10/unnamed-chunk-11-1.png" data-orig-size="1344,960" data-medium-file="https://gigadom.files.wordpress.com/2017/10/unnamed-chunk-11-1.png?w=300" data-attachment-id="5644" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2017/10/unnamed-chunk-11-1.png?w=450&h=483&fit=676%2C483" data-permalink="https://gigadom.wordpress.com/2017/10/13/practical-machine-learning-with-r-and-python-part-2/unnamed-chunk-11-1/" data-large-file="https://gigadom.files.wordpress.com/2017/10/unnamed-chunk-11-1.png?w=450&amp;h=483&amp;fit=676%2C483" alt="" data-image-title="unnamed-chunk-11-1" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-5644" data-recalc-dims="1"></p>
</div>
<div>
<h2>6b. K Fold Cross Validation – Python code</h2>
<p>The implementation of K-Fold Cross Validation Error has to be implemented and I have done this below. There is a small discrepancy in the shapes of the curves with the R plot above. Not sure why!</p>
<pre><code><span>import</span> numpy <span>as</span> np
<span>import</span> pandas <span>as</span> pd
<span>import</span> os
<span>import</span> matplotlib.pyplot <span>as</span> plt
<span>from</span> sklearn.linear_model <span>import</span> LinearRegression
<span>from</span> sklearn.cross_validation <span>import</span> train_test_split, KFold
<span>from</span> sklearn.preprocessing <span>import</span> PolynomialFeatures
<span>from</span> sklearn.metrics <span>import</span> mean_squared_error
<span># Read data</span>
autoDF =pd.read_csv(<span>&quot;auto_mpg.csv&quot;</span>,encoding=<span>&quot;ISO-8859-1&quot;</span>)
autoDF.shape
autoDF.columns
autoDF1=autoDF[[<span>'mpg'</span>,<span>'cylinder'</span>,<span>'displacement'</span>,<span>'horsepower'</span>,<span>'weight'</span>,<span>'acceleration'</span>,<span>'year'</span>]]
autoDF2 = autoDF1.apply(pd.to_numeric, errors=<span>'coerce'</span>)
<span># Drop NA rows</span>
autoDF3=autoDF2.dropna()
autoDF3.shape
<span>#X=autoDF3[['cylinder','displacement','horsepower','weight']]</span>
X=autoDF3[[<span>'horsepower'</span>]]
y=autoDF3[<span>'mpg'</span>]

<span># Create Cross Validation function</span>
<span><span>def</span> <span>computeCVError</span><span>(X,y,folds)</span>:</span>
    deg=[]
    mse=[]
    <span># For degree 1,2,3,..10</span>
    degree1=[<span>1</span>,<span>2</span>,<span>3</span>,<span>4</span>,<span>5</span>,<span>6</span>,<span>7</span>,<span>8</span>,<span>9</span>,<span>10</span>]
    
    nK=len(X)/float(folds)
    xval_err=<span>0</span>
    <span>for</span> j <span>in</span> degree1: 
        <span># Split the data into 'folds'</span>
        kf = KFold(len(X),n_folds=folds)
        <span>for</span> train_index, test_index <span>in</span> kf:
            <span># Partition the data acccording the fold indices generated</span>
            X_train, X_test = X.iloc[train_index], X.iloc[test_index]
            y_train, y_test = y.iloc[train_index], y.iloc[test_index]  

            <span># Scale the X_train and X_test as per the polynomial degree 'j'</span>
            poly = PolynomialFeatures(degree=j)             
            X_train_poly = poly.fit_transform(X_train)
            X_test_poly = poly.fit_transform(X_test)
            <span># Fit a polynomial regression</span>
            linreg = LinearRegression().fit(X_train_poly, y_train)
            <span># Compute yhat or ypred</span>
            y_pred = linreg.predict(X_test_poly)  
            <span># Compute MSE *(nK/N)</span>
            test_mse = mean_squared_error(y_test, y_pred)*float(len(X_train))/float(len(X))  
            <span># Append to list for different folds</span>
            mse.append(test_mse)
        <span># Compute the mean for poylnomial 'j' </span>
        deg.append(np.mean(mse))
        
    <span>return</span>(deg)

<span># Create and display a plot of K -Folds</span>
df=pd.DataFrame()
<span>for</span> folds <span>in</span> [<span>4</span>,<span>5</span>,<span>10</span>]:
    cvError=computeCVError(X,y,folds)
    <span>#print(cvError)</span>
    df1=pd.DataFrame(cvError)
    df=pd.concat([df,df1],axis=<span>1</span>)
    <span>#print(cvError)</span>
    
df.columns=[<span>'4-fold'</span>,<span>'5-fold'</span>,<span>'10-fold'</span>]
df=df.reindex([<span>1</span>,<span>2</span>,<span>3</span>,<span>4</span>,<span>5</span>,<span>6</span>,<span>7</span>,<span>8</span>,<span>9</span>,<span>10</span>])
df
fig2=df.plot()
fig2=plt.title(<span>&quot;K Fold Cross Validation - Cross Validation Error vs Degree of Polynomial&quot;</span>)
fig2=plt.xlabel(<span>&quot;Degree of Polynomial&quot;</span>)
fig2=plt.ylabel(<span>&quot;Cross validation Error&quot;</span>)
fig2.figure.savefig(<span>'foo2.png'</span>, bbox_inches=<span>'tight'</span>)
</code></pre>
<div>
<p>output</p>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2017/10/foo21.png?w=456" data-orig-size="615,455" data-medium-file="https://gigadom.files.wordpress.com/2017/10/foo21.png?w=456?w=300" data-attachment-id="5640" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2017/10/foo21.png?w=456" data-permalink="https://gigadom.wordpress.com/2017/10/13/practical-machine-learning-with-r-and-python-part-2/foo2-2/" data-large-file="https://gigadom.files.wordpress.com/2017/10/foo21.png?w=456?w=615" alt="" data-image-title="foo2" sizes="(max-width: 615px) 100vw, 615px" data-comments-opened="1" class="wp-image-5640" data-recalc-dims="1"></p>
</div>
<p>This concludes this 2nd part of this series. I will look into model tuning and model selection in R and Python in the coming parts. Comments, suggestions and corrections are welcome!<br>
To be continued….<br>
Watch this space!</p>
<p>Also see</p>
<ol>
<li><a rel="nofollow" href="https://gigadom.wordpress.com/2011/05/13/design-principles-of-scalable-distributed-systems/" target="_blank">Design Principles of Scalable, Distributed Systems</a></li>
<li><a rel="nofollow" href="https://gigadom.wordpress.com/2016/05/14/re-introducing-cricketr-an-r-package-to-analyze-performances-of-cricketers/" target="_blank">Re-introducing cricketr! : An R package to analyze performances of cricketers</a></li>
<li><a rel="nofollow" href="https://gigadom.wordpress.com/2014/08/07/spicing-up-a-ibm-bluemix-cloud-app-with-mongodb-and-nodeexpress/" target="_blank">Spicing up a IBM Bluemix cloud app with MongoDB and NodeExpress</a></li>
<li><a rel="nofollow" href="https://gigadom.wordpress.com/2017/09/29/using-linear-programming-lp-for-optimizing-bowling-change-or-batting-lineup/" target="_blank">Using Linear Programming (LP) for optimizing bowling change or batting lineup in T20 cricket</a></li>
<li><a rel="nofollow" href="https://gigadom.wordpress.com/2013/06/07/simulating-an-edge-shape-in-android/" target="_blank">Simulating an Edge Shape in Android</a></li>
</ol>
<p>To see all posts see <a rel="nofollow" href="https://gigadom.wordpress.com/aa-2/" target="_blank">Index of posts</a></p>
</div>
<p> <a rel="nofollow" href="http://feeds.wordpress.com/1.0/gocomments/gigadom.wordpress.com/5632/" target="_blank"><img border="0" alt="" src="http://feeds.wordpress.com/1.0/comments/gigadom.wordpress.com/5632/"></a> <img border="0" alt="" src="https://i1.wp.com/pixel.wp.com/b.gif?resize=1%2C1&ssl=1" data-recalc-dims="1"></p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://gigadom.wordpress.com/2017/10/13/practical-machine-learning-with-r-and-python-part-2/"> R – Giga thoughts …</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/XAtDMcKnHG4" height="1"></a></div>