<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/practical-machine-learning-with-r-and-python-part-4/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://gigadom.wordpress.com/2017/10/29/practical-machine-learning-with-r-and-python-part-4/"> R – Giga thoughts …</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>This is the 4th installment of my ‘Practical Machine Learning with R and Python’ series. In this part I discuss classification with Support Vector Machines (SVMs), using both a Linear and a Radial basis kernel, and Decision Trees. Further, a closer look is taken at some of the metrics associated with binary classification, namely accuracy vs precision and recall. I also touch upon Validation curves, Precision-Recall, ROC curves and AUC with equivalent code in R and Python</p>
<p>This post is a continuation of my 3 earlier posts on Practical Machine Learning in R and Python<br>
1. <a rel="nofollow" href="https://gigadom.wordpress.com/2017/10/06/practical-machine-learning-with-r-and-python-part-1/" target="_blank">Practical Machine Learning with R and Python – Part 1</a><br>
2. <a rel="nofollow" href="https://gigadom.wordpress.com/2017/10/13/practical-machine-learning-with-r-and-python-part-2/" target="_blank">Practical Machine Learning with R and Python – Part 2</a><br>
3. <a rel="nofollow" href="https://gigadom.wordpress.com/2017/10/20/practical-machine-learning-with-r-and-python-part-3/" target="_blank">Practical Machine Learning with R and Python – Part 3</a></p>
<p>The RMarkdown file with the code and the associated data files can be downloaded from Github at <a rel="nofollow" href="https://github.com/tvganesh/MachineLearning-RandPython-Part4" target="_blank">MachineLearning-RandPython-Part4</a></p>
<p>Support Vector Machines (SVM) are another useful Machine Learning model that can be used for both regression and classification problems. SVMs used in classification, compute the hyperplane, that separates the 2 classes with the maximum margin. To do this the features may be transformed into a larger multi-dimensional feature space. SVMs can be used with different kernels namely linear, polynomial or radial basis to determine the best fitting model for a given classification problem.</p>
<p>In the 2nd part of this series <a rel="nofollow" href="https://gigadom.wordpress.com/2017/10/13/practical-machine-learning-with-r-and-python-part-2/" target="_blank">Practical Machine Learning with R and Python – Part 2</a>, I had mentioned the various metrics that are used in classification ML problems namely Accuracy, Precision, Recall and F1 score. Accuracy gives the fraction of data that were correctly classified as belonging to the +ve or -ve class. However ‘accuracy’ in itself is not a good enough measure because it does not take into account the fraction of the data that were incorrectly classified. This issue becomes even more critical in different domains. For e.g a surgeon who would like to detect cancer, would like to err on the side of caution, and classify even a possibly non-cancerous patient as possibly having cancer, rather than mis-classifying a malignancy as benign. Here we would like to increase recall or sensitivity which is  given by Recall= TP/(TP+FN) or we try reduce mis-classification by either increasing the (true positives) TP or reducing (false negatives) FN</p>
<p>On the other hand, search algorithms would like to increase precision which tries to reduce the number of irrelevant results in the search result. Precision= TP/(TP+FP). In other words we do not want ‘false positives’ or irrelevant results to come in the search results and there is a need to reduce the false positives.</p>
<p>When we try to increase ‘precision’, we do so at the cost of ‘recall’, and vice-versa. I found this diagram and explanation in Wikipedia very useful Source: <a rel="nofollow" href="https://en.wikipedia.org/wiki/Precision_and_recall" target="_blank">Wikipedia</a></p>
<p>“Consider a brain surgeon tasked with removing a cancerous tumor from a patient’s brain. The surgeon needs to remove all of the tumor cells since any remaining cancer cells will regenerate the tumor. Conversely, the surgeon must not remove healthy brain cells since that would leave the patient with impaired brain function. The surgeon may be more liberal in the area of the brain she removes to ensure she has extracted all the cancer cells. This decision increases recall but reduces precision. On the other hand, the surgeon may be more conservative in the brain she removes to ensure she extracts only cancer cells. This decision increases precision but reduces recall. That is to say, greater recall increases the chances of removing healthy cells (negative outcome) and increases the chances of removing all cancer cells (positive outcome). Greater precision decreases the chances of removing healthy cells (positive outcome) but also decreases the chances of removing all cancer cells (negative outcome).”</p>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2017/10/untitled.png?w=456" data-orig-size="382,409" data-medium-file="https://gigadom.files.wordpress.com/2017/10/untitled.png?w=456?w=280" data-attachment-id="5725" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2017/10/untitled.png?w=456" data-permalink="https://gigadom.wordpress.com/2017/10/29/practical-machine-learning-with-r-and-python-part-4/untitled-107/" data-large-file="https://gigadom.files.wordpress.com/2017/10/untitled.png?w=456?w=382" alt="" data-image-title="Untitled" sizes="(max-width: 382px) 100vw, 382px" data-comments-opened="1" class="wp-image-5725" data-recalc-dims="1"></p>
<div>
<h2>1.1a. Linear SVM – R code</h2>
<p>In R code below I use SVM with linear kernel</p>
<pre><code><span>source</span><span>(</span><span>'RFunctions-1.R'</span><span>)
<span>library</span>(<span>dplyr</span>)
<span>library</span>(<span>e1071</span>)
<span>library</span>(<span>caret</span>)
<span>library</span>(<span>reshape2</span>)
<span>library</span>(<span>ggplot2</span>)</span></code></pre>
<pre><code><span># Read data. Data from SKLearn</span>
<span>cancer</span> <span>&lt;-</span> <span>read.csv</span><span>(</span><span>&quot;cancer.csv&quot;</span><span>)</span>
<span>cancer</span><span>$</span><span>target</span> <span>&lt;-</span> <span>as.factor</span><span>(</span><span>cancer</span><span>$</span><span>target</span><span>)</span>

<span># Split into training and test sets</span>
<span>train_idx</span> <span>&lt;-</span> <span>trainTestSplit</span><span>(</span><span>cancer</span>,<span>trainPercent</span><span>=</span><span>75</span>,<span>seed</span><span>=</span><span>5</span><span>)</span>
<span>train</span> <span>&lt;-</span> <span>cancer</span><span>[</span><span>train_idx</span>, <span>]</span>
<span>test</span> <span>&lt;-</span> <span>cancer</span><span>[</span><span>-</span><span>train_idx</span>, <span>]</span>

<span># Fit a linear basis kernel. DO not scale the data</span>
<span>svmfit</span><span>=</span><span>svm</span><span>(</span><span>target</span><span>~</span>., <span>data</span><span>=</span><span>train</span>, <span>kernel</span><span>=</span><span>&quot;linear&quot;</span>,<span>scale</span><span>=</span><span>FALSE</span><span>)</span>
<span>ypred</span><span>=</span><span>predict</span><span>(</span><span>svmfit</span>,<span>test</span><span>)</span>
<span>#Print a confusion matrix</span>
<span>confusionMatrix</span><span>(</span><span>ypred</span>,<span>test</span><span>$</span><span>target</span><span>)</span></code></pre>
<pre><code>## Confusion Matrix and Statistics
## 
##           Reference
## Prediction  0  1
##          0 54  3
##          1  3 82
##                                           
##                Accuracy : 0.9577          
##                  95% CI : (0.9103, 0.9843)
##     No Information Rate : 0.5986          
##     P-Value [Acc > NIR] : &lt;2e-16          
##                                           
##                   Kappa : 0.9121          
##  Mcnemar's Test P-Value : 1               
##                                           
##             Sensitivity : 0.9474          
##             Specificity : 0.9647          
##          Pos Pred Value : 0.9474          
##          Neg Pred Value : 0.9647          
##              Prevalence : 0.4014          
##          Detection Rate : 0.3803          
##    Detection Prevalence : 0.4014          
##       Balanced Accuracy : 0.9560          
##                                           
##        'Positive' Class : 0               
## </code></pre>
</div>
<div>
<h2>1.1b Linear SVM – Python code</h2>
<p>The code below creates a SVM with linear basis in Python and also dumps the corresponding classification metrics</p>
<pre><code><span>import</span> numpy <span>as</span> np
<span>import</span> pandas <span>as</span> pd
<span>import</span> os
<span>import</span> matplotlib.pyplot <span>as</span> plt
<span>from</span> sklearn.model_selection <span>import</span> train_test_split
<span>from</span> sklearn.svm <span>import</span> LinearSVC

<span>from</span> sklearn.datasets <span>import</span> make_classification, make_blobs

<span>from</span> sklearn.metrics <span>import</span> confusion_matrix
<span>from</span> matplotlib.colors <span>import</span> ListedColormap
<span>from</span> sklearn.datasets <span>import</span> load_breast_cancer
<span># Load the cancer data</span>
(X_cancer, y_cancer) = load_breast_cancer(return_X_y = <span>True</span>)
X_train, X_test, y_train, y_test = train_test_split(X_cancer, y_cancer,
                                                   random_state = <span>0</span>)
clf = LinearSVC().fit(X_train, y_train)
<span>print</span>(<span>'Breast cancer dataset'</span>)
<span>print</span>(<span>'Accuracy of Linear SVC classifier on training set: {:.2f}'</span>
     .format(clf.score(X_train, y_train)))
<span>print</span>(<span>'Accuracy of Linear SVC classifier on test set: {:.2f}'</span>
     .format(clf.score(X_test, y_test)))</code></pre>
<pre><code>## Breast cancer dataset
## Accuracy of Linear SVC classifier on training set: 0.92
## Accuracy of Linear SVC classifier on test set: 0.94</code></pre>
</div>
<div>
<h2>1.2 Dummy classifier</h2>
<p>Often when we perform classification tasks using any ML model namely logistic regression, SVM, neural networks etc. it is very useful to determine how well the ML model performs agains at dummy classifier. A dummy classifier uses some simple computation like frequency of majority class, instead of fitting and ML model. It is essential that our ML model does much better that the dummy classifier. This problem is even more important in imbalanced classes where we have only about 10% of +ve samples. If any ML model we create has a accuracy of about 0.90 then it is evident that our classifier is not doing any better than a dummy classsfier which can just take a majority count of this imbalanced class and also come up with 0.90. We need to be able to do better than that.</p>
<p>In the examples below (1.3a &amp; 1.3b) it can be seen that SVMs with ‘radial basis’ kernel with unnormalized data, for both R and Python, do not perform any better than the dummy classifier.</p>
</div>
<div>
<h2>1.2a Dummy classifier – R code</h2>
<p>R does not seem to have an explicit dummy classifier. I created a simple dummy classifier that predicts the majority class. SKlearn in Python also includes other strategies like uniform, stratified etc. but this should be possible to create in R also.</p>
<pre><code><span># Create a simple dummy classifier that computes the ratio of the majority class to the totla</span>
<span>DummyClassifierAccuracy</span> <span>&lt;-</span> <span>function</span><span>(</span><span>train</span>,<span>test</span>,<span>type</span><span>=</span><span>&quot;majority&quot;</span><span>)</span><span>{</span>
  <span>if</span><span>(</span><span>type</span><span>==</span><span>&quot;majority&quot;</span><span>)</span><span>{</span>
      <span>count</span> <span>&lt;-</span> <span>sum</span><span>(</span><span>train</span><span>$</span><span>target</span><span>==</span><span>1</span><span>)</span><span>/</span><span>dim</span><span>(</span><span>train</span><span>)</span><span>[</span><span>1</span><span>]</span>
  <span>}</span>
  <span>count</span>
<span>}</span>


<span>cancer</span> <span>&lt;-</span> <span>read.csv</span><span>(</span><span>&quot;cancer.csv&quot;</span><span>)</span>
<span>cancer</span><span>$</span><span>target</span> <span>&lt;-</span> <span>as.factor</span><span>(</span><span>cancer</span><span>$</span><span>target</span><span>)</span>

<span># Create training and test sets</span>
<span>train_idx</span> <span>&lt;-</span> <span>trainTestSplit</span><span>(</span><span>cancer</span>,<span>trainPercent</span><span>=</span><span>75</span>,<span>seed</span><span>=</span><span>5</span><span>)</span>
<span>train</span> <span>&lt;-</span> <span>cancer</span><span>[</span><span>train_idx</span>, <span>]</span>
<span>test</span> <span>&lt;-</span> <span>cancer</span><span>[</span><span>-</span><span>train_idx</span>, <span>]</span>

<span>#Dummy classifier majority class</span>
<span>acc</span><span>=</span><span>DummyClassifierAccuracy</span><span>(</span><span>train</span>,<span>test</span><span>)</span>
<span>sprintf</span><span>(</span><span>&quot;Accuracy is %f&quot;</span>,<span>acc</span><span>)</span></code></pre>
<pre><code>## [1] &quot;Accuracy is 0.638498&quot;</code></pre>
</div>
<div>
<h2>1.2b Dummy classifier – Python code</h2>
<p>This dummy classifier uses the majority class.</p>
<pre><code><span>import</span> numpy <span>as</span> np
<span>import</span> pandas <span>as</span> pd
<span>import</span> os
<span>import</span> matplotlib.pyplot <span>as</span> plt
<span>from</span> sklearn.datasets <span>import</span> load_breast_cancer
<span>from</span> sklearn.model_selection <span>import</span> train_test_split
<span>from</span> sklearn.dummy <span>import</span> DummyClassifier
<span>from</span> sklearn.metrics <span>import</span> confusion_matrix
(X_cancer, y_cancer) = load_breast_cancer(return_X_y = <span>True</span>)
X_train, X_test, y_train, y_test = train_test_split(X_cancer, y_cancer,
                                                   random_state = <span>0</span>)

<span># Negative class (0) is most frequent</span>
dummy_majority = DummyClassifier(strategy = <span>'most_frequent'</span>).fit(X_train, y_train)
y_dummy_predictions = dummy_majority.predict(X_test)

<span>print</span>(<span>'Dummy classifier accuracy on test set: {:.2f}'</span>
     .format(dummy_majority.score(X_test, y_test)))
</code></pre>
<pre><code>## Dummy classifier accuracy on test set: 0.63</code></pre>
</div>
<div>
<h2>1.3a – Radial SVM (un-normalized) – R code</h2>
<p>SVMs perform better when the data is normalized or scaled. The 2 examples below show that SVM with radial basis kernel does not perform any better than the dummy classifier</p>
<pre><code><span>library</span><span>(</span><span>dplyr</span><span>)</span>
<span>library</span><span>(</span><span>e1071</span><span>)</span>
<span>library</span><span>(</span><span>caret</span><span>)</span>
<span>library</span><span>(</span><span>reshape2</span><span>)</span>
<span>library</span><span>(</span><span>ggplot2</span><span>)</span>

<span># Radial SVM unnormalized</span>
<span>train_idx</span> <span>&lt;-</span> <span>trainTestSplit</span><span>(</span><span>cancer</span>,<span>trainPercent</span><span>=</span><span>75</span>,<span>seed</span><span>=</span><span>5</span><span>)</span>
<span>train</span> <span>&lt;-</span> <span>cancer</span><span>[</span><span>train_idx</span>, <span>]</span>
<span>test</span> <span>&lt;-</span> <span>cancer</span><span>[</span><span>-</span><span>train_idx</span>, <span>]</span>
<span># Unnormalized data</span>
<span>svmfit</span><span>=</span><span>svm</span><span>(</span><span>target</span><span>~</span>., <span>data</span><span>=</span><span>train</span>, <span>kernel</span><span>=</span><span>&quot;radial&quot;</span>,<span>cost</span><span>=</span><span>10</span>,<span>scale</span><span>=</span><span>FALSE</span><span>)</span>
<span>ypred</span><span>=</span><span>predict</span><span>(</span><span>svmfit</span>,<span>test</span><span>)</span>
<span>confusionMatrix</span><span>(</span><span>ypred</span>,<span>test</span><span>$</span><span>target</span><span>)</span></code></pre>
<pre><code>## Confusion Matrix and Statistics
## 
##           Reference
## Prediction  0  1
##          0  0  0
##          1 57 85
##                                           
##                Accuracy : 0.5986          
##                  95% CI : (0.5131, 0.6799)
##     No Information Rate : 0.5986          
##     P-Value [Acc > NIR] : 0.5363          
##                                           
##                   Kappa : 0               
##  Mcnemar's Test P-Value : 1.195e-13       
##                                           
##             Sensitivity : 0.0000          
##             Specificity : 1.0000          
##          Pos Pred Value :    NaN          
##          Neg Pred Value : 0.5986          
##              Prevalence : 0.4014          
##          Detection Rate : 0.0000          
##    Detection Prevalence : 0.0000          
##       Balanced Accuracy : 0.5000          
##                                           
##        'Positive' Class : 0               
## </code></pre>
</div>
<div>
<h2>1.4b – Radial SVM (un-normalized) – Python code</h2>
<pre><code><span>import</span> numpy <span>as</span> np
<span>import</span> pandas <span>as</span> pd
<span>import</span> os
<span>import</span> matplotlib.pyplot <span>as</span> plt
<span>from</span> sklearn.datasets <span>import</span> load_breast_cancer
<span>from</span> sklearn.model_selection <span>import</span> train_test_split
<span>from</span> sklearn.svm <span>import</span> SVC

<span># Load the cancer data</span>
(X_cancer, y_cancer) = load_breast_cancer(return_X_y = <span>True</span>)
X_train, X_test, y_train, y_test = train_test_split(X_cancer, y_cancer,
                                                   random_state = <span>0</span>)


clf = SVC(C=<span>10</span>).fit(X_train, y_train)
<span>print</span>(<span>'Breast cancer dataset (unnormalized features)'</span>)
<span>print</span>(<span>'Accuracy of RBF-kernel SVC on training set: {:.2f}'</span>
     .format(clf.score(X_train, y_train)))
<span>print</span>(<span>'Accuracy of RBF-kernel SVC on test set: {:.2f}'</span>
     .format(clf.score(X_test, y_test)))</code></pre>
<pre><code>## Breast cancer dataset (unnormalized features)
## Accuracy of RBF-kernel SVC on training set: 1.00
## Accuracy of RBF-kernel SVC on test set: 0.63</code></pre>
</div>
<div>
<h2>1.5a – Radial SVM (Normalized) -R Code</h2>
<p>The data is scaled (normalized ) before using the SVM model. The SVM model has 2 paramaters a) C – Large C (less regularization), more regularization b) gamma – Small gamma has larger decision boundary with more misclassfication, and larger gamma has tighter decision boundary</p>
<p>The R code below computes the accuracy as the regularization paramater is changed</p>
<pre><code><span>trainingAccuracy</span> <span>&lt;-</span> <span>NULL</span>
<span>testAccuracy</span> <span>&lt;-</span> <span>NULL</span>
<span>C1</span> <span>&lt;-</span> <span>c</span><span>(</span><span>.01</span>,<span>.1</span>, <span>1</span>, <span>10</span>, <span>20</span><span>)</span>
<span>for</span><span>(</span><span>i</span> <span>in</span>  <span>C1</span><span>)</span><span>{</span>
  
    <span>svmfit</span><span>=</span><span>svm</span><span>(</span><span>target</span><span>~</span>., <span>data</span><span>=</span><span>train</span>, <span>kernel</span><span>=</span><span>&quot;radial&quot;</span>,<span>cost</span><span>=</span><span>i</span>,<span>scale</span><span>=</span><span>TRUE</span><span>)</span>
    <span>ypredTrain</span> <span>&lt;-</span><span>predict</span><span>(</span><span>svmfit</span>,<span>train</span><span>)</span>
    <span>ypredTest</span><span>=</span><span>predict</span><span>(</span><span>svmfit</span>,<span>test</span><span>)</span>
    <span>a</span> <span>&lt;-</span><span>confusionMatrix</span><span>(</span><span>ypredTrain</span>,<span>train</span><span>$</span><span>target</span><span>)</span>
    <span>b</span> <span>&lt;-</span><span>confusionMatrix</span><span>(</span><span>ypredTest</span>,<span>test</span><span>$</span><span>target</span><span>)</span>
    <span>trainingAccuracy</span> <span>&lt;-</span><span>c</span><span>(</span><span>trainingAccuracy</span>,<span>a</span><span>$</span><span>overall</span><span>[</span><span>1</span><span>]</span><span>)</span>
    <span>testAccuracy</span> <span>&lt;-</span><span>c</span><span>(</span><span>testAccuracy</span>,<span>b</span><span>$</span><span>overall</span><span>[</span><span>1</span><span>]</span><span>)</span>
    
<span>}</span>
<span>print</span><span>(</span><span>trainingAccuracy</span><span>)</span></code></pre>
<pre><code>##  Accuracy  Accuracy  Accuracy  Accuracy  Accuracy 
## 0.6384977 0.9671362 0.9906103 0.9976526 1.0000000</code></pre>
<pre><code><span>print</span><span>(</span><span>testAccuracy</span><span>)</span></code></pre>
<pre><code>##  Accuracy  Accuracy  Accuracy  Accuracy  Accuracy 
## 0.5985915 0.9507042 0.9647887 0.9507042 0.9507042</code></pre>
<pre><code><span>a</span> <span>&lt;-</span><span>rbind</span><span>(</span><span>C1</span>,<span>as.numeric</span><span>(</span><span>trainingAccuracy</span><span>)</span>,<span>as.numeric</span><span>(</span><span>testAccuracy</span><span>)</span><span>)</span>
<span>b</span> <span>&lt;-</span> <span>data.frame</span><span>(</span><span>t</span><span>(</span><span>a</span><span>)</span><span>)</span>
<span>names</span><span>(</span><span>b</span><span>)</span> <span>&lt;-</span> <span>c</span><span>(</span><span>&quot;C1&quot;</span>,<span>&quot;trainingAccuracy&quot;</span>,<span>&quot;testAccuracy&quot;</span><span>)</span>
<span>df</span> <span>&lt;-</span> <span>melt</span><span>(</span><span>b</span>,<span>id</span><span>=</span><span>&quot;C1&quot;</span><span>)</span>
<span>ggplot</span><span>(</span><span>df</span><span>)</span> <span>+</span> <span>geom_line</span><span>(</span><span>aes</span><span>(</span><span>x</span><span>=</span><span>C1</span>, <span>y</span><span>=</span><span>value</span>, <span>colour</span><span>=</span><span>variable</span><span>)</span>,<span>size</span><span>=</span><span>2</span><span>)</span> <span>+</span>
    <span>xlab</span><span>(</span><span>&quot;C (SVC regularization)value&quot;</span><span>)</span> <span>+</span> <span>ylab</span><span>(</span><span>&quot;Accuracy&quot;</span><span>)</span> <span>+</span>
    <span>ggtitle</span><span>(</span><span>&quot;Training and test accuracy vs C(regularization)&quot;</span><span>)</span></code></pre>
<p><img><img data-orig-file="https://gigadom.files.wordpress.com/2017/10/fig1-11.png" data-orig-size="1344,960" data-medium-file="https://gigadom.files.wordpress.com/2017/10/fig1-11.png?w=300" data-attachment-id="5728" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2017/10/fig1-11.png?w=450&h=483&fit=676%2C483" data-permalink="https://gigadom.wordpress.com/2017/10/29/practical-machine-learning-with-r-and-python-part-4/fig1-1-3/" data-large-file="https://gigadom.files.wordpress.com/2017/10/fig1-11.png?w=450&amp;h=483&amp;fit=676%2C483" alt="" data-image-title="fig1-1" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-5728" data-recalc-dims="1"></p>
</div>
<div>
<h2>1.5b – Radial SVM (normalized) – Python</h2>
<p>The Radial basis kernel is used on normalized data for a range of ‘C’ values and the result is plotted.</p>
<pre><code><span>import</span> numpy <span>as</span> np
<span>import</span> pandas <span>as</span> pd
<span>import</span> os
<span>import</span> matplotlib.pyplot <span>as</span> plt
<span>from</span> sklearn.datasets <span>import</span> load_breast_cancer
<span>from</span> sklearn.model_selection <span>import</span> train_test_split
<span>from</span> sklearn.svm <span>import</span> SVC
<span>from</span> sklearn.preprocessing <span>import</span> MinMaxScaler
scaler = MinMaxScaler()

<span># Load the cancer data</span>
(X_cancer, y_cancer) = load_breast_cancer(return_X_y = <span>True</span>)
X_train, X_test, y_train, y_test = train_test_split(X_cancer, y_cancer,
                                                   random_state = <span>0</span>)
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)
   
<span>print</span>(<span>'Breast cancer dataset (normalized with MinMax scaling)'</span>)
trainingAccuracy=[]
testAccuracy=[]
<span>for</span> C1 <span>in</span> [.<span>01</span>,.<span>1</span>, <span>1</span>, <span>10</span>, <span>20</span>]:
    clf = SVC(C=C1).fit(X_train_scaled, y_train)
    acctrain=clf.score(X_train_scaled, y_train)
    accTest=clf.score(X_test_scaled, y_test)
    trainingAccuracy.append(acctrain)
    testAccuracy.append(accTest)
    
<span># Create a dataframe</span>
C1=[.<span>01</span>,.<span>1</span>, <span>1</span>, <span>10</span>, <span>20</span>]   
trainingAccuracy=pd.DataFrame(trainingAccuracy,index=C1)
testAccuracy=pd.DataFrame(testAccuracy,index=C1)

<span># Plot training and test R squared as a function of alpha</span>
df=pd.concat([trainingAccuracy,testAccuracy],axis=<span>1</span>)
df.columns=[<span>'trainingAccuracy'</span>,<span>'trainingAccuracy'</span>]

fig1=df.plot()
fig1=plt.title(<span>'Training and test accuracy vs C (SVC)'</span>)
fig1.figure.savefig(<span>'fig1.png'</span>, bbox_inches=<span>'tight'</span>)</code></pre>
<pre><code>## Breast cancer dataset (normalized with MinMax scaling)</code></pre>
<p>Output image: <img></p>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2017/10/fig12.png?w=456" data-orig-size="572,435" data-medium-file="https://gigadom.files.wordpress.com/2017/10/fig12.png?w=456?w=300" data-attachment-id="5730" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2017/10/fig12.png?w=456" data-permalink="https://gigadom.wordpress.com/2017/10/29/practical-machine-learning-with-r-and-python-part-4/fig1-6/" data-large-file="https://gigadom.files.wordpress.com/2017/10/fig12.png?w=456?w=572" alt="" data-image-title="fig1" sizes="(max-width: 572px) 100vw, 572px" data-comments-opened="1" class="wp-image-5730" data-recalc-dims="1"></p>
</div>
<div>
<h2>1.6a Validation curve – R code</h2>
<p>Sklearn includes code creating validation curves by varying paramaters and computing and plotting accuracy as gamma or C or changd. I did not find this R but I think this is a useful function and so I have created the R equivalent of this.</p>
<pre><code><span># The R equivalent of np.logspace</span>
<span>seqLogSpace</span> <span>&lt;-</span> <span>function</span><span>(</span><span>start</span>,<span>stop</span>,<span>len</span><span>)</span><span>{</span>
  <span>a</span><span>=</span><span>seq</span><span>(</span><span>log10</span><span>(</span><span>10</span><span>^</span><span>start</span><span>)</span>,<span>log10</span><span>(</span><span>10</span><span>^</span><span>stop</span><span>)</span>,<span>length</span><span>=</span><span>len</span><span>)</span>
  <span>10</span><span>^</span><span>a</span>
<span>}</span>

<span># Read the data. This is taken the SKlearn cancer data</span>
<span>cancer</span> <span>&lt;-</span> <span>read.csv</span><span>(</span><span>&quot;cancer.csv&quot;</span><span>)</span>
<span>cancer</span><span>$</span><span>target</span> <span>&lt;-</span> <span>as.factor</span><span>(</span><span>cancer</span><span>$</span><span>target</span><span>)</span>

<span>set.seed</span><span>(</span><span>6</span><span>)</span>

<span># Create the range of C1 in log space</span>
<span>param_range</span> <span>=</span> <span>seqLogSpace</span><span>(</span><span>-</span><span>3</span>,<span>2</span>,<span>20</span><span>)</span>
<span># Initialize the overall training and test accuracy to NULL</span>
<span>overallTrainAccuracy</span> <span>&lt;-</span> <span>NULL</span>
<span>overallTestAccuracy</span> <span>&lt;-</span> <span>NULL</span>

<span># Loop over the parameter range of Gamma</span>
<span>for</span><span>(</span><span>i</span> <span>in</span> <span>param_range</span><span>)</span><span>{</span>
    <span># Set no of folds</span>
    <span>noFolds</span><span>=</span><span>5</span>
    <span># Create the rows which fall into different folds from 1..noFolds</span>
    <span>folds</span> <span>=</span> <span>sample</span><span>(</span><span>1</span><span>:</span><span>noFolds</span>, <span>nrow</span><span>(</span><span>cancer</span><span>)</span>, <span>replace</span><span>=</span><span>TRUE</span><span>)</span> 
    <span># Initialize the training and test accuracy of folds to 0</span>
    <span>trainingAccuracy</span> <span>&lt;-</span> <span>0</span>
    <span>testAccuracy</span> <span>&lt;-</span> <span>0</span>
    
    <span># Loop through the folds</span>
    <span>for</span><span>(</span><span>j</span> <span>in</span> <span>1</span><span>:</span><span>noFolds</span><span>)</span><span>{</span>
        <span># The training is all rows for which the row is != j (k-1 folds -> training)</span>
        <span>train</span> <span>&lt;-</span> <span>cancer</span><span>[</span><span>folds</span><span>!=</span><span>j</span>,<span>]</span>
        <span># The rows which have j as the index become the test set</span>
        <span>test</span> <span>&lt;-</span> <span>cancer</span><span>[</span><span>folds</span><span>==</span><span>j</span>,<span>]</span>
        <span># Create a SVM model for this</span>
        <span>svmfit</span><span>=</span><span>svm</span><span>(</span><span>target</span><span>~</span>., <span>data</span><span>=</span><span>train</span>, <span>kernel</span><span>=</span><span>&quot;radial&quot;</span>,<span>gamma</span><span>=</span><span>i</span>,<span>scale</span><span>=</span><span>TRUE</span><span>)</span>
  
        <span># Add up all the fold accuracy for training and test separately </span>
        <span>ypredTrain</span> <span>&lt;-</span><span>predict</span><span>(</span><span>svmfit</span>,<span>train</span><span>)</span>
        <span>ypredTest</span><span>=</span><span>predict</span><span>(</span><span>svmfit</span>,<span>test</span><span>)</span>
        
        <span># Create confusion matrix </span>
        <span>a</span> <span>&lt;-</span><span>confusionMatrix</span><span>(</span><span>ypredTrain</span>,<span>train</span><span>$</span><span>target</span><span>)</span>
        <span>b</span> <span>&lt;-</span><span>confusionMatrix</span><span>(</span><span>ypredTest</span>,<span>test</span><span>$</span><span>target</span><span>)</span>
        <span># Get the accuracy</span>
        <span>trainingAccuracy</span> <span>&lt;-</span><span>trainingAccuracy</span> <span>+</span> <span>a</span><span>$</span><span>overall</span><span>[</span><span>1</span><span>]</span>
        <span>testAccuracy</span> <span>&lt;-</span><span>testAccuracy</span><span>+</span><span>b</span><span>$</span><span>overall</span><span>[</span><span>1</span><span>]</span>

    <span>}</span>
    <span># Compute the average of accuracy for K folds for number of features 'i'</span>
    <span>overallTrainAccuracy</span><span>=</span><span>c</span><span>(</span><span>overallTrainAccuracy</span>,<span>trainingAccuracy</span><span>/</span><span>noFolds</span><span>)</span>
    <span>overallTestAccuracy</span><span>=</span><span>c</span><span>(</span><span>overallTestAccuracy</span>,<span>testAccuracy</span><span>/</span><span>noFolds</span><span>)</span>
<span>}</span>
<span>#Create a dataframe</span>
<span>a</span> <span>&lt;-</span> <span>rbind</span><span>(</span><span>param_range</span>,<span>as.numeric</span><span>(</span><span>overallTrainAccuracy</span><span>)</span>,
               <span>as.numeric</span><span>(</span><span>overallTestAccuracy</span><span>)</span><span>)</span>
<span>b</span> <span>&lt;-</span> <span>data.frame</span><span>(</span><span>t</span><span>(</span><span>a</span><span>)</span><span>)</span>
<span>names</span><span>(</span><span>b</span><span>)</span> <span>&lt;-</span> <span>c</span><span>(</span><span>&quot;C1&quot;</span>,<span>&quot;trainingAccuracy&quot;</span>,<span>&quot;testAccuracy&quot;</span><span>)</span>
<span>df</span> <span>&lt;-</span> <span>melt</span><span>(</span><span>b</span>,<span>id</span><span>=</span><span>&quot;C1&quot;</span><span>)</span>
<span>#Plot in log axis</span>
<span>ggplot</span><span>(</span><span>df</span><span>)</span> <span>+</span> <span>geom_line</span><span>(</span><span>aes</span><span>(</span><span>x</span><span>=</span><span>C1</span>, <span>y</span><span>=</span><span>value</span>, <span>colour</span><span>=</span><span>variable</span><span>)</span>,<span>size</span><span>=</span><span>2</span><span>)</span> <span>+</span>
      <span>xlab</span><span>(</span><span>&quot;C (SVC regularization)value&quot;</span><span>)</span> <span>+</span> <span>ylab</span><span>(</span><span>&quot;Accuracy&quot;</span><span>)</span> <span>+</span>
      <span>ggtitle</span><span>(</span><span>&quot;Training and test accuracy vs C(regularization)&quot;</span><span>)</span> <span>+</span> <span>scale_x_log10</span><span>(</span><span>)</span></code></pre>
<p><img><img data-orig-file="https://gigadom.files.wordpress.com/2017/10/fig2-11.png" data-orig-size="1344,960" data-medium-file="https://gigadom.files.wordpress.com/2017/10/fig2-11.png?w=300" data-attachment-id="5731" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2017/10/fig2-11.png?w=450&h=483&fit=676%2C483" data-permalink="https://gigadom.wordpress.com/2017/10/29/practical-machine-learning-with-r-and-python-part-4/fig2-1-3/" data-large-file="https://gigadom.files.wordpress.com/2017/10/fig2-11.png?w=450&amp;h=483&amp;fit=676%2C483" alt="" data-image-title="fig2-1" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-5731" data-recalc-dims="1"></p>
</div>
<div>
<h2>1.6b Validation curve – Python</h2>
<p>Compute and plot the validation curve as gamma is varied.</p>
<pre><code><span>import</span> numpy <span>as</span> np
<span>import</span> pandas <span>as</span> pd
<span>import</span> os
<span>import</span> matplotlib.pyplot <span>as</span> plt
<span>from</span> sklearn.datasets <span>import</span> load_breast_cancer
<span>from</span> sklearn.model_selection <span>import</span> train_test_split
<span>from</span> sklearn.preprocessing <span>import</span> MinMaxScaler
<span>from</span> sklearn.svm <span>import</span> SVC
<span>from</span> sklearn.model_selection <span>import</span> validation_curve


<span># Load the cancer data</span>
(X_cancer, y_cancer) = load_breast_cancer(return_X_y = <span>True</span>)
scaler = MinMaxScaler()
X_scaled = scaler.fit_transform(X_cancer)

<span># Create a gamma values from 10^-3 to 10^2 with 20 equally spaced intervals</span>
param_range = np.logspace(-<span>3</span>, <span>2</span>, <span>20</span>)
<span># Compute the validation curve</span>
train_scores, test_scores = validation_curve(SVC(), X_scaled, y_cancer,
                                            param_name=<span>'gamma'</span>,
                                            param_range=param_range, cv=<span>10</span>)
                                            
<span>#Plot the figure </span>
fig2=plt.figure()

<span>#Compute the mean</span>
train_scores_mean = np.mean(train_scores, axis=<span>1</span>)
train_scores_std = np.std(train_scores, axis=<span>1</span>)
test_scores_mean = np.mean(test_scores, axis=<span>1</span>)
test_scores_std = np.std(test_scores, axis=<span>1</span>)

fig2=plt.title(<span>'Validation Curve with SVM'</span>)
fig2=plt.xlabel(<span>'$\gamma$ (gamma)'</span>)
fig2=plt.ylabel(<span>'Score'</span>)
fig2=plt.ylim(<span>0.0</span>, <span>1.1</span>)
lw = <span>2</span>

fig2=plt.semilogx(param_range, train_scores_mean, label=<span>'Training score'</span>,
            color=<span>'darkorange'</span>, lw=lw)

fig2=plt.fill_between(param_range, train_scores_mean - train_scores_std,
                train_scores_mean + train_scores_std, alpha=<span>0.2</span>,
                color=<span>'darkorange'</span>, lw=lw)

fig2=plt.semilogx(param_range, test_scores_mean, label=<span>'Cross-validation score'</span>,
            color=<span>'navy'</span>, lw=lw)

fig2=plt.fill_between(param_range, test_scores_mean - test_scores_std,
                test_scores_mean + test_scores_std, alpha=<span>0.2</span>,
                color=<span>'navy'</span>, lw=lw)
fig2.figure.savefig(<span>'fig2.png'</span>, bbox_inches=<span>'tight'</span>)
</code></pre>
<p>Output image: <img></p>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2017/10/fig21.png?w=456" data-orig-size="570,463" data-medium-file="https://gigadom.files.wordpress.com/2017/10/fig21.png?w=456?w=300" data-attachment-id="5732" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2017/10/fig21.png?w=456" data-permalink="https://gigadom.wordpress.com/2017/10/29/practical-machine-learning-with-r-and-python-part-4/fig2-4/" data-large-file="https://gigadom.files.wordpress.com/2017/10/fig21.png?w=456?w=570" alt="" data-image-title="fig2" sizes="(max-width: 570px) 100vw, 570px" data-comments-opened="1" class="wp-image-5732" data-recalc-dims="1"></p>
</div>
<div>
<h2>1.7a Validation Curve (Preventing data leakage) – Python code</h2>
<p>In this course <a rel="nofollow" href="https://www.coursera.org/learn/python-machine-learning/" target="_blank">Applied Machine Learning in Python</a>, the Professor states that when we apply the same data transformation to a entire dataset, it will cause a data leakage. “The proper way to do cross-validation when you need to scale the data is not to scale the entire dataset with a single transform, since this will indirectly leak information into the training data about the whole dataset, including the test data (see the lecture on data leakage later in the course). Instead, scaling/normalizing must be computed and applied for each cross-validation fold separately”</p>
<p>So I apply separate scaling to the training and testing folds and plot. In the lecture the Prof states that this can be done using pipelines.</p>
<pre><code><span>import</span> numpy <span>as</span> np
<span>import</span> pandas <span>as</span> pd
<span>import</span> os
<span>import</span> matplotlib.pyplot <span>as</span> plt
<span>from</span> sklearn.datasets <span>import</span> load_breast_cancer
<span>from</span> sklearn.cross_validation <span>import</span>  KFold
<span>from</span> sklearn.preprocessing <span>import</span> MinMaxScaler
<span>from</span> sklearn.svm <span>import</span> SVC

<span># Read the data</span>
(X_cancer, y_cancer) = load_breast_cancer(return_X_y = <span>True</span>)
<span># Set the parameter range</span>
param_range = np.logspace(-<span>3</span>, <span>2</span>, <span>20</span>)

<span># Set number of folds</span>
folds=<span>5</span>
<span>#Initialize</span>
overallTrainAccuracy=[]
overallTestAccuracy=[]

<span># Loop over the paramater range</span>
<span>for</span> c <span>in</span>  param_range:
    trainingAccuracy=<span>0</span>
    testAccuracy=<span>0</span>
    kf = KFold(len(X_cancer),n_folds=folds)
    <span># Partition into training and test folds</span>
    <span>for</span> train_index, test_index <span>in</span> kf:
            <span># Partition the data acccording the fold indices generated</span>
            X_train, X_test = X_cancer[train_index], X_cancer[test_index]
            y_train, y_test = y_cancer[train_index], y_cancer[test_index]  

            
            <span># Scale the X_train and X_test </span>
            scaler = MinMaxScaler()
            X_train_scaled = scaler.fit_transform(X_train)
            X_test_scaled = scaler.transform(X_test)
            <span># Fit a SVC model for each C</span>
            clf = SVC(C=c).fit(X_train_scaled, y_train)
            <span>#Compute the training and test score</span>
            acctrain=clf.score(X_train_scaled, y_train)
            accTest=clf.score(X_test_scaled, y_test)
            trainingAccuracy += np.sum(acctrain)
            testAccuracy += np.sum(accTest)
    <span># Compute the mean training and testing accuracy</span>
    overallTrainAccuracy.append(trainingAccuracy/folds)
    overallTestAccuracy.append(testAccuracy/folds)
        

overallTrainAccuracy=pd.DataFrame(overallTrainAccuracy,index=param_range)
overallTestAccuracy=pd.DataFrame(overallTestAccuracy,index=param_range)

<span># Plot training and test R squared as a function of alpha</span>
df=pd.concat([overallTrainAccuracy,overallTestAccuracy],axis=<span>1</span>)
df.columns=[<span>'trainingAccuracy'</span>,<span>'testAccuracy'</span>]


fig3=plt.title(<span>'Validation Curve with SVM'</span>)
fig3=plt.xlabel(<span>'$\gamma$ (gamma)'</span>)
fig3=plt.ylabel(<span>'Score'</span>)
fig3=plt.ylim(<span>0.5</span>, <span>1.1</span>)
lw = <span>2</span>

fig3=plt.semilogx(param_range, overallTrainAccuracy, label=<span>'Training score'</span>,
            color=<span>'darkorange'</span>, lw=lw)

fig3=plt.semilogx(param_range, overallTestAccuracy, label=<span>'Cross-validation score'</span>,
            color=<span>'navy'</span>, lw=lw)

fig3=plt.legend(loc=<span>'best'</span>)
fig3.figure.savefig(<span>'fig3.png'</span>, bbox_inches=<span>'tight'</span>)
</code></pre>
<p>Output image: <img></p>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2017/10/fig31.png?w=456" data-orig-size="570,463" data-medium-file="https://gigadom.files.wordpress.com/2017/10/fig31.png?w=456?w=300" data-attachment-id="5733" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2017/10/fig31.png?w=456" data-permalink="https://gigadom.wordpress.com/2017/10/29/practical-machine-learning-with-r-and-python-part-4/fig3-3/" data-large-file="https://gigadom.files.wordpress.com/2017/10/fig31.png?w=456?w=570" alt="" data-image-title="fig3" sizes="(max-width: 570px) 100vw, 570px" data-comments-opened="1" class="wp-image-5733" data-recalc-dims="1"></p>
</div>
<div>
<h2>1.8 a Decision trees – R code</h2>
<p>Decision trees in R can be plotted using RPart package</p>
<pre><code><span>library</span><span>(</span><span>rpart</span><span>)</span>
<span>library</span><span>(</span><span>rpart.plot</span><span>)</span></code></pre>
<pre><code><span>rpart</span> <span>=</span> <span>NULL</span>
<span># Create a decision tree</span>
<span>m</span> <span>&lt;-</span><span>rpart</span><span>(</span><span>Species</span><span>~</span>.,<span>data</span><span>=</span><span>iris</span><span>)</span>
<span>#Plot</span>
<span>rpart.plot</span><span>(</span><span>m</span>,<span>extra</span><span>=</span><span>2</span>,<span>main</span><span>=</span><span>&quot;Decision Tree - IRIS&quot;</span><span>)</span></code></pre>
<p><img> <img data-orig-file="https://gigadom.files.wordpress.com/2017/10/fig3-11.png" data-orig-size="1344,960" data-medium-file="https://gigadom.files.wordpress.com/2017/10/fig3-11.png?w=300" data-attachment-id="5734" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2017/10/fig3-11.png?w=450&h=483&fit=676%2C483" data-permalink="https://gigadom.wordpress.com/2017/10/29/practical-machine-learning-with-r-and-python-part-4/fig3-1-3/" data-large-file="https://gigadom.files.wordpress.com/2017/10/fig3-11.png?w=450&amp;h=483&amp;fit=676%2C483" alt="" data-image-title="fig3-1" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-5734" data-recalc-dims="1"></p>
<h2>1.8 b Decision trees – Python code</h2>
<pre><code><span>from</span> sklearn.datasets <span>import</span> load_iris
<span>from</span> sklearn.tree <span>import</span> DecisionTreeClassifier
<span>from</span> sklearn <span>import</span> tree
<span>from</span> sklearn.model_selection <span>import</span> train_test_split
<span>import</span> graphviz 

iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(iris.data, iris.target, random_state = <span>3</span>)
clf = DecisionTreeClassifier().fit(X_train, y_train)

<span>print</span>(<span>'Accuracy of Decision Tree classifier on training set: {:.2f}'</span>
     .format(clf.score(X_train, y_train)))
<span>print</span>(<span>'Accuracy of Decision Tree classifier on test set: {:.2f}'</span>
     .format(clf.score(X_test, y_test)))

dot_data = tree.export_graphviz(clf, out_file=<span>None</span>, 
                         feature_names=iris.feature_names,  
                         class_names=iris.target_names,  
                         filled=<span>True</span>, rounded=<span>True</span>,  
                         special_characters=<span>True</span>)  
graph = graphviz.Source(dot_data)  
graph</code></pre>
<pre><code>## Accuracy of Decision Tree classifier on training set: 1.00
## Accuracy of Decision Tree classifier on test set: 0.97</code></pre>
</div>
<div>
<h2><img data-orig-file="https://gigadom.files.wordpress.com/2017/10/untitled1.png" data-orig-size="699,514" data-medium-file="https://gigadom.files.wordpress.com/2017/10/untitled1.png?w=300" data-attachment-id="5735" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2017/10/untitled1.png?w=450&h=497&fit=676%2C497" data-permalink="https://gigadom.wordpress.com/2017/10/29/practical-machine-learning-with-r-and-python-part-4/untitled1-4/" data-large-file="https://gigadom.files.wordpress.com/2017/10/untitled1.png?w=450&amp;h=497&amp;fit=676%2C497" alt="" data-image-title="Untitled1" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-5735" data-recalc-dims="1"></h2>
<h2>1.9a Feature importance – R code</h2>
<p>I found the following <a rel="nofollow" href="https://machinelearningmastery.com/feature-selection-with-the-caret-r-package/" target="_blank">code</a> which had a snippet for feature importance. Sklean has a nice method for this. For some reason the results in R and Python are different. Any thoughts?</p>
<pre><code><span>set.seed</span><span>(</span><span>3</span><span>)</span>
<span># load the library</span>
<span>library</span><span>(</span><span>mlbench</span><span>)</span></code></pre>
<pre><code><span>library</span><span>(</span><span>caret</span><span>)</span>
<span># load the dataset</span>
<span>cancer</span> <span>&lt;-</span> <span>read.csv</span><span>(</span><span>&quot;cancer.csv&quot;</span><span>)</span>
<span>cancer</span><span>$</span><span>target</span> <span>&lt;-</span> <span>as.factor</span><span>(</span><span>cancer</span><span>$</span><span>target</span><span>)</span>
<span># Split as data</span>
<span>data</span> <span>&lt;-</span> <span>cancer</span><span>[</span>,<span>1</span><span>:</span><span>31</span><span>]</span>
<span>target</span> <span>&lt;-</span> <span>cancer</span><span>[</span>,<span>32</span><span>]</span>

<span># Train the model</span>
<span>model</span> <span>&lt;-</span> <span>train</span><span>(</span><span>data</span>, <span>target</span>, <span>method</span><span>=</span><span>&quot;rf&quot;</span>, <span>preProcess</span><span>=</span><span>&quot;scale&quot;</span>, <span>trControl</span><span>=</span><span>trainControl</span><span>(</span><span>method</span> <span>=</span> <span>&quot;cv&quot;</span><span>)</span><span>)</span></code></pre>
<pre><code><span># Compute variable importance</span>
<span>importance</span> <span>&lt;-</span> <span>varImp</span><span>(</span><span>model</span><span>)</span>
<span># summarize importance</span>
<span>print</span><span>(</span><span>importance</span><span>)</span></code></pre>
<pre><code><span># plot importance</span>
<span>plot</span><span>(</span><span>importance</span><span>)</span></code></pre>
<p><img><img data-orig-file="https://gigadom.files.wordpress.com/2017/10/fig4-11.png" data-orig-size="1344,960" data-medium-file="https://gigadom.files.wordpress.com/2017/10/fig4-11.png?w=300" data-attachment-id="5736" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2017/10/fig4-11.png?w=450&h=483&fit=676%2C483" data-permalink="https://gigadom.wordpress.com/2017/10/29/practical-machine-learning-with-r-and-python-part-4/fig4-1-3/" data-large-file="https://gigadom.files.wordpress.com/2017/10/fig4-11.png?w=450&amp;h=483&amp;fit=676%2C483" alt="" data-image-title="fig4-1" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-5736" data-recalc-dims="1"></p>
</div>
<div>
<h2>1.9b Feature importance – Python code</h2>
<pre><code><span>import</span> numpy <span>as</span> np
<span>import</span> pandas <span>as</span> pd
<span>import</span> os
<span>import</span> matplotlib.pyplot <span>as</span> plt
<span>from</span> sklearn.tree <span>import</span> DecisionTreeClassifier
<span>from</span> sklearn.model_selection <span>import</span> train_test_split
<span>from</span> sklearn.datasets <span>import</span> load_breast_cancer
<span>import</span> numpy <span>as</span> np
<span># Read the data</span>
cancer= load_breast_cancer()
(X_cancer, y_cancer) = load_breast_cancer(return_X_y = <span>True</span>)
X_train, X_test, y_train, y_test = train_test_split(X_cancer, y_cancer, random_state = <span>0</span>)
<span># Use the DecisionTreClassifier</span>
clf = DecisionTreeClassifier(max_depth = <span>4</span>, min_samples_leaf = <span>8</span>,
                            random_state = <span>0</span>).fit(X_train, y_train)

c_features=len(cancer.feature_names)
<span>print</span>(<span>'Breast cancer dataset: decision tree'</span>)
<span>print</span>(<span>'Accuracy of DT classifier on training set: {:.2f}'</span>
     .format(clf.score(X_train, y_train)))
<span>print</span>(<span>'Accuracy of DT classifier on test set: {:.2f}'</span>
     .format(clf.score(X_test, y_test)))

<span># Plot the feature importances</span>
fig4=plt.figure(figsize=(<span>10</span>,<span>6</span>),dpi=<span>80</span>)

fig4=plt.barh(range(c_features), clf.feature_importances_)
fig4=plt.xlabel(<span>&quot;Feature importance&quot;</span>)
fig4=plt.ylabel(<span>&quot;Feature name&quot;</span>)
fig4=plt.yticks(np.arange(c_features), cancer.feature_names)
fig4=plt.tight_layout()
plt.savefig(<span>'fig4.png'</span>, bbox_inches=<span>'tight'</span>)
</code></pre>
<pre><code>## Breast cancer dataset: decision tree
## Accuracy of DT classifier on training set: 0.96
## Accuracy of DT classifier on test set: 0.94</code></pre>
<p>Output image: <img></p>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2017/10/fig41.png" data-orig-size="792,472" data-medium-file="https://gigadom.files.wordpress.com/2017/10/fig41.png?w=300" data-attachment-id="5737" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2017/10/fig41.png?w=450&h=403&fit=676%2C403" data-permalink="https://gigadom.wordpress.com/2017/10/29/practical-machine-learning-with-r-and-python-part-4/fig4-3/" data-large-file="https://gigadom.files.wordpress.com/2017/10/fig41.png?w=450&amp;h=403&amp;fit=676%2C403" alt="" data-image-title="fig4" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-5737" data-recalc-dims="1"></p>
</div>
<div>
<h2>1.10a Precision-Recall, ROC curves &amp; AUC- R code</h2>
<p>I tried several R packages for plotting the Precision and Recall and AUC curve. PRROC seems to work well. The Precision-Recall curves show the tradeoff between precision and recall. The higher the precision, the lower the recall and vice versa.AUC curves that hug the top left corner indicate a high sensitivity,specificity and an excellent accuracy.</p>
<pre><code><span>source</span><span>(</span><span>&quot;RFunctions-1.R&quot;</span><span>)</span>
<span>library</span><span>(</span><span>dplyr</span><span>)</span>
<span>library</span><span>(</span><span>caret</span><span>)</span>
<span>library</span><span>(</span><span>e1071</span><span>)</span>
<span>library</span><span>(</span><span>PRROC</span><span>)</span></code></pre>
<pre><code><span># Read the data (this data is from sklearn!)</span>
<span>d</span> <span>&lt;-</span> <span>read.csv</span><span>(</span><span>&quot;digits.csv&quot;</span><span>)</span>
<span>digits</span> <span>&lt;-</span> <span>d</span><span>[</span><span>2</span><span>:</span><span>66</span><span>]</span>
<span>digits</span><span>$</span><span>X64</span> <span>&lt;-</span> <span>as.factor</span><span>(</span><span>digits</span><span>$</span><span>X64</span><span>)</span>

<span># Split as training and test sets</span>
<span>train_idx</span> <span>&lt;-</span> <span>trainTestSplit</span><span>(</span><span>digits</span>,<span>trainPercent</span><span>=</span><span>75</span>,<span>seed</span><span>=</span><span>5</span><span>)</span>
<span>train</span> <span>&lt;-</span> <span>digits</span><span>[</span><span>train_idx</span>, <span>]</span>
<span>test</span> <span>&lt;-</span> <span>digits</span><span>[</span><span>-</span><span>train_idx</span>, <span>]</span>

<span># Fit a SVM model with linear basis kernel with probabilities</span>
<span>svmfit</span><span>=</span><span>svm</span><span>(</span><span>X64</span><span>~</span>., <span>data</span><span>=</span><span>train</span>, <span>kernel</span><span>=</span><span>&quot;linear&quot;</span>,<span>scale</span><span>=</span><span>FALSE</span>,<span>probability</span><span>=</span><span>TRUE</span><span>)</span>
<span>ypred</span><span>=</span><span>predict</span><span>(</span><span>svmfit</span>,<span>test</span>,<span>probability</span><span>=</span><span>TRUE</span><span>)</span>
<span>head</span><span>(</span><span>attr</span><span>(</span><span>ypred</span>,<span>&quot;probabilities&quot;</span><span>)</span><span>)</span></code></pre>
<pre><code>##               0            1
## 6  7.395947e-01 2.604053e-01
## 8  9.999998e-01 1.842555e-07
## 12 1.655178e-05 9.999834e-01
## 13 9.649997e-01 3.500032e-02
## 15 9.994849e-01 5.150612e-04
## 16 9.999987e-01 1.280700e-06</code></pre>
<pre><code><span># Store the probability of 0s and 1s</span>
<span>m0</span><span>&lt;-</span><span>attr</span><span>(</span><span>ypred</span>,<span>&quot;probabilities&quot;</span><span>)</span><span>[</span>,<span>1</span><span>]</span>
<span>m1</span><span>&lt;-</span><span>attr</span><span>(</span><span>ypred</span>,<span>&quot;probabilities&quot;</span><span>)</span><span>[</span>,<span>2</span><span>]</span>

<span># Create a dataframe of scores</span>
<span>scores</span> <span>&lt;-</span> <span>data.frame</span><span>(</span><span>m1</span>,<span>test</span><span>$</span><span>X64</span><span>)</span>

<span># Class 0 is data points of +ve class (in this case, digit 1) and -ve class (digit 0)</span>
<span>#Compute Precision Recall</span>
<span>pr</span> <span>&lt;-</span> <span>pr.curve</span><span>(</span><span>scores.class0</span><span>=</span><span>scores</span><span>[</span><span>scores</span><span>$</span><span>test.X64</span><span>==</span><span>&quot;1&quot;</span>,<span>]</span><span>$</span><span>m1</span>,
               <span>scores.class1</span><span>=</span><span>scores</span><span>[</span><span>scores</span><span>$</span><span>test.X64</span><span>==</span><span>&quot;0&quot;</span>,<span>]</span><span>$</span><span>m1</span>,
               <span>curve</span><span>=</span><span>T</span><span>)</span>

<span># Plot precision-recall curve</span>
<span>plot</span><span>(</span><span>pr</span><span>)</span></code></pre>
<p><img><img data-orig-file="https://gigadom.files.wordpress.com/2017/10/fig5-1.png" data-orig-size="1344,960" data-medium-file="https://gigadom.files.wordpress.com/2017/10/fig5-1.png?w=300" data-attachment-id="5738" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2017/10/fig5-1.png?w=450&h=483&fit=676%2C483" data-permalink="https://gigadom.wordpress.com/2017/10/29/practical-machine-learning-with-r-and-python-part-4/fig5-1-2/" data-large-file="https://gigadom.files.wordpress.com/2017/10/fig5-1.png?w=450&amp;h=483&amp;fit=676%2C483" alt="" data-image-title="fig5-1" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-5738" data-recalc-dims="1"></p>
<pre><code><span>#Plot the ROC curve</span>
<span>roc</span><span>&lt;-</span><span>roc.curve</span><span>(</span><span>m0</span>, <span>m1</span>,<span>curve</span><span>=</span><span>TRUE</span><span>)</span>
<span>plot</span><span>(</span><span>roc</span><span>)</span></code></pre>
<p><img><img data-orig-file="https://gigadom.files.wordpress.com/2017/10/fig5-2.png" data-orig-size="1344,960" data-medium-file="https://gigadom.files.wordpress.com/2017/10/fig5-2.png?w=300" data-attachment-id="5739" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2017/10/fig5-2.png?w=450&h=483&fit=676%2C483" data-permalink="https://gigadom.wordpress.com/2017/10/29/practical-machine-learning-with-r-and-python-part-4/fig5-2-2/" data-large-file="https://gigadom.files.wordpress.com/2017/10/fig5-2.png?w=450&amp;h=483&amp;fit=676%2C483" alt="" data-image-title="fig5-2" sizes="(max-width: 676px) 100vw, 676px" data-comments-opened="1" class="wp-image-5739" data-recalc-dims="1"></p>
</div>
<div>
<h2>1.10b Precision-Recall, ROC curves &amp; AUC- Python code</h2>
<p>For Python Logistic Regression is used to plot Precision Recall, ROC curve and compute AUC</p>
<pre><code><span>import</span> numpy <span>as</span> np
<span>import</span> pandas <span>as</span> pd
<span>import</span> matplotlib.pyplot <span>as</span> plt
<span>from</span> sklearn.model_selection <span>import</span> train_test_split
<span>from</span> sklearn.linear_model <span>import</span> LogisticRegression
<span>from</span> sklearn.datasets <span>import</span> load_digits
<span>from</span> sklearn.metrics <span>import</span> precision_recall_curve
<span>from</span> sklearn.metrics <span>import</span> roc_curve, auc
<span>#Load the digits</span>
dataset = load_digits()
X, y = dataset.data, dataset.target
<span>#Create 2 classes -i) Digit 1 (from digit 1) ii) Digit 0 (from all other digits)</span>
<span># Make a copy of the target</span>
z= y.copy()
<span># Replace all non 1's as 0</span>
z[z != <span>1</span>] = <span>0</span>

X_train, X_test, y_train, y_test = train_test_split(X, z, random_state=<span>0</span>)
<span># Fit a LR model</span>
lr = LogisticRegression().fit(X_train, y_train)

<span>#Compute the decision scores</span>
y_scores_lr = lr.fit(X_train, y_train).decision_function(X_test)
y_score_list = list(zip(y_test[<span>0</span>:<span>20</span>], y_scores_lr[<span>0</span>:<span>20</span>]))

<span>#Show the decision_function scores for first 20 instances</span>
y_score_list

precision, recall, thresholds = precision_recall_curve(y_test, y_scores_lr)
closest_zero = np.argmin(np.abs(thresholds))
closest_zero_p = precision[closest_zero]
closest_zero_r = recall[closest_zero]
<span>#Plot</span>
plt.figure()
plt.xlim([<span>0.0</span>, <span>1.01</span>])
plt.ylim([<span>0.0</span>, <span>1.01</span>])
plt.plot(precision, recall, label=<span>'Precision-Recall Curve'</span>)
plt.plot(closest_zero_p, closest_zero_r, <span>'o'</span>, markersize = <span>12</span>, fillstyle = <span>'none'</span>, c=<span>'r'</span>, mew=<span>3</span>)
plt.xlabel(<span>'Precision'</span>, fontsize=<span>16</span>)
plt.ylabel(<span>'Recall'</span>, fontsize=<span>16</span>)
plt.axes().set_aspect(<span>'equal'</span>)
plt.savefig(<span>'fig5.png'</span>, bbox_inches=<span>'tight'</span>)

<span>#Compute and plot the ROC</span>
y_score_lr = lr.fit(X_train, y_train).decision_function(X_test)
fpr_lr, tpr_lr, _ = roc_curve(y_test, y_score_lr)
roc_auc_lr = auc(fpr_lr, tpr_lr)

plt.figure()
plt.xlim([-<span>0.01</span>, <span>1.00</span>])
plt.ylim([-<span>0.01</span>, <span>1.01</span>])
plt.plot(fpr_lr, tpr_lr, lw=<span>3</span>, label=<span>'LogRegr ROC curve (area = {:0.2f})'</span>.format(roc_auc_lr))
plt.xlabel(<span>'False Positive Rate'</span>, fontsize=<span>16</span>)
plt.ylabel(<span>'True Positive Rate'</span>, fontsize=<span>16</span>)
plt.title(<span>'ROC curve (1-of-10 digits classifier)'</span>, fontsize=<span>16</span>)
plt.legend(loc=<span>'lower right'</span>, fontsize=<span>13</span>)
plt.plot([<span>0</span>, <span>1</span>], [<span>0</span>, <span>1</span>], color=<span>'navy'</span>, lw=<span>3</span>, linestyle=<span>'--'</span>)
plt.axes()
plt.savefig(<span>'fig6.png'</span>, bbox_inches=<span>'tight'</span>)
</code></pre>
<div>
<p>output</p>
</div>
<div>
<p><img><img data-orig-file="https://gigadom.files.wordpress.com/2017/10/fig51.png?w=456" data-orig-size="454,442" data-medium-file="https://gigadom.files.wordpress.com/2017/10/fig51.png?w=456?w=300" data-attachment-id="5740" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2017/10/fig51.png?w=456" data-permalink="https://gigadom.wordpress.com/2017/10/29/practical-machine-learning-with-r-and-python-part-4/fig5-3/" data-large-file="https://gigadom.files.wordpress.com/2017/10/fig51.png?w=456?w=454" alt="" data-image-title="fig5" sizes="(max-width: 454px) 100vw, 454px" data-comments-opened="1" class="wp-image-5740" data-recalc-dims="1"></p>
<p>output</p>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2017/10/fig61.png?w=456" data-orig-size="584,463" data-medium-file="https://gigadom.files.wordpress.com/2017/10/fig61.png?w=456?w=300" data-attachment-id="5741" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2017/10/fig61.png?w=456" data-permalink="https://gigadom.wordpress.com/2017/10/29/practical-machine-learning-with-r-and-python-part-4/fig6-3/" data-large-file="https://gigadom.files.wordpress.com/2017/10/fig61.png?w=456?w=584" alt="" data-image-title="fig6" sizes="(max-width: 584px) 100vw, 584px" data-comments-opened="1" class="wp-image-5741" data-recalc-dims="1"></p>
</div>
</div>
<div>
<h2>1.10c Precision-Recall, ROC curves &amp; AUC- Python code</h2>
<p>In the code below classification probabilities are used to compute and plot precision-recall, roc and AUC</p>
<pre><code><span>import</span> numpy <span>as</span> np
<span>import</span> pandas <span>as</span> pd
<span>import</span> matplotlib.pyplot <span>as</span> plt
<span>from</span> sklearn.model_selection <span>import</span> train_test_split
<span>from</span> sklearn.datasets <span>import</span> load_digits
<span>from</span> sklearn.svm <span>import</span> LinearSVC
<span>from</span> sklearn.calibration <span>import</span> CalibratedClassifierCV

dataset = load_digits()
X, y = dataset.data, dataset.target
<span># Make a copy of the target</span>
z= y.copy()
<span># Replace all non 1's as 0</span>
z[z != <span>1</span>] = <span>0</span>


X_train, X_test, y_train, y_test = train_test_split(X, z, random_state=<span>0</span>)
svm = LinearSVC()
<span># Need to use CalibratedClassifierSVC to redict probabilities for lInearSVC</span>
clf = CalibratedClassifierCV(svm) 
clf.fit(X_train, y_train)
y_proba_lr = clf.predict_proba(X_test)
<span>from</span> sklearn.metrics <span>import</span> precision_recall_curve

precision, recall, thresholds = precision_recall_curve(y_test, y_proba_lr[:,<span>1</span>])
closest_zero = np.argmin(np.abs(thresholds))
closest_zero_p = precision[closest_zero]
closest_zero_r = recall[closest_zero]
<span>#plt.figure(figsize=(15,15),dpi=80)</span>
plt.figure()
plt.xlim([<span>0.0</span>, <span>1.01</span>])
plt.ylim([<span>0.0</span>, <span>1.01</span>])
plt.plot(precision, recall, label=<span>'Precision-Recall Curve'</span>)
plt.plot(closest_zero_p, closest_zero_r, <span>'o'</span>, markersize = <span>12</span>, fillstyle = <span>'none'</span>, c=<span>'r'</span>, mew=<span>3</span>)
plt.xlabel(<span>'Precision'</span>, fontsize=<span>16</span>)
plt.ylabel(<span>'Recall'</span>, fontsize=<span>16</span>)
plt.axes().set_aspect(<span>'equal'</span>)
plt.savefig(<span>'fig7.png'</span>, bbox_inches=<span>'tight'</span>)</code></pre>
<div>
<p><img>output</p>
<p><img data-orig-file="https://gigadom.files.wordpress.com/2017/10/fig71.png?w=456" data-orig-size="454,442" data-medium-file="https://gigadom.files.wordpress.com/2017/10/fig71.png?w=456?w=300" data-attachment-id="5742" data-image-meta="{" data-image-description="" src="https://gigadom.files.wordpress.com/2017/10/fig71.png?w=456" data-permalink="https://gigadom.wordpress.com/2017/10/29/practical-machine-learning-with-r-and-python-part-4/fig7-2/" data-large-file="https://gigadom.files.wordpress.com/2017/10/fig71.png?w=456?w=454" alt="" data-image-title="fig7" sizes="(max-width: 454px) 100vw, 454px" data-comments-opened="1" class="wp-image-5742" data-recalc-dims="1"></p>
<div><strong>Note</strong>: As with other posts in this series on ‘Practical Machine Learning with R and Python’,   this post is based on these 2 MOOC courses<br>
1. <a rel="nofollow" href="https://lagunita.stanford.edu/courses/HumanitiesandScience/StatLearning/Winter2015/about" target="_blank">Statistical Learning</a>, Prof Trevor Hastie &amp; Prof Robert Tibesherani, Online Stanford<br>
2. <a rel="nofollow" href="https://www.coursera.org/learn/python-machine-learning" target="_blank">Applied Machine Learning in Python</a> Prof Kevyn-Collin Thomson, University Of Michigan, Coursera
<h2>Conclusion</h2>
<p>This 4th part looked at SVMs with linear and radial basis, decision trees, precision-recall tradeoff, ROC curves and AUC.</p>
<p>Stick around for further updates. I’ll be back!<br>
Comments, suggestions and correction are welcome.</p>
<p>Also see<br>
1. <a rel="nofollow" href="https://gigadom.wordpress.com/2016/06/01/a-primer-on-qubits-quantum-gates-and-quantum-operations/" target="_blank">A primer on Qubits, Quantum gates and Quantum Operations</a><br>
2. <a rel="nofollow" href="https://gigadom.wordpress.com/2011/11/14/dabbling-with-wiener-filter-using-opencv/" target="_blank">Dabbling with Wiener filter using OpenCV</a><br>
3. <a rel="nofollow" href="https://gigadom.wordpress.com/2014/05/04/the-many-faces-of-a-programmer/" target="_blank">The mind of a programmer</a><br>
4. <a rel="nofollow" href="https://gigadom.wordpress.com/2012/01/03/sea-shells-on-the-seashore/" target="_blank">Sea shells on the seashore</a><br>
5. <a rel="nofollow" href="https://gigadom.wordpress.com/2016/04/16/yorkr-pads-up-for-the-twenty20s-part-1-analyzing-teams-match-performance/" target="_blank">yorkr pads up for the Twenty20s: Part 1- Analyzing team”s match performance</a></p>
<p>To see all posts see <a rel="nofollow" href="https://gigadom.wordpress.com/aa-2/" target="_blank">Index of posts</a></p>
</div>
</div>
</div>
<p> <a rel="nofollow" href="https://feeds.wordpress.com/1.0/gocomments/gigadom.wordpress.com/5722/" target="_blank"><img border="0" alt="" src="https://feeds.wordpress.com/1.0/comments/gigadom.wordpress.com/5722/"></a> <img border="0" alt="" src="https://i1.wp.com/pixel.wp.com/b.gif?resize=1%2C1&ssl=1" data-recalc-dims="1"></p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://gigadom.wordpress.com/2017/10/29/practical-machine-learning-with-r-and-python-part-4/"> R – Giga thoughts …</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/UOkbCAa8exo" height="1"></a></div>