<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="http://www.r-bloggers.com/vtreat-cross-frames/"></div></div><div><a data-url="http://www.r-bloggers.com/vtreat-cross-frames/" data-counturl="http://www.r-bloggers.com/vtreat-cross-frames/" data-count="horizontal" data-text="vtreat cross frames" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.win-vector.com/blog/2016/05/vtreat-cross-frames/"> R – Win-Vector Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers)</a> </div>
<h1>vtreat cross frames</h1>
<h4><em>John Mount, Nina Zumel</em></h4>
<h4><em>2016-05-05</em></h4>
<p>As a follow on to <a target="_blank" href="http://www.win-vector.com/blog/2016/04/on-nested-models/">“On Nested Models”</a> we work <a target="_blank" href="https://cran.r-project.org/">R</a> examples demonstrating “cross validated training frames” (or “cross frames”) in <a target="_blank" href="https://github.com/WinVector/vtreat">vtreat</a>.</p>
<p><span></span></p>
<p>Consider the following data frame. The outcome only depends on the “good” variables, not on the (high degree of freedom) “bad” variables. Modeling such a data set runs a high risk of overfit.</p>
<div>
<pre><code><span>set.seed</span>(<span>22626</span>)

mkData &lt;-<span> </span>function(n) {
  d &lt;-<span> </span><span>data.frame</span>(<span>xBad1=</span><span>sample</span>(<span>paste</span>(<span>'level'</span>,<span>1</span>:<span>1000</span>,<span>sep=</span><span>''</span>),n,<span>replace=</span><span>TRUE</span>),
                  <span>xBad2=</span><span>sample</span>(<span>paste</span>(<span>'level'</span>,<span>1</span>:<span>1000</span>,<span>sep=</span><span>''</span>),n,<span>replace=</span><span>TRUE</span>),
                  <span>xBad3=</span><span>sample</span>(<span>paste</span>(<span>'level'</span>,<span>1</span>:<span>1000</span>,<span>sep=</span><span>''</span>),n,<span>replace=</span><span>TRUE</span>),
                  <span>xGood1=</span><span>rnorm</span>(n),
                  <span>xGood2=</span><span>rnorm</span>(n))
  
  <span># outcome only depends on &quot;good&quot; variables</span>
  d$y &lt;-<span> </span><span>rnorm</span>(<span>nrow</span>(d))+<span>0.2</span>*d$xGood1 +<span> </span><span>0.3</span>*d$xGood2><span>0.5</span>
  <span># the random group used for splitting the data set, not a variable.</span>
  d$rgroup &lt;-<span> </span><span>sample</span>(<span>c</span>(<span>&quot;cal&quot;</span>,<span>&quot;train&quot;</span>,<span>&quot;test&quot;</span>),<span>nrow</span>(d),<span>replace=</span><span>TRUE</span>)  
  d
}

d &lt;-<span> </span><span>mkData</span>(<span>2000</span>)

<span># devtools::install_github(&quot;WinVector/WVPlots&quot;)</span>
<span># library('WVPlots')</span>
plotRes &lt;-<span> </span>function(d,predName,yName,title) {
  <span>print</span>(title)
  tab &lt;-<span> </span><span>table</span>(<span>truth=</span>d[[yName]],<span>pred=</span>d[[predName]]><span>0.5</span>)
  <span>print</span>(tab)
  diag &lt;-<span> </span><span>sum</span>(<span>vapply</span>(<span>seq_len</span>(<span>min</span>(<span>dim</span>(tab))),
                     function(i) tab[i,i],<span>numeric</span>(<span>1</span>)))
  acc &lt;-<span> </span>diag/<span>sum</span>(tab)
<span># if(requireNamespace(&quot;WVPlots&quot;,quietly=TRUE)) {</span>
<span># print(WVPlots::ROCPlot(d,predName,yName,title))</span>
<span># }</span>
  <span>print</span>(<span>paste</span>(<span>'accuracy'</span>,acc))
}</code></pre>
</div>
<div>
<h2>The Wrong Way</h2>
<p>Bad practice: use the same set of data to prepare variable encoding and train a model.</p>
<div>
<pre><code>dTrain &lt;-<span> </span>d[d$rgroup!=<span>'test'</span>,,drop=<span>FALSE</span>]
dTest &lt;-<span> </span>d[d$rgroup==<span>'test'</span>,,drop=<span>FALSE</span>]
treatments &lt;-<span> </span>vtreat::<span>designTreatmentsC</span>(dTrain,<span>c</span>(<span>'xBad1'</span>,<span>'xBad2'</span>,<span>'xBad3'</span>,<span>'xGood1'</span>,<span>'xGood2'</span>),
                                        <span>'y'</span>,<span>TRUE</span>,
  <span>rareCount=</span><span>0</span> <span># Note: usually want rareCount>0, setting to zero to illustrate problem</span>
)</code></pre>
</div>
<pre><code>## [1] &quot;desigining treatments Thu May  5 07:17:01 2016&quot;
## [1] &quot;design var xBad1 Thu May  5 07:17:01 2016&quot;
## [1] &quot;design var xBad2 Thu May  5 07:17:01 2016&quot;
## [1] &quot;design var xBad3 Thu May  5 07:17:01 2016&quot;
## [1] &quot;design var xGood1 Thu May  5 07:17:01 2016&quot;
## [1] &quot;design var xGood2 Thu May  5 07:17:01 2016&quot;
## [1] &quot;scoring treatments Thu May  5 07:17:01 2016&quot;
## [1] &quot;have treatment plan Thu May  5 07:17:01 2016&quot;
## [1] &quot;rescoring complex variables Thu May  5 07:17:01 2016&quot;
## [1] &quot;done rescoring complex variables Thu May  5 07:17:01 2016&quot;</code></pre>
<div>
<pre><code>dTrainTreated &lt;-<span> </span>vtreat::<span>prepare</span>(treatments,dTrain,
  <span>pruneSig=</span><span>c</span>() <span># Note: usually want pruneSig to be a small fraction, setting to null to illustrate problem</span>
)
m1 &lt;-<span> </span><span>glm</span>(y~xBad1_catB +<span> </span>xBad2_catB +<span> </span>xBad3_catB +<span> </span>xGood1_clean +<span> </span>xGood2_clean,
          <span>data=</span>dTrainTreated,<span>family=</span><span>binomial</span>(<span>link=</span><span>'logit'</span>))</code></pre>
</div>
<pre><code>## Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred</code></pre>
<div>
<pre><code><span>print</span>(<span>summary</span>(m1))  <span># notice low residual deviance</span></code></pre>
</div>
<pre><code>## 
## Call:
## glm(formula = y ~ xBad1_catB + xBad2_catB + xBad3_catB + xGood1_clean + 
##     xGood2_clean, family = binomial(link = &quot;logit&quot;), data = dTrainTreated)
## 
## Deviance Residuals: 
##      Min        1Q    Median        3Q       Max  
## -2.70438   0.00000   0.00000   0.03995   2.61063  
## 
## Coefficients:
##              Estimate Std. Error z value Pr(>|z|)    
## (Intercept)   -0.5074     0.3350  -1.515  0.12983    
## xBad1_catB     2.9432     0.5549   5.304 1.13e-07 ***
## xBad2_catB     2.5338     0.5857   4.326 1.52e-05 ***
## xBad3_catB     3.4172     0.6092   5.610 2.03e-08 ***
## xGood1_clean   0.7288     0.2429   3.001  0.00269 ** 
## xGood2_clean   0.7788     0.2585   3.012  0.00259 ** 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 1724.55  on 1331  degrees of freedom
## Residual deviance:  132.59  on 1326  degrees of freedom
## AIC: 144.59
## 
## Number of Fisher Scoring iterations: 12</code></pre>
<div>
<pre><code>dTrain$predM1 &lt;-<span> </span><span>predict</span>(m1,<span>newdata=</span>dTrainTreated,<span>type=</span><span>'response'</span>)
<span>plotRes</span>(dTrain,<span>'predM1'</span>,<span>'y'</span>,<span>'model1 on train'</span>)</code></pre>
</div>
<pre><code>## [1] &quot;model1 on train&quot;
##        pred
## truth   FALSE TRUE
##   FALSE   848   18
##   TRUE      6  460
## [1] &quot;accuracy 0.981981981981982&quot;</code></pre>
<div>
<pre><code>dTestTreated &lt;-<span> </span>vtreat::<span>prepare</span>(treatments,dTest,<span>pruneSig=</span><span>c</span>())
dTest$predM1 &lt;-<span> </span><span>predict</span>(m1,<span>newdata=</span>dTestTreated,<span>type=</span><span>'response'</span>)
<span>plotRes</span>(dTest,<span>'predM1'</span>,<span>'y'</span>,<span>'model1 on test'</span>)</code></pre>
</div>
<pre><code>## [1] &quot;model1 on test&quot;
##        pred
## truth   FALSE TRUE
##   FALSE   360  114
##   TRUE    153   41
## [1] &quot;accuracy 0.600299401197605&quot;</code></pre>
<p>Notice above that we see a training accuracy of 98% and a test accuracy of 60%.</p>
</div>
<div>
<h2>The Right Way: A Calibration Set</h2>
<p>Now try a proper calibration/train/test split:</p>
<div>
<pre><code>dCal &lt;-<span> </span>d[d$rgroup==<span>'cal'</span>,,drop=<span>FALSE</span>]
dTrain &lt;-<span> </span>d[d$rgroup==<span>'train'</span>,,drop=<span>FALSE</span>]
dTest &lt;-<span> </span>d[d$rgroup==<span>'test'</span>,,drop=<span>FALSE</span>]
treatments &lt;-<span> </span>vtreat::<span>designTreatmentsC</span>(dCal,<span>c</span>(<span>'xBad1'</span>,<span>'xBad2'</span>,<span>'xBad3'</span>,<span>'xGood1'</span>,<span>'xGood2'</span>),
                                        <span>'y'</span>,<span>TRUE</span>,
  <span>rareCount=</span><span>0</span> <span># Note: usually want rareCount>0, setting to zero to illustrate problem</span>
)</code></pre>
</div>
<pre><code>## [1] &quot;desigining treatments Thu May  5 07:17:01 2016&quot;
## [1] &quot;design var xBad1 Thu May  5 07:17:01 2016&quot;
## [1] &quot;design var xBad2 Thu May  5 07:17:01 2016&quot;
## [1] &quot;design var xBad3 Thu May  5 07:17:01 2016&quot;
## [1] &quot;design var xGood1 Thu May  5 07:17:01 2016&quot;
## [1] &quot;design var xGood2 Thu May  5 07:17:01 2016&quot;
## [1] &quot;scoring treatments Thu May  5 07:17:01 2016&quot;
## [1] &quot;have treatment plan Thu May  5 07:17:01 2016&quot;
## [1] &quot;rescoring complex variables Thu May  5 07:17:01 2016&quot;
## [1] &quot;done rescoring complex variables Thu May  5 07:17:02 2016&quot;</code></pre>
<div>
<pre><code>dTrainTreated &lt;-<span> </span>vtreat::<span>prepare</span>(treatments,dTrain,
  <span>pruneSig=</span><span>c</span>() <span># Note: usually want pruneSig to be a small fraction, setting to null to illustrate problem</span>
)
m1 &lt;-<span> </span><span>glm</span>(y~xBad1_catB +<span> </span>xBad2_catB +<span> </span>xBad3_catB +<span> </span>xGood1_clean +<span> </span>xGood2_clean,
          <span>data=</span>dTrainTreated,<span>family=</span><span>binomial</span>(<span>link=</span><span>'logit'</span>))
<span>print</span>(<span>summary</span>(m1))  </code></pre>
</div>
<pre><code>## 
## Call:
## glm(formula = y ~ xBad1_catB + xBad2_catB + xBad3_catB + xGood1_clean + 
##     xGood2_clean, family = binomial(link = &quot;logit&quot;), data = dTrainTreated)
## 
## Deviance Residuals: 
##     Min       1Q   Median       3Q      Max  
## -1.5853  -0.9177  -0.6876   1.1651   2.3241  
## 
## Coefficients:
##              Estimate Std. Error z value Pr(>|z|)    
## (Intercept)  -0.73798    0.12720  -5.802 6.56e-09 ***
## xBad1_catB   -0.02380    0.02637  -0.903    0.367    
## xBad2_catB   -0.02495    0.02608  -0.957    0.339    
## xBad3_catB    0.02058    0.02508   0.821    0.412    
## xGood1_clean  0.39234    0.08632   4.545 5.49e-06 ***
## xGood2_clean  0.56252    0.09673   5.816 6.04e-09 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 832.55  on 642  degrees of freedom
## Residual deviance: 769.28  on 637  degrees of freedom
## AIC: 781.28
## 
## Number of Fisher Scoring iterations: 4</code></pre>
<div>
<pre><code>dTrain$predM1 &lt;-<span> </span><span>predict</span>(m1,<span>newdata=</span>dTrainTreated,<span>type=</span><span>'response'</span>)
<span>plotRes</span>(dTrain,<span>'predM1'</span>,<span>'y'</span>,<span>'model1 on train'</span>)</code></pre>
</div>
<pre><code>## [1] &quot;model1 on train&quot;
##        pred
## truth   FALSE TRUE
##   FALSE   378   40
##   TRUE    157   68
## [1] &quot;accuracy 0.693623639191291&quot;</code></pre>
<div>
<pre><code>dTestTreated &lt;-<span> </span>vtreat::<span>prepare</span>(treatments,dTest,<span>pruneSig=</span><span>c</span>())
dTest$predM1 &lt;-<span> </span><span>predict</span>(m1,<span>newdata=</span>dTestTreated,<span>type=</span><span>'response'</span>)
<span>plotRes</span>(dTest,<span>'predM1'</span>,<span>'y'</span>,<span>'model1 on test'</span>)</code></pre>
</div>
<pre><code>## [1] &quot;model1 on test&quot;
##        pred
## truth   FALSE TRUE
##   FALSE   422   52
##   TRUE    149   45
## [1] &quot;accuracy 0.699101796407186&quot;</code></pre>
<p>Notice above that we now see training and test accuracies of 70%. We have defeated overfit in two ways: training performance is closer to test performance, and test performance is better. Also we see that the model now properly considers the “bad” variables to be insignificant.</p>
</div>
<div>
<h2>Another Right Way: Cross-Validation</h2>
<p>Below is a more statistically efficient practice: building a cross training frame.</p>
<div>
<h3>The intuition</h3>
<p>Consider any trained statistical model (in this case our treatment plan and variable selection plan) as a two-argument function <em>f(A,B)</em>. The first argument is the training data and the second argument is the application data. In our case <em>f(A,B)</em> is: <code>designTreatmentsC(A) %>% prepare(B)</code>, and it produces a treated data frame.</p>
<p>When we use the same data in both places to build our training frame, as in</p>
<blockquote>
<p><em>TrainTreated = f(TrainData,TrainData)</em>,</p>
</blockquote>
<p>we are not doing a good job simulating the future application of <em>f(,)</em>, which will be <em>f(TrainData,FutureData)</em>.</p>
<p>To improve the quality of our simulation we can call</p>
<blockquote>
<p><em>TrainTreated = f(CalibrationData,TrainData)</em></p>
</blockquote>
<p>where <em>CalibrationData</em> and <em>TrainData</em> are disjoint datasets (as we did in the earlier example) and expect this to be a good imitation of future <em>f(CalibrationData,FutureData)</em>.</p>
</div>
<div>
<h3>Cross-Validation and vtreat: The cross-frame.</h3>
<p>Another approach is to build a “cross validated” version of <em>f</em>. We split <em>TrainData</em> into a list of 3 disjoint row intervals: <em>Train1</em>,<em>Train2</em>,<em>Train3</em>. Instead of computing <em>f(TrainData,TrainData)</em> compute:</p>
<blockquote>
<p><em>TrainTreated = f(Train2+Train3,Train1) + f(Train1+Train3,Train2) + f(Train1+Train2,Train3)</em></p>
</blockquote>
<p>(where + denotes <code>rbind()</code>).</p>
<p>The idea is this looks a lot like <em>f(TrainData,TrainData)</em> except it has the important property that no row in the right-hand side is ever worked on by a model built using that row (a key characteristic that future data will have) so we have a good imitation of <em>f(TrainData,FutureData)</em>.</p>
<p>In other words: we use cross validation to simulate future data. The main thing we are doing differently is remembering that we can apply cross validation to <em>any</em> two argument function <em>f(A,B)</em> and not only to functions of the form <em>f(A,B)</em> = <code>buildModel(A) %>% scoreData(B)</code>. We can use this formulation in stacking or super-learning with <em>f(A,B)</em> of the form <code>buildSubModels(A) %>% combineModels(B)</code> (to produce a stacked or ensemble model); the idea applies to improving ensemble methods in general.</p>
<p>See:</p>
<ul>
<li>“General oracle inequalities for model selection” Charles Mitchell and Sara van de Geer</li>
<li>“On Cross-Validation and Stacking: Building seemingly predictive models on random data” Claudia Perlich and Grzegorz Swirszcz</li>
<li>“Super Learner” Mark J. van der Laan, Eric C. Polley, and Alan E. Hubbard</li>
</ul>
<p>In fact (though it was developed independently) you can think of vtreat as a superlearner.</p>
<p>In super learning cross validation techniques are used to simulate having built sub-model predictions on novel data. The simulated out of sample-applications of these sub models (and not the sub models themselves) are then used as input data for the next stage learner. In future application the actual sub-models are applied and their immediate outputs is used by the super model.</p>
<p>
<img title="superX.png" alt="SuperX" src="http://i2.wp.com/www.win-vector.com/blog/wp-content/uploads/2016/05/superX.png?resize=599%2C370" border="0" data-recalc-dims="1">
</p>
<p>In vtreat the sub-models are single variable treatments and the outer model construction is left to the practitioner (using the cross-frames for simulation and not the treatmentplan). In application the treatment plan is used.</p>
<p>
<img title="vtreatX.png" alt="VtreatX" src="http://i1.wp.com/www.win-vector.com/blog/wp-content/uploads/2016/05/vtreatX.png?resize=598%2C359" border="0" data-recalc-dims="1"></p>
</div>
<div>
<h3>Example</h3>
<p>Below is the example cross-run. The function <code>mkCrossFrameCExperiment</code> returns a treatment plan for use in preparing future data, and a cross-frame for use in fitting a model.</p>
<div>
<pre><code>dTrain &lt;-<span> </span>d[d$rgroup!=<span>'test'</span>,,drop=<span>FALSE</span>]
dTest &lt;-<span> </span>d[d$rgroup==<span>'test'</span>,,drop=<span>FALSE</span>]
prep &lt;-<span> </span>vtreat::<span>mkCrossFrameCExperiment</span>(dTrain,
                                              <span>c</span>(<span>'xBad1'</span>,<span>'xBad2'</span>,<span>'xBad3'</span>,<span>'xGood1'</span>,<span>'xGood2'</span>),
                                        <span>'y'</span>,<span>TRUE</span>,
  <span>rareCount=</span><span>0</span> <span># Note: usually want rareCount>0, setting to zero to illustrate problem</span>
)
dTrainTreated &lt;-<span> </span>prep$crossFrame
treatments &lt;-<span> </span>prep$treatments
<span>print</span>(treatments$scoreFrame[,<span>c</span>(<span>'varName'</span>,<span>'lsig'</span>,<span>'csig'</span>)])</code></pre>
</div>
<pre><code>##        varName         lsig         csig
## 1   xBad1_catP 8.172932e-01 8.172186e-01
## 2   xBad1_catB 5.675615e-01 5.676541e-01
## 3   xBad2_catP 7.446537e-01 7.441869e-01
## 4   xBad2_catB 5.792325e-01 5.793585e-01
## 5   xBad3_catP 4.356331e-01 4.342227e-01
## 6   xBad3_catB 1.786048e-01 1.770493e-01
## 7 xGood1_clean 6.529637e-12 6.072599e-12
## 8 xGood2_clean 8.584085e-21 8.286789e-21</code></pre>
<p>Now fit the model to <em>the cross-frame</em> rather than to <code>prepare(treatments, dTrain)</code> (the treated training data).</p>
<div>
<pre><code>m1 &lt;-<span> </span><span>glm</span>(y~xBad1_catB +<span> </span>xBad2_catB +<span> </span>xBad3_catB +<span> </span>xGood1_clean +<span> </span>xGood2_clean,
          <span>data=</span>dTrainTreated,<span>family=</span><span>binomial</span>(<span>link=</span><span>'logit'</span>))
<span>print</span>(<span>summary</span>(m1))  </code></pre>
</div>
<pre><code>## 
## Call:
## glm(formula = y ~ xBad1_catB + xBad2_catB + xBad3_catB + xGood1_clean + 
##     xGood2_clean, family = binomial(link = &quot;logit&quot;), data = dTrainTreated)
## 
## Deviance Residuals: 
##     Min       1Q   Median       3Q      Max  
## -1.6842  -0.9236  -0.6573   1.1824   2.3257  
## 
## Coefficients:
##               Estimate Std. Error z value Pr(>|z|)    
## (Intercept)  -0.690824   0.091313  -7.565 3.87e-14 ***
## xBad1_catB    0.001813   0.017218   0.105    0.916    
## xBad2_catB   -0.023835   0.017128  -1.392    0.164    
## xBad3_catB    0.024460   0.016978   1.441    0.150    
## xGood1_clean  0.404827   0.061885   6.542 6.09e-11 ***
## xGood2_clean  0.570083   0.064988   8.772  &lt; 2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 1724.6  on 1331  degrees of freedom
## Residual deviance: 1587.7  on 1326  degrees of freedom
## AIC: 1599.7
## 
## Number of Fisher Scoring iterations: 4</code></pre>
<div>
<pre><code>dTrain$predM1 &lt;-<span> </span><span>predict</span>(m1,<span>newdata=</span>dTrainTreated,<span>type=</span><span>'response'</span>)
<span>plotRes</span>(dTrain,<span>'predM1'</span>,<span>'y'</span>,<span>'model1 on train'</span>)</code></pre>
</div>
<pre><code>## [1] &quot;model1 on train&quot;
##        pred
## truth   FALSE TRUE
##   FALSE   776   90
##   TRUE    335  131
## [1] &quot;accuracy 0.680930930930931&quot;</code></pre>
<div>
<pre><code>dTestTreated &lt;-<span> </span>vtreat::<span>prepare</span>(treatments,dTest,<span>pruneSig=</span><span>c</span>())
dTest$predM1 &lt;-<span> </span><span>predict</span>(m1,<span>newdata=</span>dTestTreated,<span>type=</span><span>'response'</span>)
<span>plotRes</span>(dTest,<span>'predM1'</span>,<span>'y'</span>,<span>'model1 on test'</span>)</code></pre>
</div>
<pre><code>## [1] &quot;model1 on test&quot;
##        pred
## truth   FALSE TRUE
##   FALSE   421   53
##   TRUE    145   49
## [1] &quot;accuracy 0.703592814371258&quot;</code></pre>
<p>The model fit to the cross-frame behaves similarly to the model produced via the process <em>f(CalibrationData, TrainData)</em>.</p>
</div>
</div>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="http://www.r-bloggers.com/vtreat-cross-frames/"></div></div><div><a data-url="http://www.r-bloggers.com/vtreat-cross-frames/" data-counturl="http://www.r-bloggers.com/vtreat-cross-frames/" data-count="vertical" data-text="vtreat cross frames" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.win-vector.com/blog/2016/05/vtreat-cross-frames/"> R – Win-Vector Blog</a></strong>.</div>
<hr>
<a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="http://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="http://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="http://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="http://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="http://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="http://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="http://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="http://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="http://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="http://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="http://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="http://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="http://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="http://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="http://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="http://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/KNBB_Q88tAM"></a></div>