<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/machine-learning-results-in-r-one-plot-to-rule-them-all/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://datascienceplus.com/machine-learning-results-one-plot-to-rule-them-all/"> R Programming – DataScience+</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>To automate the process of modeling selection and evaluate the results with visualization, I have created some functions into my personal library and today I’m sharing the codes with you. I run them to evaluate and compare Machine Learning models as fast and easily as possible. Currently, they are designed to evaluate binary classification models results. Before we start, let me show you the final outcome so you know what we are trying to achieve here with just a simple R function:<br>
<a rel="nofollow" href="https://i1.wp.com/datascienceplus.com/wp-content/uploads/2018/07/viz_full.png?ssl=1" target="_blank"><img sizes="(max-width: 490px) 100vw, 490px" src="https://i2.wp.com/datascienceplus.com/wp-content/uploads/2018/07/viz_full-490x306.png?w=450&ssl=1" alt="" class="wp-image-15198" data-recalc-dims="1"></a></p>
<p>So, let’s start!</p>
<h2>The results object</h2>
<p>First of all, we need to have a single list with all the results to facilitate the next steps. I am assuming on this step that you already designed a model and can calculate the predictions out of your test set. So, on my list I have the following objects:</p>
<li>Project name (i.e. Fraud Score)</li>
<li>Model (the object with our model)</li>
<li>Test Scores:</li>
<ul>
<li>Index (row id, it can be a user_id, email, lead_id…)</li>
<li>Tag (known label)</li>
<li>Score (calculated with the model we are studying)</li>
</ul>
<li>Datasets:</li>
<ul>
<li>Train set</li>
<li>Test set</li>
</ul>
<li>Parameters:</li>
<ul>
<li>nfolds, ntrees, max_depth, seed, sample_rate….</li>
</ul>
<li>Variable importance</li>
<li>Metrics:</li>
<ul>
<li>log_loss</li>
<li>auc</li>
</ul>
<li>Notes (anything you’d like to write to give you a reference later on)</li>
<p>Once we automate our results object, we can start with our beautiful plots!</p>
<h2>Density Plot</h2>
<p>I have always given importance to the density plot because it gives us visual information on skewness, distribution and our model’s facility to distinguish each class. Here we can see how the model has distributed both our categories, our whole test set and the cumulative of each category (the more separate, the better).</p>
<pre>
mplot_density &lt;- function(tag, score, model_name = NA, subtitle = NA, 
                          save = FALSE, file_name = &quot;viz_distribution.png&quot;) {
  require(ggplot2)
  require(gridExtra)

  if (length(tag) != length(score)) {
    message(&quot;The tag and score vectors should be the same length.&quot;)
    stop(message(paste(&quot;Currently, tag has&quot;,length(tag),&quot;rows and score has&quot;,length(score))))
  }

  if (length(unique(tag)) != 2) {
    stop(&quot;This function is for binary models. You should only have 2 unique values for the tag value!&quot;)
  }

  out &lt;- data.frame(tag = as.character(tag),
                    score = as.numeric(score),
                    norm_score = lares::normalize(as.numeric(score)))
  
  p1 &lt;- ggplot(out) + theme_minimal() +
    geom_density(aes(x = 100 * score, group = tag, fill = as.character(tag)), 
                 alpha = 0.6, adjust = 0.25) + 
    guides(fill = guide_legend(title=&quot;Tag&quot;)) + 
    xlim(0, 100) + 
    labs(title = &quot;Score distribution for binary model&quot;,
         y = &quot;Density by tag&quot;, x = &quot;Score&quot;)
  
  p2 &lt;- ggplot(out) + theme_minimal() + 
    geom_density(aes(x = 100 * score), 
                 alpha = 0.9, adjust = 0.25, fill = &quot;deepskyblue&quot;) + 
    labs(x = &quot;&quot;, y = &quot;Density&quot;)
  
  p3 &lt;- ggplot(out) + theme_minimal() + 
    geom_line(aes(x = score * 100, y = 100 * (1 - ..y..), color = as.character(tag)), 
              stat = 'ecdf', size = 1) +
    geom_line(aes(x = score * 100, y = 100 * (1 - ..y..)), 
              stat = 'ecdf', size = 0.5, colour = &quot;black&quot;, linetype=&quot;dotted&quot;) +
    ylab('Cumulative') + xlab('') + guides(color=FALSE)
  
  if(!is.na(subtitle)) {
    p1 &lt;- p1 + labs(subtitle = subtitle)
  }
  
  if(!is.na(model_name)) {
    p1 &lt;- p1 + labs(caption = model_name)
  }
  
  if(save == TRUE) {
    png(file_name, height = 1800, width = 2100, res = 300)
    grid.arrange(
      p1, p2, p3, 
      ncol = 2, nrow = 2, heights = 2:1,
      layout_matrix = rbind(c(1,1), c(2,3)))
    dev.off()
  }
  
  return(
    grid.arrange(
      p1, p2, p3, 
      ncol = 2, nrow = 2, heights = 2:1,
      layout_matrix = rbind(c(1,1), c(2,3))))
  
}
</pre>
<p>Gives this plot:<br>
<a rel="nofollow" href="https://i2.wp.com/datascienceplus.com/wp-content/uploads/2018/07/Captura-de-pantalla-2018-07-17-a-las-4.09.35-p.-m..png?ssl=1" target="_blank"><img sizes="(max-width: 1024px) 100vw, 1024px" src="https://i0.wp.com/datascienceplus.com/wp-content/uploads/2018/07/Captura-de-pantalla-2018-07-17-a-las-4.09.35-p.-m.-1024x845.png?w=450&ssl=1" alt="" class="wp-image-15206" data-recalc-dims="1"></a></p>
<h2>ROC Curve</h2>
<p>The ROC curve will give us an idea of how our model is performing with our test set. You should know by now that if the AUC is close to 50% then the model is as good as a random selector; on the other hand, if the AUC is near 100% then you have a “perfect model” (wanting or not, you must have been giving the model the answer this whole time!). So it is always good to check this plot and check that we are getting a reasonable Area Under the Curve with a nice and closed 95% confidence range.</p>
<pre>
# ROC Curve
mplot_roc &lt;- function(tag, score, model_name = NA, subtitle = NA, interval = 0.2, plotly = FALSE,
save = FALSE, file_name = &quot;viz_roc.png&quot;) {
  require(pROC)
  require(ggplot2)

  if (length(tag) != length(score)) {
    message(&quot;The tag and score vectors should be the same length.&quot;)
    stop(message(paste(&quot;Currently, tag has&quot;,length(tag),&quot;rows and score has&quot;,length(score))))
  }

  roc &lt;- pROC::roc(tag, score, ci=T)
  coords &lt;- data.frame(
    x = rev(roc$specificities),
    y = rev(roc$sensitivities))
  ci &lt;- data.frame(roc$ci, row.names = c(&quot;min&quot;,&quot;AUC&quot;,&quot;max&quot;))

  p &lt;- ggplot(coords, aes(x = x, y = y)) +
    geom_line(colour = &quot;deepskyblue&quot;, size = 1) +
    geom_point(colour = &quot;blue3&quot;, size = 0.9, alpha = 0.4) +
    geom_segment(aes(x = 0, y = 1, xend = 1, yend = 0), alpha = 0.2, linetype = &quot;dotted&quot;) + 
    scale_x_reverse(name = &quot;% Specificity [False Positive Rate]&quot;, limits = c(1,0), 
                    breaks = seq(0, 1, interval), expand = c(0.001,0.001)) + 
    scale_y_continuous(name = &quot;% Sensitivity [True Positive Rate]&quot;, limits = c(0,1), 
                       breaks = seq(0, 1, interval), expand = c(0.001, 0.001)) +
    theme_minimal() + 
    theme(axis.ticks = element_line(color = &quot;grey80&quot;)) +
    coord_equal() + 
    ggtitle(&quot;ROC Curve: AUC&quot;) +
    annotate(&quot;text&quot;, x = 0.25, y = 0.10, vjust = 0, size = 4.2, 
             label = paste(&quot;AUC =&quot;, round(100*ci[c(&quot;AUC&quot;),],2))) +
    annotate(&quot;text&quot;, x = 0.25, y = 0.05, vjust = 0, size = 2.8, 
             label = paste0(&quot;95% CI: &quot;, 
                            round(100*ci[c(&quot;min&quot;),],2),&quot;-&quot;, 
                            round(100*ci[c(&quot;max&quot;),],2)))
  if(!is.na(subtitle)) {
    p &lt;- p + labs(subtitle = subtitle)
  }  

  if(!is.na(model_name)) {
    p &lt;- p + labs(caption = model_name)
  }

  if (plotly == TRUE) {
    require(plotly)
    p &lt;- ggplotly(p)
  }

  if (save == TRUE) {
    p &lt;- p + ggsave(file_name, width = 6, height = 6)
  }
  return(p)
}
</pre>
<p>Gives this plot:<br>
<a rel="nofollow" href="https://i1.wp.com/datascienceplus.com/wp-content/uploads/2018/07/Captura-de-pantalla-2018-07-17-a-las-4.10.41-p.-m..png?ssl=1" target="_blank"><img sizes="(max-width: 740px) 100vw, 740px" src="https://i1.wp.com/datascienceplus.com/wp-content/uploads/2018/07/Captura-de-pantalla-2018-07-17-a-las-4.10.41-p.-m..png?w=450&ssl=1" alt="" class="wp-image-15207" data-recalc-dims="1"></a></p>
<h2>Cuts by quantile</h2>
<p>If we’d have to cut the score in n equal-sized buckets, what would the score cuts be? Is the result a ladder (as it should), or a huge wall, or a valley? Is our score distribution lineal and easy to split?</p>
<pre>
mplot_cuts &lt;- function(score, splits = 10, subtitle = NA, model_name = NA, 
                       save = FALSE, file_name = &quot;viz_ncuts.png&quot;) {
  
  require(ggplot2)
  
  if (splits > 25) {
    stop(&quot;You should try with less splits!&quot;)
  }
  
  deciles &lt;- quantile(score, 
                      probs = seq((1/splits), 1, length = splits), 
                      names = TRUE)
  deciles &lt;- data.frame(cbind(Deciles=row.names(as.data.frame(deciles)),
                              Threshold=as.data.frame(deciles)))
  
  p &lt;- ggplot(deciles, 
              aes(x = reorder(Deciles, deciles), y = deciles * 100, 
                  label = round(100 * deciles, 1))) + 
    geom_col(fill=&quot;deepskyblue&quot;) + 
    xlab('') + theme_minimal() + ylab('Score') + 
    geom_text(vjust = 1.5, size = 3, inherit.aes = TRUE, colour = &quot;white&quot;, check_overlap = TRUE) +
    labs(title = paste(&quot;Cuts by score: using&quot;, splits, &quot;equal-sized buckets&quot;))
  
  if(!is.na(subtitle)) {
    p &lt;- p + labs(subtitle = subtitle)
  } 
  if(!is.na(model_name)) {
    p &lt;- p + labs(caption = model_name)
  }
  if (save == TRUE) {
    p &lt;- p + ggsave(file_name, width = 6, height = 6)
  }
  return(p)
}
</pre>
<p>Gives this plot:<br>
<a rel="nofollow" href="https://i1.wp.com/datascienceplus.com/wp-content/uploads/2018/07/Captura-de-pantalla-2018-07-17-a-las-4.12.22-p.-m..png?ssl=1" target="_blank"><img sizes="(max-width: 1024px) 100vw, 1024px" src="https://i0.wp.com/datascienceplus.com/wp-content/uploads/2018/07/Captura-de-pantalla-2018-07-17-a-las-4.12.22-p.-m.-1024x851.png?w=450&ssl=1" alt="" class="wp-image-15208" data-recalc-dims="1"></a></p>
<h2>Split and compare quantiles</h2>
<p>This parameter is the easiest to sell to the C-level guys. “Did you know that with this model, if we chop the worst 20% of leads we would have avoided 60% of the frauds and only lose 8% of our sales?” That’s what this plot will give you:</p>
<pre>
mplot_splits &lt;- function(tag, score, splits = 5, subtitle = NA, model_name = NA, facet = NA, 
                         save = FALSE, file_name = &quot;viz_splits.png&quot;) {
  
  require(ggplot2)
  require(dplyr)
  require(RColorBrewer)
  
  if (length(tag) != length(score)) {
    message(&quot;The tag and score vectors should be the same length.&quot;)
    stop(message(paste(&quot;Currently, tag has&quot;,length(tag),&quot;rows and score has&quot;,length(score))))
  }
  
  if (splits > 10) {
    stop(&quot;You should try with less splits!&quot;)
  }

  df &lt;- data.frame(tag, score, facet)
  npersplit &lt;- round(nrow(df)/splits)
  names % 
    mutate(quantile = ntile(score, splits)) %>% group_by(quantile) %>%
    summarise(n = n(), 
              max_score = round(100 * max(score), 1), 
              min_score = round(100 * min(score), 1)) %>%
    mutate(quantile_tag = paste0(quantile,&quot; (&quot;,min_score,&quot;-&quot;,max_score,&quot;)&quot;))
  
  p % 
    mutate(quantile = ntile(score, splits)) %>% 
    group_by(quantile, facet, tag) %>% tally() %>%
    ungroup() %>% group_by(facet, tag) %>% 
    arrange(desc(quantile)) %>%
    mutate(p = round(100*n/sum(n),2),
           cum = cumsum(100*n/sum(n))) %>%
    left_join(names, by = c(&quot;quantile&quot;)) %>%
    ggplot(aes(x = as.character(tag), y = p, label = as.character(p),
               fill = as.character(quantile_tag))) + theme_minimal() +
    geom_col(position = &quot;stack&quot;) +
    geom_text(size = 3, position = position_stack(vjust = 0.5), check_overlap = TRUE) +
    xlab(&quot;Tag&quot;) + ylab(&quot;Total Percentage by Tag&quot;) +
    guides(fill = guide_legend(title=paste0(&quot;~&quot;,npersplit,&quot; p/split&quot;))) +
    labs(title = &quot;Tag vs Score Splits Comparison&quot;) +
    scale_fill_brewer(palette = &quot;Spectral&quot;)
  if(!is.na(subtitle)) {
    p &lt;- p + labs(subtitle = subtitle)
  }  
  if(!is.na(model_name)) {
    p &lt;- p + labs(caption = model_name)
  }
  if(!is.na(facet)) {
    p &lt;- p + facet_grid(. ~ facet, scales = &quot;free&quot;)
  }  
  if (save == TRUE) {
    p &lt;- p + ggsave(file_name, width = 6, height = 6)
  }
  return(p)
}
</pre>
<p>Gives this plot:<br>
<a rel="nofollow" href="https://i2.wp.com/datascienceplus.com/wp-content/uploads/2018/07/Captura-de-pantalla-2018-07-17-a-las-4.13.23-p.-m..png?ssl=1" target="_blank"><img sizes="(max-width: 1024px) 100vw, 1024px" src="https://i1.wp.com/datascienceplus.com/wp-content/uploads/2018/07/Captura-de-pantalla-2018-07-17-a-las-4.13.23-p.-m.-1024x854.png?w=450&ssl=1" alt="" class="wp-image-15209" data-recalc-dims="1"></a></p>
<h2>Finally, let’s plot our results</h2>
<p>Once we have defined these functions above, we can create a new one that will bring everything together into one single plot. If you pay attention to the variables needed to create this dashboard you would notice it actually only needs two: the label or tag, and the score. You can customize the splits for the upper right plot, set a subtitle, define the model’s name, save it in a new folder, change the image’s name. </p>
<pre>
mplot_full &lt;- function(tag, score, splits = 8, subtitle = NA, model_name = NA, 
                       save = FALSE, file_name = &quot;viz_full.png&quot;, subdir = NA) {
  require(ggplot2)
  require(gridExtra)
  options(warn=-1)

  if (length(tag) != length(score)) {
    message(&quot;The tag and score vectors should be the same length.&quot;)
    stop(message(paste(&quot;Currently, tag has&quot;,length(tag),&quot;rows and score has&quot;,length(score))))
  }

  p1 &lt;- mplot_density(tag = tag, score = score, subtitle = subtitle, model_name = model_name)
  p2 &lt;- mplot_splits(tag = tag, score = score, splits = splits)
  p3 &lt;- mplot_roc(tag = tag, score = score)
  p4 &lt;- mplot_cuts(score = score)

  if(save == TRUE) {
    if (!is.na(subdir)) {
      dir.create(file.path(getwd(), subdir))
      file_name &lt;- paste(subdir, file_name, sep=&quot;/&quot;)
    }
    png(file_name, height = 2000, width = 3200, res = 300)
    grid.arrange(
      p1, p2, p3, p4,
      widths = c(1.3,1),
      layout_matrix = rbind(c(1,2), c(1,2), c(1,3), c(4,3)))
    dev.off()
  }
  return(
    grid.arrange(
      p1, p2, p3, p4,
      widths = c(1.3,1),
      layout_matrix = rbind(c(1,2), c(1,2), c(1,3), c(4,3)))
  ) 
}
</pre>
<p>That’s it. This dashboard will give us almost everything we need to visually evaluate our model’s performance into the test set.<br>
<a rel="nofollow" href="https://i1.wp.com/datascienceplus.com/wp-content/uploads/2018/07/viz_full.png?ssl=1" target="_blank"><img sizes="(max-width: 490px) 100vw, 490px" src="https://i2.wp.com/datascienceplus.com/wp-content/uploads/2018/07/viz_full-490x306.png?w=450&ssl=1" alt="" class="wp-image-15198" data-recalc-dims="1"></a></p>
<p>One bonus tip for these plots: you can set the subtitle and subdirectory before you plot everything so you don’t have to change it whenever you are trying a new model. </p>
<pre>
subtitle &lt;- paste(results$project, &quot;- AUC:&quot;, round(100 * results$auc_test, 2))
subdir &lt;- paste0(&quot;Models/&quot;, round(100*results$auc_test, 2), &quot;-&quot;, results$model_name)
</pre>
<h3>Bonus: Variables Importance</h3>
<p>If you are working with a ML algorithm that let’s you see the importance of each variable, you can use the following function to see the results:</p>
<pre>
mplot_importance &lt;- function(var, imp, colours = NA, limit = 15, model_name = NA, subtitle = NA,
                             save = FALSE, file_name = &quot;viz_importance.png&quot;, subdir = NA) {
  
  require(ggplot2)
  require(gridExtra)
  options(warn=-1)
  
  if (length(var) != length(imp)) {
    message(&quot;The variables and importance values vectors should be the same length.&quot;)
    stop(message(paste(&quot;Currently, there are&quot;,length(var),&quot;variables and&quot;,length(imp),&quot;importance values!&quot;)))
  }
  if (is.na(colours)) {
    colours &lt;- &quot;deepskyblue&quot; 
  }
  out &lt;- data.frame(var = var, imp = imp, Type = colours)
  if (length(var) &lt; limit) {
    limit &lt;- length(var)
  }
  
  output &lt;- out[1:limit,]
  
  p &lt;- ggplot(output, 
              aes(x = reorder(var, imp), y = imp * 100, 
                  label = round(100 * imp, 1))) + 
    geom_col(aes(fill = Type), width = 0.1) +
    geom_point(aes(colour = Type), size = 6) + 
    coord_flip() + xlab('') + theme_minimal() +
    ylab('Importance') + 
    geom_text(hjust = 0.5, size = 2, inherit.aes = TRUE, colour = &quot;white&quot;) +
    labs(title = paste0(&quot;Variables Importances. (&quot;, limit, &quot; / &quot;, length(var), &quot; plotted)&quot;))
  
  if (length(unique(output$Type)) == 1) {
    p &lt;- p + geom_col(fill = colours, width = 0.2) +
      geom_point(colour = colours, size = 6) + 
      guides(fill = FALSE, colour = FALSE) + 
      geom_text(hjust = 0.5, size = 2, inherit.aes = TRUE, colour = &quot;white&quot;)
  }
  if(!is.na(model_name)) {
    p &lt;- p + labs(caption = model_name)
  }
  if(!is.na(subtitle)) {
    p &lt;- p + labs(subtitle = subtitle)
  }  
  if(save == TRUE) {
    if (!is.na(subdir)) {
      dir.create(file.path(getwd(), subdir))
      file_name &lt;- paste(subdir, file_name, sep=&quot;/&quot;)
    }
    p &lt;- p + ggsave(file_name, width=7, height=6)
  }
  
  return(p)
  
}
</pre>
<p>Gives this plot:<br>
<a rel="nofollow" href="https://i0.wp.com/datascienceplus.com/wp-content/uploads/2018/07/viz_importance.png?ssl=1" target="_blank"><img sizes="(max-width: 490px) 100vw, 490px" src="https://i2.wp.com/datascienceplus.com/wp-content/uploads/2018/07/viz_importance-490x420.png?w=450&ssl=1" alt="" class="wp-image-15204" data-recalc-dims="1"></a></p>
<p>Hope you guys enjoyed this post and any further comments or suggestions are more than welcome. Not a programmer here but I surely enjoy sharing my code and ideas! Feel free to connect with me in <a rel="nofollow" href="https://www.linkedin.com/in/laresbernardo/" target="_blank">LinkedIn</a> and/or write below in the comments.</p>
<ul></ul>
<p><strong>
<p>Related Post</p>
<p></p></strong>
<ol>
<li><a rel="nofollow" href="https://datascienceplus.com/seaborn-categorical-plots-in-python/" title="Permanent Link toSeaborn Categorical Plots in Python" target="_blank">Seaborn Categorical Plots in Python</a></li>
<li><a rel="nofollow" href="https://datascienceplus.com/matplotlib-library-tutorial-with-examples-python/" title="Permanent Link toMatplotlib Library Tutorial with Examples – Python" target="_blank">Matplotlib Library Tutorial with Examples – Python</a></li>
<li><a rel="nofollow" href="https://datascienceplus.com/visualize-the-world-cup-with-r-part-1-recreating-goals-with-ggsoccer-and-ggplot2/" title="Permanent Link toVisualize the World Cup with R! Part 1: Recreating Goals with ggsoccer and ggplot2" target="_blank">Visualize the World Cup with R! Part 1: Recreating Goals with ggsoccer and ggplot2</a></li>
<li><a rel="nofollow" href="https://datascienceplus.com/creating-slopegraphs-with-r/" title="Permanent Link toCreating Slopegraphs with R" target="_blank">Creating Slopegraphs with R</a></li>
<li><a rel="nofollow" href="https://datascienceplus.com/how-to-use-paletter-to-automagically-build-palettes-from-pictures/" title="Permanent Link toHow to use paletteR to automagically build palettes from pictures" target="_blank">How to use paletteR to automagically build palettes from pictures</a></li>
</ol>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://datascienceplus.com/machine-learning-results-one-plot-to-rule-them-all/"> R Programming – DataScience+</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/C___-z5skqY" height="1"></a></div>