<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/being-successful-on-kaggle-using-mlr/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://mlr-org.github.io/Being-successful-on-Kaggle-using-mlr/"> mlr-org</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Achieving a good score on a Kaggle competition is typically quite difficult.<br>
This blog post outlines 7 tips for beginners to improve their ranking on the Kaggle leaderboards.<br>
For this purpose, I also created a <a target="_blank" rel="nofollow" href="https://www.kaggle.com/casalicchio/bike-sharing-demand/tuning-with-mlr"><em>Kernel</em></a><br>
for the <a target="_blank" rel="nofollow" href="https://www.kaggle.com/casalicchio/bike-sharing-demand"><em>Kaggle bike sharing competition</em></a><br>
that shows how the R package, <code>mlr</code>, can be used to tune a xgboost model with random search in parallel (using 16 cores). The R script scores rank 90 (of 3251) on the Kaggle leaderboard.</p>
<h2>7 Rules</h2>
<ol>
<li>Use good software</li>
<li>Understand the objective</li>
<li>Create and select features</li>
<li>Tune your model</li>
<li>Validate your model</li>
<li>Ensemble different models</li>
<li>Track your progress</li>
</ol>
<h3>1. Use good software</h3>
<p>Whether you choose R, Python or another language to work on Kaggle, you will<br>
most likely need to leverage quite a few packages to follow best practices in<br>
machine learning. To save time, you should use ‘software’<br>
that offers a standardized and well-tested interface for the important steps<br>
in your workflow:</p>
<ul>
<li>Benchmarking different machine learning algorithms (learners)</li>
<li>Optimizing hyperparameters of learners</li>
<li>Feature selection, feature engineering and dealing with missing values</li>
<li>Resampling methods for validation of learner performance</li>
<li>Parallelizing the points above</li>
</ul>
<p>Examples of ‘software’ that implement the steps above and more:</p>
<ul>
<li>For python: scikit-learn (http://scikit-learn.org/stable/auto_examples).</li>
<li>For R: <code>mlr</code> (https://mlr-org.github.io/mlr-tutorial) or <code>caret</code>.</li>
</ul>
<h3>2. Understand the objective</h3>
<p>To develop a good understanding of the Kaggle challenge, you should:</p>
<ul>
<li>Understand the problem domain:
<ul>
<li>Read the description and try to understand the aim of the competition.</li>
<li>Keep reading the forum and looking into scripts/kernels of others, learn from them!</li>
<li>Domain knowledge might help you (i.e., read publications about the topic, wikipedia is also ok).</li>
<li>Use external data if allowed (e.g., google trends, historical weather data).</li>
</ul>
</li>
<li>Explore the dataset:
<ul>
<li>Which features are numerical, categorical, ordinal or time dependent?</li>
<li>Decide how to handle <a target="_blank" rel="nofollow" href="https://mlr-org.github.io/mlr-tutorial/devel/html/impute/index.html"><em>missing values</em></a>. Some options:
<ul>
<li>Impute missing values with the mean, median or with values that are out of range (for numerical features).</li>
<li>Interpolate missing values if the feature is time dependent.</li>
<li>Introduce a new category for the missing values or use the mode (for categorical features).</li>
</ul>
</li>
<li>Do exploratory data analysis (for the lazy: wait until someone else uploads an EDA kernel).</li>
<li>Insights you learn here will inform the rest of your workflow (creating new features).</li>
</ul>
</li>
</ul>
<p>Make sure you choose an approach that directly optimizes the measure of interest!<br>
Example:</p>
<ul>
<li>The <strong>median</strong> minimizes the mean absolute error <strong>(MAE)</strong> and<br>
the <strong>mean</strong> minimizes the mean squared error <strong>(MSE)</strong>.</li>
<li>By default, many regression algorithms predict the expected <strong>mean</strong> but there<br>
are counterparts that predict the expected <strong>median</strong><br>
(e.g., linear regression vs. quantile regression).<br>
<br>
<br>
</li>
<li>For strange measures: Use algorithms where you can implement your own objective<br>
function, see e.g.
<ul>
<li><a target="_blank" rel="nofollow" href="https://www.kaggle.com/casalicchio/allstate-claims-severity/tuning-the-parameter-of-a-custom-objective-1120"><em>tuning parameters of a custom objective</em></a> or</li>
<li><a target="_blank" rel="nofollow" href="https://github.com/tqchen/xgboost/tree/master/demo#features-walkthrough"><em>customize loss function, and evaluation metric</em></a>.</li>
</ul>
</li>
</ul>
<h3>3. Create and select features:</h3>
<p>In many kaggle competitions, finding a “magic feature” can dramatically increase your ranking.<br>
Sometimes, better data beats better algorithms!<br>
You should therefore try to introduce new features containing valuable information<br>
(which can’t be found by the model) or remove noisy features (which can decrease model performance):</p>
<ul>
<li>Concat several columns</li>
<li>Multiply/Add several numerical columns</li>
<li>Count NAs per row</li>
<li>Create dummy features from factor columns</li>
<li>For time series, you could try
<ul>
<li>to add the weekday as new feature</li>
<li>to use rolling mean or median of any other numerical feature</li>
<li>to add features with a lag…</li>
</ul>
</li>
<li>Remove noisy features: <a target="_blank" rel="nofollow" href="https://mlr-org.github.io/mlr-tutorial/devel/html/feature_selection/index.html"><em>Feature selection / filtering</em></a></li>
</ul>
<h3>4. Tune your model</h3>
<p>Typically you can focus on a single model (e.g. <a target="_blank" rel="nofollow" href="https://xgboost.readthedocs.io/en/latest"><em>xgboost</em></a>) and tune its hyperparameters for optimal performance.</p>
<ul>
<li>Aim:<br>
Find the best hyperparameters that, for the given data set, optimize the pre-defined performance measure.</li>
<li>Problem:<br>
Some models have many hyperparameters that can be tuned.</li>
<li>Possible solutions:
<ul>
<li><a target="_blank" rel="nofollow" href="https://mlr-org.github.io/mlr-tutorial/devel/html/tune/index.html"><em>Grid search or random search</em></a></li>
<li>Advanced procedures such as <a target="_blank" rel="nofollow" href="https://mlr-org.github.io/mlr-tutorial/devel/html/advanced_tune/index.html"><em>irace</em></a><br>
or <a target="_blank" rel="nofollow" href="https://mlr-org.github.io/mlrMBO/articles/mlrMBO.html"><em>mbo (bayesian optimization)</em></a></li>
</ul>
</li>
</ul>
<h3>5. Validate your model</h3>
<p>Good machine learning models not only work on the data they were trained on, but<br>
also on unseen (test) data that was not used for training the model. When you use training data<br>
to make any kind of decision (like feature or model selection, hyperparameter tuning, …),<br>
the data becomes less valuable for generalization to unseen data. So if you just use the public<br>
leaderboard for testing, you might overfit to the public leaderboard and lose many ranks once the private<br>
leaderboard is revealed.<br>
A better approach is to use validation to get an estimate of performane on unseen data:</p>
<ul>
<li>First figure out how the Kaggle data was split into train and test data. Your resampling strategy should follow the same method if possible. So if kaggle uses, e.g. a feature for splitting the data, you should not use random samples for creating cross-validation folds.</li>
<li>Set up a <a target="_blank" rel="nofollow" href="https://mlr-org.github.io/mlr-tutorial/devel/html/resample"><em>resampling procedure</em></a>, e.g., cross-validation (CV) to measure your model performance</li>
<li>Improvements on your local CV score should also lead to improvements on the leaderboard.</li>
<li>If this is not the case, you can try
<ul>
<li>several CV folds (e.g., 3-fold, 5-fold, 8-fold)</li>
<li>repeated CV (e.g., 3 times 3-fold, 3 times 5-fold)</li>
<li>stratified CV</li>
</ul>
</li>
<li><code>mlr</code> offers nice <a target="_blank" rel="nofollow" href="https://mlr-org.github.io/mlr-tutorial/devel/html/benchmark_experiments/index.html#benchmark-analysis-and-visualization"><em>visualizations to benchmark</em></a> different algorithms.</li>
</ul>
<h3>6. Ensemble <strong>different</strong> models (see, e.g. <a target="_blank" rel="nofollow" href="http://mlwave.com/kaggle-ensembling-guide"><em>this guide</em></a>):</h3>
<p>After training many different models, you might want to ensemble them into one strong model using one of these methods:</p>
<ul>
<li>simple averaging or voting</li>
<li>finding optimal weights for averaging or voting</li>
<li>stacking</li>
</ul>
<h3>7. Track your progress</h3>
<p>A kaggle project might get quite messy very quickly, because you might try and prototype<br>
many different ideas. To avoid getting lost, make sure to keep track of:</p>
<ul>
<li>What preprocessing steps were used to create the data</li>
<li>What model was used for each step</li>
<li>What values were predicted in the test file</li>
<li>What local score did the model achieve</li>
<li>What public score did the model achieve</li>
</ul>
<p>If you do not want to use a tool like git, at least make sure you create subfolders<br>
for each prototype. This way you can later analyse which models you might want to ensemble<br>
or use for your final commits for the competition.</p>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://mlr-org.github.io/Being-successful-on-Kaggle-using-mlr/"> mlr-org</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/uBDTNJl5Ye4"></a></div>