<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/a-performance-benchmark-of-different-automl-frameworks/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://www.statworx.com/de/blog/a-performance-benchmark-of-different-automl-frameworks/"> r-bloggers – STATWORX</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>In a recent <a rel="nofollow" href="https://www.statworx.com/blog/a-performance-benchmark-of-google-automl-vision-using-fashion-mnist/" target="_blank">blog post</a> our CEO Sebastian Heinz wrote about Google's newest stroke of genius – <a rel="nofollow" href="https://cloud.google.com/vision/automl/docs/" target="_blank">AutoML Vision</a>. A cloud service &quot;<em>that is able to build deep learning models for image recognition completely fully automated and from scratch</em>&quot;. AutoML Vision is part of the <a rel="nofollow" href="https://www.h2o.ai/blog/the-different-flavors-of-automl/" target="_blank">current trend</a> towards the automation of machine learning tasks. This trend started with automation of hyperparameter optimization for single models (Including services like <a rel="nofollow" href="https://sigopt.com/" target="_blank">SigOpt</a>, <a rel="nofollow" href="http://jaberg.github.io/hyperopt/" target="_blank">Hyperopt</a>, <a rel="nofollow" href="https://github.com/automl/SMAC3" target="_blank">SMAC</a>), went along with automated feature engineering and selection (see my colleague <a rel="nofollow" href="https://www.statworx.com/blog/data-science/automated-feature-selection-using-bouncer" target="_blank">Lukas' blog post</a> about our <a rel="nofollow" href="https://github.com/STATWORX/bounceR" target="_blank">bounceR</a> package) towards full automation of complete data pipelines including automated model stacking (a common model ensembling technique). </p>
<p>One company at the frontier of this development is certainly <a rel="nofollow" href="https://www.statworx.com/de/blog/a-performance-benchmark-of-different-automl-frameworks/www.h2o.ai" target="_blank">h2o.ai</a>. They developed both a free Python/R library (<a rel="nofollow" href="http://docs.h2o.ai/h2o/latest-stable/h2o-docs/automl.html?_ga=2.24204030.573118300.1535804795-2033992266.1523379628" target="_blank">H2O AutoML</a>) as well as an enterprise ready software solution called <a rel="nofollow" href="https://www.h2o.ai/products/h2o-driverless-ai/" target="_blank">Driverless AI</a>. But H2O is by far not the only player on the field. This blog post will provide you with a short comparison between two freely available Auto ML solutions and compare them by predictive performance as well as general usability. </p>
<h2>H2O AutoML</h2>
<p><a rel="nofollow" href="http://docs.h2o.ai/h2o/latest-stable/h2o-docs/automl.html" target="_blank">H2O AutoML</a> is an extension to H2O's popular java based open source machine learning framework with APIs for Python and R. It automatically trains, tunes and cross-validates models (including <a rel="nofollow" href="https://www.statworx.com/blog/pushing-ordinary-least-squares-to-the-limit-with-xy/" target="_blank">Generalized Linear Models [GLM]</a>, <a rel="nofollow" href="https://www.statworx.com/blog/xgboost-tree-vs-linear/" target="_blank">Gradient Boosting Machines [GBM]</a>, Random Forest [RF], <a rel="nofollow" href="https://www.statworx.com/blog/using-machine-learning-for-causal-inference/" target="_blank">Extremely Randomized Forest [XRF]</a>, and <a rel="nofollow" href="https://www.statworx.com/de/blog/wie-lernen-neuronale-netze/" target="_blank">Neural Networks</a>). Hyperparameter optimization is done using a random search over a list of reasonable parameters (both RF and XRF are currently not tuned). In the end, H2O produces a leaderboard of models and builds two types of stacked ensembles from the base models. One including all base models, the other including only the best base model of each family. </p>
<p>Model training can be controlled by either the number of models to be trained, or the total training time. Especially the later makes model training quite transparent. One of the big advantages of H2O is that all models are parallelized out-of-the-box. </p>
<h2>auto-sklearn</h2>
<p><a rel="nofollow" href="https://automl.github.io/auto-sklearn/stable/" target="_blank">auto-sklearn</a> is an automated machine learning toolkit based on Python's Scikit-Learn Library. A detailed explanation of auto-sklearn can be found in Feurer et al. (2015). In H2O AutoML, each model was independently tuned and added to a leaderboard. In auto-sklearn, the authors combine model selection and hyperparameter optimization in what they call &quot;<em>Combined Algorithm Selection and Hyperparameter optimization</em>&quot; (CASH). This joint optimization problem is than solved using a tree-based Bayesian optimization methods called &quot;<em>Sequential Model-based Algorithm Configuration</em>&quot; (SMAC) (see Bergstra 2011).</p>
<p>So contrary to H2O AutoML, auto-sklearn optimizes a complete modeling pipeline including various data and feature preprocessing steps as well as the model selection and hyperparameter optimization. Data preprocessing includes one-hot-encoding, scaling, imputation, and balancing. Feature preprocessing includes, among others, feature agglomeration, ICA and PCA. Algorithms included in auto-sklearn are similar to those in H2O AutoML, but in addition also includes more traditional methods like k-Nearest-Neighbors (kNN), Naive Bayes, and Support Vector Machines (SVM). </p>
<p>Similar to H2O AutoML, auto-sklearn includes a final model ensemble step. Whereas H2O AutoML uses simple but efficient model stacking, auto-sklearn uses ensemble selection. A greedy method that adds individual models iteratively to the ensemble if and only if they increase the validation performance. Like H2O, auto-sklearn allows model training to be controlled by the total training time.</p>
<h2>Benchmark</h2>
<p>In order to compare the predictive performance of H2O's AutoML with auto-sklearn, one can conduct a small simulation study. My colleague André's R package <a rel="nofollow" href="https://www.statworx.com/blog/benchmarking-feature-selection-algorithms-with-xy/" target="_blank">Xy</a> offers a straightforward way to simulate regression datasets with linear, non-linear, and noisy relationships. Using multiple (ten in total) simulation runs makes the whole simulation a bit more robust. The following R code was used to simulate the data: </p>
<pre><code>library(Xy)
library(caret)
library(dplyr)
library(data.table)

# Number of datasets
n_data_set &lt;- 10

for (i in seq(n_data_set)) {
  
  # Sim settings
  n &lt;- floor(runif(1, 1000, 5000))
  n_num_vars &lt;- c(sample(2:10, 1), sample(2:10, 1))
  n_cat_vars &lt;- c(0, 0)
  n_noise_vars &lt;- sample(1:5, 1)
  inter_degree &lt;- sample(2:3, 1)
  
  # Simulate data
  sim &lt;- Xy(n = n,   
            numvars = n_num_vars,
            catvars = n_cat_vars, 
            noisevars = n_noise_vars,   
            task = Xy_task(),
            nlfun = function(x) {x^2},
            interactions = 1,
            sig = c(1,4), 
            cor = c(0),
            weights = c(-10,10),
            intercept = TRUE,
            stn = 4)
  
  # Get data and DGP
  df &lt;- sim$data
  dgp &lt;- sim$dgp
  
  # Remove Intercept
  df[, &quot;(Intercept)&quot;] &lt;- NULL
  
  # Rename columns
  names(df) &lt;- gsub(&quot;(?
<p>Since auto-sklearn is only available in Python, switching languages is necessary. Therefore, loading the raw data in Python is the next step:</p>
<pre><code lang="python">import pandas as pd

# Load data
df_train = pd.read_csv(&quot;../data/Xy/1_train.csv&quot;)
df_test = pd.read_csv(&quot;../data/Xy/1_test.csv&quot;)

# Columns
cols_train = df_train.columns.tolist()
cols_test = df_test.columns.tolist()

# Target and features
y_train = df_train.loc[:, &quot;label&quot;]
X_train = df_train.drop(&quot;label&quot;, axis=1)

y_test = df_test.loc[:, &quot;label&quot;]
X_test = df_test.drop(&quot;label&quot;, axis=1)
</code></pre>
<p>Having the data in Python, the training procedure can start. In order to make the results comparable, both frameworks used, where possible, similar settings. This included 60 minutes of training for each dataset, 5-fold crossvalidation for model evaluation and ensemble building, no preprocessing (not available in H2O AutoML and therefore deactivated in auto-sklearn), and a limitation to similar algorithms (namely GLM, RF, XRF, and GBM). </p>
<p>As previously noted, H2O supports out-of-the-box parallelization. By default, auto-sklearn only uses two cores, while also supporting more cores, at least in theory. While there is a <a rel="nofollow" href="https://automl.github.io/auto-sklearn/stable/manual.html#parallel-computation" target="_blank">manual</a> on how to do that, I was not able to get it working on my system (OSX 10.13, Python 3.6.2 Anaconda). Therefore H2O was also limited to only two cores. </p>
<pre><code lang="python">from autosklearn.regression import AutoSklearnRegressor
from autosklearn.metrics import mean_squared_error

# Settings
estimators_to_use = [&quot;random_forest&quot;, &quot;extra_trees&quot;, &quot;gradient_boosting&quot;, &quot;ridge_regression&quot;]
preprocessing_to_use = [&quot;no_preprocessing&quot;]

# Init auto-sklearn
auto_sklearn = AutoSklearnRegressor(time_left_for_this_task=60*60,
                                    per_run_time_limit=360,
                                    include_estimators=estimators_to_use,
                                    exclude_estimators=None,
                                    include_preprocessors=preprocessing_to_use,
                                    exclude_preprocessors=None,
                                    ml_memory_limit=6156,
                                    resampling_strategy=&quot;cv&quot;,
                                    resampling_strategy_arguments={&quot;folds&quot;: 5})

# Train models
auto_sklearn.fit(X=X_train.copy(), y=y_train.copy(), metric=mean_squared_error)
it_fits = auto_sklearn.refit(X=X_train.copy(), y=y_train.copy())

# Predict
y_hat = auto_sklearn.predict(X_test)

# Show results
auto_sklearn.cv_results_
auto_sklearn.sprint_statistics()
auto_sklearn.show_models()
auto_sklearn.get_models_with_weights()
</code></pre>
<pre><code lang="python">import h2o
from h2o.automl import H2OAutoML

# Shart h2o cluster
h2o.init(max_mem_size=&quot;8G&quot;, nthreads=2)

# Upload to h2o
df_train_h2o = h2o.H2OFrame(pd.concat([X_train, pd.DataFrame({&quot;target&quot;: y_train})], axis=1))
df_test_h2o = h2o.H2OFrame(X_test)

features = X_train.columns.values.tolist()
target = &quot;target&quot;

# Training
auto_h2o = H2OAutoML(max_runtime_secs=60*60)
auto_h2o.train(x=features,
               y=target,
               training_frame=df_train_h2o)

# Leaderboard
auto_h2o.leaderboard
auto_h2o = auto_h2o.leader

# Testing
df_test_hat = auto_h2o.predict(df_test_h2o)
y_hat = h2o.as_list(df_test_hat[&quot;predict&quot;])

# Close cluster
h2o.cluster().shutdown()
</code></pre>
<p>The complete code, including all simulation runs and visualization of results can be find on my <a rel="nofollow" href="https://github.com/fabianmax/ML-Automation" target="_blank">GitHub repo</a>. </p>
<h2>Results</h2>
<p>First, some words of caution: The results presented in the next sections are by no mean representative. Both H2O and the authors of auto-sklearn recommend to run their frameworks for hours, if not even days. Given ten different datasets, this was beyond the scope of a blog post. For the same reason of feasibility, the datasets are restricted to a rather small size. For a more elaborated performance comparison see for example Balaji and Allen (2018). </p>
<p>Figure 1 shows the Mean Squared Error of both frameworks produced on the test sample. The horizontal line, indicating the result from a vanilla Random Forest (from scikit-learn), serves as a benchmark. As one can see, the results are pretty similar for both frameworks and all data sets. Actually, it is a tie, with five wins for H2O and five wins for auto-sklearn. </p>
<p><img alt="results ml benchmark" src="https://i0.wp.com/www.statworx.com/wp-content/uploads/results-ml-benchmark.png?w=450&ssl=1" class="wp-image-15313" data-recalc-dims="1"></p>
<p>The percentage difference between the average errors is <img src="https://i0.wp.com/www.statworx.com/wp-content/ql-cache/quicklatex.com-849f21d8ce0018dcc78e7691f43931d1_l3.png?resize=45%2C15&ssl=1" alt="1.04%" width="45" title="Rendered by QuickLaTeX.com" data-recalc-dims="1" height="15"> in favor of auto-sklearn. Thus, auto-sklearn is on average about <img src="https://i1.wp.com/www.statworx.com/wp-content/ql-cache/quicklatex.com-955c7a8b67d095595960f12c4515296a_l3.png?resize=22%2C15&ssl=1" alt="1%" width="22" title="Rendered by QuickLaTeX.com" data-recalc-dims="1" height="15"> better than H2O. Compared with the vanilla RF, H2O's AutoML is on average <img src="https://i1.wp.com/www.statworx.com/wp-content/ql-cache/quicklatex.com-90f74692d3686e413422283c1c7aa93f_l3.png?resize=46%2C15&ssl=1" alt="23.4%" width="46" title="Rendered by QuickLaTeX.com" data-recalc-dims="1" height="15"> better than the benchmark, while auto-sklearn is <img src="https://i1.wp.com/www.statworx.com/wp-content/ql-cache/quicklatex.com-707c6e57f2d46eae5c7d1943daf0115b_l3.png?resize=46%2C15&ssl=1" alt="24.6%" width="46" title="Rendered by QuickLaTeX.com" data-recalc-dims="1" height="15"> better. </p>
<p>The sheer closeness of the results can be further illustrated when taking a look at the predicted values. Figure 2 shows exemplary the predicted values for one particular dataset against all feature values (linear, non-linear and noise features). As one can see, the estimated effects for both frameworks are almost identical and pretty close to the actual relationship.</p>
<p><img alt="visualization ml benchmark" src="https://i0.wp.com/www.statworx.com/wp-content/uploads/visualization-ml-benchmark.png?w=450&ssl=1" class="wp-image-15316" data-recalc-dims="1"></p>
<h2>Summary</h2>
<p>Automatic Machine Learning frameworks can provide promising results for standard machine learning task while keeping the manual efforts down to a minimum. This blog post compared two popular frameworks, namely H2O's AutoML and auto-sklearn. Both reached comparable results on ten simulated datasets, while outperforming vanilla models significantly. Beside predictive performance, H2O's AutoML offers some additional features like native parallelization, API for R, support for XGBoost and GPU training making it even more attractive. </p>
<h2>References</h2>
<ul>
<li>Feurer, Matthias, Aaron Klein, Katharina Eggensperger, Jost Tobias Springenberg, Manuel Blum, and Frank Hutter 2015. &quot;Efficient and Robust Automated Machine Learning.&quot; NIPS 2015. <a rel="nofollow" href="https://ml.informatik.uni-freiburg.de/papers/15-NIPS-auto-sklearn-preprint.pdf" target="_blank">https://ml.informatik.uni-freiburg.de/papers/15-NIPS-auto-sklearn-preprint.pdf</a></li>
<li>Balaji, Adithya and Alexander Allen. 2018. &quot;Benchmarking Automatic Machine Learning Frameworks.&quot; <a rel="nofollow" href="https://arxiv.org/pdf/1808.06492.pdf" target="_blank">https://arxiv.org/pdf/1808.06492.pdf</a>.</li>
<li>Bergstra, James, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. 2011. &quot;Algorithms for Hyper-Parameter Optimization.&quot; NIPS 2011. <a rel="nofollow" href="https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf" target="_blank">https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf</a></li>
</ul>
<div>
<h6>Über den Autor</h6>
<p><img src="https://i1.wp.com/www.statworx.com/wp-content/uploads/fabian-1-300x300.jpg?resize=180%2C180&ssl=1" width="180" alt="Fabian Müller" class="wp-user-avatar wp-user-avatar-180" data-recalc-dims="1" height="180"></p>
<div>
<h4>Fabian Müller</h4>
<p>Fabian ist unser Teamleiter für Data Science und betreut mit seinem Team unsere Großkunden aus der Wirtschaft. In seiner Freizeit treibt er viel Sport und ist ein großer Automobil-Fan.</p>
</div>
</div>
<p>Der Beitrag <a rel="nofollow" href="https://www.statworx.com/de/blog/a-performance-benchmark-of-different-automl-frameworks/" target="_blank">A Performance Benchmark of Different AutoML Frameworks</a> erschien zuerst auf <a rel="nofollow" href="https://www.statworx.com/de/" target="_blank">STATWORX</a>.</p>

				
		
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://www.statworx.com/de/blog/a-performance-benchmark-of-different-automl-frameworks/"> r-bloggers – STATWORX</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/7AUaonxvdyc" height="1"></a></div></code></pre>