<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/classification-in-spark-2-0-input-validation-failed-and-other-wondrous-tales/"></div></div><div><a data-url="https://www.r-bloggers.com/classification-in-spark-2-0-input-validation-failed-and-other-wondrous-tales/" data-counturl="https://www.r-bloggers.com/classification-in-spark-2-0-input-validation-failed-and-other-wondrous-tales/" data-count="horizontal" data-text="Classification in Spark 2.0: “Input validation failed” and other wondrous tales" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.nodalpoint.com/spark-classification/"> R – Nodalpoint</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Spark 2.0 has been released since last July but, despite the numerous improvements and new features, several annoyances still remain and can cause headaches, especially in the Spark machine learning APIs. Today we’ll have a look at some of them, inspired by <a target="_blank" href="http://stackoverflow.com/questions/39180678/issues-with-logistic-regression-for-multiclass-classification-using-pyspark/39207405#39207405">a recent answer of mine</a> in a Stack Overflow question (the question was about Spark 1.6 but, as we’ll see, the issue remains in Spark 2.0).</p>
<p>We’ll first try a simple binary classification problem in PySpark using <a target="_blank" href="http://spark.apache.org/docs/2.0.0/api/python/pyspark.mllib.html">Spark MLlib</a>, but, before doing so, let’s have a look at the current status of the machine learning APIs in Spark 2.0.</p>
<p>Spark MLlib was the older machine learning API for Spark, intended to be gradually replaced by the newest Spark ML library; in Spark 2.0 this terminology has changed (enough, in my opinion, to cause unnecessary confusion): now the whole machine learning functionality is termed “MLlib”, with the old MLlib being the so-called “RDD-based API”, and the (old) Spark ML library now termed the “<a target="_blank" href="http://spark.apache.org/docs/2.0.0/ml-guide.html">MLlib DataFrame-based API</a>“. The oldest RDD-based API has now entered maintenance mode, heading for gradual deprecation.</p>
<p>Truth is, whatever Databricks and the Spark architects may like to believe, there is some <em>essential</em> machine learning functionality which is still only available in the old MLlib RDD-based API, good examples being multinomial logistic regression and SVM models.</p>
<p>Now that we have hopefully justified interest to the “old” RDD-based API, let us proceed to our first example.</p>
<p>(In the code snippets below, <a target="_blank" href="http://spark.apache.org/docs/2.0.0/api/python/pyspark.mllib.html"><tt>pyspark.mllib </tt></a>corresponds to the old, RDD-based API, while <a target="_blank" href="http://spark.apache.org/docs/2.0.0/api/python/pyspark.ml.html"><tt>pyspark.ml</tt></a> corresponds to the new DataFrame-based API.)</p>
<p>The following code is slightly adapted from the <a target="_blank" href="http://spark.apache.org/docs/2.0.0/api/python/pyspark.mllib.html#pyspark.mllib.classification.LogisticRegressionModel">documentation example</a> of logistic regression in PySpark (can you spot the difference?):</p>
<pre>
>>> print spark.version
2.0.0
>>> from pyspark.mllib.classification import LogisticRegressionModel, LogisticRegressionWithSGD
>>> from pyspark.mllib.regression import LabeledPoint
>>> data = [
...     LabeledPoint(2.0, [0.0, 1.0]),
...     LabeledPoint(1.0, [1.0, 0.0])]
>>> lrm = LogisticRegressionWithSGD.train(sc.parallelize(data), iterations=10)
[...]
: org.apache.spark.SparkException: Input validation failed.
</pre>
<p>As we can see, this simple code snippet gives the super-informative error “<tt>Input validation failed</tt>“. And the situation is the same with other classifiers – here is an SVM with the same data:</p>
<pre>
>>> from pyspark.mllib.classification import SVMModel, SVMWithSGD
>>> svm = SVMWithSGD.train(sc.parallelize(data), iterations=10)
[...]
: org.apache.spark.SparkException: Input validation failed.
</pre>
<p>No matter how long you stare at the relevant <a target="_blank" href="http://spark.apache.org/docs/2.0.0/api/python/pyspark.mllib.html#pyspark.mllib.classification.SVMWithSGD">PySpark documentation</a>, you will not find the error cause, simply because it is <em>not</em> documented – to get a hint, you will have to dig in the <a target="_blank" href="https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/classification/LogisticRegression.scala#L300">Scala (!) source code</a> of <code>LogisticRegressionWithSGD</code>, where it is mentioned:</p>
<pre>
 * NOTE: Labels used in Logistic Regression should be {0, 1}
</pre>
<p>So, in binary classification, your labels cannot be whatever you like, but they need to be {0, 1} or {0.0, 1.0}…</p>
<pre>
>>> data = [
...     LabeledPoint(0.0, [0.0, 1.0]), # changed the label here from 2.0 to 0.0
...     LabeledPoint(1.0, [1.0, 0.0])]
>>> lrm = LogisticRegressionWithSGD.train(sc.parallelize(data), iterations=10) 
>>> lrm.predict([0.2, 0.5])
0
>>> svm = SVMWithSGD.train(sc.parallelize(data), iterations=10)
>>> svm.predict([0.2, 0.5])
0
</pre>
<p>A similar (again, undocumented) constraint applies for multi-class classification, where for k classes you must use the labels {0, 1, …, k-1}.</p>
<p>Interestingly enough, <a target="_blank" href="https://spark.apache.org/docs/latest/sparkr.html">SparkR</a> does not suffer from such a limitation; try the following code interactively from RStudio (where we use the <tt>iris</tt> dataset with one class removed, since <code>spark.glm</code> does not support multi-class classification):</p>
<pre>
library(SparkR, lib.loc = &quot;/home/ctsats/spark-2.0.0-bin-hadoop2.6/R/lib&quot;) # change the path accordingly here

sparkR.session(sparkHome = &quot;/home/ctsats/spark-2.0.0-bin-hadoop2.6&quot;)      # and here

df &lt;- as.DataFrame(iris[1:100,]) # keep 2 classes only
head(df)

model &lt;- spark.glm(df, Species ~ .,  family=&quot;binomial&quot;)
summary(model)

pred &lt;- predict(model, df)
showDF(pred, 100, truncate = FALSE)

sparkR.session.stop()
</pre>
<p>Since we are interested at the (very) basic functionality only, we don’t bother with splitting to training and test sets – we just run the predict function for the dataset we used for training; here is the partial output from <code>showDF</code>:</p>
<pre>
+------------+-----------+------------+-----------+----------+-----+----------------------+
|Sepal_Length|Sepal_Width|Petal_Length|Petal_Width|Species   |label|prediction            |
+------------+-----------+------------+-----------+----------+-----+----------------------+
|5.1         |3.5        |1.4         |0.2        |setosa    |1.0  |0.9999999999999999    |
|4.9         |3.0        |1.4         |0.2        |setosa    |1.0  |0.999999999999992     |
|4.7         |3.2        |1.3         |0.2        |setosa    |1.0  |0.999999999999998     |
|4.6         |3.1        |1.5         |0.2        |setosa    |1.0  |0.9999999999994831    |
[...]
|5.7         |2.9        |4.2         |1.3        |versicolor|0.0  |1.0E-16               |
|6.2         |2.9        |4.3         |1.3        |versicolor|0.0  |1.0E-16               |
|5.1         |2.5        |3.0         |1.1        |versicolor|0.0  |1.9333693386853254E-10|
|5.7         |2.8        |4.1         |1.3        |versicolor|0.0  |1.0E-16               |
+------------+-----------+------------+-----------+----------+-----+----------------------+
</pre>
<p>from which it is apparent that SparkR has performed internally the label mapping <code>setosa -> 1.0</code> and <code>versicolor -> 0.0</code> for us.</p>
<p>There is an explanation for this difference in behavior: under the hood, and unlike the PySpark examples shown above, SparkR uses the newest DataFrame-based API for the machine learning functionality; so, let’s have a quick look at this API from a PySpark point of view as well.</p>
<p>We recreate our first code snippet above, with the data now as a <tt>DataFrame</tt> instead of a <tt>LabeledPoint</tt>:</p>
<pre>
>>> print spark.version
2.0.0
>>> from pyspark.ml.classification import LogisticRegression
>>> from pyspark.ml.linalg import Vectors
>>> df = sqlContext.createDataFrame([
...     (2.0, Vectors.dense(0.0, 1.0)),
...     (1.0, Vectors.dense(1.0, 0.0))], 
...     [&quot;label&quot;, &quot;features&quot;])
>>> df.show()
+-----+---------+
|label| features|
+-----+---------+
|  2.0|[0.0,1.0]|
|  1.0|[1.0,0.0]|
+-----+---------+
>>> lr = LogisticRegression(maxIter=5, regParam=0.01, labelCol=&quot;label&quot;)
>>> model = lr.fit(df)
[...]
: org.apache.spark.SparkException: Currently, LogisticRegression with ElasticNet in ML package only supports binary classification. Found 3 in the input dataset.
</pre>
<p>Well… as you can see, our classifier complains that it has found 3 classes in the data, despite that, evidently, being not the case…</p>
<p>Changing the labels above from {1.0, 2.0} to {0.0, 1.0} resolves this issue (not shown); again, this requirement is nowhere documented in PySpark, and the error message does little to help locate the actual issue.</p>
<p>And here is our last take on weird, counter-intuitive, and undocumented features of the new DataFrame-based machine learning API…</p>
<p>One could easily argue that encoding class labels (i.e. what is actually a <em>factor</em>) with <em>floating-point</em> numbers is unnatural; and the old, RDD-based API sure permits the more natural choice of encoding the class labels as <em>integers</em>, instead:</p>
<pre>
>>> print spark.version
2.0.0
>>> from pyspark.mllib.classification import LogisticRegressionModel, LogisticRegressionWithSGD
>>> from pyspark.mllib.regression import LabeledPoint
>>> data = [
...     LabeledPoint(0, [0.0, 1.0]), # integer labels instead
...     LabeledPoint(1, [1.0, 0.0])] # of float
>>> lrm = LogisticRegressionWithSGD.train(sc.parallelize(data), iterations=10) 
>>> lrm.predict([0.2, 0.5])
0
</pre>
<p>What’s more, the binary <tt>LogisticRegression</tt> classifier from the new, DataFrame-based API also allows for integer labels:</p>
<pre>
>>> print spark.version
2.0.0
>>> from pyspark.ml.classification import LogisticRegression
>>> from pyspark.ml.linalg import Vectors
>>> df = sqlContext.createDataFrame([
...     (0, Vectors.dense(0.0, 1.0)),  # integer labels instead
...     (1, Vectors.dense(1.0, 0.0))], # of float
...     [&quot;label&quot;, &quot;features&quot;])
>>> df.show()
+-----+---------+
|label| features|
+-----+---------+
|    0|[0.0,1.0]|
|    1|[1.0,0.0]|
+-----+---------+
>>> lr = LogisticRegression(maxIter=5, regParam=0.01, labelCol=&quot;label&quot;)
>>> model = lr.fit(df) # works OK
</pre>
<p>But here is what happens when we try a <tt>DecisionTreeClassifier</tt> <em>from the very same module</em> (namely <tt>pyspark.ml.classification</tt>):</p>
<pre>
>>> print spark.version
2.0.0
>>> from pyspark.ml.classification import DecisionTreeClassifier
>>> df = sqlContext.createDataFrame([
...     (0, Vectors.dense(0.0, 1.0)),  # integer labels instead
...     (1, Vectors.dense(1.0, 0.0))], # of float
...     [&quot;label&quot;, &quot;features&quot;])
>>> dt = DecisionTreeClassifier(maxDepth=2, labelCol=&quot;label&quot;)
>>> model = dt.fit(df) 
[...]
: java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.Double
</pre>
<p>As you might have guessed by now, changing the labels back to floating-point numbers resolves the issue – but if you expect to find a reference in the relevant <a target="_blank" href="http://spark.apache.org/docs/2.0.0/api/python/pyspark.ml.html#pyspark.ml.classification.DecisionTreeClassifier">PySpark documentation</a>, or at least a hint in the general <a target="_blank" href="http://spark.apache.org/docs/2.0.0/ml-guide.html">Spark Machine Learning Library Guide</a>, well, good luck…</p>
<p>* * *</p>
<p>I will argue that</p>
<ul>
<li>Such unexpected and counter-intuitive behavior in Spark abounds</li>
<li>The documentation, especially for the Python API (PySpark), is hopelessly uninformative at such issues</li>
<li>This can cause considerable strain and frustration to both novice and seasoned data scientists alike, especially since such users are naturally expected to rely on PySpark, rather than the Scala or Java APIs</li>
</ul>
<p>Consider the following issue:</p>
<pre>
>>> print spark.version
2.0.0
>>> from pyspark.ml.linalg import Vectors
>>> x = Vectors.dense([0.0, 1.0])
>>> x
DenseVector([0.0, 1.0])
>>> -x
Traceback (most recent call last):
  File &quot;&lt;stdin>&quot;, line 1, in &lt;module>
TypeError: func() takes exactly 2 arguments (1 given)
</pre>
<p>I have <a target="_blank" href="http://www.nodalpoint.com/limitations-of-spark-mllib-local-linear-algebra-module/">written in the past</a> about this, but back then it concerned the “old” <tt>pyspark.mllib.linalg</tt> module; the reaction from the Spark community clearly implied something in the lines of “<em>It is well-known that […]</em>“; as I counter-argued (after I did my research), since it seems to be not <em>so well-known</em>, we might even consider adding it to the documentation – so I opened a <em>documentation</em> issue in <a target="_blank" href="https://issues.apache.org/jira/browse/SPARK-12372">Spark JIRA</a>. Not only it remains unresolved, but, as I have just shown above, the same behavior has been inherited by the newer <tt>pyspark.ml.linalg</tt> module, again without any relevant mention in the <a target="_blank" href="http://spark.apache.org/docs/2.0.0/api/python/pyspark.ml.html#module-pyspark.ml.linalg">documentation</a>.</p>
<p>Wondrous tales indeed…</p>
<p>The post <a target="_blank" rel="nofollow" href="http://www.nodalpoint.com/spark-classification/">Classification in Spark 2.0: “Input validation failed” and other wondrous tales</a> appeared first on <a target="_blank" rel="nofollow" href="http://www.nodalpoint.com/">Nodalpoint</a>.</p>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="https://www.r-bloggers.com/classification-in-spark-2-0-input-validation-failed-and-other-wondrous-tales/"></div></div><div><a data-url="https://www.r-bloggers.com/classification-in-spark-2-0-input-validation-failed-and-other-wondrous-tales/" data-counturl="https://www.r-bloggers.com/classification-in-spark-2-0-input-validation-failed-and-other-wondrous-tales/" data-count="vertical" data-text="Classification in Spark 2.0: “Input validation failed” and other wondrous tales" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.nodalpoint.com/spark-classification/"> R – Nodalpoint</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/NT_3p6X02Po"></a></div>