<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/big-data-2-move-into-the-big-leaguegraduate-from-r-to-sparkr/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://gigadom.in/2018/10/09/big-data-2-move-into-the-big-leaguegraduate-from-r-to-sparkr/"> R – Giga thoughts …</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<div>
<div></div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p>This post is a continuation of my earlier post <a rel="nofollow" href="https://gigadom.in/2018/10/08/big-data-1-move-into-the-big-leaguegraduate-from-python-to-pyspark/" target="_blank">Big Data-1: Move into the big league:Graduate from Python to Pyspark</a>. While the earlier post discussed parallel constructs in Python and Pyspark, this post elaborates similar and key constructs in R and SparkR. While this post just focuses on the programming part of R and SparkR it is essential to understand and fully grasp the concept of Spark, RDD and how data is distributed across the clusters. This post like the earlier post shows how if you already have a good handle of R, you can easily graduate to Big Data with SparkR</p>
<p><strong>Note 1</strong>: This notebook has also been published at Databricks community site <a rel="nofollow" href="https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/3582687578800188/4275120876911869/4574970912889634/latest.html" target="_blank">Big Data-2: Move into the big league:Graduate from R to SparkR</a></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div><strong>Note 2</strong>: You can download this RMarkdown file from Github at <a rel="nofollow" href="https://github.com/tvganesh/Big-Data-Python-to-Pyspark-and-R-to-SparkR" target="_blank">Big Data- Python to Pyspark and R to SparkR</a></div>
<div><span>1a. Read CSV- R</span></div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p>Note: To upload the CSV to databricks see the video <a rel="nofollow" href="https://www.youtube.com/watch?v=H5LxjaJgpSk" target="_blank">Upload Flat File to Databricks Table</a></p>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<pre><span># Read CSV file</span>
<span>tendulkar</span><span>=</span> <span>read.csv</span>(<span>&quot;</span><span>/dbfs/FileStore/tables/tendulkar.csv&quot;</span>,<span>stringsAsFactors</span> <span>=</span> <span>FALSE</span>,<span>na.strings</span><span>=</span><span>c</span>(<span>NA</span>,<span>&quot;</span><span>-&quot;</span>))
<span>#Check the dimensions of the dataframe</span>
<span>dim</span>(<span>tendulkar</span>)
</pre>
</div>
</div>
</div>
<div>
<div>
<div>
<div></div>
<div>
<pre>[1] 347  12</pre>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div><span>1b. Read CSV – SparkR</span></div>
<div>
<div></div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<pre><span># Load the SparkR library</span>
<span>library</span>(<span>SparkR</span>)
<span># Initiate a SparkR session</span>
<span>sparkR.session</span>()
<span>tendulkar1</span> <span>&lt;-</span> <span>read.df</span>(<span>&quot;</span><span>/FileStore/tables/tendulkar.csv&quot;</span>, 
                <span>header</span> <span>=</span> <span>&quot;</span><span>true&quot;</span>, 
                <span>delimiter</span> <span>=</span> <span>&quot;</span><span>,&quot;</span>, 
                <span>source</span> <span>=</span> <span>&quot;</span><span>csv&quot;</span>, 
                <span>inferSchema</span> <span>=</span> <span>&quot;</span><span>true&quot;</span>, 
                <span>na.strings</span> <span>=</span> <span>&quot;</span><span>&quot;</span>)

<span># Check the dimensions of the dataframe</span>
<span>dim</span>(<span>tendulkar1</span>)
</pre>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<pre>[1] 347  12</pre>
</div>
</div>
</div>
<div><span>2a. Data frame shape – R</span></div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<pre><span># Get the shape of the dataframe in R</span>
<span>dim</span>(<span>tendulkar</span>)
</pre>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<pre>[1] 347  12</pre>
</div>
</div>
</div>
<div><span>2b. Dataframe shape – SparkR</span></div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p>The same ‘dim’ command works in SparkR too!</p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<pre><span>dim</span>(<span>tendulkar1</span>)
</pre>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<pre>[1] 347  12</pre>
</div>
</div>
</div>
<div><span>3a . Dataframe columns – R</span></div>
</div>
</div>
</div>
<div>
<div>
<div>
<div></div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<pre><span># Get the names</span>
<span>names</span>(<span>tendulkar</span>) <span># Also colnames(tendulkar)</span>
</pre>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<pre> [1] &quot;Runs&quot;       &quot;Mins&quot;       &quot;BF&quot;         &quot;X4s&quot;        &quot;X6s&quot;       
 [6] &quot;SR&quot;         &quot;Pos&quot;        &quot;Dismissal&quot;  &quot;Inns&quot;       &quot;Opposition&quot;
[11] &quot;Ground&quot;     &quot;Start.Date&quot;</pre>
</div>
</div>
</div>
<div><span>3b. Dataframe columns – SparkR</span></div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<pre><span>names</span>(<span>tendulkar1</span>)
</pre>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<pre> [1] &quot;Runs&quot;       &quot;Mins&quot;       &quot;BF&quot;         &quot;4s&quot;         &quot;6s&quot;        
 [6] &quot;SR&quot;         &quot;Pos&quot;        &quot;Dismissal&quot;  &quot;Inns&quot;       &quot;Opposition&quot;
[11] &quot;Ground&quot;     &quot;Start Date&quot;</pre>
</div>
</div>
</div>
<div><span>4a. Rename columns – R</span></div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<pre><span>names</span>(<span>tendulkar</span>)<span>=</span><span>c</span>(<span>'</span><span>Runs'</span>,<span>'</span><span>Minutes'</span>,<span>'</span><span>BallsFaced'</span>,<span>'</span><span>Fours'</span>,<span>'</span><span>Sixes'</span>,<span>'</span><span>StrikeRate'</span>,<span>'</span><span>Position'</span>,<span>'</span><span>Dismissal'</span>,<span>'</span><span>Innings'</span>,<span>'</span><span>Opposition'</span>,<span>'</span><span>Ground'</span>,<span>'</span><span>StartDate'</span>)
<span>names</span>(<span>tendulkar</span>)
</pre>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<pre> [1] &quot;Runs&quot;       &quot;Minutes&quot;    &quot;BallsFaced&quot; &quot;Fours&quot;      &quot;Sixes&quot;     
 [6] &quot;StrikeRate&quot; &quot;Position&quot;   &quot;Dismissal&quot;  &quot;Innings&quot;    &quot;Opposition&quot;
[11] &quot;Ground&quot;     &quot;StartDate&quot;</pre>
</div>
</div>
</div>
<div><span>4b. Rename columns – R</span></div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<pre><span>names</span>(<span>tendulkar1</span>)<span>=</span><span>c</span>(<span>'</span><span>Runs'</span>,<span>'</span><span>Minutes'</span>,<span>'</span><span>BallsFaced'</span>,<span>'</span><span>Fours'</span>,<span>'</span><span>Sixes'</span>,<span>'</span><span>StrikeRate'</span>,<span>'</span><span>Position'</span>,<span>'</span><span>Dismissal'</span>,<span>'</span><span>Innings'</span>,<span>'</span><span>Opposition'</span>,<span>'</span><span>Ground'</span>,<span>'</span><span>StartDate'</span>)
<span>names</span>(<span>tendulkar1</span>)
</pre>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<pre> [1] &quot;Runs&quot;       &quot;Minutes&quot;    &quot;BallsFaced&quot; &quot;Fours&quot;      &quot;Sixes&quot;     
 [6] &quot;StrikeRate&quot; &quot;Position&quot;   &quot;Dismissal&quot;  &quot;Innings&quot;    &quot;Opposition&quot;
[11] &quot;Ground&quot;     &quot;StartDate&quot;</pre>
</div>
</div>
</div>
<div><span>5a. Summary – R</span></div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<pre><span>summary</span>(<span>tendulkar</span>)
</pre>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<pre>     Runs              Minutes        BallsFaced         Fours       
 Length:347         Min.   :  1.0   Min.   :  0.00   Min.   : 0.000  
 Class :character   1st Qu.: 33.0   1st Qu.: 22.00   1st Qu.: 1.000  
 Mode  :character   Median : 82.0   Median : 58.50   Median : 4.000  
                    Mean   :125.5   Mean   : 89.75   Mean   : 6.274  
                    3rd Qu.:181.0   3rd Qu.:133.25   3rd Qu.: 9.000  
                    Max.   :613.0   Max.   :436.00   Max.   :35.000  
                    NA's   :18      NA's   :19       NA's   :19      
     Sixes          StrikeRate        Position     Dismissal        
 Min.   :0.0000   Min.   :  0.00   Min.   :2.00   Length:347        
 1st Qu.:0.0000   1st Qu.: 38.09   1st Qu.:4.00   Class :character  
 Median :0.0000   Median : 52.25   Median :4.00   Mode  :character  
 Mean   :0.2097   Mean   : 51.79   Mean   :4.24                     
 3rd Qu.:0.0000   3rd Qu.: 65.09   3rd Qu.:4.00                     
 Max.   :4.0000   Max.   :166.66   Max.   :7.00                     
 NA's   :18       NA's   :20       NA's   :18                       
    Innings       Opposition           Ground           StartDate        
 Min.   :1.000   Length:347         Length:347         Length:347        
 1st Qu.:1.000   Class :character   Class :character   Class :character  
 Median :2.000   Mode  :character   Mode  :character   Mode  :character  
 Mean   :2.376                                                           
 3rd Qu.:3.000                                                           
 Max.   :4.000                                                           
 NA's   :1</pre>
</div>
</div>
</div>
<div><span>5b. Summary – SparkR</span></div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<pre><span>summary</span>(<span>tendulkar1</span>)
</pre>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<pre>SparkDataFrame[summary:string, Runs:string, Minutes:string, BallsFaced:string, Fours:string, Sixes:string, StrikeRate:string, Position:string, Dismissal:string, Innings:string, Opposition:string, Ground:string, StartDate:string]</pre>
</div>
</div>
</div>
<div><span>6a. Displaying details of dataframe with str() – R</span></div>
</div>
</div>
</div>
<div>
<div>
<div>
<div></div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<pre><span>str</span>(<span>tendulkar</span>)
</pre>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<pre>'data.frame':	347 obs. of  12 variables:
 $ Runs      : chr  &quot;15&quot; &quot;DNB&quot; &quot;59&quot; &quot;8&quot; ...
 $ Minutes   : int  28 NA 254 24 124 74 193 1 50 324 ...
 $ BallsFaced: int  24 NA 172 16 90 51 134 1 44 266 ...
 $ Fours     : int  2 NA 4 1 5 5 6 0 3 5 ...
 $ Sixes     : int  0 NA 0 0 0 0 0 0 0 0 ...
 $ StrikeRate: num  62.5 NA 34.3 50 45.5 ...
 $ Position  : int  6 NA 6 6 7 6 6 6 6 6 ...
 $ Dismissal : chr  &quot;bowled&quot; NA &quot;lbw&quot; &quot;run out&quot; ...
 $ Innings   : int  2 4 1 3 1 1 3 2 3 1 ...
 $ Opposition: chr  &quot;v Pakistan&quot; &quot;v Pakistan&quot; &quot;v Pakistan&quot; &quot;v Pakistan&quot; ...
 $ Ground    : chr  &quot;Karachi&quot; &quot;Karachi&quot; &quot;Faisalabad&quot; &quot;Faisalabad&quot; ...
 $ StartDate : chr  &quot;15-Nov-89&quot; &quot;15-Nov-89&quot; &quot;23-Nov-89&quot; &quot;23-Nov-89&quot; ...</pre>
</div>
</div>
</div>
<div><span>6b. Displaying details of dataframe with str() – SparkR</span></div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<pre><span>str</span>(<span>tendulkar1</span>)
</pre>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<pre>'SparkDataFrame': 12 variables:
 $ Runs      : chr &quot;15&quot; &quot;DNB&quot; &quot;59&quot; &quot;8&quot; &quot;41&quot; &quot;35&quot;
 $ Minutes   : chr &quot;28&quot; &quot;-&quot; &quot;254&quot; &quot;24&quot; &quot;124&quot; &quot;74&quot;
 $ BallsFaced: chr &quot;24&quot; &quot;-&quot; &quot;172&quot; &quot;16&quot; &quot;90&quot; &quot;51&quot;
 $ Fours     : chr &quot;2&quot; &quot;-&quot; &quot;4&quot; &quot;1&quot; &quot;5&quot; &quot;5&quot;
 $ Sixes     : chr &quot;0&quot; &quot;-&quot; &quot;0&quot; &quot;0&quot; &quot;0&quot; &quot;0&quot;
 $ StrikeRate: chr &quot;62.5&quot; &quot;-&quot; &quot;34.3&quot; &quot;50&quot; &quot;45.55&quot; &quot;68.62&quot;
 $ Position  : chr &quot;6&quot; &quot;-&quot; &quot;6&quot; &quot;6&quot; &quot;7&quot; &quot;6&quot;
 $ Dismissal : chr &quot;bowled&quot; &quot;-&quot; &quot;lbw&quot; &quot;run out&quot; &quot;bowled&quot; &quot;lbw&quot;
 $ Innings   : chr &quot;2&quot; &quot;4&quot; &quot;1&quot; &quot;3&quot; &quot;1&quot; &quot;1&quot;
 $ Opposition: chr &quot;v Pakistan&quot; &quot;v Pakistan&quot; &quot;v Pakistan&quot; &quot;v Pakistan&quot; &quot;v Pakistan&quot; &quot;v Pakistan&quot;
 $ Ground    : chr &quot;Karachi&quot; &quot;Karachi&quot; &quot;Faisalabad&quot; &quot;Faisalabad&quot; &quot;Lahore&quot; &quot;Sialkot&quot;
 $ StartDate : chr &quot;15-Nov-89&quot; &quot;15-Nov-89&quot; &quot;23-Nov-89&quot; &quot;23-Nov-89&quot; &quot;1-Dec-89&quot; &quot;9-Dec-89&quot;</pre>
</div>
</div>
</div>
<div><span>7a. Head &amp; tail -R</span></div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<pre><span>print</span>(<span>head</span>(<span>tendulkar</span>),<span>3</span>)
<span>print</span>(<span>tail</span>(<span>tendulkar</span>),<span>3</span>)
</pre>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<pre> Runs Minutes BallsFaced Fours Sixes StrikeRate Position Dismissal Innings
1   15      28         24     2     0      62.50        6    bowled       2
2  DNB      NA         NA    NA    NA         NA       NA             4
3   59     254        172     4     0      34.30        6       lbw       1
4    8      24         16     1     0      50.00        6   run out       3
5   41     124         90     5     0      45.55        7    bowled       1
6   35      74         51     5     0      68.62        6       lbw       1
  Opposition     Ground StartDate
1 v Pakistan    Karachi 15-Nov-89
2 v Pakistan    Karachi 15-Nov-89
3 v Pakistan Faisalabad 23-Nov-89
4 v Pakistan Faisalabad 23-Nov-89
5 v Pakistan     Lahore  1-Dec-89
6 v Pakistan    Sialkot  9-Dec-89
    Runs Minutes BallsFaced Fours Sixes StrikeRate Position Dismissal Innings
342   37     125         81     5     0      45.67        4    caught       2
343   21      71         23     2     0      91.30        4   run out       4
344   32      99         53     5     0      60.37        4       lbw       2
345    1       8          5     0     0      20.00        4       lbw       4
346   10      41         24     2     0      41.66        4       lbw       2
347   74     150        118    12     0      62.71        4    caught       2
       Opposition  Ground StartDate
342   v Australia  Mohali 14-Mar-13
343   v Australia  Mohali 14-Mar-13
344   v Australia   Delhi 22-Mar-13
345   v Australia   Delhi 22-Mar-13
346 v West Indies Kolkata  6-Nov-13
347 v West Indies  Mumbai 14-Nov-13</pre>
</div>
</div>
</div>
<div><span>7b. Head – SparkR</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<pre><span>head</span>(<span>tendulkar1</span>,<span>3</span>)
</pre>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<pre>  Runs Minutes BallsFaced Fours Sixes StrikeRate Position Dismissal Innings
1   15      28         24     2     0       62.5        6    bowled       2
2  DNB       -          -     -     -          -        -         -       4
3   59     254        172     4     0       34.3        6       lbw       1
  Opposition     Ground StartDate
1 v Pakistan    Karachi 15-Nov-89
2 v Pakistan    Karachi 15-Nov-89
3 v Pakistan Faisalabad 23-Nov-89</pre>
</div>
</div>
</div>
<div><span>8a. Determining the column types with sapply -R</span></div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<pre><span>sapply</span>(<span>tendulkar</span>,<span>class</span>)
</pre>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<pre>       Runs     Minutes  BallsFaced       Fours       Sixes  StrikeRate 
&quot;character&quot;   &quot;integer&quot;   &quot;integer&quot;   &quot;integer&quot;   &quot;integer&quot;   &quot;numeric&quot; 
   Position   Dismissal     Innings  Opposition      Ground   StartDate 
  &quot;integer&quot; &quot;character&quot;   &quot;integer&quot; &quot;character&quot; &quot;character&quot; &quot;character&quot;</pre>
</div>
</div>
</div>
<div><span>8b. Determining the column types with printSchema – SparkR</span></div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<pre><span>printSchema</span>(<span>tendulkar1</span>)
</pre>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<pre>root
 |-- Runs: string (nullable = true)
 |-- Minutes: string (nullable = true)
 |-- BallsFaced: string (nullable = true)
 |-- Fours: string (nullable = true)
 |-- Sixes: string (nullable = true)
 |-- StrikeRate: string (nullable = true)
 |-- Position: string (nullable = true)
 |-- Dismissal: string (nullable = true)
 |-- Innings: string (nullable = true)
 |-- Opposition: string (nullable = true)
 |-- Ground: string (nullable = true)
 |-- StartDate: string (nullable = true)</pre>
</div>
</div>
</div>
<div><span>9a. Selecting columns – R</span></div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<pre><span>library</span>(<span>dplyr</span>)
<span>df</span><span>=</span><span>select</span>(<span>tendulkar</span>,<span>Runs</span>,<span>BallsFaced</span>,<span>Minutes</span>)
<span>head</span>(<span>df</span>,<span>5</span>)
</pre>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<pre>  Runs BallsFaced Minutes
1   15         24      28
2  DNB         NA      NA
3   59        172     254
4    8         16      24
5   41         90     124</pre>
</div>
</div>
</div>
<div><span>9b. Selecting columns – SparkR</span></div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<pre><span>library</span>(<span>SparkR</span>)
<span>Sys.setenv</span>(<span>SPARK_HOME</span><span>=</span><span>&quot;</span><span>/usr/hdp/2.6.0.3-8/spark&quot;</span>)
<span>.libPaths</span>(<span>c</span>(<span>file.path</span>(<span>Sys.getenv</span>(<span>&quot;</span><span>SPARK_HOME&quot;</span>), <span>&quot;</span><span>R&quot;</span>, <span>&quot;</span><span>lib&quot;</span>), <span>.libPaths</span>()))
<span># Initiate a SparkR session</span>
<span>sparkR.session</span>()
<span>tendulkar1</span> <span>&lt;-</span> <span>read.df</span>(<span>&quot;</span><span>/FileStore/tables/tendulkar.csv&quot;</span>, 
                <span>header</span> <span>=</span> <span>&quot;</span><span>true&quot;</span>, 
                <span>delimiter</span> <span>=</span> <span>&quot;</span><span>,&quot;</span>, 
                <span>source</span> <span>=</span> <span>&quot;</span><span>csv&quot;</span>, 
                <span>inferSchema</span> <span>=</span> <span>&quot;</span><span>true&quot;</span>, 
                <span>na.strings</span> <span>=</span> <span>&quot;</span><span>&quot;</span>)
<span>df</span><span>=</span><span>SparkR</span><span>::</span><span>select</span>(<span>tendulkar1</span>, <span>&quot;</span><span>Runs&quot;</span>, <span>&quot;</span><span>BF&quot;</span>,<span>&quot;</span><span>Mins&quot;</span>)
<span>head</span>(<span>SparkR</span><span>::</span><span>collect</span>(<span>df</span>))
</pre>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<pre>  Runs  BF Mins
1   15  24   28
2  DNB   -    -
3   59 172  254
4    8  16   24
5   41  90  124
6   35  51   74</pre>
</div>
</div>
</div>
<div><span>10a. Filter rows by criteria – R</span></div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<pre><span>library</span>(<span>dplyr</span>)
<span>df</span><span>=</span><span>tendulkar</span> <span>%>%</span> <span>filter</span>(<span>Runs</span> <span>></span> <span>50</span>)
<span>head</span>(<span>df</span>,<span>5</span>)
</pre>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<pre>  Runs Minutes BallsFaced Fours Sixes StrikeRate Position Dismissal Innings
1  DNB      NA         NA    NA    NA         NA       NA             4
2   59     254        172     4     0      34.30        6       lbw       1
3    8      24         16     1     0      50.00        6   run out       3
4   57     193        134     6     0      42.53        6    caught       3
5   88     324        266     5     0      33.08        6    caught       1
     Opposition     Ground StartDate
1    v Pakistan    Karachi 15-Nov-89
2    v Pakistan Faisalabad 23-Nov-89
3    v Pakistan Faisalabad 23-Nov-89
4    v Pakistan    Sialkot  9-Dec-89
5 v New Zealand     Napier  9-Feb-90</pre>
</div>
</div>
</div>
<div><span>10b. Filter rows by criteria – SparkR</span></div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<pre><span>df</span><span>=</span><span>SparkR</span><span>::</span><span>filter</span>(<span>tendulkar1</span>, <span>tendulkar1</span><span>$</span><span>Runs</span> <span>></span> <span>50</span>)
<span>head</span>(<span>SparkR</span><span>::</span><span>collect</span>(<span>df</span>))
</pre>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<pre>  Runs Mins  BF 4s 6s    SR Pos Dismissal Inns     Opposition       Ground
1   59  254 172  4  0  34.3   6       lbw    1     v Pakistan   Faisalabad
2   57  193 134  6  0 42.53   6    caught    3     v Pakistan      Sialkot
3   88  324 266  5  0 33.08   6    caught    1  v New Zealand       Napier
4   68  216 136  8  0    50   6    caught    2      v England   Manchester
5  114  228 161 16  0  70.8   4    caught    2    v Australia        Perth
6  111  373 270 19  0 41.11   4    caught    2 v South Africa Johannesburg
  Start Date
1  23-Nov-89
2   9-Dec-89
3   9-Feb-90
4   9-Aug-90
5   1-Feb-92
6  26-Nov-92</pre>
</div>
</div>
</div>
<div><span>11a. Unique values -R</span></div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<pre><span>unique</span>(<span>tendulkar</span><span>$</span><span>Runs</span>)
</pre>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<pre>  [1] &quot;15&quot;   &quot;DNB&quot;  &quot;59&quot;   &quot;8&quot;    &quot;41&quot;   &quot;35&quot;   &quot;57&quot;   &quot;0&quot;    &quot;24&quot;   &quot;88&quot;  
 [11] &quot;5&quot;    &quot;10&quot;   &quot;27&quot;   &quot;68&quot;   &quot;119*&quot; &quot;21&quot;   &quot;11&quot;   &quot;16&quot;   &quot;7&quot;    &quot;40&quot;  
 [21] &quot;148*&quot; &quot;6&quot;    &quot;17&quot;   &quot;114&quot;  &quot;111&quot;  &quot;1&quot;    &quot;73&quot;   &quot;50&quot;   &quot;9*&quot;   &quot;165&quot; 
 [31] &quot;78&quot;   &quot;62&quot;   &quot;TDNB&quot; &quot;28&quot;   &quot;104*&quot; &quot;71&quot;   &quot;142&quot;  &quot;96&quot;   &quot;43&quot;   &quot;11*&quot; 
 [41] &quot;34&quot;   &quot;85&quot;   &quot;179&quot;  &quot;54&quot;   &quot;4&quot;    &quot;0*&quot;   &quot;52*&quot;  &quot;2&quot;    &quot;122&quot;  &quot;31&quot;  
 [51] &quot;177&quot;  &quot;74&quot;   &quot;42&quot;   &quot;18&quot;   &quot;61&quot;   &quot;36&quot;   &quot;169&quot;  &quot;9&quot;    &quot;15*&quot;  &quot;92&quot;  
 [61] &quot;83&quot;   &quot;143&quot;  &quot;139&quot;  &quot;23&quot;   &quot;148&quot;  &quot;13&quot;   &quot;155*&quot; &quot;79&quot;   &quot;47&quot;   &quot;113&quot; 
 [71] &quot;67&quot;   &quot;136&quot;  &quot;29&quot;   &quot;53&quot;   &quot;124*&quot; &quot;126*&quot; &quot;44*&quot;  &quot;217&quot;  &quot;116&quot;  &quot;52&quot;  
 [81] &quot;45&quot;   &quot;97&quot;   &quot;20&quot;   &quot;39&quot;   &quot;201*&quot; &quot;76&quot;   &quot;65&quot;   &quot;126&quot;  &quot;36*&quot;  &quot;69&quot;  
 [91] &quot;155&quot;  &quot;22*&quot;  &quot;103&quot;  &quot;26&quot;   &quot;90&quot;   &quot;176&quot;  &quot;117&quot;  &quot;86&quot;   &quot;12&quot;   &quot;193&quot; 
[101] &quot;16*&quot;  &quot;51&quot;   &quot;32&quot;   &quot;55&quot;   &quot;37&quot;   &quot;44&quot;   &quot;241*&quot; &quot;60*&quot;  &quot;194*&quot; &quot;3&quot;   
[111] &quot;32*&quot;  &quot;248*&quot; &quot;94&quot;   &quot;22&quot;   &quot;109&quot;  &quot;19&quot;   &quot;14&quot;   &quot;28*&quot;  &quot;63&quot;   &quot;64&quot;  
[121] &quot;101&quot;  &quot;122*&quot; &quot;91&quot;   &quot;82&quot;   &quot;56*&quot;  &quot;154*&quot; &quot;153&quot;  &quot;49&quot;   &quot;10*&quot;  &quot;103*&quot;
[131] &quot;160&quot;  &quot;100*&quot; &quot;105*&quot; &quot;100&quot;  &quot;106&quot;  &quot;84&quot;   &quot;203&quot;  &quot;98&quot;   &quot;38&quot;   &quot;214&quot; 
[141] &quot;53*&quot;  &quot;111*&quot; &quot;146&quot;  &quot;14*&quot;  &quot;56&quot;   &quot;80&quot;   &quot;25&quot;   &quot;81&quot;   &quot;13*&quot;</pre>
</div>
</div>
</div>
<div><span>11b. Unique values – SparkR</span></div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<pre><span>head</span>(<span>SparkR</span><span>::</span><span>distinct</span>(<span>tendulkar1</span>[,<span>&quot;</span><span>Runs&quot;</span>]),<span>5</span>)
</pre>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<pre>  Runs
1 119*
2    7
3   51
4  169
5  32*</pre>
</div>
</div>
</div>
<div><span>12a. Aggregate – Mean, min and max – R</span></div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<pre><span>library</span>(<span>dplyr</span>)
<span>library</span>(<span>magrittr</span>)
<span>a</span> <span>&lt;-</span> <span>tendulkar</span><span>$</span><span>Runs</span> <span>!=</span> <span>&quot;</span><span>DNB&quot;</span>
<span>tendulkar</span> <span>&lt;-</span> <span>tendulkar</span>[<span>a</span>,]
<span>dim</span>(<span>tendulkar</span>)

<span># Remove rows with 'TDNB'</span>
<span>c</span> <span>&lt;-</span> <span>tendulkar</span><span>$</span><span>Runs</span> <span>!=</span> <span>&quot;</span><span>TDNB&quot;</span>
<span>tendulkar</span> <span>&lt;-</span> <span>tendulkar</span>[<span>c</span>,]

<span># Remove rows with absent</span>
<span>d</span> <span>&lt;-</span> <span>tendulkar</span><span>$</span><span>Runs</span> <span>!=</span> <span>&quot;</span><span>absent&quot;</span>
<span>tendulkar</span> <span>&lt;-</span> <span>tendulkar</span>[<span>d</span>,]
<span>dim</span>(<span>tendulkar</span>)

<span># Remove the &quot;* indicating not out</span>
<span>tendulkar</span><span>$</span><span>Runs</span> <span>&lt;-</span> <span>as.numeric</span>(<span>gsub</span>(<span>&quot;</span><span>\\</span><span>*&quot;</span>,<span>&quot;</span><span>&quot;</span>,<span>tendulkar</span><span>$</span><span>Runs</span>))
<span>c</span> <span>&lt;-</span> <span>complete.cases</span>(<span>tendulkar</span>)

<span>#Subset the rows which are complete</span>
<span>tendulkar</span> <span>&lt;-</span> <span>tendulkar</span>[<span>c</span>,]
<span>print</span>(<span>dim</span>(<span>tendulkar</span>))
<span>df</span> <span>&lt;-</span><span>tendulkar</span> <span>%>%</span>  <span>group_by</span>(<span>Ground</span>) <span>%>%</span> <span>summarise</span>(<span>meanRuns</span><span>=</span> <span>mean</span>(<span>Runs</span>), <span>minRuns</span><span>=</span><span>min</span>(<span>Runs</span>), <span>maxRuns</span><span>=</span><span>max</span>(<span>Runs</span>)) 
<span>#names(tendulkar)</span>
<span>head</span>(<span>df</span>)
</pre>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<pre>[1] 327  12
# A tibble: 6 x 4
  Ground       meanRuns minRuns maxRuns
                   
1 Adelaide        32.6       0.    153.
2 Ahmedabad       40.1       4.    217.
3 Auckland         5.00      5.      5.
4 Bangalore       57.9       4.    214.
5 Birmingham      46.8       1.    122.
6 Bloemfontein    85.0      15.    155.</pre>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div><span>12b. Aggregate- Mean, Min, Max – SparkR</span></div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<pre><span>sparkR.session</span>()

<span>tendulkar1</span> <span>&lt;-</span> <span>read.df</span>(<span>&quot;</span><span>/FileStore/tables/tendulkar.csv&quot;</span>, 
                <span>header</span> <span>=</span> <span>&quot;</span><span>true&quot;</span>, 
                <span>delimiter</span> <span>=</span> <span>&quot;</span><span>,&quot;</span>, 
                <span>source</span> <span>=</span> <span>&quot;</span><span>csv&quot;</span>, 
                <span>inferSchema</span> <span>=</span> <span>&quot;</span><span>true&quot;</span>, 
                <span>na.strings</span> <span>=</span> <span>&quot;</span><span>&quot;</span>)

<span>print</span>(<span>dim</span>(<span>tendulkar1</span>))
<span>tendulkar1</span> <span>&lt;-</span><span>SparkR</span><span>::</span><span>filter</span>(<span>tendulkar1</span>,<span>tendulkar1</span><span>$</span><span>Runs</span> <span>!=</span> <span>&quot;</span><span>DNB&quot;</span>)
<span>print</span>(<span>dim</span>(<span>tendulkar1</span>))
<span>tendulkar1</span><span>&lt;-</span><span>SparkR</span><span>::</span><span>filter</span>(<span>tendulkar1</span>,<span>tendulkar1</span><span>$</span><span>Runs</span> <span>!=</span> <span>&quot;</span><span>TDNB&quot;</span>)
<span>print</span>(<span>dim</span>(<span>tendulkar1</span>))
<span>tendulkar1</span><span>&lt;-</span><span>SparkR</span><span>::</span><span>filter</span>(<span>tendulkar1</span>,<span>tendulkar1</span><span>$</span><span>Runs</span> <span>!=</span> <span>&quot;</span><span>absent&quot;</span>)
<span>print</span>(<span>dim</span>(<span>tendulkar1</span>))

<span># Cast the string type Runs to double</span>
<span>withColumn</span>(<span>tendulkar1</span>, <span>&quot;</span><span>Runs&quot;</span>, <span>cast</span>(<span>tendulkar1</span><span>$</span><span>Runs</span>, <span>&quot;</span><span>double&quot;</span>))
<span>head</span>(<span>SparkR</span><span>::</span><span>distinct</span>(<span>tendulkar1</span>[,<span>&quot;</span><span>Runs&quot;</span>]),<span>20</span>)
<span># Remove the &quot;* indicating not out</span>
<span>tendulkar1</span><span>$</span><span>Runs</span><span>=</span><span>SparkR</span><span>::</span><span>regexp_replace</span>(<span>tendulkar1</span><span>$</span><span>Runs</span>, <span>&quot;</span><span>\\</span><span>*&quot;</span>, <span>&quot;</span><span>&quot;</span>)
<span>head</span>(<span>SparkR</span><span>::</span><span>distinct</span>(<span>tendulkar1</span>[,<span>&quot;</span><span>Runs&quot;</span>]),<span>20</span>)
<span>df</span><span>=</span><span>SparkR</span><span>::</span><span>summarize</span>(<span>SparkR</span><span>::</span><span>groupBy</span>(<span>tendulkar1</span>, <span>tendulkar1</span><span>$</span><span>Ground</span>), <span>mean</span> <span>=</span> <span>mean</span>(<span>tendulkar1</span><span>$</span><span>Runs</span>), <span>minRuns</span><span>=</span><span>min</span>(<span>tendulkar1</span><span>$</span><span>Runs</span>),<span>maxRuns</span><span>=</span><span>max</span>(<span>tendulkar1</span><span>$</span><span>Runs</span>))
<span>head</span>(<span>df</span>,<span>20</span>)
</pre>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<pre>[1] 347  12
[1] 330  12
[1] 329  12
[1] 329  12
          Ground       mean minRuns maxRuns
1      Bangalore  54.312500       0      96
2       Adelaide  32.600000       0      61
3  Colombo (PSS)  37.200000      14      71
4   Christchurch  12.000000       0      24
5       Auckland   5.000000       5       5
6        Chennai  60.625000       0      81
7      Centurion  73.500000     111      36
8       Brisbane   7.666667       0       7
9     Birmingham  46.750000       1      40
10     Ahmedabad  40.125000     100       8
11 Colombo (RPS) 143.000000     143     143
12    Chittagong  57.800000     101      36
13     Cape Town  69.857143      14       9
14    Bridgetown  26.000000       0      92
15      Bulawayo  55.000000      36      74
16         Delhi  39.947368       0      76
17    Chandigarh  11.000000      11      11
18  Bloemfontein  85.000000      15     155
19 Colombo (SSC)  77.555556     104       8
20       Cuttack   2.000000       2       2</pre>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div><span>13a Using SQL with SparkR</span></div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<pre><span><span>sparkR.session</span><span>()</span>
tendulkar1</span> <span>&lt;-</span> <span>read.df</span>(<span>&quot;</span><span>/FileStore/tables/tendulkar.csv&quot;</span>, 
                <span>header</span> <span>=</span> <span>&quot;</span><span>true&quot;</span>, 
                <span>delimiter</span> <span>=</span> <span>&quot;</span><span>,&quot;</span>, 
                <span>source</span> <span>=</span> <span>&quot;</span><span>csv&quot;</span>, 
                <span>inferSchema</span> <span>=</span> <span>&quot;</span><span>true&quot;</span>, 
                <span>na.strings</span> <span>=</span> <span>&quot;</span><span>&quot;</span>)

<span># Register this SparkDataFrame as a temporary view.</span>
<span>createOrReplaceTempView</span>(<span>tendulkar1</span>, <span>&quot;</span><span>tendulkar2&quot;</span>)

<span># SQL statements can be run by using the sql method</span>
<span>df</span><span>=</span><span>SparkR</span><span>::</span><span>sql</span>(<span>&quot;</span><span>SELECT * FROM tendulkar2 WHERE Ground='Karachi'&quot;</span>)

<span>head</span>(<span>df</span>)

</pre>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<pre>  Runs Mins BF 4s 6s    SR Pos Dismissal Inns Opposition  Ground Start Date
1   15   28 24  2  0  62.5   6    bowled    2 v Pakistan Karachi  15-Nov-89
2  DNB    -  -  -  -     -   -         -    4 v Pakistan Karachi  15-Nov-89
3   23   49 29  5  0 79.31   4    bowled    2 v Pakistan Karachi  29-Jan-06
4   26   74 47  5  0 55.31   4    bowled    4 v Pakistan Karachi  29-Jan-06</pre>
</div>
</div>
</div>
<div><span>Conclusion</span></div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p>This post discusses some of the key constructs in R and SparkR and how one can transition from R to SparkR fairly easily. I will be adding more constructs later. Do check back!</p>
</div>
<p>You may also like<br>
1. <a rel="nofollow" href="https://gigadom.in/2016/06/05/exploring-quantum-gate-operations-with-qcsimulator/" target="_blank">Exploring Quantum Gate operations with QCSimulator</a><br>
2. <a rel="nofollow" href="https://gigadom.in/2018/02/26/deep-learning-from-first-principles-in-python-r-and-octave-part-4/" target="_blank">Deep Learning from first principles in Python, R and Octave – Part 4</a><br>
3. <a rel="nofollow" href="https://gigadom.in/2014/07/27/a-bluemix-recipe-with-mongodb-and-node-js/" target="_blank">A Bluemix recipe with MongoDB and Node.js</a><br>
4. <a rel="nofollow" href="https://gigadom.in/2017/11/07/practical-machine-learning-with-r-and-python-part-5/" target="_blank">Practical Machine Learning with R and Python – Part 5</a><br>
5. <a rel="nofollow" href="https://gigadom.in/2015/07/04/introducing-cricketr-a-r-package-to-analyze-performances-of-cricketers/" target="_blank">Introducing cricketr! : An R package to analyze performances of cricketers</a></p>
<p>To see all posts click <a rel="nofollow" href="https://gigadom.in/aa-2/" target="_blank">Index of posts</a></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://gigadom.in/2018/10/09/big-data-2-move-into-the-big-leaguegraduate-from-r-to-sparkr/"> R – Giga thoughts …</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/_XpiuF4SaL0" height="1"></a></div>