<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/visualizing-taxi-trips-between-nyc-neighborhoods-with-spark-and-microsoft-r-server/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://blog.revolutionanalytics.com/2016/12/taxi-mrs-spark.html"> Revolutions</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<div>
<div>
<p>by Ali Zaidi, Data Scientist at Microsoft</p>
<p>In <a target="_blank" rel="nofollow" href="http://blog.revolutionanalytics.com/2016/11/data-manipulation-with-sparklyr-on-azure-hdinsight.html">previous post</a> we showcased the use of the sparklyr package for manipulating large datasets using a familiar dplyr syntax on top of Spark HDInsight Clusters.</p>
<p>In this post, we will take a look at the <code>RxSpark</code> API for R, part of the <code>RevoScaleR</code> package and the <a target="_blank" rel="nofollow" href="https://azure.microsoft.com/en-us/services/hdinsight/r-server/">Microsoft R Server distribution of R on HDInsight</a>. We'll use RxSpark to visualize a dataset of 140M taxi rides between boroughs in New York City.</p>
<p><a target="_blank" rel="nofollow" href="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b8d2467bd0970c-pi"><img title="Chord_plots-4" alt="Chord_plots-4" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b8d2467bd0970c-800wi" border="0"></a></p>
<p>Dealing with data in distributed storage and programming with concurrent systems often requires learning complicated new paradigms and techniques. Statisticans and data scientists familiar wtih R are unlikely to have much experience with such systems. Fortunately, the <code>RevoScaleR</code> package abstracts away the difficult portions of distributed computation and allows the user to focus on building R code that can be automatically deployed in distributed environments.</p>
<div>
<h2>WODA – Write Once, Deploy Anywhere</h2>
<p>In a similar spirit to how <code>sparklyr</code> allowed us to reuse our functions from the <code>dplyr</code> package to manipulate Spark DataFrames, the <code>RxSpark</code> API allows a data scientist to develop code that can be deployed in a multitude of environments. This allows the developer to shift their focus from writing code that’s specific to a certain environment, and instead focus on the complex analysis of their data science problem. We call this flexibility <em>Write Once, Deploy Anywhere</em>, or <em>WODA</em> for the acronym lovers.</p>
<p>For a deeper dive into the <code>RevoScaleR</code> package, I recommend you take a look at the online course, <a target="_blank" rel="nofollow" href="https://courses.edx.org/courses/course-v1:Microsoft+DAT213x+3T2016/info">Analyzing Big Data with Microsoft R Server</a>. Much of this blogpost follows along the last section of the course, on deployment to <a target="_blank" rel="nofollow" href="https://smott.gitbooks.io/introduction-to-microsoft-r-server/content/41d-deploying-to-spark-demo.html">Spark</a>.</p>
</div>
<div>
<h2>NYC Taxi Data</h2>
<p>In this section, we will examine the ubiquitious <a target="_blank" rel="nofollow" href="http://www.nyc.gov/html/tlc/html/about/trip_record_data.shtml">NYC Taxi Dataset</a>, and showcase how we can develop data analysis pipelines that are platform invariant.</p>
</div>
</div>
<p>As we will see in this post, we can reuse code that we developed in a local compute environment (i.e., a single machine), and have it deploy automagically in our new Spark environment. <code>RevoScaleR</code> will automatically transfer the computation from a single local machine to a network of concurrent systems, without requiring any concurrent expertise from the developer.</p>
<p>The aforementioned online course provides detailed examples of examining the NYC Taxi Dataset in a local <em>compute context</em>. A <em>compute context</em> is an object from the <code>RevoScaleR</code> package that describes a compute platform to inform <code>RevoScaleR</code> where to conduct its computations. By default, it is a <code>RxLocalSeq</code> environment, which means <code>RevoScaleR</code> will conduct it’s analysis using the resources provided by the local machine, and do so in a sequential manner (i.e., if you have multiple jobs, these will be run sequentially, not in parallel). In our case, we have access to a cluster of machines on HDInsight, and a distributed compute platform through Spark, so we should inform <code>RevoScaleR</code> of our good fortune and request it use more than just a single machine to do our computation. Fortunately, we can leverage the code directly from the course demonstrations, and apply it a larger dataset through Spark.</p>
<div>
<h3>Defining our Spark Compute Context</h3>
<p>One of the major niceties of the <code>RevoScaleR</code> package is that it allows a developer to move seamlessly betweeen environments by simply specifying which compute platform the package should use for it’s next computations. When you start working with <code>RevoScaleR</code>, you are working in a local compute context, which you can see by calling on the function <code>rxGetComputeContext</code>:</p>
<pre><code>rxGetComputeContext()</code></pre>
<pre><code>## RxLocalSeq Compute Context</code></pre>
<p>If you ever change your compute environment to a different platform, you can always revert back to a local compute context by simply setting the compute context back to “local”:</p>
<pre><code>rxSetComputeContext(&quot;local&quot;)</code></pre>
<p>In our case, we would like to work in a Spark compute context, which is defined by using the <code>RxSpark</code> object. Moreover, we also want to specify to <code>RevoScaleR</code> the location of our data store, which in this case, will be HDFS. This is done through the <code>RxHdfsFileSystem</code> object:</p>
<pre><code>hdfsFS &lt;- RxHdfsFileSystem()

spark_cc &lt;- RxSpark(
                    persistentRun = TRUE, 
                    extraSparkConfig = &quot;--conf spark.speculation=true&quot;
                    )
                          

rxSetComputeContext(spark_cc)

rxGetComputeContext()</code></pre>
<pre><code>## RevoScaleR Hadoop Spark Local Object
## ------------------------------------
## hdfsShareDir : &quot;/user/RevoShare/alizaidi&quot; 
## clientShareDir : &quot;/tmp&quot; 
## hadoopRPath : &quot;/usr/bin/Revo64&quot; 
## hadoopSwitches : &quot;&quot; 
## sshUsername : &quot;alizaidi&quot; 
## sshHostname : NULL 
## sshSwitches : &quot;&quot; 
## sshProfileScript : NULL 
## sshClientDir : &quot;&quot; 
## remoteMrsVersion : structure(list(major = 9L, minor = 0L, patch = 1L), .Names = c(&quot;major&quot;, &quot;minor&quot;, &quot;patch&quot;), class = &quot;version&quot;) 
## sshStrictHostKeyChecking : &quot;ask&quot; 
## usingRunAsUserMode : FALSE 
## nameNode : &quot;default&quot; 
## jobTrackerURL : NULL 
## port : 0 
## onClusterNode : TRUE 
## showOutputWhileWaiting : TRUE 
## fileSystem : NULL 
## numExecutors : 4 
## executorCores : 11 
## executorMem : &quot;19280m&quot; 
## driverMem : &quot;4g&quot; 
## executorOverheadMem : &quot;19280m&quot; 
## extraSparkConfig : &quot;--conf spark.speculation=true&quot; 
## idleTimeout : 3600 
## persistentRun : TRUE 
## sparkReduceMethod : &quot;auto&quot; 
## suppressWarning : TRUE 
## user : &quot;alizaidi&quot; 
## rSessionPid : 62825L 
## appId : &quot;1C7804591FEA4EEEB12E2FC5963C556B&quot; 
## jobId : &quot;&quot; 
## shareDir : &quot;/var/RevoShare/alizaidi&quot; 
## revoPath : &quot;/usr/bin/Revo64&quot; 
## wait : TRUE 
## consoleOutput : FALSE 
## autoCleanup : TRUE 
## workingDir : NULL 
## dataPath : NULL 
## outDataPath : NULL 
## packagesToLoad : NULL 
## email : NULL 
## resultsTimeout : 15 
## description : &quot;spark&quot; 
## version : &quot;1.0-1&quot; 
## compatibilityRequest : &lt;environment></code></pre>
<p>For simplicity, we have used all the default arguments when defining the Spark and HDFS environmental variables, except for insisting that <code>RevoScaleR</code> reuse the existing Spark application whenever possible (the <code>persistentRun</code> parameter), and that Spark attempt to restart tasks that appear to be lagging (the <code>extraSparkConfig</code> value).</p>
<p>Now that we have moved from a local environment to a Spark compute environment, our subsequent calls to <code>rx</code> functions will kick off a Spark application which will handle the distributed nature of our computations. For the developer/data scientist, we can proceed as though we were still working with a local machine, and allow <code>RevoScaleR</code> to handle the complexities of concurrent computation. Our data is saved in HDFS in the file path defined through the <code>taxi_path</code> object below. Let’s first create a path to where we would like to save our <strong>XDF</strong> files and a pointer to our existing directory of <strong>csv</strong> files that we are hoping to import. Lastly, we will save a sample <code>data.frame</code> that we could use for testing:</p>
<pre><code>data_path &lt;- file.path(&quot;/user/RevoShare/alizaidi&quot;)
taxi_path &lt;- file.path(data_path, &quot;nyctaxi/data&quot;)
hdfs_ls &lt;- paste0(&quot;hadoop fs -ls &quot;, taxi_path)
system(hdfs_ls)

taxi_xdf &lt;- file.path(data_path, &quot;TaxiXdf&quot;)

taxi_text &lt;- RxTextData(taxi_path, fileSystem = hdfsFS)
taxi_xdf &lt;- RxXdfData(taxi_xdf, fileSystem = hdfsFS)

### Also save a sample as a data.frame
nyc_sample_df &lt;- read.csv(&quot;data/yellow_tripdata_2016-05.csv&quot;, nrows = 1000)</code></pre>
<p>An <strong>XDF</strong> file is short for an external data frame, which is the fundamental data structure for <code>RevoScaleR</code> algorithms. It is an on-disk file object, that is optimized to work with <code>RevoScaleR</code>, and will be our default data structure for this blogpost. When we save <strong>XDF</strong>s on a HDFS data store, we will use the acronym <strong>XDFD</strong>s, to emphasize the distributed nature of the <strong>XDF</strong> object. For more information about XDFs an XDFDs, particularly when stored on HDFS, please take a <a target="_blank" rel="nofollow" href="https://msdn.microsoft.com/en-us/microsoft-r/scaler-user-guide-data-import#importing-data-as-composite-xdf-files">look at the documentation here</a>.</p>
<p>Now that we have our pointers to the original data source (<code>taxi_text</code>), and the file path to where we want to save our data to (<code>taxi_xdf</code>), we can directly import our data using the simplest of RevoScaleR functions:</p>
<pre><code>rxImport(inData = taxi_text, outFile = taxi_xdf)</code></pre>
<p>Despite the fact that the data is stored in a set of directories partitioned across data nodes in a distributed cluster, our import step is the exact call that we would make when working with data stored locally!</p>
<pre><code>rxGetInfo(taxi_xdf, getVarInfo = TRUE, numRows = 5)</code></pre>
<pre><code>## File name: /user/RevoShare/alizaidi/TaxiXdf 
## Number of composite data files: 48 
## Number of observations: 138413407 
## Number of variables: 19 
## Number of blocks: 289 
## Compression type: zlib 
## Variable information: 
## Var 1: VendorID, Type: integer, Low/High: (1, 2)
## Var 2: tpep_pickup_datetime, Type: character
## Var 3: tpep_dropoff_datetime, Type: character
## Var 4: passenger_count, Type: integer, Low/High: (0, 9)
## Var 5: trip_distance, Type: numeric, Storage: float32, Low/High: (-3390583.7500, 198623008.0000)
## Var 6: pickup_longitude, Type: numeric, Storage: float32, Low/High: (-736.6166, 172.6000)
## Var 7: pickup_latitude, Type: numeric, Storage: float32, Low/High: (-78.1947, 80.6025)
## Var 8: RatecodeID, Type: integer, Low/High: (1, 99)
## Var 9: store_and_fwd_flag, Type: character
## Var 10: dropoff_longitude, Type: numeric, Storage: float32, Low/High: (-781.8333, 172.6000)
## Var 11: dropoff_latitude, Type: numeric, Storage: float32, Low/High: (-78.1947, 480.7333)
## Var 12: payment_type, Type: integer, Low/High: (1, 5)
## Var 13: fare_amount, Type: numeric, Storage: float32, Low/High: (-957.6000, 825998.6250)
## Var 14: extra, Type: numeric, Storage: float32, Low/High: (-58.5000, 648.8700)
## Var 15: mta_tax, Type: numeric, Storage: float32, Low/High: (-3.0000, 91.0000)
## Var 16: tip_amount, Type: numeric, Storage: float32, Low/High: (-440.0000, 1200.8000)
## Var 17: tolls_amount, Type: numeric, Storage: float32, Low/High: (-99.9900, 1901.4000)
## Var 18: improvement_surcharge, Type: numeric, Storage: float32, Low/High: (-0.3000, 137.6300)
## Var 19: total_amount, Type: numeric, Storage: float32, Low/High: (-958.4000, 826040.0000)
## Data (5 rows starting with row 1):
##   VendorID tpep_pickup_datetime tpep_dropoff_datetime passenger_count
## 1        1  2015-07-01 00:00:00   2015-07-01 00:15:26               1
## 2        1  2015-07-01 00:00:00   2015-07-01 00:22:22               1
## 3        1  2015-07-01 00:00:00   2015-07-01 00:07:42               1
## 4        1  2015-07-01 00:00:00   2015-07-01 00:39:37               1
## 5        1  2015-07-01 00:00:00   2015-07-01 00:05:34               1
##   trip_distance pickup_longitude pickup_latitude RatecodeID
## 1           3.5        -73.99416        40.75113          1
## 2           3.9        -73.98466        40.76849          1
## 3           2.3        -73.97889        40.76229          1
## 4           9.2        -73.99279        40.74276          1
## 5           1.1        -73.91243        40.76981          1
##   store_and_fwd_flag dropoff_longitude dropoff_latitude payment_type
## 1                  N         -73.97682         40.78857            2
## 2                  N         -74.00013         40.73490            2
## 3                  N         -74.00422         40.75253            2
## 4                  N         -73.97151         40.63715            1
## 5                  N         -73.92033         40.75744            1
##   fare_amount extra mta_tax tip_amount tolls_amount improvement_surcharge
## 1          14   0.5     0.5       0.00            0                   0.3
## 2          17   0.5     0.5       0.00            0                   0.3
## 3           9   0.5     0.5       0.00            0                   0.3
## 4          33   0.5     0.5       8.55            0                   0.3
## 5           6   0.5     0.5       2.00            0                   0.3
##   total_amount
## 1        15.30
## 2        18.30
## 3        10.30
## 4        42.85
## 5         9.30</code></pre>
<p>Even though the data is now saved across multiple nodes in a distributed environment, the data is compressed in order to improve read performance, and metadata is saved, improving querying time for simple statsitics.</p>
</div>
<div>
<h2>Transformations with XDFDs</h2>
<p>An important distinction with working with data stored in distributed file systems like HDFS in comparison to data residing on a single disk is its <em>mutability</em>. In general, it’s much more difficult to overwrite data in distributed storage, as it requires rewriting multiple non-contiguous blocks.</p>
<p>Therefore, it is often better practice to write to a new location when working with XDFDs than to overwrite existing directories. The fundamental function for data manipulation with <code>RevoScaleR</code> is the <code>rxDataStep</code> function, and it adds new columns by taking in a list of transform expressions, as described in the example below:</p>
<pre><code>taxi_tip &lt;- RxXdfData(&quot;/user/RevoShare/alizaidi/taxitipXdf&quot;,
                      fileSystem = hdfsFS)</code></pre>
<pre><code>rxDataStep(taxi_xdf, taxi_tip,
           transforms = list(tip_percent = ifelse(fare_amount > 0, 
                                                  tip_amount/fare_amount,
                                                  NA)))</code></pre>
<p>Examining results:</p>
<pre><code>rxGetInfo(taxi_tip, getVarInfo = TRUE)</code></pre>
<pre><code>## File name: /user/RevoShare/alizaidi/taxitipXdf 
## Number of composite data files: 48 
## Number of observations: 138413407 
## Number of variables: 20 
## Number of blocks: 289 
## Compression type: zlib 
## Variable information: 
## Var 1: VendorID, Type: integer, Low/High: (1, 2)
## Var 2: tpep_pickup_datetime, Type: character
## Var 3: tpep_dropoff_datetime, Type: character
## Var 4: passenger_count, Type: integer, Low/High: (0, 9)
## Var 5: trip_distance, Type: numeric, Storage: float32, Low/High: (-3390583.7500, 198623008.0000)
## Var 6: pickup_longitude, Type: numeric, Storage: float32, Low/High: (-736.6166, 172.6000)
## Var 7: pickup_latitude, Type: numeric, Storage: float32, Low/High: (-78.1947, 80.6025)
## Var 8: RatecodeID, Type: integer, Low/High: (1, 99)
## Var 9: store_and_fwd_flag, Type: character
## Var 10: dropoff_longitude, Type: numeric, Storage: float32, Low/High: (-781.8333, 172.6000)
## Var 11: dropoff_latitude, Type: numeric, Storage: float32, Low/High: (-78.1947, 480.7333)
## Var 12: payment_type, Type: integer, Low/High: (1, 5)
## Var 13: fare_amount, Type: numeric, Storage: float32, Low/High: (-957.6000, 825998.6250)
## Var 14: extra, Type: numeric, Storage: float32, Low/High: (-58.5000, 648.8700)
## Var 15: mta_tax, Type: numeric, Storage: float32, Low/High: (-3.0000, 91.0000)
## Var 16: tip_amount, Type: numeric, Storage: float32, Low/High: (-440.0000, 1200.8000)
## Var 17: tolls_amount, Type: numeric, Storage: float32, Low/High: (-99.9900, 1901.4000)
## Var 18: improvement_surcharge, Type: numeric, Storage: float32, Low/High: (-0.3000, 137.6300)
## Var 19: total_amount, Type: numeric, Storage: float32, Low/High: (-958.4000, 826040.0000)
## Var 20: tip_percent, Type: numeric, Low/High: (-1.0000, 54900.0012)</code></pre>
</div>
<div>
<h2>More Complicated Transforms</h2>
<div>
<h3>Creating Temporal Features</h3>
<p>The above manipuluation previous sections how we could create new columns by defining a list of expressions. For more complicated data maipulation steps, we could create a user-defined function (UDF) that we could apply to each chunk of our data to create new columns. This greatly expands on the possibilities of data manipulation for large datasets, as we are no longer limited to handcrafting single expressions, we could create complex functions that could even rely on external R packages. The function below uses the <code>lubridate</code> package to create a set of columns defining some temporal features; the pickup/dropoff date, hour, day of week, as well as the trip duration:</p>
<pre><code>xforms &lt;- function(data) { # transformation function for extracting some date and time features
  # require(lubridate)
  weekday_labels &lt;- c('Sun', 'Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat')
  cut_levels &lt;- c(1, 5, 9, 12, 16, 18, 22)
  hour_labels &lt;- c('1AM-5AM', '5AM-9AM', '9AM-12PM', '12PM-4PM', '4PM-6PM', '6PM-10PM', '10PM-1AM')
  
  pickup_datetime &lt;- lubridate::ymd_hms(data$tpep_pickup_datetime, tz = &quot;UTC&quot;)
  pickup_hour &lt;- addNA(cut(hour(pickup_datetime), cut_levels))
  pickup_dow &lt;- factor(wday(pickup_datetime), levels = 1:7, labels = weekday_labels)
  levels(pickup_hour) &lt;- hour_labels
  # 
  dropoff_datetime &lt;- lubridate::ymd_hms(data$tpep_dropoff_datetime, tz = &quot;UTC&quot;)
  dropoff_hour &lt;- addNA(cut(hour(dropoff_datetime), cut_levels))
  dropoff_dow &lt;- factor(wday(dropoff_datetime), levels = 1:7, labels = weekday_labels)
  levels(dropoff_hour) &lt;- hour_labels
  # 
  data$pickup_hour &lt;- pickup_hour
  data$pickup_dow &lt;- pickup_dow
  data$dropoff_hour &lt;- dropoff_hour
  data$dropoff_dow &lt;- dropoff_dow
  data$trip_duration &lt;- as.integer(lubridate::interval(pickup_datetime, dropoff_datetime))
  
  return(data)
}</code></pre>
<p>We defined our function above, and now we apply it to our dataset. In order to ensure our function works on our data, we could try it out on a sample dataset locally first by reverting to a local compute context:</p>
<pre><code>x &lt;- head(taxi_tip)
rxSetComputeContext(&quot;local&quot;)

rxDataStep(inData = x, 
           outFile = NULL, 
           transformFunc = xforms, 
           transformPackages = &quot;lubridate&quot;)</code></pre>
<pre><code>##   VendorID tpep_pickup_datetime tpep_dropoff_datetime passenger_count
## 1        1  2015-07-01 00:00:00   2015-07-01 00:15:26               1
## 2        1  2015-07-01 00:00:00   2015-07-01 00:22:22               1
## 3        1  2015-07-01 00:00:00   2015-07-01 00:07:42               1
## 4        1  2015-07-01 00:00:00   2015-07-01 00:39:37               1
## 5        1  2015-07-01 00:00:00   2015-07-01 00:05:34               1
## 6        1  2015-07-01 00:00:00   2015-07-01 00:06:46               2
##   trip_distance pickup_longitude pickup_latitude RatecodeID
## 1           3.5        -73.99416        40.75113          1
## 2           3.9        -73.98466        40.76849          1
## 3           2.3        -73.97889        40.76229          1
## 4           9.2        -73.99279        40.74276          1
## 5           1.1        -73.91243        40.76981          1
## 6           1.0        -73.95916        40.77343          1
##   store_and_fwd_flag dropoff_longitude dropoff_latitude payment_type
## 1                  N         -73.97682         40.78857            2
## 2                  N         -74.00013         40.73490            2
## 3                  N         -74.00422         40.75253            2
## 4                  N         -73.97151         40.63715            1
## 5                  N         -73.92033         40.75744            1
## 6                  N         -73.96935         40.76925            2
##   fare_amount extra mta_tax tip_amount tolls_amount improvement_surcharge
## 1        14.0   0.5     0.5       0.00            0                   0.3
## 2        17.0   0.5     0.5       0.00            0                   0.3
## 3         9.0   0.5     0.5       0.00            0                   0.3
## 4        33.0   0.5     0.5       8.55            0                   0.3
## 5         6.0   0.5     0.5       2.00            0                   0.3
## 6         6.5   0.5     0.5       0.00            0                   0.3
##   total_amount tip_percent pickup_hour pickup_dow dropoff_hour dropoff_dow
## 1        15.30   0.0000000    10PM-1AM        Wed     10PM-1AM         Wed
## 2        18.30   0.0000000    10PM-1AM        Wed     10PM-1AM         Wed
## 3        10.30   0.0000000    10PM-1AM        Wed     10PM-1AM         Wed
## 4        42.85   0.2590909    10PM-1AM        Wed     10PM-1AM         Wed
## 5         9.30   0.3333333    10PM-1AM        Wed     10PM-1AM         Wed
## 6         7.80   0.0000000    10PM-1AM        Wed     10PM-1AM         Wed
##   trip_duration
## 1           926
## 2          1342
## 3           462
## 4          2377
## 5           334
## 6           406</code></pre>
<pre><code>rxSetComputeContext(spark_cc)</code></pre>
<p>We used a local compute context to try our function on the sample locally, then reverted to the spark context for doing our analysis on the cluster.</p>
<pre><code>taxi_date &lt;- RxXdfData(&quot;/user/RevoShare/alizaidi/TaxiDatesTranf&quot;,
                       fileSystem = hdfsFS)</code></pre>
<pre><code>rxDataStep(inData = taxi_tip, 
           outFile = taxi_date, 
           transformFunc = xforms, 
           transformPackages = &quot;lubridate&quot;)</code></pre>
<pre><code>rxGetInfo(taxi_date, numRows = 5, getVarInfo = TRUE)</code></pre>
<pre><code>## File name: /user/RevoShare/alizaidi/TaxiDatesTranf 
## Number of composite data files: 48 
## Number of observations: 138413407 
## Number of variables: 25 
## Number of blocks: 289 
## Compression type: zlib 
## Variable information: 
## Var 1: VendorID, Type: integer, Low/High: (1, 2)
## Var 2: tpep_pickup_datetime, Type: character
## Var 3: tpep_dropoff_datetime, Type: character
## Var 4: passenger_count, Type: integer, Low/High: (0, 9)
## Var 5: trip_distance, Type: numeric, Storage: float32, Low/High: (-3390583.7500, 198623008.0000)
## Var 6: pickup_longitude, Type: numeric, Storage: float32, Low/High: (-736.6166, 172.6000)
## Var 7: pickup_latitude, Type: numeric, Storage: float32, Low/High: (-78.1947, 80.6025)
## Var 8: RatecodeID, Type: integer, Low/High: (1, 99)
## Var 9: store_and_fwd_flag, Type: character
## Var 10: dropoff_longitude, Type: numeric, Storage: float32, Low/High: (-781.8333, 172.6000)
## Var 11: dropoff_latitude, Type: numeric, Storage: float32, Low/High: (-78.1947, 480.7333)
## Var 12: payment_type, Type: integer, Low/High: (1, 5)
## Var 13: fare_amount, Type: numeric, Storage: float32, Low/High: (-957.6000, 825998.6250)
## Var 14: extra, Type: numeric, Storage: float32, Low/High: (-58.5000, 648.8700)
## Var 15: mta_tax, Type: numeric, Storage: float32, Low/High: (-3.0000, 91.0000)
## Var 16: tip_amount, Type: numeric, Storage: float32, Low/High: (-440.0000, 1200.8000)
## Var 17: tolls_amount, Type: numeric, Storage: float32, Low/High: (-99.9900, 1901.4000)
## Var 18: improvement_surcharge, Type: numeric, Storage: float32, Low/High: (-0.3000, 137.6300)
## Var 19: total_amount, Type: numeric, Storage: float32, Low/High: (-958.4000, 826040.0000)
## Var 20: tip_percent, Type: numeric, Low/High: (-1.0000, 54900.0012)
## Var 21: pickup_hour
##        7 factor levels: 1AM-5AM 5AM-9AM 9AM-12PM 12PM-4PM 4PM-6PM 6PM-10PM 10PM-1AM
## Var 22: pickup_dow
##        7 factor levels: Sun Mon Tue Wed Thu Fri Sat
## Var 23: dropoff_hour
##        7 factor levels: 1AM-5AM 5AM-9AM 9AM-12PM 12PM-4PM 4PM-6PM 6PM-10PM 10PM-1AM
## Var 24: dropoff_dow
##        7 factor levels: Sun Mon Tue Wed Thu Fri Sat
## Var 25: trip_duration, Type: integer, Low/High: (-631148790, 29227264)
## Data (5 rows starting with row 1):
##   VendorID tpep_pickup_datetime tpep_dropoff_datetime passenger_count
## 1        1  2015-07-01 00:00:00   2015-07-01 00:15:26               1
## 2        1  2015-07-01 00:00:00   2015-07-01 00:22:22               1
## 3        1  2015-07-01 00:00:00   2015-07-01 00:07:42               1
## 4        1  2015-07-01 00:00:00   2015-07-01 00:39:37               1
## 5        1  2015-07-01 00:00:00   2015-07-01 00:05:34               1
##   trip_distance pickup_longitude pickup_latitude RatecodeID
## 1           3.5        -73.99416        40.75113          1
## 2           3.9        -73.98466        40.76849          1
## 3           2.3        -73.97889        40.76229          1
## 4           9.2        -73.99279        40.74276          1
## 5           1.1        -73.91243        40.76981          1
##   store_and_fwd_flag dropoff_longitude dropoff_latitude payment_type
## 1                  N         -73.97682         40.78857            2
## 2                  N         -74.00013         40.73490            2
## 3                  N         -74.00422         40.75253            2
## 4                  N         -73.97151         40.63715            1
## 5                  N         -73.92033         40.75744            1
##   fare_amount extra mta_tax tip_amount tolls_amount improvement_surcharge
## 1          14   0.5     0.5       0.00            0                   0.3
## 2          17   0.5     0.5       0.00            0                   0.3
## 3           9   0.5     0.5       0.00            0                   0.3
## 4          33   0.5     0.5       8.55            0                   0.3
## 5           6   0.5     0.5       2.00            0                   0.3
##   total_amount tip_percent pickup_hour pickup_dow dropoff_hour dropoff_dow
## 1        15.30   0.0000000    10PM-1AM        Wed     10PM-1AM         Wed
## 2        18.30   0.0000000    10PM-1AM        Wed     10PM-1AM         Wed
## 3        10.30   0.0000000    10PM-1AM        Wed     10PM-1AM         Wed
## 4        42.85   0.2590909    10PM-1AM        Wed     10PM-1AM         Wed
## 5         9.30   0.3333333    10PM-1AM        Wed     10PM-1AM         Wed
##   trip_duration
## 1           926
## 2          1342
## 3           462
## 4          2377
## 5           334</code></pre>
<p>As you see from the output of the chunk above, our function creates a set of columns defining some useful temporal features. Our hope is that these temporal features will give us a better understanding of the distribution of trips as a function of temporal variables.</p>
<p>For example, we can now examine how the distribution of tip percents vary as a function of day of week and pickup hour:</p>
<pre><code>tip_dist_df &lt;- rxCube(tip_percent ~ pickup_hour + pickup_dow, data = taxi_date, returnDataFrame = TRUE)

library(ggplot2)
library(magrittr)

tip_dist_df %>% ggplot(aes(x = pickup_hour, y = pickup_dow, fill = tip_percent)) +
  geom_tile() + theme_minimal() + 
  scale_fill_continuous(label = scales::percent) +
  labs(x = &quot;Pickup Hour&quot;, y = &quot;Pickup Day of Week&quot;, fill = &quot;Tip Percent&quot;,
      title = &quot;Distribution of Tip Percents&quot;,
      subtitle = &quot;Do Passengers Tip More in the AM?&quot;)</code></pre>
<p><a target="_blank" rel="nofollow" href="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b7c8bcc43f970b-pi"><img title="Tip_percent-1" alt="Tip_percent-1" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b7c8bcc43f970b-800wi" border="0"></a></p>
</div>
<div>
<h3>Crafting Spatial Features</h3>
<p>While temporal features give us an interesting look at how taxi trips vary as a function of time, the trips data contains a wealth of information about spatial features. Currently, the spatial features are all consolidated into the coordinates columns, which are numeric values. In order to reduce the variability of the numeric features, and encode them into more sensible categories, let’s see if we can transform the coordinate columns into a column of categorical features encoding the neighborhood of pickup and dropoff.</p>
<p>Our UDF will examine the rows of our data set, and map the spatial coordinates to a shapefile that contains the neighborhoods of NYC. The shapefile we will use is from <a target="_blank" rel="nofollow" href="http://www.zillow.com/static/shp/ZillowNeighborhoods-NY.zip">Zillow</a>:</p>
<pre><code>library(rgeos)
library(sp)
library(maptools)
library(stringr)

nyc_shapefile &lt;- readShapePoly('ZillowNeighborhoods-NY/ZillowNeighborhoods-NY.shp')
mht_shapefile &lt;- subset(nyc_shapefile, str_detect(CITY, 'New York City-Manhattan'))

mht_shapefile@data$id &lt;- as.character(mht_shapefile@data$NAME)</code></pre>
<p>We will use the shapefile to map the spatial coordinates of our obsevations to the nearest neighborhood given by the shapefile:</p>
<pre><code>find_nhoods &lt;- function(data) {
  
  # extract pick-up lat and long and find their neighborhoods
  pickup_longitude &lt;- ifelse(is.na(data$pickup_longitude), 0, data$pickup_longitude)
  pickup_latitude &lt;- ifelse(is.na(data$pickup_latitude), 0, data$pickup_latitude)
  data_coords &lt;- data.frame(long = pickup_longitude, lat = pickup_latitude)
  coordinates(data_coords) &lt;- c('long', 'lat')
  nhoods &lt;- over(data_coords, shapefile)
  
  ## add only the pick-up neighborhood and city columns to the data
  data$pickup_nhood &lt;- nhoods$NAME
  data$pickup_borough &lt;- nhoods$CITY
  
  # extract drop-off lat and long and find their neighborhoods
  dropoff_longitude &lt;- ifelse(is.na(data$dropoff_longitude), 0, data$dropoff_longitude)
  dropoff_latitude &lt;- ifelse(is.na(data$dropoff_latitude), 0, data$dropoff_latitude)
  data_coords &lt;- data.frame(long = dropoff_longitude, lat = dropoff_latitude)
  coordinates(data_coords) &lt;- c('long', 'lat')
  nhoods &lt;- over(data_coords, shapefile)
  
  ## add only the drop-off neighborhood and city columns to the data  
  data$dropoff_nhood &lt;- nhoods$NAME
  data$dropoff_borough &lt;- nhoods$CITY
  
  ## return the data with the new columns added in
  data
}</code></pre>
<p>Let’s again test this function locally first:</p>
<pre><code>rxSetComputeContext(&quot;local&quot;)

head(rxDataStep(nyc_sample_df, transformFunc = find_nhoods, transformPackages = c(&quot;sp&quot;, &quot;maptools&quot;), 
                transformObjects = list(shapefile = mht_shapefile)))</code></pre>
<pre><code>##   VendorID tpep_pickup_datetime tpep_dropoff_datetime passenger_count
## 1        1  2016-05-01 00:00:00   2016-05-01 00:17:31               1
## 2        2  2016-05-01 00:00:00   2016-05-01 00:07:31               1
## 3        2  2016-05-01 00:00:00   2016-05-01 00:07:01               6
## 4        2  2016-05-01 00:00:00   2016-05-01 00:19:47               1
## 5        2  2016-05-01 00:00:00   2016-05-01 00:06:39               1
## 6        2  2016-05-01 00:00:00   2016-05-01 00:05:19               2
##   trip_distance pickup_longitude pickup_latitude RatecodeID
## 1          3.60        -73.98590        40.76804          1
## 2          1.68        -73.99158        40.74475          1
## 3          1.09        -73.99307        40.74157          1
## 4          4.21        -73.99194        40.68460          1
## 5          0.56        -74.00528        40.74019          1
## 6          0.63        -73.97929        40.75576          1
##   store_and_fwd_flag dropoff_longitude dropoff_latitude payment_type
## 1                  N         -73.98399         40.73010            1
## 2                  N         -73.97570         40.76547            1
## 3                  N         -73.98100         40.74463            1
## 4                  N         -74.00226         40.73300            1
## 5                  N         -73.99750         40.73756            1
## 6                  N         -73.98801         40.75847            1
##   fare_amount extra mta_tax tip_amount tolls_amount improvement_surcharge
## 1        15.0   0.5     0.5       1.50            0                   0.3
## 2         7.5   0.5     0.5       0.88            0                   0.3
## 3         6.5   0.5     0.5       1.56            0                   0.3
## 4        17.0   0.5     0.5       3.66            0                   0.3
## 5         6.0   0.5     0.5       1.46            0                   0.3
## 6         5.0   0.5     0.5       0.00            0                   0.3
##   total_amount pickup_nhood          pickup_borough     dropoff_nhood
## 1        17.80      Midtown New York City-Manhattan      East Village
## 2         9.68      Chelsea New York City-Manhattan      Central Park
## 3         9.36      Chelsea New York City-Manhattan          Gramercy
## 4        21.96         &lt;NA>                    &lt;NA> Greenwich Village
## 5         8.76 West Village New York City-Manhattan Greenwich Village
## 6         6.30      Midtown New York City-Manhattan           Midtown
##           dropoff_borough
## 1 New York City-Manhattan
## 2 New York City-Manhattan
## 3 New York City-Manhattan
## 4 New York City-Manhattan
## 5 New York City-Manhattan
## 6 New York City-Manhattan</code></pre>
<pre><code>rxSetComputeContext(spark_cc)</code></pre>
<p>Looks like it worked locally, let’s try it on our full dataset:</p>
<pre><code>taxi_hoods &lt;- RxXdfData(&quot;/user/RevoShare/alizaidi/TaxiHoodsXdf&quot;,
                       fileSystem = hdfsFS)</code></pre>
<pre><code>rxDataStep(taxi_date, taxi_hoods, 
           transformFunc = find_nhoods, 
           transformPackages = c(&quot;sp&quot;, &quot;maptools&quot;, &quot;rgeos&quot;), 
           transformObjects = list(shapefile = mht_shapefile))</code></pre>
<pre><code>rxGetInfo(taxi_hoods, numRows = 5)</code></pre>
<pre><code>## File name: /user/RevoShare/alizaidi/TaxiHoodsXdf 
## Number of composite data files: 48 
## Number of observations: 138413407 
## Number of variables: 29 
## Number of blocks: 289 
## Compression type: zlib 
## Data (5 rows starting with row 1):
##   VendorID tpep_pickup_datetime tpep_dropoff_datetime passenger_count
## 1        1  2015-07-01 00:00:00   2015-07-01 00:15:26               1
## 2        1  2015-07-01 00:00:00   2015-07-01 00:22:22               1
## 3        1  2015-07-01 00:00:00   2015-07-01 00:07:42               1
## 4        1  2015-07-01 00:00:00   2015-07-01 00:39:37               1
## 5        1  2015-07-01 00:00:00   2015-07-01 00:05:34               1
##   trip_distance pickup_longitude pickup_latitude RatecodeID
## 1           3.5        -73.99416        40.75113          1
## 2           3.9        -73.98466        40.76849          1
## 3           2.3        -73.97889        40.76229          1
## 4           9.2        -73.99279        40.74276          1
## 5           1.1        -73.91243        40.76981          1
##   store_and_fwd_flag dropoff_longitude dropoff_latitude payment_type
## 1                  N         -73.97682         40.78857            2
## 2                  N         -74.00013         40.73490            2
## 3                  N         -74.00422         40.75253            2
## 4                  N         -73.97151         40.63715            1
## 5                  N         -73.92033         40.75744            1
##   fare_amount extra mta_tax tip_amount tolls_amount improvement_surcharge
## 1          14   0.5     0.5       0.00            0                   0.3
## 2          17   0.5     0.5       0.00            0                   0.3
## 3           9   0.5     0.5       0.00            0                   0.3
## 4          33   0.5     0.5       8.55            0                   0.3
## 5           6   0.5     0.5       2.00            0                   0.3
##   total_amount tip_percent pickup_hour pickup_dow dropoff_hour dropoff_dow
## 1        15.30   0.0000000    10PM-1AM        Wed     10PM-1AM         Wed
## 2        18.30   0.0000000    10PM-1AM        Wed     10PM-1AM         Wed
## 3        10.30   0.0000000    10PM-1AM        Wed     10PM-1AM         Wed
## 4        42.85   0.2590909    10PM-1AM        Wed     10PM-1AM         Wed
## 5         9.30   0.3333333    10PM-1AM        Wed     10PM-1AM         Wed
##   trip_duration     pickup_nhood          pickup_borough     dropoff_nhood
## 1           926 Garment District New York City-Manhattan   Upper West Side
## 2          1342          Midtown New York City-Manhattan Greenwich Village
## 3           462          Midtown New York City-Manhattan           Chelsea
## 4          2377          Chelsea New York City-Manhattan              &lt;NA>
## 5           334             &lt;NA>                    &lt;NA>              &lt;NA>
##           dropoff_borough
## 1 New York City-Manhattan
## 2 New York City-Manhattan
## 3 New York City-Manhattan
## 4                    &lt;NA>
## 5                    &lt;NA></code></pre>
</div>
<div>
<h3>Filter Data to Manhattan Only</h3>
<p>Since Manhattan is the busiest of boroughs in NYC, we will narrow our focus to do the trips that started and ended in that borough. Moreover, we will filter out possible outlier observations as well:</p>
<pre><code>mht_xdf &lt;- RxXdfData(&quot;/user/RevoShare/alizaidi/ManhattanXdf&quot;,
                     fileSystem = hdfsFS)</code></pre>
<pre><code>rxDataStep(mht_hoods, mht_xdf, 
           rowSelection = (
             passenger_count > 0 &amp;
               trip_distance >= 0 &amp; trip_distance &lt; 30 &amp;
               trip_duration > 0 &amp; trip_duration &lt; 60*60*24 &amp;
               str_detect(pickup_borough, 'Manhattan') &amp;
               str_detect(dropoff_borough, 'Manhattan') &amp;
               !is.na(pickup_nb) &amp;
               !is.na(dropoff_nb) &amp;
               fare_amount > 0), 
           transformPackages = &quot;stringr&quot;,
           varsToDrop = c('extra', 'mta_tax', 'improvement_surcharge', 'total_amount', 
                          'pickup_borough', 'dropoff_borough', 'pickup_nhood', 'dropoff_nhood'))</code></pre>
<p>Now that we have our spatial features, let’s see if we can make a visualization of the trips as a function time of trip as well as the trip route.</p>
<pre><code>library(dplyr)

nbs_df &lt;- rxCube(~ pickup_nb + dropoff_nb + pickup_hour, data = mht_xdf, returnDataFrame = TRUE)
nbs_df &lt;- nbs_df %>% tbl_df %>%
  filter(Counts >= 100) %>% 
  mutate(width = ntile(Counts, 5))</code></pre>
<p>We used the beloved <code>dplyr</code> package in conjuction with the <code>RevoScaleR</code> <code>rxCube</code> function to tabulate the counts of trips as a function of route and pickup hour, and then filtered to only those routes that had at least 100 observations in our dataset. Finally, we used the handy <code>ntile</code> function in <code>dplyr</code> to create a variable to measure the strength of relationship of pickup and dropoff neighborhoods as a 20% percentile of all trips.</p>
<p>Now that we have the counts in a tidy <code>data.frame</code> object, we can make visualizations of the route. We’ll use the exceptional <a target="_blank" rel="nofollow" href="https://github.com/jokergoo/circlize"><code>circlize</code></a> package to make visualizations, one for each level of the <code>pickup_hour</code> factor. In order to map each of these to the visualization, we will first make a <em>functional sequence</em> which will contain the methods to make our visualization, and then use the <a target="_blank" rel="nofollow" href="http://blog.revolutionanalytics.com/2016/12/github.com/hadley/purrr"><code>purrr</code></a> package to map each level of the factor column into the functional sequence. In order to make our plots a little less “hair-ballish”, we’ll narrow our focus to a few of the most popular neighborhoods, as specified through the <code>nbs</code> vector (please don’t be offended if your favorite neighborhood was ommitted):</p>
<pre><code>library(purrr)
library(circlize)

nbs &lt;- c(&quot;Lower East Side&quot;,&quot;East Village&quot;,
          &quot;Chelsea&quot;, &quot;Midtown&quot;, &quot;Upper East Side&quot;,
         &quot;Upper West Side&quot;, &quot;Greenwich Vilalge&quot;)


chord_diag &lt;- . %>% select(pickup_nb, dropoff_nb, width) %>% chordDiagram()

chord_plot &lt;- nbs_df %>%
  filter(pickup_nb %in% nbs,
         dropoff_nb %in% nbs) %>% 
  split(.$pickup_hour) %>%
  map(chord_diag)<br><br></code></pre>
<p>Here's the chart for the morning rush hour (5AM-9AM):<br> <a target="_blank" rel="nofollow" href="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b8d2467b00970c-pi"><img title="Chord_plots-2" alt="Chord_plots-2" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b8d2467b00970c-800wi" border="0"></a></p>
</div>
</div>
<div>
<h2>Conclusion</h2>
<p>Hopefully this post provided some insight on how we could <code>RevoScaleR</code> to develop scalable data analysis pipelines that can be deployed onto Spark clusters easily. We developed some intuitive features from our data set based on temporal and spatial characteristics of the original data. Stay tuned for another post where we show how you can use these features for prediction and inference!</p>
</div>
</div>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://blog.revolutionanalytics.com/2016/12/taxi-mrs-spark.html"> Revolutions</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/TDx3MLEl6jM"></a></div>