<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="http://www.r-bloggers.com/time-series-analysis-in-arcgis/"></div></div><div><a data-url="http://www.r-bloggers.com/time-series-analysis-in-arcgis/" data-counturl="http://www.r-bloggers.com/time-series-analysis-in-arcgis/" data-count="horizontal" data-text="Time Series Analysis in ArcGIS" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://r-video-tutorial.blogspot.com/2016/07/time-series-analysis-in-arcgis.html"> R tutorial for Spatial Statistics</a></strong>, and kindly contributed to <a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>In this post I will introduce another toolbox I created to show the functions that can be added to ArcGIS by using R and the <a target="_blank" href="https://r-arcgis.github.io/">R-Bridge</a> technology.<br>In this toolbox I basically implemented the functions I showed in the previous post about <a target="_blank" href="http://r-video-tutorial.blogspot.ch/2015/05/introductory-time-series-analysis-of-us.html">time series analysis in R</a>.<br>Once again I prepared a sample dataset that I included in the GitHub archive so that you can reproduce the experiment I’m presenting here. I will start my description from there.</p>
<p><span>Dataset</span><br>As for my previous post, here I’m also including open data in shapefile from the EPA, which I downloaded for free using the custom R function I presented <a target="_blank" href="http://r-video-tutorial.blogspot.ch/2015/05/introductory-time-series-analysis-of-us.html">here</a>.<br>I downloaded only temperature data (in F) from 2013, but I kept two categorical variables: State and Address.</p>
<div><a target="_blank" href="http://i0.wp.com/2.bp.blogspot.com/-86003gp5zOc/V4iJjKjgnoI/AAAAAAAASBc/yzxisNSfeJgaHm4Ues39BTCqZAB_08R2ACLcB/s1600/Fig1.jpg?ssl=1"><img src="http://i2.wp.com/2.bp.blogspot.com/-86003gp5zOc/V4iJjKjgnoI/AAAAAAAASBc/yzxisNSfeJgaHm4Ues39BTCqZAB_08R2ACLcB/s640/Fig1.jpg?resize=456%2C208&ssl=1" border="0" data-recalc-dims="1"></a></div>
<p>As you can see from the image above the time variable is in the format year-month-day. As I mentioned in the post about the <a target="_blank" href="http://r-video-tutorial.blogspot.ch/2016/07/the-power-of-ggplot2-in-arcgis-plotting.html">plotting toolbox</a>, it is important to set this format correctly so that R can recognize it. Please refer to <a target="_blank" href="http://www.inside-r.org/r-doc/base/strptime">this page</a> for more information about the formats that R recognizes.</p>
<p><span>Time Series Plot</span><br>This type of plot is available in several packages, including <code>ggplot2</code>, which I used to create the <a target="_blank" href="http://r-video-tutorial.blogspot.ch/2016/07/the-power-of-ggplot2-in-arcgis-plotting.html">plotting toolbox</a>.  However, in my post about time series analysis I presented the package <code>xts</code>, which is very powerful for handling and plotting time-series data. For this toolbox I decided to maintain the same package and refer everything to <code>xts</code> for several reasons that I would explain along the text.<br>The first reason is related to the plotting capabilities of this package. Let’s take a look for example at the first script in the toolbox, specific for plotting time series.</p>
<div><a target="_blank" href="http://i1.wp.com/1.bp.blogspot.com/-NdBxWw-RuC0/V4jA-d0k_gI/AAAAAAAASBs/YE3lf9Dt3dwaEEca_vjwKAxhY0DAqQehgCLcB/s1600/Fig2.jpg?ssl=1"><img src="http://i2.wp.com/1.bp.blogspot.com/-NdBxWw-RuC0/V4jA-d0k_gI/AAAAAAAASBs/YE3lf9Dt3dwaEEca_vjwKAxhY0DAqQehgCLcB/s640/Fig2.jpg?resize=456%2C212&ssl=1" border="0" data-recalc-dims="1"></a></div>
<p>Similarly to the script for time series in the plotting toolbox, here users need to select the dataset (which can be a shapefile or a CSV, or any other table format that can be accessed in ArcGIS). Then they need to select the variable of interest, in the sample dataset that is Temp, which clearly stands for temperature. Another important information for R is the data/time column and its format, again please refer to my <a target="_blank" href="http://r-video-tutorial.blogspot.ch/2016/07/the-power-of-ggplot2-in-arcgis-plotting.html">previous post</a> for more information. Finally, I inserted an SQL call to subset the dataset. In this case I’m subsetting a particular station.<br>The result is the plot below:</p>
<div><a target="_blank" href="http://i2.wp.com/4.bp.blogspot.com/-faY43M_WIgg/V4jDOaQr2JI/AAAAAAAASB4/SabXKwQR8Wsw8sv-IdOs0CNkxOxwH2x6QCLcB/s1600/Fig3.jpg?ssl=1"><img src="http://i1.wp.com/4.bp.blogspot.com/-faY43M_WIgg/V4jDOaQr2JI/AAAAAAAASB4/SabXKwQR8Wsw8sv-IdOs0CNkxOxwH2x6QCLcB/s640/Fig3.jpg?resize=456%2C241&ssl=1" border="0" data-recalc-dims="1"></a></div>
<p>As you can see there are quite a few missing values in the dataset related to the station I subset. The very nice thing about the package <code>xts</code> is that with this plot it is perfectly clear where are the missing data, since along the X axis these are evident by the lack of grey tick marks.</p>
<p><span>Time Histogram</span><br>This is a simple bar chart that basically plots time against frequency of samples. The idea behind this plot is to allow users to explore the number of samples for specific time intervals in the dataset.</p>
<div><a target="_blank" href="http://i2.wp.com/4.bp.blogspot.com/-0YNUpLdFBVA/V4jK74mFbaI/AAAAAAAASCI/yzrASlip7koE-1jm3vNoonSqc8sluqyXACLcB/s1600/Fig4.jpg?ssl=1"><img src="http://i1.wp.com/4.bp.blogspot.com/-0YNUpLdFBVA/V4jK74mFbaI/AAAAAAAASCI/yzrASlip7koE-1jm3vNoonSqc8sluqyXACLcB/s640/Fig4.jpg?resize=456%2C214&ssl=1" border="0" data-recalc-dims="1"></a></div>
<p>The user interface is similar to the previous scrips. Users need to select the dataset, then the variable and then the time column and specify its format. I also included an option to select a subset of the dataset with a SQL selection. At this point I included a list to select the averaging period, and users can select between day, month or year. In this case I selected month, which means that R will loop through the months and subset the dataset for each of these. Then it will basically count the number of data sampled in each month and plot this information against the month itself. The result is the plot below:</p>
<div><a target="_blank" href="http://i1.wp.com/2.bp.blogspot.com/-VF1NS82-a0Y/V4jMA8R0BnI/AAAAAAAASCU/8VoYlHanxrkgcMi7kLq1QV6Q47pxJ0lSQCLcB/s1600/Fig5.jpg?ssl=1"><img src="http://i1.wp.com/2.bp.blogspot.com/-VF1NS82-a0Y/V4jMA8R0BnI/AAAAAAAASCU/8VoYlHanxrkgcMi7kLq1QV6Q47pxJ0lSQCLcB/s640/Fig5.jpg?resize=456%2C239&ssl=1" border="0" data-recalc-dims="1"></a></div>
<p>As you can see we can definitely gather some useful information from this plot; for example we can determine that basically this station, in the year 2013, did not have any problem.</p>
<p><span>Time Subset</span><br>In some cases we may need to subset our dataset according to a particular time period. This can be done in ArcGIS with the “Select by Attribute” tool and by using an SQL string similar to what you see in the image below:</p>
<div><a target="_blank" href="http://i0.wp.com/1.bp.blogspot.com/-C-XBngiR60s/V4jO6yIbchI/AAAAAAAASCg/yepwRI013xQmCYM4TS8vak3ANI53d8W1gCLcB/s1600/Fig7.jpg?ssl=1"><img src="http://i2.wp.com/1.bp.blogspot.com/-C-XBngiR60s/V4jO6yIbchI/AAAAAAAASCg/yepwRI013xQmCYM4TS8vak3ANI53d8W1gCLcB/s640/Fig7.jpg?resize=456%2C281&ssl=1" border="0" data-recalc-dims="1"></a></div>
<p>The package <code>xts</code> however, provides much more powerful and probably faster ways to subset by time. For example, in ArcGIS if we want to subset the whole month of June we would need to specify an SQL string like this:<br>“Time” >= ‘2013-06-01’ AND “Time” &lt; ‘2013-07-01’</p>
<p>On the contrary, in R and with the package <code>xts</code> if we wanted to do the same we would just need to use the string <code>'2013-06'</code>, and R would know to keep only the month of June. Below are some other examples of successful time subset with the package <code>xts</code> (from <a target="_blank" href="http://www.inside-r.org/packages/cran/xts/docs/xts">http://www.inside-r.org/packages/cran/xts/docs/xts</a>):</p>
<p><code>sample.xts['2013']  # all of 2013<br>sample.xts['2013-03']  # just March 2013<br>sample.xts['2013-03/']  # March 2013 to the end of the data set<br>sample.xts['2013-03/2013']  # March 2013 to the end of 2013<br>sample.xts['/'] # the whole data set<br>sample.xts['/2013'] # the beginning of the data through 2013<br>sample.xts['2013-01-03'] # just the 3rd of January 2013</code></p>
<p>With this in mind I created the following script:</p>
<div><a target="_blank" href="http://i2.wp.com/4.bp.blogspot.com/-4ZpQ9vt7IXc/V4jjR74vJTI/AAAAAAAASCw/C19e5FKn-WMWVF913rBE_PPoceaF50HlACLcB/s1600/Fig6.jpg?ssl=1"><img src="http://i1.wp.com/4.bp.blogspot.com/-4ZpQ9vt7IXc/V4jjR74vJTI/AAAAAAAASCw/C19e5FKn-WMWVF913rBE_PPoceaF50HlACLcB/s640/Fig6.jpg?resize=456%2C218&ssl=1" border="0" data-recalc-dims="1"></a></div>
<p>As you can see from the image above, here there is an option named “Subset” where users can insert one of the strings from the examples above (just the text within square brackets) and select time intervals with the same flexibility allowed in R and the package xts.<br>The result of this script is a new shapefile containing only the time included in the Subset call.</p>
<p><span>Time Average Plots</span><br>As I showed in my previous post about <a target="_blank" href="http://r-video-tutorial.blogspot.ch/2015/05/introductory-time-series-analysis-of-us.html">time series analysis</a>, with the package <code>xts</code> is possible to perform custom functions on specific time intervals with the following commands: <code>apply.daily</code>, <code>apply.weekly</code>, <code>apply.monthly</code> and <code>apply.yearly</code>.<br>In this toolbox I used these functions to compute the average, 25th and 75th percentiles for specific time intervals, which the user may choose. This is the toolbox:</p>
<div><a target="_blank" href="http://i2.wp.com/1.bp.blogspot.com/-JZJezYZuuBM/V4jm7_YW1SI/AAAAAAAASC8/PRZYafqwvFw5Ln6Ii--jM1FHHcLyhFUQgCLcB/s1600/Fig8.jpg?ssl=1"><img src="http://i2.wp.com/1.bp.blogspot.com/-JZJezYZuuBM/V4jm7_YW1SI/AAAAAAAASC8/PRZYafqwvFw5Ln6Ii--jM1FHHcLyhFUQgCLcB/s640/Fig8.jpg?resize=456%2C218&ssl=1" border="0" data-recalc-dims="1"></a></div>
<p>The only differences from the other scripts are the “Average by”, with which the user can select between day, week, month or year. Each of these will trigger the appropriate apply function. Then there is also the possibility to select the position for the plot legend: between topright, topleft, bottomright and bottomleft. Finally, users can select the output folder where the plot below will be saved, along with a CSV with the numerical values for mean, q25 and q75.</p>
<div><a target="_blank" href="http://i2.wp.com/2.bp.blogspot.com/-51BVFYLxJo0/V4j-VBe8-CI/AAAAAAAASDM/pw1hQKSxQXQHzCLkrurBo2_mZ0Hv_Jk7QCLcB/s1600/Fig9.jpg?ssl=1"><img src="http://i1.wp.com/2.bp.blogspot.com/-51BVFYLxJo0/V4j-VBe8-CI/AAAAAAAASDM/pw1hQKSxQXQHzCLkrurBo2_mZ0Hv_Jk7QCLcB/s640/Fig9.jpg?resize=456%2C244&ssl=1" border="0" data-recalc-dims="1"></a></div>
<p><span>Time Function</span><br>This is another script that provides direct access to the apply functions I presented before. Here the output is not a plot but a CSV with the results of the function, and users can input their own function directly in the GUI. Let’s take a look:</p>
<div><a target="_blank" href="http://i0.wp.com/3.bp.blogspot.com/-z1F5AFI8pA8/V4kAs9PJXlI/AAAAAAAASDY/mYXuAuykDqAx0xfJfO_OrfP1qPOU0vwcACLcB/s1600/Fig10.jpg?ssl=1"><img src="http://i1.wp.com/3.bp.blogspot.com/-z1F5AFI8pA8/V4kAs9PJXlI/AAAAAAAASDY/mYXuAuykDqAx0xfJfO_OrfP1qPOU0vwcACLcB/s640/Fig10.jpg?resize=456%2C218&ssl=1" border="0" data-recalc-dims="1"></a></div>
<p>As you can see there is a field named “Function”. Here users can insert their own custom function, written in the R language. This function takes a vector (<i>x</i>) and returns a vector and it is in the form:</p>
<p><code>function(x){sum(x>70}</code></p>
<p>Only the string within curly brackets needs to be written in the GUI. This will then be passed to the script and applied to the values averaged by day, week, month or year. Users can select this last aspect in the field “Average by”. Here for example I am calculating the number of days, for each month, with a temperature above 70 degrees Fahrenheit (21 degrees celsius) in Alaska. The results are saved in CSV in the output folder and printed on screen, as you can see from the image below.</p>
<div><a target="_blank" href="http://i2.wp.com/4.bp.blogspot.com/-epl40dOolVk/V4kDwovanaI/AAAAAAAASDk/-iuS34mW8iMuC4crfcxWrr_7RV6uQ6QgwCLcB/s1600/Fig11.jpg?ssl=1"><img src="http://i2.wp.com/4.bp.blogspot.com/-epl40dOolVk/V4kDwovanaI/AAAAAAAASDk/-iuS34mW8iMuC4crfcxWrr_7RV6uQ6QgwCLcB/s640/Fig11.jpg?resize=456%2C348&ssl=1" border="0" data-recalc-dims="1"></a></div>
<p><span>Trend Analysis</span><br>In this last script I included access to the function decompose, which I briefly described in my <a target="_blank" href="http://r-video-tutorial.blogspot.ch/2015/05/introductory-time-series-analysis-of-us.html">previous post</a>. This function does not work with <code>xts</code> time series, so the time series needs to be loaded with the standard method, <code>ts</code>, in R. This method requires the user to include the frequency of the time series. For this reason I had to add an option for this in the GUI.<br>Unfortunately, the dataset I created for this experiment only has one full year and thus making a decomposition does not make much sense, but you are invited to try with your data and it should work fine and provide you with results similar to the image below:</p>
<div><a target="_blank" href="http://i0.wp.com/2.bp.blogspot.com/-zlAye5Z8EB4/VVYL_T9Rk-I/AAAAAAAAQ7M/e-bMrdYAmeU_ul6O8f9Yz9-5SAUSL-jJACKgB/s1600/Decompose.jpeg?ssl=1"><img src="http://i0.wp.com/2.bp.blogspot.com/-zlAye5Z8EB4/VVYL_T9Rk-I/AAAAAAAAQ7M/e-bMrdYAmeU_ul6O8f9Yz9-5SAUSL-jJACKgB/s400/Decompose.jpeg?resize=400%2C377&ssl=1" border="0" data-recalc-dims="1"></a></div>
<p><span><br></span><span>Download</span><br>Once again the time-series toolbox is available for free from my GitHub page at:<br><a target="_blank" href="https://github.com/fveronesi/TimeSeries_Toolbox/">https://github.com/fveronesi/TimeSeries_Toolbox/</a></p>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="http://www.r-bloggers.com/time-series-analysis-in-arcgis/"></div></div><div><a data-url="http://www.r-bloggers.com/time-series-analysis-in-arcgis/" data-counturl="http://www.r-bloggers.com/time-series-analysis-in-arcgis/" data-count="vertical" data-text="Time Series Analysis in ArcGIS" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://r-video-tutorial.blogspot.com/2016/07/time-series-analysis-in-arcgis.html"> R tutorial for Spatial Statistics</a></strong>.</div>
<hr>
<a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="http://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="http://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="http://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="http://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="http://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="http://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="http://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="http://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="http://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="http://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="http://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="http://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="http://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="http://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="http://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="http://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/Sz-jnDZuaxg"></a></div>