<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/global-temp-geo-spatial-records-over-time/"></div></div><div><a data-url="https://www.r-bloggers.com/global-temp-geo-spatial-records-over-time/" data-counturl="https://www.r-bloggers.com/global-temp-geo-spatial-records-over-time/" data-count="horizontal" data-text="Global Temp: Geo-Spatial Records over Time" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://data-steve.github.io/glob-temp-data-pt1/"> data_steve</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><img src="https://i1.wp.com/data-steve.github.io/images/animated.gif?w=456" data-recalc-dims="1"></p>
<p>Over the past couple months there have been datavizs showing global average temp trends. They have gotten a lot of media attention, so I got to see examples fairly regularly. Several of them (e.g., <a target="_blank" href="http://www.fromthebottomoftheheap.net/2016/03/25/additive-modeling-global-temperature-series-revisited/">1</a> and <a target="_blank" href="https://rud.is/projects/hadcrut4/">2</a>) used the same HadCRUT global temperature anamolies data, analyzing it with different techniques or technologies to highlight specific statistics, namely the mean. I liked <a target="_blank" href="http://data-steve.github.io/glob-temp-data-pt1/twitter.com/ucfagls">@ucfagls’s</a> simulation work on the averaged data and <a target="_blank" href="http://data-steve.github.io/glob-temp-data-pt1/twitter.com/hrbrmstr">@hrbrmstr’s</a> incorporation of the error model’s confidence intervals over time.</p>
<p>Most of these posts above emphasize the temporal nature of the data, as it is often the processed averaged values that they are using for data; these averages aggretate what observations are present over the globe at each time interval. After staring at these a while, I began to wonder at the decades-long downturns in the global averages that begin around the 1900s and again 1940s. These downturns left me stumped to reconcile what I knew generally of industrial economic development practices going on around the world at those times.</p>
<p>I decided to take a peak at the rawest data I could easily access from public records. Some fun data-play ensued. From that, I’ll be sharing a series of posts emphasing the geo-spatial nature of the temperature records as the collection patterns evolved over time.</p>
<p>The gif above is the output of the first steps I took to process the raw HadCRUT data.</p>
<h3>HadCRUT4 data</h3>
<p>In order to get the rawest data possible for the HadCRUT4 you have to go <a target="_blank" href="http://hadobs.metoffice.com/hadcrut4/">here to get the components datasets it’s synthesized from</a>. The <a target="_blank" href="http://hadobs.metoffice.com/hadsst3/">HadSST3</a> are the sea-surface data and <a target="_blank" href="http://hadobs.metoffice.com/crutem4/">CRUTEM4</a> are the land-surface data.</p>
<p>In this post I’ll work with the sea-surface data, HadSST3. Other posts I’ll work with land-surface and compiled data, as well.</p>
<p>Three steps will be needed to complete the animated gif above:</p>
<ul>
<li>get the text data and split it up by its time intervals</li>
<li>parse and clean the data</li>
<li>create a fun gif showing recordkeeping over time</li>
</ul>
<h4>Get and split up</h4>
<div>
<pre><code>pacman::p_load(tidyverse, curl)
temp &lt;- tempfile()
curl_download(&quot;http://hadobs.metoffice.com/hadsst3/data/HadSST.3.1.1.0/ascii/HadSST.3.1.1.0.unadjusted.zip&quot;,temp)
data &lt;- readLines(unz(temp, &quot;HadSST.3.1.1.0.unadjusted.txt&quot;))
unlink(temp)

data[1:2]
[1] &quot;       1    1850 36 Rows 72 Columns.  Missing =  -99.99 Version = HadSST.3.1.1.0       Field_status = f Realisation = -1&quot;
[2] &quot; -99.99  -99.99  -99.99  -99.99  -99.99  -99.99  -99.99  -99.99  -99.99  -99.99  -99.99  -99.99  -99.99
</code></pre>
</div>
<p>The pattern in the data is first line has all the meta data like month and year the data represents and then the next 36 contain the contents of the 36×72 numeric matrix of gridded temperature readings. So I use that pattern to create a regex to define the beginning of new year’s worth of data. And then just split the matrices up into a list of matrices.</p>
<div>
<pre><code># meta-data pattern defining when each new matrix begin
ii &lt;- grepl(&quot;\\d+\\s+\\d{4}\\s+\\d+\\s+Rows&quot;, data)
sum(ii)/12+1850  # should add up to 2016+

counter &lt;- 0
index &lt;- rep(NA,length(ii))

for (j in 1:length(data)){
  if(ii[j]){
    counter=counter+1
    index[j]=counter
  } else {
    index[j] = counter
  }
}

dat_list &lt;- split(data,index)
length(dat_list)

all(sapply(dat_list,length)==37) # correctly parsed
</code></pre>
</div>
<h4>Parse and Clean</h4>
<p>The main objective of this loop is to extract the year-month data from the meta data and to define the matrix. The matrices are very sparse, especially before 1900; so a simple <code>ifelse</code> will suffice to get at gridded locations of a measurement. Again the point is the locations, not whatever values are actually present in the data. (Feel free to use whatever parallelization tool you prefer here.)</p>
<p>Since this is monthly data over many years, I mainly want to give an overall progression over 150+ years. So I’ll be plotting the matrix for every 24 months, starting with the first Dec in the data, from 1850. I intend to use all the data in the next post; so we’ll have a chance to see all the data, but I didn’t want the gif to be too big.</p>
<div>
<pre><code>yrmn_step &lt;- seq(12,2000,by=24) # every other year
yrmn_step
dat_mat &lt;- lapply(dat_list[yrmn_step], function(x) {
  
  # first line has meta data
  meta &lt;- x[1]
  yr &lt;- regmatches(meta,regexec(&quot;\\d{4}&quot;,meta))[[1]]
  mo &lt;- regmatches(meta,regexec(&quot;\\d{1,2}&quot;,meta))[[1]][1]
  mo &lt;- ifelse(as.integer(mo)&lt;10,paste0(&quot;0&quot;,mo),mo)
  
  # remaining 36 have data
  mx &lt;- x[2:37]
  mx &lt;- sapply(mx, function(y) {
    m &lt;- as.numeric(strsplit(trimws(y),&quot;\\s+&quot;)[[1]])
    ifelse(m==-99.99, 0,1)  # replace missing with 0, non-missing with 1
  })

  m &lt;- matrix(mx,36,72)
  colnames(m) &lt;-  c(paste0(seq(175,5,by=-5),&quot;W&quot;),0,paste0(seq(5,180, by=5),&quot;E&quot;))
  rownames(m) &lt;- c(paste0(seq(85,5,by=-5),&quot;S&quot;),&quot;Eq&quot;,paste0(seq(5,90, by=5),&quot;N&quot;))

  ym &lt;- paste0(yr,&quot;/&quot;,mo)
  list(ym,m)
})
</code></pre>
</div>
<h2>Animate gif</h2>
<p>I would recommend sending the output of this task to its own folder to help with the animation step later.</p>
<div>
<pre><code>plots &lt;- file.path(getwd(),&quot;plots/had_plots/&quot;)
</code></pre>
</div>
<p>Since the data is one big numerical matrix, plotting in base R using <code>image</code> seemed the fastest and most similar to other graphs you see on the government websites. I’ve included a horizonal dashed line to keep the Equator evident.</p>
<div>
<pre><code># plot every other Dec snapshots
lapply(dat_mat, function(x){
  png(paste0(plots,substr(x[1],1,4),&quot;.png&quot;))
  image(x[2][[1]], col=c(&quot;white&quot;, &quot;blue&quot;), axes=FALSE
    , main=&quot;Geo-Locate Sea-Surface Temp Readings\nby Year/Month&quot;)
  abline(h=.5, lty=&quot;dashed&quot;)
  text(y=.98,x=.5,labels=x[1], col=&quot;red&quot;, cex=1.5)
  axis(1, at = c(0,0.25, 0.5, 0.75, 1),
       labels=c(&quot;180W&quot;, &quot;90W&quot;, &quot;0&quot;, &quot;90E&quot;, &quot;180E&quot;),
       srt=45,tick=FALSE)
  axis(2, at = c(0,0.5,1),
       labels=c(&quot;90S&quot;, &quot;Eq&quot;, &quot;90N&quot;)
       ,srt=45,tick=FALSE)
  dev.off()
})
</code></pre>
</div>
<p>Now to apply the animation. I suppose you could use the <code>animation</code> package, but I just used the command line tools. I’ll show you how. First change directories to where your new pngs are. You can download <code>ffmpeg</code> from <a target="_blank" href="https://ffmpeg.org/download.html">here</a> for whatever platform. I found the <code>system</code> tool could successfully call the ffmpeg function <code>convert</code> and get it to work. (On my Mac OS, system2 could handle the <code>pwd</code> call, but not <code>convert</code>.)</p>
<div>
<pre><code># install ffmpeg from https://ffmpeg.org/download.html
setwd(plots)
system(&quot;pwd&quot;)
system(&quot;convert -loop 1850 *.png animated.gif&quot;)
</code></pre>
</div>
<p>The <code>animated.gif</code> file should be in the same folder.</p>
<p>In the next post, I’ll return to the monthly data. Using all the data this time, I’ll do some summarizations and plotting of the geo-spatial variation over time.</p>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="https://www.r-bloggers.com/global-temp-geo-spatial-records-over-time/"></div></div><div><a data-url="https://www.r-bloggers.com/global-temp-geo-spatial-records-over-time/" data-counturl="https://www.r-bloggers.com/global-temp-geo-spatial-records-over-time/" data-count="vertical" data-text="Global Temp: Geo-Spatial Records over Time" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://data-steve.github.io/glob-temp-data-pt1/"> data_steve</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/HRSKtpI4b9M"></a></div>