<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/ccafs-client-for-ccafs-general-circulation-models-data/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://ropensci.org/blog/technotes/2017/03/01/ccafs-release"> rOpenSci Blog - R</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>I've recently released the new package <a target="_blank" rel="nofollow" href="https://github.com/ropensci/ccafs">ccafs</a>, which provides access<br>
to data from Climate Change, Agriculture and Food Security<br>
(CCAFS; <a target="_blank" rel="nofollow" href="http://ccafs-climate.org/">http://ccafs-climate.org/</a>) General Circulation Models (GCM) data.<br>
GCM's are a particular type of climate model, used for weather forecasting,<br>
and climate change forecasting – read more at<br>
<a target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/General_circulation_model">https://en.wikipedia.org/wiki/General_circulation_model</a>.</p>
<p><code>ccafs</code> falls in the data client camp – its focus is on getting users<br>
data – many <a target="_blank" rel="nofollow" href="https://ropensci.org/packages/#data_access">rOpenSci packages</a><br>
fall into this area. These kinds of packages are important so that<br>
scientists don't have to recreate the wheel themselves every time, but<br>
instead use one client that everyone else uses.</p>
<p>CCAFS GCM data files are <code>.zip</code> files with a bunch of files inside. The<br>
individual files are in ARC ASCII format (<a target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/Esri_grid#ASCII">https://en.wikipedia.org/wiki/Esri_grid#ASCII</a>) –<br>
a plain text data format, but still require painful manipulation/wrangling to<br>
get into an easily consumable format. The files have a <code>.asc</code> file extension.</p>
<p>For each <code>.asc</code> file, the first 6 lines of each file indicate the reference of<br>
the grid (number of columns and rows, corner coordinates, cellsize, and missing<br>
data value), followed by the actual data values, delimited with single<br>
space characters.</p>
<p>There's a related binary format – but its proprietary, so nevermind.</p>
<p>The workflow with <code>ccafs</code> for most users will likely be as follows:</p>
<ul>
<li>Search for data they want: <code>cc_search()</code></li>
<li>Fetch/download data: <code>cc_data_fetch()</code></li>
<li>Reaad data: <code>cc_data_read()</code></li>
</ul>
<p>I'll dive into more details below.</p>
<h2>Installation</h2>
<p>First, install the package.</p>
<div>
<pre><code data-lang="r"><span></span>install.packages<span>(</span><span>&quot;ccafs&quot;</span><span>)</span>
</code></pre>
</div>
<p>Then load <code>ccafs</code></p>
<div>
<pre><code data-lang="r"><span></span><span>library</span><span>(</span><span>&quot;ccafs&quot;</span><span>)</span>
</code></pre>
</div>
<h2>Search for data</h2>
<p>Searching CCAF's data holdings is not as easy as it could be as they don't<br>
provide any programmatic way to do so. However, we provide a way to search<br>
using their web interface from R.</p>
<p>You can search by the numbers representing each possible value for<br>
each parameter. See the <code>?'ccafs-search'</code> for help on what the numbers<br>
refer to.</p>
<div>
<pre><code data-lang="r"><span></span><span>(</span>result1 <span>&lt;-</span> cc_search<span>(</span>file_set <span>=</span> <span>4</span><span>,</span> scenario <span>=</span> <span>6</span><span>,</span> model <span>=</span> <span>2</span><span>,</span> extent <span>=</span> <span>&quot;global&quot;</span><span>,</span>
  format <span>=</span> <span>&quot;ascii&quot;</span><span>,</span> period <span>=</span> <span>5</span><span>,</span> variable <span>=</span> <span>2</span><span>,</span> resolution <span>=</span> <span>3</span><span>))</span>
<span>#> [1] &quot;http://gisweb.ciat.cgiar.org/ccafs_climate/files/data/ipcc_4ar_ciat/sres_b1/2040s/bccr_bcm2_0/5min/bccr_bcm2_0_sres_b1_2040s_prec_5min_no_tile_asc.zip&quot;</span>
</code></pre>
</div>
<p>Alternatively, you can use the helper list <code>cc_params</code> where you can reference<br>
options by name; the downside is that this leads to very verbose code.</p>
<div>
<pre><code data-lang="r"><span></span><span>(</span>result2 <span>&lt;-</span> cc_search<span>(</span>file_set <span>=</span> cc_params<span>$</span>file_set<span>$</span><span>`Delta method IPCC AR4`</span><span>,</span>
                  scenario <span>=</span> cc_params<span>$</span>scenario<span>$</span><span>`SRES B1`</span><span>,</span>
                  model <span>=</span> cc_params<span>$</span>model<span>$</span>bccr_bcm2_0<span>,</span>
                  extent <span>=</span> cc_params<span>$</span>extent<span>$</span>global<span>,</span>
                  format <span>=</span> cc_params<span>$</span><span>format</span><span>$</span>ascii<span>,</span>
                  period <span>=</span> cc_params<span>$</span>period<span>$</span><span>`2040s`</span><span>,</span>
                  variable <span>=</span> cc_params<span>$</span>variable<span>$</span>Precipitation<span>,</span>
                  resolution <span>=</span> cc_params<span>$</span>resolution<span>$</span><span>`5 minutes`</span><span>))</span>
<span>#> [1] &quot;http://gisweb.ciat.cgiar.org/ccafs_climate/files/data/ipcc_4ar_ciat/sres_b1/2040s/bccr_bcm2_0/5min/bccr_bcm2_0_sres_b1_2040s_prec_5min_no_tile_asc.zip&quot;</span>
</code></pre>
</div>
<p>If you already know what you want in terms of file paths, you can query<br>
Amazon S3 directly with <code>cc_list_keys()</code> (the data file come from Amazon S3):</p>
<div>
<pre><code data-lang="r"><span></span>cc_list_keys<span>(</span>max <span>=</span> <span>3</span><span>)</span>
<span>#> # A tibble: 3 × 5</span>
<span>#> Key LastModified</span>
<span>#> &lt;chr> &lt;chr></span>
<span>#> 1 ccafs/ 2014-02-28T15:15:45.000Z</span>
<span>#> 2 ccafs/2014-05-24-01-19-33-3A0DFF1F86F3E7F7.txt 2014-07-01T02:15:51.000Z</span>
<span>#> 3 ccafs/amzn.csv 2014-02-28T15:21:32.000Z</span>
<span>#> # ... with 3 more variables: ETag &lt;chr>, Size &lt;chr>, StorageClass &lt;chr></span>
</code></pre>
</div>
<p>When using <code>cc_list_keys()</code>, you'll get not just <code>.zip</code> files that can be<br>
downloaded, but also directories. So beware that if you're going after grabbing<br>
&quot;keys&quot; for files that can be downloaded, you're looking for <code>.zip</code> files.</p>
<h2>Fetch and read data</h2>
<p>Once you get links from <code>cc_search()</code> or &quot;keys&quot; from <code>cc_list_keys()</code>, you<br>
can pass either to <code>cc_data_fetch()</code> – which normalizes the input – so it<br>
doesn't matter whether you pass in e.g.,</p>
<p><code>http://gisweb.ciat.cgiar.org/ccafs_climate/files/data/ipcc_4ar_ciat/</code><br>
<code>sres_b1/2040s/bccr_bcm2_0/5min/bccr_bcm2_0_sres_b1_2040s_prec_5min_no_tile_asc.zip</code></p>
<p>or</p>
<p><code>ccafs_climate/files/data/ipcc_4ar_ciat/sres_b1/2040s/bccr_bcm2_0/5min/</code><br>
<code>bccr_bcm2_0_sres_b1_2040s_prec_5min_no_tile_asc.zip</code></p>
<p>Let's download data with <code>cc_data_fetch()</code> using the result we got above<br>
using <code>cc_search()</code>:</p>
<div>
<pre><code data-lang="r"><span></span>xx <span>&lt;-</span> cc_data_fetch<span>(</span>result2<span>)</span>
</code></pre>
</div>
<p>Then we can read data with <code>cc_data_read()</code>:</p>
<div>
<pre><code data-lang="r"><span></span><span>(</span>dat <span>&lt;-</span> cc_data_read<span>(</span>xx<span>))</span>
<span>#> class : RasterStack</span>
<span>#> dimensions : 1800, 4320, 7776000, 12 (nrow, ncol, ncell, nlayers)</span>
<span>#> resolution : 0.08333333, 0.08333333 (x, y)</span>
<span>#> extent : -180, 180, -60, 90 (xmin, xmax, ymin, ymax)</span>
<span>#> coord. ref. : NA</span>
<span>#> names : prec_1, prec_10, prec_11, prec_12, prec_2, prec_3, prec_4, prec_5, prec_6, prec_7, prec_8, prec_9</span>
<span>#> min values : -2147483648, -2147483648, -2147483648, -2147483648, -2147483648, -2147483648, -2147483648, -2147483648, -2147483648, -2147483648, -2147483648, -2147483648</span>
<span>#> max values : 2147483647, 2147483647, 2147483647, 2147483647, 2147483647, 2147483647, 2147483647, 2147483647, 2147483647, 2147483647, 2147483647, 2147483647</span>
</code></pre>
</div>
<p>Which gives a <code>raster</code> class object, which you are likely familiar with – which<br>
opens up all the tools that deal with <code>raster</code> class objects, yay!</p>
<p>You can easily plot the data with the <code>plot</code> method from the <code>raster</code> package.</p>
<div>
<pre><code data-lang="r"><span></span><span>library</span><span>(</span><span>&quot;raster&quot;</span><span>)</span>
plot<span>(</span>dat<span>)</span>
</code></pre>
</div>
<p><img alt="plot" src="https://i2.wp.com/ropensci.org/assets/blog-images/2017-03-01-ccafs-release/unnamed-chunk-9-1.png?w=456" data-recalc-dims="1"></p>
<h2>Caching</h2>
<p>For a better user experience, we cache files for you. That means<br>
when we download data, we put the files in a known location. When a<br>
user tries to download the same data again, we look to see if it's already<br>
been downloaded, and use the cached version – if we don't have it<br>
already, we download it.</p>
<p>Of course, CCAFS may change their files, so you may not want the cached<br>
version, but the new version from them. We provide tools to inspect your<br>
cached files, and delete them.</p>
<p>List your cached files:</p>
<div>
<pre><code data-lang="r"><span></span>cc_cache_list<span>()</span>
<span>#> [1] &quot;/Users/sacmac/Library/Caches/ccafs/bcc_csm1_1_m_rcp2_6_2030s_bio_10min_r1i1p1_no_tile_asc&quot;</span>
<span>#> [2] &quot;/Users/sacmac/Library/Caches/ccafs/bcc_csm1_1_m_rcp2_6_2030s_bio_10min_r1i1p1_no_tile_asc.zip&quot;</span>
<span>#> [3] &quot;/Users/sacmac/Library/Caches/ccafs/bcc_csm1_1_m_rcp2_6_2030s_bio_10min_r1i1p1_no_tile_asc/bio_1.asc&quot;</span>
<span>#> [4] &quot;/Users/sacmac/Library/Caches/ccafs/bcc_csm1_1_m_rcp2_6_2030s_bio_10min_r1i1p1_no_tile_asc/bio_10.asc&quot;</span>
<span>#> [5] &quot;/Users/sacmac/Library/Caches/ccafs/bcc_csm1_1_m_rcp2_6_2030s_bio_10min_r1i1p1_no_tile_asc/bio_11.asc&quot;</span>
<span>#> [6] &quot;/Users/sacmac/Library/Caches/ccafs/bcc_csm1_1_m_rcp2_6_2030s_bio_10min_r1i1p1_no_tile_asc/bio_12.asc&quot;</span>
<span>#> [7] &quot;/Users/sacmac/Library/Caches/ccafs/bcc_csm1_1_m_rcp2_6_2030s_bio_10min_r1i1p1_no_tile_asc/bio_13.asc&quot;</span>
<span>#> [8] &quot;/Users/sacmac/Library/Caches/ccafs/bcc_csm1_1_m_rcp2_6_2030s_bio_10min_r1i1p1_no_tile_asc/bio_14.asc&quot;</span>
<span>#> [9] &quot;/Users/sacmac/Library/Caches/ccafs/bcc_csm1_1_m_rcp2_6_2030s_bio_10min_r1i1p1_no_tile_asc/bio_15.asc&quot;</span>
<span>#> [10] &quot;/Users/sacmac/Library/Caches/ccafs/bcc_csm1_1_m_rcp2_6_2030s_bio_10min_r1i1p1_no_tile_asc/bio_16.asc&quot;</span>
<span>...</span>
</code></pre>
</div>
<p>Get details on all files or a specific file:</p>
<div>
<pre><code data-lang="r"><span></span><span># cc_cache_details() # details for all files</span>
cc_cache_details<span>(</span>cc_cache_list<span>()[</span><span>1</span><span>])</span>
<span>#> &lt;ccafs cached files></span>
<span>#> directory: /Users/sacmac/Library/Caches/ccafs</span>
<span>#></span>
<span>#> file: /bcc_csm1_1_m_rcp2_6_2030s_bio_10min_r1i1p1_no_tile_asc</span>
<span>#> size: 0.001 mb</span>
</code></pre>
</div>
<p>Be careful with <code>cc_cache_delete_all()</code> as you will delete all your cached<br>
files.</p>
<h2>ccafs software review</h2>
<p>I want to touch briefly on the software review for this package. The reviews<br>
for <code>ccafs</code> were great, and I think the package was greatly improved via the<br>
review process.</p>
<p><a target="_blank" rel="nofollow" href="https://github.com/mikoontz">Michael Koontz</a> and <a target="_blank" rel="nofollow" href="https://github.com/manuramon">Manuel Ramon</a><br>
did reviews for <code>ccafs</code>.</p>
<p>One thing in particular that improved about <code>ccafs</code> was the user interface –<br>
that is, the programmatic interface. One feature about the interface was<br>
adding the <code>cc_search()</code> function. When I started developing <code>ccafs</code>, I didn't<br>
see a way to programmatically search CCAFS data – other than the Amazon S3<br>
data, which isn't really search, but more like listing files in a directory –<br>
so I just left it at that. During the reviews, reviewers wanted a clear workflow<br>
for potential users – the package as submitted for review didn't really have a<br>
clear workflow; it was</p>
<ul>
<li>Know what you want already (<code>cc_list_keys</code> helped get real paths at least)</li>
<li>Download data</li>
<li>Read data</li>
</ul>
<p>Which is not ideal. There should be a discovery portion to the workflow. So<br>
I decided to dig into possibly querying the CCAFS web portal itself. That panned<br>
out, and the workflow we have now is much better:</p>
<ul>
<li>Search for data with all the same variables you would on the CCAFS website</li>
<li>Download data</li>
<li>Read data</li>
</ul>
<p>This is much better!</p>
<p>As always, reviews improved the documentation a lot by pointing out areas that<br>
could use improvement – which all users will greatly benefit from.</p>
<p>A new vignette (<a target="_blank" rel="nofollow" href="https://cran.rstudio.com/web/packages/ccafs/vignettes/amazon_s3_keys.html">https://cran.rstudio.com/web/packages/ccafs/vignettes/amazon_s3_keys.html</a>)<br>
was added in the review process to explain how to get a &quot;key&quot;, a URL for CCAFS data.</p>
<h2>To Do and Feedback</h2>
<p>There's probably lots of improvements that can be made – I'm looking forward<br>
to getting feedback from users on any bugs or feature requests. One immediate<br>
thing is to <a target="_blank" rel="nofollow" href="https://github.com/ropensci/ccafs/issues/22">make the cache details more compact</a>.</p>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://ropensci.org/blog/technotes/2017/03/01/ccafs-release"> rOpenSci Blog - R</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/rTVrSbyQryI"></a></div>