<p>I've recently released the new package <a href="https://github.com/ropensci/ccafs">ccafs</a>, which provides access
to data from Climate Change, Agriculture and Food Security
(CCAFS; <a href="http://ccafs-climate.org/">http://ccafs-climate.org/</a>) General Circulation Models (GCM) data.
GCM's are a particular type of climate model, used for weather forecasting,
and climate change forecasting - read more at
<a href="https://en.wikipedia.org/wiki/General_circulation_model">https://en.wikipedia.org/wiki/General_circulation_model</a>.</p>
<p><code>ccafs</code> falls in the data client camp - its focus is on getting users
data - many <a href="https://ropensci.org/packages/#data_access">rOpenSci packages</a>
fall into this area. These kinds of packages are important so that
scientists don't have to recreate the wheel themselves every time, but
instead use one client that everyone else uses.</p>
<p>CCAFS GCM data files are <code>.zip</code> files with a bunch of files inside. The
individual files are in ARC ASCII format (<a href="https://en.wikipedia.org/wiki/Esri_grid#ASCII">https://en.wikipedia.org/wiki/Esri_grid#ASCII</a>) -
a plain text data format, but still require painful manipulation/wrangling to
get into an easily consumable format. The files have a <code>.asc</code> file extension.</p>
<p>For each <code>.asc</code> file, the first 6 lines of each file indicate the reference of
the grid (number of columns and rows, corner coordinates, cellsize, and missing
data value), followed by the actual data values, delimited with single
space characters.</p>
<p>There's a related binary format - but its proprietary, so nevermind.</p>
<p>The workflow with <code>ccafs</code> for most users will likely be as follows:</p>
<ul><li>Search for data they want: <code>cc_search()</code></li>
<li>Fetch/download data: <code>cc_data_fetch()</code></li>
<li>Reaad data: <code>cc_data_read()</code></li>
</ul><p>I'll dive into more details below.</p>
<h2>Installation</h2>
<p>First, install the package.</p>
<div><pre><code><span></span>install.packages<span>(</span><span>&quot;ccafs&quot;</span><span>)</span>
</code></pre></div>
<p>Then load <code>ccafs</code></p>
<div><pre><code><span></span><span>library</span><span>(</span><span>&quot;ccafs&quot;</span><span>)</span>
</code></pre></div>
<h2>Search for data</h2>
<p>Searching CCAF's data holdings is not as easy as it could be as they don't
provide any programmatic way to do so. However, we provide a way to search
using their web interface from R.</p>
<p>You can search by the numbers representing each possible value for
each parameter. See the <code>?'ccafs-search'</code> for help on what the numbers
refer to.</p>
<div><pre><code><span></span><span>(</span>result1 <span>&lt;-</span> cc_search<span>(</span>file_set <span>=</span> <span>4</span><span>,</span> scenario <span>=</span> <span>6</span><span>,</span> model <span>=</span> <span>2</span><span>,</span> extent <span>=</span> <span>&quot;global&quot;</span><span>,</span>
  format <span>=</span> <span>&quot;ascii&quot;</span><span>,</span> period <span>=</span> <span>5</span><span>,</span> variable <span>=</span> <span>2</span><span>,</span> resolution <span>=</span> <span>3</span><span>))</span>
<span>#> [1] &quot;http://gisweb.ciat.cgiar.org/ccafs_climate/files/data/ipcc_4ar_ciat/sres_b1/2040s/bccr_bcm2_0/5min/bccr_bcm2_0_sres_b1_2040s_prec_5min_no_tile_asc.zip&quot;</span>
</code></pre></div>
<p>Alternatively, you can use the helper list <code>cc_params</code> where you can reference
options by name; the downside is that this leads to very verbose code.</p>
<div><pre><code><span></span><span>(</span>result2 <span>&lt;-</span> cc_search<span>(</span>file_set <span>=</span> cc_params<span>$</span>file_set<span>$</span><span>`Delta method IPCC AR4`</span><span>,</span>
                  scenario <span>=</span> cc_params<span>$</span>scenario<span>$</span><span>`SRES B1`</span><span>,</span>
                  model <span>=</span> cc_params<span>$</span>model<span>$</span>bccr_bcm2_0<span>,</span>
                  extent <span>=</span> cc_params<span>$</span>extent<span>$</span>global<span>,</span>
                  format <span>=</span> cc_params<span>$</span><span>format</span><span>$</span>ascii<span>,</span>
                  period <span>=</span> cc_params<span>$</span>period<span>$</span><span>`2040s`</span><span>,</span>
                  variable <span>=</span> cc_params<span>$</span>variable<span>$</span>Precipitation<span>,</span>
                  resolution <span>=</span> cc_params<span>$</span>resolution<span>$</span><span>`5 minutes`</span><span>))</span>
<span>#> [1] &quot;http://gisweb.ciat.cgiar.org/ccafs_climate/files/data/ipcc_4ar_ciat/sres_b1/2040s/bccr_bcm2_0/5min/bccr_bcm2_0_sres_b1_2040s_prec_5min_no_tile_asc.zip&quot;</span>
</code></pre></div>
<p>If you already know what you want in terms of file paths, you can query
Amazon S3 directly with <code>cc_list_keys()</code> (the data file come from Amazon S3):</p>
<div><pre><code><span></span>cc_list_keys<span>(</span>max <span>=</span> <span>3</span><span>)</span>
<span>#> # A tibble: 3 × 5</span>
<span>#> Key LastModified</span>
<span>#> &lt;chr> &lt;chr></span>
<span>#> 1 ccafs/ 2014-02-28T15:15:45.000Z</span>
<span>#> 2 ccafs/2014-05-24-01-19-33-3A0DFF1F86F3E7F7.txt 2014-07-01T02:15:51.000Z</span>
<span>#> 3 ccafs/amzn.csv 2014-02-28T15:21:32.000Z</span>
<span>#> # ... with 3 more variables: ETag &lt;chr>, Size &lt;chr>, StorageClass &lt;chr></span>
</code></pre></div>
<p>When using <code>cc_list_keys()</code>, you'll get not just <code>.zip</code> files that can be
downloaded, but also directories. So beware that if you're going after grabbing
&quot;keys&quot; for files that can be downloaded, you're looking for <code>.zip</code> files.</p>
<h2>Fetch and read data</h2>
<p>Once you get links from <code>cc_search()</code> or &quot;keys&quot; from <code>cc_list_keys()</code>, you
can pass either to <code>cc_data_fetch()</code> - which normalizes the input - so it
doesn't matter whether you pass in e.g.,</p>
<p><code>http://gisweb.ciat.cgiar.org/ccafs_climate/files/data/ipcc_4ar_ciat/</code>
<code>sres_b1/2040s/bccr_bcm2_0/5min/bccr_bcm2_0_sres_b1_2040s_prec_5min_no_tile_asc.zip</code></p>
<p>or</p>
<p><code>ccafs_climate/files/data/ipcc_4ar_ciat/sres_b1/2040s/bccr_bcm2_0/5min/</code>
<code>bccr_bcm2_0_sres_b1_2040s_prec_5min_no_tile_asc.zip</code></p>
<p>Let's download data with <code>cc_data_fetch()</code> using the result we got above
using <code>cc_search()</code>:</p>
<div><pre><code><span></span>xx <span>&lt;-</span> cc_data_fetch<span>(</span>result2<span>)</span>
</code></pre></div>
<p>Then we can read data with <code>cc_data_read()</code>:</p>
<div><pre><code><span></span><span>(</span>dat <span>&lt;-</span> cc_data_read<span>(</span>xx<span>))</span>
<span>#> class : RasterStack</span>
<span>#> dimensions : 1800, 4320, 7776000, 12 (nrow, ncol, ncell, nlayers)</span>
<span>#> resolution : 0.08333333, 0.08333333 (x, y)</span>
<span>#> extent : -180, 180, -60, 90 (xmin, xmax, ymin, ymax)</span>
<span>#> coord. ref. : NA</span>
<span>#> names : prec_1, prec_10, prec_11, prec_12, prec_2, prec_3, prec_4, prec_5, prec_6, prec_7, prec_8, prec_9</span>
<span>#> min values : -2147483648, -2147483648, -2147483648, -2147483648, -2147483648, -2147483648, -2147483648, -2147483648, -2147483648, -2147483648, -2147483648, -2147483648</span>
<span>#> max values : 2147483647, 2147483647, 2147483647, 2147483647, 2147483647, 2147483647, 2147483647, 2147483647, 2147483647, 2147483647, 2147483647, 2147483647</span>
</code></pre></div>
<p>Which gives a <code>raster</code> class object, which you are likely familiar with - which
opens up all the tools that deal with <code>raster</code> class objects, yay!</p>
<p>You can easily plot the data with the <code>plot</code> method from the <code>raster</code> package.</p>
<div><pre><code><span></span><span>library</span><span>(</span><span>&quot;raster&quot;</span><span>)</span>
plot<span>(</span>dat<span>)</span>
</code></pre></div>
<p></p>
<h2>Caching</h2>
<p>For a better user experience, we cache files for you. That means
when we download data, we put the files in a known location. When a
user tries to download the same data again, we look to see if it's already
been downloaded, and use the cached version - if we don't have it
already, we download it.</p>
<p>Of course, CCAFS may change their files, so you may not want the cached
version, but the new version from them. We provide tools to inspect your
cached files, and delete them.</p>
<p>List your cached files:</p>
<div><pre><code><span></span>cc_cache_list<span>()</span>
<span>#> [1] &quot;/Users/sacmac/Library/Caches/ccafs/bcc_csm1_1_m_rcp2_6_2030s_bio_10min_r1i1p1_no_tile_asc&quot;</span>
<span>#> [2] &quot;/Users/sacmac/Library/Caches/ccafs/bcc_csm1_1_m_rcp2_6_2030s_bio_10min_r1i1p1_no_tile_asc.zip&quot;</span>
<span>#> [3] &quot;/Users/sacmac/Library/Caches/ccafs/bcc_csm1_1_m_rcp2_6_2030s_bio_10min_r1i1p1_no_tile_asc/bio_1.asc&quot;</span>
<span>#> [4] &quot;/Users/sacmac/Library/Caches/ccafs/bcc_csm1_1_m_rcp2_6_2030s_bio_10min_r1i1p1_no_tile_asc/bio_10.asc&quot;</span>
<span>#> [5] &quot;/Users/sacmac/Library/Caches/ccafs/bcc_csm1_1_m_rcp2_6_2030s_bio_10min_r1i1p1_no_tile_asc/bio_11.asc&quot;</span>
<span>#> [6] &quot;/Users/sacmac/Library/Caches/ccafs/bcc_csm1_1_m_rcp2_6_2030s_bio_10min_r1i1p1_no_tile_asc/bio_12.asc&quot;</span>
<span>#> [7] &quot;/Users/sacmac/Library/Caches/ccafs/bcc_csm1_1_m_rcp2_6_2030s_bio_10min_r1i1p1_no_tile_asc/bio_13.asc&quot;</span>
<span>#> [8] &quot;/Users/sacmac/Library/Caches/ccafs/bcc_csm1_1_m_rcp2_6_2030s_bio_10min_r1i1p1_no_tile_asc/bio_14.asc&quot;</span>
<span>#> [9] &quot;/Users/sacmac/Library/Caches/ccafs/bcc_csm1_1_m_rcp2_6_2030s_bio_10min_r1i1p1_no_tile_asc/bio_15.asc&quot;</span>
<span>#> [10] &quot;/Users/sacmac/Library/Caches/ccafs/bcc_csm1_1_m_rcp2_6_2030s_bio_10min_r1i1p1_no_tile_asc/bio_16.asc&quot;</span>
<span>...</span>
</code></pre></div>
<p>Get details on all files or a specific file:</p>
<div><pre><code><span></span><span># cc_cache_details() # details for all files</span>
cc_cache_details<span>(</span>cc_cache_list<span>()[</span><span>1</span><span>])</span>
<span>#> &lt;ccafs cached files></span>
<span>#> directory: /Users/sacmac/Library/Caches/ccafs</span>
<span>#></span>
<span>#> file: /bcc_csm1_1_m_rcp2_6_2030s_bio_10min_r1i1p1_no_tile_asc</span>
<span>#> size: 0.001 mb</span>
</code></pre></div>
<p>Be careful with <code>cc_cache_delete_all()</code> as you will delete all your cached
files.</p>
<h2>ccafs software review</h2>
<p>I want to touch briefly on the software review for this package. The reviews
for <code>ccafs</code> were great, and I think the package was greatly improved via the
review process.</p>
<p><a href="https://github.com/mikoontz">Michael Koontz</a> and <a href="https://github.com/manuramon">Manuel Ramon</a>
did reviews for <code>ccafs</code>.</p>
<p>One thing in particular that improved about <code>ccafs</code> was the user interface -
that is, the programmatic interface. One feature about the interface was
adding the <code>cc_search()</code> function. When I started developing <code>ccafs</code>, I didn't
see a way to programmatically search CCAFS data - other than the Amazon S3
data, which isn't really search, but more like listing files in a directory -
so I just left it at that. During the reviews, reviewers wanted a clear workflow
for potential users - the package as submitted for review didn't really have a
clear workflow; it was</p>
<ul><li>Know what you want already (<code>cc_list_keys</code> helped get real paths at least)</li>
<li>Download data</li>
<li>Read data</li>
</ul><p>Which is not ideal. There should be a discovery portion to the workflow. So
I decided to dig into possibly querying the CCAFS web portal itself. That panned
out, and the workflow we have now is much better:</p>
<ul><li>Search for data with all the same variables you would on the CCAFS website</li>
<li>Download data</li>
<li>Read data</li>
</ul><p>This is much better!</p>
<p>As always, reviews improved the documentation a lot by pointing out areas that
could use improvement - which all users will greatly benefit from.</p>
<p>A new vignette (<a href="https://cran.rstudio.com/web/packages/ccafs/vignettes/amazon_s3_keys.html">https://cran.rstudio.com/web/packages/ccafs/vignettes/amazon_s3_keys.html</a>)
was added in the review process to explain how to get a &quot;key&quot;, a URL for CCAFS data.</p>
<h2>To Do and Feedback</h2>
<p>There's probably lots of improvements that can be made - I'm looking forward
to getting feedback from users on any bugs or feature requests. One immediate
thing is to <a href="https://github.com/ropensci/ccafs/issues/22">make the cache details more compact</a>.</p>