<p>A new package <a href="https://cran.rstudio.com/web/packages/isdparser">isdparser</a> is
on CRAN. <code>isdparser</code> was in part liberated from <a href="https://github.com/ropensci/rnoaa">rnoaa</a>,
then improved. We'll use <code>isdparser</code> in <code>rnoaa</code> soon.</p>
<p><code>isdparser</code> does not download files for you from NOAA's ftp servers. The
package focuses on parsing the files, which are variable length ASCII strings
stored line by line, where each line has some mandatory data, and any amount
of optional data.</p>
<p>The data is great, and includes for example, wind speed and direction, temperature,
cloud data, sea level pressure, and more. Includes data from approximately 35,000
stations worldwide, though best coverage is in North America/Europe/Australia.
Data go all the way back to 1901, and are updated daily.</p>
<p>However, <a>the data</a> is not fun to parse,
warranting an packge to deal with the parsing.</p>
<h2>Installation</h2>
<div><pre><code>install.packages<span>(</span><span>&quot;isdparser&quot;</span><span>)</span>
</code></pre></div>
<p>If binaries aren't available, try from source:
<code>install.packages(&quot;isdparser&quot;, type = &quot;source&quot;)</code> or from GitHub:
<code>devtools::install_github(&quot;ropenscilabs/isdparser&quot;)</code></p>
<div><pre><code><span>library</span><span>(</span>isdparser<span>)</span>
<span>library</span><span>(</span>dplyr<span>)</span>
</code></pre></div>
<h2>Parse individual lines</h2>
<p>If you want to parse individual lines, use <code>isd_parse_line()</code></p>
<p>First, let's get a ISD file. There's a few that come with the package:</p>
<div><pre><code>path <span>&lt;-</span> <span>system.file</span><span>(</span><span>'extdata/024130-99999-2016.gz'</span><span>,</span> package <span>=</span> <span>&quot;isdparser&quot;</span><span>)</span>
</code></pre></div>
<p>Read in the file</p>
<div><pre><code>lns <span>&lt;-</span> <span>readLines</span><span>(</span>path<span>,</span> encoding <span>=</span> <span>&quot;latin1&quot;</span><span>)</span>
</code></pre></div>
<p>Parse a line</p>
<div><pre><code>isd_parse_line<span>(</span>lns<span>[</span><span>1</span><span>])</span>
<span>#> # A tibble: 1 × 42</span>
<span>#> total_chars usaf_station wban_station date time date_flag</span>
<span>#> &lt;dbl> &lt;chr> &lt;chr> &lt;date> &lt;chr> &lt;chr></span>
<span>#> 1 54 024130 99999 2016-01-01 0000 4</span>
<span>#> # ... with 36 more variables: latitude &lt;dbl>, longitude &lt;dbl>,</span>
<span>#> # type_code &lt;chr>, elevation &lt;dbl>, call_letter &lt;chr>, quality &lt;chr>,</span>
<span>#> # wind_direction &lt;dbl>, wind_direction_quality &lt;chr>, wind_code &lt;chr>,</span>
<span>#> # wind_speed &lt;dbl>, wind_speed_quality &lt;chr>, ceiling_height &lt;chr>,</span>
<span>#> # ceiling_height_quality &lt;chr>, ceiling_height_determination &lt;chr>,</span>
<span>#> # ceiling_height_cavok &lt;chr>, visibility_distance &lt;chr>,</span>
<span>#> # visibility_distance_quality &lt;chr>, visibility_code &lt;chr>,</span>
<span>#> # visibility_code_quality &lt;chr>, temperature &lt;dbl>,</span>
<span>#> # temperature_quality &lt;chr>, temperature_dewpoint &lt;dbl>,</span>
<span>#> # temperature_dewpoint_quality &lt;chr>, air_pressure &lt;dbl>,</span>
<span>#> # air_pressure_quality &lt;chr>,</span>
<span>#> # AW1_present_weather_observation_identifier &lt;chr>,</span>
<span>#> # AW1_automated_atmospheric_condition_code &lt;chr>,</span>
<span>#> # AW1_quality_automated_atmospheric_condition_code &lt;chr>,</span>
<span>#> # N03_original_observation &lt;chr>, N03_original_value_text &lt;chr>,</span>
<span>#> # N03_units_code &lt;chr>, N03_parameter_code &lt;chr>, REM_remarks &lt;chr>,</span>
<span>#> # REM_identifier &lt;chr>, REM_length_quantity &lt;chr>, REM_comment &lt;chr></span>
</code></pre></div>
<p>By default you get a tibble back, but you can ask for a list in return instead.</p>
<p>Parsing by line allows the user to decide how to apply parsing across lines,
whether it be <code>lapply</code> style, or for loop, etc.</p>
<h2>Parse entire files</h2>
<p>You can also parse entire ISD files.</p>
<div><pre><code>isd_parse<span>(</span>path<span>)</span>
<span>#> # A tibble: 2,601 × 42</span>
<span>#> total_chars usaf_station wban_station date time date_flag</span>
<span>#> &lt;dbl> &lt;chr> &lt;chr> &lt;date> &lt;chr> &lt;chr></span>
<span>#> 1 54 024130 99999 2016-01-01 0000 4</span>
<span>#> 2 54 024130 99999 2016-01-01 0100 4</span>
<span>#> 3 54 024130 99999 2016-01-01 0200 4</span>
<span>#> 4 54 024130 99999 2016-01-01 0300 4</span>
<span>#> 5 54 024130 99999 2016-01-01 0400 4</span>
<span>#> 6 39 024130 99999 2016-01-01 0500 4</span>
<span>#> 7 54 024130 99999 2016-01-01 0600 4</span>
<span>#> 8 39 024130 99999 2016-01-01 0700 4</span>
<span>#> 9 54 024130 99999 2016-01-01 0800 4</span>
<span>#> 10 54 024130 99999 2016-01-01 0900 4</span>
<span>#> # ... with 2,591 more rows, and 36 more variables: latitude &lt;dbl>,</span>
<span>#> # longitude &lt;dbl>, type_code &lt;chr>, elevation &lt;dbl>, call_letter &lt;chr>,</span>
<span>#> # quality &lt;chr>, wind_direction &lt;dbl>, wind_direction_quality &lt;chr>,</span>
<span>#> # wind_code &lt;chr>, wind_speed &lt;dbl>, wind_speed_quality &lt;chr>,</span>
<span>#> # ceiling_height &lt;chr>, ceiling_height_quality &lt;chr>,</span>
<span>#> # ceiling_height_determination &lt;chr>, ceiling_height_cavok &lt;chr>,</span>
<span>#> # visibility_distance &lt;chr>, visibility_distance_quality &lt;chr>,</span>
<span>#> # visibility_code &lt;chr>, visibility_code_quality &lt;chr>,</span>
<span>#> # temperature &lt;dbl>, temperature_quality &lt;chr>,</span>
<span>#> # temperature_dewpoint &lt;dbl>, temperature_dewpoint_quality &lt;chr>,</span>
<span>#> # air_pressure &lt;dbl>, air_pressure_quality &lt;chr>,</span>
<span>#> # AW1_present_weather_observation_identifier &lt;chr>,</span>
<span>#> # AW1_automated_atmospheric_condition_code &lt;chr>,</span>
<span>#> # AW1_quality_automated_atmospheric_condition_code &lt;chr>,</span>
<span>#> # N03_original_observation &lt;chr>, N03_original_value_text &lt;chr>,</span>
<span>#> # N03_units_code &lt;chr>, N03_parameter_code &lt;chr>, REM_remarks &lt;chr>,</span>
<span>#> # REM_identifier &lt;chr>, REM_length_quantity &lt;chr>, REM_comment &lt;chr></span>
</code></pre></div>
<p>Optionally, you can print progress:</p>
<div><pre><code>isd_parse<span>(</span>path<span>,</span> progress <span>=</span> <span>TRUE</span><span>)</span>
<span>#> # A tibble: 2,601 × 42</span>
<span>#> total_chars usaf_station wban_station date time date_flag</span>
<span>#> &lt;dbl> &lt;chr> &lt;chr> &lt;date> &lt;chr> &lt;chr></span>
<span>#> 1 54 024130 99999 2016-01-01 0000 4</span>
<span>#> 2 54 024130 99999 2016-01-01 0100 4</span>
<span>#> 3 54 024130 99999 2016-01-01 0200 4</span>
<span>#> 4 54 024130 99999 2016-01-01 0300 4</span>
<span>#> 5 54 024130 99999 2016-01-01 0400 4</span>
<span>#> 6 39 024130 99999 2016-01-01 0500 4</span>
<span>#> 7 54 024130 99999 2016-01-01 0600 4</span>
<span>#> 8 39 024130 99999 2016-01-01 0700 4</span>
<span>#> 9 54 024130 99999 2016-01-01 0800 4</span>
<span>#> 10 54 024130 99999 2016-01-01 0900 4</span>
<span>#> # ... with 2,591 more rows, and 36 more variables: latitude &lt;dbl>,</span>
<span>#> # longitude &lt;dbl>, type_code &lt;chr>, elevation &lt;dbl>, call_letter &lt;chr>,</span>
<span>#> # quality &lt;chr>, wind_direction &lt;dbl>, wind_direction_quality &lt;chr>,</span>
<span>#> # wind_code &lt;chr>, wind_speed &lt;dbl>, wind_speed_quality &lt;chr>,</span>
<span>#> # ceiling_height &lt;chr>, ceiling_height_quality &lt;chr>,</span>
<span>#> # ceiling_height_determination &lt;chr>, ceiling_height_cavok &lt;chr>,</span>
<span>#> # visibility_distance &lt;chr>, visibility_distance_quality &lt;chr>,</span>
<span>#> # visibility_code &lt;chr>, visibility_code_quality &lt;chr>,</span>
<span>#> # temperature &lt;dbl>, temperature_quality &lt;chr>,</span>
<span>#> # temperature_dewpoint &lt;dbl>, temperature_dewpoint_quality &lt;chr>,</span>
<span>#> # air_pressure &lt;dbl>, air_pressure_quality &lt;chr>,</span>
<span>#> # AW1_present_weather_observation_identifier &lt;chr>,</span>
<span>#> # AW1_automated_atmospheric_condition_code &lt;chr>,</span>
<span>#> # AW1_quality_automated_atmospheric_condition_code &lt;chr>,</span>
<span>#> # N03_original_observation &lt;chr>, N03_original_value_text &lt;chr>,</span>
<span>#> # N03_units_code &lt;chr>, N03_parameter_code &lt;chr>, REM_remarks &lt;chr>,</span>
<span>#> # REM_identifier &lt;chr>, REM_length_quantity &lt;chr>, REM_comment &lt;chr></span>
</code></pre></div>
<p>There's a parallel option as well, coming in handy with the larger ISD files:</p>
<div><pre><code>isd_parse<span>(</span>path<span>,</span> parallel <span>=</span> <span>TRUE</span><span>)</span>
</code></pre></div>
<h2>Visualize the data</h2>
<p>Make better date + time</p>
<div><pre><code>df <span>&lt;-</span> res <span>%>%</span>
  rowwise<span>()</span> <span>%>%</span>
  mutate<span>(</span>
    datetime <span>=</span> <span>as.POSIXct</span><span>(</span><span>strptime</span><span>(</span><span>paste</span><span>(</span><span>date</span><span>,</span> <span>paste0</span><span>(</span><span>substring</span><span>(</span>time<span>,</span> <span>1</span><span>,</span> <span>2</span><span>),</span> <span>&quot;:00:00&quot;</span><span>)),</span> <span>&quot;%Y-%m-%d %H:%M:%S&quot;</span><span>))</span>
  <span>)</span> <span>%>%</span>
  ungroup
</code></pre></div>
<p>viz</p>
<div><pre><code><span># removing some outliers (obs, look into more for serious use)</span>
<span>library</span><span>(</span>ggplot2<span>)</span>
ggplot<span>(</span>df<span>[</span>df<span>$</span>temperature <span>&lt;</span> <span>100</span><span>,],</span> aes<span>(</span>datetime<span>,</span> temperature<span>))</span> <span>+</span>
  geom_point<span>()</span> <span>+</span>
  theme_grey<span>(</span>base_size <span>=</span> <span>18</span><span>)</span>
</code></pre></div>
<p></p>
<h2>Future work</h2>
<p>I plan to improve performance via profiling and swapping out slower code for faster,
as well as possibly dropping down to C++.</p>
<p>There was already a featur request for asking for fields of interest instead of
getting all fields, so <a href="https://github.com/ropenscilabs/isdparser/issues/8">that's on the list</a>.</p>
<p>Do try out <code>isdparser</code>. Let us know of any bugs, and any feature requests!</p>