<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/parse-noaa-integrated-surface-data-files/"></div></div><div><a data-url="https://www.r-bloggers.com/parse-noaa-integrated-surface-data-files/" data-counturl="https://www.r-bloggers.com/parse-noaa-integrated-surface-data-files/" data-count="horizontal" data-text="Parse NOAA Integrated Surface Data Files" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://ropensci.org/blog/technotes/2016/11/03/isdparser-release"> rOpenSci Blog - R</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>A new package <a target="_blank" rel="nofollow" href="https://cran.rstudio.com/web/packages/isdparser">isdparser</a> is<br>
on CRAN. <code>isdparser</code> was in part liberated from <a target="_blank" rel="nofollow" href="https://github.com/ropensci/rnoaa">rnoaa</a>,<br>
then improved. We'll use <code>isdparser</code> in <code>rnoaa</code> soon.</p>
<p><code>isdparser</code> does not download files for you from NOAA's ftp servers. The<br>
package focuses on parsing the files, which are variable length ASCII strings<br>
stored line by line, where each line has some mandatory data, and any amount<br>
of optional data.</p>
<p>The data is great, and includes for example, wind speed and direction, temperature,<br>
cloud data, sea level pressure, and more. Includes data from approximately 35,000<br>
stations worldwide, though best coverage is in North America/Europe/Australia.<br>
Data go all the way back to 1901, and are updated daily.</p>
<p>However, <a target="_blank" rel="nofollow">the data</a> is not fun to parse,<br>
warranting an packge to deal with the parsing.</p>
<h2>Installation</h2>
<div>
<pre><code data-lang="r">install.packages<span>(</span><span>&quot;isdparser&quot;</span><span>)</span>
</code></pre>
</div>
<p>If binaries aren't available, try from source:<br>
<code>install.packages(&quot;isdparser&quot;, type = &quot;source&quot;)</code> or from GitHub:<br>
<code>devtools::install_github(&quot;ropenscilabs/isdparser&quot;)</code></p>
<div>
<pre><code data-lang="r"><span>library</span><span>(</span>isdparser<span>)</span>
<span>library</span><span>(</span>dplyr<span>)</span>
</code></pre>
</div>
<h2>Parse individual lines</h2>
<p>If you want to parse individual lines, use <code>isd_parse_line()</code></p>
<p>First, let's get a ISD file. There's a few that come with the package:</p>
<div>
<pre><code data-lang="r">path <span>&lt;-</span> <span>system.file</span><span>(</span><span>'extdata/024130-99999-2016.gz'</span><span>,</span> package <span>=</span> <span>&quot;isdparser&quot;</span><span>)</span>
</code></pre>
</div>
<p>Read in the file</p>
<div>
<pre><code data-lang="r">lns <span>&lt;-</span> <span>readLines</span><span>(</span>path<span>,</span> encoding <span>=</span> <span>&quot;latin1&quot;</span><span>)</span>
</code></pre>
</div>
<p>Parse a line</p>
<div>
<pre><code data-lang="r">isd_parse_line<span>(</span>lns<span>[</span><span>1</span><span>])</span>
<span>#> # A tibble: 1 × 42</span>
<span>#> total_chars usaf_station wban_station date time date_flag</span>
<span>#> &lt;dbl> &lt;chr> &lt;chr> &lt;date> &lt;chr> &lt;chr></span>
<span>#> 1 54 024130 99999 2016-01-01 0000 4</span>
<span>#> # ... with 36 more variables: latitude &lt;dbl>, longitude &lt;dbl>,</span>
<span>#> # type_code &lt;chr>, elevation &lt;dbl>, call_letter &lt;chr>, quality &lt;chr>,</span>
<span>#> # wind_direction &lt;dbl>, wind_direction_quality &lt;chr>, wind_code &lt;chr>,</span>
<span>#> # wind_speed &lt;dbl>, wind_speed_quality &lt;chr>, ceiling_height &lt;chr>,</span>
<span>#> # ceiling_height_quality &lt;chr>, ceiling_height_determination &lt;chr>,</span>
<span>#> # ceiling_height_cavok &lt;chr>, visibility_distance &lt;chr>,</span>
<span>#> # visibility_distance_quality &lt;chr>, visibility_code &lt;chr>,</span>
<span>#> # visibility_code_quality &lt;chr>, temperature &lt;dbl>,</span>
<span>#> # temperature_quality &lt;chr>, temperature_dewpoint &lt;dbl>,</span>
<span>#> # temperature_dewpoint_quality &lt;chr>, air_pressure &lt;dbl>,</span>
<span>#> # air_pressure_quality &lt;chr>,</span>
<span>#> # AW1_present_weather_observation_identifier &lt;chr>,</span>
<span>#> # AW1_automated_atmospheric_condition_code &lt;chr>,</span>
<span>#> # AW1_quality_automated_atmospheric_condition_code &lt;chr>,</span>
<span>#> # N03_original_observation &lt;chr>, N03_original_value_text &lt;chr>,</span>
<span>#> # N03_units_code &lt;chr>, N03_parameter_code &lt;chr>, REM_remarks &lt;chr>,</span>
<span>#> # REM_identifier &lt;chr>, REM_length_quantity &lt;chr>, REM_comment &lt;chr></span>
</code></pre>
</div>
<p>By default you get a tibble back, but you can ask for a list in return instead.</p>
<p>Parsing by line allows the user to decide how to apply parsing across lines,<br>
whether it be <code>lapply</code> style, or for loop, etc.</p>
<h2>Parse entire files</h2>
<p>You can also parse entire ISD files.</p>
<div>
<pre><code data-lang="r">isd_parse<span>(</span>path<span>)</span>
<span>#> # A tibble: 2,601 × 42</span>
<span>#> total_chars usaf_station wban_station date time date_flag</span>
<span>#> &lt;dbl> &lt;chr> &lt;chr> &lt;date> &lt;chr> &lt;chr></span>
<span>#> 1 54 024130 99999 2016-01-01 0000 4</span>
<span>#> 2 54 024130 99999 2016-01-01 0100 4</span>
<span>#> 3 54 024130 99999 2016-01-01 0200 4</span>
<span>#> 4 54 024130 99999 2016-01-01 0300 4</span>
<span>#> 5 54 024130 99999 2016-01-01 0400 4</span>
<span>#> 6 39 024130 99999 2016-01-01 0500 4</span>
<span>#> 7 54 024130 99999 2016-01-01 0600 4</span>
<span>#> 8 39 024130 99999 2016-01-01 0700 4</span>
<span>#> 9 54 024130 99999 2016-01-01 0800 4</span>
<span>#> 10 54 024130 99999 2016-01-01 0900 4</span>
<span>#> # ... with 2,591 more rows, and 36 more variables: latitude &lt;dbl>,</span>
<span>#> # longitude &lt;dbl>, type_code &lt;chr>, elevation &lt;dbl>, call_letter &lt;chr>,</span>
<span>#> # quality &lt;chr>, wind_direction &lt;dbl>, wind_direction_quality &lt;chr>,</span>
<span>#> # wind_code &lt;chr>, wind_speed &lt;dbl>, wind_speed_quality &lt;chr>,</span>
<span>#> # ceiling_height &lt;chr>, ceiling_height_quality &lt;chr>,</span>
<span>#> # ceiling_height_determination &lt;chr>, ceiling_height_cavok &lt;chr>,</span>
<span>#> # visibility_distance &lt;chr>, visibility_distance_quality &lt;chr>,</span>
<span>#> # visibility_code &lt;chr>, visibility_code_quality &lt;chr>,</span>
<span>#> # temperature &lt;dbl>, temperature_quality &lt;chr>,</span>
<span>#> # temperature_dewpoint &lt;dbl>, temperature_dewpoint_quality &lt;chr>,</span>
<span>#> # air_pressure &lt;dbl>, air_pressure_quality &lt;chr>,</span>
<span>#> # AW1_present_weather_observation_identifier &lt;chr>,</span>
<span>#> # AW1_automated_atmospheric_condition_code &lt;chr>,</span>
<span>#> # AW1_quality_automated_atmospheric_condition_code &lt;chr>,</span>
<span>#> # N03_original_observation &lt;chr>, N03_original_value_text &lt;chr>,</span>
<span>#> # N03_units_code &lt;chr>, N03_parameter_code &lt;chr>, REM_remarks &lt;chr>,</span>
<span>#> # REM_identifier &lt;chr>, REM_length_quantity &lt;chr>, REM_comment &lt;chr></span>
</code></pre>
</div>
<p>Optionally, you can print progress:</p>
<div>
<pre><code data-lang="r">isd_parse<span>(</span>path<span>,</span> progress <span>=</span> <span>TRUE</span><span>)</span>
<span>#> # A tibble: 2,601 × 42</span>
<span>#> total_chars usaf_station wban_station date time date_flag</span>
<span>#> &lt;dbl> &lt;chr> &lt;chr> &lt;date> &lt;chr> &lt;chr></span>
<span>#> 1 54 024130 99999 2016-01-01 0000 4</span>
<span>#> 2 54 024130 99999 2016-01-01 0100 4</span>
<span>#> 3 54 024130 99999 2016-01-01 0200 4</span>
<span>#> 4 54 024130 99999 2016-01-01 0300 4</span>
<span>#> 5 54 024130 99999 2016-01-01 0400 4</span>
<span>#> 6 39 024130 99999 2016-01-01 0500 4</span>
<span>#> 7 54 024130 99999 2016-01-01 0600 4</span>
<span>#> 8 39 024130 99999 2016-01-01 0700 4</span>
<span>#> 9 54 024130 99999 2016-01-01 0800 4</span>
<span>#> 10 54 024130 99999 2016-01-01 0900 4</span>
<span>#> # ... with 2,591 more rows, and 36 more variables: latitude &lt;dbl>,</span>
<span>#> # longitude &lt;dbl>, type_code &lt;chr>, elevation &lt;dbl>, call_letter &lt;chr>,</span>
<span>#> # quality &lt;chr>, wind_direction &lt;dbl>, wind_direction_quality &lt;chr>,</span>
<span>#> # wind_code &lt;chr>, wind_speed &lt;dbl>, wind_speed_quality &lt;chr>,</span>
<span>#> # ceiling_height &lt;chr>, ceiling_height_quality &lt;chr>,</span>
<span>#> # ceiling_height_determination &lt;chr>, ceiling_height_cavok &lt;chr>,</span>
<span>#> # visibility_distance &lt;chr>, visibility_distance_quality &lt;chr>,</span>
<span>#> # visibility_code &lt;chr>, visibility_code_quality &lt;chr>,</span>
<span>#> # temperature &lt;dbl>, temperature_quality &lt;chr>,</span>
<span>#> # temperature_dewpoint &lt;dbl>, temperature_dewpoint_quality &lt;chr>,</span>
<span>#> # air_pressure &lt;dbl>, air_pressure_quality &lt;chr>,</span>
<span>#> # AW1_present_weather_observation_identifier &lt;chr>,</span>
<span>#> # AW1_automated_atmospheric_condition_code &lt;chr>,</span>
<span>#> # AW1_quality_automated_atmospheric_condition_code &lt;chr>,</span>
<span>#> # N03_original_observation &lt;chr>, N03_original_value_text &lt;chr>,</span>
<span>#> # N03_units_code &lt;chr>, N03_parameter_code &lt;chr>, REM_remarks &lt;chr>,</span>
<span>#> # REM_identifier &lt;chr>, REM_length_quantity &lt;chr>, REM_comment &lt;chr></span>
</code></pre>
</div>
<p>There's a parallel option as well, coming in handy with the larger ISD files:</p>
<div>
<pre><code data-lang="r">isd_parse<span>(</span>path<span>,</span> parallel <span>=</span> <span>TRUE</span><span>)</span>
</code></pre>
</div>
<h2>Visualize the data</h2>
<p>Make better date + time</p>
<div>
<pre><code data-lang="r">df <span>&lt;-</span> res <span>%>%</span>
  rowwise<span>()</span> <span>%>%</span>
  mutate<span>(</span>
    datetime <span>=</span> <span>as.POSIXct</span><span>(</span><span>strptime</span><span>(</span><span>paste</span><span>(</span><span>date</span><span>,</span> <span>paste0</span><span>(</span><span>substring</span><span>(</span>time<span>,</span> <span>1</span><span>,</span> <span>2</span><span>),</span> <span>&quot;:00:00&quot;</span><span>)),</span> <span>&quot;%Y-%m-%d %H:%M:%S&quot;</span><span>))</span>
  <span>)</span> <span>%>%</span>
  ungroup
</code></pre>
</div>
<p>viz</p>
<div>
<pre><code data-lang="r"><span># removing some outliers (obs, look into more for serious use)</span>
<span>library</span><span>(</span>ggplot2<span>)</span>
ggplot<span>(</span>df<span>[</span>df<span>$</span>temperature <span>&lt;</span> <span>100</span><span>,],</span> aes<span>(</span>datetime<span>,</span> temperature<span>))</span> <span>+</span>
  geom_point<span>()</span> <span>+</span>
  theme_grey<span>(</span>base_size <span>=</span> <span>18</span><span>)</span>
</code></pre>
</div>
<p><img alt="plot of chunk unnamed-chunk-11" src="https://i1.wp.com/ropensci.org/assets/blog-images/2016-11-03-isdparser-release/unnamed-chunk-11-1.png?w=456" data-recalc-dims="1"></p>
<h2>Future work</h2>
<p>I plan to improve performance via profiling and swapping out slower code for faster,<br>
as well as possibly dropping down to C++.</p>
<p>There was already a featur request for asking for fields of interest instead of<br>
getting all fields, so <a target="_blank" rel="nofollow" href="https://github.com/ropenscilabs/isdparser/issues/8">that's on the list</a>.</p>
<p>Do try out <code>isdparser</code>. Let us know of any bugs, and any feature requests!</p>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="https://www.r-bloggers.com/parse-noaa-integrated-surface-data-files/"></div></div><div><a data-url="https://www.r-bloggers.com/parse-noaa-integrated-surface-data-files/" data-counturl="https://www.r-bloggers.com/parse-noaa-integrated-surface-data-files/" data-count="vertical" data-text="Parse NOAA Integrated Surface Data Files" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://ropensci.org/blog/technotes/2016/11/03/isdparser-release"> rOpenSci Blog - R</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/Ok-Vmcya1qs"></a></div>