<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/yet-another-power-outages-post-full-tidyverse-edition/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://rud.is/b/2017/11/02/yet-another-power-outages-post-full-tidyverse-edition/"> R – rud.is</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>This past weekend, violent windstorms raged through New England. We — along with over 500,000 other Mainers — went “dark” in the wee hours of Monday morning and (this post was published on Thursday AM) we still have no utility-provided power nor high-speed internet access. The children have turned iFeral, and being a remote worker has been made that much more challenging. Earlier in the week, even making a cellular phone call (not an Google Voice or other VoIP-ish call, just pressing buttons in the phone “app” in iOS) resulted in an “All circuits are busy” message vs human contact. (I had to repair our generator at some point between then and now, but it’s all a blur at this point).</p>
<p>Late Tuesday night, we checked the <a rel="nofollow" href="https://outagemap.cmpco.com/cmp/" target="_blank">Central Maine Power outage info</a> and were greeting with a “November 4, 2017” estimate. After regaining composure, we doubled down on the fact that we’d be extreme indoor glamping for a while longer.</p>
<p>As noted, I cope by coding and have a history of scraping Central Maine Power’s site for outage info. I ceased when I discovered the site &amp; twitter bot I mentioned in a recent post, as it does that for the entirety of the U.S. (though many power companies continue make it difficult to scrape their outage info).</p>
<p>However, I wanted to see just how many other streets were in the same position as we are (I should note that less than a mile from us there are folks with power and internet, due mostly to their proximity to “vital” resources and how screwed up the Maine power grid is). Rather than reuse existing code, I wanted to make a modern, tidyverse edition of scrapers. If you follow enough paths in the aforementioned outage site, you’ll see that you eventually come to a page with a pretty ugly <code>iframe</code> that lets you poke around counties and towns. The following code traverses that tree to get street-level outage info:</p>
<pre><code>library(rvest)
library(stringi)
library(hrbrthemes)
library(tidyverse)

# helper to make numbers from comma strings; i still find it amusing that
# this has never been a core &quot;S&quot; or base &quot;R&quot; function given that the
# central goal of both languages are to work with data
to_num &lt;- function(x) { as.numeric(stri_replace_all_fixed(x, &quot;,&quot;, &quot;&quot;)) }

# top of the tree
pg &lt;- read_html(&quot;http://www3.cmpco.com/OutageReports/CMP.html&quot;)

# basic idiom all the way down is to find the links to traverse until we get to
# the street level data, plucking the timestamp of the CMP report along the way
html_nodes(pg, &quot;a&quot;) %>% 
  map_df(~{
    
    county &lt;- stri_trans_totitle(html_text(.x))
    cpg &lt;- read_html(sprintf(&quot;http://www3.cmpco.com/OutageReports/%s&quot;, html_attr(.x, &quot;href&quot;)))
        
    message(sprintf(&quot;Processing %s...&quot;, county))
    
    html_nodes(cpg, xpath=&quot;.//td[not(@colspan = '2') and not(@align = 'right')][1]/a&quot;) %>% 
      map_df(~{
        
        town &lt;- stri_trans_totitle(html_text(.x))
        tpg &lt;- read_html(sprintf(&quot;http://www3.cmpco.com/OutageReports/%s&quot;, html_attr(.x, &quot;href&quot;)))
        
        message(sprintf(&quot;  - %s&quot;, town))
    
        html_node(tpg, xpath=&quot;.//td[contains(., 'Update:')]&quot;) %>%
          html_text() %>%
          stri_replace_first_regex(&quot;Update: &quot;, &quot;&quot;) %>%
          as.POSIXct(&quot;%b %d, %Y %I:%M %p&quot;, tz=&quot;America/New_York&quot;) -> ts
        
        html_node(tpg, &quot;table&quot;) %>% 
          html_table() %>% 
          docxtractr::assign_colnames(3) %>%  
          docxtractr::mcga() %>% # in github version
          mutate(street = stri_trans_totitle(street)) %>% 
          mutate_at(vars(-estimated_restoration, -street), .funs=to_num) %>% 
          filter(!is.na(total_customersby_street)) %>% 
          mutate(timestamp = ts) %>% 
          mutate(county = county) %>% 
          mutate(town = town) %>% 
          tbl_df()
      })
    
  }) -> xdf

xdf &lt;- mutate(xdf, estimated_restoration = as.POSIXct(estimated_restoration, &quot;%b %d, %Y %I:%M %p&quot;, tz=&quot;America/New_York&quot;))

xdf
## # A tibble: 10,601 x 7
##           street total_customersby_street customerswithout_power estimated_restoration           timestamp       county   town
##            </code></pre>