<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/reticulating-readability/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://rud.is/b/2017/08/24/reticulating-readability/"> R – rud.is</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>I needed to clean some web HTML content for a project and I usually use <code>hgr::clean_text()</code> for it and that generally works pretty well. The <code>clean_text()</code> function uses an XSLT stylesheet to try to remove all non-“main text content” from an HTML document and it usually does a good job but there are some pages that it fails miserably on since it’s more of a brute-force method than one that uses any real “intelligence” when performing the text node targeting.</p>
<p>Most modern browsers have inherent or <a rel="nofollow" href="https://github.com/MHordecki/readability-redux" target="_blank">plugin-able</a> “readability” capability, and most of those are based — at least in part — on the seminal <a rel="nofollow" href="https://github.com/masukomi/ar90-readability" target="_blank">Arc90 implementation</a>. Many programming languages have a package or module that use a similar methodology, but I’m not aware of any R ports.</p>
<p>What do I mean by “clean txt”? Well, I can’t show the URL I was having trouble processing but I <em>can</em> show an example using a recent <a rel="nofollow" href="https://ropensci.org/blog/blog/2017/08/22/visdat" target="_blank">rOpenSci blog post</a>. Here’s what the raw HTML looks like after retrieving it:</p>
<pre><code>library(xml2)
library(httr)
library(reticulate)
library(magrittr)

res &lt;- GET(&quot;https://ropensci.org/blog/blog/2017/08/22/visdat&quot;)

content(res, as=&quot;text&quot;, endoding=&quot;UTF-8&quot;)</code></pre>
<div>
## [1] &quot;\n \n\n</div>