<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/readability-redux/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://rud.is/b/2017/09/04/readability-redux/"> R – rud.is</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>I <a rel="nofollow" href="https://rud.is/b/2017/08/24/reticulating-readability/" target="_blank">recently posted</a> about using a Python module to convert HTML to usable text. Since then, a new package has hit CRAN dubbed <a rel="nofollow" href="https://cran.r-project.org/web/packages/htm2txt/index.html" target="_blank"><code>htm2txt</code></a> that is 100% R and uses regular expressions to strip tags from text.</p>
<p>I <a rel="nofollow" href="https://rud.is/rpubs/htmltst/" target="_blank">gave it a spin</a> so folks could compare some basic output, but you should definitely give <code>htm2txt</code> a try on your own conversion needs since each method produces different results.</p>
<p>On my macOS systems, the <code>htm2txt</code> calls ended up invoking XQuartz (the X11 environment on macOS) and they felt kind of sluggish (base R regular expressions don’t have a “compile” feature and can be sluggish compared to other types of regular expression computations).</p>
<p>I decided to spend some of Labor Day (in the U.S.) laboring (not for long, though) on a (currently small) <code>rJava</code>-based R package dubbed <a rel="nofollow" href="https://github.com/hrbrmstr/jericho" target="_blank"><code>jericho</code></a> which builds upon work created by Martin Jericho which is used in at-scale initiatives like the Internet Archive. Yes, I’m trading Java for Python, but the combination of Java+R has been around for much longer and there are many solved problems in Java-space that don’t need to be re-invented (if you do know a header-only, cross-platform, C++ HTML-to-text library, definitely leave a comment).</p>
<p>Is it worth it to get <code>rJava</code> up and running to use <code>jericho</code> vs <code>htm2txt</code>? Let’s take a look:</p>
<pre><code>library(jericho) # devtools::install_github(&quot;hrbrmstr/jericho&quot;)
library(microbenchmark)
library(htm2txt)
library(tidyverse)

c(
  &quot;https://medium.com/starts-with-a-bang/science-knows-if-a-nation-is-testing-nuclear-bombs-ec5db88f4526&quot;,
  &quot;https://en.wikipedia.org/wiki/Timeline_of_antisemitism&quot;,
  &quot;http://www.healthsecuritysolutions.com/2017/09/04/watch-out-more-ransomware-attacks-incoming/&quot;
) -> urls

map_chr(urls, ~paste0(read_lines(.x), collapse=&quot;\n&quot;)) -> sites_html

microbenchmark(
  jericho_txt = {
    a &lt;- html_to_text(sites_html[1])
  },
  jericho_render = {
    a &lt;- render_html_to_text(sites_html[1])
  },
  htm2txt = {
    a &lt;- htm2txt(sites_html[1])
  },
  times = 10
) -> mb1

# microbenchmark(
#   jericho_txt = {
#     a &lt;- html_to_text(sites_html[2])
#   },
#   jericho_render = {
#     a &lt;- render_html_to_text(sites_html[2])
#   },
#   htm2txt = {
#     a &lt;- htm2txt(sites_html[2])
#   },
#   times = 10
# ) -> mb2

microbenchmark(
  jericho_txt = {
    a &lt;- html_to_text(sites_html[3])
  },
  jericho_render = {
    a &lt;- render_html_to_text(sites_html[3])
  },
  htm2txt = {
    a &lt;- htm2txt(sites_html[3])
  },
  times = 10
) -> mb3</code></pre>
<p>The second benchmark is commented out because I really didn’t have time wait for it to complete (FWIW <code>jericho</code> goes fast in that test). Here’s what the other two look like:</p>
<pre><code>mb1
## Unit: milliseconds
##            expr         min          lq        mean      median          uq         max neval
##     jericho_txt    4.121872    4.294953    4.567241    4.405356    4.734923    5.621142    10
##  jericho_render    5.446296    5.564006    5.927956    5.719971    6.357465    6.785791    10
##         htm2txt 1014.858678 1021.575316 1035.342729 1029.154451 1042.642065 1082.340132    10

mb3
## Unit: milliseconds
##            expr        min         lq       mean     median         uq        max neval
##     jericho_txt   2.641352   2.814318   3.297543   3.034445   3.488639   5.437411    10
##  jericho_render   3.034765   3.143431   4.708136   3.746157   5.953550   8.931072    10
##         htm2txt 417.429658 437.493406 446.907140 445.622242 451.443907 484.563958    10</code></pre>
<p>You should run the conversion functions on your own systems to compare the results (they’re somewhat large to incorporate here). I’m fairly certain they do a comparable — if not better — job of extracting clean, pertinent text.</p>
<p>I need to separate the package into two (one for the base JAR and the other for the conversion functions) and add some more tests before a CRAN submission, but I think this would be a good addition to the budding arsenal of HTML-to-text conversion options in R.</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://rud.is/b/2017/09/04/readability-redux/"> R – rud.is</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/wqQFDIw4DTI" height="1"></a></div>