<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/in-praise-of-commonmark-wrangle-rmarkdown-files-without-regex/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://ropensci.org/technotes/2018/09/05/commonmark/"> rOpenSci - open tools for open science</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>You might have read <a rel="nofollow" href="https://ropensci.org/blog/2018/05/10/onboarding-social-weather/" target="_blank">my blog post analyzing the social weather of<br>
rOpenSci<br>
onboarding</a>,<br>
based on a text analysis of GitHub issues. I extracted text out of<br>
Markdown-formatted threads with regular expressions. I basically<br>
hammered away at the issues using tools I was familiar with until it<br>
worked! Now I know there’s a much better and cleaner way, that I’ll<br>
present in this note. Read on if you want to extract insights about<br>
text, code, links, etc. from R Markdown reports, Hugo website sources,<br>
GitHub issues… without writing messy and smelly code!</p>
<h2>Introduction to Markdown rendering and parsing</h2>
<p>This note will appear to you, dear reader, as an html page, either here<br>
on ropensci.org or on R-Bloggers, but I’m writing it as an R Markdown<br>
document, using Markdown syntax. I’ll knit it to Markdown and then<br>
Hugo’s Markdown processor,<br>
<a rel="nofollow" href="https://github.com/russross/blackfriday" target="_blank">Blackfriday</a>, will transform<br>
it to html. Elements such as <code># blabla</code> thus get transformed to<br>
<code><h1>blabla</h1></code>. Awesome!</p>
<p>The rendering of Markdown to html or XML can also be used as a way to<br>
<em>parse</em> it, <a rel="nofollow" href="https://github.com/ropensci/spelling/blob/622fc9cc200b69b3859f272c99c13c575dcb5105/R/parse-markdown.R#L12" target="_blank">which is what the <code>spelling</code> package does in order to<br>
identify text<br>
segments</a><br>
of R Markdown files, before spell checking them only, not code. I had an<br>
aha moment when seeing this <code>spelling</code> strategy: why did I ever use<br>
regex to parse Markdown for text analysis?! Transforming it to XML<br>
first, and then using XPath, would be much cleaner!</p>
<p>As a side-note, realizing how to simplify my old code made me think of<br>
<a rel="nofollow" href="https://github.com/jennybc/code-smells-and-feels" target="_blank">Jenny Bryan’s inspiring useR! keynote talk about code<br>
smells</a>. I asked her<br>
whether code full of regular expressions instead of dedicated parsing<br>
tools was a code smell, sadly it doesn’t have a specific name, but she<br>
confirmed my feeling that <em>not</em> using dedicated purpose-built tools<br>
might mean you’ll end up “re-inventing all of that logic yourself, in<br>
hacky way.”. If you have code falling under the definition below, maybe<br>
try to re-factor and if needed <a rel="nofollow" href="https://masalmon.eu/2018/07/22/wheretogethelp/" target="_blank">get<br>
help</a>.</p>
<p></p>
<blockquote class="twitter-tweet" data-lang="en">
<p lang="en" dir="ltr">
It’s that feeling when you want to do something that sounds simple but<br>
instead your code is like 10 stack overflow snippets slapped together<br>
that you could never explain to another human what they do <img alt="😰" src="https://s.w.org/images/core/emoji/2.4/72x72/1f630.png" class="wp-smiley"><br>
<a rel="nofollow" href="https://t.co/IF53AX6QvC" target="_blank">pic.twitter.com/IF53AX6QvC</a>
</p>
<p>— Dr. Alison Hill (@apreshill)<br>
<a rel="nofollow" href="https://twitter.com/apreshill/status/1035526182392000514?ref_src=twsrc%5Etfw" target="_blank">31<br>
d’agost de 2018</a>
</p></blockquote>
<p></p>
<h2>From Markdown to XML</h2>
<p>In this note I’ll use my local fork of rOpenSci’s website source, and<br>
use all the Markdown sources of blog posts as example data. The chunk<br>
below is therefore not portable, sorry about that.</p>
<pre><code>roblog &lt;- &quot;C:\\Users\\Maelle\\Documents\\ropensci\\roweb2\\content\\blog&quot;

all_posts &lt;- fs::dir_ls(roblog, regexp = &quot;*.md&quot;)
all_posts &lt;- all_posts[all_posts != &quot;_index.md&quot;]
</code></pre>
<p>My fork master branch isn’t entirely synced. It has 202 posts.</p>
<p>The code below uses the <a rel="nofollow" href="https://github.com/jeroen/commonmark" target="_blank"><code>commonmark</code><br>
package</a> to render Markdown to<br>
XML. Commonmark is a standardized specification for Markdown syntax by<br>
<a rel="nofollow" href="https://johnmacfarlane.net/tools" target="_blank">John McFarlane</a>. The <code>commonmark</code> R<br>
package by Jeroen Ooms wraps the official<br>
<a rel="nofollow" href="https://github.com/commonmark/cmark" target="_blank">cmark</a> library and is used by<br>
e.g. GitHub to render issues and readmes. Note that my function still<br>
has a hacky element, it uses a <code>blogdown</code> unexported function to strip<br>
the YAML header of posts! If you know a better way <a rel="nofollow" href="https://community.rstudio.com/t/does-blogdown-split-yaml-body-exist-as-an-exported-function-how-to-remove-the-yaml-of-a-markdown-r-markdown-document/13350/2" target="_blank">feel free to answer<br>
my question over at RStudio community discussion<br>
forum</a>.</p>
<pre><code>library(&quot;magrittr&quot;)
get_one_xml &lt;- function(md){
  md %>%
    readLines(encoding = &quot;UTF-8&quot;) %>%
    blogdown:::split_yaml_body() %>%
    .$body %>%
    commonmark::markdown_xml(extensions = TRUE) %>%
    xml2::read_xml()
}
</code></pre>
<p>See what it gives me for one post.</p>
<pre><code>get_one_xml(all_posts[42])
</code></pre>
<pre><code>## {xml_document}
## </code></pre>