<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/analysis-of-the-renert-part-2-data-processing/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.blog.rdata.lu/post/2018-01-24-analysis-of-the-renert-part-2/"> rdata.lu Blog | Data science with R</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p></p>
<p>
<img src="https://i0.wp.com/www.blog.rdata.lu/images/renert.jpg?w=456" data-recalc-dims="1">
</p>
<p><em>This is part 2 of a 3 part blog post. This post uses the data that we scraped in <a rel="nofollow" href="http://www.blog.rdata.lu/post/2018-01-22-analysis-of-the-renert-part-1/" target="_blank">part 1</a> and prepares it for further analysis, which is quite technical. If you’re only interested in the results of the analysis, skip to part 3!</em></p>
<p>First, let’s load the data that we prepared in <a rel="nofollow" href="http://www.blog.rdata.lu/post/2018-01-22-analysis-of-the-renert-part-1/" target="_blank">part 1</a>. Let’s start with the full text:</p>
<pre><code>library(&quot;tidyverse&quot;)
library(&quot;tidytext&quot;)
renert = readRDS(&quot;renert_full.rds&quot;)</code></pre>
<p>I want to study the frequencies of words, so for this, I will use a function from the <code>tidytext</code> package called <code>unnest_tokens()</code> which breaks the text down into tokens. Each token is a word, which will then make it possible to compute the frequencies of words.</p>
<p>So, let’s unnest the tokens:</p>
<pre><code>renert = renert %>%
  unnest_tokens(word, text)</code></pre>
<p>We still need to do some cleaning before continuing. In Luxembourgish, <em>the</em> is written <em>d’</em> for feminine nouns. For example <em>d’Kaz</em> for the <em>the cat</em>. There’s also a bunch of <em>’t</em>s in the text, which is <em>it</em>. For example, the second line of the first song:</p>
<p><em>’T stung Alles an der Bléi,</em></p>
<p><em>Everything (it) was on flower,</em></p>
<p>We can remove these with a couple lines of code:</p>
<pre><code>renert_tokenized = renert %>%
  mutate(word = str_replace_all(word, &quot;d'&quot;, &quot;&quot;)) %>%
  mutate(word = str_replace_all(word, &quot;'t&quot;, &quot;&quot;))</code></pre>
<p>But that’s not all! We still need to remove so called stop words. Stop words are words that are very frequent, such as “and”, and these words usually do not add anything to the analysis. There are no set rules for defining a list of stop words, so I took inspiration for the stop words in English and German, and created my own, which you can get on <a rel="nofollow" href="https://github.com/b-rodrigues/stopwords_lu" target="_blank">Github</a>.</p>
<pre><code>stopwords = read.csv(&quot;stopwords_lu.csv&quot;, header = TRUE)</code></pre>
<p>For my Luxembourgish-speaking compatriots, I’d be glad to get help to make this list better! This list is far from perfect, certainly contains typos, or even words that have no reason to be there! Please help <img alt="😅" src="https://s.w.org/images/core/emoji/2.3/72x72/1f605.png" class="wp-smiley">.</p>
<p>Using this list of stop words, I can remove words that don’t add anything to the analysis. Creating a list of stop words for the Luxembourgish language is very challenging, because there might be stop words that come from German, such as “awer”, from the German “aber”, meaning <em>but</em>, but you could also use “mä”, from the French <em>mais</em>, meaning also but. Plus, as a kid, we never really learned how to write Luxembourgish. Actually, most Luxembourguians don’t know how to write Luxembourgish 100% correctly. This is because for a very long time, Luxembourgish was used for oral communication, and French for formal written correspondence. This is changing, and more and more people are learning how to write correctly. I definitely have a lot to learn! Thus, I have certainly missed a lot of stop words in the list, but I am hopeful that others will contribute to the list and make it better! In the meantime, that’s what I’m going to use.</p>
<p>Let’s take a look at some lines of the stop words data frame:</p>
<pre><code>head(stopwords, 20)</code></pre>
<pre><code>##          word
## 1           a
## 2           à
## 3         äis
## 4          är
## 5         ärt
## 6        äert
## 7        ären
## 8         all
## 9       allem
## 10      alles
## 11   alleguer
## 12        als
## 13       also
## 14         am
## 15         an
## 16 anerefalls
## 17        ass
## 18        aus
## 19       awer
## 20        bei</code></pre>
<p>We can remove the stop words from our tokens using an <code>anti_join()</code>:</p>
<pre><code>renert_tokenized = renert_tokenized %>%
  anti_join(stopwords)</code></pre>
<pre><code>## Joining, by = &quot;word&quot;</code></pre>
<pre><code>## Warning: Column `word` joining character vector and factor, coercing into
## character vector</code></pre>
<p>Let’s save this for later use:</p>
<pre><code>saveRDS(renert_tokenized, &quot;renert_tokenized.rds&quot;)</code></pre>
<p>I now have to do the same for the data that is stored by song. Because this is a list where each element is a data frame, I have to use <code>purrr::map()</code> to map each of the functions I used before to each data frame:</p>
<pre><code>renert_songs = readRDS(&quot;renert_songs_df.rds&quot;)
renert_songs = map(renert_songs, ~unnest_tokens(., word, text))
renert_songs = map(renert_songs, ~anti_join(., stopwords))
renert_songs = map(renert_songs, ~mutate(., word = str_replace_all(word, &quot;d'&quot;, &quot;&quot;)))
renert_songs = map(renert_songs, ~mutate(., word = str_replace_all(word, &quot;'t&quot;, &quot;&quot;)))</code></pre>
<p>Let’s take a look at the object we have:</p>
<pre><code>head(renert_songs[[1]])</code></pre>
<pre><code>## # A tibble: 6 x 1
##        word
##       </code></pre>