<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/qualitative-data-science-using-rqda-to-analyse-interviews/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://r.prevos.net/qualitative-data-science/"> The Devil is in the Data</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><a rel="nofollow" href="https://kenandeen.wordpress.com/2015/01/18/quantitative-vs-qualitative-data/" target="_blank"><img src="https://kenandeen.files.wordpress.com/2015/01/quantitave-vs-qualitative.png?resize=328%2C245" alt="Qualitative Data Science" width="328" data-recalc-dims="1" height="245"></a>Qualitative data science sounds like a contradiction in terms. Data scientists generally solve problems using numerical solutions. Even the analysis of text is reduced to a numerical problem using Markov chains, topic analysis, <a rel="nofollow" href="https://r.prevos.net/tap-water-sentiment-analysis/" target="_blank">sentiment analysis</a> and other mathematical tools.</p>
<p>Scientists and professionals consider numerical methods the gold standard of analysis. There is, however, a price to pay when relying on numbers alone. Numerical analysis reduces the complexity of the social world. When analysing people, numbers present an illusion of precision and accuracy. Giving primacy to quantitative research in the social sciences comes at a high price. The dynamics of reality are reduced to statistics, losing the narrative of the people that the research aims to understand.</p>
<p>Being both an engineer and a social scientist, I acknowledge the importance of both numerical and qualitative methods. My dissertation used a mixed-method approach to review the relationship between employee behaviour and customer perception in water utilities. This article introduces some aspects of qualitative data science with an example from my dissertation. In this article, I show how I analysed data from interviews using both quantitative and qualitative methods and demonstrate why qualitative data science is better to understand text than numerical methods. The most recent version of the code is available on my <a rel="nofollow" href="https://github.com/pprevos/r.prevos.net/tree/master/Hydroinformatics/Macromarketing" target="_blank">GitHub repository</a>.</p>
<h2>Qualitative Data Science</h2>
<p>The often celebrated artificial intelligence of machine learning is impressive but does not come close to human intelligence and ability to understand the world. Many data scientists are working on automated text analysis to solve this issue (the <a rel="nofollow" href="https://cran.r-project.org/web/packages/topicmodels/" target="_blank">topicmodels</a> package is an example of such an attempt). These efforts are impressive but even the smartest text analysis algorithm is not able to derive <em>meaning</em> from text. To fully embrace all aspects of data science we need to be able to methodically undertake <a rel="nofollow" href="https://en.wikipedia.org/wiki/Qualitative_research" target="_blank">qualitative data analysis</a>.</p>
<p>The capabilities of R in numerical analysis are impressive but it can also assist with Qualitative Data Analysis (QDA). Huang Ronggui from Hong Kong developed the <a rel="nofollow" href="http://rqda.r-forge.r-project.org/" target="_blank">RQDA package</a> to analyse texts in R. RQDA assists with qualitative data analysis using a GUI front-end to analyse collections texts. The video below contains a complete course in using this software. Below the video, I share an example from my dissertation which compares qualitative and quantitative methods for analysing text.</p>
<p><iframe allowfullscreen="true" src="https://www.youtube.com/embed/videoseries?list=PL66CB2FF65368715C&hl=en_US" width="450" type="text/html"></iframe></p>
<p>For my dissertation about <a rel="nofollow" href="http://invisiblewater.org/" target="_blank">water utility marketing</a>, I interviewed seven people from various organisations. The purpose of these interviews was to learn about the value proposition for water utilities. The data consists of the transcripts of six interviews which I manually coded using RQDA. For reasons of agreed anonymity, I cannot provide the raw data file of the interviews through GitHub.</p>
<h2>Numerical Text Analysis</h2>
<p>Word clouds are a popular method for exploratory analysis of texts. The wordcloud is created with the <a rel="nofollow" href="https://cran.r-project.org/web/packages/tm/index.html" target="_blank">text mining</a> and wordcloud packages. The transcribed interviews are converted to a text corpus (the native format for the tm package) and whitespace, punctuation etc is removed. This code snippet opens the RQDA file and extracts the texts from the database. RQDA stores all text in an SQLite database and the package provides a query command to extract data.</p>
<pre>
# INIT
library(tidyverse)
library(RQDA)
library(tm)
library(wordcloud)
library(topicmodels)
library(igraph)

#Load data
library(RQDA)
openProject(&quot;Hydroinformatics/Macromarketing/stakeholders.rqda&quot;)

# Word frequency diagram
library(tm)
interviews &lt;- RQDAQuery(&quot;SELECT file FROM source&quot;)
interviews$file &lt;- apply(interviews, 1, function(x) gsub(&quot;…&quot;, &quot;...&quot;, x))
interviews$file &lt;- apply(interviews, 1, function(x) gsub(&quot;’&quot;, &quot;&quot;, x))
interviews &lt;- Corpus(VectorSource(interviews$file))
interviews &lt;- tm_map(interviews, stripWhitespace)
interviews &lt;- tm_map(interviews, content_transformer(tolower))
interviews &lt;- tm_map(interviews, removeWords, stopwords(&quot;english&quot;))
interviews &lt;- tm_map(interviews, removePunctuation)
interviews &lt;- tm_map(interviews, removeNumbers)
interviews &lt;- tm_map(interviews, removeWords, c(&quot;interviewer&quot;, &quot;interviewee&quot;))

library(wordcloud)
set.seed(1969)
wordcloud(interviews, min.freq = 10, max.words = 50, rot.per=0.35, 
 colors = brewer.pal(8, &quot;Blues&quot;)[-1:-5])
</pre>
<p><a rel="nofollow" href="https://r.prevos.net/?attachment_id=1255" target="_blank"><img data-orig-file="https://i2.wp.com/r.prevos.net/wp-content/uploads/sites/11/2018/05/wordcloud.png?fit=932%2C779&amp;ssl=1" data-orig-size="932,779" data-medium-file="https://i2.wp.com/r.prevos.net/wp-content/uploads/sites/11/2018/05/wordcloud.png?fit=300%2C251&amp;ssl=1" data-attachment-id="1255" data-image-meta="{" data-image-description="" src="https://i2.wp.com/r.prevos.net/wp-content/uploads/sites/11/2018/05/wordcloud.png?resize=584%2C488&ssl=1" data-permalink="https://r.prevos.net/qualitative-data-science/wordcloud/" data-large-file="https://i2.wp.com/r.prevos.net/wp-content/uploads/sites/11/2018/05/wordcloud.png?fit=584%2C488&amp;ssl=1" alt="Word cloud of interview transcripts." data-image-title="wordcloud" sizes="(max-width: 584px) 100vw, 584px" width="450" data-comments-opened="1" class="wp-image-1255" data-recalc-dims="1"></a></p>
<p>This word cloud makes it clear that the interviews are about water businesses and customers, which is a pretty obvious statement. The interviews are also about the opinion of the interviewees (think). While the word cloud is aesthetically pleasing and provides a quick snapshot of the content of the texts, they cannot inform us about their meaning.</p>
<p>Topic modelling is a more advanced method to extract information from the text by assessing the proximity of words to each other. The <a rel="nofollow" href="https://eight2late.wordpress.com/2015/09/29/a-gentle-introduction-to-topic-modeling-using-r/" target="_blank">topic modelling</a> package provides functions to perform this analysis. I am not an expert in this field and simply followed basic steps using default settings with four topics.</p>
<pre>
# Topic Analysis
library(topicmodels)
dtm &lt;- DocumentTermMatrix(interviews)
dtm &lt;- removeSparseTerms(dtm, 0.99)
ldaOut &lt;-LDA(dtm, k = 4)
terms(ldaOut,6)
</pre>
<p>This code converts the corpus created earlier into a <a rel="nofollow" href="https://en.wikipedia.org/wiki/Document-term_matrix" target="_blank">Document-Term Matrix</a>, which is a matrix of words and documents (the interviews) and the frequency at which each of these words occurs. The LDA function applies a <a rel="nofollow" href="https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation" target="_blank">Latent Dietrich Allocation</a> to the matrix to define the topics. The variable k defines the number of anticipated topics. An LDA is similar to clustering in multivariate data. The final output is a table with six words for each topic.</p>
<h2>Topics extracted from interviews</h2>
<table>
<thead>
<tr>
<th>Topic 1</th>
<th>Topic 2</th>
<th>Topic 3</th>
<th>Topic 4</th>
</tr>
</thead>
<tbody>
<tr>
<td>water</td>
<td>water</td>
<td>customers</td>
<td>water</td>
</tr>
<tr>
<td>think</td>
<td>think</td>
<td>water</td>
<td>think</td>
</tr>
<tr>
<td>actually</td>
<td>inaudible</td>
<td>customer</td>
<td>companies</td>
</tr>
<tr>
<td>customer</td>
<td>people</td>
<td>think</td>
<td>yeah</td>
</tr>
<tr>
<td>businesses</td>
<td>service</td>
<td>business</td>
<td>issues</td>
</tr>
<tr>
<td>customers</td>
<td>businesses</td>
<td>service</td>
<td>don’t</td>
</tr>
</tbody>
</table>
<p></p>
<p>This table does not tell me much at all about what was discussed in the interviews. Perhaps it is the frequent use of the word “water” or “think”—I did ask people their opinion about water-related issues. To make this analysis more meaningful I could perhaps manually remove the words water, yeah, and so on. This introduces bias in the analysis and reduces the reliability of the topic analysis because I would be interfering with the text.</p>
<p>Numerical text analysis sees a text as a bag of words instead of a set of meaningful words. It seems that any automated text mining needs a lot of manual cleaning to derive anything meaningful. This excursion shows that automated text analysis is not a sure-fire way to analyse the meaning of a collection of words. After a lot of trial and error to try to make this work, I decided to go back to my roots of qualitative analysis using RQDA as my tool.</p>
<h2>Qualitative Data Science Using RQA</h2>
<p>To use RQDA for qualitative data science, you first need to manually analyse each text and assign codes (topics) to parts of the text. The image below shows a question and answer and how it was coded. All marked text is blue and the codes are shown between markers. Coding a text is an iterative process that aims to extract meaning from a text. The advantage of this method compared to numerical analysis is that the researcher injects meaning into the analysis. The disadvantage is that the analysis will always be biased, which in the social sciences is unavoidable. My list of topics was based on words that appear in a marketing dictionary so that I analysed the interviews from that perspective.</p>
<div class="wp-caption"><a rel="nofollow" href="https://r.prevos.net/?attachment_id=1250" target="_blank"><img data-orig-file="https://i2.wp.com/r.prevos.net/wp-content/uploads/sites/11/2018/05/coding.png?fit=946%2C358&amp;ssl=1" data-orig-size="946,358" data-medium-file="https://i2.wp.com/r.prevos.net/wp-content/uploads/sites/11/2018/05/coding.png?fit=300%2C114&amp;ssl=1" data-attachment-id="1250" data-image-meta="{" data-image-description="" src="https://i2.wp.com/r.prevos.net/wp-content/uploads/sites/11/2018/05/coding.png?resize=584%2C221&ssl=1" data-permalink="https://r.prevos.net/qualitative-data-science/coding/" data-large-file="https://i2.wp.com/r.prevos.net/wp-content/uploads/sites/11/2018/05/coding.png?fit=584%2C221&amp;ssl=1" alt="Qualitative data science using RQDA." data-image-title="coding" sizes="(max-width: 584px) 100vw, 584px" width="450" data-comments-opened="1" class="wp-image-1250" data-recalc-dims="1"></a>
<p class="wp-caption-text">Example of text coded with RQDA.</p>
</div>
<p>My first step was to look at the occurrence of codes (themes) in each of the interviews.</p>
<pre>
codings &lt;- getCodingTable()[,4:5]
categories &lt;- RQDAQuery(&quot;SELECT filecat.name AS category, source.name AS filename 
                         FROM treefile, filecat, source 
                         WHERE treefile.catid=filecat.catid AND treefile.fid=source.id AND treefile.status=1&quot;)
codings &lt;- merge(codings, categories, all.y = TRUE)
head(codings)

# Open coding
reorder_size &lt;- function(x) {
    factor(x, levels = names(sort(table(x))))
}
ggplot(codings, aes(reorder_size(codename), fill=category)) + geom_bar(stat=&quot;count&quot;) + 
    facet_grid(~filename) + coord_flip() + 
    theme(legend.position=&quot;bottom&quot;, legend.title=element_blank()) + 
    ylab(&quot;Code frequency in interviews&quot;) + xlab(&quot;Code&quot;)
ggsave(&quot;Hydroinformatics/Macromarketing/rqda_codes.png&quot;, width=9, height=11, dpi = 300)
</pre>
<p>The code uses an internal RQDA function <em>getCodingTable</em> to obtain the primary data. The <em>RQDAQuery</em> function provides more flexibility and can be used to build more complex queries of the data. You can view the structure of the RQDA database using the <em>RQDATables</em> function.</p>
<div class="wp-caption"><img data-orig-file="https://i0.wp.com/r.prevos.net/wp-content/uploads/sites/11/2017/02/interviewcode.png?fit=1571%2C1756&amp;ssl=1" data-orig-size="1571,1756" data-medium-file="https://i0.wp.com/r.prevos.net/wp-content/uploads/sites/11/2017/02/interviewcode.png?fit=268%2C300&amp;ssl=1" data-attachment-id="388" data-image-meta="{" data-image-description="" src="https://i0.wp.com/r.prevos.net/wp-content/uploads/sites/11/2017/02/interviewcode.png?resize=584%2C653&ssl=1" data-permalink="https://r.prevos.net/qualitative-data-science/interviewcode/" data-large-file="https://i0.wp.com/r.prevos.net/wp-content/uploads/sites/11/2017/02/interviewcode.png?fit=584%2C653&amp;ssl=1" alt="Occurrence of themes from six interviews - qualitative data science" data-image-title="interviewcodes" sizes="(max-width: 584px) 100vw, 584px" width="450" data-comments-opened="1" class="wp-image-388" data-recalc-dims="1">
<p class="wp-caption-text">The occurrence of themes from six interviews.</p>
</div>
<p>This bar chart helps to explore the topics that interviewees discussed but it does not help to understand how these topic relate to each other. This method provides a better view than the ‘bag of words’ approach because the text has been given meaning.</p>
<p>RQDA provides a facility to assign each code to a code category. This structure can be visualised using a network. The network is visualised using the igraph package and the graph shows how codes relate to each other.</p>
<p>Qualitative data analysis can create value from a text by interpreting it from a given perspective. This article is not even an introduction to the science and art of qualitative data science. I hope it invites you to explore RQA and similar tools.</p>
<p>If you are interested in finding out more about how I used this analysis, then read my dissertation on <a rel="nofollow" href="http://hdl.handle.net/1959.9/561679" target="_blank">customer service in water utilities</a>.</p>
<p><a rel="nofollow" href="https://r.prevos.net/?attachment_id=1256" target="_blank"><img data-orig-file="https://i1.wp.com/r.prevos.net/wp-content/uploads/sites/11/2018/05/network.png?fit=1014%2C1016&amp;ssl=1" data-orig-size="1014,1016" data-medium-file="https://i1.wp.com/r.prevos.net/wp-content/uploads/sites/11/2018/05/network.png?fit=300%2C300&amp;ssl=1" data-attachment-id="1256" data-image-meta="{" data-image-description="" src="https://i1.wp.com/r.prevos.net/wp-content/uploads/sites/11/2018/05/network.png?resize=584%2C585&ssl=1" data-permalink="https://r.prevos.net/qualitative-data-science/network/" data-large-file="https://i1.wp.com/r.prevos.net/wp-content/uploads/sites/11/2018/05/network.png?fit=584%2C585&amp;ssl=1" alt="Qualitative data science: Using RQDA to analyse a text" data-image-title="network" sizes="(max-width: 584px) 100vw, 584px" width="450" data-comments-opened="1" class="wp-image-1256" data-recalc-dims="1"></a></p>
<pre>
# Axial coding
edges &lt;- RQDAQuery(&quot;SELECT codecat.name, freecode.name FROM codecat, freecode, treecode 
          WHERE codecat.catid=treecode.catid AND freecode.id=treecode.cid&quot;)

g &lt;- graph_from_edgelist(as.matrix(edges), directed=FALSE)
V(g)$name &lt;- gsub(&quot; &quot;, &quot;\n&quot;, V(g)$name)

c &lt;- spinglass.community(g)
par(mar=rep(0,4))
set.seed(666)
plot(c, g, 
     vertex.size=10,
     vertex.color=NA,
     vertex.frame.color=NA,
     edge.width=E(g)$weight,
     layout=layout.drl)
</pre>
<p>The post <a rel="nofollow" href="https://r.prevos.net/qualitative-data-science/" target="_blank">Qualitative Data Science: Using RQDA to analyse interviews</a> appeared first on <a rel="nofollow" href="https://r.prevos.net/" target="_blank">The Devil is in the Data</a>.</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://r.prevos.net/qualitative-data-science/"> The Devil is in the Data</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/2SuUZ6tx5ho" height="1"></a></div>