<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/writing-a-conference-abstract-the-data-science-way/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.mango-solutions.com/wp/2017/03/writing-a-conference-abstract-the-data-science-way/"> Mango Solutions » R Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><em>Adnan Fiaz</em></p>
<p>Conferences are an ideal platform to share your work with the wider community. However, as we all know, conferences require potential speakers to submit abstracts about their talk. And writing abstracts is not necessarily the most rewarding work out there. I have actually never written one so when asked to prepare abstracts for this year’s conferences I didn’t really know where to start.</p>
<div>
<p>So, I did what any sane person would do: get data. As Mango has organised a number of EARL conferences there are a good deal of abstracts available, both accepted and not accepted. In this blogpost I’m going to use the <strong>tidytext</strong> package to analyse these abstracts and see what distinguishes the accepted abstracts from the rest.</p>
<p><em>Disclaimer:</em> the objective of this blogpost is not to present a rigorous investigation into conference abstracts but rather an exploration of, and potential use for, the <strong>tidytext</strong> package.</p>
<div>
<h2>The data</h2>
<p>I don’t know what it’s like for other conferences but for EARL all abstracts are submitted through an online form. I’m not sure if these forms are stored in a database but I received them as a PDF. To convert the PDFs to text I make use of the <em>pdftotext</em> program as outlined in this <a target="_blank" rel="nofollow" href="http://www.mango-solutions.com/wp/2017/03/writing-a-conference-abstract-the-data-science-way/stackoverflow.com/questions/21445659/use-r-to-convert-pdf-files-to-text-files-for-text-mining">stackoverflow thread</a>.</p>
<pre>list_of_files &lt;- list.files(&quot;Abstracts Received/&quot;, pattern=&quot;.pdf&quot;, full.names=TRUE)
convert_pdf &lt;- function(fileName){
  outputFile &lt;- paste0('&quot;', gsub('.pdf', '.txt', fileName),'&quot;')
  inputFile &lt;- paste0('&quot;', fileName, '&quot;')
  # I turned the layout option on because the input is a somewhat tabular layout
  command &lt;- paste('&quot;C:/Program Files/xpdf/bin64/pdftotext.exe&quot; -layout', inputFile, outputFile)
  system(command, wait=FALSE)
}
result &lt;- map(list_of_files, convert_pdf)
failures &lt;- length(list_of_files) - sum(unlist(result))
if(failures>0){
  warning(sprintf(&quot;%i files failed to convert&quot;, c(failures))
}
</pre>
<p>Now that I have converted the files I can read them in and extract the relevant fields.</p>
<pre># load functions extract_abstracts(), remove_punctuation()
source('Scripts/extract_abstracts.R')

input_files &lt;- list.files(&quot;Abstracts Received/&quot;, pattern=&quot;.txt&quot;, full.names=TRUE)

acceptance_results &lt;- read_csv(&quot;Abstracts Received/acceptance_results.csv&quot;) %>% 
  mutate(Title=remove_punctuation(Title), Title=str_sub(Title, end=15)) %>% 
  rename(TitleShort=Title)

# the conversion to text created files with an &quot;incomplete final line&quot; for which
# readLines generates a warning, hence the suppressWarnings
input_data &lt;- suppressWarnings(map(input_files, readLines)) %>% 
  map_df(extract_abstracts) %>% 
  mutate(AbstractID=row_number(), TitleShort=str_sub(Title, end=15)) %>% 
  left_join(acceptance_results, by=&quot;TitleShort&quot;) %>% 
  filter(!is.na(Accepted)) %>% 
  select(AbstractID, Abstract, Accepted)

# a code chunk that ends with a plot is a good code chunk
qplot(map_int(input_data$Abstract, nchar)) + 
  labs(title=&quot;Length of abstracts&quot;, x=&quot;Number of characters&quot;, y=&quot;Number of abstracts&quot;)
</pre>
<p><img alt="" src="https://i2.wp.com/github.com/MangoTheCat/blog_analysis_abstracts/raw/master/blogpost_conference_abstracts_files/figure-html/unnamed-chunk-2-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
</div>
<div>
<h2>The analysis</h2>
<p>So, now that I have the data ready I can apply some <strong>tidytext</strong> magic. I will first convert the data into a tidy format, then clean it up a bit and finally create a few visualisations.</p>
<pre>data(stop_words)
tidy_abstracts &lt;- input_data %>% 
  mutate(Abstract=remove_punctuation(Abstract)) %>% 
  unnest_tokens(word, Abstract) %>% # abracadabra!
  anti_join(stop_words %>% filter(word!=&quot;r&quot;)) %>% # In this case R is a word
  filter(is.na(as.numeric(word))) # filter out numbers

# my personal mantra: a code chunk that ends with a plot is a good code chunk
tidy_abstracts %>% 
  count(AbstractID, Accepted) %>% 
  ggplot() +
  geom_density(aes(x=n, colour=Accepted), size=1) +
  labs(title=&quot;Distribution of number of words per abstract&quot;, x=&quot;Number of words&quot;) 
</pre>
<p><img alt="" src="https://i0.wp.com/github.com/MangoTheCat/blog_analysis_abstracts/raw/master/blogpost_conference_abstracts_files/figure-html/unnamed-chunk-3-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>The abstracts with a higher number of words have a slight advantage but I wouldn’t bet on it. There is something to be said for being succinct. But what really matters is obviously content so let’s have a look at what words are commonly used.</p>
<pre>tidy_abstracts %>% 
  count(word, Accepted,  sort=TRUE) %>% # count the number of observations per category and word
  group_by(Accepted) %>% 
  top_n(20) %>% # select the top 20 counts per category
  ungroup() %>% 
  ggplot() +
  geom_col(aes(x=word, y=n, fill=Accepted), show.legend = FALSE) +
  coord_flip() +
  labs(x=&quot;&quot;, y=&quot;Count&quot;, title=&quot;Wordcount by Acceptance category&quot;) +
  facet_grid(~ Accepted)
</pre>
<p><img alt="" src="https://i1.wp.com/github.com/MangoTheCat/blog_analysis_abstracts/raw/master/blogpost_conference_abstracts_files/figure-html/unnamed-chunk-4-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>Certainly an interesting graph! It may have been better to show the proportions instead of counts as the number of abstracts in each category are not equal. Nevertheless, the conclusion remains the same. The words “r” and “data” are clearly the most common. However, what is more interesting is that abstracts in the “yes” category use certain words significantly more often than abstracts in the “no” category and vice versa (more often because a missing bar doesn’t necessarily mean a zero observation). For example, the words “science”, “production” and “performance” occur more often in the “yes” category. Vice versa, the words “tools”, “product”, “package” and “company(ies)” occur more often in the “no” category. Also, the word “application” occurs in its singular form in the “no” category and in its plural form in the “yes” category. Certainly, at EARL we like our applications to be plural, it is in the name after all.</p>
<p>There is one important caveat with the above analysis and that is to do with the frequency of words within abstracts. The overall frequencies aren’t really that high and one abstract’s usage of a particular word can make it seem more important than it really is. Luckily the <strong>tidytext</strong> package provides a solution for that as I can now easily calculate the <a target="_blank" rel="nofollow" href="http://tidytextmining.com/tfidf.html#term-frequency-in-jane-austens-novels">TF-IDF</a> score.</p>
<pre>tidy_abstracts %>% 
  count(Accepted, word, sort=TRUE) %>% # count the number of observations per category and word
  bind_tf_idf(word, Accepted, n) %>% # calculate tf-idf
  group_by(Accepted) %>% 
  top_n(10) %>% # select the top 10 scores per category
  ungroup() %>% 
  ggplot() +
  geom_col(aes(x=word, tf_idf, fill=Accepted), show.legend = FALSE) + 
  labs(x=&quot;&quot;, y=&quot;TF-IDF&quot;, title=&quot;TF-IDF by Acceptance category&quot;) +
  coord_flip() +
  facet_grid(~ Accepted)
</pre>
<p><img alt="" src="https://i0.wp.com/github.com/MangoTheCat/blog_analysis_abstracts/raw/master/blogpost_conference_abstracts_files/figure-html/unnamed-chunk-5-1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>Note that I have aggregated the counts over the Acceptance category as I’m interested in what words are important within a category and not within a particular abstract. There isn’t an obvious pattern visible in the results but I can certainly hypothesise. Words like “algorithm”, “effects”, “visualize”, “ml” and “optimization” point strongly towards the application side of things. Whereas words like “concept”, “objects” and “statement” are softer and more generic. XBRL is the odd one out here but <a target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/XBRL">interesting</a> in it’s own right, whoever submitted that abstract should perhaps consider re-submitting as it’s quite unique.</p>
</div>
<div>
<h2>Next Steps</h2>
<p>That’s it for this blogpost but here are some next steps I would do if I had more time:</p>
<ul>
<li>Add more abstracts from previous years / other conferences</li>
<li>Analyse combination of words (n-grams) to work towards what kind of sentences should go into an abstract</li>
<li>The content isn’t the only thing that matters. By adding more metadata (time of submission, previously presented, etc.) the model can be made more accurate</li>
<li>Try out topic modeling on the accepted abstracts to help with deciding what streams would make sense</li>
<li>Train a neural network with all abstracts and generate a winning abstract [insert evil laugh]</li>
</ul>
</div>
<div>
<h2>Conclusion</h2>
<p>In this blogpost I have explored text data taken from abstract submissions to the EARL conference using the fabulous <strong>tidytext</strong> package. I analysed words from abstracts that were accepted versus those that weren’t and also compared their TF-IDF score. If you want to know more about the <strong>tidytext</strong> package come to the <a target="_blank" rel="nofollow" href="http://www.londonr.org/download/Workshop%20-%20Web%20Scraping%20and%20Text%20Analysis%20in%20R.pdf">Web Scraping and Text Mining</a> workshop my colleagues Nic Crane and Beth Ashlee will be giving preceding the LondonR meetup this Tuesday the 28th of March. Also, if this blogpost has made you want to write an abstract, we are still accepting submissions for <a target="_blank" rel="nofollow" href="https://earlconf.com/abstracts/">EARL London</a> and <a target="_blank" rel="nofollow" href="https://earlconf.com/abstracts/">EARL San Fransisco</a> (I promise I won’t use it for a blogpost).</p>
<p>As always, the code for this post can be found on <a target="_blank" rel="nofollow" href="https://github.com/MangoTheCat/blog_analysis_abstracts">GitHub</a>.</p>
</div>
</div>
<p></p>
<p><br>
</p>
<p> </p>

<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.mango-solutions.com/wp/2017/03/writing-a-conference-abstract-the-data-science-way/"> Mango Solutions » R Blog</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/GEPyO5NiLwA"></a></div>