<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/cant-stop-at-21-twitter-recipe-22-tying-up-loose-threads/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://rud.is/b/2018/01/15/cant-stop-at-21-twitter-recipe-22-tying-up-loose-threads/"> R – rud.is</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>NOTE: The likelihood of this recipe being added to the recent <a rel="nofollow" href="http://rud.is/books/21-recipes" target="_blank">practice <code>bookdown</code> book</a> is slim, but I’ll try to keep the same format for the blog post.</p>
<h3>Problem</h3>
<p>You want to collect all the tweets in a Twitter tweet thread</p>
<h3>Solution</h3>
<p>Use a few key functions in <code>rtweet</code> to piece the thread elements back together.</p>
<h3>Discussion</h3>
<p>In Twitterland, a “thread” is a series of tweets by an author that are in a reply chain to each other which enables them to be displayed sequentially to form a larger &amp; (ostensibly) more cohesive message. Even with the recent 280 character tweet-length increase, threads are still popular and used daily. They’re very easy to distinguish on Twitter but there is no Twitter API call to collect up all the pieces of these threads. </p>
<p>Let’s build a function — <code>get_thread()</code> — that will take as input a starting thread URL or status id and return a data frame of all the tweets in the thread (in order). As a bonus, we’ll also include a way to include all first-level retweets and replies to each threaded tweet (that, too, happens quite a bit).</p>
<p>There are documentation snippets in the code block (below), but the essence of the function is:</p>
<ul>
<li>first, finding the tweet that belongs to the status id to get some metadata</li>
<li>then doing a search for tweets from the author that occurred after that tweet (we do this to save on API calls and we grab a bunch of them)</li>
<li>rather than do a bunch of things by hand, we make from/to pairs to feed in as vertex edges into <code>igraph</code></li>
<li>once that’s done, separate out the graph into unique subgraphs and find the one containing the starting status id</li>
<li>since that subgraph is just a set of status ids, rebuild the data frame from it and put it in order.</li>
</ul>
<p>There may be occasions where we want to grab the replies or RTs of any of the original thread tweets. They aren’t always useful, but when they are it’d be good to have this context. So, we’ll add an option that — if <code>TRUE</code> — will cause the function to go down the list of threaded tweets and pull the first-level replies and RTs (excluding the ones from the author). We’ll do this using the Twitter search API as it’ll ultimately save on API calls and it puts the filtering closer to the data (I’m generally “a fan” of putting computation as close to the data as possible for any given task). If there were any, they’ll be in a <code>replies</code> column which can be unnested at-will.</p>
<p>Here’s the complete function:</p>
<pre><code>get_thread &lt;- function(first_status, include_replies=FALSE, .timeline_history=3000) {

  require(rtweet, quietly=TRUE, warn.conflicts=FALSE)
  require(igraph, quietly=TRUE, warn.conflicts=FALSE)
  require(tidyverse, quietly=TRUE, warn.conflicts=FALSE)

  first_status &lt;- if (str_detect(first_status[1], &quot;^http[s]://&quot;)) basename(first_status[1]) else first_status[1]

  # get first status
  orig &lt;- rtweet::lookup_tweets(first_status)

  # grab the author's timeline to search
  author_timeline &lt;- rtweet::get_timeline(orig$screen_name, n=.timeline_history, since_id=first_status)

  # build a data frame containing from/to pairs (anything the author
  # replied to) that also includes the `first_status` id.
  suppressWarnings(
    dplyr::filter(
      author_timeline,
      (status_id == first_status) | (reply_to_screen_name == orig$screen_name)
    ) %>%
      dplyr::select(status_id, reply_to_status_id) %>%
      igraph::graph_from_data_frame() -> g
  ) # build a graph from this

  # decompose the graph into unique subgraphs and return them to data frames
  igraph::decompose(g) %>%
    purrr::map(igraph::as_data_frame) -> threads_dfs

  # find the thread with our `first_status` ids

  thread_df &lt;- purrr::keep(threads_dfs, ~any(which(unique(unlist(.x, use.names=FALSE)) == first_status)))

  # BONUS: we get them in the order we need!
  thread_order &lt;- purrr::discard(rev(unique(unlist(thread_df))), str_detect, &quot;NA&quot;)

  # filter out the thread from the timeline corpus &amp; sort it
  dplyr::filter(author_timeline, status_id %in% pull(thread_df[[1]], from)) %>%
    dplyr::mutate(status_id = factor(status_id, levels=thread_order)) %>%
    dplyr::arrange(status_id) -> tweet_thread

  if (include_replies) {
    # for each status, lookup 1st-level references to it, excluding ones from the original author
    mutate(
      tweet_thread,
      replies = purrr::map(
        as.character(status_id),
        ~rtweet::search_tweets(sprintf(&quot;%s -from:%s&quot;, .x, orig$screen_name[1]))
      )
    ) -> tweet_thread
  }

  class(tweet_thread) &lt;-  c(&quot;tweet_thread&quot;, class(tweet_thread))

  return(tweet_thread)

}</code></pre>
<p>Now, if we grab <a rel="nofollow" href="https://twitter.com/petersagal/status/952910499825451009" target="_blank">this thread</a>, the function will return the following:</p>
<pre><code>xdf &lt;- get_thread(&quot;https://twitter.com/petersagal/status/952910499825451009&quot;)

glimpse(select(xdf, 1:5))
## Observations: 10
## Variables: 5
## $ status_id   </code></pre>