<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/using-httr-to-detect-https-redirects/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://petermeissner.de/blog/2018/11/07/using-httr-to-detect-redirects/"> petermeissner</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<h1>The Summary</h1>
<p>In this short note I will write about the <a rel="nofollow" href="https://cran.r-project.org/package=httr" target="_blank">httr</a> package and my need to detect<br>
whether or not an HTTP request had been redirected or not – it turns out this is quite easy. Along the way I will also show how to access information of an HTTP-conversation other than the actual content to be retrieved.</p>
<h1>The Problem</h1>
<p>I am the creator and maintainer of the <a rel="nofollow" href="https://cran.r-project.org/package=robotstxt" target="_blank">robotstxt</a> package an R package that enables users to retrieve and parse robots.txt files and ultimately is designed to do access permission checking for web resources.</p>
<p>Recently a <a rel="nofollow" href="https://github.com/ropensci/robotstxt/issues/27" target="_blank">discussion</a> came up about how to interpret permissions in case of sub-domains and HTTP redirects. Long story short: In case of robots.txt files redirects are suspicious and users should at least be informed about it happening so they might take appropriate action.</p>
<p>So, I set out to find a way to check whether or not a robots.txt files requested via the httr package has gone through one or more redirects prior to its retrieval.</p>
<p><a rel="nofollow" href="https://cran.r-project.org/package=httr" target="_blank">httr’s</a> automatic handling of redirects is one of its many wonderful features and happens silently in the background. Despite the fact that <a rel="nofollow" href="https://cran.r-project.org/package=httr" target="_blank">httr</a> hides this process it turns out that the location negotiation process is transparently logged within the return value of httr’s HTTP functions (e.g. <code>httr::GET()</code>). Furthermore it is easy to tap into that information for further processing.</p>
<h1>The Request</h1>
<p>Now let’s get our hands dirty with some real HTTP-requests done via <a rel="nofollow" href="https://cran.r-project.org/package=httr" target="_blank">httr</a>. For this we use <a rel="nofollow" href="https://httpbin.org/" target="_blank">httpbin.org</a> a service allowing to test numerous HTTP-interaction scenarios – e.g. simple HTTP GET requests with redirection.</p>
<p>When executing an HTTP GET request against the <a rel="nofollow" href="https://petermeissner.de/blog/2018/11/07/using-httr-to-detect-redirects/httpbin.org/redirect/2" target="_blank">httpbin.org/redirect/2</a> endpoint it leads to two redirects before finally providing a resource. At first glance the result and status code looks pretty normal …</p>
<div>
<div>
<pre><code><span># executing HTTP request</span><span>
</span><span>r</span><span> </span><span>&lt;-</span><span> </span><span>httr</span><span>::</span><span>GET</span><span>(</span><span>&quot;httpbin.org/redirect/2&quot;</span><span>)</span><span>
</span></code></pre>
</div>
</div>
<p><em>… the status is 200 (everything OK)…</em></p>
<div>
<div>
<pre><code><span># HTTP status code</span><span>
</span><span>r</span><span>$</span><span>status_code</span><span>
</span></code></pre>
</div>
</div>
<div>
<div>
<pre><code>## [1] 200
</code></pre>
</div>
</div>
<p><em>… and we get some content back.</em></p>
<div>
<div>
<pre><code><span># parsed content retrieved</span><span>
</span><span>httr</span><span>::</span><span>content</span><span>(</span><span>r</span><span>)</span><span>
</span></code></pre>
</div>
</div>
<div>
<div>
<pre><code>## $args
## named list()
## 
## $headers
## $headers$Accept
## [1] &quot;application/json, text/xml, application/xml, */*&quot;
## 
## $headers$`Accept-Encoding`
## [1] &quot;gzip, deflate&quot;
## 
## $headers$Connection
## [1] &quot;close&quot;
## 
## $headers$Host
## [1] &quot;httpbin.org&quot;
## 
## $headers$`User-Agent`
## [1] &quot;libcurl/7.59.0 r-curl/3.2 httr/1.3.1&quot;
## 
## 
## $origin
## [1] &quot;91.249.21.24&quot;
## 
## $url
## [1] &quot;http://httpbin.org/get&quot;
</code></pre>
</div>
</div>
<h1>The Solution</h1>
<p>So far, so good. If we look further into the response object we see that among its items there are two of particular interest: <code>headers</code> and <code>all_headers</code>. While the former only gives back headers and response meta information about the last response, the latter is a list of headers and response information for all responses.</p>
<div>
<div>
<pre><code><span># items of response object</span><span>
</span><span>names</span><span>(</span><span>r</span><span>)</span><span>
</span></code></pre>
</div>
</div>
<div>
<div>
<pre><code>##  [1] &quot;url&quot;         &quot;status_code&quot; &quot;headers&quot;     &quot;all_headers&quot; &quot;cookies&quot;    
##  [6] &quot;content&quot;     &quot;date&quot;        &quot;times&quot;       &quot;request&quot;     &quot;handle&quot;
</code></pre>
</div>
</div>
<div>
<div>
<pre><code><span># contents of all_headers item</span><span>
</span><span>names</span><span>(</span><span>r</span><span>$</span><span>all_headers</span><span>[[</span><span>1</span><span>]])</span><span>
</span></code></pre>
</div>
</div>
<div>
<div>
<pre><code>## [1] &quot;status&quot;  &quot;version&quot; &quot;headers&quot;
</code></pre>
</div>
</div>
<p>The solution to the initial problem now can be written down as neat little function which (1) extracts all status codes logged in <code>all_headers</code> and (2) checks if any of them equals some <a rel="nofollow" href="https://en.wikipedia.org/wiki/List_of_HTTP_status_codes#3xx_Redirection" target="_blank"><code>3xx</code></a> status code (3xx is mainly but not exclusively about redirection but can be considered always suspicious in the problem at hand).</p>
<div>
<div>
<pre><code><span>#' http_was_redirected</span><span>
</span><span>#'</span><span>
</span><span>#' @param response an httr response object, e.g. from a call to httr::GET()</span><span>
</span><span>#'</span><span>
</span><span>#' @return logical of length 1 indicating whether or not any redirect happened </span><span>
</span><span>#' during the HTTP request</span><span>
</span><span>#'</span><span>
</span><span>#' @export</span><span>
</span><span>#'</span><span>
</span><span>http_was_redirected</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>response</span><span>){</span><span>
</span><span># extract status </span><span>
</span><span>status</span><span> </span><span>&lt;-</span><span> </span><span>vapply</span><span>(</span><span>
</span><span>X</span><span> </span><span>=</span><span> </span><span>response</span><span>$</span><span>all_headers</span><span>,</span><span> </span><span>FUN</span><span> </span><span>=</span><span> </span><span>`[[`</span><span>,</span><span> </span><span>FUN.VALUE</span><span> </span><span>=</span><span> </span><span>integer</span><span>(</span><span>1</span><span>),</span><span>
</span><span>&quot;status&quot;</span><span>
</span><span>)</span><span>
</span><span># check status and return</span><span>
</span><span>any</span><span>(</span><span>status</span><span> </span><span>>=</span><span> </span><span>300</span><span> </span><span>&amp;</span><span> </span><span>status</span><span> </span><span></span><span> </span><span>400</span><span>)</span><span>
</span><span>}</span><span>
</span></code></pre>
</div>
</div>
<h1>The Bonus</h1>
<p>A more specific question in regard to redirects is whether or not a redirect<br>
did not only change the path but also entailed a domain change – robots.txt conventions clearly state that each domain and subdomain have to provide their own robots.txt files.</p>
<p>The following function makes use of the <code>all_headers</code> item from the response object again. In addition it uses the <code>domain()</code> function provided by the <a rel="nofollow" href="https://cran.r-project.org/package=urltools" target="_blank">urltools</a> package to extract the domain part of an <a rel="nofollow" href="https://en.wikipedia.org/wiki/URL" target="_blank">URL</a>. If any location header shows a domain not equal to that of the original URL requested a domain change must have happened along the way.</p>
<div>
<div>
<pre><code><span>#' http_domain_changed</span><span>
</span><span>#'</span><span>
</span><span>#' @param response an httr response object, e.g. from a call to httr::GET()</span><span>
</span><span>#'</span><span>
</span><span>#' @return logical of length 1 indicating whether or not any domain change </span><span>
</span><span>#' happened during the HTTP request</span><span>
</span><span>#'</span><span>
</span><span>#' @export</span><span>
</span><span>#'</span><span>
</span><span>http_domain_changed</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>response</span><span>){</span><span>
</span><span># get domain of origignal HTTP request</span><span>
</span><span>orig_domain</span><span> </span><span>&lt;-</span><span> </span><span>urltools</span><span>::</span><span>domain</span><span>(</span><span>response</span><span>$</span><span>request</span><span>$</span><span>url</span><span>)</span><span>
</span><span># extract location headers</span><span>
</span><span>location</span><span> </span><span>&lt;-</span><span> </span><span>unlist</span><span>(</span><span>
</span><span>lapply</span><span>(</span><span>
</span><span>X</span><span> </span><span>=</span><span> </span><span>response</span><span>$</span><span>all_headers</span><span>,</span><span> </span><span>FUN</span><span> </span><span>=</span><span> </span><span>function</span><span>(</span><span>x</span><span>){</span><span>
</span><span>x</span><span>$</span><span>headers</span><span>$</span><span>location</span><span>
</span><span>}</span><span>
</span><span>)</span><span>
</span><span>)</span><span>
</span><span># new domains</span><span>
</span><span>new_domains</span><span> </span><span>&lt;-</span><span> </span><span>urltools</span><span>::</span><span>domain</span><span>(</span><span>location</span><span>)</span><span>
</span><span># check domains in location against original domain</span><span>
</span><span>any</span><span>(</span><span> </span><span>!</span><span>is.na</span><span>(</span><span>new_domains</span><span>)</span><span> </span><span>&amp;</span><span> </span><span>new_domains</span><span> </span><span>!=</span><span> </span><span>orig_domain</span><span> </span><span>)</span><span>
</span><span>}</span><span>
</span></code></pre>
</div>
</div>
<h1>Some Resources</h1>
<ul>
<li><a rel="nofollow" href="https://cran.r-project.org/package=httr" target="_blank">https://cran.r-project.org/package=httr</a></li>
<li><a rel="nofollow" href="https://cran.r-project.org/package=robotstxt" target="_blank">https://cran.r-project.org/package=robotstxt</a></li>
<li><a rel="nofollow" href="https://cran.r-project.org/package=urltools" target="_blank">https://cran.r-project.org/package=urltools</a></li>
<li><a rel="nofollow" href="https://developers.google.com/search/reference/robots_txt?hl=en" target="_blank">https://developers.google.com/search/reference/robots_txt?hl=en</a></li>
<li><a rel="nofollow" href="https://en.wikipedia.org/wiki/Robots_exclusion_standard" target="_blank">https://en.wikipedia.org/wiki/Robots_exclusion_standard</a></li>
<li><a rel="nofollow" href="https://www.webnots.com/what-is-http/" target="_blank">https://www.webnots.com/what-is-http/</a></li>
<li><a rel="nofollow" href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Messages" target="_blank">https://developer.mozilla.org/en-US/docs/Web/HTTP/Messages</a></li>
<li><a rel="nofollow" href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Redirections" target="_blank">https://developer.mozilla.org/en-US/docs/Web/HTTP/Redirections</a></li>
</ul>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://petermeissner.de/blog/2018/11/07/using-httr-to-detect-redirects/"> petermeissner</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/wxKlUc3x6B8" height="1"></a></div>