<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/a-workaround-for-when-anti-ddos-also-means-anti-data/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://rud.is/b/2017/12/10/a-workaround-for-when-anti-ddos-also-means-anti-data/"> R – rud.is</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>More sites are turning to services like Cloudflare due to just how stupid-easy it is to DDoS a site. Sometimes the DDoS is intentional (malicious). Sometimes it’s because <em>your</em> bot didn’t play nice (stop that, btw). Sadly, at some point, most of us with “vital” sites are going to have to pay protection money to one of these services unless law enforcement or ISPs do a better job stopping DDoS (killing the plethora of pwnd IoT devices that make up one of the largest for-rent DDoS services out there would be a good start).</p>
<p>Soapbox aside, sites like this one — <a rel="nofollow" href="https://www.bitmarket.pl/docs.php?file=api_public.html" target="_blank">https://www.bitmarket.pl/docs.php?file=api_public.html</a> — (which was giving an SO poster trouble) have DDoS protection enabled <em>but</em> they also want you to be able to automate the downloads (this one even calls it an “API”). However, try to grab one of the files there with your browser and you’ll likely see a Cloudflare interstitial page which eventually gets you the data.</p>
<p>Try the same thing with <code>download.file()</code> or <code>httr::GET()</code> and you’ll run into trouble since neither of those two functions have a way to perform the javascript challenge execution which ultimately is posted (well, GETted in this case) to a checker endpoint which eventually redirects to the original URL with enough <img alt="🍪" src="https://i1.wp.com/s.w.org/images/core/emoji/2.3/72x72/1f36a.png?w=456&ssl=1" class="wp-smiley" data-recalc-dims="1"><img alt="🍪" src="https://i1.wp.com/s.w.org/images/core/emoji/2.3/72x72/1f36a.png?w=456&ssl=1" class="wp-smiley" data-recalc-dims="1"><img alt="🍪" src="https://i1.wp.com/s.w.org/images/core/emoji/2.3/72x72/1f36a.png?w=456&ssl=1" class="wp-smiley" data-recalc-dims="1"> to ensure you won’t be bothered again.</p>
<p>Cloudflare has captcha and other types of interstitials, but if you happen on the 503+javascript challenge one, have I got a package for you! Meet: <a rel="nofollow" href="https://github.com/hrbrmstr/cfhttr" target="_blank"><code>cfhttr</code><img alt="🔗" src="https://i1.wp.com/s.w.org/images/core/emoji/2.3/72x72/1f517.png?w=456&ssl=1" class="wp-smiley" data-recalc-dims="1"></a>.</p>
<p>The singular function (for now) — <code>cf_GET()</code> — does the following:</p>
<ul>
<li>Makes an <code>httr::GET()</code> call with the initial URL</li>
<li>Checks to ensure it’s both on Cloudflare and is using the javascript challenge protection scheme</li>
<li>Slices the javascript and tweaks it enough to enable running it in <code>V8</code></li>
<li>Retrieves the challenge computation from <code>V8</code></li>
<li>Posts (well, <code>httr::GET()</code>s it since that’s what Cloudflare expects) the challenge form with the proper <code>Referer</code> header and hopefully passes the test so you get your content.</li>
</ul>
<pre><code>devtools::install_github(&quot;hrbrmstr/cfhttr&quot;)

library(cfhttr)

res &lt;- cf_GET(&quot;https://www.bitmarket.pl/graphs/BTCPLN/90m.json&quot;)
## Waiting 5 seconds...

str(httr::content(res, as=&quot;parsed&quot;))
## List of 90
##  $ :List of 6
##   ..$ time : int 1512908160
##   ..$ open : chr &quot;48000.00000000&quot;
##   ..$ high : chr &quot;48100.00000000&quot;
##   ..$ low  : chr &quot;48000.00000000&quot;
##   ..$ close: chr &quot;48100.00000000&quot;
##   ..$ vol  : chr &quot;0.00124821&quot;
##  $ :List of 6
##   ..$ time : int 1512908220
##   ..$ open : chr &quot;48100.00000000&quot;
##   ..$ high : chr &quot;48100.00000000&quot;
##   ..$ low  : chr &quot;48100.00000000&quot;
##   ..$ close: chr &quot;48100.00000000&quot;
##   ..$ vol  : chr &quot;0.00000000&quot;
##  $ :List of 6
##   ..$ time : int 1512908280
##   ..$ open : chr &quot;48100.00000000&quot;
##   ..$ high : chr &quot;48100.00000000&quot;
##   ..$ low  : chr &quot;48100.00000000&quot;
##   ..$ close: chr &quot;48100.00000000&quot;
##   ..$ vol  : chr &quot;0.00000000&quot;
## ...</code></pre>
<h2>FIN</h2>
<p>If you end up using this in workflows and run into a problem, it likely means that Cloudflare changed the challenge code page. Please <a rel="nofollow" href="https://github.com/hrbrmstr/cfhttr/issues" target="_blank">file an issue</a> so I can update the code.</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://rud.is/b/2017/12/10/a-workaround-for-when-anti-ddos-also-means-anti-data/"> R – rud.is</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/TMaselT3KvA" height="1"></a></div>