<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/what-makes-an-r-talk-popular-scraping-user2017-attendance-information-to-find-out/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://deanattali.com/blog/user2017/"> Dean Attali's R Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<div>
<a rel="nofollow" href="https://daattali.com/shiny/user2017/" target="_blank">Click here to explore the data for yourself</a>
</div>
<p>First off — I’ll admit that was my poor attempt at a click-bait title. But if you’re still reading the next paragraph, that means it was successful!</p>
<h1>Table of contents</h1>
<ul>
<li><a rel="nofollow" href="http://deanattali.com/blog/user2017/#background" target="_blank">Background</a></li>
<li><a rel="nofollow" href="http://deanattali.com/blog/user2017/#number1" target="_blank">Am I number 1?!</a></li>
<li><a rel="nofollow" href="http://deanattali.com/blog/user2017/#scrapeindividual" target="_blank">Scraping individual event information from the useR Sched website</a></li>
<li><a rel="nofollow" href="http://deanattali.com/blog/user2017/#scrapeall" target="_blank">Scrape all events data into one dataset</a></li>
<li><a rel="nofollow" href="http://deanattali.com/blog/user2017/#bigmoment" target="_blank">The moment we all (or at least I) waited for!</a></li>
<li><a rel="nofollow" href="http://deanattali.com/blog/user2017/#normalize" target="_blank">Normalizing for parallel sessions</a></li>
<li><a rel="nofollow" href="http://deanattali.com/blog/user2017/#visualize" target="_blank">Visualizing the data and making observations</a>
<ul>
<li><a rel="nofollow" href="http://deanattali.com/blog/user2017/#type_v_attendance" target="_blank">Want attendnace? Don’t sponsor the conference, try to get invited!</a></li>
<li><a rel="nofollow" href="http://deanattali.com/blog/user2017/#time_v_attendance" target="_blank">Does attendance drop off with time?</a></li>
<li><a rel="nofollow" href="http://deanattali.com/blog/user2017/#type_attendance_ratio" target="_blank">Quick look at attendance ratios</a></li>
<li><a rel="nofollow" href="http://deanattali.com/blog/user2017/#top_rooms" target="_blank">Are some rooms better than others?</a></li>
<li><a rel="nofollow" href="http://deanattali.com/blog/user2017/#correlation" target="_blank">Correlation obviously implies causation</a></li>
</ul>
</li>
<li><a rel="nofollow" href="http://deanattali.com/blog/user2017/#improvements" target="_blank">Improvements</a></li>
<li><a rel="nofollow" href="http://deanattali.com/blog/user2017/#explore" target="_blank">Explore or reproduce the data yourself</a></li>
</ul>
<h2>Background</h2>
<p>Last week I attended the <a rel="nofollow" href="https://user2017.brussels/" target="_blank">useR 2017 conference</a> in Belgium<a rel="nofollow" href="http://deanattali.com/blog/user2017/#footnote" target="_blank">*</a> — where 1,161 R users from all around the world gathered for 4 days to live and breath R. I still consider myself somewhat new to R (less than 4 years), so I still get extremely excited to go to these conferences! So excited, in fact, that I was fine parting with over $2,000 out of my own pocket to attend…</p>
<p>A few weeks before the conference, all attendees received an email with an invitation to a <a rel="nofollow" href="https://user2017.sched.com/" target="_blank">scheduling app/website</a> called Sched. We could use this website to browse through all the events at the conference – keynote talks, lightning talks, regular talks, tutorials, coffee breaks, lunch breaks, and anything else that was in the schedule. We could also mark which events we planned to attend. There was incentive for us to select the talks we planned on attending, because there were often several talks in parallel, and Sched could show me my personalized schedule so that I’d know where to go for the talks I’m interested in.</p>
<p><a rel="nofollow" href="https://i2.wp.com/deanattali.com/img/blog/user2017/sched.PNG" target="_blank"><img alt="Sched screenshot" src="https://i2.wp.com/deanattali.com/img/blog/user2017/sched.PNG?w=456" data-recalc-dims="1"></a></p>
<blockquote>
<p>Screenshot of the Sched website. Checkmarks indicate events that I marked as attending.</p>
</blockquote>
<p>If you click on any event, you are taken to a new page that shows a more detailed view of that event. It will list all sorts of metadata for the event, such as its location, start time, end time, description, and name of speaker when applicable. Each event also keeps track of how many people added it to their own schedules – which was the interesting piece of information for me. This essentially acts as a popularity gauge, by showing how many attendees are interested in each event.</p>
<p><a rel="nofollow" href="https://i0.wp.com/deanattali.com/img/blog/user2017/scheddetails.PNG" target="_blank"><img alt="Sched detailed screenshot" src="https://i0.wp.com/deanattali.com/img/blog/user2017/scheddetails.PNG?w=456" data-recalc-dims="1"></a></p>
<blockquote>
<p>Detailed view of an event</p>
</blockquote>
<h2>Am I number 1?!</h2>
<p>As I was prepping for my lightning talk, I wanted to see how many people said they’d attend my talk (even though I realize the number of RSVP is a very bad estimate for the real attendance). So I went to the <a rel="nofollow" href="https://user2017.sched.com/event/AxrJ/plot-colour-helper-finally-an-easy-way-to-pick-colours-for-your-r-plots" target="_blank">event page for my talk</a> and saw that it had 159 attendees.</p>
<p><em>Great!</em></p>
<p>But I had no idea how that compares with other lightning talks… I <em>obviously</em> had to investigate the matter further to see where I rank, so I opened the pages of another ~20 lightning talks and checked their attendance numbers. It seemed like I had more attendees than all the talks I checked. Then I thought <em>“why not do this properly? Instead of spending 10 minutes manually opening all the lightning talk pages and looking at that number, I could spend 10 times longer doing it automatically with R!”</em></p>
<p>(By the way, <a rel="nofollow" href="http://deanattali.com/colourpicker-user2017/" target="_blank">you can watch my talk here</a> if you’re interested – it’s about a tool I built for making it easier to select colours for your plots.)</p>
<h2>Scraping individual event information from the useR Sched website</h2>
<p>For anyone who’s not familiar with the term, <strong>scraping</strong> a webpage means to programmatically extract information from it. What I was manually doing – scanning the main Sched page for any lightning talks, clicking on them, recording the number of attendees from each page – that can be automated, and this process would be called scraping.</p>
<p>Before useR started, I already had the idea that I wanted to scrape the useR schedule website to find out how my talk compared to others. But I haven’t found the time to do it. While at useR, I met <a rel="nofollow" href="https://twitter.com/daroczig" target="_blank">Gergely Daróczi</a>, who runs an R meetup group in Budapest, Hungary, and he invited me to give a talk at their next meetup which would be about highlights from useR. I decided this was a good reason for me to actually go through with my mini project, and told him I would scrape the schedule and present any interesting findings.</p>
<p>Next week, while on my Wi-Fi-equipped train to Budapest, I had a few hours to kill so I decided it was a good time to work on this. The best way to scrape the web with R is using the <a rel="nofollow" href="https://cran.r-project.org/package=rvest" target="_blank"><code>rvest</code></a> package, and it is very easy to do so.</p>
<p>I noticed that every event page had the same URL pattern of <code>https://user2017.sched.com/event/</code>