<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/scrapping-data-about-australian-politicians-with-rselenium/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://levashov.biz/scrapping-data-about-australian-politicians-with-rselenium/"> R – Alex Levashov – eCommerce Consultant (Melbourne, Australia)</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>While there is more and more data available in structured formats (CSV, JSON) through initiatives like OpenData, sometimes nicely formatted data still not publicly available.</p>
<p>When I decided to conduct a little study of what Australian politicians from the major party post in Twitter. So I decide to find list of Twitter accounts of the Members of Parliaments with the references to what party they belong. Unfortunately my search for this data in any structured format wasn’t successful (if you know where to get it – you are welcome to comment).</p>
<p>The information I needed can be taken from <a rel="nofollow" href="https://www.aph.gov.au/Senators_and_Members/Parliamentarian_Search_Results?q=&mem=1&par=-1&gen=0&ps=96" target="_blank">Parliament of Australia</a> website. The issue is that it is presented as HTML pages like one below:</p>
<p><img sizes="(max-width: 899px) 100vw, 899px" src="https://i1.wp.com/levashov.biz/wp-content/uploads/2018/11/APH-screenshot.png?w=450&ssl=1" alt="APH website screenshot" class="wp-image-814" data-wp-pid="814" data-recalc-dims="1"></p>
<p>It contains all the information I needed. Let’s see how it can be extracted using RSelenium package.</p>
<p>First we need to install the package and load it.</p>
<p><code>install.packages(&quot;RSelenium&quot;)<br>
library(RSelenium)</code></p>
<p>Also make an empty dataframe for the data I want to collect</p>
<p><code><br>
mps &lt;- data.frame(matrix(ncol = 4, nrow = 0))<br>
colnames(mps)&lt;- c('name', 'electorate', 'party', 'twitter')<br>
</code></p>
<p>Next thing we need to do is to initialize Selenium driver. Selenium is different from other R packages used to scrap web. It actually loads web pages into real browser and allows you to work with that pages. Selenuim use is much wider than data scrapping: for example it is a very popular tool for automated testing of websites.</p>
<p>The code below initializes Google Chrome driver on local machine and open browser session controlled in our case from R.</p>
<p><code><br>
rD &lt;- rsDriver()<br>
remDr &lt;- remoteDriver(port = 4567L, browserName = &quot;chrome&quot;)<br>
remDr$open()</code></p>
<p>This is the only way I could run it on Windows 10 machine without use of Docker containers and other virtualization mechanisms. Firefox didn’t work for me. I admit that haven’t tried Safari and IE/Edge.</p>
<p>You may also run RSelenium sessions from remote machine including such services as BrowserStack.</p>
<p>Next prepare a function for scrapping the data about individual MP – name, electorate, party, twitter handle.</p>
<p><code><br>
onempdata &lt;- function (webElems, ref) {<br>
one &lt;- data.frame(matrix(ncol = 4, nrow = 1))<br>
colnames(one)&lt;- c('name', 'electorate', 'party', 'twitter')<br>
name &lt;- webElems[[ref]]$findChildElement(using='class', value='title')<br>
one$name &lt;- as.character(name$getElementText())<br>
electorate &lt;- webElems[[ref]]$findChildElement(using='css selector', value='dd:nth-child(2)')<br>
one$electorate &lt;- as.character(electorate$getElementText())<br>
## party data is badly structured, will get it another way<br>
### getting twitter, since not all MPs have it, we need to catch errors to avoid stop<br>
twitter &lt;- tryCatch({<br>
suppressMessages({<br>
webElems[[ref]]$findChildElement(using = 'css selector', value = '.fa-twitter')<br>
})<br>
},<br>
error = function(e) {<br>
NA_character_<br>
}<br>
)<br>
# only collect Twitter handle if it exists<br>
if (class(twitter)!='character'){<br>
twitter$clickElement()<br>
Sys.sleep(4)<br>
windows &lt;- remDr$getWindowHandles() #get list of open windows<br>
remDr$switchToWindow(windows[[2]]) # switch to Twitter window<br>
tt &lt;- as.character(remDr$getCurrentUrl())# collect URL<br>
remDr$closeWindow() # Close Twitter window<br>
remDr$switchToWindow(windows[[1]]) #switch back to main window<br>
one$twitter &lt;- as.character(tt)<br>
}<br>
else one$twitter &lt;- NA_character_ #if no Twitter return NA<br>
# return one row dataframe with all data about one MP<br>
return(one)<br>
}</code></p>
<p>Several notes about the code above:</p>
<ul>
<li>remDr$findElements is the main function to search web page content. The search can be done using xpath, css selector, classes. Variation of the function called like webElems[[ref]]$findChildElement look for the child objects within webElems[ref] object. There is a nice Chrome plugin to find web page elements xpath or class called <a rel="nofollow" href="https://selectorgadget.com/" target="_blank">SelectorGadget</a></li>
<li>To extract text <code>getElementText() </code>&lt; function that can be applied to webElement class of objects used</li>
<li>We need to process exceptions to avoid function stopping when no Twitter handle available (yes, not all MPs have it)</li>
<li>I was unable to find a way to extract Twitter handle directly from the web page. Maybe it is possible, but I haven’t found a way to get it. So I found a work around. R code controlled browser user actually visits Twitter accounts if they exists and we capture that page URL to extract Twitter handle later</li>
</ul>
<p>Next we define a function to get data about all MPs who are member of particular party. We use here APH website feature that allows to filter MPs by party, so each party has specific URL like <a rel="nofollow" href="https://www.aph.gov.au/Senators_and_Members/Parliamentarian_Search_Results?q=&mem=1&par=1&gen=0&ps=96" target="_blank">this one</a> displays only Labor MPs.<br>
<code><br>
collectPartyData &lt;- function (url, party){<br>
remDr$navigate(url)<br>
# empty dataframe to collect one party data and return as a function result<br>
fmps &lt;- data.frame(matrix(ncol = 4, nrow = 0))<br>
colnames(fmps)&lt;- c('name', 'electorate', 'party', 'twitter')<br>
webElems1 &lt;- remDr$findElements(using = 'xpath', value = '//*[contains(concat( &quot; &quot;, @class, &quot; &quot; ), concat( &quot; &quot;, &quot;padding-top&quot;, &quot; &quot; ))]')<br>
for (i in seq_along(webElems1)){<br>
one &lt;- data.frame(matrix(ncol = 4, nrow = 1))<br>
colnames(one)&lt;- c('name', 'electorate', 'party', 'twitter')<br>
one &lt;- onempdata(webElems1, i)<br>
one$party &lt;- party<br>
fmps &lt;- rbind(fmps, one)<br>
}<br>
return (fmps)<br>
}<br>
</code></p>
<p>Now it is time to run the data collection using the functions defined.<br>
<code>## run with Labour MPS<br>
url &lt;- &quot;https://www.aph.gov.au/Senators_and_Members/Parliamentarian_Search_Results?q=&amp;mem=1&amp;par=1&amp;gen=0&amp;ps=96&quot;<br>
mps &lt;- rbind(mps,collectPartyData(url, &quot;Australian Labor Party&quot;))<br>
## run with Liberal MPS<br>
url &lt;- &quot;https://www.aph.gov.au/Senators_and_Members/Parliamentarian_Search_Results?q=&amp;mem=1&amp;par=2&amp;gen=0&amp;ps=96&quot;<br>
mps &lt;- rbind(mps,collectPartyData(url, &quot;Liberal Party of Australia&quot;))<br>
## run with Nationals<br>
url &lt;- &quot;https://www.aph.gov.au/Senators_and_Members/Parliamentarian_Search_Results?q=&amp;mem=1&amp;par=3&amp;gen=0&amp;ps=96&quot;<br>
mps &lt;- rbind(mps,collectPartyData(url, &quot;The Nationals&quot;))<br>
## run with Greens<br>
url &lt;- &quot;https://www.aph.gov.au/Senators_and_Members/Parliamentarian_Search_Results?q=&amp;mem=1&amp;par=9&amp;gen=0&amp;ps=96&quot;<br>
mps &lt;- rbind(mps,collectPartyData(url, &quot;Australian Greens&quot;))<br>
## run with independent<br>
url &lt;- &quot;https://www.aph.gov.au/Senators_and_Members/Parliamentarian_Search_Results?q=&amp;mem=1&amp;par=4&amp;gen=0&amp;ps=96&quot;<br>
mps &lt;- rbind(mps,collectPartyData(url, &quot;Independent&quot;))<br>
## run with Centre Alliance<br>
url &lt;- &quot;https://www.aph.gov.au/Senators_and_Members/Parliamentarian_Search_Results?q=&amp;mem=1&amp;par=25&amp;gen=0&amp;ps=96&quot;<br>
mps &lt;- rbind(mps,collectPartyData(url, &quot;Centre Alliance&quot;))<br>
### run with Katter party<br>
url &lt;- &quot;https://www.aph.gov.au/Senators_and_Members/Parliamentarian_Search_Results?q=&amp;mem=1&amp;par=15&amp;gen=0&amp;ps=96&quot;<br>
mps &lt;- rbind(mps,collectPartyData(url, &quot;Katter's Australian Party&quot;))<br>
### closing webdriver session<br>
remDr$close() </code><br>
Here it is, we now have a nice data frame with all MPs with their parties, electorate and Twitter account (if they have it).<br>
We can apply a quick formatting to leave just Twitter handle/account name part of URL, it is better for future use and save the results in CSV file.</p>
<p><code>mps$twitter &lt;- gsub(&quot;https://twitter.com/&quot;,&quot;&quot;, mps$twitter)<br>
write.csv(mps, &quot;mps.csv&quot;)</code></p>
<p>That’s it, with help of RSelenium we have collected the data from the website and can use it for further analysis, which will be part of my future posts.<br>
If you don’t really keen to explore data collection and just need the resulting CSV file you can take it from <a rel="nofollow" href="https://github.com/alevashov/aupolitics/blob/master/mps.csv" target="_blank">Github</a>.</p>
<p>Full code is also available in my <a rel="nofollow" href="https://github.com/alevashov/aupolitics" target="_blank">Github repository</a>, the file related with this post is <a rel="nofollow" href="https://github.com/alevashov/aupolitics/blob/master/data-collection.R" target="_blank">date-collection.R</a></p>
<p><a rel="nofollow" href="https://www.addtoany.com/add_to/facebook?linkurl=https%3A%2F%2Flevashov.biz%2Fscrapping-data-about-australian-politicians-with-rselenium%2F&linkname=Scrapping%20data%20about%20Australian%20politicians%20with%20RSelenium" title="Facebook" target="_blank"></a><a rel="nofollow" href="https://www.addtoany.com/add_to/twitter?linkurl=https%3A%2F%2Flevashov.biz%2Fscrapping-data-about-australian-politicians-with-rselenium%2F&linkname=Scrapping%20data%20about%20Australian%20politicians%20with%20RSelenium" title="Twitter" target="_blank"></a><a rel="nofollow" href="https://www.addtoany.com/add_to/google_plus?linkurl=https%3A%2F%2Flevashov.biz%2Fscrapping-data-about-australian-politicians-with-rselenium%2F&linkname=Scrapping%20data%20about%20Australian%20politicians%20with%20RSelenium" title="Google+" target="_blank"></a><a rel="nofollow" href="https://www.addtoany.com/add_to/linkedin?linkurl=https%3A%2F%2Flevashov.biz%2Fscrapping-data-about-australian-politicians-with-rselenium%2F&linkname=Scrapping%20data%20about%20Australian%20politicians%20with%20RSelenium" title="LinkedIn" target="_blank"></a><a rel="nofollow" href="https://www.addtoany.com/add_to/reddit?linkurl=https%3A%2F%2Flevashov.biz%2Fscrapping-data-about-australian-politicians-with-rselenium%2F&linkname=Scrapping%20data%20about%20Australian%20politicians%20with%20RSelenium" title="Reddit" target="_blank"></a><a data-a2a-title="Scrapping data about Australian politicians with RSelenium" rel="nofollow" href="https://www.addtoany.com/share#url=https%3A%2F%2Flevashov.biz%2Fscrapping-data-about-australian-politicians-with-rselenium%2F&amp;%23038;title=Scrapping%20data%20about%20Australian%20politicians%20with%20RSelenium" data-a2a-url="https://levashov.biz/scrapping-data-about-australian-politicians-with-rselenium/" target="_blank"></a></p>
<p>The post <a rel="nofollow" href="https://levashov.biz/scrapping-data-about-australian-politicians-with-rselenium/" target="_blank">Scrapping data about Australian politicians with RSelenium</a> appeared first on <a rel="nofollow" href="https://levashov.biz/" target="_blank">Alex Levashov – eCommerce Consultant (Melbourne, Australia)</a>.</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://levashov.biz/scrapping-data-about-australian-politicians-with-rselenium/"> R – Alex Levashov – eCommerce Consultant (Melbourne, Australia)</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/b5BOqt2J_oQ" height="1"></a></div>