<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/controlling-expenses-on-ali-express-with-rselenium/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://r-addict.com/2017/01/08/RSelenium-at-TriCity-and-AliExpress.html"> http://r-addict.com</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><img src="https://i2.wp.com/r-addict.com/images/fulls/rsel.png?w=456" data-recalc-dims="1"> Due to the incrising interest in the Internet and due to the its rising number of users, one can notice the surprising growth in the demand for analyzing data and information in the Internet that were left by users and for users. Many companies and institutions base their business decisions on the extensive research of social media portals and Internet forums, where users leave reviews on various products and brands. <strong>Not only the same analysis, but also the ability to obtain data from the Internet, is a key part of the puzzle…</strong></p>
<p>… one can read in the description of the second talk (by Krzysztof Słomczynski (<a target="_blank" rel="nofollow" href="https://github.com/krzyslom">krzyslom</a>)) of the first meetup of the new Tri-City R Users Group – <a target="_blank" rel="nofollow" href="https://www.meetup.com/Trojmiejska-Grupa-Entuzjastow-R/events/236257032/">meet(R) in Tricity!</a>. The meeting will be held on this Thursday (12-01-2017) at the Gdańsk University of Technology.</p>
<ul>
<li><a target="_blank" rel="nofollow" href="http://r-addict.com/2017/01/08/RSelenium-at-TriCity-and-AliExpress.html#rselenium">RSelenium</a></li>
<li><a target="_blank" rel="nofollow" href="http://r-addict.com/2017/01/08/RSelenium-at-TriCity-and-AliExpress.html#launching-rselenium">Launching RSelenium</a></li>
<li><a target="_blank" rel="nofollow" href="http://r-addict.com/2017/01/08/RSelenium-at-TriCity-and-AliExpress.html#logging-to-ali-express">Logging to Ali Express</a></li>
<li><a target="_blank" rel="nofollow" href="http://r-addict.com/2017/01/08/RSelenium-at-TriCity-and-AliExpress.html#few-basic-rselenium-functions">Few basic RSelenium functions</a></li>
<li><a target="_blank" rel="nofollow" href="http://r-addict.com/2017/01/08/RSelenium-at-TriCity-and-AliExpress.html#extracting-information-from-ali-express-allowed-only-for-logged-user">Extracting information from Ali Express allowed only for logged user</a></li>
<li><a target="_blank" rel="nofollow" href="http://r-addict.com/2017/01/08/RSelenium-at-TriCity-and-AliExpress.html#plotting-expenses">Plotting expenses</a></li>
</ul>
<h1>RSelenium</h1>
<p>Motivated by that upcoming talk I took a tour through <a target="_blank" rel="nofollow" href="https://cran.r-project.org/web/packages/RSelenium/index.html">RSelenium</a> vignettes: <a target="_blank" rel="nofollow" href="https://cran.r-project.org/web/packages/RSelenium/vignettes/RSelenium-basics.html">RSelenium basics</a> and <a target="_blank" rel="nofollow" href="https://cran.r-project.org/web/packages/RSelenium/vignettes/RSelenium-docker.html">RSelenium Docker</a> to launch my first Docker container with Selenium Server. If you are not yet motivated to use Docker containers, then have a look at this post <a target="_blank" rel="nofollow" href="http://r-addict.com/2016/05/13/Docker-Motivation.html">R 3.3.0 is another motivation for Docker</a>.</p>
<p>RSelenium is an R interface that connects to <a target="_blank" rel="nofollow" href="http://docs.seleniumhq.org/">Selenium</a> (Server), which is a project focused on automating web browsers and enables to create a regular web browser session that can be controlled with command lines. Such Internet browsing automatization is a huge trigger for web-harvesting because with RSelenium you are able to simulate a real user and to pass keys to the browser session (such as user login and password). With such a possibility you are able to log into any portal automatically (or manually) and to fill bot security captchas code (this rather manually). You can also interact with web elements that first need to be clicked to show information which are in the demand to be web-scraped.</p>
<h1>Launching RSelenium</h1>
<p>I used <a target="_blank" rel="nofollow" href="https://hub.docker.com/r/selenium/">Selenium Docker container</a> to launch Selenium Server which is available on Docker hub. The following command launches Selenium Server and binds it to the localhost on the port 4445. It also binds the remote desktop on the port 5901 so that we can run VNC viewer (Vinagre) to observe operations performed on a fake web session.</p>
<figure>
<pre><code data-lang="r"><span>docker</span><span> </span><span>run</span><span> </span><span>-</span><span>d</span><span> </span><span>-</span><span>p</span><span> </span><span>5901</span><span>:</span><span>5900</span><span> </span><span>-</span><span>p</span><span> </span><span>127.0.0.1</span><span>:</span><span>4445</span><span>:</span><span>4444</span><span> </span><span>selenium</span><span>/</span><span>standalone</span><span>-</span><span>firefox</span><span>-</span><span>debug</span><span>:</span><span>2.53.1</span><span>
</span><span>#</span><span> </span><span>then</span><span> </span><span>open</span><span> </span><span>vinagre</span><span> </span><span>at</span><span> </span><span>port</span><span> </span><span>127.0.0.1</span><span>:</span><span>5901</span><span> </span><span>for</span><span> </span><span>VCN</span><span> </span><span>method</span></code></pre>
</figure>
<p>When Docker container is running, we are able to establish a connection with Selenium Server from R with the following commands</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>RSelenium</span><span>)</span><span>
</span><span>remDr</span><span> </span><span>&lt;-</span><span> </span><span>remoteDriver</span><span>(</span><span>port</span><span> </span><span>=</span><span> </span><span>4445L</span><span>)</span><span>
</span><span>remDr</span><span>$</span><span>open</span><span>(</span><span>silent</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span></code></pre>
</figure>
<h1>Logging to Ali Express</h1>
<p>The result of previous commands is a <code>remoteDriver</code> object, in this situation called <code>remDr</code>. This S4 object can be used to navigate through the web browser session. The below command navigates to the Ali Express logging page.</p>
<figure>
<pre><code data-lang="r"><span>remDr</span><span>$</span><span>navigate</span><span>(</span><span>&quot;https://login.aliexpress.com/&quot;</span><span>)</span></code></pre>
</figure>
<p>In this situation you can see than in the remote desktop VNC viewer we have entered Ali Express.</p>
<p><img src="https://i0.wp.com/r-addict.com/images/fulls/ali55.png?w=456" data-recalc-dims="1"></p>
<p>You can log in in 2 ways: by filling fields manually or by finding fields by their properties (like <code>ir</code> or <code>class</code> name) and by sending keys (like <code>password</code> and <code>user name</code>) to them.</p>
<figure>
<pre><code data-lang="r"><span>remDr</span><span>$</span><span>findElement</span><span>(</span><span>&quot;id&quot;</span><span>,</span><span> </span><span>&quot;fm-login-id&quot;</span><span>)</span><span>$</span><span>sendKeysToElement</span><span>(</span><span>list</span><span>(</span><span>login</span><span>))</span><span>
</span><span>remDr</span><span>$</span><span>findElement</span><span>(</span><span>&quot;id&quot;</span><span>,</span><span> </span><span>&quot;fm-login-password&quot;</span><span>)</span><span>$</span><span>sendKeysToElement</span><span>(</span><span>list</span><span>(</span><span>password</span><span>))</span></code></pre>
</figure>
<p>Then you can click on the <code>sign in</code> button with</p>
<figure>
<pre><code data-lang="r"><span>remDr</span><span>$</span><span>findElement</span><span>(</span><span>&quot;id&quot;</span><span>,</span><span> </span><span>&quot;fm-login-submit&quot;</span><span>)</span><span>$</span><span>clickElement</span><span>()</span></code></pre>
</figure>
<p><img src="https://i2.wp.com/r-addict.com/images/fulls/ali_automatic_login.png?w=456" data-recalc-dims="1"> You can check what are properties of a field with <code>inspect element</code> in any web browser, so that you will know how to navigate to the element with <code>findElement</code> by it’s <code>id</code> or by it’s <code>class</code> or even <code>css</code>. For this portal I wasn’t successful with sending keys so I logged manually.</p>
<p>However, for Facebook it worked like a charm</p>
<figure>
<pre><code data-lang="r"><span>remDr</span><span>$</span><span>navigate</span><span>(</span><span>&quot;https://www.facebook.com/&quot;</span><span>)</span><span> </span><span>remDr</span><span>$</span><span>findElement</span><span>(</span><span>&quot;id&quot;</span><span>,</span><span> </span><span>&quot;email&quot;</span><span>)</span><span>$</span><span>sendKeysToElement</span><span>(</span><span>list</span><span>(</span><span>login</span><span>))</span><span>
</span><span>remDr</span><span>$</span><span>findElement</span><span>(</span><span>&quot;id&quot;</span><span>,</span><span> </span><span>&quot;pass&quot;</span><span>)</span><span>$</span><span>sendKeysToElement</span><span>(</span><span>list</span><span>(</span><span>password</span><span>))</span><span>
</span><span>remDr</span><span>$</span><span>findElement</span><span>(</span><span>&quot;id&quot;</span><span>,</span><span> </span><span>&quot;loginbutton&quot;</span><span>)</span><span>$</span><span>clickElement</span><span>()</span></code></pre>
</figure>
<h1>Few basic RSelenium functions</h1>
<p>In this moment I think few comments about basics of RSelenium commands are required. With <code>findElement</code> method you can get the first element on the web page that suits the searching criterion, with <code>findElements</code> you can find all such elements. On each such element (or on the list of elements) you can perform further operations like</p>
<ul>
<li>sending data to element (<code>sendKeysToElement</code>)</li>
<li>clicking the element (<code>clickElement</code>)</li>
<li>finding elements within that element (<code>findChildsElement</code>)</li>
<li>extracting the text from element (<code>getElementText</code>)</li>
<li>highlighting the element (<code>highlightElement</code>)</li>
</ul>
<p>and many more!</p>
<h1>Extracting information from Ali Express allowed only for logged user</h1>
<p>Ali Express is a market portal where you can order products mainly from China. They are of low quality but also of a cheap price, that’s why this portal is very popular, even though the long period of a home delivery (which is free in many cases). You can buy <code>earrings</code> or <code>clothes</code> for few cents!</p>
<p><strong>How much money did <code>I</code> <img alt="🙂" class="wp-smiley" src="https://s.w.org/images/core/emoji/2/72x72/1f642.png"> spend for all my transactions?</strong></p>
<p>From <code>My orders</code> panel, allowed for a logged user I found out that for each page of orders (I had 32 pages of history of my transactions) I can extract whole body of a transaction, and then from that body I can extract the amount that I payed in dollars, check if the transaction wasn’t cancelled and get the ID of an operation. I just need to properly specify the names of classes of HTML tags/objects I need.</p>
<p>I will use <code>order</code> and <code>transaction</code> interchangeably.</p>
<figure>
<pre><code data-lang="r"><span># get all orders from the current page
</span><span>remDr</span><span>$</span><span>findElements</span><span>(</span><span>&quot;class&quot;</span><span>,</span><span> </span><span>&quot;order-item-wraper&quot;</span><span>)</span><span> </span><span>-></span><span> </span><span>orders</span><span>
</span><span># extract id, amount and status of the first order
</span><span>orders</span><span>[[</span><span>1</span><span>]]</span><span>$</span><span>findChildElement</span><span>(</span><span>&quot;class&quot;</span><span>,</span><span> </span><span>&quot;amount-num&quot;</span><span>)</span><span>$</span><span>getElementText</span><span>()[[</span><span>1</span><span>]]</span><span> </span><span>-></span><span> </span><span>amount</span><span>
</span><span>orders</span><span>[[</span><span>1</span><span>]]</span><span>$</span><span>findChildElement</span><span>(</span><span>&quot;class&quot;</span><span>,</span><span> </span><span>&quot;product-action&quot;</span><span>)</span><span>$</span><span>getElementText</span><span>()[[</span><span>1</span><span>]]</span><span> </span><span>-></span><span> </span><span>payed</span><span>
</span><span>orders</span><span>[[</span><span>1</span><span>]]</span><span>$</span><span>findChildElement</span><span>(</span><span>&quot;class&quot;</span><span>,</span><span> </span><span>&quot;info-body&quot;</span><span>)</span><span>$</span><span>getElementText</span><span>()[[</span><span>1</span><span>]]</span><span> </span><span>-></span><span> </span><span>id</span><span>
</span><span># the order date is the second element found with
</span><span>orders</span><span>[[</span><span>1</span><span>]]</span><span>$</span><span>findChildElements</span><span>(</span><span>&quot;class&quot;</span><span>,</span><span> </span><span>&quot;info-body&quot;</span><span>)[[</span><span>2</span><span>]]</span><span>$</span><span>getElementText</span><span>()[[</span><span>1</span><span>]]</span></code></pre>
</figure>
<p><img src="https://i2.wp.com/r-addict.com/images/fulls/ali1.png?w=456" data-recalc-dims="1"><br>
<img src="https://i1.wp.com/r-addict.com/images/fulls/ali999.png?w=456" data-recalc-dims="1"><br>
<img src="https://i1.wp.com/r-addict.com/images/fulls/ali1234.png?w=456" data-recalc-dims="1"></p>
<p>For all 32 pages of orders’ history the following <code>for loop</code> extracts information for each sub page and then navigates to the next sub page of orders’ history.</p>
<figure>
<pre><code data-lang="r"><span>payments</span><span> </span><span>&lt;-</span><span> </span><span>list</span><span>()</span><span>
</span><span>for</span><span>(</span><span>i</span><span> </span><span>in</span><span> </span><span>1</span><span>:</span><span>32</span><span>){</span><span>
</span><span>remDr</span><span>$</span><span>findElements</span><span>(</span><span>&quot;class&quot;</span><span>,</span><span> </span><span>&quot;order-item-wraper&quot;</span><span>)</span><span> </span><span>-></span><span> </span><span>orders</span><span>
</span><span>do.call</span><span>(</span><span>rbind</span><span>,</span><span>
</span><span>lapply</span><span>(</span><span>orders</span><span>,</span><span> </span><span>function</span><span>(</span><span>element</span><span>){</span><span>
</span><span>element</span><span>$</span><span>findChildElement</span><span>(</span><span>&quot;class&quot;</span><span>,</span><span> </span><span>&quot;amount-num&quot;</span><span>)</span><span>$</span><span>getElementText</span><span>()[[</span><span>1</span><span>]]</span><span> </span><span>-></span><span> </span><span>amount</span><span>
</span><span>element</span><span>$</span><span>findChildElement</span><span>(</span><span>&quot;class&quot;</span><span>,</span><span> </span><span>&quot;product-action&quot;</span><span>)</span><span>$</span><span>getElementText</span><span>()[[</span><span>1</span><span>]]</span><span> </span><span>-></span><span> </span><span>payed</span><span>
</span><span>element</span><span>$</span><span>findChildElement</span><span>(</span><span>&quot;class&quot;</span><span>,</span><span> </span><span>&quot;info-body&quot;</span><span>)</span><span>$</span><span>getElementText</span><span>()[[</span><span>1</span><span>]]</span><span> </span><span>-></span><span> </span><span>id</span><span>
</span><span>data.frame</span><span>(</span><span>amount</span><span> </span><span>=</span><span> </span><span>amount</span><span>,</span><span>
</span><span>payed</span><span> </span><span>=</span><span> </span><span>payed</span><span>,</span><span>
</span><span>id</span><span> </span><span>=</span><span> </span><span>id</span><span>)</span><span>
</span><span>}))</span><span> </span><span>-></span><span> </span><span>payments</span><span>[[</span><span>i</span><span>]]</span><span>
</span><span>remDr</span><span>$</span><span>findElement</span><span>(</span><span>&quot;class name&quot;</span><span>,</span><span> </span><span>&quot;ui-pagination-next&quot;</span><span>)</span><span>$</span><span>clickElement</span><span>()</span><span>
</span><span>Sys.sleep</span><span>(</span><span>1</span><span>);</span><span> </span><span>cat</span><span>(</span><span>i</span><span>,</span><span> </span><span>&quot;\r&quot;</span><span>)</span><span>
</span><span>}</span></code></pre>
</figure>
<h1>Plotting expenses</h1>
<p>The extracted information is a plain text so it requires some text manipulation to achieve the tidy data, that can be plotted. Additionally I filter orders to those that haven’t been cancelled.</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>dplyr</span><span>)</span><span>
</span><span>library</span><span>(</span><span>stringi</span><span>)</span><span>
</span><span>do.call</span><span>(</span><span>rbind</span><span>,</span><span> </span><span>payments</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>filter</span><span>(</span><span>grepl</span><span>(</span><span>'Open Dispute'</span><span>,</span><span> </span><span>payed</span><span>))</span><span> </span><span>%>%</span><span> </span><span># </span><span> </span><span>mutate</span><span>(</span><span>amount</span><span> </span><span>=</span><span> </span><span>as.numeric</span><span>(</span><span>unlist</span><span>(</span><span>
</span><span>stri_extract_all_regex</span><span>(</span><span>amount</span><span>,</span><span> </span><span>&quot;[0-9\\.]+&quot;</span><span>))))</span><span> </span><span>-></span><span> </span><span>aliexpress</span></code></pre>
</figure>
<figure>
<pre><code data-lang="r"><span>head</span><span>(</span><span>aliexpress</span><span>)</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">  amount        payed             id
1   0.24 Open Dispute 81238841078231
2   1.46 Open Dispute 81235899648231
3   0.66 Open Dispute 81320512758231
4   1.07 Open Dispute 81192904428231
5   1.85 Open Dispute 81150862778231
6   0.61 Open Dispute 81116930588231</code></pre>
</figure>
<p>The plot of cumulative expenses can be obtained with the following code. I can’t believe that 450 $ was spend over 8 months! The result of the code is the main photo of this post.</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>ggthemr</span><span>);</span><span>ggthemr</span><span>(</span><span>&quot;camoflauge&quot;</span><span>)</span><span>
</span><span>aliexpress</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>id</span><span> </span><span>=</span><span> </span><span>as.numeric</span><span>(</span><span>as.character</span><span>(</span><span>id</span><span>)))</span><span> </span><span>%>%</span><span>
</span><span>arrange</span><span>(</span><span>id</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>cum_amount</span><span> </span><span>=</span><span> </span><span>cumsum</span><span>(</span><span>amount</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>ggplot</span><span>(</span><span>aes</span><span>(</span><span>id</span><span>,</span><span> </span><span>cum_amount</span><span>,</span><span> </span><span>group</span><span> </span><span>=</span><span> </span><span>1</span><span>))</span><span> </span><span>+</span><span> </span><span>geom_line</span><span>()</span><span> </span><span>+</span><span>
</span><span>theme</span><span>(</span><span>axis.ticks.x</span><span>=</span><span>element_blank</span><span>(),</span><span>
</span><span>axis.text.x</span><span>=</span><span>element_blank</span><span>())</span><span> </span><span>+</span><span>
</span><span>labs</span><span>(</span><span>title</span><span> </span><span>=</span><span> </span><span>&quot;Cumulative Ali Express expenses&quot;</span><span>,</span><span>
</span><span>subtitle</span><span> </span><span>=</span><span> </span><span>&quot;web-harvested with RSelenium&quot;</span><span>,</span><span>
</span><span>caption</span><span> </span><span>=</span><span> </span><span>&quot;source: r-addict.com&quot;</span><span>)</span><span> </span><span>+</span><span>
</span><span>ylab</span><span>(</span><span>&quot;Cumulative sum of spendings in $&quot;</span><span>)</span><span> </span><span>+</span><span> </span><span>xlab</span><span>(</span><span>NULL</span><span>)</span></code></pre>
</figure>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://r-addict.com/2017/01/08/RSelenium-at-TriCity-and-AliExpress.html"> http://r-addict.com</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/lKxGKQ15eHQ"></a></div>