<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/can-we-predict-the-crawling-of-the-google-bot/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://keyword-hero.com/can-predict-crawling-google-bot"> Keyword Hero – The SEO Revolution</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Logfile analysis allows website owners a deep insight about how Google and other search engines crawl their pages. How often does a bot come by a page, which pages are rarely or not at all visited by the bots? For which pages does the bot get errors? All these and more questions can be answered by a log file analysis.</p>
<p>Unfortunately, a log file analysis is extremely time-consuming in many cases, as very few employees have access to the log files, especially in large organizations. If you have managed to access this treasure trove of data, the evaluation of the usually huge files can cause you headaches.</p>
<p>For collecting our logs we used <a rel="nofollow" href="https://log-hero.com/" target="_blank">log-hero.com</a>, a tool which stores the logs directly in Google Analytics and we can simply query the data there via the API and compare this data directly with user data from the main account as needed in the analysis.</p>
<h2>Get the Bot Data from Google Analytics</h2>
<p>To get to know the paths the google crawler travels along our page, we first need the data provided through <a rel="nofollow" href="https://log-hero.com/" target="_blank">Log Hero</a>. We can easily acccess the data through an API call to Google Analyitcs. We like to use the rga package, but you may prefer the googleAnalyticsR. Both deliver the data in the same format, so it’s mostly a matter of personal preference. It is very important though to use the rga package from <a rel="nofollow" href="https://github.com/skardhamar/rga" target="_blank">https://github.com/skardhamar/rga</a> and not it’s capital letter cousin RGA.</p>
<pre>rga.open(instance=&quot;ga&quot;,
client.id = &quot;xxxxxxxxxxxxxx.apps.googleusercontent.com&quot;,
client.secret = &quot;xxxxxxxxxxxxxxxxx&quot;)
logdata &lt;- ga$getData(&quot;177217805&quot;, start.date = '2018-06-01', end.date = '2018-10-28', dimensions = &quot;ga:year,ga:date,ga:pagePath&quot;, metrics = &quot;ga:hits&quot;, filters = &quot;ga:dimension2=@Google;ga:dimension4==200&quot;, batch = TRUE</pre>
<p> </p>
<div>
<h2>Follow the Googlebot</h2>
<p>We do some data wrangling to put the steps of the crawler into a format that is accepted by the clickstream package. Now we know the paths the crawlers is taking and are able to find a structure within these paths. A natural model to be applied here is markov chains which are easily fitted and plotted.</p>
<pre># Use four lettters of md4 sum to make plot readable
logdata$pagePathMD5 = substr(unlist(lapply(logdata$pagePath,FUN = digest::sha1)),1,4)
logdata %>% 
  filter(hits > 10) ->
  filtered_logdata

# Data wrangling for clickstream format
path &lt;- ddply(filtered_logdata, &quot;year&quot;,function(logdata)paste(logdata$pagePathMD5,collapse = &quot;,&quot;))
newchar &lt;- as.character(path$V1)
csf &lt;- file(description = &quot;clickstream.txt&quot;)
writeLines(newchar,csf)
cls &lt;- readClickstreams(csf, header = TRUE)

# Fit and plot MC
mc &lt;- fitMarkovChain(cls)
plot(mc, edge.label = NULL)</pre>

</div>
<div>
<h2><img sizes="(max-width: 1344px) 100vw, 1344px" src="https://i1.wp.com/keyword-hero.com/wp-content/uploads/2018/11/Markov_Chain_Googlebot.png?w=450&ssl=1" alt="Markov_Chain_Googlebot" class="wp-image-2244" data-recalc-dims="1"></h2>
<h2>But how often will Google visit us again?</h2>
<p>With the help of the high-level prediction package of prophet, we can make easy and surprisingly accurate predictions of how many times the crawler will visit the page in the future – by learning from past data. There is however some erratic component to the crawler behaviour – or rather we are missing some information.</p>
<pre>prophet_input = logdata[,c(&quot;date&quot;,&quot;hits&quot;)]

colnames(prophet_input)&lt;-c(&quot;ds&quot;, &quot;y&quot;)

m &lt;- prophet(prophet_input)</pre>

<pre>## Initial log joint probability = -3.33717
## Optimization terminated normally:
## Convergence detected: relative gradient magnitude is below tolerance</pre>

<pre>future &lt;- make_future_dataframe(m, periods = 60)
forecast &lt;- predict(m, future)
plot(m, forecast)</pre>

<div>
<h2><img sizes="(max-width: 1344px) 100vw, 1344px" src="https://i0.wp.com/keyword-hero.com/wp-content/uploads/2018/11/Prophet-googlebot.png?w=450&ssl=1" alt="Prophet googlebot" class="wp-image-2243" data-recalc-dims="1"></h2>
<h2>Estimating the hits of a single page</h2>
<p>Now if we add a new page or otherwise want to predcit how often a single page will be visited by the crawler, we would like to build a prediction model. For this, we will use 4 sources of data:</p>
<ol>
<li>The Loghero data – Where was the crawler in the past and how often?</li>
<li>The standard Google Analytics Data – Where and how long do human users stay?</li>
<li>Screaming Frog page data – Meta data of each page (text ratio, word count, size…)</li>
<li>Screaming Frog link data – (Unique) in and outlinks to the page</li>
</ol>
<p>So let us gather all this data. We will be using the googleAnalyticsR package this time.</p>
<pre># GA data
googleAnalyticsR::ga_auth(new_user = TRUE)

# Loghero Data
df_loghero = google_analytics_3(id = &quot;>your_LOGHERO_ID&lt;&quot;,start = '2018-05-01',end = '2018-10-15',
                       dimensions = c(&quot;ga:dimension2&quot;,&quot;ga:dimension1&quot;, &quot;ga:dimension4&quot; ,&quot;ga:pagePath&quot;),
                       metrics = c(&quot;ga:hits&quot;,&quot;ga:metric1&quot;),
                       filters = &quot;ga:dimension2=~^Googlebot(\\-Mobile)?&quot;,
                       samplingLevel = &quot;WALK&quot;)

# Google Analytics data
df_user_ga = google_analytics_3(id = &quot;>your_GA_ID&quot;,start = '2018-05-01',end = '2018-08-15',
                                dimensions = c(&quot;ga:pagePath&quot;,&quot;ga:hostname&quot;),
                                metrics = c(&quot;ga:pageviews&quot;,&quot;ga:avgTimeOnPage&quot;,&quot;entrances&quot;),
                                samplingLevel = &quot;WALK&quot;)

# Screaming Frog data
# Written to csv files in screaming frog application
page_data = fread(input = &quot;internal_html.csv&quot;)
page_inlinks = fread(input = &quot;all_inlinks.csv&quot;)</pre>
<p>To merge all our data together we have to do some basic data wrangling.</p>
<pre>library(googleAnalyticsR)
library(igraph)
library(ranger)
library(data.table)
library(dplyr)

# Feature Engineering and Type Casting
df_loghero$avgMetric1 = df_loghero$metric1 / as.numeric(df_loghero$hits) 
df_loghero$hits = as.factor(df_loghero$hits)

# Build into a single file
page_data$pagePath = gsub(pattern = &quot;http://ohren-reinigen.de&quot;,&quot;&quot;,page_data$Address)

df_loghero$pagePath&lt;-gsub(&quot;sitemap&quot;, &quot;sitemap&quot;,df_loghero$pagePath)
df_loghero$pagePath&lt;-gsub(&quot;.*sitemap.*&quot;, &quot;sitemap&quot;,df_loghero$pagePath)
df_loghero$pagePath&lt;-gsub(&quot;^\\/$&quot;, &quot;Home&quot;,df_loghero$pagePath)

df_user_ga$pagePath&lt;-gsub(&quot;sitemap&quot;, &quot;sitemap&quot;,df_user_ga$pagePath)
df_user_ga$pagePath&lt;-gsub(&quot;.*sitemap.*&quot;, &quot;sitemap&quot;,df_user_ga$pagePath)
df_user_ga$pagePath&lt;-gsub(&quot;^\\/$&quot;, &quot;Home&quot;,df_user_ga$pagePath)

# summarise 
df_user_ga %>% 
  group_by(pagePath) %>% 
  summarise(pageviews = sum(pageviews),
            avgTimeOnPage = mean(avgTimeOnPage),
            entrances = sum(entrances)) %>% 
  ungroup() ->
  df_user_ga

df_loghero %>% 
  group_by(pagePath) %>% 
  summarise(hits = sum(as.numeric(hits)),
            avgMetric1 = mean(avgMetric1)) ->
  df_loghero

# merge 1
df_merged = merge(page_data, df_loghero, by = &quot;pagePath&quot;)
df_merged = merge(df_merged, df_user_ga, by = &quot;pagePath&quot;, all.x = TRUE)


# Basic imputation
df_merged$pageviews[is.na(df_merged$pageviews)] = mean(df_merged$pageviews,na.rm = TRUE)
df_merged$avgTimeOnPage[is.na(df_merged$avgTimeOnPage)] = mean(df_merged$avgTimeOnPage,na.rm = TRUE)
df_merged$entrances[is.na(df_merged$entrances)] = mean(df_merged$entrances,na.rm = TRUE)


# Use only R standard variable names
names(df_merged) = make.names(names(df_merged))</pre>
<p>We will now use to basic models which we will train on a subset and test on test set. First, we will use a Random Forest to estimate a basic regression, ignoring the ordinal factor scale of counting the visits discreetly, but enjoying the non-linearity. In the second model, we will use a generalize linear model with poisson link and thusly incorporate the fact we are predicting a count variable but losing non-linearity in the proess.</p>
<pre># Modelling

# Train Test Split
train_idx = sample(1:nrow(df_merged),size = round(2*nrow(df_merged)/3),replace = FALSE)
train = df_merged[train_idx,]
test = df_merged[-train_idx,]

# Ranger basic Model
mod_rf = ranger(formula = hits ~ Word.Count
                              + Unique.Inlinks
                              + Unique.Outlinks
                              + Inlinks
                              + Outlinks
                              + Text.Ratio
                              + Response.Time
                              + Size
                              + pageviews
                              + avgTimeOnPage
                              + entrances
                 , data = train, num.trees = 500,importance = &quot;impurity&quot;)

mod_glm = glm(formula = hits ~ Word.Count
                           + Unique.Inlinks
                           + Unique.Outlinks
                           + Inlinks
                           + Outlinks
                           + Text.Ratio
                           + Response.Time
                           + Size
                           + pageviews
                           + avgTimeOnPage
                           + entrances,
                           data = train,
              family = poisson)

# Predict values
test$pred_rf = round(predict(mod_rf, data = test)$predictions)
test$pred_glm = round(predict(mod_glm, newdata = test))</pre>
<p>We have trained the models and now predicted the number of crawler visits for pages the model hast never seen. Let’s compare the predicted values with the actual ones:</p>
<pre>test %>% 
  dplyr::select(pagePath,hits,pred_rf,pred_glm) %>% 
  arrange(desc(hits))</pre>

<table>
<thead>
<tr>
<th align="left">pagePath</th>
<th align="right">hits</th>
<th align="right">pred_rf</th>
<th align="right">pred_glm</th>
</tr>
</thead>
<tbody>
<tr>
<td align="left">/ohrenschmalz/</td>
<td align="right">197</td>
<td align="right">126</td>
<td align="right">5</td>
</tr>
<tr>
<td align="left">/ohren-reinigen/ohrenkerzen/</td>
<td align="right">195</td>
<td align="right">74</td>
<td align="right">5</td>
</tr>
<tr>
<td align="left">/ohren-reinigen/ohren-reinigen-bei-haustieren/</td>
<td align="right">127</td>
<td align="right">120</td>
<td align="right">6</td>
</tr>
<tr>
<td align="left">/ohren-reinigen/ohrenreiniger/</td>
<td align="right">79</td>
<td align="right">72</td>
<td align="right">4</td>
</tr>
<tr>
<td align="left">/otoskop/</td>
<td align="right">69</td>
<td align="right">41</td>
<td align="right">4</td>
</tr>
<tr>
<td align="left">/reinigungsanleitung/</td>
<td align="right">63</td>
<td align="right">44</td>
<td align="right">5</td>
</tr>
<tr>
<td align="left">/ohren-reinigen/wattestaebchen/</td>
<td align="right">47</td>
<td align="right">97</td>
<td align="right">4</td>
</tr>
<tr>
<td align="left">/testsieger/</td>
<td align="right">38</td>
<td align="right">53</td>
<td align="right">2</td>
</tr>
</tbody>
</table>
<pre>test %>% 
  dplyr::select(pagePath,hits,pred_rf,pred_glm) %>% 
  arrange(desc(hits)) %>% 
  mutate(rank = rank(1/hits, ties.method = &quot;min&quot;), 
         rank_rf = rank(1/pred_rf, ties.method = &quot;min&quot;), 
         rank_glm = rank(1/pred_glm, ties.method = &quot;min&quot;)) %>% 
  dplyr::select(pagePath, rank, rank_rf, rank_glm)</pre>

<table>
<thead>
<tr>
<th align="left">pagePath</th>
<th align="right">rank</th>
<th align="right">rank_rf</th>
<th align="right">rank_glm</th>
</tr>
</thead>
<tbody>
<tr>
<td align="left">/ohrenschmalz/</td>
<td align="right">1</td>
<td align="right">1</td>
<td align="right">2</td>
</tr>
<tr>
<td align="left">/ohren-reinigen/ohrenkerzen/</td>
<td align="right">2</td>
<td align="right">4</td>
<td align="right">2</td>
</tr>
<tr>
<td align="left">/ohren-reinigen/ohren-reinigen-bei-haustieren/</td>
<td align="right">3</td>
<td align="right">2</td>
<td align="right">1</td>
</tr>
<tr>
<td align="left">/ohren-reinigen/ohrenreiniger/</td>
<td align="right">4</td>
<td align="right">5</td>
<td align="right">5</td>
</tr>
<tr>
<td align="left">/otoskop/</td>
<td align="right">5</td>
<td align="right">8</td>
<td align="right">5</td>
</tr>
<tr>
<td align="left">/reinigungsanleitung/</td>
<td align="right">6</td>
<td align="right">7</td>
<td align="right">2</td>
</tr>
<tr>
<td align="left">/ohren-reinigen/wattestaebchen/</td>
<td align="right">7</td>
<td align="right">3</td>
<td align="right">5</td>
</tr>
<tr>
<td align="left">/testsieger/</td>
<td align="right">8</td>
<td align="right">6</td>
<td align="right">8</td>
</tr>
</tbody>
</table>
<p>As we can see, the random forest is far more capable of dealing with the large differences in scale in between the pages than the glm. But if we only consider the rank of the pages, both models acutally work pretty well. From an user perspective, the concret number of predicted visits is not as important as the fact which pages are visited more often. The GLM of course offers to take a look at the beta values:</p>
<pre>summary(mod_glm)$coefficients</pre>

<pre>## Estimate Std. Error z value Pr(>|z|)
## (Intercept) 11.7771264452 1.884972e+00 6.2479049 4.159949e-10
## Word.Count 0.0063256324 7.856065e-04 8.0519097 8.151201e-16
## Unique.Inlinks 0.0967094083 3.276334e-02 2.9517569 3.159715e-03
## Unique.Outlinks 0.2747712141 1.120261e-01 2.4527436 1.417714e-02
## Inlinks 0.0062746899 6.595578e-03 0.9513481 3.414277e-01
## Outlinks -0.0886505758 4.462267e-02 -1.9866712 4.695885e-02
## Text.Ratio -0.4113764679 7.342136e-02 -5.6029539 2.107293e-08
## Response.Time 2.2484413075 1.285903e+00 1.7485306 8.037219e-02
## Size -0.0002508579 6.675199e-05 -3.7580583 1.712370e-04
## pageviews -0.0004068211 1.399437e-04 -2.9070336 3.648740e-03
## avgTimeOnPage 0.0059767188 1.986728e-03 3.0083225 2.626942e-03
## entrances 0.0004635700 1.624705e-04 2.8532566 4.327367e-03</pre>
<p> </p>
<p>With these information you can further tune and adjust your homepage and find out what the crawler is actually doing.</p>
</div>
<h2>Further thoughts to predict the Google Bot</h2>
<p>This article shows the first experiments we can do with the data from Log Hero for individual pages. Even further, we managed to produce viable proof-of-concept models for crawler behaviour prediction within a few dozen lines of code. Investing more time and thought may lead to very sophisticate models directly suited to your needs.</p>
<p>The idea now is to add more and more data points to the prediction and to find out what were the most important factors in our prediction – the page speed, behaviour/traffic by real users, in- and outlinks?</p>
<p>We are very happy if you could share your results with us, if you can do similar analyses (or even the same with the code above) analyses or give further tips for perhaps relevant data points.</p>
</div>
<p>Der Beitrag <a rel="nofollow" href="https://keyword-hero.com/can-predict-crawling-google-bot" target="_blank">Can we predict the crawling of the Google-Bot?</a> erschien zuerst auf <a rel="nofollow" href="https://keyword-hero.com/" target="_blank">Keyword Hero – The SEO Revolution</a>.</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://keyword-hero.com/can-predict-crawling-google-bot"> Keyword Hero – The SEO Revolution</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/9xfp1V5U5Zk" height="1"></a></div>