<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/using-r-to-study-the-evolution-of-tennis/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://msperlin.github.io/2017-02-05-R-and-Tennis/"> R and Finance</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p> An analysis of point by point data – </p>
<p>I’m a big fan of Tennis. When I’m not working in the university, you can<br>
probably find me in my favourite tennis club,<br>
<a target="_blank" rel="nofollow" href="http://www.sogipa.com.br/web/">Sogipa</a>. What is so great about Tennis?<br>
It is a sport that challenges your physically and mentally. It forces<br>
you to your limit in both aspects. The more you play, the more you love<br>
it and the more you want to learn about the game.</p>
<p>Tennis is also very different from the university work, which is mainly<br>
a brain effort of writing, speaking and coding. I find it very healthy<br>
to be able to momentarily distance myself from the day to day work, even<br>
for a few hours. As my close friends can testify, I won’t shut up about<br>
tennis and try to convince other people to start playing. Now, let’s get<br>
to part of this blog that matters.</p>
<p>In this post I will combine two things that I love, R and Tennis. As you<br>
will see soon enough, you can find a large database of information about<br>
tennis matches publicly available in the internet. In my first post in<br>
this topic, I will describe the evolution and effects in the game over<br>
time, specially a change in the equipment back in 2000.</p>
<h2>The data</h2>
<p>The data gathered for this post is from the great repository of <a target="_blank" rel="nofollow" href="https://github.com/JeffSackmann/tennis_MatchChartingProject">Jeff<br>
Sackmann</a>.<br>
There you can find csv files with information from ATP matches, WTA and<br>
more. Among the repository choices, the most impressive is the <a target="_blank" rel="nofollow" href="https://github.com/JeffSackmann/tennis_MatchChartingProject">Match<br>
Charting<br>
Project</a>.<br>
It contains point by point data for thousands of matches. Yes, that’s<br>
right, all points of the match are registered following particular<br>
guidelines. This is a collective effort, where the public can send<br>
statistics to the project. The level of details is amazing, you can<br>
simulate the whole match based on the csv files.</p>
<p>Now, lets get some code going. Our first step in this analysis is to<br>
download and unzip the dataset.</p>
<div>
<pre><code>zip.file &lt;- 'TennisData.zip'

if (!file.exists(zip.file)){
  download.file('https://github.com/JeffSackmann/tennis_MatchChartingProject/archive/master.zip', destfile = zip.file)
  
}

unzip(zip.file)
</code></pre>
</div>
<p>Let’s look at the contents of the folder from the zip file.</p>
<div>
<pre><code>list.files(path = 'tennis_MatchChartingProject-master/')

##  [1] &quot;charting-m-matches.csv&quot;              
##  [2] &quot;charting-m-points.csv&quot;               
##  [3] &quot;charting-m-stats-KeyPointsReturn.csv&quot;
##  [4] &quot;charting-m-stats-KeyPointsServe.csv&quot; 
##  [5] &quot;charting-m-stats-NetPoints.csv&quot;      
##  [6] &quot;charting-m-stats-Overview.csv&quot;       
##  [7] &quot;charting-m-stats-Rally.csv&quot;          
##  [8] &quot;charting-m-stats-ReturnDepth.csv&quot;    
##  [9] &quot;charting-m-stats-ReturnOutcomes.csv&quot; 
## [10] &quot;charting-m-stats-ServeBasics.csv&quot;    
## [11] &quot;charting-m-stats-ServeDirection.csv&quot; 
## [12] &quot;charting-m-stats-ServeInfluence.csv&quot; 
## [13] &quot;charting-m-stats-ShotDir.csv&quot;        
## [14] &quot;charting-m-stats-ShotDirection.csv&quot;  
## [15] &quot;charting-m-stats-ShotDirOutcomes.csv&quot;
## [16] &quot;charting-m-stats-ShotTypes.csv&quot;      
## [17] &quot;charting-m-stats-SnV.csv&quot;            
## [18] &quot;charting-m-stats-SvBreakSplit.csv&quot;   
## [19] &quot;charting-m-stats-SvBreakTotal.csv&quot;   
## [20] &quot;charting-w-matches.csv&quot;              
## [21] &quot;charting-w-points.csv&quot;               
## [22] &quot;charting-w-stats-KeyPointsReturn.csv&quot;
## [23] &quot;charting-w-stats-KeyPointsServe.csv&quot; 
## [24] &quot;charting-w-stats-NetPoints.csv&quot;      
## [25] &quot;charting-w-stats-Overview.csv&quot;       
## [26] &quot;charting-w-stats-Rally.csv&quot;          
## [27] &quot;charting-w-stats-ReturnDepth.csv&quot;    
## [28] &quot;charting-w-stats-ReturnOutcomes.csv&quot; 
## [29] &quot;charting-w-stats-ServeBasics.csv&quot;    
## [30] &quot;charting-w-stats-ServeDirection.csv&quot; 
## [31] &quot;charting-w-stats-ServeInfluence.csv&quot; 
## [32] &quot;charting-w-stats-ShotDirection.csv&quot;  
## [33] &quot;charting-w-stats-ShotDirOutcomes.csv&quot;
## [34] &quot;charting-w-stats-ShotTypes.csv&quot;      
## [35] &quot;charting-w-stats-SnV.csv&quot;            
## [36] &quot;charting-w-stats-SvBreakSplit.csv&quot;   
## [37] &quot;charting-w-stats-SvBreakTotal.csv&quot;   
## [38] &quot;charting-w-ufe-fe.csv&quot;               
## [39] &quot;MatchChart 0.1.5.xlsm&quot;               
## [40] &quot;README.md&quot;
</code></pre>
</div>
<p>That’s a lot of files! But, once you read the manual, the structure is<br>
quite simple. The <code>m</code> and <code>w</code> in each file means <em>men</em> (masculine) and<br>
<em>women</em> (feminine) tennis matches. The files with <code>stats</code> in their names<br>
are the product of a particular aspect of the game. All analysis that<br>
resulted in these files are based on the <strong>master files</strong><br>
<code>charting-m-points.csv</code> and <code>charting-f-points.csv</code>.</p>
<p>These are the largest files from the repository, containing point by<br>
point data. For example, the first row of column <code>1st</code> in<br>
<code>charting-m-points.csv</code> shows the code <code>6b29f3f3f3b3f1f2f3b3u1*</code>. The<br>
first letter indicates the first serve. The value 6 means a serve <em>down<br>
the T</em>, that is, a serve in the “T” located in the middle of the court.<br>
Each following shot is represented by a number that indicates direction<br>
and depth (e.g. 4=out wide) and a letter that determines shot type (e.g.<br>
f=forehand).</p>
<p>The game keeps moving forward and each pair of number and letter sets<br>
the configuration of shots from the players, back and forth. Eventually,<br>
the point ends when someone misses or hits a winner. For the example<br>
code <code>6b29f3f3f3b3f1f2f3b3u1*</code>, the last symbol <code>*</code> indicates that<br>
someone has hit a winner and that event ended the point. As you can see,<br>
this is an extremely detailed registration of a match. We only scratched<br>
the surface, there are many more details about the coding system in file<br>
<code>MatchChart 0.1.5.xlsm</code>.</p>
<p>Our first analysis will be to look into the volume of registered games<br>
over time. We will combine men and women data in the analysis.</p>
<div>
<pre><code>library(dplyr)
library(ggplot2)
library(readr)
library(stringr)

# files
my.f.m &lt;- 'tennis_MatchChartingProject-master/charting-m-matches.csv'
my.f.w &lt;- 'tennis_MatchChartingProject-master/charting-w-matches.csv'

# set cols for read_csv
my.cols &lt;- cols(
  match_id = col_character(),
  `Player 1` = col_character(),
  `Player 2` = col_character(),
  `Pl 1 hand` = col_character(),
  `Pl 2 hand` = col_character(),
  Gender = col_character(),
  Date = col_character(),
  Tournament = col_character(),
  Round = col_character(),
  Time = col_character(),
  Court = col_character(),
  Surface = col_character(),
  Umpire = col_character(),
  `Best of` = col_integer(),
  `Final TB?` = col_integer(),
  `Charted by` = col_character()
)

# read and bind!
my.df.matches &lt;- bind_rows(read_csv(my.f.m, col_types = my.cols) %>% mutate(Gender = 'Men'),
                           read_csv(my.f.w, col_types = my.cols) %>% mutate(Gender = 'Women'))

# create year col
my.df.matches$Date &lt;- as.Date(my.df.matches$Date,'%Y%m%d')
my.df.matches$Year &lt;- format(my.df.matches$Date,'%Y')

# plot it!
p &lt;- my.df.matches %>%
  group_by(Year,Gender) %>%
  summarise(`Number of matches` = n()) %>%
  ggplot(aes(x=as.numeric(Year),y=`Number of matches`,color=factor(Gender))) +
  geom_line(size=2)

print(p)
</code></pre>
</div>
<p><img alt="" src="https://i2.wp.com/msperlin.github.io/img/2017-02-05-R-and-Tennis_files/figure-markdown_strict/matches-1.png?w=456" data-recalc-dims="1"></p>
<p>As you can see, most of the data was registered for matches after 2010.<br>
Interestingly, the number of men and women games follow a very similar<br>
pattern. The drop in 2016 can be explained by the registration process.<br>
It is likely that people watch recorded matches in order to fill out the<br>
points. I would predict that the drop will be lower once the data is<br>
again downloaded from the repository in 2017.</p>
<p>Now, lets look at the distribution of court types. First, as with any<br>
open field in a database, there are lots of problems with the court type<br>
field. If we just count the cases, we get:</p>
<div>
<pre><code>temp.tab &lt;- table(my.df.matches$Surface)
print(temp.tab)

## 
##            Carpet     Carpet (hard)              Clay Court des Princes 
##                 3                 1               583                 1 
##             Grass              hard              Hard        Hard Court 
##               259                 1              1410                 1 
##       Hard Indoor      Hard Outdoor       Indoor Clay       Indoor hard 
##                 1                 1                 1                10 
##       Indoor Hard      Outdoor Clay 
##                28                 1
</code></pre>
</div>
<p>Clearly there are registration errors. We can clean this up it by just<br>
keeping the court types with more than 30 matches.</p>
<div>
<pre><code># remove
temp.tab &lt;- temp.tab[temp.tab>30]
surf.to.keep &lt;- names(temp.tab)

# find idx of Not 
idx &lt;- !(my.df.matches$Surface %in% surf.to.keep)

# set to NA
my.df.matches$Surface[idx] &lt;- NA
</code></pre>
</div>
<p>And now we plot the result.</p>
<div>
<pre><code># plot percentage of court type over years
p &lt;- ggplot(my.df.matches, aes(x = as.numeric(Year), fill = Surface))
p &lt;- p + geom_bar(position = 'fill') 

print(p)

## Warning: Removed 1 rows containing non-finite values (stat_count).
</code></pre>
</div>
<p><img alt="" src="https://i1.wp.com/msperlin.github.io/img/2017-02-05-R-and-Tennis_files/figure-markdown_strict/unnamed-chunk-5-1.png?w=456" data-recalc-dims="1"></p>
<p>It seems that hard courts dominated the game over the years. I’m not<br>
sure why that is. As a player, I find it easier to play in clay court as<br>
you can slide with your legs and that puts lower pressure on the knees.<br>
Perhaps hard courts are easier to maintain and that economic argument is<br>
extrapolated to tournaments as well. If anyone has a better guess,<br>
please leave it at the comments.</p>
<h2>Tennis matches before and after 2000</h2>
<p>Tennis is a sport that depends on its equipment. As anyone who plays it<br>
can testify, there is a lot of personal choices regarding brands, weight<br>
distribution of the racket, strings, grips, brand of tennis balls and so<br>
on. A particular type of player will benefit the most from a particular<br>
type of racket and string.</p>
<p>The physical and mechanical details of tennis makes it that any change<br>
in equipment will impact the game. A simple example: if you play more<br>
than five games with a set of balls (usually three), they become flat<br>
and travel differently in the air. The game becomes slower and that<br>
makes it more difficulty to hit winner shots, giving an advantage to a<br>
particular strategy of playing. This explains why new tennis balls are<br>
constantly introduced within a professional game.</p>
<p>One of the biggest systematic changes in tennis happened around 2000.<br>
The problem was that the game was losing audience. People were bored of<br>
watching the same type of plays and strategies over and over. A tennis<br>
match before 2000 was mostly composed of <strong>aces</strong>, when a player hit a<br>
speedy and angular serve, without chance for the adversary to prepare<br>
and return the ball or a <strong>serve and volley</strong> strategy, were the server<br>
would hit a slower ball, but would immediately run to the net in order<br>
to close the point. If you can find some example of this strategy<br>
<a target="_blank" rel="nofollow" href="https://www.youtube.com/watch?v=kOtC2UGY77I">here</a>. I can personally<br>
relate to that boredom. Most of the fun in watching matches comes<br>
happens when players are exchanging balls back and forth over the court,<br>
one trying to outsmart or overpower the other.</p>
<p>The success of both strategies, aces and serve-volley, are related to<br>
the speed of the ball. Aces are easier to hit if the ball travels<br>
faster. Likewise, a high velocity of the ball does not give chance for<br>
the adversary to prepare and counter attack the serve-volley. If the<br>
ball from the serve comes slower, the opponent can prepare and hit a<br>
passing shot or a lob (high ball) that counters a possible net approach.</p>
<p>Back in 2000, the ITF (International Tennis Federation) changed its<br>
regulation in order to introduce three new types of balls that would<br>
decrease the speed in hard courts and increase it in clay courts. The<br>
idea was to make it so that the tennis matches became more interesting<br>
for the audience to watch. It is not clear if this change was also set<br>
in women tennis, but we will study its effect in the data as well. You<br>
can find more details about the change in <a target="_blank" rel="nofollow" href="http://www.independent.co.uk/sport/tennis/itf-introduces-three-types-of-balls-to-counter-power-game-9206610.html">this<br>
article</a>.</p>
<h3>Percentage of aces</h3>
<p>Now, let’s look at the impact of this change over the percentage of aces<br>
from the total points of the game. We will also separate the results by<br>
type of surface and gender. Here, we will use the <em>stats</em> files from the<br>
repository. The information we need is in <em>ServeBasics</em>.</p>
<div>
<pre><code># filenames
my.f.m &lt;- 'tennis_MatchChartingProject-master/charting-m-stats-ServeBasics.csv'
my.f.w &lt;- 'tennis_MatchChartingProject-master/charting-w-stats-ServeBasics.csv'

# set cols
my.cols &lt;- cols(
  match_id = col_character(),
  row = col_character(),
  pts = col_integer(),
  pts_won = col_integer(),
  aces = col_integer(),
  unret = col_integer(),
  forced_err = col_integer(),
  pts_won_lte_3_shots = col_integer(),
  wide = col_integer(),
  body = col_integer(),
  t = col_integer()
)

# build dataframe
my.df.serve &lt;- bind_rows(read_csv(my.f.m, col_types = my.cols) %>% mutate(Gender = 'Men'),
                         read_csv(my.f.w, col_types = my.cols) %>% mutate(Gender = 'Women'))

# set dates
my.df.serve$Date &lt;- as.Date(my.df.serve$match_id,'%Y%m%d')
my.df.serve$Year &lt;- format(my.df.serve$Date,'%Y')

# get surface
idx &lt;- match(my.df.serve$match_id, my.df.matches$match_id)
my.df.serve$Surface &lt;- my.df.matches$Surface[idx]

# plot it!
p &lt;- filter(my.df.serve,str_detect(row,'Total'),!is.na(Surface)) %>%
  group_by(Year,Gender,Surface) %>%
  summarise(`Percent of Aces` = sum(aces)/sum(pts)) %>%
  ggplot(aes(x=as.numeric(Year),y=`Percent of Aces`,color = Gender)) +
  geom_point()+geom_smooth()+
  facet_wrap(~Surface)

print(p)
</code></pre>
</div>
<p><img alt="" src="https://i1.wp.com/msperlin.github.io/img/2017-02-05-R-and-Tennis_files/figure-markdown_strict/aces-1.png?w=456" data-recalc-dims="1"></p>
<p>The previou result show exactly what we expected. Looking at man’s<br>
tennis, the proportion of aces in clay did not show a significant change<br>
in 2000. As a matter of fact, the proportion has increased over the<br>
years. Clearly there is a break for grass and hard courts in 2000. It<br>
seems that players were investing heavily in making aces. This changed<br>
dramatically once the new balls were introduced.</p>
<p>What I’ve found surprising from the figures is how the difference is not<br>
that big. Back in 1995, an <em>average</em> men’s tennis match in a hard court<br>
had around 15% of all points from aces. This decreased to only around 8%<br>
in 2016. I would be very surprised if someone from an audience even<br>
noticed such a change. I was expecting a bigger difference. Perhaps,<br>
looking at games from the top 50 players this difference would be more<br>
noticeable.</p>
<p>As for the woman’s tennis, I don’t acknowledge the same effect. We see<br>
from the previous figures that the percentage of aces in the match are<br>
in its highest values in the past few years. This result suggests that<br>
the balls were not changed in female matches.</p>
<h3>Percentage of Net points</h3>
<p>Another information to analyze is the percentage of net points. If the<br>
change in the balls also affected this part of the game, then we can<br>
expect that the proportion of net points has decreased after 2000. As<br>
mentioned before, a net approach is easier to counter if the balls<br>
travels slower. Let’s have a look in what the data tells us.</p>
<div>
<pre><code># files 
my.f.m &lt;- 'tennis_MatchChartingProject-master/charting-m-stats-NetPoints.csv'
my.f.w &lt;- 'tennis_MatchChartingProject-master/charting-w-stats-NetPoints.csv'

# set cols
my.cols &lt;- cols(
  match_id = col_character(),
  player = col_integer(),
  row = col_character(),
  net_pts = col_integer(),
  pts_won = col_integer(),
  net_winner = col_integer(),
  induced_forced = col_integer(),
  net_unforced = col_integer(),
  passed_at_net = col_integer(),
  passing_shot_induced_forced = col_integer(),
  total_shots = col_integer()
)

# load data from csv
my.df.netpts &lt;- bind_rows(read_csv(my.f.m, col_types = my.cols) %>% mutate(Gender = 'Men'),
                          read_csv(my.f.w, col_types = my.cols) %>% mutate(Gender = 'Women'))

# get dates and years
my.df.netpts$Date &lt;- as.Date(my.df.netpts$match_id,'%Y%m%d')
my.df.netpts$Year &lt;- format(my.df.netpts$Date,'%Y')

# get surface
idx &lt;- match(my.df.netpts$match_id, my.df.matches$match_id)
my.df.netpts$Surface &lt;- my.df.matches$Surface[idx]

# plot it!
p &lt;- filter(my.df.netpts,!is.na(Surface)) %>%
  group_by(Year, Gender, Surface) %>%
  summarise(`Percentage of net points` = sum(net_pts)/sum(total_shots)) %>%
  ggplot(aes(x=as.numeric(Year),y=`Percentage of net points`, color = Gender)) +
  geom_point()+geom_smooth() + facet_wrap(~Surface)

print(p)
</code></pre>
</div>
<p><img alt="" src="https://i1.wp.com/msperlin.github.io/img/2017-02-05-R-and-Tennis_files/figure-markdown_strict/netpts-1.png?w=456" data-recalc-dims="1"></p>
<p>As expected, we find similar information from the previous analysis. Not<br>
much of a change in seen in clay courts after 2000. This is the opposite<br>
of hard courts, where the percentage of net point decreased dramatically<br>
after 2000. In this case, however, the different is much larger. The<br>
percentage of net points back in 2000 was around 35%. That decreased to<br>
approximately 20% in 2016. A steeper difference from the change in the<br>
proportion of aces seen before.</p>
<p>As for women tennis, we again don’t see a big change. This result also<br>
suggests that the new balls were not implemented in woman’s tennis.</p>
<h2>Conclusion</h2>
<p>In this post I showed some simple graphical analysis of the evolution of<br>
tennis over time. I find that the change in the tennis ball’s<br>
composition back in 2000 made a significant impact in the game,<br>
specially in grass and hard courts. The proportion of aces and net<br>
points changed significantly.</p>
<p>In the next post, scheduled for next week, I will make a data analysis<br>
of tennis players. I will focus first in the big names such as Federer<br>
and Djokovic and later will analyze the Brazilian scene and the records<br>
of the greatest Brazilian player (so far, i hope <img alt="🙂" class="wp-smiley" src="https://s.w.org/images/core/emoji/2/72x72/1f642.png"> ), <a target="_blank" rel="nofollow" href="https://www.google.com.br/search?q=tennis&newwindow=1&safe=off&source=lnms&tbm=isch&sa=X&ved=0ahUKEwjO6PrvqvbRAhUDRSYKHfR_AJkQ_AUICCgB&biw=1360&bih=648#newwindow=1&safe=off&tbm=isch&q=tennis+guga">Gustavo Kuerten<br>
–<br>
Guga</a>.<br>
As you can see, I am a big fan of him.</p>
<p>I hope you enjoyed this post. Fell free to leave your message at the<br>
comments.</p>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://msperlin.github.io/2017-02-05-R-and-Tennis/"> R and Finance</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/EegLKpG9lS0"></a></div>