Today, we continue our work on sampling so that we can run models on subsets of our data and then test the accuracy of the models on data not included in those subsets. In the machine learning prediction world, these two data sets are often called training data and testing data, but we’re not going to do any machine learning prediction today. We’ll stay with our good’ol Fama French regression models for the reasons explained last time: the goal is to explore a new method of sampling our data and I prefer to do that in the context of a familiar model and data set. In 2019, we’ll start expanding our horizons to different models and data, but for today it’s more of a tools exploration.
Loyal readers of this blog (if that’s you, thank you!) will remember that we undertook a similar project in the previous post, when we used k-fold cross-validation. Today, we will use rolling origin sampling of the data, which differs from k-fold cross-validation in the sense that with rolling origin we explicitly sample based on the dates of our observation. With rolling origin, our first sample starts on the first day, our second sample starts on the second day, our third sample starts on third day. Or, we could have the second sample start on the twentieth day, or we could have it start again on the first day and just include an extra day. Either way, we are aware of the order of our data when sampling. With k-fold cross-validation, we were randomly shuffling and then selecting observations. This distinction can be particularly important for economic time series where we believe that the order of our observations is important.
Let’s get to it.
First, we need our data and, as usual, we’ll import data for daily prices of 5 ETFs, convert them to returns (have a look here for a refresher on that code flow), then import the 5 Fama French factor data and join it to our 5 ETF returns data. Here’s the code to make that happen (this code was covered in detail in this post:
library(tidyverse)
library(tidyquant)
library(rsample)
library(broom)
library(timetk)
symbols % reduce(merge) %__%
`colnames%
gather(asset, returns, -date) %__% group_by(asset) %__% mutate(returns = (log(returns) - log(lag(returns)))) %__% na.omit()
factors_data_address % select(-RF)
data_joined_tidy %
left_join(Global_5_Factors, by = &quot;date&quot;) %__% na.omit()
After running that code, we have an object called data_joined_tidy. It holds daily returns for 5 ETFs and the Fama French factors. Here’s a look at the first row for each ETF rows.
data_joined_tidy %__% slice(1)
# A tibble: 5 x 8
# Groups: asset [5]
date asset returns MKT SMB HML RMW CMA
1 2013-01-02 AGG -0.00117 0.0199 -0.0043 0.0028 -0.0028 -0.0023
2 2013-01-02 EEM 0.0194 0.0199 -0.0043 0.0028 -0.0028 -0.0023
3 2013-01-02 EFA 0.0154 0.0199 -0.0043 0.0028 -0.0028 -0.0023
4 2013-01-02 IJS 0.0271 0.0199 -0.0043 0.0028 -0.0028 -0.0023
5 2013-01-02 SPY 0.0253 0.0199 -0.0043 0.0028 -0.0028 -0.0023
For today, let’s work with just the daily returns of EEM.
eem_for_roll % filter(asset == &quot;EEM&quot;)
We are going to regress those EEM daily returns on the Fama French factors and need a way to measure the accuracy of our various factor models. Previously, we measured our models by looking at the adjusted R-squared, when the models were run on the entirety of the data. Last time, we use k-fold cross-validation to split the data into a bunch of subsets, then ran our model on the subsets and measured the accuracy against the holdouts from the subsets. Now, let’s use the rolling_origin() function from rsample to split our data into analysis and assessment sets in a time-aware way, then calculate RMSEs.
The rolling_origin() function needs a few arguments set. We first set data to be eem_for_roll Then, we assign initial to be 50 - this tells the function that the size of our first sample is 50 days. Our first chunk of analysis data will be the first 50 days of EEM returns. Next, we assign assess to be 5 - this tells the function that our assessment data is the 5 days of EEM returns following those first 50 days. Finally, we set cumulative to be FALSE - this tells the functions that each of splits is a 50 days. The first split is the first 50 days, starting on day 1 and ending on day 50. The next split starts on day 2 and ends on day 51. If we were to set cumulative = TRUE, the first split would be 50 days. The next split would be 51 days, the next split would be 52 days. And so on. The analysis split days would be accumulating. For now, we will leave it at cumulative = FALSE.
For that reason, we will append _sliding to the name of our object because the start of our window will slide each time.
library(rsample)
roll_eem_sliding % slice(c(1,50))
# A tibble: 2 x 8
# Groups: asset [1]
date asset returns MKT SMB HML RMW CMA
1 2013-01-02 EEM 0.0194 0.0199 -0.0043 0.0028 -0.0028 -0.0023
2 2013-03-14 EEM 0.00418 0.0078 0.0021 0.0016 -0.002 0.0026
And we know split 2 begins on day 2 and ends on day 51:
analysis(roll_eem_sliding$splits[[2]]) %__% slice(c(1,50))
# A tibble: 2 x 8
# Groups: asset [1]
date asset returns MKT SMB HML RMW CMA
1 2013-01-03 EEM -0.00710 -0.0021 0.00120 0.000600 0.0008 0.0013
2 2013-03-15 EEM -0.00908 0.0025 0.006 -0.0001 0.0018 -0.0001
Now, we can start using our data to fit and then assess our model. The code flow is very similar to our previous post, but I’ll go ahead and run through it anyway.
First, let’s create a function that takes a split as an argument, fits a 3-factor Fama French model and calculates the root mean squared error. We will use the rmse() function from yardstick to measure our model. RMSE stands for root mean-squared error. It’s the sum of the squared differences between our fitted values and the actual values in the assessment data. A lower RMSE is better!
library(yardstick)
ff_three_rmse % head()
# A tibble: 6 x 3
splits id model
* 1 Slice0001 0.00531
2 Slice0002 0.00621
3 Slice0003 0.00642
4 Slice0004 0.00647
5 Slice0005 0.00654
6 Slice0006 0.00649
This is the same as we did last time. Now, let’s get functional. We will define three models that we wish to test with via our rolling origin splits and RMSE, then pass those models to one function.
First, we use as.formula() to define our three models.
mod_form_1 %
select(start_date, everything())
rmse_3_models_with_start_date %__% head()
# A tibble: 6 x 6
start_date splits id model_1_rmse model_2_rmse model_3_rmse
* 1 2013-03-15 Slice… 0.00601 0.00531 0.00496
2 2013-03-18 Slice… 0.00548 0.00621 0.00596
3 2013-03-19 Slice… 0.00547 0.00642 0.00617
4 2013-03-20 Slice… 0.00556 0.00647 0.00615
5 2013-03-21 Slice… 0.00544 0.00654 0.00613
6 2013-03-22 Slice… 0.00539 0.00649 0.00600
We can head to ggplot for some visualizing. I’d like to plot all of my RMSE’s in different colors and the best way to do that is to gather() this data to tidy format, with a column called model and a column called value. It’s necessary to coerce to a data frame first, using as.data.frame().
rmse_3_models_with_start_date %__%
as.data.frame() %__% select(-splits, -id) %__% gather(model, value, -start_date) %__% head()
start_date model value
1 2013-03-15 model_1_rmse 0.006011868
2 2013-03-18 model_1_rmse 0.005483091
3 2013-03-19 model_1_rmse 0.005470834
4 2013-03-20 model_1_rmse 0.005557170
5 2013-03-21 model_1_rmse 0.005439921
6 2013-03-22 model_1_rmse 0.005391862
Next, we can use some of our familiar ggplot methods to plot our RMSEs over time, and see if we notice this model degrading or improving in different periods.
rmse_3_models_with_start_date %__%
as.data.frame() %__% select(-splits, -id) %__% gather(model, value, -start_date) %__% ggplot(aes(x = start_date, y = value, color = model)) +
geom_point() +
facet_wrap(~model, nrow = 2) +
scale_x_date(breaks = scales::pretty_breaks(n = 10)) +
theme(axis.text.x = element_text(angle = 45, vjust = 1, hjust=1))
All of the models have a jump in RMSE, meaning they performed worse, around the end of 2017. We aren’t focused on the theory of these models today but if we were, this would be a good place to start investigating. This is just the beginning of our exploration of rolling_origin, but I love how it opens the door for ways to think about visualizing our models.
And finally, those same public service announcements from last time are still live, so I’ll mention them one more time.
Thanks to everyone who has checked out my new book! The price just got lowered for the holidays. See on Amazon or on the CRC homepage (okay, that was more of an announcement about my book).
Applications are open for the Battlefin alternative data contest and RStudio is one of the tools you can use to analyze the data. Check it out here. They’ll announce 25 finalists in January who will get to compete for a cash prize and connect with some quant hedge funds. Go get’em!
A special thanks to Bruce Fox who suggested we might want to expand on the Monte Carlo simulation in the book to take account of different distributions implied by historical returns and different market regimes that might arise. Today’s rolling origin framework will also lay the foundation, I hope, for implementing some of Bruce’s ideas in January.
Thanks for reading and see you next time.