We are ready to demo our new new experimental package for Algorithmic Trading, flyingfox, which uses reticulate to to bring Quantopian’s open source algorithmic trading Python library, Zipline, to R. The flyingfox library is part of our NEW Business Science Labs innovation lab, which is dedicated to bringing experimental packages to our followers early on so they can test them out and let us know what they think before they make their way to CRAN. This article includes a long-form code tutorial on how to perform backtest optimizations of trading algorithms via grid search and parallel processing. In this article, we’ll show you how to use the combination of tibbletime (time-based extension of tibble) + furrr (a parallel-processing compliment to purrr) + flyingfox (Zipline in R) to develop a backtested trading algorithm that can be optimized via grid search and parallel processing. We are releasing this article as a compliment to the R/Finance Conference presentation “A Time Series Platform For The Tidyverse”, which Matt will present on Saturday (June 2nd, 2018). Enjoy!
New: Business Science Labs
I (Davis) am excited to introduce a new open source initiative called Business Science Labs. A lot of the experimental work we do is done behind the scenes, and much of it you don’t see early on. What you do see is a “refined” version of what we think you need based on our perception, which is not always reality. We aim to change this. Starting today, we have created Business Science Labs, which is aimed at bringing our experimental software to you earlier so you can test it out and let us know your thoughts!
Our first initiative is to bring Quantopian’s Open Source algorithmic trading Python library, Zipline, to R via an experimental package called flyingfox (built using the awesome reticulate package).
What We’re Going To Learn
Introducing Business Science Labs is exciting, but we really want to educate you on some new packages! In this tutorial, we are going to go over how to backtest algorithmic trading strategies using parallel processing and Quantopian’s Zipline infrastructure in R. You’ll gain exposure to a tibbletime, furrr, and our experimental flyingfox package. The general progression is:
tibbletime: What it is and why it’s essential to performing scalable time-based calculations in the tidyverse
furrr: Why you need to know this package for speeding up code by processing purrr in parallel
flyingfox: The story behind the package, and how you can use it to test algorithmic trading strategies
tibbletime + furrr + flyingfox: Putting it all together to perform parallelized algorithmic trading strategies and analyze time-based performance
Here’s an example of the grid search we perform to determine which are the best combinations of short and long moving averages for the stock symbol JPM (JP Morgan).
Here’s an example of the time series showing the order (buy/sell) points determined by the moving average crossovers, and the effect on the portfolio value.
Algorithmic Trading Strategies And Backtesting
Algorithmic trading is nothing new. Financial companies have been performing algorithmic trading for years as a way of attempting to “beat” the market. It can be very difficult to do, but some traders have successfully applied advanced algorithms to yield significant profits.
Using an algorithm to trade boils down to buying and selling. In the simplest case, when an algorithm detects an asset (a stock) is going to go higher, a buy order is placed. Conversely, when the algorithm detects that an asset is going to go lower, a sell order is placed. Positions are managed by buying and selling all or part of the portfolio of assets. To keep things simple, we’ll focus on just the full buy/sell orders.
One very basic method of algorithmic trading is using short and long moving averages to detect shifts in trend. The crossover is the point where a buy/sell order would take place. The figure below shows the price of Halliburton (symbol “HAL”), which a trader would have an initial position in of say 10,000 shares. In a hypothetical case, the trader could use a combination of a 20 day short moving average and a 150 day long moving average and look for buy/sell points at the crossovers. If the trader hypothetically sold his/her position in full on the sell and bought the position back in full, then the trader would stand to avoid a delta loss of approximately $5/share during the downswing, or $50,000.
Backtesting is a strategy that is used to detect how a trading strategy would have performed in the past. It’s impossible to know what the future will bring, but using trading strategies that work in the past helps to instill confidence in an algorithm.
Quantopian is a platform designed to enable anyone to develop algorithmic trading strategies. To help its community, Quantopian provides several open source tools. The one we’ll focus on is Zipline for backtesting. There’s one downside: it’s only available in Python.
With the advent of the reticulate package, which enables porting any Python library to R, we took it upon ourselves to test out the viability of porting Zipline to R. Our experiment is called flyingfox.
RStudio Cloud Experiment Sandbox
In this code-based tutorial, we’ll use an experimental package called flyingfox. It has several dependencies including Python that require setup time and effort. For those that want to test out flyingfox quickly, we’ve created a FREE RStudio Cloud Sandbox for running experiments. You can access the Cloud Sandbox here for FREE: https://rstudio.cloud/project/38291
Packages Needed For Backtest Optimization
The meat of this code-tutorial is the section Backtest Optimization Using tibbletime + furrr + flyingfox . However, before we get to it, we’ll go over the three main packages used to do high-performance backtesting optimizations:
tibbletime: What it is, and why it’s essential to performing scalable time-based calculations in the tidyverse
furrr: Why you need to know this package for speeding up code by processing purrr in parallel
flyingfox: How to use it to test algorithmic trading strategies
Putting It All Together: tibbletime + furrr + flyingfox for backtesting optimizations performed using parallel processing and grid search!
Install &amp; Load Libraries
Install Packages
For this post, you’ll need to install development version of flyingfox.
devtools::install_github(&quot;DavisVaughan/flyingfox&quot;)If you are on windows, you should also install the development version of furrr.
devtools::install_github(&quot;DavisVaughan/furrr&quot;)Other packages you’ll need include tibbletime, furrr, and tidyverse. We’ll also load tidyquant mainly for the ggplot2 themes. We’ll install ggrepel to repel overlapping plot labels. You can install these from CRAN using install.packages().
Load Packages
library(tidyverse)
library(tibbletime)
library(furrr)
library(flyingfox)
library(tidyquant)
library(ggrepel)We’ll cover how a few packages work before jumping into backtesting and optimizations.
1. tibbletime
The tibbletime package is a cornerstone for future time series software development efforts at Business Science. We have major plans for this package. Here are some key benefits:
Time periods down to milliseconds are supported
Because this is a tibble under the hood, we are able to leverage existing packages for analysis without reinventing the wheel
Scalable grouped analysis is at your fingertips because of collapse_by() and integration with group_by()
It’s best to learn now, and we’ll go over the basics along with a few commonly used functions: collapse_by(), rollify(), filter_time(), and as_period().
First, let’s get some data. We’ll use the FANG data set that comes with tibbletime, which includes stock prices for FB, AMZN, NFLX, and GOOG. We recommend using the tidyquant package to get this or other stock data.
data(&quot;FANG&quot;)
FANG## # A tibble: 4,032 x 8
## symbol date open high low close volume adjusted
## ## 1 FB 2013-01-02 27.4 28.2 27.4 28 69846400 28 ## 2 FB 2013-01-03 27.9 28.5 27.6 27.8 63140600 27.8
## 3 FB 2013-01-04 28.0 28.9 27.8 28.8 72715400 28.8
## 4 FB 2013-01-07 28.7 29.8 28.6 29.4 83781800 29.4
## 5 FB 2013-01-08 29.5 29.6 28.9 29.1 45871300 29.1
## 6 FB 2013-01-09 29.7 30.6 29.5 30.6 104787700 30.6
## 7 FB 2013-01-10 30.6 31.5 30.3 31.3 95316400 31.3
## 8 FB 2013-01-11 31.3 32.0 31.1 31.7 89598000 31.7
## 9 FB 2013-01-14 32.1 32.2 30.6 31.0 98892800 31.0
## 10 FB 2013-01-15 30.6 31.7 29.9 30.1 173242600 30.1
## # ... with 4,022 more rowsNext, you’ll need to convert thistbl_df object to a tbl_time object using the tbl_time() function.
FANG_time %
tbl_time(date) %__%
group_by(symbol)
FANG_time## # A time tibble: 4,032 x 8
## # Index: date
## # Groups: symbol [4]
## symbol date open high low close volume adjusted
## ## 1 FB 2013-01-02 27.4 28.2 27.4 28 69846400 28 ## 2 FB 2013-01-03 27.9 28.5 27.6 27.8 63140600 27.8
## 3 FB 2013-01-04 28.0 28.9 27.8 28.8 72715400 28.8
## 4 FB 2013-01-07 28.7 29.8 28.6 29.4 83781800 29.4
## 5 FB 2013-01-08 29.5 29.6 28.9 29.1 45871300 29.1
## 6 FB 2013-01-09 29.7 30.6 29.5 30.6 104787700 30.6
## 7 FB 2013-01-10 30.6 31.5 30.3 31.3 95316400 31.3
## 8 FB 2013-01-11 31.3 32.0 31.1 31.7 89598000 31.7
## 9 FB 2013-01-14 32.1 32.2 30.6 31.0 98892800 31.0
## 10 FB 2013-01-15 30.6 31.7 29.9 30.1 173242600 30.1
## # ... with 4,022 more rowscollapse_by()
Beautiful. Now we have a time-aware tibble. Let’s test out some functions. First, let’s take a look at collapse_by(), which is used for grouped operations. We’ll collapse by “year”, and calculate the average price for each of the stocks.
FANG_time %__%
collapse_by(period = &quot;year&quot;) %__%
group_by(symbol, date) %__%
summarise(mean_adj = mean(adjusted))## # A time tibble: 16 x 3
## # Index: date
## # Groups: symbol [?]
## symbol date mean_adj
## ## 1 AMZN 2013-12-31 298. ## 2 AMZN 2014-12-31 333. ## 3 AMZN 2015-12-31 478. ## 4 AMZN 2016-12-30 700. ## 5 FB 2013-12-31 35.5
## 6 FB 2014-12-31 68.8
## 7 FB 2015-12-31 88.8
## 8 FB 2016-12-30 117. ## 9 GOOG 2013-12-31 442. ## 10 GOOG 2014-12-31 561. ## 11 GOOG 2015-12-31 602. ## 12 GOOG 2016-12-30 743. ## 13 NFLX 2013-12-31 35.3
## 14 NFLX 2014-12-31 57.5
## 15 NFLX 2015-12-31 91.9
## 16 NFLX 2016-12-30 102.rollify()
Next, let’s take a look at rollify(). Remember the chart of Halliburton prices at the beginning. It was created using rollify(), which turns any function into a rolling function. Here’s the code for the chart. Notice how we create two rolling functions using mean() and supplying the appropriate window argument.
hal_tbl %
ggplot(aes(date, value, color = key, linetype = key)) +
geom_line() +
theme_tq() +
scale_color_manual(values = c(palette_light()[[1]], &quot;blue&quot;, &quot;red&quot;)) +
labs(title = &quot;Halliburton Asset Price (HAL)&quot;, subtitle = &quot;Strategy: Short MA = 20, Long MA = 150&quot;,
x = &quot;Date&quot;, y = &quot;Adjusted Stock Price&quot;) +
annotate(&quot;point&quot;, x = as_date(&quot;2017-04-07&quot;), y = 49.25, colour = &quot;red&quot;, size = 5) +
annotate(&quot;text&quot;, label = &quot;Sell Signal&quot;, x = as_date(&quot;2017-04-07&quot;), y = 49.25 + 2.5, color = &quot;red&quot;, hjust = 0) +
annotate(&quot;point&quot;, x = as_date(&quot;2017-10-09&quot;), y = 44.25, colour = &quot;blue&quot;, size = 5) +
annotate(&quot;text&quot;, label = &quot;Buy Signal&quot;, x = as_date(&quot;2017-10-09&quot;), y = 44.25 + 2.5, color = &quot;blue&quot;)
filter_time()
Let’s check out filter_time(), which enables easier subsetting of time-based indexes. Let’s redo the chart above, instead focusing in on sell and buy signals, which occur after February 2017. We can convert the previously stored hal_ma_tbl to a tbl_time object, group by the “key” column, and then filter using the time function format filter_time(&quot;2017-03-01&quot; ~ &quot;end&quot;). We then reuse the plotting code above.
hal_ma_time %
as_tbl_time(date) %__%
group_by(key) hal_ma_time %__%
filter_time(&quot;2017-03-01&quot; ~ &quot;end&quot;) %__%
ggplot(aes(date, value, color = key, linetype = key)) +
geom_line() +
theme_tq() +
scale_color_manual(values = c(palette_light()[[1]], &quot;blue&quot;, &quot;red&quot;)) +
labs(title = &quot;Halliburton Asset Price (HAL)&quot;, subtitle = &quot;2016 through 2017&quot;,
x = &quot;Date&quot;, y = &quot;Adjusted Stock Price&quot;) +
annotate(&quot;point&quot;, x = as_date(&quot;2017-04-07&quot;), y = 49.25, colour = &quot;red&quot;, size = 5) +
annotate(&quot;text&quot;, label = &quot;Sell Signal&quot;, x = as_date(&quot;2017-04-07&quot;), y = 49.25 + 2.5, color = &quot;red&quot;, hjust = 0) +
annotate(&quot;point&quot;, x = as_date(&quot;2017-10-09&quot;), y = 44.25, colour = &quot;blue&quot;, size = 5) +
annotate(&quot;text&quot;, label = &quot;Buy Signal&quot;, x = as_date(&quot;2017-10-09&quot;), y = 44.25 + 2.5, color = &quot;blue&quot;)
as_period()
We can use the as_period() function to change the periodicity to a less granular level (e.g. going from daily to monthly). Here we convert the HAL share prices from daily periodicity to monthly periodicity.
hal_ma_time %__%
as_period(period = &quot;monthly&quot;, side = &quot;end&quot;) %__%
ggplot(aes(date, value, color = key, linetype = key)) +
geom_line() +
theme_tq() +
scale_color_manual(values = c(palette_light()[[1]], &quot;blue&quot;, &quot;red&quot;)) +
labs(title = &quot;Halliburton Asset Price (HAL)&quot;, subtitle = &quot;Periodicity Changed To Monthly&quot;,
x = &quot;Date&quot;, y = &quot;Adjusted Stock Price&quot;)
Next, let’s check out a new package for parallel processing using purrr: furrr!
2. furrr
The furrr package combines the awesome powers of future for parallel processing with purrr for iteration. Let’s break these up into pieces by purrr, future, and furrr.
purrr
The purrr package is used for iteration over a number of different types of generic objects in R, including vectors, lists, and tibbles. The main function used is map(), which comes in several varieties (e.g. map(), map2(), pmap(), map_chr(), map_lgl(), map_df(), etc). Here’s a basic example to get the class() of the columns of the FANG_time variable. Using map() iterates over the columns of the data frame returning a list containing the contents of the function applied to each column.
FANG_time %__%
map(class)## $symbol
## [1] &quot;character&quot;
## ## $date
## [1] &quot;Date&quot;
## ## $open
## [1] &quot;numeric&quot;
## ## $high
## [1] &quot;numeric&quot;
## ## $low
## [1] &quot;numeric&quot;
## ## $close
## [1] &quot;numeric&quot;
## ## $volume
## [1] &quot;numeric&quot;
## ## $adjusted
## [1] &quot;numeric&quot;future
The future package enables parallel processing. Here are a few important points:
future is a unified interface for parallel programming in R.
You set a “plan” for how code should be executed, call future() with an expression to evaluate, and call value() to retrieve the value. The first future() call sends off the code to another R process and is non-blocking so you can keep running R code in this session. It only blocks once you call value().
Global variables and packages are automatically identified and exported for you!
Now, the major point: If you’re familiar with purrr, you can take advantage of future parallel processing with furrr!
furrr = future + purrr
furrr takes the best parts of purrr and combines it with the parallel processing capabilities of future to create a powerful package with an interface instantly familiar to anyone who has used purrr before. All you need to do is two things:
Setup a plan() such as plan(&quot;multiprocess&quot;)
Change map() (or other purrr function) to future_map() (or other compatible furrr function)
Every purrr function has a compatible furrr function. For example, map_df() has future_map_df(). Set a plan, and run future_map_df() and that is all there
is to it!
furrr Example: Multiple Moving Averages
Say you would like to process not a single moving average but multiple moving averages for a given data set. We can create a custom function, multi_roller(), that uses map_dfc() to iteratively map a rollify()-ed mean() based on a sequence of windows. Here’s the function and how it works when a user supplies a data frame, a column in the data frame to perform moving averages, and a sequence of moving averages.
multi_roller %
bind_cols(data, .)
return(ret_tbl)
}We can test the function out with the FB stock prices from FANG. We’ll ungroup, filter by FB, and select the important columns, then pass the data frame to the multi_roller() function with window = seq(10, 100, by = 10). Great, it’s working!
FANG_time %__%
ungroup() %__%
filter(symbol == &quot;FB&quot;) %__%
select(symbol, date, adjusted) %__%
multi_roller(adjusted, mean, window = seq(10, 100, by = 10), sep = &quot;ma_&quot;)## # A time tibble: 1,008 x 13
## # Index: date
## symbol date adjusted ma_10 ma_20 ma_30 ma_40 ma_50 ma_60
## ## 1 FB 2013-01-02 28 NA NA NA NA NA NA
## 2 FB 2013-01-03 27.8 NA NA NA NA NA NA
## 3 FB 2013-01-04 28.8 NA NA NA NA NA NA
## 4 FB 2013-01-07 29.4 NA NA NA NA NA NA
## 5 FB 2013-01-08 29.1 NA NA NA NA NA NA
## 6 FB 2013-01-09 30.6 NA NA NA NA NA NA
## 7 FB 2013-01-10 31.3 NA NA NA NA NA NA
## 8 FB 2013-01-11 31.7 NA NA NA NA NA NA
## 9 FB 2013-01-14 31.0 NA NA NA NA NA NA
## 10 FB 2013-01-15 30.1 29.8 NA NA NA NA NA
## # ... with 998 more rows, and 4 more variables: ma_70 ,
## # ma_80 , ma_90 , ma_100 We can apply this multi_roller() function to a nested data frame. Let’s try it on our FANG_time data set. We’ll select the columns of interest (symbol, date, and adjusted), group by symbol, and use the nest() function to get the data nested.
FANG_time_nested %
select(symbol, date, adjusted) %__%
group_by(symbol) %__%
nest() FANG_time_nested## # A tibble: 4 x 2
## symbol data ## ## 1 FB ## 2 AMZN ## 3 NFLX ## 4 GOOG Next, we can perform a rowwise map using the combination of mutate() and map(). We apply multi_roller() as an argument to map() along with data (variable being mapped), and the additional static arguments, col and window.
FANG_time_nested %__%
mutate(ma = map(data, multi_roller, col = adjusted, ..f = mean,
window = seq(10, 100, by = 10), sep = &quot;ma_&quot;)) %__%
select(-data) %__%
unnest()## # A time tibble: 4,032 x 13
## # Index: date
## symbol date adjusted ma_10 ma_20 ma_30 ma_40 ma_50 ma_60
## ## 1 FB 2013-01-02 28 NA NA NA NA NA NA
## 2 FB 2013-01-03 27.8 NA NA NA NA NA NA
## 3 FB 2013-01-04 28.8 NA NA NA NA NA NA
## 4 FB 2013-01-07 29.4 NA NA NA NA NA NA
## 5 FB 2013-01-08 29.1 NA NA NA NA NA NA
## 6 FB 2013-01-09 30.6 NA NA NA NA NA NA
## 7 FB 2013-01-10 31.3 NA NA NA NA NA NA
## 8 FB 2013-01-11 31.7 NA NA NA NA NA NA
## 9 FB 2013-01-14 31.0 NA NA NA NA NA NA
## 10 FB 2013-01-15 30.1 29.8 NA NA NA NA NA
## # ... with 4,022 more rows, and 4 more variables: ma_70 ,
## # ma_80 , ma_90 , ma_100 Great, we have our moving averages. But…
What if instead of 10 moving averages, we had 500? This would take a really long time to run on many stocks. Solution: Parallelize with furrr!
There are two ways we could do this since there are two maps:
Parallelize the map() internal to the multi_roller() function
Parallelize the rowwise map() applied to each symbol
We’ll choose the former (1) to show off furrr.
To make the multi_roller_parallel() function, copy the multi_roller() function and do 2 things:
Add plan(&quot;multiprocess&quot;) at the beginning
Change map_dfc() to future_map_dfc()
That’s it!
multi_roller_parallel %
bind_cols(data, .)
return(ret_tbl)
}In the previous rowwise map, switch out multi_roller() for multi_roller_parallel() and change the window = 2:500. Sit back and let the function run in parallel using each of your computer cores.
FANG_time_nested %__%
mutate(ma = map(data, multi_roller_parallel, col = adjusted, ..f = mean, window = 2:500, sep = &quot;ma_&quot;)) %__%
select(-data) %__%
unnest()## # A time tibble: 4,032 x 502
## # Index: date
## symbol date adjusted ma_2 ma_3 ma_4 ma_5 ma_6 ma_7
## ## 1 FB 2013-01-02 28 NA NA NA NA NA NA ## 2 FB 2013-01-03 27.8 27.9 NA NA NA NA NA ## 3 FB 2013-01-04 28.8 28.3 28.2 NA NA NA NA ## 4 FB 2013-01-07 29.4 29.1 28.6 28.5 NA NA NA ## 5 FB 2013-01-08 29.1 29.2 29.1 28.8 28.6 NA NA ## 6 FB 2013-01-09 30.6 29.8 29.7 29.5 29.1 28.9 NA ## 7 FB 2013-01-10 31.3 30.9 30.3 30.1 29.8 29.5 29.3
## 8 FB 2013-01-11 31.7 31.5 31.2 30.7 30.4 30.1 29.8
## 9 FB 2013-01-14 31.0 31.3 31.3 31.1 30.7 30.5 30.3
## 10 FB 2013-01-15 30.1 30.5 30.9 31.0 30.9 30.6 30.4
## # ... with 4,022 more rows, and 493 more variables: ma_8 ,
## # ma_9 , ma_10 , ma_11 , ma_12 , ma_13 ,
## # ma_14 , ma_15 , ma_16 , ma_17 , ma_18 ,
## # ma_19 , ma_20 , ma_21 , ma_22 , ma_23 ,
## # ma_24 , ma_25 , ma_26 , ma_27 , ma_28 ,
## # ma_29 , ma_30 , ma_31 , ma_32 , ma_33 ,
## # ma_34 , ma_35 , ma_36 , ma_37 , ma_38 ,
## # ma_39 , ma_40 , ma_41 , ma_42 , ma_43 ,
## # ma_44 , ma_45 , ma_46 , ma_47 , ma_48 ,
## # ma_49 , ma_50 , ma_51 , ma_52 , ma_53 ,
## # ma_54 , ma_55 , ma_56 , ma_57 , ma_58 ,
## # ma_59 , ma_60 , ma_61 , ma_62 , ma_63 ,
## # ma_64 , ma_65 , ma_66 , ma_67 , ma_68 ,
## # ma_69 , ma_70 , ma_71 , ma_72 , ma_73 ,
## # ma_74 , ma_75 , ma_76 , ma_77 , ma_78 ,
## # ma_79 , ma_80 , ma_81 , ma_82 , ma_83 ,
## # ma_84 , ma_85 , ma_86 , ma_87 , ma_88 ,
## # ma_89 , ma_90 , ma_91 , ma_92 , ma_93 ,
## # ma_94 , ma_95 , ma_96 , ma_97 , ma_98 ,
## # ma_99 , ma_100 , ma_101 , ma_102 ,
## # ma_103 , ma_104 , ma_105 , ma_106 ,
## # ma_107 , ...Bam! 500 moving averages run in parallel in fraction of the time it would take running in series.
3. flyingfox
We have one final package we need to demo prior to jumping into our Algorithmic Trading Backtest Optimization: flyingfox.
What is Quantopian?
Quantopian is a company that has setup a community-driven platform for everyone (from traders to home-gamers) enabling development of algorithmic trading strategies. The one downside is they only use Python.
What is Zipline?
Zipline is a Python module open-sourced by Quantopian to help traders back-test their trading algorithms. Here are some quick facts about Quantopian’s Zipline Python module for backtesting algorithmic trading strategies:
It is used to develop and backtest financial algorithms using Python.
It includes an event-driven backtester (really good at preventing look-ahead bias)
Algorithms consist of two main functions:
initialize(): You write an initialize() function that is called once at the beginning of the backtest. This sets up variables for use in the backtest, schedules functions to be called daily, monthly, etc, and let’s you set slippage or commission for the backtest.
handle_data(): You then write a handle_data() function that is called once per day (or minute) that implements the trading logic. You can place orders, retrieve historical pricing data, record metrics for performance evalutation and more.
Extra facts: You can use any Python module inside the handle_data() function, so you have a lot of flexibility here.
What is reticulate?
The reticulate package from RStudio is an interface with Python. It smartly takes care of (most) conversion between R and Python objects.
Can you combine them?
Yes, and that’s exactly what we did. We used reticulate to access the Zipline Python module from R!
What is the benefit to R users?
What if you could write your initialize() and handle_data() functions in R utilizing any financial or time series R package for your analysis and then have them called from Python and Zipline?
Introducing flyingfox: An R interface to Zipline
flyingfox integrates the Zipline backtesting module in R! Further, it takes care of the overhead with creating the main infrastructure functions initialize() and handle_data() by enabling the user to set up:
fly_initialize(): R version of Zipline’s initialize()
fly_handle_data(): R version of Zipline’s handle_data()
flyingfox takes care of passing these functions to Python and Zipline.
Why “Flying Fox”?
Zipline just doesn’t quite make for a good hex sticker. A flying fox is a synonym for zipliners, and it’s hard to argue that this majestic animal wouldn’t create a killer hex sticker.
Getting Started With flyingfox: Moving Average Crossover
Let’s do a Moving Average Crossover example using the following strategy:
Using JP Morgan (JPM) stock prices
If the 20 day short-term moving average crosses above the 150 day long-term moving average, buy 100% into JP Morgan
If 20 day crosses below the 150 day, sell all of the current JPM position
Setup
Setup can take a while and take up some computer space due to ingesting data (which is where Zipline saves every major asset to your computer). We recommend one of two options:
No weight option (for people that just want to try it out): Use our flyingfox sandbox on RStudio Cloud. You can connect here: https://rstudio.cloud/project/38291
Heavy weight option (for people that want to expand and really test it): Follow the instructions on my GitHub page to fly_ingest() data. The setup and data ingesting process are discussed here: https://github.com/DavisVaughan/flyingfox. Keep in mind that this is still a work in progress. We recommend doing the no weight option as a first start.
Initialize
First, write the R function for initialize(). It must take context as an argument. This is where you store variables used later, which are accessed via context$variable. We’ll store context$i = 0L to initialize the tracking of days, and context$asset = fly_symbol(&quot;JPM&quot;) to trade the JPM symbol. You can select any symbol that you’d like (provided Quantopian pulls it from Quandl).
fly_initialize %
pull(portfolio_value) %__%
tail(1)## [1] 9187.849We can turn this into a function and map it to all of the columns to obtain the “final_portfolio_value” for each of the grid search combinations.
results_tbl %
mutate(final_portfolio_value = map_dbl(results, ~ pull(.x, portfolio_value) %__% tail(1)))
results_tbl## # A tibble: 20 x 5
## short_ma long_ma f results final_portfolio_value
## ## 1 20 150 9188.
## 2 40 150 9277.
## 3 60 150 9823.
## 4 80 150 10816.
## 5 100 150 10697.
## 6 20 200 9065.
## 7 40 200 10612.
## 8 60 200 11444.
## 9 80 200 11366.
## 10 100 200 11473.
## 11 20 250 9522.
## 12 40 250 10898.
## 13 60 250 11494.
## 14 80 250 11494.
## 15 100 250 11494.
## 16 20 300 10951.
## 17 40 300 11494.
## 18 60 300 11494.
## 19 80 300 11494.
## 20 100 300 11494.Visualizing The Backtest Performance Results
Now let’s visualize the results to see which combinations of short and long moving averages maximize the portfolio value. It’s clear that short __= 60 days and long __= 200 days maximize the return. But, why?
results_tbl %__%
ggplot(aes(long_ma, short_ma, color = final_portfolio_value)) +
geom_point(aes(size = final_portfolio_value)) +
geom_label_repel(
aes(label = scales::dollar(round(final_portfolio_value, 0))), vjust = -1, color = palette_light()[[1]]) +
theme_tq() +
scale_color_gradient(low = palette_light()[[1]], high = palette_light()[[3]]) +
scale_size(range = c(1, 12)) +
labs(
title = &quot;JPM Strategy: Backtest Grid Search Optimization&quot;,
subtitle = &quot;Short and Fast Moving Averages: Optimum = 60 days (Short MA), 300 Days (Long MA)&quot;,
x = &quot;Long Moving Average (Days)&quot;, y = &quot;Short Moving Average (Days)&quot;,
color = &quot;Final Portfolio Value&quot;, size = &quot;Final Portfolio Value&quot;
) +
theme(legend.direction = &quot;vertical&quot;,
legend.position = &quot;right&quot;)
Let’s get the transaction information (buy/sell) by unnesting the results and determining which transactions are buys and sells.
transactions_tbl %
unnest(results) %__%
select(short_ma:date, portfolio_value, transactions) %__%
mutate(flag_transactions = map_lgl(transactions, is.tibble)) %__%
filter(flag_transactions == TRUE) %__%
unnest() %__%
mutate(transaction_type = ifelse(amount __= 0, &quot;buy&quot;, &quot;sell&quot;)) %__%
mutate(
short_ma = as.factor(short_ma) %__% fct_rev(),
long_ma = as.factor(long_ma)
)
transactions_tbl %__%
glimpse()## Observations: 105
## Variables: 13
## $ short_ma 20, 20, 20, 20, 20, 20, 20, 20, 20,...
## $ long_ma 150, 150, 150, 150, 150, 150, 150, ...
## $ final_portfolio_value 9187.849, 9187.849, 9187.849, 9187....
## $ date 2014-03-13 20:00:00, 2014-03-14 20...
## $ portfolio_value 9994.890, 9888.191, 9404.815, 9400....
## $ flag_transactions TRUE, TRUE, TRUE, TRUE, TRUE, TRUE,...
## $ commission NA, NA, NA, NA, NA, NA, NA, NA, NA,...
## $ amount 172, 2, -174, 161, 2, -163, 147, -1...
## $ sid [...
## $ order_id &quot;d8a3551cb36c4ff985c42e4d78e00ef8&quot;,...
## $ price 57.44871, 56.82840, 54.02298, 57.44...
## $ dt 2014-03-13 21:00:00, 2014-03-14 21...
## $ transaction_type &quot;buy&quot;, &quot;buy&quot;, &quot;sell&quot;, &quot;buy&quot;, &quot;buy&quot;,...Finally, we can visualize the portfolio value over time for each combination of short and long moving averages. By plotting the buy/sell transactions, we can see the effect on a stock with a bullish trend. The portfolios with the optimal performance are those that were bought and held rather than sold using the moving average crossover. For this particular stock, the benefit of downside protection via the moving average crossover costs the portfolio during the bullish uptrend.
results_tbl %__%
unnest(results) %__%
mutate(
short_ma = as.factor(short_ma) %__% fct_rev(),
long_ma = as.factor(long_ma)
) %__%
group_by(short_ma, long_ma) %__%
filter_time(&quot;2014&quot; ~ &quot;end&quot;) %__%
ggplot(aes(date, portfolio_value)) +
geom_line(color = palette_light()[[1]]) +
geom_vline(aes(xintercept = date), color = &quot;blue&quot;,
data = filter(transactions_tbl, transaction_type == &quot;buy&quot;)) +
geom_vline(aes(xintercept = date), color = &quot;red&quot;,
data = filter(transactions_tbl, transaction_type == &quot;sell&quot;)) +
facet_grid(short_ma ~ long_ma, labeller = label_both) +
theme_tq() +
scale_y_continuous(labels = scales::dollar) +
scale_x_datetime(date_labels = &quot;%Y&quot;, date_breaks = &quot;1 year&quot;) +
labs(
title = &quot;JPM Strategy: Portfolio Value Over Time With Transactions&quot;,
subtitle = &quot;Bull Market: Buy and Hold Strategy is Optimal Vs Short/Long Moving Average Strategies&quot;,
x = &quot;Date&quot;, y = &quot;Portfolio Value&quot;
)
Conclusions
We’ve covered a lot of ground in this article. Congrats if you’ve made it through. You’ve now been exposed to three cool packages:
tibbletime: For time-series in the tidyverse
furrr: Our parallel processing extension to purrr
flyingfox: Our experimental package brought to you as part of our Business Science Labs initiative
Further, you’ve seen how to apply all three of these packages to perform grid search backtest optimization of your trading algorithm.
Business Science University If you are looking to take the next step and learn Data Science For Business (DS4B), you should consider Business Science University. Our goal is to empower data scientists through teaching the tools and techniques we implement every day. You’ll learn:
Data Science Framework: Business Science Problem Framework
Tidy Eval
H2O Automated Machine Learning
LIME Feature Explanations
Sensitivity Analysis
Tying data science to financial improvement
All while solving a REAL WORLD CHURN PROBLEM: Employee Turnover!
Get 15% OFF in June!
DS4B Virtual Workshop: Predicting Employee Attrition Did you know that an organization that loses 200 high performing employees per year is essentially losing $15M/year in lost productivity? Many organizations don’t realize this because it’s an indirect cost. It goes unnoticed. What if you could use data science to predict and explain turnover in a way that managers could make better decisions and executives would see results? You will learn the tools to do so in our Virtual Workshop. Here’s an example of a Shiny app you will create.
Get 15% OFF in June!
Shiny App That Predicts Attrition and Recommends Management Strategies, Taught in HR 301
Our first Data Science For Business Virtual Workshop teaches you how to solve this employee attrition problem in four courses that are fully integrated:
HR 201: Predicting Employee Attrition with h2o and lime
HR 301 (Coming Soon): Building A Shiny Web Application
HR 302 (EST Q4): Data Story Telling With RMarkdown Reports and Presentations
HR 303 (EST Q4): Building An R Package For Your Organization, tidyattrition
The Virtual Workshop is intended for intermediate and advanced R users. It’s code intensive (like these articles), but also teaches you fundamentals of data science consulting including CRISP-DM and the Business Science Problem Framework. The content bridges the gap between data science and the business, making you even more effective and improving your organization in the process.
Get 15% OFF in June!
Don’t Miss A Beat
Sign up for the Business Science blog to stay updated
Enroll in Business Science University to learn how to solve real-world data science problems from Business Science
Check out our Open Source Software
Connect With Business Science If you like our software (anomalize, tidyquant, tibbletime, timetk, and sweep), our courses, and our company, you can connect with us:
business-science on GitHub
Business Science, LLC on LinkedIn
bizScienc on twitter
Business Science, LLC on Facebook