The document discusses making. It focuses on the process of creating or producing something. Some key aspects of making include coming up with an idea, gathering materials, and putting in the effort to turn ideas into a finished product or creation through hands-on work. The document emphasizes that making allows one to be creative and productive by turning thoughts into reality.

1. Making the Grade:
A look inside the algorithm evaluation process
Jessica Stauth, PhD - VP Quant Strategy

2. Quantopian CONFIDENTIAL AND PROPRIETARY - NOT FOR DISTRIBUTION WITHOUT THE WRITTEN CONSENT OF QUANTOPIANQuantopian
This presentation is for informational purposes only and does not constitute an offer to sell, a solicitation to
buy, or a recommendation for any security; nor does it constitute an offer to provide investment advisory or
other services by Quantopian, Inc. ("Quantopian"). Nothing contained herein constitutes investment advice or
offers any opinion with respect to the suitability of any security, and any views expressed herein should not
be taken as advice to buy, sell, or hold any security or as an endorsement of any security or company. In
preparing the information contained herein, Quantopian has not taken into account the investment needs,
objectives, and financial circumstances of any particular investor. Any views expressed and data illustrated
herein were prepared based upon information, believed to be reliable, available to Quantopian at the time of
publication. Quantopian makes no guarantees as to their accuracy or completeness. All information is subject
to change and may quickly become unreliable for various reasons, including changes in market conditions or
economic circumstances.
Disclaimer
2

3. Quantopian CONFIDENTIAL AND PROPRIETARY - NOT FOR DISTRIBUTION WITHOUT THE WRITTEN CONSENT OF QUANTOPIAN 3Quantopian
Investment
Vehicle

4. Quantopian CONFIDENTIAL AND PROPRIETARY - NOT FOR DISTRIBUTION WITHOUT THE WRITTEN CONSENT OF QUANTOPIAN 4Quantopian
Investment
Vehicle

5. Quantopian CONFIDENTIAL AND PROPRIETARY - NOT FOR DISTRIBUTION WITHOUT THE WRITTEN CONSENT OF QUANTOPIAN 5Quantopian
1. ALGORITHM IS BORN
Quantopian’s model validation process
includes a rigorous definition of “out-of
-sample” data as data accumulated
after the code was frozen. We typically
require six months of out-of-sample
data before considering an algorithm
for inclusion in the Fund.
The investment team reviews each
algorithm that meets our investment
criteria AND passes out-of-sample
validation.
Once an algorithm is selected the
author is invited to a formal diligence
process which includes completion
of:
▪ Due diligence questionnaire
▪ Video or phone interview(s)
▪ Background check
▪ IP license agreement
▪ Operational and Performance
checks
Upon successful completion this
package is reviewed and approved
(or not) by our CIO in a formal,
periodic investment committee
meeting.
Algorithm selection process
Code is version-controlled and time-
stamped in Quantopian’s database.
We extract over 50 features spanning
performance, risk, structure, and
author behavior.
2. TRUE OUT-OF-SAMPLE 3. AUTHOR DILIGENCE

6. So you think you have a great
algorithm?
CONFIDENTIAL AND PROPRIETARY - NOT FOR DISTRIBUTION WITHOUT THE WRITTEN CONSENT OF QUANTOPIAN

7. So you think you have a great
algorithm?
CONFIDENTIAL AND PROPRIETARY - NOT FOR DISTRIBUTION WITHOUT THE WRITTEN CONSENT OF QUANTOPIAN

8. Quantopian CONFIDENTIAL AND PROPRIETARY - NOT FOR DISTRIBUTION WITHOUT THE WRITTEN CONSENT OF QUANTOPIAN 8
Returns-Based Analysis - Example Codenamed Illinois
Spider
The illustration reflects the backtested performance of an algorithm for the period of January 4, 2010 through August 28, 2016. This algorithm is not and has never been traded by Quantopian in any capacity. Results do
not include estimated transaction costs.
Backtested performance has certain inherent limitations. For instance, such performance does not incorporate the impact that material economic and market factors might have had on the performance of the algorithm.
As a result, actual performance will vary from the results presented, and that variance may be material. Past performance is not necessarily reflective of future results.Please see the Appendix hereto for important
definitions of the Performance statistics illustrated above.

9. Quantopian CONFIDENTIAL AND PROPRIETARY - NOT FOR DISTRIBUTION WITHOUT THE WRITTEN CONSENT OF QUANTOPIAN 9
The illustration reflects the backtested performance of an algorithm for the period of January 4, 2010 through February 27, 2017. This algorithm is not and has never been traded by Quantopian in any capacity. Results
do not include estimated transaction costs.
Backtested performance has certain inherent limitations. For instance, such performance does not incorporate the impact that material economic and market factors might have had on the performance of the algorithm.
As a result, actual performance will vary from the results presented, and that variance may be material. Past performance is not necessarily reflective of future results.Please see the Appendix hereto for important
definitions of the Performance statistics illustrated above.
Returns-Based Analysis - Example Codenamed Illinois
Spider
Problem: Out of sample performance is not
living up to the in-sample expectation.

10. Quantopian CONFIDENTIAL AND PROPRIETARY - NOT FOR DISTRIBUTION WITHOUT THE WRITTEN CONSENT OF QUANTOPIAN
▪ Out of sample results that don’t match in-
sample results raise the concern of overfitting.
▪ Academic result: 6 months of ‘out of sample’
(daily) returns with a Sharpe ratio of 1.0 yields
a 75% probability of positive future Sharpe
ratio1.
10
1. Bailey, D. & Lopez de Prado, M. (2012) The Sharpe Ratio Efficient Frontier
http://ssrn.com/abstract=1821643

11. Quantopian CONFIDENTIAL AND PROPRIETARY - NOT FOR DISTRIBUTION WITHOUT THE WRITTEN CONSENT OF QUANTOPIAN 11
Returns-Based Analysis - Example Codenamed Vermont
Trout
The illustration reflects the backtested performance of an algorithm for the period of January 4, 2010 through February 27, 2017. This algorithm is not and has never been traded by Quantopian in any capacity. Results
do not include estimated transaction costs.
Backtested performance has certain inherent limitations. For instance, such performance does not incorporate the impact that material economic and market factors might have had on the performance of the algorithm.
As a result, actual performance will vary from the results presented, and that variance may be material. Past performance is not necessarily reflective of future results.Please see the Appendix hereto for important
definitions of the Performance statistics illustrated above.
Out of sample looks possibly in-line with in-
sample results and returns fall within the “cone
of expectation”

12. Quantopian CONFIDENTIAL AND PROPRIETARY - NOT FOR DISTRIBUTION WITHOUT THE WRITTEN CONSENT OF QUANTOPIAN 12
Problem: Algo is Net Long and Long Beta (as
high as 0.5 measured on a rolling single
factor regression).
The illustration reflects the backtested performance of an algorithm for the period of January 4, 2010 through February 27, 2017. This algorithm is not and has never been traded by Quantopian in any capacity. Results
do not include estimated transaction costs.
Backtested performance has certain inherent limitations. For instance, such performance does not incorporate the impact that material economic and market factors might have had on the performance of the algorithm.
As a result, actual performance will vary from the results presented, and that variance may be material. Past performance is not necessarily reflective of future results.Please see the Appendix hereto for important
definitions of the Performance statistics illustrated above.
Returns-Based Analysis - Example Codenamed Vermont
Trout

13. Quantopian CONFIDENTIAL AND PROPRIETARY - NOT FOR DISTRIBUTION WITHOUT THE WRITTEN CONSENT OF QUANTOPIAN 13
▪ Common factor risk, such a dollar, market, and sector
exposure do not diversify away with portfolio construction
and must be tightly controlled at the individual algorithm
level.
In some cases post-hoc hedging can be used to ‘correct’
common factor risk, but hedging comes at a cost and
degrades expected performance.
Ideally we look for algorithms that have low common factor
risk exposures by design and derive returns mostly from
stock selection.

14. Quantopian CONFIDENTIAL AND PROPRIETARY - NOT FOR DISTRIBUTION WITHOUT THE WRITTEN CONSENT OF QUANTOPIAN 14
Returns-Based Analysis - Example Codenamed Michigan
Hippopotamus
The illustration reflects the backtested performance of an algorithm for the period of January 4, 2010 through February 27, 2017. This algorithm is not and has never been traded by Quantopian in any capacity. Results
do not include estimated transaction costs.
Backtested performance has certain inherent limitations. For instance, such performance does not incorporate the impact that material economic and market factors might have had on the performance of the algorithm.
As a result, actual performance will vary from the results presented, and that variance may be material. Past performance is not necessarily reflective of future results.Please see the Appendix hereto for important
definitions of the Performance statistics illustrated above.
Out of sample looks plausibly in-line with in-
sample results and returns are within the “cone
of expectation”

15. Quantopian CONFIDENTIAL AND PROPRIETARY - NOT FOR DISTRIBUTION WITHOUT THE WRITTEN CONSENT OF QUANTOPIAN 15
The illustration reflects the backtested performance of an algorithm for the period of January 4, 2010 through February 27, 2017. This algorithm is not and has never been traded by Quantopian in any capacity. Results
do not include estimated transaction costs.
Backtested performance has certain inherent limitations. For instance, such performance does not incorporate the impact that material economic and market factors might have had on the performance of the algorithm.
As a result, actual performance will vary from the results presented, and that variance may be material. Past performance is not necessarily reflective of future results.Please see the Appendix hereto for important
definitions of the Performance statistics illustrated above.
Returns-Based Analysis - Example Codenamed Michigan
Hippopotamus
Problems:
- Beta exposure of ~ +0.25 - +0.50
- Substantial concentration in a single long
position
- Possibly a blend of two approaches

16. Quantopian CONFIDENTIAL AND PROPRIETARY - NOT FOR DISTRIBUTION WITHOUT THE WRITTEN CONSENT OF QUANTOPIAN 16
▪ Algorithms that combine more than one
style, e.g. in this case a long/short book and
a single timed long-only position are difficult
to evaluate without more context.
In order to proceed with diligence on this type
of algorithm we would likely need to work
collaboratively with the author to separate
the components and evaluate each on a
standalone basis.

17. Quantopian CONFIDENTIAL AND PROPRIETARY - NOT FOR DISTRIBUTION WITHOUT THE WRITTEN CONSENT OF QUANTOPIAN 17
Returns-Based Analysis - Example Codenamed Georgia
Pigeon
Quantopian
Summary Statistics
Cumulative Gross Returns
The illustration reflects the backtested performance of an algorithm for the period of January 4, 2010 through February 24, 2017. This algorithm is not and has never been traded by Quantopian in any capacity. Results
do not include estimated transaction costs.
Backtested performance has certain inherent limitations. For instance, such performance does not incorporate the impact that material economic and market factors might have had on the performance of the algorithm.
As a result, actual performance will vary from the results presented, and that variance may be material. Past performance is not necessarily reflective of future results.Please see the Appendix hereto for important
definitions of the Performance statistics illustrated above.
Out of sample looks quite consistent with in-sample
results and returns are within the “cone of expectation”

18. Quantopian CONFIDENTIAL AND PROPRIETARY - NOT FOR DISTRIBUTION WITHOUT THE WRITTEN CONSENT OF QUANTOPIAN 18Quantopian
The illustration reflects the backtested performance of an algorithm for the period of January 4, 2010 through February 24, 2017. This algorithm is not and has never been traded by Quantopian in any capacity. Results
do not include estimated transaction costs.
Backtested performance has certain inherent limitations. For instance, such performance does not incorporate the impact that material economic and market factors might have had on the performance of the algorithm.
As a result, actual performance will vary from the results presented, and that variance may be material. Past performance is not necessarily reflective of future results.Please see the Appendix hereto for important
definitions of the Performance statistics illustrated above.
Returns-Based Analysis - Example Codenamed Georgia
Pigeon
Rolling factor risk, Sharpe, and drawdown as well as calendarized returns all look good too.

19. Quantopian CONFIDENTIAL AND PROPRIETARY - NOT FOR DISTRIBUTION WITHOUT THE WRITTEN CONSENT OF QUANTOPIAN 19Quantopian
Problem: Algo derives substantial fraction of
its PNL from names outside of the Q1500
tradeable universe.
Limit universe to the Q1500 names…
The illustration reflects the backtested performance of an algorithm for the period of January 4, 2010 through February 24, 2017. This algorithm is not and has never been traded by Quantopian in any capacity. Results
do not include estimated transaction costs.
Backtested performance has certain inherent limitations. For instance, such performance does not incorporate the impact that material economic and market factors might have had on the performance of the algorithm.
As a result, actual performance will vary from the results presented, and that variance may be material. Past performance is not necessarily reflective of future results.Please see the Appendix hereto for important
definitions of the Performance statistics illustrated above.
Returns-Based Analysis - Example Codenamed Georgia
Pigeon
Net and single position exposures look well controlled and turnover of ~ 20% /day is ok…

20. Quantopian CONFIDENTIAL AND PROPRIETARY - NOT FOR DISTRIBUTION WITHOUT THE WRITTEN CONSENT OF QUANTOPIAN
Returns-Based Analysis - Example Codenamed Georgia
Pigeon
20
The illustration reflects the backtested performance of an algorithm for the period of January 4, 2010 through February 24, 2017. This algorithm is not and has never been traded by Quantopian in any capacity. Results
do not include estimated transaction costs.
Backtested performance has certain inherent limitations. For instance, such performance does not incorporate the impact that material economic and market factors might have had on the performance of the algorithm.
As a result, actual performance will vary from the results presented, and that variance may be material. Past performance is not necessarily reflective of future results.Please see the Appendix hereto for important
definitions of the Performance statistics illustrated above.
Summary Statistics
Cumulative Gross Returns
Cumulative Gross Returns
Limit to Q1500 Liquid Trade-ables

21. Quantopian CONFIDENTIAL AND PROPRIETARY - NOT FOR DISTRIBUTION WITHOUT THE WRITTEN CONSENT OF QUANTOPIAN 21
▪ Quant strategies can pick up on apparent
inefficiencies in stocks that are not
actually trade-able due to liquidity, borrow
availability, borrow cost, etc.
August 2016: Q1500 universe definition
was released as a single line API import:
from quantopian.pipeline.filters import Q1500US

22. Quantopian CONFIDENTIAL AND PROPRIETARY - NOT FOR DISTRIBUTION WITHOUT THE WRITTEN CONSENT OF QUANTOPIAN
▪ So, what have we learned here?
▪ Algorithms that don’t make the grade
can still be extremely valuable.
▪ Failure modes discovered in the
selection process are fed back into
product and educational content
development.
22

23. Quantopian CONFIDENTIAL AND PROPRIETARY - NOT FOR DISTRIBUTION WITHOUT THE WRITTEN CONSENT OF QUANTOPIAN 23
Education Materials developed with top academics

24. Quantopian CONFIDENTIAL AND PROPRIETARY - NOT FOR DISTRIBUTION WITHOUT THE WRITTEN CONSENT OF QUANTOPIAN 24Quantopian
Open Source

25. Quantopian CONFIDENTIAL AND PROPRIETARY - NOT FOR DISTRIBUTION WITHOUT THE WRITTEN CONSENT OF QUANTOPIAN 25
Thank you.
Questions?
jstauth@quantopian.com

26. Quantopian CONFIDENTIAL AND PROPRIETARY - NOT FOR DISTRIBUTION WITHOUT THE WRITTEN CONSENT OF QUANTOPIANQuantopian 26
Appendix

27. Quantopian CONFIDENTIAL AND PROPRIETARY - NOT FOR DISTRIBUTION WITHOUT THE WRITTEN CONSENT OF QUANTOPIAN 27Quantopian
alpha_spy is the annualized returns in excess of returns resulting from correlation with the S&P 500 index over the same period.
annual_return is the annualized returns.
annual_volatility is the annualized volatility of daily returns.
beta_spy is the correlation between daily returns and daily S&P 500 index returns.
beta_std is the standard deviation of rolling 6 month beta to the S&P 500.
calmar_ratio is the annualized returns divided by the maximum peak to trough drawdown.
common_sense_ratio is equal to the tail ratio multiplied by one plus the annual returns.
drawdown_area is the annualized area of drawdown periods (bounded by the high water mark and cumulative returns curve).
kurtosis is the kurtosis of daily returns distribution.
max_drawdown is the maximum peak to trough drawdown in the cumulative returns curve (on a percentage basis).
omega_ratio is calculated by dividing a portfolio’s cumulative positive returns by its cumulative negative returns.
sharpe_ratio is calculated by dividing a portfolio’s excess return by the standard deviation of its returns.
sharpe_ratio_last_year is the Sharpe ratio over the last year.
sharpe_std is the standard deviation of rolling 6 month Sharpe ratio
skewness is the skewness of daily returns distribution
sortino_ratio is the mean of the daily returns divided by the standard deviation of the positive values from the daily returns.
stability is the R-squared error of a linear fit to the cumulative log returns.
tail_ratio is the ratio between the 95th and (absolute) 5th percentile of the daily returns distribution.
Index of Definitions Used Herein