This talk details the challenges addressed during the development of Zipline's new Pipeline API, which provides a high-level expression language allowing users to describe computations on rolling windows of continuously-adjusted financial data. We discuss the notion of "perspectival" time-series data, arguing that this concept provides a useful framework for formally reasoning about financial data in the face of domain oddities like stock splits, dividends, and restatements. We also consider the architectural and performance benefits of developing an API focused on symbolic computation, drawing comparisons to several recent developments in the Python numerical ecosystem.

8. Large
Problem
Medium
Problem
Medium
Problem
Medium
Problem
Small
Problem
Small
Problem
Small
Problem
Small
Problem
Small
Problem
Small
Problem
Small
Problem
Small
Problem

14. mean median first last stddev
rank() zscore()

15. factor1 >
factor2
& |
rank() percentile()

16. cross_product()

17. Factors Filters
Classifiers
(2)

21. a + a + a 3a

22. SQL
numpy

29. In [3]:
Out[3]:
from zipline.assets import AssetFinder
finder = AssetFinder("sqlite:///data/assets.db")
lifetimes = finder.lifetimes(
dates=pd.date_range('2001-01-01', '2015-10-01'),
include_start_date=True,
)
lifetimes.head(5)

30. In [4]: daily_count = lifetimes.sum(axis=1)
daily_count.plot(title="Companies in Existence by Day");

32. In [5]: AAPL_prices = pd.read_csv(
'data_public/AAPL-split.csv',
parse_dates=['Date'],
index_col='Date',
)
def plot_prices(prices):
price_plot = prices.plot(title='AAPL Price', grid=False)
price_plot.set_ylabel("Price", rotation='horizontal', labelpad=50)
price_plot.vlines(
['2014-05-08'], 0, 700,
label="$3.05 Dividend",
linestyles='dotted',
colors='black',
)
price_plot.vlines(
['2014-06-09'], 0, 700,
label="7:1 Split",
linestyles='--',
colors='black',
)
price_plot.legend()
sns.despine()
return price_plot

33. In [6]: plot_prices(AAPL_prices);

34. In [7]: naive_returns = AAPL_prices.pct_change()
naive_returns.plot();

49. In [8]: from bcolz import open
from humanize import naturalsize
all_prices = open('data/equity_daily_bars.bcolz')
min_offset = min(all_prices.attrs['calendar_offset'].itervalues())
max_offset = max(all_prices.attrs['calendar_offset'].itervalues())
calendar = pd.DatetimeIndex(all_prices.attrs['calendar'])[min_offset:max_offset]
nassets = len(lifetimes.columns)
ndates = len(calendar)
nfields = len(('id', 'open', 'high', 'low', 'close', 'volume', 'date'))
print "Number of Assets: %d" % nassets
print "Number of Dates: %d" % ndates
print "Naive Dataset Size: %s" % naturalsize(
nassets * ndates * nfields * 8
)
Number of Assets: 20353
Number of Dates: 3480
Naive Dataset Size: 4.0 GB

50. In [9]: !du -h -d0 data/equity_daily_bars.bcolz
!du -h -d0 data/adjustments.db
299M data/equity_daily_bars.bcolz
30M data/adjustments.db

51. In [9]: !du -h -d0 data/equity_daily_bars.bcolz
!du -h -d0 data/adjustments.db
299M data/equity_daily_bars.bcolz
30M data/adjustments.db

54. In [10]: import pandas as pd
from zipline.utils.tradingcalendar import trading_day
from zipline.pipeline.data import USEquityPricing
from zipline.pipeline.loaders import USEquityPricingLoader
loader = USEquityPricingLoader.from_files(
'data/equity_daily_bars.bcolz',
'data/adjustments.db'
)
dates = pd.date_range(
'2014-5-20',
'2014-06-30',
freq=trading_day,
tz='UTC',
)

55. In [11]:
Out[11]:
# load_adjusted_array() returns a dictionary mapping columns to instances of `Ad
justedArray`.
(closes,) = loader.load_adjusted_array(
columns=[USEquityPricing.close],
dates=dates,
assets=pd.Int64Index([24, 5061]),
mask=None,
).values()
closes
Adjusted Array:
Data:
array([[ 604.4 , 39.74],
[ 604.55, 39.69],
[ 606.28, 40.35],
...,
[ 90.35, 42.02],
[ 90.92, 41.73],
[ 91.96, 42.24]])
Adjustments:
{13: [Float64Multiply(first_row=0, last_row=13, first_col=0, last_col=0, value=0
.142860)]}

56. In [14]:
Out[14]:
dates_iter = iter(dates[4:])
window = closes.traverse(5)
window
_Float64AdjustedArrayWindow
Window Length: 5
Current Buffer:
[[ 604.4 39.74 ]
[ 604.55 39.69 ]
[ 606.28 40.35 ]
[ 607.33 40.105]
[ 614.14 40.12 ]]
Remaining Adjustments:
{13: [Float64Multiply(first_row=0, last_row=13, first_col=0, last_col=0, value=0
.142860)]}

57. In [15]: # This cell is run multiple times to show the numbers scrolling up until we hit
the split.
data = next(window)
print data
print next(dates_iter)
[[ 604.4 39.74 ]
[ 604.55 39.69 ]
[ 606.28 40.35 ]
[ 607.33 40.105]
[ 614.14 40.12 ]]
2014-05-27 00:00:00+00:00

59. float bool

60. dask