Bradlee Speice

Event Studies and Earnings Releases

2016-06-08T00:00:00-04:00

Or, being suspicious of market insiders.

Use the button below to show the code I've used to generate this article. Because there is a significant amount more code involved than most other posts I've written, it's hidden by default to allow people to concentrate on the important bits.

In [1]:

from IPython.display import HTML

HTML('''<script>
code_show=true; 
function code_toggle() {
 if (code_show){
 $('div.input').hide();
 } else {
 $('div.input').show();
 }
 code_show = !code_show
} 
$( document ).ready(code_toggle);
</script>
<form action="javascript:code_toggle()"><input type="submit" value="Click here to toggle on/off the raw code."></form>''')

Out[1]:

The Market Just Knew¶

I recently saw two examples of stock charts that have kept me thinking for a while. And now that the semester is complete, I finally have enough time to really look at them and give them the treatment they deserve. The first is good old Apple:

In [2]:

from secrets import QUANDL_KEY
import matplotlib.pyplot as plt
from matplotlib.dates import date2num
from matplotlib.finance import candlestick_ohlc
from matplotlib.dates import DateFormatter, WeekdayLocator,\
    DayLocator, MONDAY
import quandl
from datetime import datetime
import pandas as pd
%matplotlib inline

def fetch_ticker(ticker, start, end):
    # Quandl is currently giving me issues with returning
    # the entire dataset and not slicing server-side.
    # So instead, we'll do it client-side!
    q_format = '%Y-%m-%d'
    ticker_data = quandl.get('YAHOO/' + ticker,
                             start_date=start.strftime(q_format),
                             end_date=end.strftime(q_format),
                             authtoken=QUANDL_KEY)
    return ticker_data

def ohlc_dataframe(data, ax=None):
    # Much of this code re-used from:
    # http://matplotlib.org/examples/pylab_examples/finance_demo.html
    if ax is None:
        f, ax = plt.subplots()
    
    vals = [(date2num(date), *(data.loc[date]))
            for date in data.index]
    candlestick_ohlc(ax, vals)
    
    mondays = WeekdayLocator(MONDAY)
    alldays = DayLocator()
    weekFormatter = DateFormatter('%b %d')
    ax.xaxis.set_major_locator(mondays)
    ax.xaxis.set_minor_locator(alldays)
    ax.xaxis.set_major_formatter(weekFormatter)
    return ax

AAPL = fetch_ticker('AAPL', datetime(2016, 3, 1), datetime(2016, 5, 1))
ax = ohlc_dataframe(AAPL)
plt.vlines(date2num(datetime(2016, 4, 26, 12)),
           ax.get_ylim()[0], ax.get_ylim()[1],
           color='b',
          label='Earnings Release')
plt.legend(loc=3)
plt.title("Apple Price 3/1/2016 - 5/1/2016");

The second chart is from Facebook:

In [3]:

FB = fetch_ticker('FB', datetime(2016, 3, 1), datetime(2016, 5, 5))
ax = ohlc_dataframe(FB)
plt.vlines(date2num(datetime(2016, 4, 27, 12)),
           ax.get_ylim()[0], ax.get_ylim()[1],
           color='b', label='Earnings Release')
plt.title('Facebook Price 3/5/2016 - 5/5/2016')
plt.legend(loc=2);

These two charts demonstrate two very specific phonomena: how the market prepares for earnings releases. Let's look at those charts again, but with some extra information. As we're about the see, the market "knew" in advance that Apple was going to perform poorly. The market expected that Facebook was going to perform poorly, and instead shot the lights out. Let's see that trend in action:

In [4]:

def plot_hilo(ax, start, end, data):
    ax.plot([date2num(start), date2num(end)],
            [data.loc[start]['High'], data.loc[end]['High']],
            color='b')
    ax.plot([date2num(start), date2num(end)],
            [data.loc[start]['Low'], data.loc[end]['Low']],
            color='b')

f, axarr = plt.subplots(1, 2)

ax_aapl = axarr[0]
ax_fb = axarr[1]

# Plot the AAPL trend up and down
ohlc_dataframe(AAPL, ax=ax_aapl)
plot_hilo(ax_aapl, datetime(2016, 3, 1), datetime(2016, 4, 15), AAPL)
plot_hilo(ax_aapl, datetime(2016, 4, 18), datetime(2016, 4, 26), AAPL)
ax_aapl.vlines(date2num(datetime(2016, 4, 26, 12)),
               ax_aapl.get_ylim()[0], ax_aapl.get_ylim()[1],
               color='g', label='Earnings Release')
ax_aapl.legend(loc=2)
ax_aapl.set_title('AAPL Price History')

# Plot the FB trend down and up
ohlc_dataframe(FB, ax=ax_fb)
plot_hilo(ax_fb, datetime(2016, 3, 30), datetime(2016, 4, 27), FB)
plot_hilo(ax_fb, datetime(2016, 4, 28), datetime(2016, 5, 5), FB)
ax_fb.vlines(date2num(datetime(2016, 4, 27, 12)),
             ax_fb.get_ylim()[0], ax_fb.get_ylim()[1],
             color='g', label='Earnings Release')
ax_fb.legend(loc=2)
ax_fb.set_title('FB Price History')

f.set_size_inches(18, 6)

As we can see above, the market broke a prevailing trend on Apple in order to go down, and ultimately predict the earnings release. For Facebook, the opposite happened. While the trend was down, the earnings were fantastic and the market corrected itself much higher.

Formulating the Question¶

While these are two specific examples, there are plenty of other examples you could cite one way or another. Even if the preponderance of evidence shows that the market correctly predicts earnings releases, we need not accuse people of collusion; for a company like Apple with many suppliers we can generally forecast how Apple has done based on those same suppliers.

The question then, is this: how well does the market predict the earnings releases? It's an incredibly broad question that I want to disect in a couple of different ways:

Given a stock that has been trending down over the past N days before an earnings release, how likely does it continue downward after the release?
Given a stock trending up, how likely does it continue up?
Is there a difference in accuracy between large- and small-cap stocks?
How often, and for how long, do markets trend before an earnings release?

I want to especially thank Alejandro Saltiel for helping me retrieve the data. He's great. And now for all of the interesting bits.

Event Studies¶

Before we go too much further, I want to introduce the actual event study. Each chart intends to capture a lot of information and present an easy-to-understand pattern:

In [5]:

import numpy as np
import pandas as pd
from pandas.tseries.holiday import USFederalHolidayCalendar
from pandas.tseries.offsets import CustomBusinessDay
from datetime import datetime, timedelta

# If you remove rules, it removes them from *all* calendars
# To ensure we don't pop rules we don't want to, first make
# sure to fully copy the object
trade_calendar = USFederalHolidayCalendar()
trade_calendar.rules.pop(6) # Remove Columbus day
trade_calendar.rules.pop(7) # Remove Veteran's day
TradeDay = lambda days: CustomBusinessDay(days, calendar=trade_calendar)

def plot_study(array):
    # Given a 2-d array, we assume the event happens at index `lookback`,
    # and create all of our summary statistics from there.
    lookback = int((array.shape[1] - 1) / 2)
    norm_factor = np.repeat(array[:,lookback].reshape(-1, 1), array.shape[1], axis=1)
    centered_data = array / norm_factor - 1
    lookforward = centered_data.shape[1] - lookback
    means = centered_data.mean(axis=0)
    lookforward_data = centered_data[:,lookforward:]
    std_dev = np.hstack([0, lookforward_data.std(axis=0)])
    maxes = lookforward_data.max(axis=0)
    mins = lookforward_data.min(axis=0)
    
    f, axarr = plt.subplots(1, 2)
    range_begin = -lookback
    range_end = lookforward
    axarr[0].plot(range(range_begin, range_end), means)
    axarr[1].plot(range(range_begin, range_end), means)
    axarr[0].fill_between(range(0, range_end),
                     means[-lookforward:] + std_dev,
                     means[-lookforward:] - std_dev,
                    alpha=.5, label="$\pm$ 1 s.d.")
    axarr[1].fill_between(range(0, range_end),
                     means[-lookforward:] + std_dev,
                     means[-lookforward:] - std_dev,
                    alpha=.5, label="$\pm$ 1 s.d.")
    
    max_err = maxes - means[-lookforward+1:]
    min_err = means[-lookforward+1:] - mins
    axarr[0].errorbar(range(1, range_end),
                  means[-lookforward+1:],
                  yerr=[min_err, max_err], label='Max & Min')
    axarr[0].legend(loc=2)
    axarr[1].legend(loc=2)
    
    axarr[0].set_xlim((-lookback-1, lookback+1))
    axarr[1].set_xlim((-lookback-1, lookback+1))
    
def plot_study_small(array):
    # Given a 2-d array, we assume the event happens at index `lookback`,
    # and create all of our summary statistics from there.
    lookback = int((array.shape[1] - 1) / 2)
    norm_factor = np.repeat(array[:,lookback].reshape(-1, 1), array.shape[1], axis=1)
    centered_data = array / norm_factor - 1
    lookforward = centered_data.shape[1] - lookback
    means = centered_data.mean(axis=0)
    lookforward_data = centered_data[:,lookforward:]
    std_dev = np.hstack([0, lookforward_data.std(axis=0)])
    maxes = lookforward_data.max(axis=0)
    mins = lookforward_data.min(axis=0)
    
    range_begin = -lookback
    range_end = lookforward
    plt.plot(range(range_begin, range_end), means)
    plt.fill_between(range(0, range_end),
                     means[-lookforward:] + std_dev,
                     means[-lookforward:] - std_dev,
                    alpha=.5, label="$\pm$ 1 s.d.")
    
    max_err = maxes - means[-lookforward+1:]
    min_err = means[-lookforward+1:] - mins
    plt.errorbar(range(1, range_end),
                  means[-lookforward+1:],
                  yerr=[min_err, max_err], label='Max & Min')
    plt.legend(loc=2)
    plt.xlim((-lookback-1, lookback+1))
    
def fetch_event_data(ticker, events, horizon=5):
    # Use horizon+1 to account for including the day of the event,
    # and half-open interval - that is, for a horizon of 5,
    # we should be including 11 events. Additionally, using the
    # CustomBusinessDay means we automatically handle issues if
    # for example a company reports Friday afternoon - the date
    # calculator will turn this into a "Saturday" release, but
    # we effectively shift that to Monday with the logic below.
    td_back = TradeDay(horizon+1)
    td_forward = TradeDay(horizon+1)
    
    start_date = min(events) - td_back
    end_date = max(events) + td_forward
    total_data = fetch_ticker(ticker, start_date, end_date)
    event_data = [total_data.ix[event-td_back:event+td_forward]\
                      [0:horizon*2+1]\
                      ['Adjusted Close']
                  for event in events]
    return np.array(event_data)

# Generate a couple of random events

event_dates = [datetime(2016, 5, 27) - timedelta(days=1) - TradeDay(x*20) for x in range(1, 40)]
data = fetch_event_data('CELG', event_dates)
plot_study_small(data)
plt.legend(loc=3)
plt.gcf().set_size_inches(12, 6);


plt.annotate('Mean price for days leading up to each event',
             (-5, -.01), (-4.5, .025),
             arrowprops=dict(facecolor='black', shrink=0.05))
plt.annotate('', (-.1, .005), (-.5, .02),
             arrowprops={'facecolor': 'black', 'shrink': .05})
plt.annotate('$\pm$ 1 std. dev. each day', (5, .055), (2.5, .085),
            arrowprops={'facecolor': 'black', 'shrink': .05})
plt.annotate('Min/Max each day', (.9, -.07), (-1, -.1),
            arrowprops={'facecolor': 'black', 'shrink': .05});

And as a quick textual explanation as well:

The blue line represents the mean price for each day, represented as a percentage of the price on the '0-day'. For example, if we defined an 'event' as whenever the stock price dropped for three days, we would see a decreasing blue line to the left of the 0-day.
The blue shaded area represents one standard deviation above and below the mean price for each day following an event. This is intended to give us an idea of what the stock price does in general following an event.
The green bars are the minimum and maximum price for each day following an event. This instructs us as to how much it's possible for the stock to move.

The first type of event I want to study is how stocks perform when they've been trending down over the past couple of days prior to a release. However, we need to clarify what exactly is meant by "trending down." To do so, we'll use the following metric: the midpoint between each day's opening and closing price goes down over a period of N days.

It's probably helpful to have an example:

In [6]:

f, axarr = plt.subplots(1, 2)
f.set_size_inches(18, 6)

FB_plot = axarr[0]
ohlc_dataframe(FB[datetime(2016, 4, 18):], FB_plot)

FB_truncated = FB[datetime(2016, 4, 18):datetime(2016, 4, 27)]
midpoint = FB_truncated['Open']/2 + FB_truncated['Close']/2
FB_plot.plot(FB_truncated.index, midpoint, label='Midpoint')
FB_plot.vlines(date2num(datetime(2016, 4, 27, 12)),
               ax_fb.get_ylim()[0], ax_fb.get_ylim()[1],
               color='g', label='Earnings Release')
FB_plot.legend(loc=2)
FB_plot.set_title('FB Midpoint Plot')

AAPL_plot = axarr[1]
ohlc_dataframe(AAPL[datetime(2016, 4, 10):], AAPL_plot)
AAPL_truncated = AAPL[datetime(2016, 4, 10):datetime(2016, 4, 26)]
midpoint = AAPL_truncated['Open']/2 + AAPL_truncated['Close']/2
AAPL_plot.plot(AAPL_truncated.index, midpoint, label='Midpoint')
AAPL_plot.vlines(date2num(datetime(2016, 4, 26, 12)),
                 ax_aapl.get_ylim()[0], ax_aapl.get_ylim()[1],
                 color='g', label='Earnings Release')
AAPL_plot.legend(loc=3)
AAPL_plot.set_title('AAPL Midpoint Plot');

Given these charts, we can see that FB was trending down for the four days preceding the earnings release, and AAPL was trending down for a whopping 8 days (we don't count the peak day). This will define the methodology that we will use for the study.

So what are the results? For a given horizon, how well does the market actually perform?

In [7]:

# Read in the events for each stock;
# The file was created using the first code block in the Appendix
import yaml
from dateutil.parser import parse
from progressbar import ProgressBar

data_str = open('earnings_dates.yaml', 'r').read()
# Need to remove invalid lines
filtered = filter(lambda x: '{' not in x, data_str.split('\n'))
earnings_data = yaml.load('\n'.join(filtered))

# Convert our earnings data into a list of (ticker, date) pairs
# to make it easy to work with.
# This is horribly inefficient, but should get us what we need
ticker_dates = []
for ticker, date_list in earnings_data.items():
    for iso_str in date_list:
        ticker_dates.append((ticker, parse(iso_str)))

def does_trend_down(ticker, event, horizon):
    # Figure out if the `event` has a downtrend for
    # the `horizon` days preceding it
    # As an interpretation note: it is assumed that
    # the closing price of day `event` is the reference
    # point, and we want `horizon` days before that.
    # The price_data.hdf was created in the second appendix code block
    try:
        ticker_data = pd.read_hdf('price_data.hdf', ticker)
        data = ticker_data[event-TradeDay(horizon):event]
        midpoints = data['Open']/2 + data['Close']/2

        # Shift dates one forward into the future and subtract
        # Effectively: do we trend down over all days?
        elems = midpoints - midpoints.shift(1)
        return len(elems)-1 == len(elems.dropna()[elems <= 0])
    except KeyError:
        # If the stock doesn't exist, it doesn't qualify as trending down
        # Mostly this is here to make sure the entire analysis doesn't 
        # blow up if there were issues in data retrieval
        return False

def study_trend(horizon, trend_function):
    five_day_events = np.zeros((1, horizon*2 + 1))
    invalid_events = []
    for ticker, event in ProgressBar()(ticker_dates):
        if trend_function(ticker, event, horizon):
            ticker_data = pd.read_hdf('price_data.hdf', ticker)
            event_data = ticker_data[event-TradeDay(horizon):event+TradeDay(horizon)]['Close']

            try:
                five_day_events = np.vstack([five_day_events, event_data])
            except ValueError:
                # Sometimes we don't get exactly the right number of values due to calendar
                # issues. I've fixed most everything I can, and the few issues that are left
                # I assume don't systemically bias the results (i.e. data could be missing
                # because it doesn't exist, etc.). After running through, ~1% of events get
                # discarded this way
                invalid_events.append((ticker, event))
            

    # Remove our initial zero row
    five_day_events = five_day_events[1:,:]
    plot_study(five_day_events)
    plt.gcf().suptitle('Action over {} days: {} events'
                       .format(horizon,five_day_events.shape[0]))
    plt.gcf().set_size_inches(18, 6)
    
# Start with a 5 day study
study_trend(5, does_trend_down)

100% (47578 of 47578) |###########################################################| Elapsed Time: 0:21:38 Time: 0:21:38

When a stock has been trending down for 5 days, once the earnings are announced it really doesn't move on average. However, the variability is incredible. This implies two important things:

The market is just as often wrong about an earnings announcement before it happens as it is correct
The incredible width of the min/max bars and standard deviation area tell us that the market reacts violently after the earnings are released.

Let's repeat the same study, but over a time horizon of 8 days and 3 days. Presumably if a stock has been going down for 8 days at a time before the earnings, the market should be more accurate.

In [8]:

# 8 day study next
study_trend(8, does_trend_down)

100% (47578 of 47578) |###########################################################| Elapsed Time: 0:20:29 Time: 0:20:29

However, looking only at stocks that trended down for 8 days prior to a release, the same pattern emerges: on average, the stock doesn't move, but the market reaction is often incredibly violent.

In [9]:

# 3 day study after that
study_trend(3, does_trend_down)

100% (47578 of 47578) |###########################################################| Elapsed Time: 0:26:26 Time: 0:26:26

Finally, when we look at a 3-day horizon, we start getting some incredible outliers. Stocks have a potential to move over ~300% up, and the standard deviation width is again, incredible. The results for a 3-day horizon follow the same pattern we've seen in the 5- and 8-day horizons.

We're now going to repeat the analysis, but do it for uptrends instead. That is, instead of looking at stocks that have been trending down over the past number of days, we focus only on stocks that have been trending up.

In [10]:

def does_trend_up(ticker, event, horizon):
    # Figure out if the `event` has an uptrend for
    # the `horizon` days preceding it
    # As an interpretation note: it is assumed that
    # the closing price of day `event` is the reference
    # point, and we want `horizon` days before that.
    # The price_data.hdf was created in the second appendix code block
    try:
        ticker_data = pd.read_hdf('price_data.hdf', ticker)
        data = ticker_data[event-TradeDay(horizon):event]
        midpoints = data['Open']/2 + data['Close']/2

        # Shift dates one forward into the future and subtract
        # Effectively: do we trend down over all days?
        elems = midpoints - midpoints.shift(1)
        return len(elems)-1 == len(elems.dropna()[elems >= 0])
    except KeyError:
        # If the stock doesn't exist, it doesn't qualify as trending down
        # Mostly this is here to make sure the entire analysis doesn't 
        # blow up if there were issues in data retrieval
        return False

study_trend(5, does_trend_up)

100% (47578 of 47578) |###########################################################| Elapsed Time: 0:22:51 Time: 0:22:51

The patterns here are very similar. With the exception of noting that stocks can go to nearly 400% after an earnings announcement (most likely this included a takeover announcement, etc.), we still see large min/max bars and wide standard deviation of returns.

We'll repeat the pattern for stocks going up for both 8 and 3 days straight, but at this point, the results should be very predictable:

In [11]:

study_trend(8, does_trend_up)

100% (47578 of 47578) |###########################################################| Elapsed Time: 0:20:51 Time: 0:20:51

In [12]:

study_trend(3, does_trend_up)

100% (47578 of 47578) |###########################################################| Elapsed Time: 0:26:56 Time: 0:26:56

Conclusion and Summary¶

I guess the most important thing to summarize with is this: looking at the entire market, stock performance prior to an earnings release has no bearing on the stock's performance. Honestly: given the huge variability of returns after an earnings release, even when the stock has been trending for a long time, you're best off divesting before an earnings release and letting the market sort itself out.

However, there is a big caveat. These results are taken when we look at the entire market. So while we can say that the market as a whole knows nothing and just reacts violently, I want to take a closer look into this data. Does the market typically perform poorly on large-cap/high liquidity stocks? Do smaller companies have investors that know them better and can thus predict performance better? Are specific market sectors better at prediction? Presumably technology stocks are more volatile than the industrials.

So there are some more interesting questions I still want to ask with this data. Knowing that the hard work of data processing is largely already done, it should be fairly simple to continue this analysis and get much more refined with it. Until next time.

Appendix¶

Export event data for Russell 3000 companies:

import pandas as pd
from html.parser import HTMLParser
from datetime import datetime, timedelta
import requests
import re
from dateutil import parser
import progressbar
from concurrent import futures
import yaml

class EarningsParser(HTMLParser):
    store_dates = False
    earnings_offset = None
    dates = []

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.dates = []

    def handle_starttag(self, tag, attrs):
        if tag == 'table':
            self.store_dates = True

    def handle_data(self, data):
        if self.store_dates:
            match = re.match(r'\d+/\d+/\d+', data)
            if match:
                self.dates.append(match.group(0))

        # If a company reports before the bell, record the earnings date
        # being at midnight the day before. Ex: WMT reports 5/19/2016,
        # but we want the reference point to be the closing price on 5/18/2016
        if 'After Close' in data:
            self.earnings_offset = timedelta(days=0)
        elif 'Before Open' in data:
            self.earnings_offset = timedelta(days=-1)

    def handle_endtag(self, tag):
        if tag == 'table':
            self.store_dates = False

def earnings_releases(ticker):
    #print("Looking up ticker {}".format(ticker))
    user_agent = 'Mozilla/5.0 (Windows NT 10.0; WOW64; rv:46.0) '\
        'Gecko/20100101 Firefox/46.0'
    headers = {'user-agent': user_agent}
    base_url = 'http://www.streetinsider.com/ec_earnings.php?q={}'\
        .format(ticker)
    e = EarningsParser()
    s = requests.Session()
    a = requests.adapters.HTTPAdapter(max_retries=0)
    s.mount('http://', a)
    e.feed(str(s.get(base_url, headers=headers).content))

    if e.earnings_offset is not None:
        dates = map(lambda x: parser.parse(x) + e.earnings_offset, e.dates)
        past = filter(lambda x: x < datetime.now(), dates)
        return list(map(lambda d: d.isoformat(), past))

# Use a Russell-3000 ETF tracker (ticker IWV) to get a list of holdings
r3000 = pd.read_csv('https://www.ishares.com/us/products/239714/'
                    'ishares-russell-3000-etf/1449138789749.ajax?'
                    'fileType=csv&fileName=IWV_holdings&dataType=fund',
                    header=10)
r3000_equities = r3000[(r3000['Exchange'] == 'NASDAQ') |
                       (r3000['Exchange'] == 'New York Stock Exchange Inc.')]

dates_file = open('earnings_dates.yaml', 'w')

with futures.ThreadPoolExecutor(max_workers=8) as pool:
    fs = {pool.submit(earnings_releases, r3000_equities.ix[t]['Ticker']): t
          for t in r3000_equities.index}
    pbar = progressbar.ProgressBar(term_width=80,
                                   max_value=r3000_equities.index.max())

    for future in futures.as_completed(fs):
        i = fs[future]
        pbar.update(i)
        dates_file.write(yaml.dump({r3000_equities.ix[i]['Ticker']:
            future.result()}))

Downloading stock price data needed for the event studies:

from secrets import QUANDL_KEY
import pandas as pd
import yaml
from dateutil.parser import parse
from datetime import timedelta
import quandl
from progressbar import ProgressBar

def fetch_ticker(ticker, start, end):
    # Quandl is currently giving me issues with returning
    # the entire dataset and not slicing server-side.
    # So instead, we'll do it client-side!
    q_format = '%Y-%m-%d'
    ticker_data = quandl.get('YAHOO/' + ticker,
                             start_date=start.strftime(q_format),
                             end_date=end.strftime(q_format),
                             authtoken=QUANDL_KEY)
    return ticker_data

data_str = open('earnings_dates.yaml', 'r').read()
# Need to remove invalid lines
filtered = filter(lambda x: '{' not in x, data_str.split('\n'))
earnings_data = yaml.load('\n'.join(filtered))

# Get the first 1500 keys - split up into two statements
# because of Quandl rate limits
tickers = list(earnings_data.keys())

price_dict = {}
invalid_tickers = []
for ticker in ProgressBar()(tickers[0:1500]):
    try:
        # Replace '.' with '-' in name for some tickers
        fixed = ticker.replace('.', '-')
        event_strs = earnings_data[ticker]
        events = [parse(event) for event in event_strs]
        td = timedelta(days=20)
        price_dict[ticker] = fetch_ticker(fixed,
            min(events)-td, max(events)+td)
    except quandl.NotFoundError:
        invalid_tickers.append(ticker)

# Execute this after 10 minutes have passed
for ticker in ProgressBar()(tickers[1500:]):
    try:
        # Replace '.' with '-' in name for some tickers
        fixed = ticker.replace('.', '-')
        event_strs = earnings_data[ticker]
        events = [parse(event) for event in event_strs]
        td = timedelta(days=20)
        price_dict[ticker] = fetch_ticker(fixed,
            min(events)-td, max(events)+td)
    except quandl.NotFoundError:
        invalid_tickers.append(ticker)

prices_store = pd.HDFStore('price_data.hdf')
for ticker, prices in price_dict.items():
    prices_store[ticker] = prices

The Unfair Casino

2016-05-15T00:00:00-04:00

Or, how to get thrown out of a casino because you're a mathematician.

In the ongoing eternal cycle of mathematicians asking generally useless questions about probability, I dreamt up another one. The scenario is as follows:

You're playing a game with two die, and you do not get to see what the outcome of the die are on each roll. All you get to see is their sum. Given an arbitrarily long list of the sum of two rolls, can you determine if one or both die are loaded, and what those loadings are?

Proving we can detect cheating¶

My first question is simply, is this possible? There's a lot of trivial cases that make it obvious that there's cheating going on. But there are some edge cases that might give us more difficulty. First though, let's get a picture of what the fair distribution looks like. In principle, we can only detect cheating if the distribution of the fair die differs from the distribution of the loaded die.

In [1]:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline

fair_1 = np.random.randint(1, 7, 10000)
fair_2 = np.random.randint(1, 7, 10000)

pd.Series(fair_1 + fair_2).plot(kind='hist', bins=11);
plt.title('Fair Distribution');

This distribution makes sense: there are many ways to make a 7 (the most frequent observed value) and very few ways to make a 12 or 2; an important symmetry. As a special note, you can notice that the sum of two fair dice is a discrete case of the Triangle Distribution, which is itself a special case of the Irwin-Hall Distribution.

The Edge Cases¶

Given that we understand how the results of two fair dice are distributed, let's see some of the interesting edge cases that come up. This will give us assurance that when a casino is cheating, it is detectable (given sufficient data). To make this as hard as possible, we will think of scenarios where the expected value of the sum of loaded dice is the same as the expected value of the sum of fair dice.

Edge Case 1¶

What happens when one die is biased low, and one die is biased high? That is, where:

\begin{align} \begin{array}{cc} D_1 = \left\{ \begin{array}{lr} 1 & w.p. 1/3\\ 2 & w.p. 1/3\\ 3 & w.p. 1/12\\ 4 & w.p. 1/12\\ 5 & w.p. 1/12\\ 6 & w.p. 1/12 \end{array} \right. & D_2 = \left\{ \begin{array}{lr} 1 & w.p. 1/12\\ 2 & w.p. 1/12\\ 3 & w.p. 1/12\\ 4 & w.p. 1/12\\ 5 & w.p. 1/3\\ 6 & w.p. 1/3 \end{array} \right. \\ \mathbb{E}[D_1] = 2.5 & \mathbb{E}[D_2] = 4.5 \end{array}\\ \mathbb{E}[D_1 + D_2] = 7 = \mathbb{E}[D_{fair} + D_{fair}] \end{align}

In [2]:

def unfair_die(p_vals, n):
    x = np.random.multinomial(1, p_vals, n)
    return x.nonzero()[1] + 1

d1 = [1/3, 1/3, 1/12, 1/12, 1/12, 1/12]
d2 = [1/12, 1/12, 1/12, 1/12, 1/3, 1/3]

x1 = unfair_die(d1, 10000)
x2 = unfair_die(d2, 10000)

pd.Series(x1 + x2).plot(kind='hist', bins=11);
plt.title('$D_1$ biased low, $D_2$ biased high');

We can see that while the 7 value remains the most likely (as expected), the distribution is not so nicely shaped any more.

Edge Case 2: When one die is loaded low, and one is loaded high, we've seen how we can detect them. How about when two die are loaded both low and high? That is, we have the following distribution:

\begin{align} \begin{array}{cc} D_1 = \left\{ \begin{array}{lr} 1 & w.p. 1/3\\ 2 & w.p. 1/12\\ 3 & w.p. 1/12\\ 4 & w.p. 1/12\\ 5 & w.p. 1/12\\ 6 & w.p. 1/3 \end{array} \right. & D_2 = \left\{ \begin{array}{lr} 1 & w.p. 1/3\\ 2 & w.p. 1/12\\ 3 & w.p. 1/12\\ 4 & w.p. 1/12\\ 5 & w.p. 1/12\\ 6 & w.p. 1/3 \end{array} \right. \\ \mathbb{E}[D_1] = 3.5 & \mathbb{E}[D_2] = 3.5 \end{array}\\ \mathbb{E}[D_1 + D_2] = 7 = \mathbb{E}[D_{fair} + D_{fair}] \end{align}

We can see even that the expected value of each individual die is the same as the fair die! However, the distribution (if we are doing this correctly) should still be skewed:

In [3]:

d1 = [1/3, 1/12, 1/12, 1/12, 1/12, 1/3]
d2 = d1

x1 = unfair_die(d1, 10000)
x2 = unfair_die(d2, 10000)

pd.Series(x1 + x2).plot(kind='hist', bins=11)
plt.title("$D_1$ and $D_2$ biased to 1 and 6");

In a very un-subtle way, we have of course made the values 2 and 12 far more likely.

Detection Conclusion¶

There are some trivial examples of cheating that are easy to detect: whenever the expected value of the sum of two fair dice deviates from the expected value for the sum of two fair dice, we can immediately conclude that there is cheating at stake.

The interesting edge cases occur when the expected value of the sum of loaded dice matches the expected value of the sum of fair dice. Considering the above examples (and a couple more I ran through in developing this), we have seen that in every circumstance having two unfair dice leads to a distribution of results different from the fair results.

We can thus finally state: just by looking at the distribution of results from this game, we can immediately conclude whether there is cheating.

Simulated Annealing¶

What we really would like to do though, is see if there is any way to determine how exactly the dice are loaded. This is significantly more complicated, but we can borrow some algorithms from Machine Learning to figure out exactly how to perform this process. I'm using the Simulated Annealing algorithm, and I discuss why this works and why I chose it over some of the alternatives in the justification. If you don't care about how I set up the model and just want to see the code, check out the actual code.

Simulated Annealing is a variation of the Metropolis-Hastings Algorithm, but the important thing for us is: Simulated Annealing allows us to quickly optimize high-dimensional problems. But what exactly are we trying to optimize? Ideally, we want a function that can tell us whether one distribution for the dice better explains the results than another distribution. This is known as the likelihood function.

Deriving the Likelihood function¶

To derive our likelihood function, we want to know: what is the probability of seeing a specific result given those hidden parameters? This is actually a surprisingly difficult problem. While we can do a lot of calculations by hand, we need a more general solution since we will be working with very some interesting die distributions.

We first note that the sum of two dice can take on 11 different values - 2 through 12. This implies that each individual sum follows a Categorical distribution. That is:

\begin{align} \mathcal{L(x)} = \left\{ \begin{array}{lr} p_2 & x = 2\\ p_3 & x = 3\\ \ldots & \\ p_{11} & x = 11\\ p_{12} & x = 12 \end{array} \right. \end{align}

Where each $p_i$ is the probability of seeing that specific result. However, we need to calculate what each probability is! I'll save you the details, but this author explains how to do it.

Now, we would like to know the likelihood of our entire data-set. This is trivial:

\begin{align} \mathcal{L(\mathbf{X})} &= \prod_{i=1}^n L(x) \end{align}

However, it's typically much easier to work with the $\log(\mathcal{L})$ function instead. This is critically important from a computational perspective: when you multiply so many small numbers together (i.e. the product of $L(x)$ terms) the computer suffers from rounding error; if we don't control for this, we will find that no matter the distributions we choose for each die, the "likelihood" will be close to zero because the computer is not precise enough.

\begin{align} \log(\mathcal{L}) &= \sum_{i=1}^n \log(L) \end{align}

The process of Simulated Annealing¶

The means by which we optimize our likelihood function is the simulated annealing algorithm. The way it works is as follows:

Start with a random guess for the parameters we are trying to optimize. In our case we are trying to guess the distribution of two dice, and so we "optimize" until we have a distribution that matches the data.
For each iteration of the algorithm:
1. Generate a new "proposed" set of parameters based on the current parameters - i.e. slightly modify the current parameters to get a new set of parameters.
2. Calculate the value of $\log(\mathcal{L})$ for each set of parameters. If the function value for the proposed parameter set is higher than for the current, automatically switch to the new parameter set and continue the next iteration.
3. Given the new parameter set performs worse, determine a probability of switching to the new parameter set anyways: $\mathcal{P}(p_{current}, p_{proposed})$
4. Switch to the new parameter set with probability $\mathcal{P}$. If you fail to switch, begin the next iteration.
The algorithm is complete after we fail to make a transition $n$ times in a row.

If everything goes according to plan, we will have a value that is close to the true distribution of each die.

The actual code¶

We start by defining the score function. This will tell us how well the proposed die densities actually explain the results.

In [5]:

import numpy as np
from numpy import polynomial

def density_coef(d1_density, d2_density):
    # Calculating the probabilities of each outcome was taken
    # from this author: http://math.stackexchange.com/a/1710392/320784
    d1_p = polynomial.Polynomial(d1_density)
    d2_p = polynomial.Polynomial(d2_density)
    coefs = (d1_p * d2_p).coef
    return coefs

def score(x, d1_density, d2_density):
    # We've now got the probabilities of each event, but we need
    # to shift the array a bit so we can use the x values to actually
    # index into it. This will allow us to do all the calculations
    # incredibly quickly
    coefs = density_coef(d1_density, d2_density)
    coefs = np.hstack((0, 0, coefs))
    return np.log(coefs[x]).sum()

Afterward, we need to write something to permute the proposal densities. We make random modifications, and eventually the best one survives.

In [6]:

def permute(d1_density, d2_density):
    # To ensure we have legitimate densities, we will randomly
    # increase one die face probability by `change`,
    # and decrease one by `change`.
    # This means there are something less than (1/`change`)^12 possibilities
    # we are trying to search over.
    change = .01
    
    d1_index1, d1_index2 = np.random.randint(0, 6, 2)
    d2_index1, d2_index2 = np.random.randint(0, 6, 2)
    
    # Also make sure to copy. I've had some weird aliasing issues
    # in the past that made everything blow up.
    new_d1 = np.float64(np.copy(d1_density))
    new_d2 = np.float64(np.copy(d2_density))
    
    # While this doesn't account for the possibility that some
    # values go negative, in practice this never happens
    new_d1[d1_index1] += change
    new_d1[d1_index2] -= change
    new_d2[d2_index1] += change
    new_d2[d2_index2] -= change
    
    return new_d1, new_d2

Now we've got the main algorithm code to do. This is what brings all the pieces together.

In [7]:

def optimize(data, conv_count=10, max_iter=1e4):
    switch_failures = 0
    iter_count = 0
    
    # Start with guessing fair dice
    cur_d1 = np.repeat(1/6, 6)
    cur_d2 = np.repeat(1/6, 6)
    cur_score = score(data, cur_d1, cur_d2)
    
    # Keep track of our best guesses - may not be
    # what we end with
    max_score = cur_score
    max_d1 = cur_d1
    max_d2 = cur_d2
    
    # Optimization stops when we have failed to switch `conv_count`
    # times (presumably because we have a great guess), or we reach
    # the maximum number of iterations.
    while switch_failures < conv_count and iter_count < max_iter:
        iter_count += 1
        if iter_count % (max_iter / 10) == 0:
            print('Iteration: {}; Current score (higher is better): {}'.format(
                    iter_count, cur_score))
                
        new_d1, new_d2 = permute(cur_d1, cur_d2)
        new_score = score(data, new_d1, new_d2)
        
        if new_score > max_score:
            max_score = new_score
            max_d1 = new_d1
            max_d2 = new_d2
            
        if new_score > cur_score:
            # If the new permutation beats the old one,
            # automatically select it.
            cur_score = new_score
            cur_d1 = new_d1
            cur_d2 = new_d2
            switch_failures = 0
        else:
            # We didn't beat the current score, but allow
            # for possibly switching anyways.
            accept_prob = np.exp(new_score - cur_score)
            coin_toss = np.random.rand()
            if coin_toss < accept_prob:
                # We randomly switch to the new distribution
                cur_score = new_score
                cur_d1 = new_d1
                cur_d2 = new_d2
                switch_failures = 0
            else:
                switch_failures += 1
            
    # Return both our best guess, and the ending guess
    return max_d1, max_d2, cur_d1, cur_d2

And now we have finished the hard work!

Catching the Casino¶

Let's go through a couple of scenarios and see if we can catch the casino cheating with some loaded dice. In every scenario we start with an assumption of fair dice, and then try our hand to figure out what the actual distribution was.

Attempt 1¶

The casino is using two dice that are both biased low. How well can we recover the distribution?

In [8]:

import time
def simulate_casino(d1_dist, d2_dist, n=10000):
    d1_vals = unfair_die(d1_dist, n)
    d2_vals = unfair_die(d2_dist, n)
    
    start = time.perf_counter()
    max_d1, max_d2, final_d1, final_d2 = optimize(d1_vals + d2_vals)
    end = time.perf_counter()
    print("Simulated Annealing time: {:.02f}s".format(end - start))
    
    coef_range = np.arange(2, 13) - .5
    plt.subplot(221)
    plt.bar(coef_range, density_coef(d1_dist, d2_dist), width=1)
    plt.title('True Distribution')
    
    plt.subplot(222)
    plt.hist(d1_vals + d2_vals, bins=11)
    plt.title('Empirical Distribution')
    
    plt.subplot(223)
    plt.bar(coef_range, density_coef(max_d1, max_d2), width=1)
    plt.title('Recovered Distribution')
    
    plt.gcf().set_size_inches(10, 10)


simulate_casino([2/9, 2/9, 2/9, 1/9, 1/9, 1/9],
                [2/9, 2/9, 2/9, 1/9, 1/9, 1/9])

Iteration: 1000; Current score (higher is better): -22147.004400281654
Simulated Annealing time: 0.30s

Attempt 2¶

The casino now uses dice that are both biased towards 1 and 6.

In [9]:

simulate_casino([1/3, 1/12, 1/12, 1/12, 1/12, 1/3],
                [1/3, 1/12, 1/12, 1/12, 1/12, 1/3])

Simulated Annealing time: 0.08s

Attempt 3¶

The casino will now use one die biased towards 1 and 6, and one die towards 3 and 4.

In [10]:

simulate_casino([1/3, 1/12, 1/12, 1/12, 1/12, 1/3],
                [1/12, 1/12, 1/3, 1/3, 1/12, 1/12])

Simulated Annealing time: 0.09s

Attempt 4¶

We'll now finally go to a fair casino to make sure that we can still recognize a positive result.

In [11]:

simulate_casino(np.repeat(1/6, 6), np.repeat(1/6, 6))

Simulated Annealing time: 0.02s

Attempt 5¶

We've so far been working with a large amount of data - 10,000 data points. Can we now scale things back to only 250 throws? We'll start with two dice biased high.

In [12]:

simulate_casino([1/9, 1/9, 1/9, 2/9, 2/9, 2/9],
                [1/9, 1/9, 1/9, 2/9, 2/9, 2/9],
                n=250)

Iteration: 1000; Current score (higher is better): -551.6995384525453
Iteration: 2000; Current score (higher is better): -547.7803673440676
Iteration: 3000; Current score (higher is better): -547.9805613193807
Iteration: 4000; Current score (higher is better): -546.7574874775273
Iteration: 5000; Current score (higher is better): -549.5798007672656
Iteration: 6000; Current score (higher is better): -545.0354060154496
Iteration: 7000; Current score (higher is better): -550.1134504086606
Iteration: 8000; Current score (higher is better): -549.9306537114975
Iteration: 9000; Current score (higher is better): -550.7075182119111
Iteration: 10000; Current score (higher is better): -549.400679551826
Simulated Annealing time: 1.94s

The results are surprisingly good. While the actual optimization process took much longer to finish than in the other examples, we still have a very good guess. As a caveat though: the recovered distribution tends to overfit the data. That is, if the data doesn't fit the underlying distribution well, the model will also fail.

Conclusion¶

Given the results above, we can see that we have indeed come up with a very good algorithm to determine the distribution of two dice given their results. As a benefit, we have even seen that results come back very quickly; it's not uncommon for the optimization to converge within a tenth of a second.

Additionally, we have seen that the algorithm can intuit the distribution even when there is not much data. While the final example shows that we can 'overfit' on the dataset, we can still get valuable information from a relatively small dataset.

We can declare at long last: the mathematicians have again triumphed over the casino.

Justification of Simulated Annealing¶

Why Simulated Annealing?¶

So why even use an algorithm with a fancy title like Simulated Annealing? First of all, because the title is sexy. Second of all, because this is a reasonably complicated problem to try and solve. We have a parameter space where each value $p_{ij} \in (0, 1); i, j \in \{1, \ldots, 6\}$, for a total of 12 different variables we are trying to optimize over. Additionally, given a 12-dimensional function we are trying to optimize, simulated annealing makes sure that we don't fall into a local minimum.

Why not something else?¶

This is a fair question. There are two classes of algorithms that can also be used to solve this problem: Non-linear optimization methods, and the EM algorithm.

I chose not to use non-linear optimization simply because I'm a bit concerned that it will trap me in a local maximum. Instead of running multiple different optimizations from different starting points, I can just use simulated annealing to take that into account. In addition, throughout the course of testing the simulated annealing code converged incredibly quickly - far more quickly than any non-linear solver would be able to accomplish.
The EM Algorithm was originally what I intended to write this blog post with. Indeed, the post was inspired by the crooked casino example which uses the EM algorithm to solve it. However, after modeling the likelihood function I realized that the algebra would very quickly get out of hand. Trying to compute all the polynomial terms would not be fun, which would be needed to actually optimize for each parameter. So while the EM algorithm would likely be much faster in raw speed terms, the amount of time needed to program and verify it meant that I was far better off using a different method for optimization.

Tick Tock...

2016-04-06T00:00:00-04:00

If all we have is a finite number of heartbeats left, what about me?

Warning: this one is a bit creepier. But that's what you get when you come up with data science ideas as you're drifting off to sleep.

2.5 Billion¶

If PBS is right, that's the total number of heartbeats we get. Approximately once every second that number goes down, and down, and down again...

In [1]:

total_heartbeats = 2500000000

I got a Fitbit this past Christmas season, mostly because I was interested in the data and trying to work on some data science projects with it. This is going to be the first project, but there will likely be more (and not nearly as morbid). My idea was: If this is the final number that I'm running up against, how far have I come, and how far am I likely to go? I've currently had about 3 months' time to estimate what my data will look like, so let's go ahead and see: given a lifetime 2.5 billion heart beats, how much time do I have left?

Statistical Considerations¶

Since I'm starting to work with health data, there are a few considerations I think are important before I start digging through my data.

The concept of 2.5 billion as an agreed-upon number is tenuous at best. I've seen anywhere from 2.21 billion to 3.4 billion so even if I knew exactly how many times my heart had beaten so far, the ending result is suspect at best. I'm using 2.5 billion because that seems to be about the midpoint of the estimates I've seen so far.
Most of the numbers I've seen so far are based on extrapolating number of heart beats from life expectancy. As life expectancy goes up, the number of expected heart beats goes up too.
My estimation of the number of heartbeats in my life so far is based on 3 months worth of data, and I'm extrapolating an entire lifetime based on this.

So while the ending number is not useful in any medical context, it is still an interesting project to work with the data I have on hand.

Getting the data¶

Fitbit has an API available for people to pull their personal data off the system. It requires registering an application, authentication with OAuth, and some other complicated things. If you're not interested in how I fetch the data, skip here.

Registering an application¶

I've already registered a personal application with Fitbit, so I can go ahead and retrieve things like the client secret from a file.

In [2]:

# Import all the OAuth secret information from a local file
from secrets import CLIENT_SECRET, CLIENT_ID, CALLBACK_URL

Handling OAuth 2¶

So, all the people that know what OAuth 2 is know what's coming next. For those who don't: OAuth is how people allow applications to access other data without having to know your password. Essentially the dialog goes like this:

Application: I've got a user here who wants to use my application, but I need their data.
Fitbit: OK, what data do you need access to, and for how long?
Application: I need all of these scopes, and for this amount of time.
Fitbit: OK, let me check with the user to make sure they really want to do this.

Fitbit: User, do you really want to let this application have your data?
User: I do! And to prove it, here's my password.
Fitbit: OK, everything checks out. I'll let the application access your data.

Fitbit: Application, you can access the user's data. Use this special value whenever you need to request data from me.
Application: Thank you, now give me all the data.

Effectively, this allows an application to gain access to a user's data without ever needing to know the user's password. That way, even if the other application is hacked, the user's original data remains safe. Plus, the user can let the data service know to stop providing the application access any time they want. All in all, very secure.

It does make handling small requests a bit challenging, but I'll go through the steps here. We'll be using the Implicit Grant workflow, as it requires fewer steps in processing.

First, we need to set up the URL the user would visit to authenticate:

In [3]:

import urllib

FITBIT_URI = 'https://www.fitbit.com/oauth2/authorize'
params = {
    # If we need more than one scope, must be a CSV string
    'scope': 'heartrate',
    'response_type': 'token',
    'expires_in': 86400, # 1 day
    'redirect_uri': CALLBACK_URL,
    'client_id': CLIENT_ID
}

request_url = FITBIT_URI + '?' + urllib.parse.urlencode(params)

Now, here you would print out the request URL, go visit it, and get the full URL that it sends you back to. Because that is very sensitive information (specifically containing my CLIENT_ID that I'd really rather not share on the internet), I've skipped that step in the code here, but it happens in the background.

In [6]:

# The `response_url` variable contains the full URL that
# FitBit sent back to us, but most importantly,
# contains the token we need for authorization.
access_token = dict(urllib.parse.parse_qsl(response_url))['access_token']

Requesting the data¶

Now that we've actually set up our access via the access_token, it's time to get the actual heart rate data. I'll be using data from January 1, 2016 through March 31, 2016, and extrapolating wildly from that.

Fitbit only lets us fetch intraday data one day at a time, so I'll create a date range using pandas and iterate through that to pull down all the data.

In [7]:

from requests_oauthlib import OAuth2Session
import pandas as pd
from datetime import datetime

session = OAuth2Session(token={
        'access_token': access_token,
        'token_type': 'Bearer'
    })

format_str = '%Y-%m-%d'
start_date = datetime(2016, 1, 1)
end_date = datetime(2016, 3, 31)
dr = pd.date_range(start_date, end_date)

url = 'https://api.fitbit.com/1/user/-/activities/heart/date/{0}/1d/1min.json'
hr_responses = [session.get(url.format(d.strftime(format_str))) for d in dr]

def record_to_df(record):
    if 'activities-heart' not in record:
        return None
    date_str = record['activities-heart'][0]['dateTime']
    df = pd.DataFrame(record['activities-heart-intraday']['dataset'])
        
    df.index = df['time'].apply(
        lambda x: datetime.strptime(date_str + ' ' + x, '%Y-%m-%d %H:%M:%S'))
    return df

hr_dataframes = [record_to_df(record.json()) for record in hr_responses]
hr_df_concat = pd.concat(hr_dataframes)


# There are some minutes with missing data, so we need to correct that
full_daterange = pd.date_range(hr_df_concat.index[0],
                              hr_df_concat.index[-1],
                              freq='min')
hr_df_full = hr_df_concat.reindex(full_daterange, method='nearest')

print("Heartbeats from {} to {}: {}".format(hr_df_full.index[0],
                                            hr_df_full.index[-1],
                                            hr_df_full['value'].sum()))

Heartbeats from 2016-01-01 00:00:00 to 2016-03-31 23:59:00: 8139060

And now we've retrieved all the available heart rate data for January 1^st through March 31^st! Let's get to the actual analysis.

Wild Extrapolations from Small Data¶

A fundamental issue of this data is that it's pretty small. I'm using 3 months of data to make predictions about my entire life. But, purely as an exercise, I'll move forward.

How many heartbeats so far?¶

The first step is figuring out how many of the 2.5 billion heartbeats I've used so far. We're going to try and work backward from the present day to when I was born to get that number. The easy part comes first: going back to January 1^st, 1992. That's because I can generalize how many 3-month increments there were between now and then, account for leap years, and call that section done.

Between January 1992 and January 2016 there were 96 quarters, and 6 leap days. The number we're looking for is:

\begin{equation} hr_q \cdot n - hr_d \cdot (n-m) \end{equation}

$hr_q$: Number of heartbeats per quarter
$hr_d$: Number of heartbeats on leap day
$n$: Number of quarters, in this case 96
$m$: Number of leap days, in this case 6

In [8]:

quarterly_count = hr_df_full['value'].sum()
leap_day_count = hr_df_full[(hr_df_full.index.month == 2) &
                            (hr_df_full.index.day == 29)]['value'].sum()
num_quarters = 96
leap_days = 6

jan_92_jan_16 = quarterly_count * num_quarters - leap_day_count * (num_quarters - leap_days)
jan_92_jan_16

Out[8]:

773609400

So between January 1992 and January 2016 I've used $\approx$ 774 million heartbeats. Now, I need to go back to my exact birthday. I'm going to first find on average how many heartbeats I use in a minute, and multiply that by the number of minutes between my birthday and January 1992.

For privacy purposes I'll put the code here that I'm using, but without any identifying information:

In [9]:

minute_mean = hr_df_full['value'].mean()
# Don't you wish you knew?
# birthday_minutes = ???

birthday_heartbeats = birthday_minutes * minute_mean

heartbeats_until_2016 = int(birthday_heartbeats + jan_92_jan_16)
remaining_2016 = total_heartbeats - heartbeats_until_2016

print("Heartbeats so far: {}".format(heartbeats_until_2016))
print("Remaining heartbeats: {}".format(remaining_2016))

Heartbeats so far: 775804660
Remaining heartbeats: 1724195340

It would appear that my heart has beaten 775,804,660 times between my moment of birth and January 1^st 2016, and that I have 1.72 billion left.

How many heartbeats longer?¶

Now comes the tricky bit. I know how many heart beats I've used so far, and how many I have remaining, so I'd like to come up with a (relatively) accurate estimate of when exactly my heart should give out. We'll do this in a few steps, increasing in granularity.

First step, how many heartbeats do I use in a 4-year period? I have data for a single quarter including leap day, so I want to know:

\begin{equation} hr_q \cdot n - hr_d \cdot (n - m) \end{equation}

$hr_q$: Heartbeats per quarter
$hr_d$: Heartbeats per leap day
$n$: Number of quarters = 16
$m$: Number of leap days = 1

In [10]:

heartbeats_4year = quarterly_count * 16 - leap_day_count * (16 - 1)
heartbeats_4year

Out[10]:

128934900

Now, I can fast forward from 2016 the number of periods of 4 years I have left.

In [11]:

four_year_periods = remaining_2016 // heartbeats_4year
remaining_4y = remaining_2016 - four_year_periods * heartbeats_4year

print("Four year periods remaining: {}".format(four_year_periods))
print("Remaining heartbeats after 4 year periods: {}".format(remaining_4y))

Four year periods remaining: 13
Remaining heartbeats after 4 year periods: 48041640

Given that there are 13 four-year periods left, I can move from 2016 all the way to 2068, and find that I will have 48 million heart beats left. Let's drop down to figuring out how many quarters that is. I know that 2068 will have a leap day (unless someone finally decides to get rid of them), so I'll subtract that out first. Then, I'm left to figure out how many quarters exactly are left.

In [12]:

remaining_leap = remaining_4y - leap_day_count
# Ignore leap day in the data set
heartbeats_quarter = hr_df_full[(hr_df_full.index.month != 2) &
                                (hr_df_full.index.day != 29)]['value'].sum()
quarters_left = remaining_leap // heartbeats_quarter
remaining_year = remaining_leap - quarters_left * heartbeats_quarter

print("Quarters left starting 2068: {}".format(quarters_left))
print("Remaining heartbeats after that: {}".format(remaining_year))

Quarters left starting 2068: 8
Remaining heartbeats after that: 4760716

So, that analysis gets me through until January 1^st 2070. Final step, using that minute estimate to figure out how many minutes past that I'm predicted to have:

In [13]:

from datetime import timedelta

base = datetime(2070, 1, 1)
minutes_left = remaining_year // minute_mean

kaput = timedelta(minutes=minutes_left)
base + kaput

Out[13]:

datetime.datetime(2070, 2, 23, 5, 28)

According to this, I've got until February 23^rd, 2070 at 5:28 PM in the evening before my heart gives out.

Summary¶

Well, that's kind of a creepy date to know. As I said at the top though, this number is totally useless in any medical context. It ignores the rate at which we continue to get better at making people live longer, and is extrapolating from 3 months' worth of data the rest of my life. Additionally, throughout my time developing this post I made many minor mistakes. I think they're all fixed now, but it's easy to mix a number up here or there and the analysis gets thrown off by a couple years.

Even still, I think philosophically humans have a desire to know how much time we have left in the world. Man is but a breath, and it's scary to think just how quickly that date may be coming up. This analysis asks an important question though: what are you going to do with the time you have left?

Thanks for sticking with me on this one, I promise it will be much less depressing next time!

Tweet Like Me

2016-03-28T00:00:00-04:00

An experiment in creating a robot that will imitate me on Twitter.

So, I'm taking a Machine Learning course this semester in school, and one of the topics we keep coming back to is natural language processing and the 'bag of words' data structure. That is, given a sentence:

How much wood would a woodchuck chuck if a woodchuck could chuck wood?

We can represent that sentence as the following list:

{ How: 1 much: 1 wood: 2 would: 2 a: 2 woodchuck: 2 chuck: 2 if: 1 }

Ignoring where the words happened, we're just interested in how often the words occurred. That got me thinking: I wonder what would happen if I built a robot that just imitated how often I said things? It's dangerous territory when computer scientists ask "what if," but I got curious enough I wanted to follow through.

The Objective¶

Given an input list of Tweets, build up the following things:

The distribution of starting words; since there are no "prior" words to go from, we need to treat this as a special case.
The distribution of words given a previous word; for example, every time I use the word woodchuck in the example sentence, there is a 50% chance it is followed by chuck and a 50% chance it is followed by could. I need this distribution for all words.
The distribution of quantity of hashtags; Do I most often use just one? Two? Do they follow something like a Poisson distribution?
Distribution of hashtags; Given a number of hashtags, what is the actual content? I'll treat hashtags as separate from the content of a tweet.

The Data¶

I'm using as input my tweet history. I don't really use Twitter anymore, but it seems like a fun use of the dataset. I'd like to eventually build this to a point where I can imitate anyone on Twitter using their last 100 tweets or so, but I'll start with this as example code.

The Algorithm¶

I'll be using the NLTK library for doing a lot of the heavy lifting. First, let's import the data:

In [1]:

import pandas as pd

tweets = pd.read_csv('tweets.csv')
text = tweets.text

# Don't include tweets in reply to or mentioning people
replies = text.str.contains('@')
text_norep = text.loc[~replies]

And now that we've got data, let's start crunching. First, tokenize and build out the distribution of first word:

In [2]:

from nltk.tokenize import TweetTokenizer
tknzr = TweetTokenizer()
tokens = text_norep.map(tknzr.tokenize)

first_words = tokens.map(lambda x: x[0])
first_words_alpha = first_words[first_words.str.isalpha()]
first_word_dist = first_words_alpha.value_counts() / len(first_words_alpha)

Next, we need to build out the conditional distributions. That is, what is the probability of the next word given the current word is $X$? This one is a bit more involved. First, find all unique words, and then find what words proceed them. This can probably be done in a more efficient manner than I'm currently doing here, but we'll ignore that for the moment.

In [3]:

from functools import reduce

# Get all possible words
all_words = reduce(lambda x, y: x+y, tokens, [])
unique_words = set(all_words)
actual_words = set([x if x[0] != '.' else None for x in unique_words])

word_dist = {}
for word in iter(actual_words):
    indices = [i for i, j in enumerate(all_words) if j == word]
    proceeding = [all_words[i+1] for i in indices]
    word_dist[word] = proceeding

Now that we've got the tweet analysis done, it's time for the fun part: hashtags! Let's count how many hashtags are in each tweet, I want to get a sense of the distribution.

In [4]:

import matplotlib.pyplot as plt
%matplotlib inline

hashtags = text_norep.str.count('#')
bins = hashtags.unique().max()
hashtags.plot(kind='hist', bins=bins)

Out[4]:

<matplotlib.axes._subplots.AxesSubplot at 0x18e59dc28d0>

That looks like a Poisson distribution, kind of as I expected. I'm guessing my number of hashtags per tweet is $\sim Poi(1)$, but let's actually find the most likely estimator which in this case is just $\bar{\lambda}$:

In [5]:

mle = hashtags.mean()
mle

Out[5]:

0.870236869207003

Pretty close! So we can now simulate how many hashtags are in a tweet. Let's also find what hashtags are actually used:

In [6]:

hashtags = [x for x in all_words if x[0] == '#']
n_hashtags = len(hashtags)

unique_hashtags = list(set([x for x in unique_words if x[0] == '#']))
hashtag_dist = pd.DataFrame({'hashtags': unique_hashtags,
                             'prob': [all_words.count(h) / n_hashtags
                                      for h in unique_hashtags]})
len(hashtag_dist)

Out[6]:

Turns out I have used 603 different hashtags during my time on Twitter. That means I was using a unique hashtag for about every third tweet.

In better news though, we now have all the data we need to go about actually constructing tweets! The process will happen in a few steps:

Randomly select what the first word will be.
Randomly select the number of hashtags for this tweet, and then select the actual hashtags.
Fill in the remaining space of 140 characters with random words taken from my tweets.

And hopefully, we won't have anything too crazy come out the other end. The way we do the selection follows a Multinomial Distribution: given a lot of different values with specific probability, pick one. Let's give a quick example:

x: .33
y: .5
z: .17

That is, I pick x with probability 33%, y with probability 50%, and so on. In context of our sentence construction, I've built out the probabilities of specific words already - now I just need to simulate that distribution. Time for the engine to actually be developed!

In [7]:

import numpy as np

def multinom_sim(n, vals, probs):
    occurrences = np.random.multinomial(n, probs)
    results = occurrences * vals
    return ' '.join(results[results != ''])

def sim_n_hashtags(hashtag_freq):
    return np.random.poisson(hashtag_freq)

def sim_hashtags(n, hashtag_dist):
    return multinom_sim(n, hashtag_dist.hashtags, hashtag_dist.prob)

def sim_first_word(first_word_dist):
    probs = np.float64(first_word_dist.values)
    return multinom_sim(1, first_word_dist.reset_index()['index'], probs)

def sim_next_word(current, word_dist):
    dist = pd.Series(word_dist[current])
    probs = np.ones(len(dist)) / len(dist)
    return multinom_sim(1, dist, probs)

Pulling it all together¶

I've now built out all the code I need to actually simulate a sentence written by me. Let's try doing an example with five words and a single hashtag:

In [8]:

first = sim_first_word(first_word_dist)
second = sim_next_word(first, word_dist)
third = sim_next_word(second, word_dist)
fourth = sim_next_word(third, word_dist)
fifth = sim_next_word(fourth, word_dist)
hashtag = sim_hashtags(1, hashtag_dist)

' '.join((first, second, third, fourth, fifth, hashtag))

Out[8]:

'My first all-nighter of friends #oldschool'

Let's go ahead and put everything together! We're going to simulate a first word, simulate the hashtags, and then simulate to fill the gap until we've either taken up all the space or reached a period.

In [9]:

def simulate_tweet():
    chars_remaining = 140
    first = sim_first_word(first_word_dist)
    n_hash = sim_n_hashtags(mle)
    hashtags = sim_hashtags(n_hash, hashtag_dist)
    
    chars_remaining -= len(first) + len(hashtags)
    
    tweet = first
    current = first
    while chars_remaining > len(tweet) + len(hashtags) and current[0] != '.' and current[0] != '!':
        current = sim_next_word(current, word_dist)
        tweet += ' ' + current
        
    tweet = tweet[:-2] + tweet[-1]
        
    return ' '.join((tweet, hashtags)).strip()

The results¶

And now for something completely different: twenty random tweets dreamed up by my computer and my Twitter data. Here you go:

In [12]:

for i in range(0, 20):
    print(simulate_tweet())
    print()

Also , I'm at 8 this morning. #thursdaysgohard #ornot

Turns out of us breathe the code will want to my undergraduate career is becoming more night trying ? Religion is now as a chane #HYPE

You know what recursion is to review the UNCC. #ornot

There are really sore 3 bonfires in my first writing the library ground floor if awesome. #realtalk #impressed

So we can make it out there's nothing but I'm not let us so hot I could think I may be good. #SwingDance

Happy Christmas , at Harris Teeter to be be godly or Roman Catholic ). #4b392b#4b392b #Isaiah26

For context , I in the most decisive factor of the same for homework. #accomplishment

Freaking done. #loveyouall

New blog post : Don't jump in a quiz in with a knife fight. #haskell #earlybirthday

God shows me legitimately want to get some food and one day.

Stormed the queen city. #mindblown

The day of a cold at least outside right before the semester ..

Finished with the way back. #winners

Waking up , OJ , I feel like Nick Jonas today.

First draft of so hard drive. #humansvszombies

Eric Whitacre is the wise creation.

Ethics paper first , music in close to everyone who just be posting up with my sin , and Jerry Springr #TheLittleThings

Love that you know enough time I've eaten at 8 PM. #deepthoughts #stillblownaway

Lead. #ThinkingTooMuch #Christmas

Aamazing conference when you married #DepartmentOfRedundancyDepartment Yep , but there's a legitimate challenge.

...Which all ended up being a whole lot more nonsensical than I had hoped for. There are some good ones, so I'll call that an accomplishment! I was banking on grammar not being an issue: since my tweets use impeccable grammar, the program modeled off them should have pretty good grammar as well. There are going to be some hilarious edge cases (I'm looking at you, Ethics paper first, music in close to everyone) that make no sense, and some hilarious edge cases (Waking up, OJ, I feel like Nick Jonas today) that make me feel like I should have a Twitter rap career. On the whole though, the structure came out alright.

Moving on from here¶

During class we also talked about an interesting idea: trying to analyze corporate documents and corporate speech. I'd be interested to know what this analysis applied to something like a couple of bank press releases could do. By any means, the code needs some work to clean it up before I get that far.

For further reading¶

I'm pretty confident I re-invented a couple wheels along the way - what I'm doing feels a lot like what Markov Chain Monte Carlo is intended to do. But I've never worked explicitly with that before, so more research is needed.

Predicting Santander Customer Happiness

2016-03-05T00:00:00-05:00

My first Kaggle competition¶

It's time! After embarking on a Machine Learning class this semester, and with a Saturday in which I don't have much planned, I wanted to put this class and training to work. It's my first competition submission. I want to walk you guys through how I'm approaching this problem, because I thought it would be really neat. The competition is Banco Santander's Santander Customer Satisfaction competition. It seemed like an easy enough problem I could actually make decent progress on it.

Data Exploration¶

First up: we need to load our data and do some exploratory work. Because we're going to be using this data for model selection prior to testing, we need to make a further split. I've already gone ahead and done this work, please see the code in the appendix below.

In [1]:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

# Record how long it takes to run the notebook - I'm curious.
from datetime import datetime
start = datetime.now()

dataset = pd.read_csv('split_train.csv')
dataset.index = dataset.ID
X = dataset.drop(['TARGET', 'ID', 'ID.1'], 1)
y = dataset.TARGET

In [2]:

y.unique()

Out[2]:

array([0, 1], dtype=int64)

In [3]:

len(X.columns)

Out[3]:

Okay, so there are only two classes we're predicting: 1 for unsatisfied customers, 0 for satisfied customers. I would have preferred this to be something more like a regression, or predicting multiple classes: maybe the customer isn't the most happy, but is nowhere near closing their accounts. For now though, that's just the data we're working with.

Now, I'd like to make a scatter matrix of everything going on. Unfortunately as noted above, we have 369 different features. There's no way I can graphically make sense of that much data to start with.

We're also not told what the data actually represents: Are these survey results? Average time between contact with a customer care person? Frequency of contacting a customer care person? The idea is that I need to reduce the number of dimensions we're predicting across.

Dimensionality Reduction pt. 1 - Binary Classifiers¶

My first attempt to reduce the data dimensionality is to find all the binary classifiers in the dataset (i.e. 0 or 1 values) and see if any of those are good (or anti-good) predictors of the final data.

In [4]:

cols = X.columns
b_class = []
for c in cols:
    if len(X[c].unique()) == 2:
        b_class.append(c)
        
len(b_class)

Out[4]:

So there are 111 features in the dataset that are a binary label. Let's see if any of them are good at predicting the users satisfaction!

In [5]:

# First we need to `binarize` the data to 0-1; some of the labels are {0, 1},
# some are {0, 3}, etc.
from sklearn.preprocessing import binarize
X_bin = binarize(X[b_class])

accuracy = [np.mean(X_bin[:,i] == y) for i in range(0, len(b_class))]
acc_df = pd.DataFrame({"Accuracy": accuracy}, index=b_class)
acc_df.describe()

Out[5]:

	Accuracy
count	111.000000
mean	0.905159
std	0.180602
min	0.043598
25%	0.937329
50%	0.959372
75%	0.960837
max	0.960837

Wow! Looks like we've got some incredibly predictive features! So much so that we should be a bit concerned. My initial guess for what's happening is that we have a sparsity issue: so many of the values are 0, and these likely happen to line up with satisfied customers.

So the question we must now answer, which I likely should have asked long before now: What exactly is the distribution of un/satisfied customers?

In [6]:

unsat = y[y == 1].count()
print("Satisfied customers: {}; Unsatisfied customers: {}".format(len(y) - unsat, unsat))
naive_guess = np.mean(y == np.zeros(len(y)))
print("Naive guess accuracy: {}".format(naive_guess))

Satisfied customers: 51131; Unsatisfied customers: 2083
Naive guess accuracy: 0.9608561656706882

This is a bit discouraging. A naive guess of "always satisfied" performs as well as our best individual binary classifier. What this tells me then, is that these data columns aren't incredibly helpful in prediction. I'd be interested in a polynomial expansion of this data-set, but for now, that's more computation than I want to take on.

Dimensionality Reduction pt. 2 - LDA¶

Knowing that our naive guess performs so well is a blessing and a curse:

Curse: The threshold for performance is incredibly high: We can only "improve" over the naive guess by 4%
Blessing: All the binary classification features we just discovered are worthless on their own. We can throw them out and reduce the data dimensionality from 369 to 111.

Now, in removing these features from the dataset, I'm not saying that there is no "information" contained within them. There might be. But the only way we'd know is through a polynomial expansion, and I'm not going to take that on within this post.

My initial thought for a "next guess" is to use the LDA model for dimensionality reduction. However, it can only reduce dimensions to $1 - p$, with $p$ being the number of classes. Since this is a binary classification, every LDA model that I try will have dimensionality one; when I actually try this, the predictor ends up being slightly less accurate than the naive guess.

Instead, let's take a different approach to dimensionality reduction: principle components analysis. This allows us to perform the dimensionality reduction without worrying about the number of classes. Then, we'll use a Gaussian Naive Bayes model to actually do the prediction. This model is chosen simply because it doesn't take a long time to fit and compute; because PCA will take so long, I just want a prediction at the end of this. We can worry about using a more sophisticated LDA/QDA/SVM model later.

Now into the actual process: We're going to test out PCA dimensionality reduction from 1 - 20 dimensions, and then predict using a Gaussian Naive Bayes model. The 20 dimensions upper limit was selected because the accuracy never improves after you get beyond that (I found out by running it myself). Hopefully, we'll find that we can create a model better than the naive guess.

In [7]:

from sklearn.naive_bayes import GaussianNB
from sklearn.decomposition import PCA

X_no_bin = X.drop(b_class, 1)

def evaluate_gnb(dims):
    pca = PCA(n_components=dims)
    X_xform = pca.fit_transform(X_no_bin)
    
    gnb = GaussianNB()
    gnb.fit(X_xform, y)
    return gnb.score(X_xform, y)

dim_range = np.arange(1, 21)
plt.plot(dim_range, [evaluate_gnb(dim) for dim in dim_range], label="Gaussian NB Accuracy")
plt.axhline(naive_guess, label="Naive Guess", c='k')
plt.axhline(1 - naive_guess, label="Inverse Naive Guess", c='k')
plt.gcf().set_size_inches(12, 6)
plt.legend();

**sigh...** After all the effort and computational power, we're still at square one: we have yet to beat out the naive guess threshold. With PCA in play we end up performing terribly, but not terribly enough that we can guess against ourselves.

Let's try one last-ditch attempt using the entire data set:

In [8]:

def evaluate_gnb_full(dims):
    pca = PCA(n_components=dims)
    X_xform = pca.fit_transform(X)
    
    gnb = GaussianNB()
    gnb.fit(X_xform, y)
    return gnb.score(X_xform, y)

dim_range = np.arange(1, 21)
plt.plot(dim_range, [evaluate_gnb(dim) for dim in dim_range], label="Gaussian NB Accuracy")
plt.axhline(naive_guess, label="Naive Guess", c='k')
plt.axhline(1 - naive_guess, label="Inverse Naive Guess", c='k')
plt.gcf().set_size_inches(12, 6)
plt.legend();

Nothing. It is interesting to note that the graphs are almost exactly the same: This would imply again that the variables we removed earlier (all the binary classifiers) indeed have almost no predictive power. It seems this problem is high-dimensional, but with almost no data that can actually inform our decisions.

Summary for Day 1¶

After spending a couple hours with this dataset, there seems to be a fundamental issue in play: We have very high-dimensional data, and it has no bearing on our ability to actually predict customer satisfaction. This can be a huge issue: it implies that no matter what model we use, we fundamentally can't perform well. I'm sure most of this is because I'm not an experienced data scientist. Even so, we have yet to develop a strategy that can actually beat out the village idiot; so far, the bank is best off just assuming all its customers are satisfied. Hopefully more to come soon.

In [9]:

end = datetime.now()
print("Running time: {}".format(end - start))

Running time: 0:00:58.715714

Appendix¶

Code used to split the initial training data:

from sklearn.cross_validation import train_test_split
data = pd.read_csv('train.csv')
data.index = data.ID

data_train, data_validate = train_test_split(
    data, train_size=.7)

data_train.to_csv('split_train.csv')
data_validate.to_csv('split_validate.csv')

Profitability using the Investment Formula

2016-02-26T00:00:00-05:00

Profitability using the Investment Formula¶

I've previously talked about crafting an Investment Formula that would guarantee making money if you could predict which direction the stock market was going to go. This is going to be the first in a series of posts trying to flesh out what an actual investment strategy based on this formula would look like.

But first, the formula doesn't take into account two very important things: leverage, and the number of days invested. That's why I want to set up what I'm going to call the Profitability Score.

The definition is going to be very simple:

$p$: Profit made once you exit the investment
$i$: Initial investment into the asset
$m$: Maximum investment in the asset
$l = m / i$: The maximum leverage of an investment, as the ratio of maximum invested to initial investment
$d$: The number of days it takes to turn a profit

$s = \frac{1000 p}{i(l + d)} = \frac{1000 p}{m + i\cdot d}$

Crazy, right? The score is simply the (normalized) profit you made divided by the leverage plus days invested. The $\cdot 1000$ is just to turn the number into something more reasonable - people don't like hearing something with a profitability score of .001 for example.

Theoretical Justification¶

The formula itself is designed to be simple in principle: I like making a profit, and I want to penalize the leverage you incur and days you have to invest. Ideally, we want to have a stock that goes up all the time. However, the investment formula takes advantage of a different case: trying to profit from highly volatile assets. If we can make money when the investment only has one day up, let's do it!

Even so, there are two potential issues: First, stocks that trend upward will have a higher profitability score - both leverage and days invested will be 1. To protect against only investing in this trend, I can do things like taking $\log(d)$. I don't want to start biasing the scoring function until I have a practical reason to do so, so right now I'll leave it standing.

The second issue is how to penalize leverage and days invested relative to each other. As it currently stands, a leverage of 6x with only 1 day invested is the same as leveraging 2x with 3 days invested. In the future, I'd again want to look at making the impact of days invested smaller - I can get over an extra 3 days in the market if it means that I don't have to incur a highly leveraged position.

So there could be things about the scoring function we change in the future, but I want to run some actual tests before we start worrying about things like that!

Running a simulation¶

This won't be an incredibly rigorous backtest, I just want to see some results from the work so far. Let's set up the simulation code again, and start looking into some random stocks. If you've read the last blog post, you can skip over the code. The only difference is that it's been ported to python to make the data-wrangling easier. Julia doesn't yet support some of the multi-index things I'm trying to do.

In [19]:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from Quandl import get as qget
%matplotlib inline
api_key = ''

profitability = lambda p, i, m, d: 1000*p / (m + i*d)

def is_profitable(current_price, purchase_history, open_history):
    shares = (purchase_history / open_history).sum()
    return current_price * shares > sum(purchase_history)

def daily_investment(current_open, current_close, purchase_history, open_history):
    t1 = current_close / current_open - 1
    t2 = (purchase_history - purchase_history * current_close / open_history).sum()
    return t2 / t1

def simulate_day(open_vals, close_vals, init, expected, bias):
    invested = np.array([init])
    
    day = 1
    profitable = is_profitable(close_vals[day-1], invested, open_vals[0:len(invested)]) \
        or is_profitable(open_vals[day], invested, open_vals[0:len(invested)])

    while not profitable:
        expected_close = open_vals[day] * expected
        todays_purchase = daily_investment(open_vals[day], expected_close, invested, open_vals[0:day])
        invested = np.append(invested, todays_purchase + bias)
        # expected_profit = expected_close * (invested / open_vals[0:len(invested)]).sum() - invested.sum()
        day += 1
        profitable = is_profitable(close_vals[day-1], invested, open_vals[0:len(invested)]) \
            or is_profitable(open_vals[day], invested, open_vals[0:len(invested)])
            
    shares = (invested / open_vals[0:len(invested)]).sum()
    
    # Make sure we can't see into the future - we know either today's close or tomorrow's open
    # will be profitable, but we need to check which one.
    if is_profitable(close_vals[day-1], invested, open_vals[0:len(invested)]):
        ending_price = close_vals[day-1]
    else:
        ending_price = open_vals[day]
        
    profit = shares * ending_price - sum(invested)
    return invested, profit

def simulate_ts(name, start, end, initial, expected, bias):
    ticker_info = qget(name, trim_start=start, api_key=api_key)
    evaluation_times = ticker_info[:end].index
    
    # Handle Google vs. YFinance data
    if "Adjusted Close" in ticker_info.columns:
        close_column = "Adjusted Close"
    else:
        close_column = "Close"
        
    sim = {d: simulate_day(ticker_info[d:]["Open"], ticker_info[d:][close_column],
                       100, 1.02, 10) for d in evaluation_times}
    sim_series = pd.Series(sim)
    result = pd.DataFrame()
    result["profit"] = sim_series.apply(lambda x: x[1])
    result["max"] = sim_series.apply(lambda x: max(x[0]))
    result["days"] = sim_series.apply(lambda x: len(x[0]))
    result["score"] = sim_series.apply(lambda x: profitability(x[1], x[0][0], max(x[0]), len(x[0])))
    result["investments"] = sim_series.apply(lambda x: x[0])
    
    return result

def simulate_tickers(tickers):
    from datetime import datetime
    results = {}
    for ticker in tickers:
        start = datetime(2015, 1, 1)
        results_df = simulate_ts(ticker, start, datetime(2016, 1, 1), 100, 1.01, 10)
        results[ticker] = results_df
        
    return pd.concat(list(results.values()), keys=list(results.keys()), axis=1)

And now the interesting part¶

Let's start looking into the data! FANG stocks have been big over the past year, let's see how they look:

In [7]:

fang_df = simulate_tickers(["YAHOO/FB", "YAHOO/AAPL", "YAHOO/NFLX", "YAHOO/GOOG"])

In [8]:

fang_df.xs('days', axis=1, level=1).hist()
plt.gcf().set_size_inches(18, 8);
plt.gcf().suptitle("Distribution of Days Until Profitability", fontsize=18);

In [10]:

fang_df.xs('score', axis=1, level=1).plot()
plt.gcf().set_size_inches(18, 6)
plt.gcf().suptitle("Profitability score over time", fontsize=18);

Let's think about these graphs. First, the histogram. What we like seeing is a lot of 1's - that means there were a lot of days that the stock went up and we didn't have to worry about actually implementing the strategy - we were able to close the trade at a profit.

Looking at the profitability score over time though is a bit more interesting. First off, stocks that are more volatile will tend to have a higher profitability score, no two ways about that. However, Netflix consistently outperformed on this metric. We know that 2015 was a good year for Netflix, so that's a (small) sign the strategy is performing as expected.

The final interesting note happens around the end of August 2015. Around this period, the markets were selling off in a big way due to issues in China (not unlike what's happening now). Even so, all of the FANG stocks saw an uptick in profitability around this time. This is another sign that the strategy being developed performs better during periods of volatility, rather than from riding markets up or down.

What about FANG vs. some cyclicals?

In [13]:

cyclic_df = simulate_tickers(["YAHOO/X", "YAHOO/CAT", "YAHOO/NFLX", "YAHOO/GOOG"])

In [14]:

cyclic_df.xs('days', axis=1, level=1).hist()
plt.gcf().set_size_inches(18, 8);
plt.gcf().suptitle("Distribution of Days Until Profitability", fontsize=18);

In [15]:

cyclic_df.xs('score', axis=1, level=1).plot()
plt.gcf().set_size_inches(18, 6)
plt.gcf().suptitle("Profitability score over time", fontsize=18);

Some more interesting results come from this as well. First off, US Steel (X) has a much smoother distribution of days until profitability - it doesn't have a huge number of values at 1 and then drop off. Intuitively, we're not terribly large fans of this, we want a stock to go up! However, on the profitability score it is the only serious contender to Netflix.

Second, we see the same trend around August - the algorithm performs well in volatile markets.

For a final test, let's try some biotech and ETFs!

In [21]:

biotech_df = simulate_tickers(['YAHOO/REGN', 'YAHOO/CELG', 'GOOG/NASDAQ_BIB', 'GOOG/NASDAQ_IBB'])

In [22]:

biotech_df.xs('days', axis=1, level=1).hist()
plt.gcf().set_size_inches(18, 8);
plt.gcf().suptitle("Distribution of Days Until Profitability", fontsize=18);

In [23]:

biotech_df.xs('score', axis=1, level=1).plot()
plt.gcf().set_size_inches(18, 6)
plt.gcf().suptitle("Profitability score over time", fontsize=18);

In this example, we don't see a whole lot of interesting things: the scores are all fairly close together with notable exceptions in late August, and mid-October.

What is interesting is that during the volatile period, the ETF's performed significantly better than the stocks did in terms of profitability. The leveraged ETF (BIB) performed far above anyone else, and it appears that indeed, it is most profitable during volatile periods. Even so, it was far more likely to take multiple days to give a return. Its count of 1-day investments trails the other ETF and both stocks by a decent margin.

And consider me an OCD freak, but I just really like Celgene's distribution - it looks nice and smooth.

Summary and plans for the next post¶

So far I'm really enjoying playing with this strategy - there's a lot of depth here to understand, though the preliminary results seem to indicate that it profits mostly from taking the other side of a volatile trade. I'd be interested to run results later on data from January - It's been a particularly volatile start to the year so it would be neat to see whether this strategy would work then.

For the next post, I want to start playing with some of the parameters: How do the bias and expected close influence the process? The values have been fairly conservative so far, it will be interesting to see how the simulations respond afterward.

Guaranteed Money Maker

2016-02-03T00:00:00-05:00

If you can see into the future, that is.¶

My previous class in Stochastic Calculus covered a lot of interesting topics, and the important one for today is the Gambler's Ruin problem. If you're interested in some of the theory behind it, also make sure to check out random walks. The important bit is that we studied the Martingale Betting Strategy, which describes for us a guaranteed way to eventually make money.

The strategy goes like this: You are going to toss a fair coin with a friend. If you guess heads or tails correctly, you get back double the money you bet. If you guess incorrectly, you lose money. How should you bet?

The correct answer is that you should double your bet each time you lose. Then when you finally win, you'll be guaranteed to make back everything you lost and then $1 extra! Consider the scenario:

You bet $1, and guess incorrectly. You're 1 dollar in the hole.
You bet $2, and guess incorrectly. You're 3 dollars in the hole now.
You bet $4, and guess incorrectly. You're 7 dollars in the hole.
You bet $8, and guess correctly! You now get back those 8 dollars you bet, plus 8 extra for winning, for a total profit of one dollar!

Mathematically, we can prove that as long as you have unlimited money to bet, you are guaranteed to make money.

Applying the Martingale Strategy¶

But we're all realistic people, and once you start talking about "unlimited money" eyebrows should be raised. Even still, this is an interesting strategy to investigate, and I want to apply it to the stock market. As long as we can guarantee there's a single day in which the stock goes up, we should be able to make money right? The question is just how much we have to invest to guarantee this.

Now it's time for the math. We'll use the following definitions:

$o_i$ = the share price at the opening of day $i$
$c_i$ = the share price at the close of day $i$
$d_i$ = the amount of money we want to invest at the beginning of day $i$

With those definitions in place, I'd like to present the formula that is guaranteed to make you money. I call it Bradlee's Investment Formula:

$c_n \sum_{i=1}^n \frac{d_i}{o_i} > \sum_{i=1}^{n} d_i$

It might not look like much, but if you can manage to make it so that this formula holds true, you will be guaranteed to make money. The intuition behind the formula is this: The closing share price times the number of shares you have purchased ends up greater than the amount of money you invested.

That is, on day $n$, if you know what the closing price will be you can set up the amount of money you invest that day to guarantee you make money. I'll even teach you to figure out how much money that is! Take a look:

$ \begin{align} c_n \sum_{i=1}^{n-1} \frac{d_i}{o_i} + \frac{c_nd_n}{o_n} &> \sum_{i=1}^{n-1}d_i + d_n\\ \frac{c_nd_n}{o_n} - d_n &> \sum_{i=1}^{n-1}(d_i - \frac{c_nd_i}{o_i})\\ d_n (\frac{c_n - o_n}{o_n}) &> \sum_{i=1}^{n-1} d_i(1 - \frac{c_n}{o_i})\\ d_n &> \frac{o_n}{c_n - o_n} \sum_{i=1}^{n-1} d_i(1 - \frac{1}{o_i}) \end{align}$

If you invest exactly $d_n$ that day, you'll break even. But if you can make sure the money you invest is greater than that quantity on the right (which requires that you have a crystal ball tell you the stock's closing price) you are guaranteed to make money!

Interesting Implications¶

On a more serious note though, the formula above tells us a couple of interesting things:

It's impossible to make money without the closing price at some point being greater than the opening price (or vice-versa if you are short selling) - there is no amount of money you can invest that will turn things in your favor.
Close prices of the past aren't important if you're concerned about the bottom line. While chart technicians use price history to make judgment calls, in the end, the closing price on anything other than the last day is irrelevant.
It's possible to make money as long as there is a single day where the closing price is greater than the opening price! You might have to invest a lot to do so, but it's possible.
You must make a prediction about where the stock will close at if you want to know how much to invest. That is, we can set up our investment for the day to make money if the stock goes up 1%, but if it only goes up .5% we'll still lose money.
It's possible the winning move is to scale back your position. Consider the scenario:
- You invest money and the stock closes down the day .5%
- You invest tomorrow expecting the stock to go up 1%
- The winning investment to break even (assuming a 1% increase) is to scale back the position, since the shares you purchased at the beginning would then be profitable

Running the simulation¶

So now that we've defined our investment formula,we need to tweak a couple things in order to make an investment strategy we can actually work with. There are two issues we need to address:

The formula only tells us how much to invest if we want to break even ($d_n$). If we actually want to turn a profit, we need to invest more than that, which we will refer to as the bias.
The formula assumes we know what the closing price will be on any given day. If we don't know this, we can still invest assuming the stock price will close at a level we choose. If the price doesn't meet this objective, we try again tomorrow! This predetermined closing price will be referred to as the expectation.

Now that we've defined our bias and expectation, we can actually build a strategy we can simulate. Much like the martingale strategy told you to bet twice your previous bet in order to make money, we've designed a system that tells us how much to bet in order to make money as well.

Now, let's get to the code!

In [1]:

using Quandl
api_key = ""
daily_investment = function(current_open, current_close, purchase_history, open_history)
    # We're not going to safeguard against divide by 0 - that's the user's responsibility
    t1 = current_close / current_open - 1
    t2 = sum(purchase_history - purchase_history*current_close ./ open_history)
    return t2 / t1
end;

And let's code a way to run simulations quickly:

In [2]:

is_profitable = function(current_price, purchase_history, open_history)
    shares = sum(purchase_history ./ open_history)
    return current_price*shares > sum(purchase_history)
end

simulate = function(name, start, init, expected, bias)
    ticker_info = quandlget(name, from=start, api_key=api_key)
    open_vals = ticker_info["Open"].values
    close_vals = ticker_info["Close"].values
    invested = [init]
    
    # The simulation stops once we've made a profit
    day = 1
    profitable = is_profitable(close_vals[day], invested, open_vals[1:length(invested)]) ||
        is_profitable(open_vals[day+1], invested, open_vals[1:length(invested)])
    while !profitable
        expected_close = open_vals[day+1] * expected
        todays_purchase = daily_investment(open_vals[day+1], expected_close, invested, open_vals[1:day])
        invested = [invested; todays_purchase + bias]
        # expected_profit = expected_close * sum(invested ./ open_vals[1:length(invested)]) - sum(invested)
        day += 1
        profitable = is_profitable(close_vals[day], invested, open_vals[1:length(invested)]) ||
            is_profitable(open_vals[day+1], invested, open_vals[1:length(invested)])
    end
    
    shares = sum(invested ./ open_vals[1:length(invested)])
    max_profit = max(close_vals[day], open_vals[day+1])
    profit = shares * max_profit - sum(invested)
    return (invested, profit)
end

sim_summary = function(investments, profit)
    leverages = [sum(investments[1:i]) for i=1:length(investments)]
    max_leverage = maximum(leverages) / investments[1]
    println("Max leverage: $(max_leverage)")
    println("Days invested: $(length(investments))")
    println("Profit: $profit")
end;

Now, let's get some data and run a simulation! Our first test:

We'll invest 100 dollars in LMT, and expect that the stock will close up 1% every day. We'll invest $d_n$ + 10 dollars every day that we haven't turned a profit, and end the simulation once we've made a profit.

In [3]:

investments, profit = simulate("YAHOO/LMT", Date(2015, 11, 29), 100, 1.01, 10)
sim_summary(investments, profit)

Max leverage: 5.590373200042106
Days invested: 5
Profit: 0.6894803101560001

The result: We need to invest 5.6x our initial position over a period of 5 days to make approximately .69¢

Now let's try the same thing, but we'll assume the stock closes up 2% instead.

In [4]:

investments, profit = simulate("YAHOO/LMT", Date(2015, 11, 29), 100, 1.02, 10)
sim_summary(investments, profit)

Max leverage: 1.854949900247809
Days invested: 25
Profit: 0.08304813163696423

In this example, we only get up to a 1.85x leveraged position, but it takes 25 days to turn a profit of 8¢

Summary¶

We've defined an investment strategy that can tell us how much to invest when we know what the closing position of a stock will be. We can tweak the strategy to actually make money, but plenty of work needs to be done so that we can optimize the money invested.

In the next post I'm going to post more information about some backtests and strategy tests on this strategy (unless of course this experiment actually produces a significant profit potential, and then I'm keeping it for myself).

Side note and disclaimer¶

The claims made in this presentation about being able to guarantee making money are intended as a joke and do not constitute investment advice of any sort.

Cloudy in Seattle

2016-01-23T00:00:00-05:00

In [1]:

import pickle
import pandas as pd
import numpy as np
from bokeh.plotting import output_notebook, figure, show
from bokeh.palettes import RdBu4 as Palette
from datetime import datetime
import warnings

output_notebook()

BokehJS successfully loaded.

After taking some time to explore how the weather in North Carolina stacked up over the past years, I was interested in doing the same analysis for other cities. Growing up with family from Binghamton, NY I was always told it was very cloudy there. And Seattle has a nasty reputation for being very depressing and cloudy. All said, the cities I want to examine are:

Binghamton, NY
Cary, NC
Seattle, WA
New York City, NY

I'd be interested to try this analysis worldwide at some point - comparing London and Seattle might be an interesting analysis. For now though, we'll stick with trying out the US data.

There will be plenty of charts. I want to know: How has average cloud cover and precipitation chance changed over the years for each city mentioned? This will hopefully tell us whether Seattle has actually earned its reputation for being a depressing city.

In [2]:

city_forecasts = pickle.load(open('city_forecasts.p', 'rb'))
forecasts_df = pd.DataFrame.from_dict(city_forecasts)

In [3]:

cities = ['binghamton', 'cary', 'nyc', 'seattle']
city_colors = {cities[i]: Palette[i] for i in range(0, 4)}

def safe_cover(frame):
    if frame and 'cloudCover' in frame:
        return frame['cloudCover']
    else:
        return np.NaN

def monthly_avg_cloudcover(city, year, month):
    dates = pd.DatetimeIndex(start=datetime(year, month, 1, 12),
                             end=datetime(year, month + 1, 1, 12),
                             freq='D', closed='left')
    cloud_cover_vals = list(map(lambda x: safe_cover(forecasts_df[city][x]['currently']), dates))
    cloud_cover_samples = len(list(filter(lambda x: x is not np.NaN, cloud_cover_vals)))
    # Ignore an issue with nanmean having all NaN values. We'll discuss the data issues below.
    with warnings.catch_warnings():
        warnings.simplefilter('ignore')
        return np.nanmean(cloud_cover_vals), cloud_cover_samples

In [4]:

years = range(1990, 2016)
def city_avg_cc(city, month):
    return [monthly_avg_cloudcover(city, y, month) for y in years]

months = [
    ('July', 7),
    ('August', 8),
    ('September', 9),
    ('October', 10),
    ('November', 11)
]

for month, month_id in months:
    month_averages = {city: city_avg_cc(city, month_id) for city in cities}
    f = figure(title="{} Average Cloud Cover".format(month),
               x_axis_label='Year',
               y_axis_label='Cloud Cover Percentage')
    for city in cities:
        f.line(years, [x[0] for x in month_averages[city]],
               legend=city, color=city_colors[city])
    show(f)

Well, as it so happens it looks like there are some data issues. July's data is a bit sporadic, and 2013 seems to be missing from most months as well. I think really only two things can really be confirmed here:

Seattle, specifically for the months of October and November, is in fact significantly more cloudy on average than are other cities
All cities surveyed have seen average cloud cover decline over the months studied. There are data issues, but the trend seems clear.

Let's now move from cloud cover data to looking at average rainfall chance.

In [5]:

def safe_precip(frame):
    if frame and 'precipProbability' in frame:
        return frame['precipProbability']
    else:
        return np.NaN

def monthly_avg_precip(city, year, month):
    dates = pd.DatetimeIndex(start=datetime(year, month, 1, 12),
                             end=datetime(year, month + 1, 1, 12),
                             freq='D', closed='left')
    precip_vals = list(map(lambda x: safe_precip(forecasts_df[city][x]['currently']), dates))
    precip_samples = len(list(filter(lambda x: x is not np.NaN, precip_vals)))
    # Ignore an issue with nanmean having all NaN values. We'll discuss the data issues below.
    with warnings.catch_warnings():
        warnings.simplefilter('ignore')
        return np.nanmean(precip_vals), precip_samples

def city_avg_precip(city, month):
    return [monthly_avg_precip(city, y, month) for y in years]

for month, month_id in months:
    month_averages = {city: city_avg_cc(city, month_id) for city in cities}
    f = figure(title="{} Average Precipitation Chance".format(month),
               x_axis_label='Year',
               y_axis_label='Precipitation Chance Percentage')
    for city in cities:
        f.line(years, [x[0] for x in month_averages[city]],
               legend=city, color=city_colors[city])
    show(f)

The same data issue caveats apply here: 2013 seems to be missing some data, and July has some issues as well. However, this seems to confirm the trends we saw with cloud cover:

Seattle, specifically for the months of August, October, and November has had a consistently higher chance of rain than other cities surveyed.
Average precipitation chance, just like average cloud cover, has been trending down over time.

Conclusion¶

I have to admit I was a bit surprised after doing this analysis. Seattle showed a higher average cloud cover and average precipitation chance than did the other cities surveyed. Maybe Seattle is actually an objectively more depressing city to live in.

Well that's all for weather data at the moment. It's been a great experiment, but I think this is about as far as I'll be able to get with weather data without some domain knowledge. Talk again soon!

Bradlee Speice

Event Studies and Earnings Releases

The Market Just Knew¶

Formulating the Question¶

Event Studies¶

Event Type 1: Trending down over the past N days¶

Event Type 2: Trending up for N days¶

Conclusion and Summary¶

Appendix¶

The Unfair Casino

Proving we can detect cheating¶

The Edge Cases¶

Edge Case 1¶

Detection Conclusion¶

Simulated Annealing¶

Deriving the Likelihood function¶

The process of Simulated Annealing¶

The actual code¶

Catching the Casino¶

Attempt 1¶

Attempt 2¶

Attempt 3¶

Attempt 4¶

Attempt 5¶

Conclusion¶

Justification of Simulated Annealing¶

Why Simulated Annealing?¶

Why not something else?¶

Tick Tock...

2.5 Billion¶

Statistical Considerations¶

Getting the data¶

Registering an application¶

Handling OAuth 2¶

Requesting the data¶

Wild Extrapolations from Small Data¶

How many heartbeats so far?¶

How many heartbeats longer?¶

Summary¶

Tweet Like Me

The Objective¶

The Data¶

The Algorithm¶

Pulling it all together¶

The results¶

Moving on from here¶

For further reading¶

Predicting Santander Customer Happiness

My first Kaggle competition¶

Data Exploration¶

Dimensionality Reduction pt. 1 - Binary Classifiers¶

Dimensionality Reduction pt. 2 - LDA¶

Summary for Day 1¶

Appendix¶

Profitability using the Investment Formula

Profitability using the Investment Formula¶

Theoretical Justification¶

Running a simulation¶

And now the interesting part¶

Summary and plans for the next post¶

Guaranteed Money Maker

If you can see into the future, that is.¶

Applying the Martingale Strategy¶

Interesting Implications¶

Running the simulation¶

Summary¶

Side note and disclaimer¶

Cloudy in Seattle

Conclusion¶