{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from dateutil import parser as dtparser\n",
    "from dateutil.relativedelta import relativedelta\n",
    "from datetime import datetime\n",
    "from html.parser import HTMLParser\n",
    "from copy import copy\n",
    "import Quandl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Testing Cramer\n",
    "\n",
    "Pursuant to attending a graduate school studying Financial Engineering, I've been a fan of the [Mad Money][3] TV show featuring the bombastic Jim Cramer. One of the things that he's said is that you shouldn't use the futures to predict where the stock market is going to go. But he says it often enough, I've begun to wonder - who is he trying to convince?\n",
    "\n",
    "It makes sense that because futures on things like the S&P 500 are traded continuously, they would price in market information before the stock market opens. So is Cramer right to be convinced that strategies based on the futures are a poor idea? I wanted to test it out.\n",
    "\n",
    "The first question is where to get the future's data. I've been part of [Seeking Alpha][2] for a bit, and they publish the [Wall Street Breakfast][3] newsletter which contains daily future's returns as of 6:20 AM EST. I'd be interested in using that data to see if we can actually make some money.\n",
    "\n",
    "First though, let's get the data:\n",
    "\n",
    "# Downloading Futures data from Seeking Alpha\n",
    "\n",
    "We're going to define two HTML parsing classes - one to get the article URL's from a page, and one to get the actual data from each article.\n",
    "\n",
    "[1]: http://www.cnbc.com/mad-money/\n",
    "[2]: http://seekingalpha.com/\n",
    "[3]: http://seekingalpha.com/author/wall-street-breakfast?s=wall-street-breakfast"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class ArticleListParser(HTMLParser):\n",
    "    \"\"\"Given a web page with articles on it, parse out the article links\"\"\"\n",
    "    \n",
    "    articles = []\n",
    "    \n",
    "    def handle_starttag(self, tag, attrs):\n",
    "        #if tag == 'div' and (\"id\", \"author_articles_wrapper\") in attrs:\n",
    "        #    self.fetch_links = True\n",
    "        if tag == 'a' and ('class', 'dashboard_article_link') in attrs:\n",
    "            href = list(filter(lambda x: x[0] == 'href', attrs))[0][1]\n",
    "            self.articles.append(href)\n",
    "            \n",
    "base_url = \"http://seekingalpha.com/author/wall-street-breakfast/articles\"\n",
    "article_page_urls = [base_url] + [base_url + '/{}'.format(i) for i in range(2, 20)]\n",
    "\n",
    "global_articles = []\n",
    "for page in article_page_urls:\n",
    "    # We need to switch the user agent, as SA blocks the standard requests agent\n",
    "    articles_html = requests.get(page,\n",
    "                                headers={\"User-Agent\": \"Wget/1.13.4\"})\n",
    "    parser = ArticleListParser()\n",
    "    parser.feed(articles_html.text)\n",
    "    global_articles += (parser.articles)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "class ArticleReturnParser(HTMLParser):\n",
    "    \"Given an article, parse out the futures returns in it\"\n",
    "    \n",
    "    record_font_tags = False\n",
    "    in_font_tag = False\n",
    "    counter = 0\n",
    "    # data = {} # See __init__\n",
    "    \n",
    "    def __init__(self, *args, **kwargs):\n",
    "        super().__init__(*args, **kwargs)\n",
    "        self.data = {}\n",
    "    \n",
    "    def handle_starttag(self, tag, attrs):\n",
    "        if tag == 'span' and ('itemprop', 'datePublished') in attrs:\n",
    "            date_string = list(filter(lambda x: x[0] == 'content', attrs))[0][1]\n",
    "            date = dtparser.parse(date_string)\n",
    "            self.data['date'] = date\n",
    "            \n",
    "        self.in_font_tag = tag == 'font'\n",
    "        \n",
    "    def safe_float(self, string):\n",
    "        try:\n",
    "            return float(string[:-1]) / 100\n",
    "        except ValueError:\n",
    "            return np.NaN\n",
    "            \n",
    "    def handle_data(self, content):\n",
    "        if not self.record_font_tags and \"Futures at 6\" in content:\n",
    "            self.record_font_tags = True\n",
    "            \n",
    "        if self.record_font_tags and self.in_font_tag:\n",
    "            if self.counter == 0:\n",
    "                self.data['DOW'] = self.safe_float(content)\n",
    "            elif self.counter == 1:\n",
    "                self.data['S&P'] = self.safe_float(content)\n",
    "            elif self.counter == 2:\n",
    "                self.data['NASDAQ'] = self.safe_float(content)\n",
    "            elif self.counter == 3:\n",
    "                self.data['Crude'] = self.safe_float(content)\n",
    "            elif self.counter == 4:\n",
    "                self.data['Gold'] = self.safe_float(content)\n",
    "            \n",
    "            self.counter += 1\n",
    "            \n",
    "    def handle_endtag(self, tag):\n",
    "        self.in_font_tag = False\n",
    "\n",
    "def retrieve_data(url):\n",
    "    sa = \"http://seekingalpha.com\"\n",
    "    article_html = requests.get(sa + url,\n",
    "                               headers={\"User-Agent\": \"Wget/1.13.4\"})\n",
    "    parser = ArticleReturnParser()\n",
    "    parser.feed(article_html.text)\n",
    "    parser.data.update({\"url\": url})\n",
    "    parser.data.update({\"text\": article_html.text})\n",
    "    return parser.data\n",
    "\n",
    "# This copy **MUST** be in place. I'm not sure why,\n",
    "# as you'd think that the data being returned would already\n",
    "# represent a different memory location. Even so, it blows up\n",
    "# if you don't do this.\n",
    "article_list = list(set(global_articles))\n",
    "article_data = [copy(retrieve_data(url)) for url in article_list]\n",
    "# If there's an issue downloading the article, drop it.\n",
    "article_df = pd.DataFrame.from_dict(article_data).dropna()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fetching the Returns data\n",
    "\n",
    "Now that we have the futures data, we're going to compare across 4 different indices - the S&P 500 index, Dow Jones Industrial, Russell 2000, and NASDAQ 100. Let's get the data off of Quandl to make things easier!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# article_df is sorted by date, so we get the first row.\n",
    "start_date = article_df.sort_values(by='date').iloc[0]['date'] - relativedelta(days=1)\n",
    "SPY = Quandl.get(\"GOOG/NYSE_SPY\", trim_start=start_date)\n",
    "DJIA = Quandl.get(\"GOOG/AMS_DIA\", trim_start=start_date)\n",
    "RUSS = Quandl.get(\"GOOG/AMEX_IWM\", trim_start=start_date)\n",
    "NASDAQ = Quandl.get(\"GOOG/EPA_QQQ\", trim_start=start_date)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Running the Comparison\n",
    "\n",
    "There are two types of tests I want to determine: How accurate each futures category is at predicting the index's opening change over the close before, and predicting the index's daily return.\n",
    "\n",
    "Let's first calculate how good each future is at predicting the opening return over the previous day. I expect that the futures will be more than 50% accurate, since the information is recorded 3 hours before the markets open."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Articles Checked: \n",
      "        DJIA  NASDAQ  RUSS  SPY\n",
      "Crude    268     268   271  271\n",
      "DOW      268     268   271  271\n",
      "Gold     268     268   271  271\n",
      "NASDAQ   268     268   271  271\n",
      "S&P      268     268   271  271\n",
      "\n",
      "Prediction Accuracy:\n",
      "            DJIA    NASDAQ      RUSS       SPY\n",
      "Crude   0.544776  0.522388  0.601476  0.590406\n",
      "DOW     0.611940  0.604478  0.804428  0.841328\n",
      "Gold    0.462687  0.455224  0.464945  0.476015\n",
      "NASDAQ  0.615672  0.608209  0.797048  0.830258\n",
      "S&P     0.604478  0.597015  0.811808  0.848708\n"
     ]
    }
   ],
   "source": [
    "def calculate_opening_ret(frame):\n",
    "    # I'm not a huge fan of the appending for loop,\n",
    "    # but it's a bit verbose for a comprehension\n",
    "    data = {}\n",
    "    for i in range(1, len(frame)):\n",
    "        date = frame.iloc[i].name\n",
    "        prior_close = frame.iloc[i-1]['Close']\n",
    "        open_val = frame.iloc[i]['Open']\n",
    "        data[date] = (open_val - prior_close) / prior_close\n",
    "        \n",
    "    return data\n",
    "\n",
    "SPY_open_ret = calculate_opening_ret(SPY)\n",
    "DJIA_open_ret = calculate_opening_ret(DJIA)\n",
    "RUSS_open_ret = calculate_opening_ret(RUSS)\n",
    "NASDAQ_open_ret = calculate_opening_ret(NASDAQ)\n",
    "\n",
    "def signs_match(list_1, list_2):\n",
    "    # This is a surprisingly difficult task - we have to match\n",
    "    # up the dates in order to check if opening returns actually match\n",
    "    index_dict_dt = {key.to_datetime(): list_2[key] for key in list_2.keys()}\n",
    "    \n",
    "    matches = []\n",
    "    for row in list_1.iterrows():\n",
    "        row_dt = row[1][1]\n",
    "        row_value = row[1][0]\n",
    "        index_dt = datetime(row_dt.year, row_dt.month, row_dt.day)\n",
    "        if index_dt in list_2:\n",
    "            index_value = list_2[index_dt]\n",
    "            if (row_value > 0 and index_value > 0) or \\\n",
    "                (row_value < 0 and index_value < 0) or \\\n",
    "                (row_value == 0 and index_value == 0):\n",
    "                    matches += [1]\n",
    "            else:\n",
    "                matches += [0]\n",
    "            #print(\"{}\".format(list_2[index_dt]))\n",
    "    return matches\n",
    "    \n",
    "    \n",
    "prediction_dict = {}\n",
    "matches_dict = {}\n",
    "count_dict = {}\n",
    "index_dict = {\"SPY\": SPY_open_ret, \"DJIA\": DJIA_open_ret, \"RUSS\": RUSS_open_ret, \"NASDAQ\": NASDAQ_open_ret}\n",
    "indices = [\"SPY\", \"DJIA\", \"RUSS\", \"NASDAQ\"]\n",
    "futures = [\"Crude\", \"Gold\", \"DOW\", \"NASDAQ\", \"S&P\"]\n",
    "for index in indices:\n",
    "    matches_dict[index] = {future: signs_match(article_df[[future, 'date']],\n",
    "                                               index_dict[index]) for future in futures}\n",
    "    count_dict[index] = {future: len(matches_dict[index][future]) for future in futures}\n",
    "    prediction_dict[index] = {future: np.mean(matches_dict[index][future])\n",
    "                              for future in futures}\n",
    "print(\"Articles Checked: \")\n",
    "print(pd.DataFrame.from_dict(count_dict))\n",
    "print()\n",
    "print(\"Prediction Accuracy:\")\n",
    "print(pd.DataFrame.from_dict(prediction_dict))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This data is very interesting. Some insights:\n",
    "\n",
    "- Both DOW and NASDAQ futures are pretty bad at predicting their actual market openings\n",
    "- NASDAQ and Dow are fairly unpredictable; Russell 2000 and S&P are very predictable\n",
    "- Gold is a poor predictor in general - intuitively Gold should move inverse to the market, but it appears to be about as accurate as a coin flip.\n",
    "\n",
    "All said though it appears that futures data is important for determining market direction for both the S&P 500 and Russell 2000. Cramer is half-right: futures data isn't very helpful for the Dow and NASDAQ indices, but is great for the S&P and Russell indices.\n",
    "\n",
    "# The next step - Predicting the close\n",
    "\n",
    "Given the code we currently have, I'd like to predict the close of the market as well. We can re-use most of the code, so let's see what happens:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Articles Checked:\n",
      "        DJIA  NASDAQ  RUSS  SPY\n",
      "Crude    268     268   271  271\n",
      "DOW      268     268   271  271\n",
      "Gold     268     268   271  271\n",
      "NASDAQ   268     268   271  271\n",
      "S&P      268     268   271  271\n",
      "\n",
      "Prediction Accuracy:\n",
      "            DJIA    NASDAQ      RUSS       SPY\n",
      "Crude   0.533582  0.529851  0.501845  0.542435\n",
      "DOW     0.589552  0.608209  0.535055  0.535055\n",
      "Gold    0.455224  0.451493  0.483395  0.512915\n",
      "NASDAQ  0.582090  0.626866  0.531365  0.538745\n",
      "S&P     0.585821  0.608209  0.535055  0.535055\n"
     ]
    }
   ],
   "source": [
    "def calculate_closing_ret(frame):\n",
    "    # I'm not a huge fan of the appending for loop,\n",
    "    # but it's a bit verbose for a comprehension\n",
    "    data = {}\n",
    "    for i in range(0, len(frame)):\n",
    "        date = frame.iloc[i].name\n",
    "        open_val = frame.iloc[i]['Open']\n",
    "        close_val = frame.iloc[i]['Close']\n",
    "        data[date] = (close_val - open_val) / open_val\n",
    "        \n",
    "    return data\n",
    "\n",
    "SPY_close_ret = calculate_closing_ret(SPY)\n",
    "DJIA_close_ret = calculate_closing_ret(DJIA)\n",
    "RUSS_close_ret = calculate_closing_ret(RUSS)\n",
    "NASDAQ_close_ret = calculate_closing_ret(NASDAQ)\n",
    "\n",
    "def signs_match(list_1, list_2):\n",
    "    # This is a surprisingly difficult task - we have to match\n",
    "    # up the dates in order to check if opening returns actually match\n",
    "    index_dict_dt = {key.to_datetime(): list_2[key] for key in list_2.keys()}\n",
    "    \n",
    "    matches = []\n",
    "    for row in list_1.iterrows():\n",
    "        row_dt = row[1][1]\n",
    "        row_value = row[1][0]\n",
    "        index_dt = datetime(row_dt.year, row_dt.month, row_dt.day)\n",
    "        if index_dt in list_2:\n",
    "            index_value = list_2[index_dt]\n",
    "            if (row_value > 0 and index_value > 0) or \\\n",
    "                (row_value < 0 and index_value < 0) or \\\n",
    "                (row_value == 0 and index_value == 0):\n",
    "                    matches += [1]\n",
    "            else:\n",
    "                matches += [0]\n",
    "            #print(\"{}\".format(list_2[index_dt]))\n",
    "    return matches\n",
    "    \n",
    "    \n",
    "matches_dict = {}\n",
    "count_dict = {}\n",
    "prediction_dict = {}\n",
    "index_dict = {\"SPY\": SPY_close_ret, \"DJIA\": DJIA_close_ret,\n",
    "              \"RUSS\": RUSS_close_ret, \"NASDAQ\": NASDAQ_close_ret}\n",
    "indices = [\"SPY\", \"DJIA\", \"RUSS\", \"NASDAQ\"]\n",
    "futures = [\"Crude\", \"Gold\", \"DOW\", \"NASDAQ\", \"S&P\"]\n",
    "for index in indices:\n",
    "    matches_dict[index] = {future: signs_match(article_df[[future, 'date']],\n",
    "                                               index_dict[index]) for future in futures}\n",
    "    count_dict[index] = {future: len(matches_dict[index][future]) for future in futures}\n",
    "    prediction_dict[index] = {future: np.mean(matches_dict[index][future])\n",
    "                              for future in futures}\n",
    "    \n",
    "print(\"Articles Checked:\")\n",
    "print(pd.DataFrame.from_dict(count_dict))\n",
    "print()\n",
    "print(\"Prediction Accuracy:\")\n",
    "print(pd.DataFrame.from_dict(prediction_dict))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, it appears that the futures data is terrible at predicting market close. NASDAQ predicting NASDAQ is the most interesting data point, but 63% accuracy isn't accurate enough to make money consistently.\n",
    "\n",
    "# Final sentiments\n",
    "\n",
    "The data bears out very close to what I expected would happen:\n",
    "\n",
    "- Futures data is more accurate than a coin flip for predicting openings, which makes sense since it is recorded only 3 hours before the actual opening\n",
    "- Futures data is about as acccurate as a coin flip for predicting closings, which means there is no money to be made in trying to predict the market direction for the day given the futures data.\n",
    "\n",
    "In summary:\n",
    "\n",
    "- Cramer is half right: Futures data is not good for predicting the market open of the Dow and NASDAQ indices. Contrary to Cramer though, it is very good for predicting the S&P and Russell indices - we can achieve an accuracy slightly over 80% for each. \n",
    "- Making money in the market is hard. We can't just go to the futures and treat them as an oracle for where the market will close.\n",
    "\n",
    "I hope you've enjoyed this, I quite enjoyed taking a deep dive in the analytics this way. I'll be posting more soon!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
