speice.io/2015/12/testing-cramer/index.html

<!doctype html><html lang=en dir=ltr class="blog-wrapper blog-post-page plugin-blog plugin-id-default" data-has-hydrated=false><meta charset=UTF-8><meta name=generator content="Docusaurus v3.6.0"><title data-rh=true>Testing Cramer | The Old Speice Guy</title><meta data-rh=true name=viewport content="width=device-width,initial-scale=1.0"><meta data-rh=true name=twitter:card content=summary_large_image><meta data-rh=true property=og:url content=https://speice.io/2015/12/testing-cramer><meta data-rh=true property=og:locale content=en><meta data-rh=true name=docusaurus_locale content=en><meta data-rh=true name=docusaurus_tag content=default><meta data-rh=true name=docsearch:language content=en><meta data-rh=true name=docsearch:docusaurus_tag content=default><meta data-rh=true property=og:title content="Testing Cramer | The Old Speice Guy"><meta data-rh=true name=description content="Pursuant to attending a graduate school studying Financial Engineering, I've been a fan of the Mad Money TV show featuring the bombastic Jim Cramer. One of the things that he's said is that you shouldn't use the futures to predict where the stock market is going to go. But he says it often enough, I've begun to wonder - who is he trying to convince?"><meta data-rh=true property=og:description content="Pursuant to attending a graduate school studying Financial Engineering, I've been a fan of the Mad Money TV show featuring the bombastic Jim Cramer. One of the things that he's said is that you shouldn't use the futures to predict where the stock market is going to go. But he says it often enough, I've begun to wonder - who is he trying to convince?"><meta data-rh=true property=og:type content=article><meta data-rh=true property=article:published_time content=2015-12-26T12:00:00.000Z><link data-rh=true rel=icon href=/img/favicon.ico><link data-rh=true rel=canonical href=https://speice.io/2015/12/testing-cramer><link data-rh=true rel=alternate href=https://speice.io/2015/12/testing-cramer hreflang=en><link data-rh=true rel=alternate href=https://speice.io/2015/12/testing-cramer hreflang=x-default><script data-rh=true type=application/ld+json>{"@context":"https://schema.org","@id":"https://speice.io/2015/12/testing-cramer","@type":"BlogPosting","author":{"@type":"Person","name":"Bradlee Speice"},"dateModified":"2024-11-03T23:57:32.000Z","datePublished":"2015-12-26T12:00:00.000Z","description":"Pursuant to attending a graduate school studying Financial Engineering, I've been a fan of the Mad Money TV show featuring the bombastic Jim Cramer. One of the things that he's said is that you shouldn't use the futures to predict where the stock market is going to go. But he says it often enough, I've begun to wonder - who is he trying to convince?","headline":"Testing Cramer","isPartOf":{"@id":"https://speice.io/","@type":"Blog","name":"Blog"},"keywords":[],"mainEntityOfPage":"https://speice.io/2015/12/testing-cramer","name":"Testing Cramer","url":"https://speice.io/2015/12/testing-cramer"}</script><link rel=alternate type=application/rss+xml href=/rss.xml title="The Old Speice Guy RSS Feed"><link rel=alternate type=application/atom+xml href=/atom.xml title="The Old Speice Guy Atom Feed"><link rel=stylesheet href=https://cdn.jsdelivr.net/npm/katex@0.13.24/dist/katex.min.css integrity=sha384-odtC+0UGzzFL/6PNoE8rX/SPcQDXBJ+uRepguP4QkPCm2LBxH3FA3y+fKSiJ+AmM crossorigin><link rel=stylesheet href=/assets/css/styles.ae6ff4a3.css><script src=/assets/js/runtime~main.751b419d.js defer></script><script src=/assets/js/main.62ce6156.js defer></script><body class=navigation-with-keyboard><script>!function(){var t,e=function(){try{return new URLSearchParams(window.location.search).get("docusaurus-theme")}catch(t){}}()||function(){try{return window.localStorage.getItem("theme")}catch(t){}}();t=null!==e?e:"light",document.documentElement.setAttribute("data-theme",t)}(),function(){try{for(var[t,e]of new URLSearchParams(window.location.search).entries())if(t.startsWith("docusaurus-data-")){var a=t.replace("docusaurus-data-","data-");document.documentElement.setAttribute(a,e)}}catch(t){}}()</script><div id=__docusaurus><div role=region aria-label="Skip to main content"><a class=skipToContent_fXgn href=#__docusaurus_skipToContent_fallback>Skip to main content</a></div><nav aria-label=Main class="navbar navbar--fixed-top"><div class=navbar__inner><div class=navbar__items><button aria-label="Toggle navigation bar" aria-expanded=false class="navbar__toggle clean-btn" type=button><svg width=30 height=30 viewBox="0 0 30 30" aria-hidden=true><path stroke=currentColor stroke-linecap=round stroke-miterlimit=10 stroke-width=2 d="M4 7h22M4 15h22M4 23h22"/></svg></button><a class=navbar__brand href=/><div class=navbar__logo><img src=/img/logo.svg alt="Sierpinski Gasket" class="themedComponent_mlkZ themedComponent--light_NVdE"><img src=/img/logo-dark.svg alt="Sierpinski Gasket" class="themedComponent_mlkZ themedComponent--dark_xIcU"></div><b class="navbar__title text--truncate">The Old Speice Guy</b></a></div><div class="navbar__items navbar__items--right"><a href=https://github.com/bspeice target=_blank rel="noopener noreferrer" class="navbar__item navbar__link header-github-link"></a><div class="toggle_vylO colorModeToggle_DEke"><button class="clean-btn toggleButton_gllP toggleButtonDisabled_aARS" type=button disabled title="Switch between dark and light mode (currently light mode)" aria-label="Switch between dark and light mode (currently light mode)" aria-live=polite aria-pressed=false><svg viewBox="0 0 24 24" width=24 height=24 class=lightToggleIcon_pyhR><path fill=currentColor d="M12,9c1.65,0,3,1.35,3,3s-1.35,3-3,3s-3-1.35-3-3S10.35,9,12,9 M12,7c-2.76,0-5,2.24-5,5s2.24,5,5,5s5-2.24,5-5 S14.76,7,12,7L12,7z M2,13l2,0c0.55,0,1-0.45,1-1s-0.45-1-1-1l-2,0c-0.55,0-1,0.45-1,1S1.45,13,2,13z M20,13l2,0c0.55,0,1-0.45,1-1 s-0.45-1-1-1l-2,0c-0.55,0-1,0.45-1,1S19.45,13,20,13z M11,2v2c0,0.55,0.45,1,1,1s1-0.45,1-1V2c0-0.55-0.45-1-1-1S11,1.45,11,2z M11,20v2c0,0.55,0.45,1,1,1s1-0.45,1-1v-2c0-0.55-0.45-1-1-1C11.45,19,11,19.45,11,20z M5.99,4.58c-0.39-0.39-1.03-0.39-1.41,0 c-0.39,0.39-0.39,1.03,0,1.41l1.06,1.06c0.39,0.39,1.03,0.39,1.41,0s0.39-1.03,0-1.41L5.99,4.58z M18.36,16.95 c-0.39-0.39-1.03-0.39-1.41,0c-0.39,0.39-0.39,1.03,0,1.41l1.06,1.06c0.39,0.39,1.03,0.39,1.41,0c0.39-0.39,0.39-1.03,0-1.41 L18.36,16.95z M19.42,5.99c0.39-0.39,0.39-1.03,0-1.41c-0.39-0.39-1.03-0.39-1.41,0l-1.06,1.06c-0.39,0.39-0.39,1.03,0,1.41 s1.03,0.39,1.41,0L19.42,5.99z M7.05,18.36c0.39-0.39,0.39-1.03,0-1.41c-0.39-0.39-1.03-0.39-1.41,0l-1.06,1.06 c-0.39,0.39-0.39,1.03,0,1.41s1.03,0.39,1.41,0L7.05,18.36z"/></svg><svg viewBox="0 0 24 24" width=24 height=24 class=darkToggleIcon_wfgR><path fill=currentColor d="M9.37,5.51C9.19,6.15,9.1,6.82,9.1,7.5c0,4.08,3.32,7.4,7.4,7.4c0.68,0,1.35-0.09,1.99-0.27C17.45,17.19,14.93,19,12,19 c-3.86,0-7-3.14-7-7C5,9.07,6.81,6.55,9.37,5.51z M12,3c-4.97,0-9,4.03-9,9s4.03,9,9,9s9-4.03,9-9c0-0.46-0.04-0.92-0.1-1.36 c-0.98,1.37-2.58,2.26-4.4,2.26c-2.98,0-5.4-2.42-5.4-5.4c0-1.81,0.89-3.42,2.26-4.4C12.92,3.04,12.46,3,12,3L12,3z"/></svg></button></div><div class=navbarSearchContainer_Bca1><div class=navbar__search><span aria-label="expand searchbar" role=button class=search-icon tabindex=0></span><input id=search_input_react type=search placeholder=Loading... aria-label=Search class="navbar__search-input search-bar" disabled></div></div></div></div><div role=presentation class=navbar-sidebar__backdrop></div></nav><div id=__docusaurus_skipToContent_fallback class="main-wrapper mainWrapper_z2l0"><div class="container margin-vert--lg"><div class=row><aside class="col col--3"><nav class="sidebar_re4s thin-scrollbar" aria-label="Blog recent posts navigation"><div class="sidebarItemTitle_pO2u margin-bottom--md">All posts</div><div role=group><h3>2022</h3><ul class="sidebarItemList_Yudw clean-list"><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/2011/11/webpack-industrial-complex>The webpack industrial complex</a></ul></div><div role=group><h3>2019</h3><ul class="sidebarItemList_Yudw clean-list"><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/2019/12/release-the-gil>Release the GIL</a></ul><ul class="sidebarItemList_Yudw clean-list"><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/2019/09/binary-format-shootout>Binary format shootout</a></ul><ul class="sidebarItemList_Yudw clean-list"><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/2019/06/high-performance-systems>On building high performance systems</a></ul><ul class="sidebarItemList_Yudw clean-list"><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/2019/05/making-bread>Making bread</a></ul><div role=group><h4>Allocations in Rust</h4><ul><ul class="sidebarItemList_Yudw clean-list"><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/2019/02/understanding-allocations-in-rust>Foreword</a><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/2019/02/the-whole-world>Global memory</a><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/2019/02/stacking-up>Fixed memory</a><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/2019/02/a-heaping-helping>Dynamic memory</a><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/2019/02/08/compiler-optimizations>Compiler optimizations</a><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/2019/02/summary>Summary</a></ul></ul></div></div><div role=group><h3>2018</h3><ul class="sidebarItemList_Yudw clean-list"><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/2018/12/allocation-safety>QADAPT - debug_assert! for allocations</a></ul><ul class="sidebarItemList_Yudw clean-list"><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/2018/12/what-small-business-really-means>More "what companies really mean"</a></ul><ul class="sidebarItemList_Yudw clean-list"><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/2018/10/case-study-optimization>A case study in heaptrack</a></ul><ul class="sidebarItemList_Yudw clean-list"><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/2018/09/isomorphic-apps>Isomorphic desktop apps with Rust</a></ul><ul class="sidebarItemList_Yudw clean-list"><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/2018/09/primitives-in-rust-are-weird>Primitives in Rust are weird (and cool)</a></ul><ul class="sidebarItemList_Yudw clean-list"><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/2018/06/dateutil-parser-to-rust>What I learned porting dateutil to Rust</a></ul><ul class="sidebarItemList_Yudw clean-list"><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/2018/05/hello>Hello!</a></ul><div role=group><h4>Captain's Cookbook</h4><ul><ul class="sidebarItemList_Yudw clean-list"><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/2018/01/captains-cookbook-part-1>Project setup</a><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/2018/01/captains-cookbook-part-2>Practical usage</a></ul></ul></div></div><div role=group><h3>2016</h3><ul class="sidebarItemList_Yudw clean-list"><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/2016/11/pca-audio-compression>PCA audio compression</a></ul><ul class="sidebarItemList_Yudw clean-list"><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/2016/10/rustic-repodcasting>A Rustic re-podcasting server</a></ul><ul class="sidebarItemList_Yudw clean-list"><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/2016/06/event-studies-and-earnings-releases>Event studies and earnings releases</a></ul><ul class="sidebarItemList_Yudw clean-list"><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/2016/05/the-unfair-casino>The unfair casino</a></ul><ul class="sidebarItemList_Yudw clean-list"><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/2016/04/tick-tock>Tick tock...</a></ul><ul class="sidebarItemList_Yudw clean-list"><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/2016/03/tweet-like-me>Tweet like me</a></ul><ul class="sidebarItemList_Yudw clean-list"><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/2016/03/predicting-santander-customer-happiness>Predicting Santander customer happiness</a></ul><ul class="sidebarItemList_Yudw clean-list"><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/2016/02/profitability-using-the-investment-formula>Profitability using the investment formula</a></ul><ul class="sidebarItemList_Yudw clean-list"><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/2016/02/guaranteed-money-maker>Guaranteed money maker</a></ul><ul class="sidebarItemList_Yudw clean-list"><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/2016/01/cloudy-in-seattle>Cloudy in Seattle</a></ul><ul class="sidebarItemList_Yudw clean-list"><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/2016/01/complaining-about-the-weather>Complaining about the weather</a></ul></div><div role=group><h3>2015</h3><ul class="sidebarItemList_Yudw clean-list"><li class=sidebarItem__DBe><a aria-current=page class="sidebarItemLink_mo7H sidebarItemLinkActive_I1ZP" href=/2015/12/testing-cramer>Testing Cramer</a></ul><ul class="sidebarItemList_Yudw clean-list"><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/2015/11/autocallable>Autocallable Bonds</a></ul><ul class="sidebarItemList_Yudw clean-list"><li class=sidebarItem__DBe><a class=sidebarItemLink_mo7H href=/2015/11/welcome>Welcome, and an algorithm</a></ul></div></nav></aside><main class="col col--7"><article><header><h1 class=title_f1Hy>Testing Cramer</h1><div class="container_mt6G margin-vert--md"><time datetime=2015-12-26T12:00:00.000Z>December 26, 2015</time> · <!-- -->9 min read</div><div class="margin-top--md margin-bottom--sm row"><div class="col col--12 authorCol_Hf19"><div class="avatar margin-bottom--sm"><div class="avatar__intro authorDetails_lV9A"><div class=avatar__name><span class=authorName_yefp>Bradlee Speice</span></div><div class=authorSocials_rSDt><a href=https://github.com/bspeice target=_blank rel="noopener noreferrer" class=authorSocialLink_owbf title=GitHub><svg viewBox="0 0 256 250" width=1em height=1em class="authorSocialLink_owbf githubSvg_Uu4N" style=--dark:#000;--light:#fff preserveAspectRatio=xMidYMid><path d="M128.001 0C57.317 0 0 57.307 0 128.001c0 56.554 36.676 104.535 87.535 121.46 6.397 1.185 8.746-2.777 8.746-6.158 0-3.052-.12-13.135-.174-23.83-35.61 7.742-43.124-15.103-43.124-15.103-5.823-14.795-14.213-18.73-14.213-18.73-11.613-7.944.876-7.78.876-7.78 12.853.902 19.621 13.19 19.621 13.19 11.417 19.568 29.945 13.911 37.249 10.64 1.149-8.272 4.466-13.92 8.127-17.116-28.431-3.236-58.318-14.212-58.318-63.258 0-13.975 5-25.394 13.188-34.358-1.329-3.224-5.71-16.242 1.24-33.874 0 0 10.749-3.44 35.21 13.121 10.21-2.836 21.16-4.258 32.038-4.307 10.878.049 21.837 1.47 32.066 4.307 24.431-16.56 35.165-13.12 35.165-13.12 6.967 17.63 2.584 30.65 1.255 33.873 8.207 8.964 13.173 20.383 13.173 34.358 0 49.163-29.944 59.988-58.447 63.157 4.591 3.972 8.682 11.762 8.682 23.704 0 17.126-.148 30.91-.148 35.126 0 3.407 2.304 7.398 8.792 6.14C219.37 232.5 256 184.537 256 128.002 256 57.307 198.691 0 128.001 0Zm-80.06 182.34c-.282.636-1.283.827-2.194.39-.929-.417-1.45-1.284-1.15-1.922.276-.655 1.279-.838 2.205-.399.93.418 1.46 1.293 1.139 1.931Zm6.296 5.618c-.61.566-1.804.303-2.614-.591-.837-.892-.994-2.086-.375-2.66.63-.566 1.787-.301 2.626.591.838.903 1 2.088.363 2.66Zm4.32 7.188c-.785.545-2.067.034-2.86-1.104-.784-1.138-.784-2.503.017-3.05.795-.547 2.058-.055 2.861 1.075.782 1.157.782 2.522-.019 3.08Zm7.304 8.325c-.701.774-2.196.566-3.29-.49-1.119-1.032-1.43-2.496-.726-3.27.71-.776 2.213-.558 3.315.49 1.11 1.03 1.45 2.505.701 3.27Zm9.442 2.81c-.31 1.003-1.75 1.459-3.199 1.033-1.448-.439-2.395-1.613-2.103-2.626.301-1.01 1.747-1.484 3.207-1.028 1.446.436 2.396 1.602 2.095 2.622Zm10.744 1.193c.036 1.055-1.193 1.93-2.715 1.95-1.53.034-2.769-.82-2.786-1.86 0-1.065 1.202-1.932 2.733-1.958 1.522-.03 2.768.818 2.768 1.868Zm10.555-.405c.182 1.03-.875 2.088-2.387 2.37-1.485.271-2.861-.365-3.05-1.386-.184-1.056.893-2.114 2.376-2.387 1.514-.263 2.868.356 3.061 1.403Z"/></svg></a></div></div></div></div></div></header><div id=__blog-post-container class=markdown><p>Pursuant to attending a graduate school studying Financial Engineering, I've been a fan of the <a href=http://www.cnbc.com/mad-money/ target=_blank rel="noopener noreferrer">Mad Money</a> TV show featuring the bombastic Jim Cramer. One of the things that he's said is that you shouldn't use the futures to predict where the stock market is going to go. But he says it often enough, I've begun to wonder - who is he trying to convince?</p>
<p>It makes sense that because futures on things like the S&P 500 are traded continuously, they would price in market information before the stock market opens. So is Cramer right to be convinced that strategies based on the futures are a poor idea? I wanted to test it out.</p>
<p>The first question is where to get the future's data. I've been part of <a href=http://seekingalpha.com/ target=_blank rel="noopener noreferrer">Seeking Alpha</a> for a bit, and they publish the <a href="http://seekingalpha.com/author/wall-street-breakfast?s=wall-street-breakfast" target=_blank rel="noopener noreferrer">Wall Street Breakfast</a> newsletter which contains daily future's returns as of 6:20 AM EST. I'd be interested in using that data to see if we can actually make some money.</p>
<p>First though, let's get the data:</p>
<h2 class="anchor anchorWithStickyNavbar_LWe7" id=downloading-futures-data-from-seeking-alpha>Downloading Futures data from Seeking Alpha<a href=#downloading-futures-data-from-seeking-alpha class=hash-link aria-label="Direct link to Downloading Futures data from Seeking Alpha" title="Direct link to Downloading Futures data from Seeking Alpha"></a></h2>
<p>We're going to define two HTML parsing classes - one to get the article URL's from a page, and one to get the actual data from each article.</p>
<div class="language-python codeBlockContainer_Ckt0 theme-code-block" style="--prism-background-color:hsl(230, 1%, 98%);--prism-color:hsl(230, 8%, 24%)"><div class=codeBlockContent_biex><pre tabindex=0 class="prism-code language-python codeBlock_bY9V thin-scrollbar" style="background-color:hsl(230, 1%, 98%);color:hsl(230, 8%, 24%)"><code class=codeBlockLines_e6Vv><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token keyword" style="color:hsl(301, 63%, 40%)">class</span><span class="token plain"> </span><span class="token class-name" style="color:hsl(35, 99%, 36%)">ArticleListParser</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">HTMLParser</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    </span><span class="token triple-quoted-string string" style="color:hsl(119, 34%, 47%)">"""Given a web page with articles on it, parse out the article links"""</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    </span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    articles </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    </span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">def</span><span class="token plain"> </span><span class="token function" style="color:hsl(221, 87%, 60%)">handle_starttag</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">self</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> tag</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> attrs</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">        </span><span class="token comment" style="color:hsl(230, 4%, 64%)">#if tag == 'div' and ("id", "author_articles_wrapper") in attrs:</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">        </span><span class="token comment" style="color:hsl(230, 4%, 64%)">#    self.fetch_links = True</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">        </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">if</span><span class="token plain"> tag </span><span class="token operator" style="color:hsl(221, 87%, 60%)">==</span><span class="token plain"> </span><span class="token string" style="color:hsl(119, 34%, 47%)">'a'</span><span class="token plain"> </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">and</span><span class="token plain"> </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token string" style="color:hsl(119, 34%, 47%)">'class'</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> </span><span class="token string" style="color:hsl(119, 34%, 47%)">'dashboard_article_link'</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"> </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">in</span><span class="token plain"> attrs</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">            href </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> </span><span class="token builtin" style="color:hsl(119, 34%, 47%)">list</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token builtin" style="color:hsl(119, 34%, 47%)">filter</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token keyword" style="color:hsl(301, 63%, 40%)">lambda</span><span class="token plain"> x</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"> x</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token number" style="color:hsl(35, 99%, 36%)">0</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"> </span><span class="token operator" style="color:hsl(221, 87%, 60%)">==</span><span class="token plain"> </span><span class="token string" style="color:hsl(119, 34%, 47%)">'href'</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> attrs</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token number" style="color:hsl(35, 99%, 36%)">0</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token number" style="color:hsl(35, 99%, 36%)">1</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">            self</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">articles</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">append</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">href</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">            </span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">base_url </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> </span><span class="token string" style="color:hsl(119, 34%, 47%)">"http://seekingalpha.com/author/wall-street-breakfast/articles"</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">article_page_urls </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token plain">base_url</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"> </span><span class="token operator" style="color:hsl(221, 87%, 60%)">+</span><span class="token plain"> </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token plain">base_url </span><span class="token operator" style="color:hsl(221, 87%, 60%)">+</span><span class="token plain"> </span><span class="token string" style="color:hsl(119, 34%, 47%)">'/{}'</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token builtin" style="color:hsl(119, 34%, 47%)">format</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">i</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"> </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">for</span><span class="token plain"> i </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">in</span><span class="token plain"> </span><span class="token builtin" style="color:hsl(119, 34%, 47%)">range</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token number" style="color:hsl(35, 99%, 36%)">2</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> </span><span class="token number" style="color:hsl(35, 99%, 36%)">20</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain" style=display:inline-block></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">global_articles </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain"></span><span class="token keyword" style="color:hsl(301, 63%, 40%)">for</span><span class="token plain"> page </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">in</span><span class="token plain"> article_page_urls</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    </span><span class="token comment" style="color:hsl(230, 4%, 64%)"># We need to switch the user agent, as SA blocks the standard requests agent</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    articles_html </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> requests</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">get</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">page</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">                                headers</span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">{</span><span class="token string" style="color:hsl(119, 34%, 47%)">"User-Agent"</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"> </span><span class="token string" style="color:hsl(119, 34%, 47%)">"Wget/1.13.4"</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">}</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    parser </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> ArticleListParser</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    parser</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">feed</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">articles_html</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">text</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    global_articles </span><span class="token operator" style="color:hsl(221, 87%, 60%)">+=</span><span class="token plain"> </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">parser</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">articles</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><br></span></code></pre><div class=buttonGroup__atx><button type=button aria-label="Copy code to clipboard" title=Copy class=clean-btn><span class=copyButtonIcons_eSgA aria-hidden=true><svg viewBox="0 0 24 24" class=copyButtonIcon_y97N><path fill=currentColor d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"/></svg><svg viewBox="0 0 24 24" class=copyButtonSuccessIcon_LjdS><path fill=currentColor d=M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z /></svg></span></button></div></div></div>
<div class="language-python codeBlockContainer_Ckt0 theme-code-block" style="--prism-background-color:hsl(230, 1%, 98%);--prism-color:hsl(230, 8%, 24%)"><div class=codeBlockContent_biex><pre tabindex=0 class="prism-code language-python codeBlock_bY9V thin-scrollbar" style="background-color:hsl(230, 1%, 98%);color:hsl(230, 8%, 24%)"><code class=codeBlockLines_e6Vv><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token keyword" style="color:hsl(301, 63%, 40%)">class</span><span class="token plain"> </span><span class="token class-name" style="color:hsl(35, 99%, 36%)">ArticleReturnParser</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">HTMLParser</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    </span><span class="token string" style="color:hsl(119, 34%, 47%)">"Given an article, parse out the futures returns in it"</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    </span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    record_font_tags </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> </span><span class="token boolean" style="color:hsl(35, 99%, 36%)">False</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    in_font_tag </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> </span><span class="token boolean" style="color:hsl(35, 99%, 36%)">False</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    counter </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> </span><span class="token number" style="color:hsl(35, 99%, 36%)">0</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    </span><span class="token comment" style="color:hsl(230, 4%, 64%)"># data = {} # See __init__</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    </span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">def</span><span class="token plain"> </span><span class="token function" style="color:hsl(221, 87%, 60%)">__init__</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">self</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> </span><span class="token operator" style="color:hsl(221, 87%, 60%)">*</span><span class="token plain">args</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> </span><span class="token operator" style="color:hsl(221, 87%, 60%)">**</span><span class="token plain">kwargs</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">        </span><span class="token builtin" style="color:hsl(119, 34%, 47%)">super</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">__init__</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token operator" style="color:hsl(221, 87%, 60%)">*</span><span class="token plain">args</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> </span><span class="token operator" style="color:hsl(221, 87%, 60%)">**</span><span class="token plain">kwargs</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">        self</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">data </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">{</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">}</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    </span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">def</span><span class="token plain"> </span><span class="token function" style="color:hsl(221, 87%, 60%)">handle_starttag</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">self</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> tag</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> attrs</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">        </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">if</span><span class="token plain"> tag </span><span class="token operator" style="color:hsl(221, 87%, 60%)">==</span><span class="token plain"> </span><span class="token string" style="color:hsl(119, 34%, 47%)">'span'</span><span class="token plain"> </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">and</span><span class="token plain"> </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token string" style="color:hsl(119, 34%, 47%)">'itemprop'</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> </span><span class="token string" style="color:hsl(119, 34%, 47%)">'datePublished'</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"> </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">in</span><span class="token plain"> attrs</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">            date_string </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> </span><span class="token builtin" style="color:hsl(119, 34%, 47%)">list</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token builtin" style="color:hsl(119, 34%, 47%)">filter</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token keyword" style="color:hsl(301, 63%, 40%)">lambda</span><span class="token plain"> x</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"> x</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token number" style="color:hsl(35, 99%, 36%)">0</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"> </span><span class="token operator" style="color:hsl(221, 87%, 60%)">==</span><span class="token plain"> </span><span class="token string" style="color:hsl(119, 34%, 47%)">'content'</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> attrs</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token number" style="color:hsl(35, 99%, 36%)">0</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token number" style="color:hsl(35, 99%, 36%)">1</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">            date </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> dtparser</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">parse</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">date_string</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">            self</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">data</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token string" style="color:hsl(119, 34%, 47%)">'date'</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"> </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> date</span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">            </span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">        self</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">in_font_tag </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> tag </span><span class="token operator" style="color:hsl(221, 87%, 60%)">==</span><span class="token plain"> </span><span class="token string" style="color:hsl(119, 34%, 47%)">'font'</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">        </span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">def</span><span class="token plain"> </span><span class="token function" style="color:hsl(221, 87%, 60%)">safe_float</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">self</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> string</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">        </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">try</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">            </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">return</span><span class="token plain"> </span><span class="token builtin" style="color:hsl(119, 34%, 47%)">float</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">string</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token operator" style="color:hsl(221, 87%, 60%)">-</span><span class="token number" style="color:hsl(35, 99%, 36%)">1</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"> </span><span class="token operator" style="color:hsl(221, 87%, 60%)">/</span><span class="token plain"> </span><span class="token number" style="color:hsl(35, 99%, 36%)">100</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">        </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">except</span><span class="token plain"> ValueError</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">            </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">return</span><span class="token plain"> np</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">NaN</span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">            </span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">def</span><span class="token plain"> </span><span class="token function" style="color:hsl(221, 87%, 60%)">handle_data</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">self</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> content</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">        </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">if</span><span class="token plain"> </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">not</span><span class="token plain"> self</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">record_font_tags </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">and</span><span class="token plain"> </span><span class="token string" style="color:hsl(119, 34%, 47%)">"Futures at 6"</span><span class="token plain"> </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">in</span><span class="token plain"> content</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">            self</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">record_font_tags </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> </span><span class="token boolean" style="color:hsl(35, 99%, 36%)">True</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">            </span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">        </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">if</span><span class="token plain"> self</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">record_font_tags </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">and</span><span class="token plain"> self</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">in_font_tag</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">            </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">if</span><span class="token plain"> self</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">counter </span><span class="token operator" style="color:hsl(221, 87%, 60%)">==</span><span class="token plain"> </span><span class="token number" style="color:hsl(35, 99%, 36%)">0</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">                self</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">data</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token string" style="color:hsl(119, 34%, 47%)">'DOW'</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"> </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> self</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">safe_float</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">content</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">            </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">elif</span><span class="token plain"> self</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">counter </span><span class="token operator" style="color:hsl(221, 87%, 60%)">==</span><span class="token plain"> </span><span class="token number" style="color:hsl(35, 99%, 36%)">1</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">                self</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">data</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token string" style="color:hsl(119, 34%, 47%)">'S&P'</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"> </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> self</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">safe_float</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">content</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">            </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">elif</span><span class="token plain"> self</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">counter </span><span class="token operator" style="color:hsl(221, 87%, 60%)">==</span><span class="token plain"> </span><span class="token number" style="color:hsl(35, 99%, 36%)">2</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">                self</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">data</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token string" style="color:hsl(119, 34%, 47%)">'NASDAQ'</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"> </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> self</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">safe_float</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">content</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">            </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">elif</span><span class="token plain"> self</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">counter </span><span class="token operator" style="color:hsl(221, 87%, 60%)">==</span><span class="token plain"> </span><span class="token number" style="color:hsl(35, 99%, 36%)">3</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">                self</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">data</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token string" style="color:hsl(119, 34%, 47%)">'Crude'</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"> </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> self</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">safe_float</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">content</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">            </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">elif</span><span class="token plain"> self</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">counter </span><span class="token operator" style="color:hsl(221, 87%, 60%)">==</span><span class="token plain"> </span><span class="token number" style="color:hsl(35, 99%, 36%)">4</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">                self</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">data</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token string" style="color:hsl(119, 34%, 47%)">'Gold'</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"> </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> self</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">safe_float</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">content</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">            </span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">            self</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">counter </span><span class="token operator" style="color:hsl(221, 87%, 60%)">+=</span><span class="token plain"> </span><span class="token number" style="color:hsl(35, 99%, 36%)">1</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">            </span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">def</span><span class="token plain"> </span><span class="token function" style="color:hsl(221, 87%, 60%)">handle_endtag</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">self</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> tag</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">        self</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">in_font_tag </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> </span><span class="token boolean" style="color:hsl(35, 99%, 36%)">False</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain" style=display:inline-block></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain"></span><span class="token keyword" style="color:hsl(301, 63%, 40%)">def</span><span class="token plain"> </span><span class="token function" style="color:hsl(221, 87%, 60%)">retrieve_data</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">url</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    sa </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> </span><span class="token string" style="color:hsl(119, 34%, 47%)">"http://seekingalpha.com"</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    article_html </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> requests</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">get</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">sa </span><span class="token operator" style="color:hsl(221, 87%, 60%)">+</span><span class="token plain"> url</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">                               headers</span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">{</span><span class="token string" style="color:hsl(119, 34%, 47%)">"User-Agent"</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"> </span><span class="token string" style="color:hsl(119, 34%, 47%)">"Wget/1.13.4"</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">}</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    parser </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> ArticleReturnParser</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    parser</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">feed</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">article_html</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">text</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    parser</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">data</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">update</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">{</span><span class="token string" style="color:hsl(119, 34%, 47%)">"url"</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"> url</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">}</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    parser</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">data</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">update</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">{</span><span class="token string" style="color:hsl(119, 34%, 47%)">"text"</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"> article_html</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">text</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">}</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">return</span><span class="token plain"> parser</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">data</span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain" style=display:inline-block></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain"></span><span class="token comment" style="color:hsl(230, 4%, 64%)"># This copy **MUST** be in place. I'm not sure why,</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain"></span><span class="token comment" style="color:hsl(230, 4%, 64%)"># as you'd think that the data being returned would already</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain"></span><span class="token comment" style="color:hsl(230, 4%, 64%)"># represent a different memory location. Even so, it blows up</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain"></span><span class="token comment" style="color:hsl(230, 4%, 64%)"># if you don't do this.</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">article_list </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> </span><span class="token builtin" style="color:hsl(119, 34%, 47%)">list</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token builtin" style="color:hsl(119, 34%, 47%)">set</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">global_articles</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">article_data </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token plain">copy</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">retrieve_data</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">url</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"> </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">for</span><span class="token plain"> url </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">in</span><span class="token plain"> article_list</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain"></span><span class="token comment" style="color:hsl(230, 4%, 64%)"># If there's an issue downloading the article, drop it.</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">article_df </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> pd</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">DataFrame</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">from_dict</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">article_data</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">dropna</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><br></span></code></pre><div class=buttonGroup__atx><button type=button aria-label="Copy code to clipboard" title=Copy class=clean-btn><span class=copyButtonIcons_eSgA aria-hidden=true><svg viewBox="0 0 24 24" class=copyButtonIcon_y97N><path fill=currentColor d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"/></svg><svg viewBox="0 0 24 24" class=copyButtonSuccessIcon_LjdS><path fill=currentColor d=M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z /></svg></span></button></div></div></div>
<h2 class="anchor anchorWithStickyNavbar_LWe7" id=fetching-the-returns-data>Fetching the Returns data<a href=#fetching-the-returns-data class=hash-link aria-label="Direct link to Fetching the Returns data" title="Direct link to Fetching the Returns data"></a></h2>
<p>Now that we have the futures data, we're going to compare across 4 different indices - the S&P 500 index, Dow Jones Industrial, Russell 2000, and NASDAQ 100. Let's get the data off of Quandl to make things easier!</p>
<div class="language-python codeBlockContainer_Ckt0 theme-code-block" style="--prism-background-color:hsl(230, 1%, 98%);--prism-color:hsl(230, 8%, 24%)"><div class=codeBlockContent_biex><pre tabindex=0 class="prism-code language-python codeBlock_bY9V thin-scrollbar" style="background-color:hsl(230, 1%, 98%);color:hsl(230, 8%, 24%)"><code class=codeBlockLines_e6Vv><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token comment" style="color:hsl(230, 4%, 64%)"># article_df is sorted by date, so we get the first row.</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">start_date </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> article_df</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">sort_values</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">by</span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token string" style="color:hsl(119, 34%, 47%)">'date'</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">iloc</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token number" style="color:hsl(35, 99%, 36%)">0</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token string" style="color:hsl(119, 34%, 47%)">'date'</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"> </span><span class="token operator" style="color:hsl(221, 87%, 60%)">-</span><span class="token plain"> relativedelta</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">days</span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token number" style="color:hsl(35, 99%, 36%)">1</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">SPY </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> Quandl</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">get</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token string" style="color:hsl(119, 34%, 47%)">"GOOG/NYSE_SPY"</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> trim_start</span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain">start_date</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">DJIA </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> Quandl</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">get</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token string" style="color:hsl(119, 34%, 47%)">"GOOG/AMS_DIA"</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> trim_start</span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain">start_date</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">RUSS </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> Quandl</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">get</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token string" style="color:hsl(119, 34%, 47%)">"GOOG/AMEX_IWM"</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> trim_start</span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain">start_date</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">NASDAQ </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> Quandl</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">get</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token string" style="color:hsl(119, 34%, 47%)">"GOOG/EPA_QQQ"</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> trim_start</span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain">start_date</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><br></span></code></pre><div class=buttonGroup__atx><button type=button aria-label="Copy code to clipboard" title=Copy class=clean-btn><span class=copyButtonIcons_eSgA aria-hidden=true><svg viewBox="0 0 24 24" class=copyButtonIcon_y97N><path fill=currentColor d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"/></svg><svg viewBox="0 0 24 24" class=copyButtonSuccessIcon_LjdS><path fill=currentColor d=M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z /></svg></span></button></div></div></div>
<h2 class="anchor anchorWithStickyNavbar_LWe7" id=running-the-comparison>Running the Comparison<a href=#running-the-comparison class=hash-link aria-label="Direct link to Running the Comparison" title="Direct link to Running the Comparison"></a></h2>
<p>There are two types of tests I want to determine: How accurate each futures category is at predicting the index's opening change over the close before, and predicting the index's daily return.</p>
<p>Let's first calculate how good each future is at predicting the opening return over the previous day. I expect that the futures will be more than 50% accurate, since the information is recorded 3 hours before the markets open.</p>
<div class="language-python codeBlockContainer_Ckt0 theme-code-block" style="--prism-background-color:hsl(230, 1%, 98%);--prism-color:hsl(230, 8%, 24%)"><div class=codeBlockContent_biex><pre tabindex=0 class="prism-code language-python codeBlock_bY9V thin-scrollbar" style="background-color:hsl(230, 1%, 98%);color:hsl(230, 8%, 24%)"><code class=codeBlockLines_e6Vv><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token keyword" style="color:hsl(301, 63%, 40%)">def</span><span class="token plain"> </span><span class="token function" style="color:hsl(221, 87%, 60%)">calculate_opening_ret</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">frame</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    </span><span class="token comment" style="color:hsl(230, 4%, 64%)"># I'm not a huge fan of the appending for loop,</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    </span><span class="token comment" style="color:hsl(230, 4%, 64%)"># but it's a bit verbose for a comprehension</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    data </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">{</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">}</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">for</span><span class="token plain"> i </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">in</span><span class="token plain"> </span><span class="token builtin" style="color:hsl(119, 34%, 47%)">range</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token number" style="color:hsl(35, 99%, 36%)">1</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> </span><span class="token builtin" style="color:hsl(119, 34%, 47%)">len</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">frame</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">        date </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> frame</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">iloc</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token plain">i</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">name</span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">        prior_close </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> frame</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">iloc</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token plain">i</span><span class="token operator" style="color:hsl(221, 87%, 60%)">-</span><span class="token number" style="color:hsl(35, 99%, 36%)">1</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token string" style="color:hsl(119, 34%, 47%)">'Close'</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">        open_val </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> frame</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">iloc</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token plain">i</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token string" style="color:hsl(119, 34%, 47%)">'Open'</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">        data</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token plain">date</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"> </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">open_val </span><span class="token operator" style="color:hsl(221, 87%, 60%)">-</span><span class="token plain"> prior_close</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"> </span><span class="token operator" style="color:hsl(221, 87%, 60%)">/</span><span class="token plain"> prior_close</span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">        </span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">return</span><span class="token plain"> data</span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain" style=display:inline-block></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">SPY_open_ret </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> calculate_opening_ret</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">SPY</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">DJIA_open_ret </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> calculate_opening_ret</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">DJIA</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">RUSS_open_ret </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> calculate_opening_ret</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">RUSS</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">NASDAQ_open_ret </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> calculate_opening_ret</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">NASDAQ</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain" style=display:inline-block></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain"></span><span class="token keyword" style="color:hsl(301, 63%, 40%)">def</span><span class="token plain"> </span><span class="token function" style="color:hsl(221, 87%, 60%)">signs_match</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">list_1</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> list_2</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    </span><span class="token comment" style="color:hsl(230, 4%, 64%)"># This is a surprisingly difficult task - we have to match</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    </span><span class="token comment" style="color:hsl(230, 4%, 64%)"># up the dates in order to check if opening returns actually match</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    index_dict_dt </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">{</span><span class="token plain">key</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">to_datetime</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"> list_2</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token plain">key</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"> </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">for</span><span class="token plain"> key </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">in</span><span class="token plain"> list_2</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">keys</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">}</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    </span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    matches </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">for</span><span class="token plain"> row </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">in</span><span class="token plain"> list_1</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">iterrows</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">        row_dt </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> row</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token number" style="color:hsl(35, 99%, 36%)">1</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token number" style="color:hsl(35, 99%, 36%)">1</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">        row_value </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> row</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token number" style="color:hsl(35, 99%, 36%)">1</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token number" style="color:hsl(35, 99%, 36%)">0</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">        index_dt </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> datetime</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">row_dt</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">year</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> row_dt</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">month</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> row_dt</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">day</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">        </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">if</span><span class="token plain"> index_dt </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">in</span><span class="token plain"> list_2</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">            index_value </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> list_2</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token plain">index_dt</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">            </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">if</span><span class="token plain"> </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">row_value </span><span class="token operator" style="color:hsl(221, 87%, 60%)">></span><span class="token plain"> </span><span class="token number" style="color:hsl(35, 99%, 36%)">0</span><span class="token plain"> </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">and</span><span class="token plain"> index_value </span><span class="token operator" style="color:hsl(221, 87%, 60%)">></span><span class="token plain"> </span><span class="token number" style="color:hsl(35, 99%, 36%)">0</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"> </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">or</span><span class="token plain"> \</span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">                </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">row_value </span><span class="token operator" style="color:hsl(221, 87%, 60%)">&lt;</span><span class="token plain"> </span><span class="token number" style="color:hsl(35, 99%, 36%)">0</span><span class="token plain"> </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">and</span><span class="token plain"> index_value </span><span class="token operator" style="color:hsl(221, 87%, 60%)">&lt;</span><span class="token plain"> </span><span class="token number" style="color:hsl(35, 99%, 36%)">0</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"> </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">or</span><span class="token plain"> \</span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">                </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">row_value </span><span class="token operator" style="color:hsl(221, 87%, 60%)">==</span><span class="token plain"> </span><span class="token number" style="color:hsl(35, 99%, 36%)">0</span><span class="token plain"> </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">and</span><span class="token plain"> index_value </span><span class="token operator" style="color:hsl(221, 87%, 60%)">==</span><span class="token plain"> </span><span class="token number" style="color:hsl(35, 99%, 36%)">0</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">                    matches </span><span class="token operator" style="color:hsl(221, 87%, 60%)">+=</span><span class="token plain"> </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token number" style="color:hsl(35, 99%, 36%)">1</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">            </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">else</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">                matches </span><span class="token operator" style="color:hsl(221, 87%, 60%)">+=</span><span class="token plain"> </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token number" style="color:hsl(35, 99%, 36%)">0</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">            </span><span class="token comment" style="color:hsl(230, 4%, 64%)">#print("{}".format(list_2[index_dt]))</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">return</span><span class="token plain"> matches</span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    </span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    </span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">prediction_dict </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">{</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">}</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">matches_dict </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">{</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">}</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">count_dict </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">{</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">}</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">index_dict </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">{</span><span class="token string" style="color:hsl(119, 34%, 47%)">"SPY"</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"> SPY_open_ret</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> </span><span class="token string" style="color:hsl(119, 34%, 47%)">"DJIA"</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"> DJIA_open_ret</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> </span><span class="token string" style="color:hsl(119, 34%, 47%)">"RUSS"</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"> RUSS_open_ret</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> </span><span class="token string" style="color:hsl(119, 34%, 47%)">"NASDAQ"</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"> NASDAQ_open_ret</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">}</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">indices </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token string" style="color:hsl(119, 34%, 47%)">"SPY"</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> </span><span class="token string" style="color:hsl(119, 34%, 47%)">"DJIA"</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> </span><span class="token string" style="color:hsl(119, 34%, 47%)">"RUSS"</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> </span><span class="token string" style="color:hsl(119, 34%, 47%)">"NASDAQ"</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">futures </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token string" style="color:hsl(119, 34%, 47%)">"Crude"</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> </span><span class="token string" style="color:hsl(119, 34%, 47%)">"Gold"</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> </span><span class="token string" style="color:hsl(119, 34%, 47%)">"DOW"</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> </span><span class="token string" style="color:hsl(119, 34%, 47%)">"NASDAQ"</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> </span><span class="token string" style="color:hsl(119, 34%, 47%)">"S&P"</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain"></span><span class="token keyword" style="color:hsl(301, 63%, 40%)">for</span><span class="token plain"> index </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">in</span><span class="token plain"> indices</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    matches_dict</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token plain">index</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"> </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">{</span><span class="token plain">future</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"> signs_match</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">article_df</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token plain">future</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> </span><span class="token string" style="color:hsl(119, 34%, 47%)">'date'</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">                                               index_dict</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token plain">index</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"> </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">for</span><span class="token plain"> future </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">in</span><span class="token plain"> futures</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">}</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    count_dict</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token plain">index</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"> </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">{</span><span class="token plain">future</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"> </span><span class="token builtin" style="color:hsl(119, 34%, 47%)">len</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">matches_dict</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token plain">index</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token plain">future</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"> </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">for</span><span class="token plain"> future </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">in</span><span class="token plain"> futures</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">}</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    prediction_dict</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token plain">index</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"> </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">{</span><span class="token plain">future</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"> np</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">mean</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">matches_dict</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token plain">index</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token plain">future</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">                              </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">for</span><span class="token plain"> future </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">in</span><span class="token plain"> futures</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">}</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain"></span><span class="token keyword" style="color:hsl(301, 63%, 40%)">print</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token string" style="color:hsl(119, 34%, 47%)">"Articles Checked: "</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain"></span><span class="token keyword" style="color:hsl(301, 63%, 40%)">print</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">pd</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">DataFrame</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">from_dict</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">count_dict</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain"></span><span class="token keyword" style="color:hsl(301, 63%, 40%)">print</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain"></span><span class="token keyword" style="color:hsl(301, 63%, 40%)">print</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token string" style="color:hsl(119, 34%, 47%)">"Prediction Accuracy:"</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain"></span><span class="token keyword" style="color:hsl(301, 63%, 40%)">print</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">pd</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">DataFrame</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">from_dict</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">prediction_dict</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><br></span></code></pre><div class=buttonGroup__atx><button type=button aria-label="Copy code to clipboard" title=Copy class=clean-btn><span class=copyButtonIcons_eSgA aria-hidden=true><svg viewBox="0 0 24 24" class=copyButtonIcon_y97N><path fill=currentColor d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"/></svg><svg viewBox="0 0 24 24" class=copyButtonSuccessIcon_LjdS><path fill=currentColor d=M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z /></svg></span></button></div></div></div>
<div class="codeBlockContainer_Ckt0 theme-code-block" style="--prism-background-color:hsl(230, 1%, 98%);--prism-color:hsl(230, 8%, 24%)"><div class=codeBlockContent_biex><pre tabindex=0 class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="background-color:hsl(230, 1%, 98%);color:hsl(230, 8%, 24%)"><code class=codeBlockLines_e6Vv><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    Articles Checked: </span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">            DJIA  NASDAQ  RUSS  SPY</span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    Crude    268     268   271  271</span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    DOW      268     268   271  271</span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    Gold     268     268   271  271</span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    NASDAQ   268     268   271  271</span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    S&P      268     268   271  271</span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    </span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    Prediction Accuracy:</span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">                DJIA    NASDAQ      RUSS       SPY</span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    Crude   0.544776  0.522388  0.601476  0.590406</span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    DOW     0.611940  0.604478  0.804428  0.841328</span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    Gold    0.462687  0.455224  0.464945  0.476015</span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    NASDAQ  0.615672  0.608209  0.797048  0.830258</span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    S&P     0.604478  0.597015  0.811808  0.848708</span><br></span></code></pre><div class=buttonGroup__atx><button type=button aria-label="Copy code to clipboard" title=Copy class=clean-btn><span class=copyButtonIcons_eSgA aria-hidden=true><svg viewBox="0 0 24 24" class=copyButtonIcon_y97N><path fill=currentColor d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"/></svg><svg viewBox="0 0 24 24" class=copyButtonSuccessIcon_LjdS><path fill=currentColor d=M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z /></svg></span></button></div></div></div>
<p>This data is very interesting. Some insights:</p>
<ul>
<li>Both DOW and NASDAQ futures are pretty bad at predicting their actual market openings</li>
<li>NASDAQ and Dow are fairly unpredictable; Russell 2000 and S&P are very predictable</li>
<li>Gold is a poor predictor in general - intuitively Gold should move inverse to the market, but it appears to be about as accurate as a coin flip.</li>
</ul>
<p>All said though it appears that futures data is important for determining market direction for both the S&P 500 and Russell 2000. Cramer is half-right: futures data isn't very helpful for the Dow and NASDAQ indices, but is great for the S&P and Russell indices.</p>
<h2 class="anchor anchorWithStickyNavbar_LWe7" id=the-next-step---predicting-the-close>The next step - Predicting the close<a href=#the-next-step---predicting-the-close class=hash-link aria-label="Direct link to The next step - Predicting the close" title="Direct link to The next step - Predicting the close"></a></h2>
<p>Given the code we currently have, I'd like to predict the close of the market as well. We can re-use most of the code, so let's see what happens:</p>
<div class="language-python codeBlockContainer_Ckt0 theme-code-block" style="--prism-background-color:hsl(230, 1%, 98%);--prism-color:hsl(230, 8%, 24%)"><div class=codeBlockContent_biex><pre tabindex=0 class="prism-code language-python codeBlock_bY9V thin-scrollbar" style="background-color:hsl(230, 1%, 98%);color:hsl(230, 8%, 24%)"><code class=codeBlockLines_e6Vv><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token keyword" style="color:hsl(301, 63%, 40%)">def</span><span class="token plain"> </span><span class="token function" style="color:hsl(221, 87%, 60%)">calculate_closing_ret</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">frame</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    </span><span class="token comment" style="color:hsl(230, 4%, 64%)"># I'm not a huge fan of the appending for loop,</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    </span><span class="token comment" style="color:hsl(230, 4%, 64%)"># but it's a bit verbose for a comprehension</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    data </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">{</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">}</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">for</span><span class="token plain"> i </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">in</span><span class="token plain"> </span><span class="token builtin" style="color:hsl(119, 34%, 47%)">range</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token number" style="color:hsl(35, 99%, 36%)">0</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> </span><span class="token builtin" style="color:hsl(119, 34%, 47%)">len</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">frame</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">        date </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> frame</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">iloc</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token plain">i</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">name</span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">        open_val </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> frame</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">iloc</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token plain">i</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token string" style="color:hsl(119, 34%, 47%)">'Open'</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">        close_val </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> frame</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">iloc</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token plain">i</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token string" style="color:hsl(119, 34%, 47%)">'Close'</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">        data</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token plain">date</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"> </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">close_val </span><span class="token operator" style="color:hsl(221, 87%, 60%)">-</span><span class="token plain"> open_val</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"> </span><span class="token operator" style="color:hsl(221, 87%, 60%)">/</span><span class="token plain"> open_val</span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">        </span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">return</span><span class="token plain"> data</span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain" style=display:inline-block></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">SPY_close_ret </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> calculate_closing_ret</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">SPY</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">DJIA_close_ret </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> calculate_closing_ret</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">DJIA</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">RUSS_close_ret </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> calculate_closing_ret</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">RUSS</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">NASDAQ_close_ret </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> calculate_closing_ret</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">NASDAQ</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain" style=display:inline-block></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain"></span><span class="token keyword" style="color:hsl(301, 63%, 40%)">def</span><span class="token plain"> </span><span class="token function" style="color:hsl(221, 87%, 60%)">signs_match</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">list_1</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> list_2</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    </span><span class="token comment" style="color:hsl(230, 4%, 64%)"># This is a surprisingly difficult task - we have to match</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    </span><span class="token comment" style="color:hsl(230, 4%, 64%)"># up the dates in order to check if opening returns actually match</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    index_dict_dt </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">{</span><span class="token plain">key</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">to_datetime</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"> list_2</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token plain">key</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"> </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">for</span><span class="token plain"> key </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">in</span><span class="token plain"> list_2</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">keys</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">}</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    </span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    matches </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">for</span><span class="token plain"> row </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">in</span><span class="token plain"> list_1</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">iterrows</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">        row_dt </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> row</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token number" style="color:hsl(35, 99%, 36%)">1</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token number" style="color:hsl(35, 99%, 36%)">1</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">        row_value </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> row</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token number" style="color:hsl(35, 99%, 36%)">1</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token number" style="color:hsl(35, 99%, 36%)">0</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">        index_dt </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> datetime</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">row_dt</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">year</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> row_dt</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">month</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> row_dt</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">day</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">        </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">if</span><span class="token plain"> index_dt </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">in</span><span class="token plain"> list_2</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">            index_value </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> list_2</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token plain">index_dt</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">            </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">if</span><span class="token plain"> </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">row_value </span><span class="token operator" style="color:hsl(221, 87%, 60%)">></span><span class="token plain"> </span><span class="token number" style="color:hsl(35, 99%, 36%)">0</span><span class="token plain"> </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">and</span><span class="token plain"> index_value </span><span class="token operator" style="color:hsl(221, 87%, 60%)">></span><span class="token plain"> </span><span class="token number" style="color:hsl(35, 99%, 36%)">0</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"> </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">or</span><span class="token plain"> \</span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">                </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">row_value </span><span class="token operator" style="color:hsl(221, 87%, 60%)">&lt;</span><span class="token plain"> </span><span class="token number" style="color:hsl(35, 99%, 36%)">0</span><span class="token plain"> </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">and</span><span class="token plain"> index_value </span><span class="token operator" style="color:hsl(221, 87%, 60%)">&lt;</span><span class="token plain"> </span><span class="token number" style="color:hsl(35, 99%, 36%)">0</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"> </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">or</span><span class="token plain"> \</span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">                </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">row_value </span><span class="token operator" style="color:hsl(221, 87%, 60%)">==</span><span class="token plain"> </span><span class="token number" style="color:hsl(35, 99%, 36%)">0</span><span class="token plain"> </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">and</span><span class="token plain"> index_value </span><span class="token operator" style="color:hsl(221, 87%, 60%)">==</span><span class="token plain"> </span><span class="token number" style="color:hsl(35, 99%, 36%)">0</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">                    matches </span><span class="token operator" style="color:hsl(221, 87%, 60%)">+=</span><span class="token plain"> </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token number" style="color:hsl(35, 99%, 36%)">1</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">            </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">else</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">                matches </span><span class="token operator" style="color:hsl(221, 87%, 60%)">+=</span><span class="token plain"> </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token number" style="color:hsl(35, 99%, 36%)">0</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">            </span><span class="token comment" style="color:hsl(230, 4%, 64%)">#print("{}".format(list_2[index_dt]))</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">return</span><span class="token plain"> matches</span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    </span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    </span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">matches_dict </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">{</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">}</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">count_dict </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">{</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">}</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">prediction_dict </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">{</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">}</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">index_dict </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">{</span><span class="token string" style="color:hsl(119, 34%, 47%)">"SPY"</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"> SPY_close_ret</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> </span><span class="token string" style="color:hsl(119, 34%, 47%)">"DJIA"</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"> DJIA_close_ret</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">              </span><span class="token string" style="color:hsl(119, 34%, 47%)">"RUSS"</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"> RUSS_close_ret</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> </span><span class="token string" style="color:hsl(119, 34%, 47%)">"NASDAQ"</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"> NASDAQ_close_ret</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">}</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">indices </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token string" style="color:hsl(119, 34%, 47%)">"SPY"</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> </span><span class="token string" style="color:hsl(119, 34%, 47%)">"DJIA"</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> </span><span class="token string" style="color:hsl(119, 34%, 47%)">"RUSS"</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> </span><span class="token string" style="color:hsl(119, 34%, 47%)">"NASDAQ"</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">futures </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token string" style="color:hsl(119, 34%, 47%)">"Crude"</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> </span><span class="token string" style="color:hsl(119, 34%, 47%)">"Gold"</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> </span><span class="token string" style="color:hsl(119, 34%, 47%)">"DOW"</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> </span><span class="token string" style="color:hsl(119, 34%, 47%)">"NASDAQ"</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> </span><span class="token string" style="color:hsl(119, 34%, 47%)">"S&P"</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain"></span><span class="token keyword" style="color:hsl(301, 63%, 40%)">for</span><span class="token plain"> index </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">in</span><span class="token plain"> indices</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    matches_dict</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token plain">index</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"> </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">{</span><span class="token plain">future</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"> signs_match</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">article_df</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token plain">future</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"> </span><span class="token string" style="color:hsl(119, 34%, 47%)">'date'</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">,</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">                                               index_dict</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token plain">index</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"> </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">for</span><span class="token plain"> future </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">in</span><span class="token plain"> futures</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">}</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    count_dict</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token plain">index</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"> </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">{</span><span class="token plain">future</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"> </span><span class="token builtin" style="color:hsl(119, 34%, 47%)">len</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">matches_dict</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token plain">index</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token plain">future</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"> </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">for</span><span class="token plain"> future </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">in</span><span class="token plain"> futures</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">}</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    prediction_dict</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token plain">index</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token plain"> </span><span class="token operator" style="color:hsl(221, 87%, 60%)">=</span><span class="token plain"> </span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">{</span><span class="token plain">future</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">:</span><span class="token plain"> np</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">mean</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">matches_dict</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token plain">index</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">[</span><span class="token plain">future</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">]</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">                              </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">for</span><span class="token plain"> future </span><span class="token keyword" style="color:hsl(301, 63%, 40%)">in</span><span class="token plain"> futures</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">}</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    </span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain"></span><span class="token keyword" style="color:hsl(301, 63%, 40%)">print</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token string" style="color:hsl(119, 34%, 47%)">"Articles Checked:"</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain"></span><span class="token keyword" style="color:hsl(301, 63%, 40%)">print</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">pd</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">DataFrame</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">from_dict</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">count_dict</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain"></span><span class="token keyword" style="color:hsl(301, 63%, 40%)">print</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain"></span><span class="token keyword" style="color:hsl(301, 63%, 40%)">print</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token string" style="color:hsl(119, 34%, 47%)">"Prediction Accuracy:"</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token plain"></span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain"></span><span class="token keyword" style="color:hsl(301, 63%, 40%)">print</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">pd</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">DataFrame</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">.</span><span class="token plain">from_dict</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">(</span><span class="token plain">prediction_dict</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><span class="token punctuation" style="color:hsl(119, 34%, 47%)">)</span><br></span></code></pre><div class=buttonGroup__atx><button type=button aria-label="Copy code to clipboard" title=Copy class=clean-btn><span class=copyButtonIcons_eSgA aria-hidden=true><svg viewBox="0 0 24 24" class=copyButtonIcon_y97N><path fill=currentColor d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"/></svg><svg viewBox="0 0 24 24" class=copyButtonSuccessIcon_LjdS><path fill=currentColor d=M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z /></svg></span></button></div></div></div>
<div class="codeBlockContainer_Ckt0 theme-code-block" style="--prism-background-color:hsl(230, 1%, 98%);--prism-color:hsl(230, 8%, 24%)"><div class=codeBlockContent_biex><pre tabindex=0 class="prism-code language-text codeBlock_bY9V thin-scrollbar" style="background-color:hsl(230, 1%, 98%);color:hsl(230, 8%, 24%)"><code class=codeBlockLines_e6Vv><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    Articles Checked:</span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">            DJIA  NASDAQ  RUSS  SPY</span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    Crude    268     268   271  271</span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    DOW      268     268   271  271</span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    Gold     268     268   271  271</span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    NASDAQ   268     268   271  271</span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    S&P      268     268   271  271</span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    </span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    Prediction Accuracy:</span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">                DJIA    NASDAQ      RUSS       SPY</span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    Crude   0.533582  0.529851  0.501845  0.542435</span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    DOW     0.589552  0.608209  0.535055  0.535055</span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    Gold    0.455224  0.451493  0.483395  0.512915</span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    NASDAQ  0.582090  0.626866  0.531365  0.538745</span><br></span><span class=token-line style="color:hsl(230, 8%, 24%)"><span class="token plain">    S&P     0.585821  0.608209  0.535055  0.535055</span><br></span></code></pre><div class=buttonGroup__atx><button type=button aria-label="Copy code to clipboard" title=Copy class=clean-btn><span class=copyButtonIcons_eSgA aria-hidden=true><svg viewBox="0 0 24 24" class=copyButtonIcon_y97N><path fill=currentColor d="M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"/></svg><svg viewBox="0 0 24 24" class=copyButtonSuccessIcon_LjdS><path fill=currentColor d=M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z /></svg></span></button></div></div></div>
<p>Well, it appears that the futures data is terrible at predicting market close. NASDAQ predicting NASDAQ is the most interesting data point, but 63% accuracy isn't accurate enough to make money consistently.</p>
<h2 class="anchor anchorWithStickyNavbar_LWe7" id=final-sentiments>Final sentiments<a href=#final-sentiments class=hash-link aria-label="Direct link to Final sentiments" title="Direct link to Final sentiments"></a></h2>
<p>The data bears out very close to what I expected would happen:</p>
<ul>
<li>Futures data is more accurate than a coin flip for predicting openings, which makes sense since it is recorded only 3 hours before the actual opening</li>
<li>Futures data is about as acccurate as a coin flip for predicting closings, which means there is no money to be made in trying to predict the market direction for the day given the futures data.</li>
</ul>
<p>In summary:</p>
<ul>
<li>Cramer is half right: Futures data is not good for predicting the market open of the Dow and NASDAQ indices. Contrary to Cramer though, it is very good for predicting the S&P and Russell indices - we can achieve an accuracy slightly over 80% for each.</li>
<li>Making money in the market is hard. We can't just go to the futures and treat them as an oracle for where the market will close.</li>
</ul>
<p>I hope you've enjoyed this, I quite enjoyed taking a deep dive in the analytics this way. I'll be posting more soon!</div></article><nav class="pagination-nav docusaurus-mt-lg" aria-label="Blog post page navigation"><a class="pagination-nav__link pagination-nav__link--prev" href=/2015/11/autocallable><div class=pagination-nav__sublabel>Older post</div><div class=pagination-nav__label>Autocallable Bonds</div></a><a class="pagination-nav__link pagination-nav__link--next" href=/2016/01/complaining-about-the-weather><div class=pagination-nav__sublabel>Newer post</div><div class=pagination-nav__label>Complaining about the weather</div></a></nav></main><div class="col col--2"><div class="tableOfContents_bqdL thin-scrollbar"><ul class="table-of-contents table-of-contents__left-border"><li><a href=#downloading-futures-data-from-seeking-alpha class="table-of-contents__link toc-highlight">Downloading Futures data from Seeking Alpha</a><li><a href=#fetching-the-returns-data class="table-of-contents__link toc-highlight">Fetching the Returns data</a><li><a href=#running-the-comparison class="table-of-contents__link toc-highlight">Running the Comparison</a><li><a href=#the-next-step---predicting-the-close class="table-of-contents__link toc-highlight">The next step - Predicting the close</a><li><a href=#final-sentiments class="table-of-contents__link toc-highlight">Final sentiments</a></ul></div></div></div></div></div><footer class=footer><div class="container container-fluid"><div class="footer__bottom text--center"><div class=footer__copyright>Copyright © 2024 Bradlee Speice</div></div></div></footer></div>