<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="http://www.r-bloggers.com/economist-liquidity-index/"></div></div><div><a data-url="http://www.r-bloggers.com/economist-liquidity-index/" data-counturl="http://www.r-bloggers.com/economist-liquidity-index/" data-count="horizontal" data-text="Economist Liquidity Index" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://scweiss.blogspot.com/2016/06/economist-liquidity-index.html"> sweissblaug</a></strong>, and kindly contributed to <a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers)</a> </div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<h1><span>Introduction</span></h1>
<h1>
<div></div>
<div>
<div>A typical article that discusses economics is awash with words that allude economics connection with water or a liquid. For Example: Markets get saturated when a competing country dumps its products on them Stock prices can get frothy and might develop into a bubble A credit squeeze will dampen expectations and sink prices People often talk of soaking the rich’s slush funds Floating more shares will dilute the value of a stock curiously in my opinion: During a credit drought a mortgage will become underwater and business might need a bailout and of course prices and inflate or deflate It is tempting to see the relation of how money flows through an economy and that prices inflate or deflate as money supply increases or decreases. And there has been serious effort to model money and markets as a flow.</div>
<div>The analogy of money or markets as a liquid that flows has a long history in economics. Francois Quesnay discussed the topic in 1758 as the <a target="_blank" href="https://en.wikipedia.org/wiki/Tableau_%C3%A9conomique">Tableau economique</a>. Irving Fisher saw markets as the as <a target="_blank" href="http://judson.blogs.nytimes.com/2009/06/02/guest-column-like-water-for-money/">“the flowing water..”</a>. Machines that model the economy as water called <a target="_blank" href="http://judson.blogs.nytimes.com/2009/06/02/guest-column-like-water-for-money/">“Phillipes Machines”</a>were built in the 1940’s. Perhaps even Monetarists could be included in this line of thought by considering the equation <a target="_blank" href="https://en.wikipedia.org/wiki/Velocity_of_money">MV=PQ</a> (money velocity = prices * quantity) by discussing the flow of money through an economy. However, from a theoretical point of view It is hard to reconcile this concept with<a target="_blank" href="https://research.stlouisfed.org/fred2/series/BASE">recent developments</a>.. but thats another topic.</div>
<div>This post will explore how layman financial / economics articles that discuss the ‘liquidity’ of markets evolve over time using The Economist articles since 1998.</div>
<div>Since articles can discuss multiple topics (politics, regions, science, etc), paragraphs are the basic unit of data (bag of words) and each is assign a probability to whether the paragraph discusses economics.</div>
<div>If a paragraph includes a ‘liquidity’ word (discussed below) then the paragraphs probability of economics will be used as part of the index (to account for possibility that the article is not about economics).</div>
<div>An aggregated score by week is then calculated. Below is the resulting scaled time series of all ‘liquidity’ words along ‘inflat’ and ‘deflat’ (since these are actually technical terms I felt that they should be treated separately from the other liquidity words such as “bubble” in which some economists find that term vacuous).</div>
<div><a target="_blank" href="https://4.bp.blogspot.com/-9MgKBnmvRhM/V1eKCMCB94I/AAAAAAAACWI/sFUl4V3ZcawK24UnRfskX27mxeh2-Y-9QCLcB/s1600/TS_Indicies.png"><img height="492" width="640" src="https://4.bp.blogspot.com/-9MgKBnmvRhM/V1eKCMCB94I/AAAAAAAACWI/sFUl4V3ZcawK24UnRfskX27mxeh2-Y-9QCLcB/s640/TS_Indicies.png" border="0"></a></div>
<div>Of course this is a very messy index, but we do see spikes of “liquidity” index around the financial crises at the end of 2007, a rise in ‘inflat’ index before that date and at least two spikes in ‘deflat’ index around 2003 and 2015.</div>
<div>To smooth out the signal, I made a simple local linear trend model for each index and extracted the states. Below are the time series of the states.</div>
<div></div>
<p>
<div></div>
<p> <a target="_blank" href="https://2.bp.blogspot.com/-a5Duu-XYdXw/V1eKCAOlV4I/AAAAAAAACWE/7N_c1RszjYoQCaRXKKuMNM-xBOZS1prggCLcB/s1600/indicty_states.png"><img height="495" width="640" src="https://2.bp.blogspot.com/-a5Duu-XYdXw/V1eKCAOlV4I/AAAAAAAACWE/7N_c1RszjYoQCaRXKKuMNM-xBOZS1prggCLcB/s640/indicty_states.png" border="0"></a>
<div></div>
<div> From around 2005 to mid 2007 ‘inflat’ occurred at an increasing rate in Economists articles. You can see this index sink very fast, just as the ‘liquidity index’ reached its height during the financial crises. Finally, it’s noteworthy that the most recent blip of discussion is deflation in 2015.</div>
<div>I thought the liquidity index I created looked like the VIX index so I plotted both scaled series below.</div>
<div><a target="_blank" href="https://3.bp.blogspot.com/-L1D_LF-c52Y/V1eKCGTzNVI/AAAAAAAACWA/JRF1pVNBpjwlZD-NpopUwpMRSTYEfBj1QCLcB/s1600/indicy_states_vix.png"><img height="492" width="640" src="https://3.bp.blogspot.com/-L1D_LF-c52Y/V1eKCGTzNVI/AAAAAAAACWA/JRF1pVNBpjwlZD-NpopUwpMRSTYEfBj1QCLcB/s640/indicy_states_vix.png" border="0"></a></div>
<div>They do appear to increase and decrease at same rate, but not perfectly. But maybe if I’m too busy to read The Economist for a particular week, I can just check the VIX and infer how much they talk about liquidity.</div>
<div>Outline of Code: </div>
<div></div>
<ul>
<li>Scraped the economist articles Classified which paragraphs discussed business / economics / financial topics. </li>
<li>Applied model to all paragraphs and ended with prob(economics) – find more information on methodology <a target="_blank" href="http://scweiss.blogspot.com/2016/05/when-predictions-are-more-accurate-than.html">here</a> </li>
<li><span>Created corpus of words that are affiliated with the concept of ‘liquidity’ or ‘water’ in relation to economics – did this through word2vec and manually seeing which words were associated with liquidity. for instance, i found illiquid was similar to liquid so included that in the corpus. i continued this manual process until i couldn’t find any more words. I thought word2vec could be more precise in this regard and I could just find a “liquidity vector” thats similar to the words I’m looking for but this wasn’t the case. Below are words I found that I thought were appropriate </span><span>(“soak”,“mop”, “bubble”,“burst”,“pop”,“frothy”,“saturate”, “liquid”,“deflat”,“choppy”,“topsy”,“mop”,“squeeze”,“damp”,“headwinds”,“dilute”,“sink”, “flow”,“inflat”,“solvent”,“ripple”,“pressure”,“expansion”,“untapped”,“reservoir”,“tap”,“funnel”,“fusion”,“pump”,“bail”,“absorb”,“intake”,“draining”,“meltdown”,“bursts”,“implos”,“explos”,“ripple”,“flotations”,“float”,“sink”,“dump”,“circul”,“spew”,“swell”,“pour”,“splashing”,“flood”,“draught”,“dry”,“reservoirs”, “reservoirs”,“fluid”,“droplets”,“dissolve”) </span></li>
<li>Created an ‘index’ by multiplying the prob(economics)*Indicator(word included) for each paragraph and summing this value by week </li>
<li>Built state-space models for index to smooth values </li>
<li>Found liquidity state sort of follows VIX index.</li>
</ul>
<p>
<pre><code><span>year_links</span><span>=</span><span>function</span><span>(</span><span>year</span><span>)</span><span>{</span><br>  <span>url</span><span>=</span><span>paste0</span><span>(</span><span>&quot;http://www.economist.com/printedition/covers?print_region=76980&amp;date_filter%5Bvalue%5D%5Byear%5D=&quot;</span>,<span>year</span><span>)</span><br>  <span>doc</span> <span>=</span> <span>htmlTreeParse</span><span>(</span><span>url</span>, <span>useInternalNodes</span> <span>=</span> <span>T</span><span>)</span><br>  <span>text</span> <span>=</span> <span>xpathSApply</span><span>(</span><span>doc</span>, <span>&quot;//a&quot;</span>, <span>xmlValue</span><span>)</span> <br>  <span>src</span> <span>=</span> <span>xpathApply</span><span>(</span><span>doc</span>, <span>&quot;//a[@href]&quot;</span>, <span>xmlGetAttr</span>, <span>&quot;href&quot;</span><span>)</span><br>  <span>links</span><span>=</span><span>unlist</span><span>(</span><span>src</span><span>)</span><br>  <span>links</span><span>=</span><span>links</span><span>[</span><span>grep</span><span>(</span><span>paste0</span><span>(</span><span>&quot;printedition/&quot;</span>,<span>year</span><span>)</span>,<span>links</span><span>)</span><span>]</span><br>  <span>links_editions</span><span>=</span><span>paste0</span><span>(</span><span>&quot;http://www.economist.com&quot;</span>,<span>links</span><span>)</span><br>  <span>links_editions</span><span>=</span><span>unique</span><span>(</span><span>links_editions</span><span>)</span><br>  <span>return</span><span>(</span><span>links_editions</span><span>)</span><br><span>}</span><br><br><br><span>weekly_links</span><span>=</span><span>lapply</span><span>(</span><span>1998</span><span>:</span><span>2015</span>, <span>year_links</span><span>)</span><br><span>weekly_links</span><span>=</span><span>unlist</span><span>(</span><span>weekly_links</span><span>)</span><br><span># </span><br><span># doc = htmlTreeParse(links_editions[1], useInternalNodes = T)</span><br><span># src = xpathApply(doc2, &quot;//a[@href]&quot;, xmlGetAttr, &quot;href&quot;)</span><br><span># links_week=unlist(src)</span><br><span># links_week=links_week[grep(&quot;news&quot;,links_week)]</span><br><span># </span><br><span># </span><br><span># </span><br><span>links_editions</span><span>=</span><span>weekly_links</span><span>[</span><span>2</span><span>]</span><br><span>scrape_articles</span><span>=</span><span>function</span><span>(</span><span>links_editions</span><span>)</span><span>{</span><br>  <span>doc</span> <span>=</span> <span>htmlTreeParse</span><span>(</span><span>links_editions</span>, <span>useInternalNodes</span> <span>=</span> <span>T</span><span>)</span><br>  <span>src</span> <span>=</span> <span>xpathApply</span><span>(</span><span>doc</span>, <span>&quot;//a[@href]&quot;</span>, <span>xmlGetAttr</span>, <span>&quot;href&quot;</span><span>)</span><br>  <span>links_week</span><span>=</span><span>unlist</span><span>(</span><span>src</span><span>)</span><br>  <span>links_week</span><span>=</span><span>links_week</span><span>[</span><span>unique</span><span>(</span><span>c</span><span>(</span><span>grep</span><span>(</span><span>&quot;node&quot;</span>,<span>links_week</span><span>)</span>, <span>grep</span><span>(</span><span>&quot;news&quot;</span>,<span>links_week</span><span>)</span><span>)</span><span>)</span><span>]</span><br>  <span>links_week</span><span>=</span><span>paste0</span><span>(</span><span>&quot;http://www.economist.com&quot;</span>,<span>links_week</span><span>)</span><br>  <span>text1</span><span>=</span><span>vector</span><span>(</span><span>length</span><span>=</span><span>length</span><span>(</span><span>links_week</span><span>)</span><span>)</span><br>  <span>for</span><span>(</span><span>i</span> <span>in</span> <span>1</span><span>:</span><span>length</span><span>(</span><span>links_week</span><span>)</span><span>)</span><span>{</span><br>    <span>doc</span><span>=</span> <span>tryCatch</span><span>(</span><span>htmlTreeParse</span><span>(</span><span>links_week</span><span>[</span><span>i</span><span>]</span>, <span>useInternalNodes</span> <span>=</span> <span>T</span><span>)</span>, <span>error</span><span>=</span><span>function</span><span>(</span><span>e</span><span>)</span> <span>NULL</span><span>)</span><br>    <span>if</span><span>(</span><span>!</span> <span>is.null</span><span>(</span><span>doc</span><span>)</span><span>)</span> <span>{</span><span>text</span> <span>=</span> <span>xpathSApply</span><span>(</span><span>doc</span>, <span>'//*[contains(concat( &quot; &quot;, @class, &quot; &quot; ), concat( &quot; &quot;, &quot;main-content&quot;, &quot; &quot; ))]'</span>, <span>xmlValue</span><span>)</span> <span>}</span><br>    <span>if</span><span>(</span><span>length</span><span>(</span><span>text</span><span>)</span><span>!=</span><span>0</span><span>)</span> <span>{</span><span>text1</span><span>[</span><span>i</span><span>]</span><span>=</span><span>text</span><span>}</span><br>  <span>}</span><br>  <span>return</span><span>(</span><span>cbind</span><span>(</span><span>edition</span><span>=</span><span>rep</span><span>(</span><span>links_editions</span>,<span>length</span><span>(</span><span>links_week</span><span>)</span><span>)</span>,<span>link</span><span>=</span><span>links_week</span>, <span>text1</span><span>)</span><span>)</span><br><span>}</span><br><span>articles</span><span>=</span><span>lapply</span><span>(</span><span>weekly_links</span>,<span>scrape_articles</span><span>)</span><br><span>articles1</span><span>=</span><span>do.call</span><span>(</span><span>rbind</span>,<span>articles</span><span>)</span><br><br><span>#extract dates</span><br><span>articles1</span><span>=</span><span>cbind</span><span>(</span><span>articles1</span>,<span>date</span><span>=</span><span>gsub</span><span>(</span><span>&quot;http://www.economist.com/printedition/&quot;</span>,<span>&quot;&quot;</span>, <span>articles1</span><span>[</span>,<span>1</span><span>]</span><span>)</span><span>)</span><br><span>#remove small articles</span><br><span>articles1</span><span>=</span><span>articles1</span><span>[</span><span>-</span><span>which</span><span>(</span><span>nchar</span><span>(</span><span>articles1</span><span>[</span>,<span>3</span><span>]</span><span>)</span><span>&lt;</span><span>550</span><span>)</span>,<span>]</span><br><span>articles1</span><span>=</span><span>cbind</span><span>(</span><span>articles1</span>,<span>numeric_date</span><span>=</span><span>as.numeric</span><span>(</span><span>as.Date</span><span>(</span><span>articles1</span><span>[</span>,<span>&quot;date&quot;</span><span>]</span><span>)</span><span>)</span><span>)</span><br><br><span>library</span><span>(</span><span>stringr</span><span>)</span><br><br><span>##clean up text /split articles into paragraphs</span><br><span>articles_text</span><span>=</span><span>str_replace_all</span><span>(</span><span>articles1</span><span>[</span>,<span>3</span><span>]</span>, <span>&quot;[^[:alnum:]]&quot;</span>, <span>&quot; &quot;</span><span>)</span><br><span>link_articles</span><span>=</span><span>rep</span><span>(</span> <span>unlist</span><span>(</span><span>lapply</span><span>(</span><span>paragraphs</span>,<span>length</span><span>)</span><span>)</span><span>)</span><br><span>paragraphs</span><span>=</span><span>strsplit</span><span>(</span><span>articles1</span><span>[</span>,<span>3</span><span>]</span>, <span>&quot;\n&quot;</span><span>)</span><br><span>date_articles</span><span>=</span><span>rep</span><span>(</span><span>articles1</span><span>[</span>,<span>&quot;date&quot;</span><span>]</span>,<span>unlist</span><span>(</span><span>lapply</span><span>(</span><span>paragraphs</span>,<span>length</span><span>)</span><span>)</span><span>)</span><br><span>link_articles</span><span>=</span><span>rep</span><span>(</span><span>articles1</span><span>[</span>,<span>&quot;link&quot;</span><span>]</span>,<span>unlist</span><span>(</span><span>lapply</span><span>(</span><span>paragraphs</span>,<span>length</span><span>)</span><span>)</span><span>)</span><br><br><br><span>##remove small size paragraphs</span><br><span>paragraphs_1</span><span>=</span><span>unlist</span><span>(</span><span>paragraphs</span><span>)</span><br><span>paragraphs_2</span><span>=</span><span>paragraphs_1</span><span>[</span><span>-</span><span>which</span><span>(</span><span>nchar</span><span>(</span><span>paragraphs_1</span><span>)</span><span>&lt;</span><span>90</span><span>)</span><span>]</span><br><span>link_articles_2</span><span>=</span><span>link_articles</span><span>[</span><span>-</span><span>which</span><span>(</span><span>nchar</span><span>(</span><span>paragraphs_1</span><span>)</span><span>&lt;</span><span>90</span><span>)</span><span>]</span><br><span>date_articles_2</span><span>=</span><span>date_articles</span><span>[</span><span>-</span><span>which</span><span>(</span><span>nchar</span><span>(</span><span>paragraphs_1</span><span>)</span><span>&lt;</span><span>90</span><span>)</span><span>]</span><br><span>link_articles_2</span><span>=</span><span>link_articles</span><span>[</span><span>-</span><span>which</span><span>(</span><span>nchar</span><span>(</span><span>paragraphs_1</span><span>)</span><span>&lt;</span><span>90</span><span>)</span><span>]</span><br><span>paragraph_date</span><span>=</span><span>cbind</span><span>(</span><span>date_articles_2</span>,<span>paragraphs_2</span><span>)</span><br><span>economist_paragraph_section</span><span>=</span><span>unlist</span><span>(</span><span>lapply</span><span>(</span><span>strsplit</span><span>(</span><span>link_articles_2</span>,<span>&quot;/&quot;</span><span>)</span>, <span>function</span><span>(</span><span>x</span><span>)</span> <span>x</span><span>[</span><span>5</span><span>]</span><span>)</span><span>)</span><br><br><span>#create article class</span><br><span>economist_sections</span><span>=</span><span>c</span><span>(</span><span>&quot;united-states&quot;</span>,<span>&quot;middle-east-and-africa&quot;</span>,<span>&quot;europe&quot;</span>,<span>&quot;finance-and-economics&quot;</span>,<span>&quot;business&quot;</span>,<span>&quot;britain&quot;</span>,<span>&quot;books-and-arts&quot;</span>,<span>&quot;asia&quot;</span>,<span>&quot;science-and-technology&quot;</span>,<span>&quot;americas&quot;</span>,<span>&quot;china&quot;</span>,<span>&quot;business-and-finance&quot;</span>,<span>&quot;international&quot;</span><span>)</span><br><span>economist_paragraph_section_target</span><span>=</span><span>economist_paragraph_section</span><br><span>economist_paragraph_section_target</span><span>[</span><span>-</span><span>which</span><span>(</span><span>economist_paragraph_section_target</span> <span>%in%</span> <span>economist_sections</span><span>)</span><span>]</span><span>=</span><span>&quot;&quot;</span><br><br><span>paragraphs_3</span><span>=</span><span>str_replace_all</span><span>(</span><span>paragraphs_2</span>, <span>&quot;[^[:alnum:]]&quot;</span>, <span>&quot; &quot;</span><span>)</span><br><span>paragraphs_3</span><span>=</span><span>iconv</span><span>(</span><span>paragraphs_3</span>, <span>to</span><span>=</span><span>'ASCII//TRANSLIT'</span><span>)</span><br><br><br><span>##create term document matrix for classification </span><br><span>it</span> <span>&lt;-</span> <span>itoken</span><span>(</span><span>paragraphs_3</span>, <br>             <span>preprocess_function</span> <span>=</span> <span>tolower</span>, <br>             <span>tokenizer</span> <span>=</span> <span>word_tokenizer</span>, <br>             <span>ids</span> <span>=</span> <span>1</span><span>:</span><span>length</span><span>(</span><span>paragraphs_3</span><span>)</span><span>)</span><br><br><span>vocab</span> <span>&lt;-</span> <span>create_vocabulary</span><span>(</span><span>it</span>, <span>stopwords</span> <span>=</span> <span>sw</span><span>)</span><br><br><span># Each element of list represents document</span><br><span>tokens</span> <span>&lt;-</span> <span>paragraphs_3</span><span>%>%</span> <br>  <span>tolower</span><span>(</span><span>)</span> <span>%>%</span> <br>  <span>word_tokenizer</span><span>(</span><span>)</span><br><span>it</span> <span>&lt;-</span> <span>itoken</span><span>(</span><span>tokens</span>, <span>ids</span> <span>=</span> <span>1</span><span>:</span><span>length</span><span>(</span><span>paragraphs_3</span><span>)</span><span>)</span><br><span>vocab</span> <span>&lt;-</span> <span>create_vocabulary</span><span>(</span><span>it</span>, <span>stopwords</span> <span>=</span> <span>sw</span><span>)</span><br><span>it</span> <span>&lt;-</span> <span>itoken</span><span>(</span><span>tokens</span>, <span>ids</span> <span>=</span> <span>1</span><span>:</span><span>length</span><span>(</span><span>paragraphs_3</span><span>)</span><span>)</span><br><span># Or</span><br><span># it &lt;- itoken(movie_review$review, tolower, word_tokenizer, ids = movie_review$id)</span><br><span>vectorizer</span> <span>&lt;-</span> <span>vocab_vectorizer</span><span>(</span><span>vocab</span><span>)</span><br><span>dtm</span> <span>&lt;-</span> <span>create_dtm</span><span>(</span><span>it</span>, <span>vectorizer</span><span>)</span><br><br><br><span>dtm_1</span><span>=</span><span>dtm</span><span>[</span><span>-</span><span>which</span><span>(</span><span>economist_paragraph_section_target</span><span>==</span><span>&quot;&quot;</span><span>)</span>,<span>]</span><br><span>economist_paragraph_section_target_1</span><span>=</span><span>economist_paragraph_section_target</span><span>[</span><span>-</span><span>which</span><span>(</span><span>economist_paragraph_section_target</span><span>==</span><span>&quot;&quot;</span><span>)</span><span>]</span><br><span>economist_paragraph_section_target_2</span><span>=</span><span>rep</span><span>(</span><span>0</span>,<span>length</span><span>(</span><span>economist_paragraph_section_target_1</span><span>)</span><span>)</span><br><span>economist_paragraph_section_target_2</span><span>[</span><span>which</span><span>(</span><span>economist_paragraph_section_target_1</span> <span>%in%</span> <span>c</span><span>(</span><span>&quot;business&quot;</span>,<span>&quot;business-and-finance&quot;</span>,<span>&quot;finance-and-economics&quot;</span><span>)</span><span>)</span><span>]</span><span>=</span><span>1</span><br><span>word_sums</span><span>=</span><span>colSums</span><span>(</span><span>dtm_1</span><span>)</span><br><span>dtm_1</span><span>=</span><span>dtm_1</span><span>[</span>,<span>-</span><span>which</span><span>(</span><span>word_sums</span><span>&lt;</span><span>10</span><span>)</span><span>]</span><br><br><br><span>#build model and predict on all text</span><br><span>library</span><span>(</span><span>glmnet</span><span>)</span><br><span>require</span><span>(</span><span>doMC</span><span>)</span><br><span>registerDoMC</span><span>(</span><span>cores</span><span>=</span><span>4</span><span>)</span><br><br><span>glmnet</span><span>=</span><span>cv.glmnet</span><span>(</span><span>dtm_1</span>,<span>economist_paragraph_section_target_2</span>,<span>parallel</span><span>=</span><span>TRUE</span>,<span>family</span><span>=</span><span>&quot;binomial&quot;</span>,<span>type.measure</span><span>=</span><span>&quot;auc&quot;</span><span>)</span><br><span>pred_glmnet</span><span>=</span><span>predict</span><span>(</span><span>dtm</span><span>[</span>,<span>-</span><span>which</span><span>(</span><span>word_sums</span><span>&lt;</span><span>10</span><span>)</span><span>]</span>, <span>s</span><span>=</span><span>'lambda.min'</span><span>)</span><br><span>paragraphs_3</span><span>[</span><span>order</span><span>(</span><span>pred_glmnet</span>,<span>decreasing</span><span>=</span><span>TRUE</span><span>)</span><span>[</span><span>1</span><span>:</span><span>10</span><span>]</span><span>]</span><br><br><span>pred_glmnet</span><span>=</span><span>dtm</span><span>[</span>,<span>-</span><span>which</span><span>(</span><span>word_sums</span><span>&lt;</span><span>10</span><span>)</span><span>]</span><span>%*%</span><span>coef</span><span>(</span><span>glmnet</span>,<span>s</span><span>=</span><span>'lambda.min'</span><span>)</span><span>[</span><span>-</span><span>1</span>,<span>]</span><span>+</span><span>coef</span><span>(</span><span>glmnet</span>,<span>s</span><span>=</span><span>'lambda.min'</span><span>)</span><span>[</span><span>1</span>,<span>]</span><br><span>prob_glmnet</span><span>=</span><span>1</span><span>-</span><span>exp</span><span>(</span><span>-</span><span>pred_glmnet</span><span>[</span>,<span>1</span><span>]</span><span>)</span><span>/</span><span>(</span><span>1</span><span>+</span><span>exp</span><span>(</span><span>-</span><span>pred_glmnet</span><span>[</span>,<span>1</span><span>]</span><span>)</span><span>)</span><br><br><br><br><span>##words to use</span><br><span>words</span><span>=</span><span>unique</span><span>(</span><span>c</span><span>(</span><span>&quot;soak&quot;</span>,<span>&quot;mop&quot;</span>, <span>&quot;bubble&quot;</span>,<span>&quot;burst&quot;</span>,<span>&quot;pop&quot;</span>,<span>&quot;frothy&quot;</span>,<span>&quot;saturate&quot;</span>, <span>&quot;liquid&quot;</span>,<span>&quot;deflat&quot;</span>,<span>&quot;choppy&quot;</span>,<span>&quot;topsy&quot;</span>,<span>&quot;mop&quot;</span>,<span>&quot;squeeze&quot;</span>,<span>&quot;damp&quot;</span>,<span>&quot;headwinds&quot;</span>,<span>&quot;dilute&quot;</span>,<span>&quot;sink&quot;</span>,<br>               <span>&quot;flow&quot;</span>,<span>&quot;inflat&quot;</span>,<span>&quot;solvent&quot;</span>,<span>&quot;ripple&quot;</span>,<span>&quot;pressure&quot;</span>,<span>&quot;expansion&quot;</span>,<span>&quot;untapped&quot;</span>,<span>&quot;reservoir&quot;</span>,<span>&quot;tap&quot;</span>,<span>&quot;funnel&quot;</span>,<span>&quot;fusion&quot;</span>,<span>&quot;pump&quot;</span>,<span>&quot;bail&quot;</span>,<br>               <span>&quot;absorb&quot;</span>,<span>&quot;intake&quot;</span>,<span>&quot;draining&quot;</span>,<span>&quot;meltdown&quot;</span>,<span>&quot;bursts&quot;</span>,<span>&quot;implos&quot;</span>,<span>&quot;explos&quot;</span>,<span>&quot;ripple&quot;</span>,<span>&quot;flotations&quot;</span>,<span>&quot;float&quot;</span> ,<span>&quot;sink&quot;</span>,<span>&quot;dump&quot;</span>,<span>&quot;circul&quot;</span>,<br>               <span>&quot;spew&quot;</span>,<span>&quot;swell&quot;</span>,<span>&quot;pressure&quot;</span>,<span>&quot;pour&quot;</span>,<span>&quot;splashing&quot;</span>,<span>&quot;flood&quot;</span>,<span>&quot;draught&quot;</span>,<span>&quot;dry&quot;</span>,<span>&quot;reservoirs&quot;</span>,<br>               <span>&quot;reservoirs&quot;</span>,<span>&quot;fluid&quot;</span>,<span>&quot;droplets&quot;</span>,<span>&quot;dissolve&quot;</span><span>)</span><span>)</span><br><br><span>business_economics_predictions</span><span>=</span><span>prob_glmnet</span><br><span>library</span><span>(</span><span>parallel</span><span>)</span><br><br><span>#see if word is in a paragraph</span><br><span>is_included</span><span>=</span><span>mclapply</span><span>(</span><span>words</span>, <span>function</span><span>(</span><span>x</span><span>)</span> <span>grepl</span><span>(</span><span>x</span>,<span>paragraphs_3</span>,<span>ignore.case</span> <span>=</span> <span>FALSE</span><span>)</span>, <span>mc.cores</span><span>=</span><span>4</span><span>)</span><br><span>is_included_1</span><span>=</span><span>do.call</span><span>(</span><span>cbind</span>,<span>is_included</span><span>)</span><br><span>liquidity_words</span><span>=</span><span>rowSums</span><span>(</span><span>is_included_1</span><span>[</span>,<span>-</span><span>which</span><span>(</span><span>words</span> <span>%in%</span> <span>c</span><span>(</span><span>&quot;inflat&quot;</span>, <span>&quot;deflat&quot;</span><span>)</span><span>)</span><span>]</span><span>)</span><span>*</span><span>business_economics_predictions</span><br><span>inflat_words</span><span>=</span><span>(</span><span>is_included_1</span><span>[</span>,<span>which</span><span>(</span><span>words</span> <span>%in%</span> <span>c</span><span>(</span><span>&quot;inflat&quot;</span><span>)</span><span>)</span><span>]</span><span>)</span><span>*</span><span>business_economics_predictions</span><br><span>deflat_words</span><span>=</span><span>(</span><span>is_included_1</span><span>[</span>,<span>which</span><span>(</span><span>words</span> <span>%in%</span> <span>c</span><span>(</span><span>&quot;deflat&quot;</span><span>)</span><span>)</span><span>]</span><span>)</span><span>*</span><span>business_economics_predictions</span><br><span>all_words</span><span>=</span><span>(</span><span>is_included_1</span><span>)</span><span>*</span><span>business_economics_predictions</span><br><span>date_articles_3</span><span>=</span><span>date_articles_2</span><br><br><br><span>##aggregate by week</span><br><span>library</span><span>(</span><span>dplyr</span><span>)</span><br><span>count_dates</span><span>=</span><span>as.data.frame</span><span>(</span><span>cbind</span><span>(</span><span>liquidity_words</span>,<span>inflat_words</span>,<span>deflat_words</span>,<span>date</span><span>=</span><span>date_articles_2</span><span>)</span><span>)</span><br><span>count_dates</span><span>[</span>,<span>1</span><span>]</span><span>=</span><span>as.numeric</span><span>(</span><span>as.character</span><span>(</span><span>count_dates</span><span>[</span>,<span>1</span><span>]</span><span>)</span><span>)</span><br><span>count_dates</span><span>[</span>,<span>2</span><span>]</span><span>=</span><span>as.numeric</span><span>(</span><span>as.character</span><span>(</span><span>count_dates</span><span>[</span>,<span>2</span><span>]</span><span>)</span><span>)</span><br><span>count_dates</span><span>[</span>,<span>3</span><span>]</span><span>=</span><span>as.numeric</span><span>(</span><span>as.character</span><span>(</span><span>count_dates</span><span>[</span>,<span>3</span><span>]</span><span>)</span><span>)</span><br><span>count_dates</span><span>$</span><span>prob</span><span>=</span><span>business_economics_predictions</span><br><span>data_date</span> <span>&lt;-</span> <span>group_by</span><span>(</span><span>count_dates</span>, <span>date</span><span>)</span><br><span>index_by_date</span><span>=</span><span>summarise</span><span>(</span><span>data_date</span>, <span>liquidity_words</span><span>=</span><span>sum</span><span>(</span><span>liquidity_words</span><span>)</span>,<span>inflat_words</span><span>=</span><span>sum</span><span>(</span><span>inflat_words</span><span>)</span>,<span>deflat_words</span><span>=</span><span>sum</span><span>(</span><span>deflat_words</span><span>)</span>,<span>prob</span><span>=</span><span>sum</span><span>(</span><span>prob</span><span>)</span><span>)</span><br><span>index_by_date_1</span><span>=</span><span>as.data.frame</span><span>(</span><span>index_by_date</span><span>)</span><br><span>index_by_date_scaled</span><span>=</span><span>index_by_date_1</span> <br><span>index_by_date_scaled</span><span>[</span>,<span>-</span><span>1</span><span>]</span><span>=</span><span>apply</span><span>(</span><span>index_by_date_scaled</span><span>[</span>,<span>-</span><span>1</span><span>]</span>,<span>2</span>,<span>scale</span><span>)</span> <br><span>index_by_date_melt</span><span>=</span><span>melt</span><span>(</span><span>index_by_date_scaled</span><span>[</span>,<span>-</span><span>5</span><span>]</span><span>)</span><br><br><br><span>library</span><span>(</span><span>reshape</span><span>)</span><br><span>library</span><span>(</span><span>ggplot2</span><span>)</span><br><br><span>index_plot</span><span>=</span><span>ggplot</span><span>(</span><span>index_by_date_melt</span>, <span>aes</span><span>(</span><span>x</span><span>=</span><span>as.Date</span><span>(</span><span>date</span><span>)</span>, <span>y</span><span>=</span><span>value</span>, <span>group</span><span>=</span><span>variable</span>,<span>colour</span><span>=</span><span>variable</span><span>)</span><span>)</span> <span>+</span>  <span>geom_line</span><span>(</span><span>)</span><span>+</span> <span>scale_x_date</span><span>(</span><span>)</span><span>+</span> <span>theme_classic</span><span>(</span><span>)</span><span>+</span><span>labs</span><span>(</span><span>title</span><span>=</span><span>&quot;Time Series of Indicies&quot;</span>,<span>x</span><span>=</span><span>&quot;Date (by Week)&quot;</span>,<span>y</span><span>=</span><span>&quot;Scaled Value of Index&quot;</span><span>)</span><br><br><br><br><br><br><span>code</span><span>=</span><span>&quot;<br><br>data {<br>int n;<br>vector[n] y;<br>real&lt;lower=0> theta1_mean;<br>real&lt;lower=0> theta1_sd;<br>}<br><br>parameters {<br>real&lt;lower=0> sigma_v;<br>real&lt;lower=0> sigma_w;<br><br>vector[n] theta_innov;<br><br>}<br><br>transformed parameters {<br>vector[n] theta;<br><br>theta[1] &lt;- theta1_mean + theta1_sd * theta_innov[1];<br>for (t in 2:n) {<br>theta[t] &lt;- theta[t - 1] + sigma_w * theta_innov[t]; <br>}<br><br><br>}<br><br>model {<br>theta_innov ~ normal(0, 1);<br>y ~ normal(theta, sigma_v);<br>}<br>&quot;</span><br><br><span>local_linear_trend_state</span><span>=</span><span>function</span><span>(</span><span>y</span><span>)</span><span>{</span><br>  <span>fit</span><span>=</span><span>stan</span><span>(</span><span>model_code</span><span>=</span><span>code</span>, <span>data</span><span>=</span><span>list</span><span>(</span><span>y</span><span>=</span><span>y</span>, <span>n</span><span>=</span><span>length</span><span>(</span><span>y</span><span>)</span>, <span>theta1_mean</span><span>=</span><span>y</span><span>[</span><span>1</span><span>]</span>, <span>theta1_sd</span><span>=</span><span>10</span><span>)</span>, <span>iter</span><span>=</span><span>2000</span>,<span>chains</span><span>=</span><span>3</span><span>)</span><br>  <span>mat</span><span>=</span><span>as.matrix</span><span>(</span><span>fit</span><span>)</span><br>  <span>theta</span><span>=</span><span>mat</span><span>[</span>,<span>grep</span><span>(</span><span>&quot;theta&quot;</span>, <span>colnames</span><span>(</span><span>mat</span><span>)</span><span>)</span><span>]</span><br>  <span>return</span><span>(</span><span>colMeans</span><span>(</span><span>theta</span><span>[</span>,<span>-</span><span>c</span><span>(</span><span>1</span><span>:</span><span>ncol</span><span>(</span><span>theta</span><span>)</span><span>/</span><span>2</span><span>)</span><span>]</span><span>)</span><span>)</span><br><span>}</span><br><br><span>index_by_date_1</span><span>$</span><span>all_liquidity</span><span>=</span><span>rowSums</span><span>(</span><span>index_by_date_1</span><span>[</span>,<span>c</span><span>(</span><span>&quot;liquidity_words&quot;</span> ,<span>&quot;inflat_words&quot;</span> ,<span>&quot;deflat_words&quot;</span><span>)</span><span>]</span><span>)</span><br><span>states</span><span>=</span><span>lapply</span><span>(</span><span>index_by_date_1</span><span>[</span>,<span>-</span><span>1</span><span>]</span>,<span>local_linear_trend_state</span><span>)</span><br><span>states</span><span>=</span><span>do.call</span><span>(</span><span>cbind</span>,<span>states</span><span>)</span><br><span>states_1</span><span>=</span><span>states</span><br><span>states_1</span><span>=</span><span>data.frame</span><span>(</span><span>date</span><span>=</span><span>index_by_date_1</span><span>[</span>,<span>&quot;date&quot;</span><span>]</span>, <span>states_1</span><span>)</span><br><span>states_1</span><span>[</span>,<span>-</span><span>1</span><span>]</span><span>=</span><span>apply</span><span>(</span><span>states_1</span><span>[</span>,<span>-</span><span>1</span><span>]</span>,<span>2</span>,<span>scale</span><span>)</span><br><br><span>library</span><span>(</span><span>reshape</span><span>)</span><br><span>states_melt</span><span>=</span><span>melt</span><span>(</span><span>states_1</span><span>[</span>,<span>-</span><span>5</span><span>]</span><span>)</span><br><span>library</span><span>(</span><span>ggplot2</span><span>)</span><br><span>states_plot</span><span>=</span><span>ggplot</span><span>(</span><span>states_melt</span>, <span>aes</span><span>(</span><span>x</span><span>=</span><span>as.Date</span><span>(</span><span>date</span><span>)</span>, <span>y</span><span>=</span><span>value</span>, <span>group</span><span>=</span><span>variable</span>,<span>colour</span><span>=</span><span>variable</span><span>)</span><span>)</span> <span>+</span>  <span>geom_line</span><span>(</span><span>)</span><span>+</span> <span>scale_x_date</span><span>(</span><span>)</span><span>+</span> <span>theme_classic</span><span>(</span><span>)</span><span>+</span><span>labs</span><span>(</span><span>title</span><span>=</span><span>&quot;Time Series of Indicy States&quot;</span>,<span>x</span><span>=</span><span>&quot;Date (by Week)&quot;</span>,<span>y</span><span>=</span><span>&quot;Scaled Value of Index&quot;</span><span>)</span><br><br><span>library</span><span>(</span><span>Quandl</span><span>)</span><br><span>library</span><span>(</span><span>Quandl</span><span>)</span><br><span>vix</span><span>=</span><span>Quandl</span><span>(</span><span>&quot;CBOE/VIX&quot;</span><span>)</span><br><br><span>states_1</span><span>[</span>,<span>1</span><span>]</span><span>=</span><span>as.Date</span><span>(</span><span>states_1</span><span>[</span>,<span>1</span><span>]</span><span>)</span><span>-</span><span>1</span><br><span>states_2</span><span>=</span><span>merge</span><span>(</span><span>states_1</span>,<span>vix</span>, <span>by.x</span><span>=</span><span>&quot;date&quot;</span>,<span>by.y</span><span>=</span><span>&quot;Date&quot;</span>, <span>all</span><span>=</span><span>TRUE</span><span>)</span><br><span>states_2</span><span>[</span>,<span>1</span><span>]</span><span>=</span><span>as.factor</span><span>(</span><span>states_2</span><span>[</span>,<span>1</span><span>]</span><span>)</span><br><span>states_2</span><span>[</span>,<span>-</span><span>1</span><span>]</span><span>=</span><span>apply</span><span>(</span><span>states_2</span><span>[</span>,<span>-</span><span>1</span><span>]</span>,<span>2</span>,<span>scale</span><span>)</span><br><span>states_3_melt</span><span>=</span><span>melt</span><span>(</span><span>states_2</span><span>[</span><span>-</span><span>which</span><span>(</span><span>is.na</span><span>(</span><span>states_2</span><span>[</span>,<span>2</span><span>]</span><span>)</span><span>)</span>,<span>c</span><span>(</span><span>&quot;date&quot;</span>,<span>&quot;liquidity_words&quot;</span>,<span>&quot;VIX Close&quot;</span><span>)</span><span>]</span><span>)</span></code></pre>
<div><code><span><br></span></code></div>
</div>
</h1>
</div>
</div>
<div></div>
</div>
</div>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="http://www.r-bloggers.com/economist-liquidity-index/"></div></div><div><a data-url="http://www.r-bloggers.com/economist-liquidity-index/" data-counturl="http://www.r-bloggers.com/economist-liquidity-index/" data-count="vertical" data-text="Economist Liquidity Index" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://scweiss.blogspot.com/2016/06/economist-liquidity-index.html"> sweissblaug</a></strong>.</div>
<hr>
<a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="http://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="http://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="http://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="http://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="http://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="http://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="http://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="http://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="http://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="http://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="http://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="http://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="http://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="http://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="http://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="http://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/7tNkWVH-aNQ"></a></div>