<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/how-to-check-data-quality-using-r/"></div></div><div><a data-url="https://www.r-bloggers.com/how-to-check-data-quality-using-r/" data-counturl="https://www.r-bloggers.com/how-to-check-data-quality-using-r/" data-count="horizontal" data-text="How to Check Data Quality using R" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.quantinsti.com/blog/use-clean-data-r/"> R programming</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>
<div>
<div><a data-wpel-link="internal" data-site="" target="_blank" href="http://www.facebook.com/sharer.php?u=http://www.quantinsti.com/blog/use-clean-data-r/"><img title="Facebook" alt="Share on Facebook" src="http://i0.wp.com/www.quantinsti.com/wp-content/plugins/simple-share-buttons-adder/buttons/somacro/facebook.png?w=900" data-recalc-dims="1"></a><a data-wpel-link="internal" data-site="" target="_blank" href="http://twitter.com/share?url=http://www.quantinsti.com/blog/use-clean-data-r/&text=How+to+Check+Data+Quality+using+R+"><img title="Twitter" alt="Tweet about this on Twitter" src="http://i0.wp.com/www.quantinsti.com/wp-content/plugins/simple-share-buttons-adder/buttons/somacro/twitter.png?w=900" data-recalc-dims="1"></a><a data-wpel-link="internal" data-site="linkedin" target="_blank" href="http://www.linkedin.com/shareArticle?mini=true&url=http://www.quantinsti.com/blog/use-clean-data-r/"><img title="LinkedIn" alt="Share on LinkedIn" src="http://i0.wp.com/www.quantinsti.com/wp-content/plugins/simple-share-buttons-adder/buttons/somacro/linkedin.png?w=900" data-recalc-dims="1"></a><a data-wpel-link="internal" data-site="" target="_blank" href="https://plus.google.com/share?url=http://www.quantinsti.com/blog/use-clean-data-r/"><img title="Google+" alt="Share on Google+" src="http://i1.wp.com/www.quantinsti.com/wp-content/plugins/simple-share-buttons-adder/buttons/somacro/google.png?w=900" data-recalc-dims="1"></a></div>
</div>
<p> </p>
<p><img sizes="(max-width: 869px) 100vw, 869px" alt="How to check data quality" class="wp-image-5157" src="http://i2.wp.com/www.quantinsti.com/wp-content/uploads/2016/08/How-to-check-data-quality.jpg?resize=869%2C415" data-recalc-dims="1"></p>
<p>By <a data-wpel-link="external" target="_blank" rel="external noopener noreferrer" href="https://www.linkedin.com/in/milind-paradkar-b37292107">Milind Paradkar</a></p>
<h3><strong>Do You Use Clean Data?</strong></h3>
<p>Always go for clean data! Why is it that experienced traders/authors stress this point in their trading articles/books so often? As a novice trader, you might be using the freely available data from sources like Google or Yahoo finance. Do such sources provide accurate, quality data?</p>
<p>We decided to do a quick check and took a sample of 143 stocks listed on the National Stock Exchange of India Ltd (NSE). For these stocks, we downloaded the 1-minute intraday data for the period 1/08/2016 – 19/08/2016. The aim was to check whether Google finance captured every 1-minute bar during this period for each of the 143 stocks.</p>
<p>NSE’s trading session starts at 9:15 am and ends at 15:30 pm IST, thus comprising of 375 minutes. For 14 trading sessions, we should have 5250 data points for each of these stocks. We wrote a simple code in R to perform the check.</p>
<p>Here is our finding. Out of the 143 stocks scanned, 89 stocks had data points less than 5250, that’s more than 60% of our sample set!! The table shown below lists downs 10 such stocks from those 89 stocks.</p>
<p><img sizes="(max-width: 226px) 100vw, 226px" alt="Symbols" class="wp-image-5156" src="http://i2.wp.com/www.quantinsti.com/wp-content/uploads/2016/08/Symbols.png?resize=226%2C250" data-recalc-dims="1"></p>
<p>Let’s take the case of PAGEIND. Google finance has captured only 4348 1-minute data points for the stock, thus missing 902 points!!</p>
<p>Example – Missing the 1306 minute bar on 20160801:</p>
<p><img sizes="(max-width: 447px) 100vw, 447px" alt="Missing the 1306 minute bar on 20160801" class="wp-image-5155" src="http://i0.wp.com/www.quantinsti.com/wp-content/uploads/2016/08/Missing-the-1306-minute-bar-on-20160801.png?resize=447%2C62" data-recalc-dims="1"></p>
<p>Example – Missing the 1032 minute bar on 20160802:</p>
<p><img sizes="(max-width: 446px) 100vw, 446px" alt="Missing the 1032 minute bar on 20160802" class="wp-image-5154" src="http://i2.wp.com/www.quantinsti.com/wp-content/uploads/2016/08/Missing-the-1032-minute-bar-on-20160802.png?resize=446%2C60" data-recalc-dims="1"></p>
<p>If a trader is running an intraday strategy which generates buy/sell signals based on 1-minute bars, the strategy is bound to give some false signals.</p>
<p>As can be seen from the quick check above, data quality from free sources or from cheap data vendors is not always guaranteed. Many of the cheap data vendors source the data from Yahoo finance and provide it to their clients. Poor data feed is a big issue faced by many traders and you will find many traders complaining about the same on various trading forums.</p>
<p>Backtesting a trading strategy using such data will give false results. If are using the data in live trading and in case there is a server problem with Google or Yahoo finance, it will lead to a delay in the data feed. As a trader, you don’t want to be in a position where you have an open trade, and the data feed stops or is delayed. When trading with real money, one is always advised to use quality data from reliable data vendors. After all, Data is Everything!</p>
<h3><strong>Next Step</strong></h3>
<p>If you’re a retail trader interested in learning various aspects of Algorithmic trading, check out the <a data-wpel-link="internal" target="_blank" href="http://www.quantinsti.com/epat/">Executive Programme in Algorithmic Trading (EPAT)</a>. The course covers training modules like Statistics &amp; Econometrics, Financial Computing &amp; Technology, and Algorithmic &amp; Quantitative Trading. The course equips you with the required skillsets to be a successful trader.</p>
<h3><strong>Download Data Files</strong></h3>
<ul>
<li>Do You Use Clean Data.rar
<ul>
<li>15 Day Intraday Historical Data.zip</li>
<li>F&amp;O Stock List.csv</li>
<li>R code – Google_Data_Quality_Check.txt</li>
<li>R code – Stock price data.txt</li>
</ul>
</li>
</ul>
<p><span></span></p>
<p><a data-wpel-link="internal" target="_blank" href="http://www.quantinsti.com/wp-content/uploads/2016/08/Do-You-Use-Clean-Data.rar"> 
<div><a target="_blank" href="http://www.quantinsti.com/wp-content/uploads/2016/08/Do-You-Use-Clean-Data.rar"><i></i>Download</a></div>
<p> </p></a>
<p>
<div>
<div><a data-wpel-link="internal" data-site="" target="_blank" href="http://www.facebook.com/sharer.php?u=http://www.quantinsti.com/blog/use-clean-data-r/"><img title="Facebook" alt="Share on Facebook" src="http://i0.wp.com/www.quantinsti.com/wp-content/plugins/simple-share-buttons-adder/buttons/somacro/facebook.png?w=900" data-recalc-dims="1"></a><a data-wpel-link="internal" data-site="" target="_blank" href="http://twitter.com/share?url=http://www.quantinsti.com/blog/use-clean-data-r/&text=How+to+Check+Data+Quality+using+R+"><img title="Twitter" alt="Tweet about this on Twitter" src="http://i0.wp.com/www.quantinsti.com/wp-content/plugins/simple-share-buttons-adder/buttons/somacro/twitter.png?w=900" data-recalc-dims="1"></a><a data-wpel-link="internal" data-site="linkedin" target="_blank" href="http://www.linkedin.com/shareArticle?mini=true&url=http://www.quantinsti.com/blog/use-clean-data-r/"><img title="LinkedIn" alt="Share on LinkedIn" src="http://i0.wp.com/www.quantinsti.com/wp-content/plugins/simple-share-buttons-adder/buttons/somacro/linkedin.png?w=900" data-recalc-dims="1"></a><a data-wpel-link="internal" data-site="" target="_blank" href="https://plus.google.com/share?url=http://www.quantinsti.com/blog/use-clean-data-r/"><img title="Google+" alt="Share on Google+" src="http://i1.wp.com/www.quantinsti.com/wp-content/plugins/simple-share-buttons-adder/buttons/somacro/google.png?w=900" data-recalc-dims="1"></a></div>
</div>
<p>The post <a data-wpel-link="internal" target="_blank" rel="nofollow" href="http://www.quantinsti.com/blog/use-clean-data-r/">How to Check Data Quality using R</a> appeared first on <a data-wpel-link="internal" target="_blank" rel="nofollow" href="http://www.quantinsti.com/"></a>.</p>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="https://www.r-bloggers.com/how-to-check-data-quality-using-r/"></div></div><div><a data-url="https://www.r-bloggers.com/how-to-check-data-quality-using-r/" data-counturl="https://www.r-bloggers.com/how-to-check-data-quality-using-r/" data-count="vertical" data-text="How to Check Data Quality using R" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.quantinsti.com/blog/use-clean-data-r/"> R programming</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/eRN-5fFH6ls"></a></div>