<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="http://www.r-bloggers.com/best-practices-for-logging-computational-systems-in-r-and-python/"></div></div><div><a data-url="http://www.r-bloggers.com/best-practices-for-logging-computational-systems-in-r-and-python/" data-counturl="http://www.r-bloggers.com/best-practices-for-logging-computational-systems-in-r-and-python/" data-count="horizontal" data-text="Best practices for logging computational systems in R and Python" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://cartesianfaith.com/2016/07/07/best-practices-for-logging-computational-systems-in-r-and-python/"> R – Cartesian Faith</a></strong>, and kindly contributed to <a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>As is the case with most quant software, it’s a bit different from run-of-the-mill software. The somewhat prosaic world of logging is one such place where there are some differences. What’s different about quant systems? First, they have multiple run modes. Particularly in finance, models often run in real-time but also historically. Models may also run incrementally or in batch. These scenarios each have their own logging needs. In particular, real-time processes typically need to show information about every event, whereas batch processes might only log every <img title="n" alt="n" class="latex" src="https://s0.wp.com/latex.php?latex=n&bg=ffffff&%23038;fg=333333&%23038;s=0"> events/samples. Alternatively, incremental processes may require a lot of debugging information, whereas the performance of batch processes will suffer with too much logging.</p>
<p><img height="325" width="325" src="http://www.ergo101.com/media/Printer-Stands/bottom_feed_printer_stand-rolling_printer_carts_02-21701_325"></p>
<p>A good approach that balances these different scenarios is to log messages to both the console and a file. For interactive use the console is most convenient since the feedback is immediate. For long batch processes or running as a system, having the information in a log file is ideal. Trying to search for a log event in the console is a good way to waste a lot of time, whereas as a file, you can take advantage of standard UNIX tools like <code>grep</code> and <code>sed</code>. Another nice thing about two loggers is that you can assign different log levels to each one. The console can be verbose, while the file can be more terse to ease searching for information. The same approach works in both Python and R.</p>
<h3>Python</h3>
<p>Logging comes as “included batteries” in Python. The stock <code>logging</code> package is powerful and flexible. The drawback is that it can be onerous to setup. In large systems, it’s not uncommon to see a dozen logging configurations interspersed in code, each with their own setup, message format, and log levels. It’s much better to standardize the logging and apply it <em>once</em> at the top-level of the application. I usually talk about software being like an onion or a cabbage, with multiple layers of abstraction. The outer layers are tough and designed to protect against the outside world. As you go towards the center, it is clean and pure. The consequence is that the core assumes correct data types and whatnot, while it’s the responsibility of the outer layers to ensure proper data types and formats. This makes testing easier, which I discussed in a prior post. It also makes it easier to re-use functions since the pure core is more readily shared, while the use-case-specific or environment-specific code is wrapped up in the outer layers.</p>
<p>Anyway, back to logging. Following this principle, logging configuration is application specific, so it should be in the outer layers. In Python, there is usually a main function that acts as the entry point of a script, such as</p>
<pre>
if __name__ == '__main__':
  # Call app entry point
</pre>
<p>This is a good place to initialize your logging. Then all modules within your package/system <em>assume</em> logging is configured. If it isn’t Python will tell you that logging hasn’t been initialized, so you know when this assumption is faulty.</p>
<p>Python logging demands intialization. The simplest is to use a one-liner such as <code>logging.basicConfig(level=logging.INFO)</code>, which sets the ROOT logger level to INFO and prints to stdout. But you’ll grow out of this in a few days, particularly because the default message format is not great. Better to use a configuration file to manage the logging system. The default ini-style log configurations are somewhat arcane and opaque due to the way sections have to be referenced by other variables. The newer YAML syntax is more intuitive. As a markup language YAML is becoming more commonplace, so it’s a reasonable replacement.</p>
<p>Our goals for logging include:</p>
<ol>
<li>define a useful message format that is both informative and easy to read;</li>
<li>a verbose logger that writes to the console;</li>
<li>a less verbose logger that writes to a file.</li>
</ol>
<p>The following configuration satisfies the above goals. Typically, this configuration can be placed in a <code>conf</code> directory that contains all configuration files.</p>
<pre>
version: 1
formatters:
  simpleFormater:
    format: '%(levelname)7s [%(asctime)s] %(name)s: %(message)s'
    datefmt: '%Y-%m-%d %H:%M:%S'
  
handlers:
  console:
    class: logging.StreamHandler
    formatter: simpleFormater
    stream: ext://sys.stdout
  file:
    level: INFO
    class : logging.FileHandler
    formatter: simpleFormater
    filename: example.log

loggers:
  clogger:
    handlers: [console]
  flogger:
    handlers: [file]

root:
  level: DEBUG
  handlers: [console, file]
</pre>
<p>Let’s call this config file <code>example_log.yaml</code>. Using the logging configuration is easy: open the file and tell the logging system to use it.</p>
<pre>
import logging.config, yaml
with open('example_log.yaml') as f:
  D = yaml.load(f)
  logging.config.dictConfig(D)
</pre>
<p>Now in your modules, you create a logger and log messages as regular.</p>
<pre>
logger = logging.getLogger(__name__)
logger.info(&quot;This is an INFO log message&quot;)
logger.warning(&quot;This is a WARNING message&quot;)
logger.debug(&quot;This is a DEBUG message&quot;)
</pre>
<p>In this example, all three messages will be output to the console, but only two will be written to disk.</p>
<p>This approach works for most purposes. If you want to change the name of the log file, update the <code>filename</code> property of the file handler from <code>example.log</code> to your preferred log file name. For additional customizations, refer to the Python <a target="_blank" href="https://docs.python.org/2/howto/logging-cookbook.html">Logging Cookbook</a>.</p>
<h3>R</h3>
<p>In R, logging is not a in the included batteries. To fill this void, I wrote the <code>futile.logger</code> package to provide an easy to use logging facility in R. This package has similar semantics to Python’s logging. The primary difference is that a convenient initial configuration is provided by default. In fact, to get console output, no configuration is needed at all! For example, after installing from <a target="_blank" href="https://cran.r-project.org/web/packages/futile.logger/index.html">CRAN</a> or <a target="_blank" href="https://github.com/zatonovo/futile.logger">github</a>, you can log immediately to the console with a threshold of INFO</p>
<pre>
> flog.info(&quot;My first log message&quot;)
INFO [2016-07-07 15:44:44] My first log message
</pre>
<p><code>futile.logger</code> also supports multiple loggers and can write to files. All interaction is done via functions, where a logger is specified. By default, the ROOT logger is used, so if you don’t need more than that, don’t worry about the logger name. For example, to change the log threshold to DEBUG, use <code>flog.threshold</code>.<br>
> flog.threshold(DEBUG)<br>
> flog.debug(“My debug log message”)<br>
DEBUG [2016-07-07 15:52:21] My debug log message<br>
[/sourcecode]</p>
<p>This means during an interactive run, you can quickly turn on DEBUG level messages, while the normal code runs at INFO level. If you want to write to both console and file, specify a new appender using <code>flog.appender</code>. Many appenders come bundled with <code>futile.logger</code>. Here we’ll use the tee appender to write to console and file.</p>
<pre>
> flog.appender(appender.tee('example.log'))
NULL
> flog.info(&quot;Writing to file&quot;)
INFO [2016-07-07 15:54:16] Writing to file
</pre>
<p>Unlike Python logging, <code>futile.logger</code> does not support configuration files. Instead, a code-based configuration must be used. The same principle can be followed though, where the configuration should be at the top level of the code. For R packages, this can be in the <code>.onLoad(libname, pkgname)</code> function. More information is available <a target="_blank" href="https://github.com/zatonovo/futile.logger">in the source</a>.</p>
<h3>Other Considerations</h3>
<p>While logging greatly helps debugging code and understanding what a system is doing, there are a few things to be wary of. First, make sure not to accidentally commit your log files into your repository. The best way to prevent this is to add <code>*.log</code> in your .gitignore file.</p>
<h3>Conclusion</h3>
<p>With a bit of forethought, logging can be beneficial to your system and development process. By using the recipes above, you can quickly take advantage of logging in your data science projects.</p>
<p> <a target="_blank" rel="nofollow" href="http://feeds.wordpress.com/1.0/gocomments/cartesianfaith.wordpress.com/2161/"><img alt="" src="http://feeds.wordpress.com/1.0/comments/cartesianfaith.wordpress.com/2161/" border="0"></a> <img alt="" src="http://i1.wp.com/pixel.wp.com/b.gif?resize=1%2C1&ssl=1" border="0" data-recalc-dims="1"></p>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="http://www.r-bloggers.com/best-practices-for-logging-computational-systems-in-r-and-python/"></div></div><div><a data-url="http://www.r-bloggers.com/best-practices-for-logging-computational-systems-in-r-and-python/" data-counturl="http://www.r-bloggers.com/best-practices-for-logging-computational-systems-in-r-and-python/" data-count="vertical" data-text="Best practices for logging computational systems in R and Python" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://cartesianfaith.com/2016/07/07/best-practices-for-logging-computational-systems-in-r-and-python/"> R – Cartesian Faith</a></strong>.</div>
<hr>
<a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="http://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="http://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="http://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="http://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="http://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="http://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="http://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="http://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="http://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="http://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="http://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="http://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="http://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="http://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="http://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="http://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/vKB7vlPty_E"></a></div>