Posts Tagged ‘slopegraph’

Slopegraph As A Service

@adammontville posited that Figure 15 from this year’s DBIR could use some slopegraph love. As I am not one to back down from a reasonable challenge, I obliged.

Here’s the original chart (produced by @jayjacobs):

figure15-orig

and, here’s a very quick slopegraph version of it:

figure15-slope

You can click on both/either for a larger version. If I had more time, I could have made the slopegraph version nicer, but it conveys a story fairly well the way it is, especially with the highlight on the two biggest changes between 2008 & 2012.

Two problems with the modified visualization are (a) multi-column slopegraphs blend into a parallel coordinate or plain old line graph pretty quickly (thus, reducing their slopegraph-y goodness); and, (b) the diversity of the year-over-year DBIR data set makes the comparison between years almost pointless (as the DBIR itself points out).

I also generated a proper/traditional slopegraph, comparing 2008 to 2012:

figure15-true-slope

The visualization is far more compact and, if the goal was to show the change between 2008 and 2012, it provides a much clearer view of what has and has not changed.

Businessweek’s #Spiffy Rank-order Stock Market Slopegraph

Businessweek’s bleeding-edge approach to typography, layout and overall design is one of the features that keeps me reading the magazine in print form. The design team also often delves into experiments with data visualization and short-form infographics and the most recent issue (Sept 3, 2012) is no exception. Given my proclivity towards slopegraphs, I felt compelled to comment both on their “U.S. Stocks Lead the World” slopegraph:

I think they did a fine job combining some of the aspects of a bubble chart with a rank-order slopegraph. Normally, annotation would be necessary as most slopegraphs are comparing values instead of position; however there is sufficient labeling and consistent use of sizing, colors and other visualization hints to overcome most — if not all — of the problems usually found when using a slopegraph.

Kudos to both Lu Wang and Rita Nazareth!

Just When You Thought It Was Safe To Make A Slopegraph

Thanks to a nice call-out post link on Flowing Data in my RSS feeds this morning, I found Naomi Robbins’ Effective Graphs Forbes blog, perused the archives a bit and came across her post on arrow charts.

She presented a nice comparison between (ugh) pie charts, arrow charts and slopegraphs. Sadly, both the NPR slopegraph and Peltier’s slopegraph included in the article committed some of the cardinal sins of slopegraphs I have pointed out previously. Let’s take a look (click on each graphic to make them bigger):

<center

  • Use of binning/rounding without annotation
  • Use of binning/rounding but not to show rate of change
  • Stacking labels (presenting rank where none exists)

More faithful representations would be explicit rounding/binning (to only show rate of change):

or the full scale version (warning: huge slopegraph) to accurately show the value differences and rate of change:

The data set is small, so transcription is not really be an issue, but here is is for you if you want to play with it some more.

This is definitely a case where her arrow charts are a solid alternative to slopegraphs, so definitely check out her post.

Slopegraphs in Python – Failed States Index (Part 1)

The Fund For Peace (FFP) and Foreign Policy jointly released the 2012 version of the “failed states index” (FSI). From the FFP site, the FSI:

…focuses on the indicators of risk and is based on thousands of articles and reports that are processed by our CAST Software from electronically available sources.

I read it every year (mostly due to being an ardent reader of Foreign Policy magazine) and find the rankings, methodology & insights quite intriguing. With my recent work on slopegraphs, I thought this would be a good data set to play with to determine what – if any – features were necessary to support rank order (and to provide some impetus to finally refactor the code to support multi-column slopegraphs…more on that later).

However, I was not looking forward to transcribing the data from the Flash visualization on the Foreign Policy web site. There are HTML grids on the FFP site but I really just wanted the overall rankings (i.e. no sub-indices) and noticed this interesting scrollable mini-grid on one of the FFP FSI pages:

Thankfully[?] it’s an IFRAME and I was able to pull 2010, 2011 & 2012 data in a very usable format by manipulating this URL: http://www.fundforpeace.org/global/tables/fsiindex2010_sml.htm.

After some quick transformations, I had two CSV files for a 2010-2012 comparison and a 2011-2012 comparison.

(Before continuing, I feel the need to point out that the data, methodology, etc is 100% Copyright © 2012 The Fund for Peace as they overtly point out many times on their site.)

When I threw the data into the slopegraph tool, it was immediately obvious that I was missing something important: the ability to specify sort order for the data. For most slopegraphs, the code works well since our brains expect the larger values on the top. For a rank-order slopegraph, that sort order (for the most part) should be ascending vs descending to best represent changes in rank position. It does feel odd that being “#1″ in the FSI actually means you’re really a loser, but I didn’t make the rules for their index.

So, PySlopegraph now handles two column rank order slopegraphs and, as you’ll see in part two, also handles multi-column slopegraphs (but that bit needs some work). The code will be up on github in a couple days as I’ve also got some half-finished support for Processing.js and Paper.js that I want to finish before another push. If anyone needs it sooner, just @ or DM me.


Now, For The Data

The “Top 25″ (that sounds way too positive for what it really means) slopegraph is the easiest to read (as it’s the smallest). It is also where Foreign Policy & FFP focus some dataviz effort as well (though they do have visualizations for all the data). Here’s the slopegraph showing the rank order chance from 2010 to 2012:

The full slopegraphs are tall slopegraphs (I’ve been prototyping some ways to make tall ones more useful, but that’s nowhere near ready for public consumption). You may just want to grab the two PDFs and look there vs in this post:

Rank Order Comparison :: 2010/2012

Rank Order Comparison :: 2011/2012

While it requires scrolling, the changes in rank are immediately noticeable as is the fact that the the FFP folk allow for ties that leave “holes” in the table. I think you really get a feel for which countries are stable, improving and declining very quickly with the slopegraph version, but I’d like to hear your thoughts if you have an opine you’d like to share.

Stay tuned for part two!

Slopegraphs in Python – Gratuitous Raphaël Animations

UPDATE: It seems my use of <script async> optimization for Raphaël busted the inline slopegraph generation. Will work on tweaking the example posts to wait for Raphaël to load when I get some time.

So, I had to alter this to start after a user interaction. It loaded fine as a static, local page but seems to get a bit wonky embedded in a complex page. I also see some artifacts in Chrome but not in Safari. Still, not a bad foray into basic animation.

Animate Slopegraph


Slopegraphs in Python – Formatting Tweaks

There were enough eye-catching glitches in the experimental javascript support and the ugly large-number display in the spam example post that I felt compelled to make a couple formatting tweaks in the code. I also didn’t have time to do “real” work on the codebase this weekend.

So, along with spacing adjustments, there’s now an “add_commas” non-mandatory option that will toss commas in large numbers so they’re easy to read. Here’s an example of the new output (both the Raphaël display and commas):

As usual, it’s up on github

Slopegraphs Everywhere

Not much progress over the weekend on my latest obsession (been busy enjoying some non-rainy days here in Maine). So, here are some other slopegraph implementations/resources I’ve found through mining the internets:

Slopegraphs in Python – Experimental Raphaël Support

In preparation for the upcoming 1.0 release and with the hopes of laying a foundation for more interactive slopegraphs, I threw together some rudimentary output support over lunch today for Raphaël, which means that all you have to do is generate a new slopegraph with the “js” output type and include the salient portions of the generated html/css/javascript into a web page (along with including the Raphaël script code).

The next github push will have this update. Here’s an example of the output, using the classic Tufte example chart:

It’s definitely a bit rough around the edges (my eyes immediately fixate upon spacing discrepancies) and lacking any interactivity, but the basic building blocks are in place. It also does not render on my Android phone (HTC Incredible 2) but it does render in Chrome, Safari & on my iPad. Embedding a Raphaël graphic in a web page will definitely have advantages over a PNG or PDF in most situations even if it’s not interactive, so I’ll probably keep the support in regardless of whether I continue to improve upon it.

As I was playing with the code, I kept thinking how neat it would be if there was a Raphaël Cairosurface” option. Perhaps that will be a side project if all goes well, since it would not be that much more complicated (in fact, it may be less complicated) than the Cairo SVG surface code.

Slopegraphs in Python – Log Scales & Spam Data Analysis

Given the focus on actual development of the PySlopegraph tool in most of the blog posts of late, folks may be wondering why an infosec/inforisk guy is obsessing so much on a tool and not talking security. Besides the fixation on filling a void and promoting an underused visualization tool, I do believe there is a place for slopegraphs in infosec data analysis and will utilize some data from McAfee’s recent Q1 2012 Threat Report [PDF] to illustrate how one might use slopegraphs in interpreting the “Spam Volume” data presented in the “Messaging Threats” section (pages 11 & 12 of the report).

The report shows individual graphs of spam volume per country from April of 2011 through March of 2012. Each individual graph conveys useful information, but I put together two slopegraphs that each show alternate and aggregate views which let you compare spam volume data relative to each country (versus just in-country).

When first doing this exploration, the scale problem reared it’s ugly head again since the United States is a huge spam outlier and causes the chart to be as tall as my youngest son when printed. I really wanted to show relative spam volume between countries as well as the increase or decrease between years in one chart and — after chatting with @maximumyin a bit — decided to test out using a log scale option for the charting (click for larger image):

This chart — Spam Volume by Country — instantly shows that:

  • overall volume has declined for most countries
  • two countries have remained steady
  • one country (Germany) has increased

The next chart – Spam Volume Percentage by Country — also needed to be presented on a log scale and has some equally compelling information:

Despite holding steady count-wise, the United States percentage of global spam actually increased and is joined by seven other countries, with Germany having the second largest percentage increase. Both charts present an opportunity to further explore why the values changed (since the best metrics are supposed to both inform and be actionable in some way).

I’m going to extract some more data from the McAfee report and some other security reports to show how slopegraphs can be used to interpret the data. Feedback on both the views and the use of the log scale would be greatly appreciated by general data scientists as well as those in the infosec community.

Slopegraphs in Python – Exploring Binning/Rounding

One of the last items for the 1.0 release is support for multiple columns of data. That will require some additional refactoring, so I’ve been procrastinating by exploring the recent “fudging” discovery. Despite claims to the contrary on other sites, there are more folks playing with slopegraphs than you might imagine. The inspiration for today’s installment comes from Jon Custer (@stuffisthings). He has a two partTelling Stories with Data” series that does some exploration of export data with slopegraphs. In his “Slopegraph Strikes Back” post, Jon does a spiffy job discussing data visualization fundamentals and walks the reader through his re-design of a chart on commodities ranking, including a commentary on an aspect of slopegraphs that I’ve been noticing as I’ve been doing my exploring: the ‘scale’ problem (which I began to point out in the aforementioned “fudging” post).

The data set Jon is working with allows for a great exploration as to what works best when trying to convey a message with slopegraphs. I took the values from one of the tables he extracted:

and made a “raw” slopegraph from them (focusing on the “top 10″). The graphic won’t even come close to fitting in this post but you can grab the PDF of it and see how scale is the primary enemy of slopegraphs. It does show how gold and precious metal ores have skyrocketed from 1998 to 2007, but it’s hardly an engaging and easy to read visualization (unless you really like using your scroll wheel).

Jon grok’d this point, too, and decided to focus on the power law ranking and use the slopegraph to present the rate of change of each commodity:

While he didn’t “pull a Tufte” and just include values without caveat (see left & right 90° side labels), I still believe that there needs to be either increased annotation or the inclusion of base tabular data. Using my PySlopegraph code (forgot to mention the name change), I worked up a version of Jon’s visualization that I believe provides a clean, honest view of the data (click for larger view):

Because the chart is still based on the percentages that are fairly precise:

1
2
3
4
5
6
7
8
9
10
11
12
13
"Coconuts, Brazil nuts, cashews",17.93,0.93
Coffee,12.93,3.91
Fish,7.89,5.04
Tobacco,7.25,3.19
Gold,6.62,18.63
Tea,4.14,1.32
Cotton,4.01,1.36
Cloves,3.58,0.29
Diamonds,3.44,0.58
Mounted stones,2.44,1.5
Vegetables,1.61,1.73
Wheat,0.54,1.38
"Precious metal ores",0,6.76

I finally added an option to the PySlopegraph configuration file for rounding (NOTE: rounding != true binning). If you add the “round_precision” option with a value that supports Python’s round function’s little-known second parameter (arbitrary positional rounding), you can have the values round to decimal or tens/hundreds/etc places which will help with scaling issues, but will also group items (in ways that you may not have originally intended). For this chart, if we use a value of “1″ (first decimal rounding precision…use negative values for rounding on the whole integer side of the decimal) it’s still unreadable due to the scale it imposes by that precision, so I ended up using the nearest whole integer rounding option (value of “0″) and also included the table of actual values, along with annotating the “rate of change” nature of the slopes.

This (again) defeats the “no wasted ink (pixels?)” component of Tufte’s original creation, but I believe it’s necessary for some types of slopegraphs to ensure the chart can stand on it’s own. I’m definitely becoming more convinced that many slopegraphs are more suited for an interactive visualization where you can encode more information in rollovers/popups/etc plus allow for switching of view from percentage, power-law ranking or raw numeric comparison.

For those interested in playing with this particular data set, it’ll be included in the next github code push, which will also include the rounding feature.

Optimization WordPress Plugins & Solutions by W3 EDGE