Development Archives - Page 4 of 5

Category Archives: Development

Slopegraphs in Python – Log Scales & Spam Data Analysis

2012-06-07 – 17:32
Posted in Charts & Graphs, Development, Information Security, Programming, Python
Tagged slopegraph, table-chart, tufte
Leave a Comment

Given the focus on actual development of the PySlopegraph tool in most of the blog posts of late, folks may be wondering why an infosec/inforisk guy is obsessing so much on a tool and not talking security. Besides the fixation on filling a void and promoting an underused visualization tool, I do believe there is a place for slopegraphs in infosec data analysis and will utilize some data from McAfee’s recent Q1 2012 Threat Report [PDF] to illustrate how one might use slopegraphs in interpreting the “Spam Volume” data presented in the “Messaging Threats” section (pages 11 & 12 of the report).

The report shows individual graphs of spam volume per country from April of 2011 through March of 2012. Each individual graph conveys useful information, but I put together two slopegraphs that each show alternate and aggregate views which let you compare spam volume data relative to each country (versus just in-country).

When first doing this exploration, the scale problem reared it’s ugly head again since the United States is a huge spam outlier and causes the chart to be as tall as my youngest son when printed. I really wanted to show relative spam volume between countries as well as the increase or decrease between years in one chart and — after chatting with @maximumyin a bit — decided to test out using a log scale option for the charting (click for larger image):

This chart — Spam Volume by Country — instantly shows that:

overall volume has declined for most countries
two countries have remained steady
one country (Germany) has increased

The next chart – Spam Volume Percentage by Country — also needed to be presented on a log scale and has some equally compelling information:

Despite holding steady count-wise, the United States percentage of global spam actually increased and is joined by seven other countries, with Germany having the second largest percentage increase. Both charts present an opportunity to further explore why the values changed (since the best metrics are supposed to both inform and be actionable in some way).

I’m going to extract some more data from the McAfee report and some other security reports to show how slopegraphs can be used to interpret the data. Feedback on both the views and the use of the log scale would be greatly appreciated by general data scientists as well as those in the infosec community.

Slopegraphs in Python – Exploring Binning/Rounding

2012-06-07 – 12:17
Posted in Charts & Graphs, Development, Programming, Python
Tagged slopegraph, table-chart, tufte
Comments (1)

One of the last items for the 1.0 release is support for multiple columns of data. That will require some additional refactoring, so I’ve been procrastinating by exploring the recent “fudging” discovery. Despite claims to the contrary on other sites, there are more folks playing with slopegraphs than you might imagine. The inspiration for today’s installment comes from Jon Custer (@stuffisthings). He has a two part “Telling Stories with Data” series that does some exploration of export data with slopegraphs. In his “Slopegraph Strikes Back” post, Jon does a spiffy job discussing data visualization fundamentals and walks the reader through his re-design of a chart on commodities ranking, including a commentary on an aspect of slopegraphs that I’ve been noticing as I’ve been doing my exploring: the ‘scale’ problem (which I began to point out in the aforementioned “fudging” post).

The data set Jon is working with allows for a great exploration as to what works best when trying to convey a message with slopegraphs. I took the values from one of the tables he extracted:

and made a “raw” slopegraph from them (focusing on the “top 10”). The graphic won’t even come close to fitting in this post but you can grab the PDF of it and see how scale is the primary enemy of slopegraphs. It does show how gold and precious metal ores have skyrocketed from 1998 to 2007, but it’s hardly an engaging and easy to read visualization (unless you really like using your scroll wheel).

Jon grok’d this point, too, and decided to focus on the power law ranking and use the slopegraph to present the rate of change of each commodity:

While he didn’t “pull a Tufte” and just include values without caveat (see left & right 90° side labels), I still believe that there needs to be either increased annotation or the inclusion of base tabular data. Using my PySlopegraph code (forgot to mention the name change), I worked up a version of Jon’s visualization that I believe provides a clean, honest view of the data (click for larger view):

Because the chart is still based on the percentages that are fairly precise:

"Coconuts, Brazil nuts, cashews",17.93,0.93

```
Coffee,12.93,3.91
```
```
Fish,7.89,5.04
```
```
Tobacco,7.25,3.19
```
```
Gold,6.62,18.63
```
```
Tea,4.14,1.32
```
```
Cotton,4.01,1.36
```
```
Cloves,3.58,0.29
```
```
Diamonds,3.44,0.58
```
```
Mounted stones,2.44,1.5
```
```
Vegetables,1.61,1.73
```
```
Wheat,0.54,1.38
```
```
"Precious metal ores",0,6.76
```

I finally added an option to the PySlopegraph configuration file for rounding (NOTE: rounding != true binning). If you add the “round_precision” option with a value that supports Python’s round function’s little-known second parameter (arbitrary positional rounding), you can have the values round to decimal or tens/hundreds/etc places which will help with scaling issues, but will also group items (in ways that you may not have originally intended). For this chart, if we use a value of “1” (first decimal rounding precision…use negative values for rounding on the whole integer side of the decimal) it’s still unreadable due to the scale it imposes by that precision, so I ended up using the nearest whole integer rounding option (value of “0”) and also included the table of actual values, along with annotating the “rate of change” nature of the slopes.

This (again) defeats the “no wasted ink (pixels?)” component of Tufte’s original creation, but I believe it’s necessary for some types of slopegraphs to ensure the chart can stand on it’s own. I’m definitely becoming more convinced that many slopegraphs are more suited for an interactive visualization where you can encode more information in rollovers/popups/etc plus allow for switching of view from percentage, power-law ranking or raw numeric comparison.

For those interested in playing with this particular data set, it’ll be included in the next github code push, which will also include the rounding feature.

Slopegraphs in Python – Slope Colors

2012-06-05 – 22:43
Posted in Charts & Graphs, Development, Programming, Python
Tagged slopegraph, table-chart, tufte
Comments (1)

As the codebase gets closer to the 1.0 stretch we now have the addition of slope colors for when values go up/down or remain constant between points. The code still only handles two columns of data, but the intent is for each segment to also be colored appropriately (up/down/same) in a multi-column layout.

I was scanning for ‘slopegraph’ again via a few search engines and came across Chris Conley’s (@ResearchChat) Education and Health Care – Using Slopegraphs to Understand Complex Systems. I really like what Chris has done with the slopegraph formatting and copied the LINH data example to the project. As you can see, Chris came up with a pretty neat way to handle the overlapping data/label issue (and one which I may “borrow” when expanding my slopegraph generator):

Since Chris used colors, it seemed like a fitting example to use to show off the newest feature of the slopegraph code. Here’s the output for the same data with my implementation.

Both “slope_up_color” & “slope_down_color” (Lines 23-24) control the slope color.

This example also showed that the ‘header’ processing needs some tweaking. The code currently assumes the header label width will be less than or equal to the width of the data labels. We’ll need to do some bounding box fitting and/or canvas expansion to enable more robust header text formatting.

Given the minor tweak, no code inclusion here but yet-another handy link to github.

```
{
```
```
 
```
```
"label_font_family" : "Arial Narrow",
```
```
"label_font_size" : "9",
```
```
 
```
```
"header_font_family" : "Arial Narrow",
```
```
"header_font_size" : "10",
```
```
 
```
```
"x_margin" : "20",
```
```
"y_margin" : "30",
```
```
 
```
```
"line_width" : "1.0",
```
```
 
```
```
"slope_length" : "300",
```
```
 
```

"labels" : [ "# Below Average Indicators", "# Above Average Indicators" ],

```
 
```
```
"header_color" : "000000",
```
```
"background_color" : "FFFFFF",
```
```
"label_color" : "111111",
```
```
"value_color" : "999999",
```
```
"slope_color" : "222222",
```
```
"slope_up_color" : "B0B465",
```
```
"slope_down_color" : "A13E52",
```
```
 
```
```
"value_format_string" : "%2d",
```
```
 
```
```
"input" : "examples/linh.csv",
```
```
"output" : "examples/output/linh",
```
```
"format" : "pdf",
```
```
 
```

"description" : "2011 report from ICES; LINH Indicators",

"source" : "http://cconley.ca/2011/07/18/education-and-health-care-using-slopegraphs-to-understand-complex-systems/"

```
 
```
```
}
```

Slopegraphs in Python – More Output Tweaks

2012-06-05 – 12:12
Posted in Charts & Graphs, Development, Programming, Python
Tagged slopegraph, table-chart, tufte
Leave a Comment

The best way to explain this release will be to walk you through an updated configuration file:

```
{
```
```
 
```
```
"label_font_family" : "Palatino",
```
```
"label_font_size" : "9",
```
```
 
```
```
"header_font_family" : "Palatino",
```
```
"header_font_size" : "10",
```
```
 
```
```
"x_margin" : "20",
```
```
"y_margin" : "30",
```
```
 
```
```
"line_width" : "0.5",
```
```
 
```
```
"slope_length" : "150",
```
```
 
```
```
"labels" : [ "1970", "1979" ],
```
```
 
```
```
"header_color" : "000000",
```
```
"background_color" : "FFFFFF",
```
```
"label_color" : "111111",
```
```
"value_color" : "999999",
```
```
"slope_color" : "AAAAAA",
```
```
 
```
```
"value_format_string" : "%2d",
```
```
 
```
```
"input" : "receipts.csv",
```
```
"output" : "receipts",
```
```
"format" : "svg",
```
```
 
```

"description" : "Current Receipts of Government as a Percentage of Gross Domestic Product, 1970 & 1979",

"source" : "Tufte, Edward. The Visual Display of Quantitative Information. Cheshire, Connecticut: Graphics Press; 1983; p. 158"

```
 
```
```
}
```

I added the ability to include column headers and separated the font specifications for both the column data/labels and the headers (Lines 2-7). You’re not required to use headers, so just leave out the header font specification and the “labels” option (Line 16) if you don’t want them (it keys off of the font spec, tho). You can also color headers via the “header_color” option (line 18).

If you use the keyword “transparent” for the “background_color” config option (Line 19, tho it’s not transparent in this example) it will leave out the fill, which is useful for blog posts or embedding in other documents. Works best for PNG & PDF output.

If you want to use a different value for width of the space for the slopelines, you can tweak this via the “slope_length” option (Line 14). This is setting the stage for multi-column slopegraphs.

When exchanging some communications with @jayjacobs regarding slopegraphs and seeing his spiffy use of them for incident data anlysis, it became readily apparent that I needed to include a way of formatting the column data values, so there’s a “value_format_string” option, now, that works with Pythonic sprintf formats.

Finally, I added “description” and “source” options that the code does not yet process, but allows for documenting the configuration a bit, since there’s no good way to embed comments in a JSON-format configuration file.

As always, the code’s up on github and also below:

```
import csv
```
```
import cairo
```
```
import argparse
```
```
import json
```
```
 
```
```
def split(input, size):
```

    return [input[start:start+size] for start in range(0, len(input), size)]

```
 
```
```
class Slopegraph:
```
```
 
```
```
    starts = {} # starting "points"
```
```
    ends = {} # ending "points"
```

    pairs = [] # base pair array for the final plotting

```
 
```
```
    def readCSV(self, filename):
```
```
 
```

        slopeReader = csv.reader(open(filename, 'rb'), delimiter=',', quotechar='"')

```
 
```
```
        for row in slopeReader:
```
```
 
```

            # add chosen values (need start/end for each CSV row) to the final plotting array.

```
 
```
```
            lab = row[0] # label
```

            beg = float(row[1]) # left vals

            end = float(row[2]) # right vals

```
 
```

            self.pairs.append( (float(beg), float(end)) )

```
 
```

            # combine labels of common values into one string

```
 
```
```
            if beg in self.starts:
```

                self.starts[beg] = self.starts[beg] + "; " + lab

```
            else:
```
```
                self.starts[beg] = lab
```
```
 
```
```
 
```
```
            if end in self.ends:
```

                self.ends[end] = self.ends[end] + "; " + lab

```
            else:
```
```
                self.ends[end] = lab
```
```
 
```
```
 
```
```
    def sortKeys(self):
```
```
 
```

        # sort all the values (in the event the CSV wasn't) so

        # we can determine the smallest increment we need to use

        # when stacking the labels and plotting points

```
 
```

        self.startSorted = [(k, self.starts[k]) for k in sorted(self.starts)]

        self.endSorted = [(k, self.ends[k]) for k in sorted(self.ends)]

```
 
```

        self.startKeys = sorted(self.starts.keys())

        self.delta = max(self.startSorted)

        for i in range(len(self.startKeys)):

            if (i+1 <= len(self.startKeys)-1):

                currDelta = float(self.startKeys[i+1]) - float(self.startKeys[i])

                if (currDelta < self.delta):

                    self.delta = currDelta

```
 
```

        self.endKeys = sorted(self.ends.keys())

        for i in range(len(self.endKeys)):

            if (i+1 <= len(self.endKeys)-1):

                currDelta = float(self.endKeys[i+1]) - float(self.endKeys[i])

                if (currDelta < self.delta):

                    self.delta = currDelta

```
 
```
```
 
```
```
    def findExtremes(self):
```
```
 
```

        # we also need to find the absolute min & max values

        # so we know how to scale the plots

```
 
```

        self.lowest = min(self.startKeys)

        if (min(self.endKeys) < self.lowest) : self.lowest = min(self.endKeys)

```
 
```

        self.highest = max(self.startKeys)

        if (max(self.endKeys) > self.highest) : self.highest = max(self.endKeys)

```
 
```
```
        self.delta = float(self.delta)
```

        self.lowest = float(self.lowest)

        self.highest = float(self.highest)

```
 
```
```
 
```

    def calculateExtents(self, filename, format, valueFormatString):

```
 
```
```
        if (format == "pdf"):
```

            surface = cairo.PDFSurface (filename, 8.5*72, 11*72)

```
        elif (format == "ps"):
```

            surface = cairo.PSSurface(filename, 8.5*72, 11*72)

```
            surface.set_eps(True)
```
```
        elif (format == "svg"):
```

            surface = cairo.SVGSurface (filename, 8.5*72, 11*72)

```
        elif (format == "png"):
```

            surface = cairo.ImageSurface (cairo.FORMAT_ARGB32, int(8.5*72), int(11*72))

```
        else:
```

            surface = cairo.PDFSurface (filename, 8.5*72, 11*72)

```
 
```
```
        cr = cairo.Context(surface)
```
```
        cr.save()
```

        cr.select_font_face(self.LABEL_FONT_FAMILY, cairo.FONT_SLANT_NORMAL, cairo.FONT_WEIGHT_NORMAL)

        cr.set_font_size(self.LABEL_FONT_SIZE)

        cr.set_line_width(self.LINE_WIDTH)

```
 
```

        # find the *real* maximum label width (not just based on number of chars)

```
 
```
```
        maxLabelWidth = 0
```
```
        maxNumWidth = 0
```
```
 
```

        for k in sorted(self.startKeys):

```
            s1 = self.starts[k]
```

            xbearing, ybearing, self.sWidth, self.sHeight, xadvance, yadvance = (cr.text_extents(s1))

            if (self.sWidth > maxLabelWidth) : maxLabelWidth = self.sWidth

            xbearing, ybearing, self.startMaxLabelWidth, startMaxLabelHeight, xadvance, yadvance = (cr.text_extents(valueFormatString % (k)))

            if (self.startMaxLabelWidth > maxNumWidth) : maxNumWidth = self.startMaxLabelWidth

```
 
```
```
        self.sWidth = maxLabelWidth
```

        self.startMaxLabelWidth = maxNumWidth

```
 
```
```
        maxLabelWidth = 0
```
```
        maxNumWidth = 0
```
```
 
```
```
        for k in sorted(self.endKeys):
```
```
            e1 = self.ends[k]
```

            xbearing, ybearing, self.eWidth, eHeight, xadvance, yadvance = (cr.text_extents(e1))

            if (self.eWidth > maxLabelWidth) : maxLabelWidth = self.eWidth

            xbearing, ybearing, self.endMaxLabelWidth, endMaxLabelHeight, xadvance, yadvance = (cr.text_extents(valueFormatString % (k)))

            if (self.endMaxLabelWidth > maxNumWidth) : maxNumWidth = self.endMaxLabelWidth

```
 
```
```
        self.eWidth = maxLabelWidth
```

        self.endMaxLabelWidth = maxNumWidth

```
 
```
```
        cr.restore()
```
```
        cr.show_page()
```
```
        surface.finish()
```
```
 
```

        self.width = self.X_MARGIN + self.sWidth + self.SPACE_WIDTH + self.startMaxLabelWidth + self.SPACE_WIDTH + self.SLOPE_LENGTH + self.SPACE_WIDTH + self.endMaxLabelWidth + self.SPACE_WIDTH + self.eWidth + self.X_MARGIN ;

        self.height = (self.Y_MARGIN * 2) + (((self.highest - self.lowest) / self.delta) * self.LINE_HEIGHT)

```
 
```
```
        self.HEADER_SPACE = 0.0
```

        if (self.HEADER_FONT_FAMILY != None):

            self.HEADER_SPACE = self.HEADER_FONT_SIZE + 2*self.LINE_HEIGHT

            self.height += self.HEADER_SPACE

```
 
```
```
 
```

    def makeSlopegraph(self, filename, config):

```
 
```

        (lab_r,lab_g,lab_b) = split(config["label_color"],2)

```
        LAB_R = (int(lab_r, 16)/255.0)
```
```
        LAB_G = (int(lab_g, 16)/255.0)
```
```
        LAB_B = (int(lab_b, 16)/255.0)
```
```
 
```

        (val_r,val_g,val_b) = split(config["value_color"],2)

```
        VAL_R = (int(val_r, 16)/255.0)
```
```
        VAL_G = (int(val_g, 16)/255.0)
```
```
        VAL_B = (int(val_b, 16)/255.0)
```
```
 
```

        (line_r,line_g,line_b) = split(config["slope_color"],2)

        LINE_R = (int(line_r, 16)/255.0)

        LINE_G = (int(line_g, 16)/255.0)

        LINE_B = (int(line_b, 16)/255.0)

```
 
```

        if (config["background_color"] != "transparent"):

            (bg_r,bg_g,bg_b) = split(config["background_color"],2)

            BG_R = (int(bg_r, 16)/255.0)

            BG_G = (int(bg_g, 16)/255.0)

            BG_B = (int(bg_b, 16)/255.0)

```
 
```
```
        if (config['format'] == "pdf"):
```

            surface = cairo.PDFSurface (filename, self.width, self.height)

        elif (config['format'] == "ps"):

            surface = cairo.PSSurface(filename, self.width, self.height)

```
            surface.set_eps(True)
```

        elif (config['format'] == "svg"):

            surface = cairo.SVGSurface (filename, self.width, self.height)

        elif (config['format'] == "png"):

            surface = cairo.ImageSurface (cairo.FORMAT_ARGB32, int(self.width), int(self.height))

```
        else:
```

            surface = cairo.PDFSurface (filename, self.width, self.height)

```
 
```
```
        cr = cairo.Context(surface)
```
```
 
```
```
        cr.save()
```
```
 
```

        cr.set_line_width(self.LINE_WIDTH)

```
 
```

        if (config["background_color"] != "transparent"):

            cr.set_source_rgb(BG_R,BG_G,BG_B)

            cr.rectangle(0,0,self.width,self.height)

```
            cr.fill()
```
```
 
```
```
        # draw headers (if present)
```
```
 
```

        if (self.HEADER_FONT_FAMILY != None):

```
 
```

            (header_r,header_g,header_b) = split(config["header_color"],2)

            HEADER_R = (int(header_r, 16)/255.0)

            HEADER_G = (int(header_g, 16)/255.0)

            HEADER_B = (int(header_b, 16)/255.0)

```
 
```
```
            cr.save()
```
```
 
```

            cr.select_font_face(self.HEADER_FONT_FAMILY, cairo.FONT_SLANT_NORMAL, cairo.FONT_WEIGHT_BOLD)

            cr.set_font_size(self.HEADER_FONT_SIZE)

            cr.set_source_rgb(HEADER_R,HEADER_G,HEADER_B)

```
 
```

            xbearing, ybearing, hWidth, hHeight, xadvance, yadvance = (cr.text_extents(config["labels"][0]))

            cr.move_to(self.X_MARGIN + self.sWidth - hWidth, self.Y_MARGIN + self.HEADER_FONT_SIZE)

            cr.show_text(config["labels"][0])

```
 
```

            xbearing, ybearing, hWidth, hHeight, xadvance, yadvance = (cr.text_extents(config["labels"][1]))

            cr.move_to(self.width - self.X_MARGIN - self.SPACE_WIDTH - self.eWidth, self.Y_MARGIN + self.HEADER_FONT_SIZE)

            cr.show_text(config["labels"][1])

```
 
```
```
            cr.stroke()
```
```
 
```
```
            cr.restore()
```
```
 
```

        # draw start labels at the correct positions

```
 
```

        cr.select_font_face(self.LABEL_FONT_FAMILY, cairo.FONT_SLANT_NORMAL, cairo.FONT_WEIGHT_NORMAL)

        cr.set_font_size(self.LABEL_FONT_SIZE)

```
 
```

        valueFormatString = config["value_format_string"]

```
 
```

        for k in sorted(self.startKeys):

```
 
```
```
            val = float(k)
```
```
            label = self.starts[k]
```

            xbearing, ybearing, lWidth, lHeight, xadvance, yadvance = (cr.text_extents(label))

            xbearing, ybearing, kWidth, kHeight, xadvance, yadvance = (cr.text_extents(valueFormatString % (val)))

```
 
```

            cr.set_source_rgb(LAB_R,LAB_G,LAB_B)

            cr.move_to(self.X_MARGIN + (self.sWidth - lWidth), self.Y_MARGIN + self.HEADER_SPACE + (self.highest - val) * self.LINE_HEIGHT * (1/self.delta))

```
            cr.show_text(label)
```
```
 
```

            cr.set_source_rgb(VAL_R,VAL_G,VAL_B)

            cr.move_to(self.X_MARGIN + self.sWidth + self.SPACE_WIDTH + (self.startMaxLabelWidth - kWidth), self.Y_MARGIN + self.HEADER_SPACE + (self.highest - val) * self.LINE_HEIGHT * (1/self.delta))

            cr.show_text(valueFormatString % (val))

```
 
```
```
            cr.stroke()
```
```
 
```

        # draw end labels at the correct positions

```
 
```
```
        for k in sorted(self.endKeys):
```
```
 
```
```
            val = float(k)
```
```
            label = self.ends[k]
```

            xbearing, ybearing, lWidth, lHeight, xadvance, yadvance = (cr.text_extents(label))

```
 
```

            cr.set_source_rgb(VAL_R,VAL_G,VAL_B)

            cr.move_to(self.width - self.X_MARGIN - self.SPACE_WIDTH - self.eWidth - self.SPACE_WIDTH - self.endMaxLabelWidth, self.Y_MARGIN + self.HEADER_SPACE + (self.highest - val) * self.LINE_HEIGHT * (1/self.delta))

            cr.show_text(valueFormatString % (val))

```
 
```

            cr.set_source_rgb(LAB_R,LAB_G,LAB_B)

            cr.move_to(self.width - self.X_MARGIN - self.SPACE_WIDTH - self.eWidth, self.Y_MARGIN + self.HEADER_SPACE + (self.highest - val) * self.LINE_HEIGHT * (1/self.delta))

```
            cr.show_text(label)
```
```
 
```
```
            cr.stroke()
```
```
 
```
```
        # do the actual plotting
```
```
 
```

        cr.set_line_width(self.LINE_WIDTH)

        cr.set_source_rgb(LINE_R, LINE_G, LINE_B)

```
 
```
```
        for s1,e1 in self.pairs:
```

            cr.move_to(self.X_MARGIN + self.sWidth + self.SPACE_WIDTH + self.startMaxLabelWidth + self.LINE_START_DELTA, self.Y_MARGIN + self.HEADER_SPACE + (self.highest - s1) * self.LINE_HEIGHT * (1/self.delta) - self.LINE_HEIGHT/4)

            cr.line_to(self.width - self.X_MARGIN - self.eWidth - self.SPACE_WIDTH - self.endMaxLabelWidth - self.LINE_START_DELTA, self.Y_MARGIN + self.HEADER_SPACE + (self.highest - e1) * self.LINE_HEIGHT * (1/self.delta) - self.LINE_HEIGHT/4)

```
            cr.stroke()
```
```
 
```
```
        cr.restore()
```
```
        cr.show_page()
```
```
 
```
```
        if (config['format'] == "png"):
```

            surface.write_to_png(filename)

```
 
```
```
        surface.finish()    
```
```
 
```
```
    def __init__(self, config):
```
```
 
```

        # since some methods need these, make them local to the class

```
 
```

        self.LABEL_FONT_FAMILY = config["label_font_family"]

        self.LABEL_FONT_SIZE = float(config["label_font_size"])

```
 
```

        if "header_font_family" in config:

            self.HEADER_FONT_FAMILY = config["header_font_family"]

            self.HEADER_FONT_SIZE = float(config["header_font_size"])

```
        else:
```

            self.HEADER_FONT_FAMILY = None

            self.HEADER_FONT_SIZE = None

```
 
```

        self.X_MARGIN = float(config["x_margin"])

        self.Y_MARGIN = float(config["y_margin"])

        self.LINE_WIDTH = float(config["line_width"])

```
 
```
```
        if "slope_length" in config:
```

            self.SLOPE_LENGTH = float(config["slope_length"])

```
        else:
```
```
            self.SLOPE_LENGTH = 300
```
```
 
```

        self.SPACE_WIDTH = self.LABEL_FONT_SIZE / 2.0

        self.LINE_HEIGHT = self.LABEL_FONT_SIZE + (self.LABEL_FONT_SIZE / 2.0)

        self.LINE_START_DELTA = 1.5*self.SPACE_WIDTH

```
 
```

        OUTPUT_FILE = config["output"] + "." + config["format"]

```
 
```

        # process the values & make the slopegraph

```
 
```
```
        self.readCSV(config["input"])
```
```
        self.sortKeys()
```
```
        self.findExtremes()
```

        self.calculateExtents(OUTPUT_FILE, config["format"], config["value_format_string"])

        self.makeSlopegraph(OUTPUT_FILE, config)

```
 
```
```
 
```
```
def main():
```
```
 
```

    parser = argparse.ArgumentParser(description="Creates a slopegraph from a CSV source")

    parser.add_argument("--config",required=True,

                    help="config file name to use for  slopegraph creation",)

```
    args = parser.parse_args()
```
```
 
```
```
    if args.config:
```
```
 
```
```
        json_data = open(args.config)
```
```
        config = json.load(json_data)
```
```
        json_data.close()
```
```
 
```
```
        Slopegraph(config)
```
```
 
```
```
    return(0)
```
```
 
```
```
if __name__ == "__main__":
```
```
    main()
```

Slopegraphs – The Quintessential Tufte GDP Example Revisited

2012-06-05 – 11:17
Posted in Charts & Graphs, Development, Programming, Python
Tagged sloepgraph
Comments (1)

If you’re even remotely familiar with slopegraphs, then you’ll recognize Tufte’s classic 1970-1979 GDP chart example (click for larger version):

What you may not notice initially, however, is that Tufte — well — cheated. Yes, I said it. Cheated. I can show it by zooming into the “Belgium/Canada/Finland” grouping in the left column:

The value “35.2” shows up twice and is actually equivalent (on the scale) to the “34.9” value. If the numbers were actually representative of the positional elements, the slopegraph should, in fact, look more like this (you’ll need to click on the image to view it, as it’s big):

You may want to even grab the PDF version as it’s a bit more legible if you’re going to peruse it for any length of time.

One of the strengths of the slopegraph chart is to have every bit of ink be useful. In the GDP chart, however, while the slopes are accurate (as you’ll see below) you’re a bit misled by the juxtaposition of similar values on either column and the use of detailed values when clearly the chart data has been rounded. Here’s more of what the actual chart represents (click the graphic for larger version):

If you compare this version (made with my code) with Tufte’s example, you’ll see the similarity (except that I group common values on a single line). I only discovered this after inputting the GDP numbers from Tufte’s chart into an example configuration that I could use with my code.

The fact that Tufte rounded (perhaps “binned” may be a better word) the values does not make the results less useful nor make the “Britain” outlier any less significant (it’s even more pronounced in my huge, “unbinned” version). But, his graphic does seem misleading (to me) knowing that they are rounded values. I believe something like this would have been a more honest reproduction,even though it adds ink (click the graphic for larger version):

I believe a cue about the rounding and the inclusion of the actual values make this a more honest chart.

As a result of this digging, I’m going to be working on some “binning” code for my slopegraph implementation. I’m anticipating having to use a more interactive model when it comes to some data sources (think hover/zoom for data detail) and will probably look at using Raphael or D3 for them.

I’m curious as to what you think of the “undeclared rounding” and how you’d present the information given the obvious need for some type of “binning”.

Slopegraphs in Python – The Great Refactor

2012-06-02 – 18:47
Posted in Charts & Graphs, Development, Python
Tagged slopegraph, table-chart, tufte
Leave a Comment

Despite being on holiday, I had a spare hour to refactor the code (@mrshrbrmstr was joining the 1% in the hotel spa). It’s up on github and now sports a spiffy JSON-format config file. You now must execute the slopegraph.py script with a “--config FILENAME” argument. The configuration file lets you specify the “theme” as well as the input file and output format (you can only use PDF for the moment).

Here’s a sample config file included in the github push (there’s another one there too):

```
{
```
```
 
```
```
"font_family" : "Palatino",
```
```
"font_size" : "20",
```
```
 
```
```
"x_margin" : "20",
```
```
"y_margin" : "30",
```
```
 
```
```
"line_width" : "0.5",
```
```
 
```
```
"background_color" : "DEC299",
```
```
"label_color" : "687D64",
```
```
"value_color" : "949258",
```
```
"slope_color" : "61514C",
```
```
 
```
```
"value_format_string" : "%2d",
```
```
 
```
```
"input" : "television.csv",
```
```
"output" : "television",
```
```
"format" : "pdf"
```
```
 
```
```
}
```

Included in the refactor is the ability to use a sprintf-like format string for the label value output to make the slopegraphs a tad prettier. Also included with the refactor is a new limitation of the CSV file requiring a

"LABEL, VALUE, VALUE"

format in preparation for support for multiple columns. As @jayjacobs said to me, it’s easy to reformat data into the CSV file format, and, he’s right (as usual).

Plans for the next revision include:

Specifying a transparent background
Specifying PDF|PS|SVG|PNG format output
Allowing for an arbitrary number of columns for the slopegraph
Optional column labels as well as slopepgraph title (with themeing)
Line color change by slope up/same/down value (will most likely be pushed out, tho)

Here’s the whole source:

```
import csv
```
```
import cairo
```
```
import argparse
```
```
import json
```
```
 
```
```
def split(input, size):
```

	return [input[start:start+size] for start in range(0, len(input), size)]

```
 
```
```
class Slopegraph:
```
```
 
```
```
	SLOPEGRAPH_CANVAS_SIZE = 300
```
```
 
```
```
	starts = {} # starting "points"
```
```
	ends = {} # ending "points"
```

	pairs = [] # base pair array for the final plotting

```
 
```
```
	def readCSV(self, filename):
```
```
 
```

		slopeReader = csv.reader(open(filename, 'rb'), delimiter=',', quotechar='"')

```
 
```
```
		for row in slopeReader:
```
```
 
```

			# add chosen values (need start/end for each CSV row) to the final plotting array.

```
 
```
```
			lab = row[0] # label
```
```
			beg = float(row[1]) # left vals
```
```
			end = float(row[2]) # right vals
```
```
 
```

			self.pairs.append( (float(beg), float(end)) )

```
 
```

			# combine labels of common values into one string

```
 
```
```
			if beg in self.starts:
```

				self.starts[beg] = self.starts[beg] + "; " + lab

```
			else:
```
```
				self.starts[beg] = lab
```
```
 
```
```
 
```
```
			if end in self.ends:
```

				self.ends[end] = self.ends[end] + "; " + lab

```
			else:
```
```
				self.ends[end] = lab
```
```
 
```
```
 
```
```
	def sortKeys(self):
```
```
 
```

		# sort all the values (in the event the CSV wasn't) so

		# we can determine the smallest increment we need to use

		# when stacking the labels and plotting points

```
 
```

		self.startSorted = [(k, self.starts[k]) for k in sorted(self.starts)]

		self.endSorted = [(k, self.ends[k]) for k in sorted(self.ends)]

```
 
```

		self.startKeys = sorted(self.starts.keys())

```
		self.delta = max(self.startSorted)
```
```
		for i in range(len(self.startKeys)):
```
```
			if (i+1 <= len(self.startKeys)-1):
```

				currDelta = float(self.startKeys[i+1]) - float(self.startKeys[i])

```
				if (currDelta < self.delta):
```
```
					self.delta = currDelta
```
```
 
```

		self.endKeys = sorted(self.ends.keys())

```
		for i in range(len(self.endKeys)):
```
```
			if (i+1 <= len(self.endKeys)-1):
```

				currDelta = float(self.endKeys[i+1]) - float(self.endKeys[i])

```
				if (currDelta < self.delta):
```
```
					self.delta = currDelta
```
```
 
```
```
 
```
```
	def findExtremes(self):
```
```
 
```

		# we also need to find the absolute min & max values

```
		# so we know how to scale the plots
```
```
 
```
```
		self.lowest = min(self.startKeys)
```

		if (min(self.endKeys) < self.lowest) : self.lowest = min(self.endKeys)

```
 
```
```
		self.highest = max(self.startKeys)
```

		if (max(self.endKeys) > self.highest) : self.highest = max(self.endKeys)

```
 
```
```
		self.delta = float(self.delta)
```
```
		self.lowest = float(self.lowest)
```
```
		self.highest = float(self.highest)
```
```
 
```
```
 
```

	def calculateExtents(self, filename, format, valueFormatString):

```
 
```

		surface = cairo.PDFSurface (filename, 8.5*72, 11*72)

```
		cr = cairo.Context (surface)
```
```
		cr.save()
```

		cr.select_font_face(self.FONT_FAMILY, cairo.FONT_SLANT_NORMAL, cairo.FONT_WEIGHT_NORMAL)

```
		cr.set_font_size(self.FONT_SIZE)
```
```
		cr.set_line_width(self.LINE_WIDTH)
```
```
 
```

		# find the *real* maximum label width (not just based on number of chars)

```
 
```
```
		maxLabelWidth = 0
```
```
		maxNumWidth = 0
```
```
 
```
```
		for k in sorted(self.startKeys):
```
```
			s1 = self.starts[k]
```

			xbearing, ybearing, self.sWidth, self.sHeight, xadvance, yadvance = (cr.text_extents(s1))

			if (self.sWidth > maxLabelWidth) : maxLabelWidth = self.sWidth

			xbearing, ybearing, self.startMaxLabelWidth, startMaxLabelHeight, xadvance, yadvance = (cr.text_extents(valueFormatString % (k)))

			if (self.startMaxLabelWidth > maxNumWidth) : maxNumWidth = self.startMaxLabelWidth

```
 
```
```
		self.sWidth = maxLabelWidth
```
```
		self.startMaxLabelWidth = maxNumWidth
```
```
 
```
```
		maxLabelWidth = 0
```
```
		maxNumWidth = 0
```
```
 
```
```
		for k in sorted(self.endKeys):
```
```
			e1 = self.ends[k]
```

			xbearing, ybearing, self.eWidth, eHeight, xadvance, yadvance = (cr.text_extents(e1))

			if (self.eWidth > maxLabelWidth) : maxLabelWidth = self.eWidth

			xbearing, ybearing, self.endMaxLabelWidth, endMaxLabelHeight, xadvance, yadvance = (cr.text_extents(valueFormatString % (k)))

			if (self.endMaxLabelWidth > maxNumWidth) : maxNumWidth = self.endMaxLabelWidth

```
 
```
```
		self.eWidth = maxLabelWidth
```
```
		self.endMaxLabelWidth = maxNumWidth	
```
```
 
```
```
		cr.restore()
```
```
		cr.show_page()
```
```
		surface.finish()
```
```
 
```

		self.width = self.X_MARGIN + self.sWidth + self.SPACE_WIDTH + self.startMaxLabelWidth + self.SPACE_WIDTH + self.SLOPEGRAPH_CANVAS_SIZE + self.SPACE_WIDTH + self.endMaxLabelWidth + self.SPACE_WIDTH + self.eWidth + self.X_MARGIN ;

		self.height = (self.Y_MARGIN * 2) + (((self.highest - self.lowest) / self.delta) * self.LINE_HEIGHT)

```
 
```
```
 
```

	def makeSlopegraph(self, filename, config):

```
 
```

		(lab_r,lab_g,lab_b) = split(config["label_color"],2)

		(val_r,val_g,val_b) = split(config["value_color"],2)

		(line_r,line_g,line_b) = split(config["slope_color"],2)

		(bg_r,bg_g,bg_b) = split(config["background_color"],2)

```
 
```
```
		LAB_R = (int(lab_r, 16)/255.0)
```
```
		LAB_G = (int(lab_g, 16)/255.0)
```
```
		LAB_B = (int(lab_b, 16)/255.0)
```
```
 
```
```
		VAL_R = (int(val_r, 16)/255.0)
```
```
		VAL_G = (int(val_g, 16)/255.0)
```
```
		VAL_B = (int(val_b, 16)/255.0)
```
```
 
```
```
		LINE_R = (int(line_r, 16)/255.0)
```
```
		LINE_G = (int(line_g, 16)/255.0)
```
```
		LINE_B = (int(line_b, 16)/255.0)
```
```
 
```
```
		BG_R = (int(bg_r, 16)/255.0)
```
```
		BG_G = (int(bg_g, 16)/255.0)
```
```
		BG_B = (int(bg_b, 16)/255.0)
```
```
 
```

		surface = cairo.PDFSurface (filename, self.width, self.height)

```
		cr = cairo.Context(surface)
```
```
 
```
```
		cr.save()
```
```
 
```

		cr.select_font_face(self.FONT_FAMILY, cairo.FONT_SLANT_NORMAL, cairo.FONT_WEIGHT_NORMAL)

```
		cr.set_font_size(self.FONT_SIZE)
```
```
 
```
```
		cr.set_line_width(self.LINE_WIDTH)
```
```
 
```
```
		cr.set_source_rgb(BG_R,BG_G,BG_B)
```

		cr.rectangle(0,0,self.width,self.height)

```
		cr.fill()
```
```
 
```

		# draw start labels at the correct positions

```
 
```

		valueFormatString = config["value_format_string"]

```
 
```
```
		for k in sorted(self.startKeys):
```
```
 
```
```
			val = float(k)
```
```
			label = self.starts[k]
```

			xbearing, ybearing, lWidth, lHeight, xadvance, yadvance = (cr.text_extents(label))

			xbearing, ybearing, kWidth, kHeight, xadvance, yadvance = (cr.text_extents(valueFormatString % (val)))

```
 
```
```
			cr.set_source_rgb(LAB_R,LAB_G,LAB_B)
```

			cr.move_to(self.X_MARGIN + (self.sWidth - lWidth), self.Y_MARGIN + (self.highest - val) * self.LINE_HEIGHT * (1/self.delta))

```
			cr.show_text(label)
```
```
 
```
```
			cr.set_source_rgb(VAL_R,VAL_G,VAL_B)
```

			cr.move_to(self.X_MARGIN + self.sWidth + self.SPACE_WIDTH + (self.startMaxLabelWidth - kWidth), self.Y_MARGIN + (self.highest - val) * self.LINE_HEIGHT * (1/self.delta))

			cr.show_text(valueFormatString % (val))

```
 
```
```
			cr.stroke()
```
```
 
```

		# draw end labels at the correct positions

```
 
```
```
		for k in sorted(self.endKeys):
```
```
 
```
```
			val = float(k)
```
```
			label = self.ends[k]
```

			xbearing, ybearing, lWidth, lHeight, xadvance, yadvance = (cr.text_extents(label))

```
 
```
```
			cr.set_source_rgb(VAL_R,VAL_G,VAL_B)
```

			cr.move_to(self.width - self.X_MARGIN - self.SPACE_WIDTH - self.eWidth - self.SPACE_WIDTH - self.endMaxLabelWidth, self.Y_MARGIN + (self.highest - val) * self.LINE_HEIGHT * (1/self.delta))

			cr.show_text(valueFormatString % (val))

```
 
```
```
			cr.set_source_rgb(LAB_R,LAB_G,LAB_B)
```

			cr.move_to(self.width - self.X_MARGIN - self.SPACE_WIDTH - self.eWidth, self.Y_MARGIN + (self.highest - val) * self.LINE_HEIGHT * (1/self.delta))

```
			cr.show_text(label)
```
```
 
```
```
			cr.stroke()
```
```
 
```
```
		# do the actual plotting
```
```
 
```
```
		cr.set_line_width(self.LINE_WIDTH)
```

		cr.set_source_rgb(LINE_R, LINE_G, LINE_B)

```
 
```
```
		for s1,e1 in self.pairs:
```

			cr.move_to(self.X_MARGIN + self.sWidth + self.SPACE_WIDTH + self.startMaxLabelWidth + self.LINE_START_DELTA, self.Y_MARGIN + (self.highest - s1) * self.LINE_HEIGHT * (1/self.delta) - self.LINE_HEIGHT/4)

			cr.line_to(self.width - self.X_MARGIN - self.eWidth - self.SPACE_WIDTH - self.endMaxLabelWidth - self.LINE_START_DELTA, self.Y_MARGIN + (self.highest - e1) * self.LINE_HEIGHT * (1/self.delta) - self.LINE_HEIGHT/4)

```
			cr.stroke()
```
```
 
```
```
		cr.restore()
```
```
		cr.show_page()
```
```
		surface.finish()	
```
```
 
```
```
 
```
```
	def __init__(self, config):
```
```
 
```

		# a couple methods need these so make them local to the class

```
 
```

		self.FONT_FAMILY = config["font_family"]

		self.LINE_WIDTH = float(config["line_width"])

		self.X_MARGIN = float(config["x_margin"])

		self.Y_MARGIN = float(config["y_margin"])

		self.FONT_SIZE = float(config["font_size"])

		self.SPACE_WIDTH = self.FONT_SIZE / 2.0

		self.LINE_HEIGHT = self.FONT_SIZE + (self.FONT_SIZE / 2.0)

		self.LINE_START_DELTA = 1.5*self.SPACE_WIDTH

```
 
```

		OUTPUT_FILE = config["output"] + "." + config["format"]

```
 
```

		# process the values & make the slopegraph

```
 
```
```
		self.readCSV(config["input"])
```
```
		self.sortKeys()
```
```
		self.findExtremes()
```

		self.calculateExtents(OUTPUT_FILE, config["format"], config["value_format_string"])

		self.makeSlopegraph(OUTPUT_FILE, config)

```
 
```
```
 
```
```
def main():
```
```
 
```

	parser = argparse.ArgumentParser(description="Creates a slopegraph from a CSV source")

	parser.add_argument("--config",required=True,

					help="config file name to use for  slopegraph creation",)

```
	args = parser.parse_args()
```
```
 
```
```
	if args.config:
```
```
 
```
```
		json_data = open(args.config)
```
```
		config = json.load(json_data)
```
```
		json_data.close()
```
```
 
```
```
		Slopegraph(config)
```
```
 
```
```
	return(0)
```
```
 
```
```
if __name__ == "__main__":
```
```
	main()
```

Slopegraphs in Python – Attention to Detail

2012-05-31 – 06:04
Posted in Charts & Graphs, Development, Python
Tagged table-chart, tufte
Leave a Comment

In the previous installment, a foundation was laid for “parameterizing” fonts, colors and overall slopegraph size. However, a big failing in all this code (up until now) was the reliance on character string length to determine label width. When working with fonts, the font metrics are more important since a lowercase ‘l’ will have a smaller font width than an uppercase ‘D’. So, while ‘llll’ is “longer” than ‘DDD’, it may not be wider (especially in a sans-serif font, but likely in any font):

To solve this problem, we need to use a temporary Cairo surface to compute font metrics for each label & value so we know what the maximum width of both for the starting and ending points. It’s a simple concept & calculation, but very important to ensure everything lines up well.

# find the *real* maximum label width (not just based on number of chars)

```
 
```
```
maxLabelWidth = 0
```
```
maxNumWidth = 0
```
```
 
```
```
for k in sorted(startKeys):
```
```
	s1 = starts[k]
```

	xbearing, ybearing, sWidth, sHeight, xadvance, yadvance = (cr.text_extents(s1))

	if (sWidth > maxLabelWidth) : maxLabelWidth = sWidth

	xbearing, ybearing, startMaxLabelWidth, startMaxLabelHeight, xadvance, yadvance = (cr.text_extents(str(k)))

	if (startMaxLabelWidth > maxNumWidth) : maxNumWidth = startMaxLabelWidth

```
 
```
```
sWidth = maxLabelWidth
```
```
startMaxLabelWidth = maxNumWidth
```
```
 
```
```
maxWidth = 0
```
```
maxNumWidth = 0
```
```
 
```
```
for k in sorted(endKeys):
```
```
	e1 = ends[k]
```

	xbearing, ybearing, eWidth, eHeight, xadvance, yadvance = (cr.text_extents(e1))

	if (eWidth > maxLabelWidth) : maxLabelWidth = eWidth

	xbearing, ybearing, endMaxLabelWidth, endMaxLabelHeight, xadvance, yadvance = (cr.text_extents(str(k)))

	if (endMaxLabelWidth > maxNumWidth) : maxNumWidth = endMaxLabelWidth

```
 
```
```
eWidth = maxLabelWidth
```
```
endMaxLabelWidth = maxNumWidth
```

I tossed some “anomalies” into the sample data set to show both how adaptable the vertical scale is as well as demonstrate the label alignments:

Updates are in github.

Slopegraphs In Python – colors, fonts & alignment

2012-05-29 – 21:49
Posted in Charts & Graphs, Development, Python
Tagged slopegraph, table-chart, tufte
Comments (1)

On the heels of last evening’s release of Slopegraphs in Python post comes some minor tweaks:

Complete alignment control of labels & and values
Colors (for background, lines, labels & values) — I picked a random pattern from Adobe’s Kuler
A font change (to prove width calculations work)

…and a new example slopegraph:

As promised, the latest revisions are in github.

Some notes for aspiring Python/Cairo hackers:

RGB colors are 0-1 in Cairo, so divide your 0-255 values by 255 to get the corresponding Cairo value and make sure you’re doing float arithmetic in Python
It turns out simple Cairo font rendering (ie. non-Pango) does not interpret well in Illustrator from a Cairo PDF surface, so if you do plan on post-processing the slopegraphs, use a postscript/EPS surface:
surface = cairo.PSSurface ("slopegraph.ps", width, height)
surface.set_eps(True)
(I’ll be incorporating an output option as this gets closer to 1.0)
If you do plan on using this at all, grab the Bitstream Vera Serif font (link goes to FontSquirrel, but you can find it almost everywhere as it’s free)