rud.is

Slopegraphs – The Quintessential Tufte GDP Example Revisited

2012-06-05 – 11:17
Posted in Charts & Graphs, Development, Programming, Python
Tagged sloepgraph
Comments (1)

If you’re even remotely familiar with slopegraphs, then you’ll recognize Tufte’s classic 1970-1979 GDP chart example (click for larger version):

What you may not notice initially, however, is that Tufte — well — cheated. Yes, I said it. Cheated. I can show it by zooming into the “Belgium/Canada/Finland” grouping in the left column:

The value “35.2” shows up twice and is actually equivalent (on the scale) to the “34.9” value. If the numbers were actually representative of the positional elements, the slopegraph should, in fact, look more like this (you’ll need to click on the image to view it, as it’s big):

You may want to even grab the PDF version as it’s a bit more legible if you’re going to peruse it for any length of time.

One of the strengths of the slopegraph chart is to have every bit of ink be useful. In the GDP chart, however, while the slopes are accurate (as you’ll see below) you’re a bit misled by the juxtaposition of similar values on either column and the use of detailed values when clearly the chart data has been rounded. Here’s more of what the actual chart represents (click the graphic for larger version):

If you compare this version (made with my code) with Tufte’s example, you’ll see the similarity (except that I group common values on a single line). I only discovered this after inputting the GDP numbers from Tufte’s chart into an example configuration that I could use with my code.

The fact that Tufte rounded (perhaps “binned” may be a better word) the values does not make the results less useful nor make the “Britain” outlier any less significant (it’s even more pronounced in my huge, “unbinned” version). But, his graphic does seem misleading (to me) knowing that they are rounded values. I believe something like this would have been a more honest reproduction,even though it adds ink (click the graphic for larger version):

I believe a cue about the rounding and the inclusion of the actual values make this a more honest chart.

As a result of this digging, I’m going to be working on some “binning” code for my slopegraph implementation. I’m anticipating having to use a more interactive model when it comes to some data sources (think hover/zoom for data detail) and will probably look at using Raphael or D3 for them.

I’m curious as to what you think of the “undeclared rounding” and how you’d present the information given the obvious need for some type of “binning”.

Slopegraphs in Python – The Great Refactor

2012-06-02 – 18:47
Posted in Charts & Graphs, Development, Python
Tagged slopegraph, table-chart, tufte
Leave a Comment

Despite being on holiday, I had a spare hour to refactor the code (@mrshrbrmstr was joining the 1% in the hotel spa). It’s up on github and now sports a spiffy JSON-format config file. You now must execute the slopegraph.py script with a “--config FILENAME” argument. The configuration file lets you specify the “theme” as well as the input file and output format (you can only use PDF for the moment).

Here’s a sample config file included in the github push (there’s another one there too):

```
{
```
```
 
```
```
"font_family" : "Palatino",
```
```
"font_size" : "20",
```
```
 
```
```
"x_margin" : "20",
```
```
"y_margin" : "30",
```
```
 
```
```
"line_width" : "0.5",
```
```
 
```
```
"background_color" : "DEC299",
```
```
"label_color" : "687D64",
```
```
"value_color" : "949258",
```
```
"slope_color" : "61514C",
```
```
 
```
```
"value_format_string" : "%2d",
```
```
 
```
```
"input" : "television.csv",
```
```
"output" : "television",
```
```
"format" : "pdf"
```
```
 
```
```
}
```

Included in the refactor is the ability to use a sprintf-like format string for the label value output to make the slopegraphs a tad prettier. Also included with the refactor is a new limitation of the CSV file requiring a

"LABEL, VALUE, VALUE"

format in preparation for support for multiple columns. As @jayjacobs said to me, it’s easy to reformat data into the CSV file format, and, he’s right (as usual).

Plans for the next revision include:

Specifying a transparent background
Specifying PDF|PS|SVG|PNG format output
Allowing for an arbitrary number of columns for the slopegraph
Optional column labels as well as slopepgraph title (with themeing)
Line color change by slope up/same/down value (will most likely be pushed out, tho)

Here’s the whole source:

```
import csv
```
```
import cairo
```
```
import argparse
```
```
import json
```
```
 
```
```
def split(input, size):
```

	return [input[start:start+size] for start in range(0, len(input), size)]

```
 
```
```
class Slopegraph:
```
```
 
```
```
	SLOPEGRAPH_CANVAS_SIZE = 300
```
```
 
```
```
	starts = {} # starting "points"
```
```
	ends = {} # ending "points"
```

	pairs = [] # base pair array for the final plotting

```
 
```
```
	def readCSV(self, filename):
```
```
 
```

		slopeReader = csv.reader(open(filename, 'rb'), delimiter=',', quotechar='"')

```
 
```
```
		for row in slopeReader:
```
```
 
```

			# add chosen values (need start/end for each CSV row) to the final plotting array.

```
 
```
```
			lab = row[0] # label
```
```
			beg = float(row[1]) # left vals
```
```
			end = float(row[2]) # right vals
```
```
 
```

			self.pairs.append( (float(beg), float(end)) )

```
 
```

			# combine labels of common values into one string

```
 
```
```
			if beg in self.starts:
```

				self.starts[beg] = self.starts[beg] + "; " + lab

```
			else:
```
```
				self.starts[beg] = lab
```
```
 
```
```
 
```
```
			if end in self.ends:
```

				self.ends[end] = self.ends[end] + "; " + lab

```
			else:
```
```
				self.ends[end] = lab
```
```
 
```
```
 
```
```
	def sortKeys(self):
```
```
 
```

		# sort all the values (in the event the CSV wasn't) so

		# we can determine the smallest increment we need to use

		# when stacking the labels and plotting points

```
 
```

		self.startSorted = [(k, self.starts[k]) for k in sorted(self.starts)]

		self.endSorted = [(k, self.ends[k]) for k in sorted(self.ends)]

```
 
```

		self.startKeys = sorted(self.starts.keys())

```
		self.delta = max(self.startSorted)
```
```
		for i in range(len(self.startKeys)):
```
```
			if (i+1 <= len(self.startKeys)-1):
```

				currDelta = float(self.startKeys[i+1]) - float(self.startKeys[i])

```
				if (currDelta < self.delta):
```
```
					self.delta = currDelta
```
```
 
```

		self.endKeys = sorted(self.ends.keys())

```
		for i in range(len(self.endKeys)):
```
```
			if (i+1 <= len(self.endKeys)-1):
```

				currDelta = float(self.endKeys[i+1]) - float(self.endKeys[i])

```
				if (currDelta < self.delta):
```
```
					self.delta = currDelta
```
```
 
```
```
 
```
```
	def findExtremes(self):
```
```
 
```

		# we also need to find the absolute min & max values

```
		# so we know how to scale the plots
```
```
 
```
```
		self.lowest = min(self.startKeys)
```

		if (min(self.endKeys) < self.lowest) : self.lowest = min(self.endKeys)

```
 
```
```
		self.highest = max(self.startKeys)
```

		if (max(self.endKeys) > self.highest) : self.highest = max(self.endKeys)

```
 
```
```
		self.delta = float(self.delta)
```
```
		self.lowest = float(self.lowest)
```
```
		self.highest = float(self.highest)
```
```
 
```
```
 
```

	def calculateExtents(self, filename, format, valueFormatString):

```
 
```

		surface = cairo.PDFSurface (filename, 8.5*72, 11*72)

```
		cr = cairo.Context (surface)
```
```
		cr.save()
```

		cr.select_font_face(self.FONT_FAMILY, cairo.FONT_SLANT_NORMAL, cairo.FONT_WEIGHT_NORMAL)

```
		cr.set_font_size(self.FONT_SIZE)
```
```
		cr.set_line_width(self.LINE_WIDTH)
```
```
 
```

		# find the *real* maximum label width (not just based on number of chars)

```
 
```
```
		maxLabelWidth = 0
```
```
		maxNumWidth = 0
```
```
 
```
```
		for k in sorted(self.startKeys):
```
```
			s1 = self.starts[k]
```

			xbearing, ybearing, self.sWidth, self.sHeight, xadvance, yadvance = (cr.text_extents(s1))

			if (self.sWidth > maxLabelWidth) : maxLabelWidth = self.sWidth

			xbearing, ybearing, self.startMaxLabelWidth, startMaxLabelHeight, xadvance, yadvance = (cr.text_extents(valueFormatString % (k)))

			if (self.startMaxLabelWidth > maxNumWidth) : maxNumWidth = self.startMaxLabelWidth

```
 
```
```
		self.sWidth = maxLabelWidth
```
```
		self.startMaxLabelWidth = maxNumWidth
```
```
 
```
```
		maxLabelWidth = 0
```
```
		maxNumWidth = 0
```
```
 
```
```
		for k in sorted(self.endKeys):
```
```
			e1 = self.ends[k]
```

			xbearing, ybearing, self.eWidth, eHeight, xadvance, yadvance = (cr.text_extents(e1))

			if (self.eWidth > maxLabelWidth) : maxLabelWidth = self.eWidth

			xbearing, ybearing, self.endMaxLabelWidth, endMaxLabelHeight, xadvance, yadvance = (cr.text_extents(valueFormatString % (k)))

			if (self.endMaxLabelWidth > maxNumWidth) : maxNumWidth = self.endMaxLabelWidth

```
 
```
```
		self.eWidth = maxLabelWidth
```
```
		self.endMaxLabelWidth = maxNumWidth	
```
```
 
```
```
		cr.restore()
```
```
		cr.show_page()
```
```
		surface.finish()
```
```
 
```

		self.width = self.X_MARGIN + self.sWidth + self.SPACE_WIDTH + self.startMaxLabelWidth + self.SPACE_WIDTH + self.SLOPEGRAPH_CANVAS_SIZE + self.SPACE_WIDTH + self.endMaxLabelWidth + self.SPACE_WIDTH + self.eWidth + self.X_MARGIN ;

		self.height = (self.Y_MARGIN * 2) + (((self.highest - self.lowest) / self.delta) * self.LINE_HEIGHT)

```
 
```
```
 
```

	def makeSlopegraph(self, filename, config):

```
 
```

		(lab_r,lab_g,lab_b) = split(config["label_color"],2)

		(val_r,val_g,val_b) = split(config["value_color"],2)

		(line_r,line_g,line_b) = split(config["slope_color"],2)

		(bg_r,bg_g,bg_b) = split(config["background_color"],2)

```
 
```
```
		LAB_R = (int(lab_r, 16)/255.0)
```
```
		LAB_G = (int(lab_g, 16)/255.0)
```
```
		LAB_B = (int(lab_b, 16)/255.0)
```
```
 
```
```
		VAL_R = (int(val_r, 16)/255.0)
```
```
		VAL_G = (int(val_g, 16)/255.0)
```
```
		VAL_B = (int(val_b, 16)/255.0)
```
```
 
```
```
		LINE_R = (int(line_r, 16)/255.0)
```
```
		LINE_G = (int(line_g, 16)/255.0)
```
```
		LINE_B = (int(line_b, 16)/255.0)
```
```
 
```
```
		BG_R = (int(bg_r, 16)/255.0)
```
```
		BG_G = (int(bg_g, 16)/255.0)
```
```
		BG_B = (int(bg_b, 16)/255.0)
```
```
 
```

		surface = cairo.PDFSurface (filename, self.width, self.height)

```
		cr = cairo.Context(surface)
```
```
 
```
```
		cr.save()
```
```
 
```

		cr.select_font_face(self.FONT_FAMILY, cairo.FONT_SLANT_NORMAL, cairo.FONT_WEIGHT_NORMAL)

```
		cr.set_font_size(self.FONT_SIZE)
```
```
 
```
```
		cr.set_line_width(self.LINE_WIDTH)
```
```
 
```
```
		cr.set_source_rgb(BG_R,BG_G,BG_B)
```

		cr.rectangle(0,0,self.width,self.height)

```
		cr.fill()
```
```
 
```

		# draw start labels at the correct positions

```
 
```

		valueFormatString = config["value_format_string"]

```
 
```
```
		for k in sorted(self.startKeys):
```
```
 
```
```
			val = float(k)
```
```
			label = self.starts[k]
```

			xbearing, ybearing, lWidth, lHeight, xadvance, yadvance = (cr.text_extents(label))

			xbearing, ybearing, kWidth, kHeight, xadvance, yadvance = (cr.text_extents(valueFormatString % (val)))

```
 
```
```
			cr.set_source_rgb(LAB_R,LAB_G,LAB_B)
```

			cr.move_to(self.X_MARGIN + (self.sWidth - lWidth), self.Y_MARGIN + (self.highest - val) * self.LINE_HEIGHT * (1/self.delta))

```
			cr.show_text(label)
```
```
 
```
```
			cr.set_source_rgb(VAL_R,VAL_G,VAL_B)
```

			cr.move_to(self.X_MARGIN + self.sWidth + self.SPACE_WIDTH + (self.startMaxLabelWidth - kWidth), self.Y_MARGIN + (self.highest - val) * self.LINE_HEIGHT * (1/self.delta))

			cr.show_text(valueFormatString % (val))

```
 
```
```
			cr.stroke()
```
```
 
```

		# draw end labels at the correct positions

```
 
```
```
		for k in sorted(self.endKeys):
```
```
 
```
```
			val = float(k)
```
```
			label = self.ends[k]
```

			xbearing, ybearing, lWidth, lHeight, xadvance, yadvance = (cr.text_extents(label))

```
 
```
```
			cr.set_source_rgb(VAL_R,VAL_G,VAL_B)
```

			cr.move_to(self.width - self.X_MARGIN - self.SPACE_WIDTH - self.eWidth - self.SPACE_WIDTH - self.endMaxLabelWidth, self.Y_MARGIN + (self.highest - val) * self.LINE_HEIGHT * (1/self.delta))

			cr.show_text(valueFormatString % (val))

```
 
```
```
			cr.set_source_rgb(LAB_R,LAB_G,LAB_B)
```

			cr.move_to(self.width - self.X_MARGIN - self.SPACE_WIDTH - self.eWidth, self.Y_MARGIN + (self.highest - val) * self.LINE_HEIGHT * (1/self.delta))

```
			cr.show_text(label)
```
```
 
```
```
			cr.stroke()
```
```
 
```
```
		# do the actual plotting
```
```
 
```
```
		cr.set_line_width(self.LINE_WIDTH)
```

		cr.set_source_rgb(LINE_R, LINE_G, LINE_B)

```
 
```
```
		for s1,e1 in self.pairs:
```

			cr.move_to(self.X_MARGIN + self.sWidth + self.SPACE_WIDTH + self.startMaxLabelWidth + self.LINE_START_DELTA, self.Y_MARGIN + (self.highest - s1) * self.LINE_HEIGHT * (1/self.delta) - self.LINE_HEIGHT/4)

			cr.line_to(self.width - self.X_MARGIN - self.eWidth - self.SPACE_WIDTH - self.endMaxLabelWidth - self.LINE_START_DELTA, self.Y_MARGIN + (self.highest - e1) * self.LINE_HEIGHT * (1/self.delta) - self.LINE_HEIGHT/4)

```
			cr.stroke()
```
```
 
```
```
		cr.restore()
```
```
		cr.show_page()
```
```
		surface.finish()	
```
```
 
```
```
 
```
```
	def __init__(self, config):
```
```
 
```

		# a couple methods need these so make them local to the class

```
 
```

		self.FONT_FAMILY = config["font_family"]

		self.LINE_WIDTH = float(config["line_width"])

		self.X_MARGIN = float(config["x_margin"])

		self.Y_MARGIN = float(config["y_margin"])

		self.FONT_SIZE = float(config["font_size"])

		self.SPACE_WIDTH = self.FONT_SIZE / 2.0

		self.LINE_HEIGHT = self.FONT_SIZE + (self.FONT_SIZE / 2.0)

		self.LINE_START_DELTA = 1.5*self.SPACE_WIDTH

```
 
```

		OUTPUT_FILE = config["output"] + "." + config["format"]

```
 
```

		# process the values & make the slopegraph

```
 
```
```
		self.readCSV(config["input"])
```
```
		self.sortKeys()
```
```
		self.findExtremes()
```

		self.calculateExtents(OUTPUT_FILE, config["format"], config["value_format_string"])

		self.makeSlopegraph(OUTPUT_FILE, config)

```
 
```
```
 
```
```
def main():
```
```
 
```

	parser = argparse.ArgumentParser(description="Creates a slopegraph from a CSV source")

	parser.add_argument("--config",required=True,

					help="config file name to use for  slopegraph creation",)

```
	args = parser.parse_args()
```
```
 
```
```
	if args.config:
```
```
 
```
```
		json_data = open(args.config)
```
```
		config = json.load(json_data)
```
```
		json_data.close()
```
```
 
```
```
		Slopegraph(config)
```
```
 
```
```
	return(0)
```
```
 
```
```
if __name__ == "__main__":
```
```
	main()
```

Slopegraphs in Python – Attention to Detail

2012-05-31 – 06:04
Posted in Charts & Graphs, Development, Python
Tagged table-chart, tufte
Leave a Comment

In the previous installment, a foundation was laid for “parameterizing” fonts, colors and overall slopegraph size. However, a big failing in all this code (up until now) was the reliance on character string length to determine label width. When working with fonts, the font metrics are more important since a lowercase ‘l’ will have a smaller font width than an uppercase ‘D’. So, while ‘llll’ is “longer” than ‘DDD’, it may not be wider (especially in a sans-serif font, but likely in any font):

To solve this problem, we need to use a temporary Cairo surface to compute font metrics for each label & value so we know what the maximum width of both for the starting and ending points. It’s a simple concept & calculation, but very important to ensure everything lines up well.

# find the *real* maximum label width (not just based on number of chars)

```
 
```
```
maxLabelWidth = 0
```
```
maxNumWidth = 0
```
```
 
```
```
for k in sorted(startKeys):
```
```
	s1 = starts[k]
```

	xbearing, ybearing, sWidth, sHeight, xadvance, yadvance = (cr.text_extents(s1))

	if (sWidth > maxLabelWidth) : maxLabelWidth = sWidth

	xbearing, ybearing, startMaxLabelWidth, startMaxLabelHeight, xadvance, yadvance = (cr.text_extents(str(k)))

	if (startMaxLabelWidth > maxNumWidth) : maxNumWidth = startMaxLabelWidth

```
 
```
```
sWidth = maxLabelWidth
```
```
startMaxLabelWidth = maxNumWidth
```
```
 
```
```
maxWidth = 0
```
```
maxNumWidth = 0
```
```
 
```
```
for k in sorted(endKeys):
```
```
	e1 = ends[k]
```

	xbearing, ybearing, eWidth, eHeight, xadvance, yadvance = (cr.text_extents(e1))

	if (eWidth > maxLabelWidth) : maxLabelWidth = eWidth

	xbearing, ybearing, endMaxLabelWidth, endMaxLabelHeight, xadvance, yadvance = (cr.text_extents(str(k)))

	if (endMaxLabelWidth > maxNumWidth) : maxNumWidth = endMaxLabelWidth

```
 
```
```
eWidth = maxLabelWidth
```
```
endMaxLabelWidth = maxNumWidth
```

I tossed some “anomalies” into the sample data set to show both how adaptable the vertical scale is as well as demonstrate the label alignments:

Updates are in github.

Slopegraphs In Python – colors, fonts & alignment

2012-05-29 – 21:49
Posted in Charts & Graphs, Development, Python
Tagged slopegraph, table-chart, tufte
Comments (1)

On the heels of last evening’s release of Slopegraphs in Python post comes some minor tweaks:

Complete alignment control of labels & and values
Colors (for background, lines, labels & values) — I picked a random pattern from Adobe’s Kuler
A font change (to prove width calculations work)

…and a new example slopegraph:

As promised, the latest revisions are in github.

Some notes for aspiring Python/Cairo hackers:

RGB colors are 0-1 in Cairo, so divide your 0-255 values by 255 to get the corresponding Cairo value and make sure you’re doing float arithmetic in Python
It turns out simple Cairo font rendering (ie. non-Pango) does not interpret well in Illustrator from a Cairo PDF surface, so if you do plan on post-processing the slopegraphs, use a postscript/EPS surface:
surface = cairo.PSSurface ("slopegraph.ps", width, height)
surface.set_eps(True)
(I’ll be incorporating an output option as this gets closer to 1.0)
If you do plan on using this at all, grab the Bitstream Vera Serif font (link goes to FontSquirrel, but you can find it almost everywhere as it’s free)

An Homage To Flame

(Sing to the tune of “Fame – Remember My name” …
Here’s some YouTube background music)

They’ve been lookin’ at me, but they never did see—
no, no trace of me did they detect;
Gave me time to collect all the data at rest.
I’ve got so much in me: LUA, zlib & sqlite3–
I can infect the USB in your hand. Don’t you know who I am?
Remember my name [FLAME]

I’ve been around forever. Capturing packets on the fly. [HIGH]
My botnet is comin’ together. When researchers see me they’ll cry. [FLAME]
I even infected Lebanon. Lit up the Middle East with my FLAME. [FLAME]
I’ve been around forever. They will remember my name.

[REMEMBER, REMEMBER, REMEMBER, REMEMBER, REMEMBER, ]

I’m not packed up too tight (I take up 20 megabytes).
With no kill date, I’ll never stop.
Give me your mic and I’ll take all you’ve got to give.
Finding me will be tough. Too much (you’ll say ‘enough’!)
I can ride your net but not break (it). Yeah, I got what it takes.

FLAME!

[REMEMBER, REMEMBER, REMEMBER, REMEMBER, REMEMBER, REMEMBER]

FLAME!…

Slopegraphs in Python

2012-05-28 – 21:01
Posted in Charts & Graphs, Development, Open Source, OS X, Python
Tagged slopegraph, table-chart, tufte
Comments (2)

(NOTE: You can keep up with progress best at github, but can always search on “slopegraph” here or just hit the tag page: “slopegraph” regularly)

I’ve been a bit obsessed with slopegraphs (a.k.a “Tufte table-chart”) of late and very dissatisfied with the lack of tools to make this particular visualization tool more prevalent. While my ultimate goal is to have a user-friendly modern web app or platform app that’s as easy as a “drag & drop” of a CSV file, this first foray will require a bit (not much, really!) of elbow grease to be used.

For those who want to get right to the code, head on over to github and have a look (I’ll post all updates there). Setup, sample & source are also below.

First, you’ll need a modern Python install. I did all the development on Mac OS Mountain Lion (beta) with the stock Python 2.7 build. You’ll also need the Cairo 2D graphics library which built and installed perfectly from source, even on ML, so it should work fine for you. If you want something besides PDF rendering, you may need additional libraries, but PDF is decent for hi-res embedding, converting to jpg/png (see below) and tweaking in programs like Illustrator.

If you search for “Gender Comparisons” in the comments on this post at Tufte’s blog, you’ll see what I was trying to reproduce in this bit of skeleton code (below). By modifying the CSV file you’re using [line 21] and then which fields are relevant [lines 45-47] you should be able to make your own basic slopegraphs without much trouble.

If you catch any glitches, add some tweak or have a slopegraph “wish list”, let me know here, twitter (@hrbrmstr) or over at github.

```
# slopegraph.py
```
```
#
```
```
# Author: Bob Rudis (@hrbrmstr)
```
```
#
```

# Basic Python skeleton to do simple two value slopegraphs

# with output to PDF (most useful form for me...Cairo has tons of options)

```
#
```

# Find out more about & download Cairo here:

```
# http://cairographics.org/
```
```
#
```

# 2012-05-28 - 0.5 - Initial github release. Still needs some polish

```
#
```
```
 
```
```
import csv
```
```
import cairo
```
```
 
```

# original data source: http://www.calvin.edu/~stob/data/television.csv

```
 
```
```
# get a CSV file to work with 
```
```
 
```

slopeReader = csv.reader(open('television.csv', 'rb'), delimiter=',', quotechar='"')

```
 
```
```
starts = {} # starting "points"/
```
```
ends = {} # ending "points"
```
```
 
```

# Need to refactor label max width into font calculations

# as there's no guarantee the longest (character-wise)

```
# label is the widest one
```
```
 
```
```
startLabelMaxLen = 0
```
```
endLabelMaxLen = 0
```
```
 
```

# build a base pair array for the final plotting

# wastes memory, but simplifies plotting

```
 
```
```
pairs = []
```
```
 
```
```
for row in slopeReader:
```
```
 
```

	# add chosen values (need start/end for each CSV row)

	# to the final plotting array. Try this sample with

	# row[1] (average life span) instead of row[5] to see some

```
	# of the scaling in action
```
```
 
```
```
	lab = row[0] # label
```
```
	beg = row[5] # male life span
```
```
	end = row[4] # female life span
```
```
 
```

	pairs.append( (float(beg), float(end)) )

```
 
```

	# combine labels of common values into one string

	# also (as noted previously, inappropriately) find the

```
	# longest one
```
```
 
```
```
	if beg in starts:
```

		starts[beg] = starts[beg] + "; " + lab

```
	else:
```
```
		starts[beg] = lab
```
```
 
```

	if ((len(starts[beg]) + len(beg)) > startLabelMaxLen):

		startLabelMaxLen = len(starts[beg]) + len(beg)

```
		s1 = starts[beg]
```
```
 
```
```
 
```
```
	if end in ends:
```
```
		ends[end] = ends[end] + "; " + lab
```
```
	else:
```
```
		ends[end] = lab
```
```
 
```

	if ((len(ends[end]) + len(end)) > endLabelMaxLen):

		endLabelMaxLen = len(ends[end]) + len(end)

```
		e1 = ends[end]
```
```
 
```

# sort all the values (in the event the CSV wasn't) so

# we can determine the smallest increment we need to use

# when stacking the labels and plotting points

```
 
```

startSorted = [(k, starts[k]) for k in sorted(starts)]

endSorted = [(k, ends[k]) for k in sorted(ends)]

```
 
```
```
startKeys = sorted(starts.keys())
```
```
delta = max(startSorted)
```
```
for i in range(len(startKeys)):
```
```
	if (i+1 <= len(startKeys)-1):
```

		currDelta = float(startKeys[i+1]) - float(startKeys[i])

```
		if (currDelta < delta):
```
```
			delta = currDelta
```
```
 
```
```
endKeys = sorted(ends.keys())
```
```
for i in range(len(endKeys)):
```
```
	if (i+1 <= len(endKeys)-1):
```

		currDelta = float(endKeys[i+1]) - float(endKeys[i])

```
		if (currDelta < delta):
```
```
			delta = currDelta
```
```
 
```

# we also need to find the absolute min & max values

```
# so we know how to scale the plots
```
```
 
```
```
lowest = min(startKeys)
```

if (min(endKeys) < lowest) : lowest = min(endKeys)

```
 
```
```
highest = max(startKeys)
```

if (max(endKeys) > highest) : highest = max(endKeys)

```
 
```

# just making sure everything's a number

# probably should move some of this to the csv reader section

```
 
```
```
delta = float(delta)
```
```
lowest = float(lowest)
```
```
highest = float(highest)
```

startLabelMaxLen = float(startLabelMaxLen)

```
endLabelMaxLen = float(endLabelMaxLen)
```
```
 
```

# setup line width and font-size for the Cairo

# you can change these and the constants should

```
# scale the plots accordingly
```
```
 
```
```
FONT_SIZE = 9
```
```
LINE_WIDTH = 0.5
```
```
 
```

# there has to be a better way to get a base "surface"

# to do font calculations besides this. we're just making

# this Cairo surface to we know the max pixel width

# (font extents) of the labels in order to scale the graph

# accurately (since width/height are based, in part, on it)

```
 
```
```
filename = 'slopegraph.pdf'
```

surface = cairo.PDFSurface (filename, 8.5*72, 11*72)

```
cr = cairo.Context (surface)
```
```
cr.save()
```

cr.select_font_face("Sans", cairo.FONT_SLANT_NORMAL, cairo.FONT_WEIGHT_NORMAL)

```
cr.set_font_size(FONT_SIZE)
```
```
cr.set_line_width(LINE_WIDTH)
```

xbearing, ybearing, sWidth, sHeight, xadvance, yadvance = (cr.text_extents(s1))

xbearing, ybearing, eWidth, eHeight, xadvance, yadvance = (cr.text_extents(e1))

xbearing, ybearing, spaceWidth, spaceHeight, xadvance, yadvance = (cr.text_extents(" "))

```
cr.restore()
```
```
cr.show_page()
```
```
surface.finish()
```
```
 
```

# setup some more constants for plotting

# all of these are malleable and should cascade nicely

```
 
```
```
X_MARGIN = 10
```
```
Y_MARGIN = 10
```
```
SLOPEGRAPH_CANVAS_SIZE = 200
```
```
spaceWidth = 5
```
```
LINE_HEIGHT = 15
```
```
PLOT_LINE_WIDTH = 0.5
```
```
 
```

width = (X_MARGIN * 2) + sWidth + spaceWidth + SLOPEGRAPH_CANVAS_SIZE + spaceWidth + eWidth

height = (Y_MARGIN * 2) + (((highest - lowest + 1) / delta) * LINE_HEIGHT)

```
 
```
```
# create the real Cairo surface/canvas
```
```
 
```
```
filename = 'slopegraph.pdf'
```

surface = cairo.PDFSurface (filename, width, height)

```
cr = cairo.Context (surface)
```
```
 
```
```
cr.save()
```
```
 
```

cr.select_font_face("Sans", cairo.FONT_SLANT_NORMAL, cairo.FONT_WEIGHT_NORMAL)

```
cr.set_font_size(FONT_SIZE)
```
```
 
```
```
cr.set_line_width(LINE_WIDTH)
```

cr.set_source_rgba (0, 0, 0) # need to make this a constant

```
 
```

# draw start labels at the correct positions

# cheating a bit here as the code doesn't (yet) line up

```
# the actual data values
```
```
 
```
```
for k in sorted(startKeys):
```
```
 
```
```
	label = starts[k]
```

	xbearing, ybearing, lWidth, lHeight, xadvance, yadvance = (cr.text_extents(label))

```
 
```
```
	val = float(k)
```
```
 
```

	cr.move_to(X_MARGIN + (sWidth - lWidth), Y_MARGIN + (highest - val) * LINE_HEIGHT * (1/delta) + LINE_HEIGHT/2)

```
	cr.show_text(label + " " + k)
```
```
	cr.stroke()
```
```
 
```

# draw end labels at the correct positions

# cheating a bit here as the code doesn't (yet) line up

```
# the actual data values
```
```
 
```
```
for k in sorted(endKeys):
```
```
 
```
```
	label = ends[k]
```

	xbearing, ybearing, lWidth, lHeight, xadvance, yadvance = (cr.text_extents(label))

```
 
```
```
	val = float(k)
```
```
 
```

	cr.move_to(width - X_MARGIN - eWidth - (4*spaceWidth), Y_MARGIN + (highest - val) * LINE_HEIGHT * (1/delta) + LINE_HEIGHT/2)

```
	cr.show_text(k + " " + label)
```
```
	cr.stroke()
```
```
 
```
```
# do the actual plotting
```
```
 
```
```
cr.set_line_width(PLOT_LINE_WIDTH)
```

cr.set_source_rgba (0.75, 0.75, 0.75) # need to make this a constant

```
 
```
```
for s1,e1 in pairs:
```

	cr.move_to(X_MARGIN + sWidth + spaceWidth + 20, Y_MARGIN + (highest - s1) * LINE_HEIGHT * (1/delta) + LINE_HEIGHT/2)

	cr.line_to(width - X_MARGIN - eWidth - spaceWidth - 20, Y_MARGIN + (highest - e1) * LINE_HEIGHT * (1/delta) + LINE_HEIGHT/2)

```
	cr.stroke()
```
```
 
```
```
cr.restore()
```
```
cr.show_page()
```
```
surface.finish()
```

New SecurID Soft Token Cloning Weakness : What’s The Risk?

I posted a link to Twitter earlier on a recent discovery of the ability to clone RSA SecurID soft tokens:

https://twitter.com/hrbrmstr/status/204908233645764609

It (rightfully so) received some critical responses by @wh1t3rabbit & @wikidsystems since, apart from what the hypesters may say, this is a low-risk weakness.

@Wh1t3Rabbit @hrbrmstr once you own the machine and can enumerate SIDs in AD, aren't you in? And, don't you still need the PIN?
— Nick Owen (@wikidsystems) May 22, 2012

Think about it. Just looking at the two most likely threat actors & actions: an insider trying to siphon off soft tokens and an external attacker using crafted malware to grab soft tokens. The former (most likely) knows your organization is using soft tokens (and probably has one herself). The latter is unlikely to just try to blanket siphon off soft tokens so they’ll have to do some research to target an organization (which costs time/money).

Once a victim (or set of victims) is identified, the cloning steps would have to be perfectly executed (and, I’m not convinced that’s a given). Let’s say that this is a given, though. Now both the insider and external agent have access to the bits to clone a token. It is easier for the insider to get that data, but the external attacker has to exfiltrate successfully it somehow (more complexity/time/cost).

To be useful, the attacker needs the user id, PIN and – in most implementations – a password. An insider would (most likely) know the user id (since she probably has one herself) but that data would require more time/effort/cost to the external attacker (think opportunistic keylogger/screenscraper with successful exfiltration). For both attackers, getting the password requires either social engineering or the use of a keylogger. Even then, there’s a time-limit of 90 days or less (since, if you’re using soft tokens, you probably have a 90 day password policy). That shrinks the amount of time the attack can be successful.

Now, both attackers need to know where this soft token can be used and have direct access to those systems. Again, probably easier for an insider and fairly costly for an external attacker.

Looking at this, there’s definitely a greater risk associated with an insider from this weakness than there is from an external party (as pointed out by the aforementioned twitter commentators). As @wikidsystems further pointed out, this also shows the inherent positives of multi-factor authentication :: you need far more component parts to execute a successful attack, making the whole thing very costly to obtain. Security economics FTW!

My comment has been that if using the TPM store for Windows-based SecurID soft token implementations negates this weakness, then why not do it? Does the added deployment & management complexity really cost that much?

In the end, I would categorize this weakness as a low risk to most organizations using soft tokens with a non-TPM storage configuration. Unless you know you’re a nation-state target (my opine for the origin of the attacker) – and, even then, you’re probably using hard tokens – far too many celestial bodies need to align for this weakness to be exploited successfully.

NOTE: This post was not meant to be a comprehensive risk assessment of the weakness and does not cover all attack scenarios. I left out many, including Windows desktop administrators and privileged script access. I was merely trying to do my part to counter whatever hype ensues from this weakness. Comments on those vectors or the analysis in general are most welcome.

Feudal Japan’s Haiku Clubs A Pre-cursor of Modern-day Twitter?

2012-05-21 – 17:35
Posted in haiku, japan, samurai, twitter
Leave a Comment

I recently finished watching “Japan: Memoirs of a Secret Empire” [ Netflix | PBS | Amazon ]. It’s a three-part documentary that centers around the 16th & 17th centuries (where a vast majority of my favorite samurai movies are set).

In the third part there is a segment on the four-tier class system. It goes into some detail on how the decline of the samurai (due to a time of significant peace) and the increase in trade facilitated the erosion of societal taboos that previously prevented classes from mingling. Apart from the creation of a new merchant-inspired class and an increase in the frequenting of courtesans there was also the mention of “haiku clubs” where (quoting from the documentary) “members chose pen names to obscure their social rank. That way, the classes could mingle freely“.

In a similar way, Twitter (and many forums before it) is the great equalizer with the added similarity of brevity (and often a gravity well for some modern day haiku fanatics). I don’t have any statistics, but I do wonder if Twitter is having a more equalizing impact on societies where class and overt discrimination are still wildly prevalent (think India or Saudi Arabia).

as avatars tweet
modern words become equal
history repeats