hrbrmstr, Author at rud.is

Author Archives: hrbrmstr

Don't look at me…I do what he does — just slower. #rstats avuncular • ?Resistance Fighter • Cook • Christian • [Master] Chef des Données de Sécurité @ @rapid7

Slopegraphs in Python – The Great Refactor

2012-06-02 – 18:47
Posted in Charts & Graphs, Development, Python
Tagged slopegraph, table-chart, tufte
Leave a Comment

Despite being on holiday, I had a spare hour to refactor the code (@mrshrbrmstr was joining the 1% in the hotel spa). It’s up on github and now sports a spiffy JSON-format config file. You now must execute the slopegraph.py script with a “--config FILENAME” argument. The configuration file lets you specify the “theme” as well as the input file and output format (you can only use PDF for the moment).

Here’s a sample config file included in the github push (there’s another one there too):

```
{
```
```
 
```
```
"font_family" : "Palatino",
```
```
"font_size" : "20",
```
```
 
```
```
"x_margin" : "20",
```
```
"y_margin" : "30",
```
```
 
```
```
"line_width" : "0.5",
```
```
 
```
```
"background_color" : "DEC299",
```
```
"label_color" : "687D64",
```
```
"value_color" : "949258",
```
```
"slope_color" : "61514C",
```
```
 
```
```
"value_format_string" : "%2d",
```
```
 
```
```
"input" : "television.csv",
```
```
"output" : "television",
```
```
"format" : "pdf"
```
```
 
```
```
}
```

Included in the refactor is the ability to use a sprintf-like format string for the label value output to make the slopegraphs a tad prettier. Also included with the refactor is a new limitation of the CSV file requiring a

"LABEL, VALUE, VALUE"

format in preparation for support for multiple columns. As @jayjacobs said to me, it’s easy to reformat data into the CSV file format, and, he’s right (as usual).

Plans for the next revision include:

Specifying a transparent background
Specifying PDF|PS|SVG|PNG format output
Allowing for an arbitrary number of columns for the slopegraph
Optional column labels as well as slopepgraph title (with themeing)
Line color change by slope up/same/down value (will most likely be pushed out, tho)

Here’s the whole source:

```
import csv
```
```
import cairo
```
```
import argparse
```
```
import json
```
```
 
```
```
def split(input, size):
```

	return [input[start:start+size] for start in range(0, len(input), size)]

```
 
```
```
class Slopegraph:
```
```
 
```
```
	SLOPEGRAPH_CANVAS_SIZE = 300
```
```
 
```
```
	starts = {} # starting "points"
```
```
	ends = {} # ending "points"
```

	pairs = [] # base pair array for the final plotting

```
 
```
```
	def readCSV(self, filename):
```
```
 
```

		slopeReader = csv.reader(open(filename, 'rb'), delimiter=',', quotechar='"')

```
 
```
```
		for row in slopeReader:
```
```
 
```

			# add chosen values (need start/end for each CSV row) to the final plotting array.

```
 
```
```
			lab = row[0] # label
```
```
			beg = float(row[1]) # left vals
```
```
			end = float(row[2]) # right vals
```
```
 
```

			self.pairs.append( (float(beg), float(end)) )

```
 
```

			# combine labels of common values into one string

```
 
```
```
			if beg in self.starts:
```

				self.starts[beg] = self.starts[beg] + "; " + lab

```
			else:
```
```
				self.starts[beg] = lab
```
```
 
```
```
 
```
```
			if end in self.ends:
```

				self.ends[end] = self.ends[end] + "; " + lab

```
			else:
```
```
				self.ends[end] = lab
```
```
 
```
```
 
```
```
	def sortKeys(self):
```
```
 
```

		# sort all the values (in the event the CSV wasn't) so

		# we can determine the smallest increment we need to use

		# when stacking the labels and plotting points

```
 
```

		self.startSorted = [(k, self.starts[k]) for k in sorted(self.starts)]

		self.endSorted = [(k, self.ends[k]) for k in sorted(self.ends)]

```
 
```

		self.startKeys = sorted(self.starts.keys())

```
		self.delta = max(self.startSorted)
```
```
		for i in range(len(self.startKeys)):
```
```
			if (i+1 <= len(self.startKeys)-1):
```

				currDelta = float(self.startKeys[i+1]) - float(self.startKeys[i])

```
				if (currDelta < self.delta):
```
```
					self.delta = currDelta
```
```
 
```

		self.endKeys = sorted(self.ends.keys())

```
		for i in range(len(self.endKeys)):
```
```
			if (i+1 <= len(self.endKeys)-1):
```

				currDelta = float(self.endKeys[i+1]) - float(self.endKeys[i])

```
				if (currDelta < self.delta):
```
```
					self.delta = currDelta
```
```
 
```
```
 
```
```
	def findExtremes(self):
```
```
 
```

		# we also need to find the absolute min & max values

```
		# so we know how to scale the plots
```
```
 
```
```
		self.lowest = min(self.startKeys)
```

		if (min(self.endKeys) < self.lowest) : self.lowest = min(self.endKeys)

```
 
```
```
		self.highest = max(self.startKeys)
```

		if (max(self.endKeys) > self.highest) : self.highest = max(self.endKeys)

```
 
```
```
		self.delta = float(self.delta)
```
```
		self.lowest = float(self.lowest)
```
```
		self.highest = float(self.highest)
```
```
 
```
```
 
```

	def calculateExtents(self, filename, format, valueFormatString):

```
 
```

		surface = cairo.PDFSurface (filename, 8.5*72, 11*72)

```
		cr = cairo.Context (surface)
```
```
		cr.save()
```

		cr.select_font_face(self.FONT_FAMILY, cairo.FONT_SLANT_NORMAL, cairo.FONT_WEIGHT_NORMAL)

```
		cr.set_font_size(self.FONT_SIZE)
```
```
		cr.set_line_width(self.LINE_WIDTH)
```
```
 
```

		# find the *real* maximum label width (not just based on number of chars)

```
 
```
```
		maxLabelWidth = 0
```
```
		maxNumWidth = 0
```
```
 
```
```
		for k in sorted(self.startKeys):
```
```
			s1 = self.starts[k]
```

			xbearing, ybearing, self.sWidth, self.sHeight, xadvance, yadvance = (cr.text_extents(s1))

			if (self.sWidth > maxLabelWidth) : maxLabelWidth = self.sWidth

			xbearing, ybearing, self.startMaxLabelWidth, startMaxLabelHeight, xadvance, yadvance = (cr.text_extents(valueFormatString % (k)))

			if (self.startMaxLabelWidth > maxNumWidth) : maxNumWidth = self.startMaxLabelWidth

```
 
```
```
		self.sWidth = maxLabelWidth
```
```
		self.startMaxLabelWidth = maxNumWidth
```
```
 
```
```
		maxLabelWidth = 0
```
```
		maxNumWidth = 0
```
```
 
```
```
		for k in sorted(self.endKeys):
```
```
			e1 = self.ends[k]
```

			xbearing, ybearing, self.eWidth, eHeight, xadvance, yadvance = (cr.text_extents(e1))

			if (self.eWidth > maxLabelWidth) : maxLabelWidth = self.eWidth

			xbearing, ybearing, self.endMaxLabelWidth, endMaxLabelHeight, xadvance, yadvance = (cr.text_extents(valueFormatString % (k)))

			if (self.endMaxLabelWidth > maxNumWidth) : maxNumWidth = self.endMaxLabelWidth

```
 
```
```
		self.eWidth = maxLabelWidth
```
```
		self.endMaxLabelWidth = maxNumWidth	
```
```
 
```
```
		cr.restore()
```
```
		cr.show_page()
```
```
		surface.finish()
```
```
 
```

		self.width = self.X_MARGIN + self.sWidth + self.SPACE_WIDTH + self.startMaxLabelWidth + self.SPACE_WIDTH + self.SLOPEGRAPH_CANVAS_SIZE + self.SPACE_WIDTH + self.endMaxLabelWidth + self.SPACE_WIDTH + self.eWidth + self.X_MARGIN ;

		self.height = (self.Y_MARGIN * 2) + (((self.highest - self.lowest) / self.delta) * self.LINE_HEIGHT)

```
 
```
```
 
```

	def makeSlopegraph(self, filename, config):

```
 
```

		(lab_r,lab_g,lab_b) = split(config["label_color"],2)

		(val_r,val_g,val_b) = split(config["value_color"],2)

		(line_r,line_g,line_b) = split(config["slope_color"],2)

		(bg_r,bg_g,bg_b) = split(config["background_color"],2)

```
 
```
```
		LAB_R = (int(lab_r, 16)/255.0)
```
```
		LAB_G = (int(lab_g, 16)/255.0)
```
```
		LAB_B = (int(lab_b, 16)/255.0)
```
```
 
```
```
		VAL_R = (int(val_r, 16)/255.0)
```
```
		VAL_G = (int(val_g, 16)/255.0)
```
```
		VAL_B = (int(val_b, 16)/255.0)
```
```
 
```
```
		LINE_R = (int(line_r, 16)/255.0)
```
```
		LINE_G = (int(line_g, 16)/255.0)
```
```
		LINE_B = (int(line_b, 16)/255.0)
```
```
 
```
```
		BG_R = (int(bg_r, 16)/255.0)
```
```
		BG_G = (int(bg_g, 16)/255.0)
```
```
		BG_B = (int(bg_b, 16)/255.0)
```
```
 
```

		surface = cairo.PDFSurface (filename, self.width, self.height)

```
		cr = cairo.Context(surface)
```
```
 
```
```
		cr.save()
```
```
 
```

		cr.select_font_face(self.FONT_FAMILY, cairo.FONT_SLANT_NORMAL, cairo.FONT_WEIGHT_NORMAL)

```
		cr.set_font_size(self.FONT_SIZE)
```
```
 
```
```
		cr.set_line_width(self.LINE_WIDTH)
```
```
 
```
```
		cr.set_source_rgb(BG_R,BG_G,BG_B)
```

		cr.rectangle(0,0,self.width,self.height)

```
		cr.fill()
```
```
 
```

		# draw start labels at the correct positions

```
 
```

		valueFormatString = config["value_format_string"]

```
 
```
```
		for k in sorted(self.startKeys):
```
```
 
```
```
			val = float(k)
```
```
			label = self.starts[k]
```

			xbearing, ybearing, lWidth, lHeight, xadvance, yadvance = (cr.text_extents(label))

			xbearing, ybearing, kWidth, kHeight, xadvance, yadvance = (cr.text_extents(valueFormatString % (val)))

```
 
```
```
			cr.set_source_rgb(LAB_R,LAB_G,LAB_B)
```

			cr.move_to(self.X_MARGIN + (self.sWidth - lWidth), self.Y_MARGIN + (self.highest - val) * self.LINE_HEIGHT * (1/self.delta))

```
			cr.show_text(label)
```
```
 
```
```
			cr.set_source_rgb(VAL_R,VAL_G,VAL_B)
```

			cr.move_to(self.X_MARGIN + self.sWidth + self.SPACE_WIDTH + (self.startMaxLabelWidth - kWidth), self.Y_MARGIN + (self.highest - val) * self.LINE_HEIGHT * (1/self.delta))

			cr.show_text(valueFormatString % (val))

```
 
```
```
			cr.stroke()
```
```
 
```

		# draw end labels at the correct positions

```
 
```
```
		for k in sorted(self.endKeys):
```
```
 
```
```
			val = float(k)
```
```
			label = self.ends[k]
```

			xbearing, ybearing, lWidth, lHeight, xadvance, yadvance = (cr.text_extents(label))

```
 
```
```
			cr.set_source_rgb(VAL_R,VAL_G,VAL_B)
```

			cr.move_to(self.width - self.X_MARGIN - self.SPACE_WIDTH - self.eWidth - self.SPACE_WIDTH - self.endMaxLabelWidth, self.Y_MARGIN + (self.highest - val) * self.LINE_HEIGHT * (1/self.delta))

			cr.show_text(valueFormatString % (val))

```
 
```
```
			cr.set_source_rgb(LAB_R,LAB_G,LAB_B)
```

			cr.move_to(self.width - self.X_MARGIN - self.SPACE_WIDTH - self.eWidth, self.Y_MARGIN + (self.highest - val) * self.LINE_HEIGHT * (1/self.delta))

```
			cr.show_text(label)
```
```
 
```
```
			cr.stroke()
```
```
 
```
```
		# do the actual plotting
```
```
 
```
```
		cr.set_line_width(self.LINE_WIDTH)
```

		cr.set_source_rgb(LINE_R, LINE_G, LINE_B)

```
 
```
```
		for s1,e1 in self.pairs:
```

			cr.move_to(self.X_MARGIN + self.sWidth + self.SPACE_WIDTH + self.startMaxLabelWidth + self.LINE_START_DELTA, self.Y_MARGIN + (self.highest - s1) * self.LINE_HEIGHT * (1/self.delta) - self.LINE_HEIGHT/4)

			cr.line_to(self.width - self.X_MARGIN - self.eWidth - self.SPACE_WIDTH - self.endMaxLabelWidth - self.LINE_START_DELTA, self.Y_MARGIN + (self.highest - e1) * self.LINE_HEIGHT * (1/self.delta) - self.LINE_HEIGHT/4)

```
			cr.stroke()
```
```
 
```
```
		cr.restore()
```
```
		cr.show_page()
```
```
		surface.finish()	
```
```
 
```
```
 
```
```
	def __init__(self, config):
```
```
 
```

		# a couple methods need these so make them local to the class

```
 
```

		self.FONT_FAMILY = config["font_family"]

		self.LINE_WIDTH = float(config["line_width"])

		self.X_MARGIN = float(config["x_margin"])

		self.Y_MARGIN = float(config["y_margin"])

		self.FONT_SIZE = float(config["font_size"])

		self.SPACE_WIDTH = self.FONT_SIZE / 2.0

		self.LINE_HEIGHT = self.FONT_SIZE + (self.FONT_SIZE / 2.0)

		self.LINE_START_DELTA = 1.5*self.SPACE_WIDTH

```
 
```

		OUTPUT_FILE = config["output"] + "." + config["format"]

```
 
```

		# process the values & make the slopegraph

```
 
```
```
		self.readCSV(config["input"])
```
```
		self.sortKeys()
```
```
		self.findExtremes()
```

		self.calculateExtents(OUTPUT_FILE, config["format"], config["value_format_string"])

		self.makeSlopegraph(OUTPUT_FILE, config)

```
 
```
```
 
```
```
def main():
```
```
 
```

	parser = argparse.ArgumentParser(description="Creates a slopegraph from a CSV source")

	parser.add_argument("--config",required=True,

					help="config file name to use for  slopegraph creation",)

```
	args = parser.parse_args()
```
```
 
```
```
	if args.config:
```
```
 
```
```
		json_data = open(args.config)
```
```
		config = json.load(json_data)
```
```
		json_data.close()
```
```
 
```
```
		Slopegraph(config)
```
```
 
```
```
	return(0)
```
```
 
```
```
if __name__ == "__main__":
```
```
	main()
```

Slopegraphs in Python – Attention to Detail

2012-05-31 – 06:04
Posted in Charts & Graphs, Development, Python
Tagged table-chart, tufte
Leave a Comment

In the previous installment, a foundation was laid for “parameterizing” fonts, colors and overall slopegraph size. However, a big failing in all this code (up until now) was the reliance on character string length to determine label width. When working with fonts, the font metrics are more important since a lowercase ‘l’ will have a smaller font width than an uppercase ‘D’. So, while ‘llll’ is “longer” than ‘DDD’, it may not be wider (especially in a sans-serif font, but likely in any font):

To solve this problem, we need to use a temporary Cairo surface to compute font metrics for each label & value so we know what the maximum width of both for the starting and ending points. It’s a simple concept & calculation, but very important to ensure everything lines up well.

# find the *real* maximum label width (not just based on number of chars)

```
 
```
```
maxLabelWidth = 0
```
```
maxNumWidth = 0
```
```
 
```
```
for k in sorted(startKeys):
```
```
	s1 = starts[k]
```

	xbearing, ybearing, sWidth, sHeight, xadvance, yadvance = (cr.text_extents(s1))

	if (sWidth > maxLabelWidth) : maxLabelWidth = sWidth

	xbearing, ybearing, startMaxLabelWidth, startMaxLabelHeight, xadvance, yadvance = (cr.text_extents(str(k)))

	if (startMaxLabelWidth > maxNumWidth) : maxNumWidth = startMaxLabelWidth

```
 
```
```
sWidth = maxLabelWidth
```
```
startMaxLabelWidth = maxNumWidth
```
```
 
```
```
maxWidth = 0
```
```
maxNumWidth = 0
```
```
 
```
```
for k in sorted(endKeys):
```
```
	e1 = ends[k]
```

	xbearing, ybearing, eWidth, eHeight, xadvance, yadvance = (cr.text_extents(e1))

	if (eWidth > maxLabelWidth) : maxLabelWidth = eWidth

	xbearing, ybearing, endMaxLabelWidth, endMaxLabelHeight, xadvance, yadvance = (cr.text_extents(str(k)))

	if (endMaxLabelWidth > maxNumWidth) : maxNumWidth = endMaxLabelWidth

```
 
```
```
eWidth = maxLabelWidth
```
```
endMaxLabelWidth = maxNumWidth
```

I tossed some “anomalies” into the sample data set to show both how adaptable the vertical scale is as well as demonstrate the label alignments:

Updates are in github.

Slopegraphs In Python – colors, fonts & alignment

2012-05-29 – 21:49
Posted in Charts & Graphs, Development, Python
Tagged slopegraph, table-chart, tufte
Comments (1)

On the heels of last evening’s release of Slopegraphs in Python post comes some minor tweaks:

Complete alignment control of labels & and values
Colors (for background, lines, labels & values) — I picked a random pattern from Adobe’s Kuler
A font change (to prove width calculations work)

…and a new example slopegraph:

As promised, the latest revisions are in github.

Some notes for aspiring Python/Cairo hackers:

RGB colors are 0-1 in Cairo, so divide your 0-255 values by 255 to get the corresponding Cairo value and make sure you’re doing float arithmetic in Python
It turns out simple Cairo font rendering (ie. non-Pango) does not interpret well in Illustrator from a Cairo PDF surface, so if you do plan on post-processing the slopegraphs, use a postscript/EPS surface:
surface = cairo.PSSurface ("slopegraph.ps", width, height)
surface.set_eps(True)
(I’ll be incorporating an output option as this gets closer to 1.0)
If you do plan on using this at all, grab the Bitstream Vera Serif font (link goes to FontSquirrel, but you can find it almost everywhere as it’s free)

An Homage To Flame

(Sing to the tune of “Fame – Remember My name” …
Here’s some YouTube background music)

They’ve been lookin’ at me, but they never did see—
no, no trace of me did they detect;
Gave me time to collect all the data at rest.
I’ve got so much in me: LUA, zlib & sqlite3–
I can infect the USB in your hand. Don’t you know who I am?
Remember my name [FLAME]

I’ve been around forever. Capturing packets on the fly. [HIGH]
My botnet is comin’ together. When researchers see me they’ll cry. [FLAME]
I even infected Lebanon. Lit up the Middle East with my FLAME. [FLAME]
I’ve been around forever. They will remember my name.

[REMEMBER, REMEMBER, REMEMBER, REMEMBER, REMEMBER, ]

I’m not packed up too tight (I take up 20 megabytes).
With no kill date, I’ll never stop.
Give me your mic and I’ll take all you’ve got to give.
Finding me will be tough. Too much (you’ll say ‘enough’!)
I can ride your net but not break (it). Yeah, I got what it takes.

FLAME!

[REMEMBER, REMEMBER, REMEMBER, REMEMBER, REMEMBER, REMEMBER]

FLAME!…

Slopegraphs in Python

2012-05-28 – 21:01
Posted in Charts & Graphs, Development, Open Source, OS X, Python
Tagged slopegraph, table-chart, tufte
Comments (2)

(NOTE: You can keep up with progress best at github, but can always search on “slopegraph” here or just hit the tag page: “slopegraph” regularly)

I’ve been a bit obsessed with slopegraphs (a.k.a “Tufte table-chart”) of late and very dissatisfied with the lack of tools to make this particular visualization tool more prevalent. While my ultimate goal is to have a user-friendly modern web app or platform app that’s as easy as a “drag & drop” of a CSV file, this first foray will require a bit (not much, really!) of elbow grease to be used.

For those who want to get right to the code, head on over to github and have a look (I’ll post all updates there). Setup, sample & source are also below.

First, you’ll need a modern Python install. I did all the development on Mac OS Mountain Lion (beta) with the stock Python 2.7 build. You’ll also need the Cairo 2D graphics library which built and installed perfectly from source, even on ML, so it should work fine for you. If you want something besides PDF rendering, you may need additional libraries, but PDF is decent for hi-res embedding, converting to jpg/png (see below) and tweaking in programs like Illustrator.

If you search for “Gender Comparisons” in the comments on this post at Tufte’s blog, you’ll see what I was trying to reproduce in this bit of skeleton code (below). By modifying the CSV file you’re using [line 21] and then which fields are relevant [lines 45-47] you should be able to make your own basic slopegraphs without much trouble.

If you catch any glitches, add some tweak or have a slopegraph “wish list”, let me know here, twitter (@hrbrmstr) or over at github.

```
# slopegraph.py
```
```
#
```
```
# Author: Bob Rudis (@hrbrmstr)
```
```
#
```

# Basic Python skeleton to do simple two value slopegraphs

# with output to PDF (most useful form for me...Cairo has tons of options)

```
#
```

# Find out more about & download Cairo here:

```
# http://cairographics.org/
```
```
#
```

# 2012-05-28 - 0.5 - Initial github release. Still needs some polish

```
#
```
```
 
```
```
import csv
```
```
import cairo
```
```
 
```

# original data source: http://www.calvin.edu/~stob/data/television.csv

```
 
```
```
# get a CSV file to work with 
```
```
 
```

slopeReader = csv.reader(open('television.csv', 'rb'), delimiter=',', quotechar='"')

```
 
```
```
starts = {} # starting "points"/
```
```
ends = {} # ending "points"
```
```
 
```

# Need to refactor label max width into font calculations

# as there's no guarantee the longest (character-wise)

```
# label is the widest one
```
```
 
```
```
startLabelMaxLen = 0
```
```
endLabelMaxLen = 0
```
```
 
```

# build a base pair array for the final plotting

# wastes memory, but simplifies plotting

```
 
```
```
pairs = []
```
```
 
```
```
for row in slopeReader:
```
```
 
```

	# add chosen values (need start/end for each CSV row)

	# to the final plotting array. Try this sample with

	# row[1] (average life span) instead of row[5] to see some

```
	# of the scaling in action
```
```
 
```
```
	lab = row[0] # label
```
```
	beg = row[5] # male life span
```
```
	end = row[4] # female life span
```
```
 
```

	pairs.append( (float(beg), float(end)) )

```
 
```

	# combine labels of common values into one string

	# also (as noted previously, inappropriately) find the

```
	# longest one
```
```
 
```
```
	if beg in starts:
```

		starts[beg] = starts[beg] + "; " + lab

```
	else:
```
```
		starts[beg] = lab
```
```
 
```

	if ((len(starts[beg]) + len(beg)) > startLabelMaxLen):

		startLabelMaxLen = len(starts[beg]) + len(beg)

```
		s1 = starts[beg]
```
```
 
```
```
 
```
```
	if end in ends:
```
```
		ends[end] = ends[end] + "; " + lab
```
```
	else:
```
```
		ends[end] = lab
```
```
 
```

	if ((len(ends[end]) + len(end)) > endLabelMaxLen):

		endLabelMaxLen = len(ends[end]) + len(end)

```
		e1 = ends[end]
```
```
 
```

# sort all the values (in the event the CSV wasn't) so

# we can determine the smallest increment we need to use

# when stacking the labels and plotting points

```
 
```

startSorted = [(k, starts[k]) for k in sorted(starts)]

endSorted = [(k, ends[k]) for k in sorted(ends)]

```
 
```
```
startKeys = sorted(starts.keys())
```
```
delta = max(startSorted)
```
```
for i in range(len(startKeys)):
```
```
	if (i+1 <= len(startKeys)-1):
```

		currDelta = float(startKeys[i+1]) - float(startKeys[i])

```
		if (currDelta < delta):
```
```
			delta = currDelta
```
```
 
```
```
endKeys = sorted(ends.keys())
```
```
for i in range(len(endKeys)):
```
```
	if (i+1 <= len(endKeys)-1):
```

		currDelta = float(endKeys[i+1]) - float(endKeys[i])

```
		if (currDelta < delta):
```
```
			delta = currDelta
```
```
 
```

# we also need to find the absolute min & max values

```
# so we know how to scale the plots
```
```
 
```
```
lowest = min(startKeys)
```

if (min(endKeys) < lowest) : lowest = min(endKeys)

```
 
```
```
highest = max(startKeys)
```

if (max(endKeys) > highest) : highest = max(endKeys)

```
 
```

# just making sure everything's a number

# probably should move some of this to the csv reader section

```
 
```
```
delta = float(delta)
```
```
lowest = float(lowest)
```
```
highest = float(highest)
```

startLabelMaxLen = float(startLabelMaxLen)

```
endLabelMaxLen = float(endLabelMaxLen)
```
```
 
```

# setup line width and font-size for the Cairo

# you can change these and the constants should

```
# scale the plots accordingly
```
```
 
```
```
FONT_SIZE = 9
```
```
LINE_WIDTH = 0.5
```
```
 
```

# there has to be a better way to get a base "surface"

# to do font calculations besides this. we're just making

# this Cairo surface to we know the max pixel width

# (font extents) of the labels in order to scale the graph

# accurately (since width/height are based, in part, on it)

```
 
```
```
filename = 'slopegraph.pdf'
```

surface = cairo.PDFSurface (filename, 8.5*72, 11*72)

```
cr = cairo.Context (surface)
```
```
cr.save()
```

cr.select_font_face("Sans", cairo.FONT_SLANT_NORMAL, cairo.FONT_WEIGHT_NORMAL)

```
cr.set_font_size(FONT_SIZE)
```
```
cr.set_line_width(LINE_WIDTH)
```

xbearing, ybearing, sWidth, sHeight, xadvance, yadvance = (cr.text_extents(s1))

xbearing, ybearing, eWidth, eHeight, xadvance, yadvance = (cr.text_extents(e1))

xbearing, ybearing, spaceWidth, spaceHeight, xadvance, yadvance = (cr.text_extents(" "))

```
cr.restore()
```
```
cr.show_page()
```
```
surface.finish()
```
```
 
```

# setup some more constants for plotting

# all of these are malleable and should cascade nicely

```
 
```
```
X_MARGIN = 10
```
```
Y_MARGIN = 10
```
```
SLOPEGRAPH_CANVAS_SIZE = 200
```
```
spaceWidth = 5
```
```
LINE_HEIGHT = 15
```
```
PLOT_LINE_WIDTH = 0.5
```
```
 
```

width = (X_MARGIN * 2) + sWidth + spaceWidth + SLOPEGRAPH_CANVAS_SIZE + spaceWidth + eWidth

height = (Y_MARGIN * 2) + (((highest - lowest + 1) / delta) * LINE_HEIGHT)

```
 
```
```
# create the real Cairo surface/canvas
```
```
 
```
```
filename = 'slopegraph.pdf'
```

surface = cairo.PDFSurface (filename, width, height)

```
cr = cairo.Context (surface)
```
```
 
```
```
cr.save()
```
```
 
```

cr.select_font_face("Sans", cairo.FONT_SLANT_NORMAL, cairo.FONT_WEIGHT_NORMAL)

```
cr.set_font_size(FONT_SIZE)
```
```
 
```
```
cr.set_line_width(LINE_WIDTH)
```

cr.set_source_rgba (0, 0, 0) # need to make this a constant

```
 
```

# draw start labels at the correct positions

# cheating a bit here as the code doesn't (yet) line up

```
# the actual data values
```
```
 
```
```
for k in sorted(startKeys):
```
```
 
```
```
	label = starts[k]
```

	xbearing, ybearing, lWidth, lHeight, xadvance, yadvance = (cr.text_extents(label))

```
 
```
```
	val = float(k)
```
```
 
```

	cr.move_to(X_MARGIN + (sWidth - lWidth), Y_MARGIN + (highest - val) * LINE_HEIGHT * (1/delta) + LINE_HEIGHT/2)

```
	cr.show_text(label + " " + k)
```
```
	cr.stroke()
```
```
 
```

# draw end labels at the correct positions

# cheating a bit here as the code doesn't (yet) line up

```
# the actual data values
```
```
 
```
```
for k in sorted(endKeys):
```
```
 
```
```
	label = ends[k]
```

	xbearing, ybearing, lWidth, lHeight, xadvance, yadvance = (cr.text_extents(label))

```
 
```
```
	val = float(k)
```
```
 
```

	cr.move_to(width - X_MARGIN - eWidth - (4*spaceWidth), Y_MARGIN + (highest - val) * LINE_HEIGHT * (1/delta) + LINE_HEIGHT/2)

```
	cr.show_text(k + " " + label)
```
```
	cr.stroke()
```
```
 
```
```
# do the actual plotting
```
```
 
```
```
cr.set_line_width(PLOT_LINE_WIDTH)
```

cr.set_source_rgba (0.75, 0.75, 0.75) # need to make this a constant

```
 
```
```
for s1,e1 in pairs:
```

	cr.move_to(X_MARGIN + sWidth + spaceWidth + 20, Y_MARGIN + (highest - s1) * LINE_HEIGHT * (1/delta) + LINE_HEIGHT/2)

	cr.line_to(width - X_MARGIN - eWidth - spaceWidth - 20, Y_MARGIN + (highest - e1) * LINE_HEIGHT * (1/delta) + LINE_HEIGHT/2)

```
	cr.stroke()
```
```
 
```
```
cr.restore()
```
```
cr.show_page()
```
```
surface.finish()
```

New SecurID Soft Token Cloning Weakness : What’s The Risk?

I posted a link to Twitter earlier on a recent discovery of the ability to clone RSA SecurID soft tokens:

https://twitter.com/hrbrmstr/status/204908233645764609

It (rightfully so) received some critical responses by @wh1t3rabbit & @wikidsystems since, apart from what the hypesters may say, this is a low-risk weakness.

@Wh1t3Rabbit @hrbrmstr once you own the machine and can enumerate SIDs in AD, aren't you in? And, don't you still need the PIN?
— Nick Owen (@wikidsystems) May 22, 2012

Think about it. Just looking at the two most likely threat actors & actions: an insider trying to siphon off soft tokens and an external attacker using crafted malware to grab soft tokens. The former (most likely) knows your organization is using soft tokens (and probably has one herself). The latter is unlikely to just try to blanket siphon off soft tokens so they’ll have to do some research to target an organization (which costs time/money).

Once a victim (or set of victims) is identified, the cloning steps would have to be perfectly executed (and, I’m not convinced that’s a given). Let’s say that this is a given, though. Now both the insider and external agent have access to the bits to clone a token. It is easier for the insider to get that data, but the external attacker has to exfiltrate successfully it somehow (more complexity/time/cost).

To be useful, the attacker needs the user id, PIN and – in most implementations – a password. An insider would (most likely) know the user id (since she probably has one herself) but that data would require more time/effort/cost to the external attacker (think opportunistic keylogger/screenscraper with successful exfiltration). For both attackers, getting the password requires either social engineering or the use of a keylogger. Even then, there’s a time-limit of 90 days or less (since, if you’re using soft tokens, you probably have a 90 day password policy). That shrinks the amount of time the attack can be successful.

Now, both attackers need to know where this soft token can be used and have direct access to those systems. Again, probably easier for an insider and fairly costly for an external attacker.

Looking at this, there’s definitely a greater risk associated with an insider from this weakness than there is from an external party (as pointed out by the aforementioned twitter commentators). As @wikidsystems further pointed out, this also shows the inherent positives of multi-factor authentication :: you need far more component parts to execute a successful attack, making the whole thing very costly to obtain. Security economics FTW!

My comment has been that if using the TPM store for Windows-based SecurID soft token implementations negates this weakness, then why not do it? Does the added deployment & management complexity really cost that much?

In the end, I would categorize this weakness as a low risk to most organizations using soft tokens with a non-TPM storage configuration. Unless you know you’re a nation-state target (my opine for the origin of the attacker) – and, even then, you’re probably using hard tokens – far too many celestial bodies need to align for this weakness to be exploited successfully.

NOTE: This post was not meant to be a comprehensive risk assessment of the weakness and does not cover all attack scenarios. I left out many, including Windows desktop administrators and privileged script access. I was merely trying to do my part to counter whatever hype ensues from this weakness. Comments on those vectors or the analysis in general are most welcome.

Feudal Japan’s Haiku Clubs A Pre-cursor of Modern-day Twitter?

2012-05-21 – 17:35
Posted in haiku, japan, samurai, twitter
Leave a Comment

I recently finished watching “Japan: Memoirs of a Secret Empire” [ Netflix | PBS | Amazon ]. It’s a three-part documentary that centers around the 16th & 17th centuries (where a vast majority of my favorite samurai movies are set).

In the third part there is a segment on the four-tier class system. It goes into some detail on how the decline of the samurai (due to a time of significant peace) and the increase in trade facilitated the erosion of societal taboos that previously prevented classes from mingling. Apart from the creation of a new merchant-inspired class and an increase in the frequenting of courtesans there was also the mention of “haiku clubs” where (quoting from the documentary) “members chose pen names to obscure their social rank. That way, the classes could mingle freely“.

In a similar way, Twitter (and many forums before it) is the great equalizer with the added similarity of brevity (and often a gravity well for some modern day haiku fanatics). I don’t have any statistics, but I do wonder if Twitter is having a more equalizing impact on societies where class and overt discrimination are still wildly prevalent (think India or Saudi Arabia).

as avatars tweet
modern words become equal
history repeats

Mountain Lion, AirPlay, Screen Captures & DRM

I’ve been wanting to post this entry for a while, but I didn’t have the opportunity to compel an extra pair of hands to assist with some necessary, salient portions of it until tonight.

For those who were hoping Mountain Lion’s AirPlay would be a revolutionary step in the “your content, wherever you want it” battle, I fear you may be in for a bit of a disappointment. Quite by accident, I stumbled upon some eerie signs that location-aware video DRM will be alive and well as an integrated part of Apple’s forthcoming release of Mountain Lion.

Since I have a shiny, new 1080p Apple TV and a device capable of running Mountain Lion, I’ve been experimenting with the awesomeness that is AirPlay. Despite claims to the contrary, most of the time routing MacBook Pro Desktop video & system audio to the tiny black box of happiness works flawlessly (even more so since the recent Apple TV update). It’s been a treat to be able to play owned-backup copies of my favorite samurai videos (I should buy stock in Criterion) with subtitles via VLC.

However, on a lark I tried to play one of my Avengers : Earth’s Mightiest Heroes episodes (we subscribe via iTunes) using QuickTime Player and, much to my chagrin, discovered that there lies within the heart of the tame Mountain Lion, a DRM beast.

The easiest way to show this is with what happens when I try to take a fullscreen snapshot with Skitch as I’m playing the DRM-laden episode:

I managed to get #3 to help me record a video (albeit crappy) of what happens when I try to route the Desktop video to the living room TV via AirPlay: (youtube link)

In case it’s not obvious, the video plays fine on the Desktop (in QuickTime) prior to the AirPlay route, but goes equally as blank when the AirPlay device is chosen, yet reverts to playing when AirPlay is disabled.

This means the API hooks are there to prevent DRM-laden content from being used with AirPlay (or snapped via screen capture) and that, in turn, means your hopes of AirPlaying Hulu, Netflix and Amazon Video content may be dashed despite all of those services working now (in the betas).

Video is the last content area to understand the need to be open. Amazon & Apple sell untainted music and even Tor is going DRM free (joining #spiffy folks like O’Reilly Media).

I own that episode of The Avengers yet am not able to do with it as I please. Yes, I could have streamed it over the Internet from iCloud to the Apple TV or even routed it via the local iTunes to the Apple TV, but I wanted to use QuickTime (though, just for a test). What’s to stop Apple or other companies from requiring a special streaming license if you want the ability to use AirPlay or just disabling it altogether in favor of forcing you to use something as horrific as Google TV (full disclosure, I own a Logitech Google TV box, too)?

Combine these restrictions with the inevitable “you will only be able to use Apple-authorized apps in Mac OS” in a post-Mountain Lion release and your hopes of using VLC (or any other player that will not conform to draconian rules) to bypass this silliness will be equally as dashed as your naive AirPlay ones. If there’s no guarantee you’ll be able to get your content to the screen of your choice, why would you choose to remain legal (moral arguments notwithstanding)?

I hope, in the long run, Apple manages to figure out an sane, amenable solution to this silliness. In the meantime, I’m going to pop in a DVD and crank through some Godzilla flicks. At least I can be fairly certain that should work for quite a while longer.

Street sign photo via jbonnain