Javascript Archives - Page 3 of 4

Category Archives: Javascript

Preparing For The February 2014 Book Launch

2013-11-26 – 19:05
Posted in Data Analysis, data driven security, Data Visualization, DataVis, DataViz, HTML5, Information Security, Javascript
Leave a Comment

Data Driven Security launches in February 2014. @jayjacobs & I have seen half of the book in PDF form so far and it’s almost unbelievable that this journey is almost over.

Data_Driven_Security___Amazon_Sales_Rank_Tracker

We setup a live Amazon “sales rank” tracker over at the book’s web site and provided some Python and JavaScript code to show folks how use the AWS API in conjunction with the dygraphs charting library to do the same for any ISBN. In the coming weeks, we’ll have a Google App Engine component you can clone to setup something similar without the need for your own server(s).

Scraping Content From Google Groups

2013-09-25 – 11:57
Posted in Development, hacks, Javascript
Comments (1)

I was helping a friend out who wanted to build a word cloud from the text in Google Groups posts. If you’ve made any efforts to try to get content out of Google Groups you know that the only way to do so is to ensure you subscribe to the group posts via e-mail, then extract all those messages. If you don’t e-mail subscribe to a group, there really is no way to create an archive of the content.

After hacking around a bit and failing, I pulled up the mobile version of the group. You can do that for any Google Group by using the following URL and filling in GROUPNAME for the group you’re interested in: https://groups.google.com/forum/m/#!topic/GROUPNAME.

Then, you’ll need to navigate to a thread, use the double-down arrow to expand all the items in the thread, open up the JavaScript inspector on one of the posts and look for <div dir="ltr">. If that surrounds the post, the following hack will work. Google only seems to add this left-to-right attribute on newer groups, so if you have an older group you need to work with, you’ll need to figure out a different selector (which is coming up in a bit).

With all of the posts expanded, paste the following code into the JavaScript console:

nl = document.querySelectorAll('[dir=ltr]');
s="" ; 
for (i=0; i<nl.length; i++) {
  s = s + nl[i].textContent + "<br/><br/>";
}; 
nw = window.open(); 
nd = nw.document; 
nd.write(s); 
nd.close()

and hit return (I have it spaced out in the code above just for clarity; it will all fit on one line which makes it easier to execute in the console).

Untitled_and_input_text_not_updated_-_Google_Groups

You should get a new browser window (so, you may need to temporarily enable popups on Google Groups for this to work) with the text of all the posts in it. I only put the double <br/> tags in there for the purposes of this example. I just needed the raw text, but you can mark the posts any way you’d like.

You can tweak this hack in many ways to pull as much post metadata as you need since it’s all wrapped in heavily marked <div>s and the base technique should work in a GreaseMonkey or TamperMonkey userscript for those of you with time to code one up.

This hack only lessens the tedium a small amount. You still need to go topic by topic in the group if you want all the content. There’s probably a way to get that navigation automation coded into the script as well. Thankfully, I didn’t need to do that this time around.

If you have other ways to free Google Groups content, drop a note in the comments.

Visualizing Risky Words

2013-03-06 – 23:05
Posted in d3, Data Visualization, DataVis, DataViz, Javascript
Leave a Comment

NOTE: Parts [2], [3] & [4] are also now up.

Inspired by a post by @bfist who created the following word cloud in Ruby from VZ RISK INTSUM posts (visit the link or select the visualization to go to the post):

I ♥ word clouds as much as anyone and usually run Presidential proclamations & SOTU addresses through a word cloud generator just to see what the current year’s foci are.

However, word clouds rarely provide what the creator of the visualization intends. Without performing more strict corpus analysis, one is really just getting a font-based frequency counter. While pretty, it’s not a good idea to derive too much meaning from a simple frequency count since there are phrase & sentence structure components to consider as well as concepts such as stemming (e.g. “risks” & “risk” are most likely the same thing, one is just plural…that’s a simplistic definition/example, though).

I really liked Jim Vallandingham’s Building a Bubble Cloud walk through on how he made a version of @mbostock’s NYTimes convention word counts and decided to both run a rudimentary stem on the VZ RISK INTSUM corpus along with a principal component analysis [PDF] to find the core text influencers and feed the output to a modified version of the bubble cloud:

You can select the graphic to go to the “interactive”/larger version of it. I had intended to make selecting a circle bring up the relevant documents from the post corpus, but that will have to be a task for another day.

It’s noteworthy that both @bfist’s work and this modified version share many of the same core “important” words. With some stemming refinement and further stopword removal (e.g. “week” was in the original run of this visualization and is of no value for this risk-oriented visualization, so I made it part of the stopword list), this could be a really good way to get an overview of what the risky year was all about.

I won’t promise anything, but I’ll try to get the R code cleaned up enough to post. It’s really basic tm & PCA work, so no rocket-science degree is required. Fork @vlandham’s github repo & follow the aforelinked tutorial for the crunchy D3-goodness bits.

Election Trackers Moved From Flash to HTML5

2012-11-27 – 18:56
Posted in Browsers, Charts & Graphs, d3, HTML5, Javascript
Leave a Comment

For those inclined to click, I was interviewed by Fahmida Rashid (@fahmiwrite) over at Sourceforge’s HTML5 center a few weeks ago (right after the elections) due to my tweets on the use of HTML5 tech over Flash. Here’s one of them:

https://twitter.com/hrbrmstr/status/266006111256207361

While a tad inaccurate (one site did use Flash with an HTML fallback and some international sites are still stuck in the 1990s), it is still a good sign of how the modern web is progressing.

I can honestly say I’ve never seen my last name used so many times in one article :-)

More DIY ZeroAccess GeoIP Fun : jQuery/D3 Choropleths

2012-10-05 – 19:48
Posted in Charts & Graphs, d3, Javascript
Comments (1)

In the spirit of the previous example this one shows you how to do a quick, country-based choropleth in D3/jQuery with some help from the command-line since not everyone is equipped to kick out some R and most folks I know are very handy at a terminal prompt.

I took the ZeroAccessGeoIPs.csv file and ran it through a quick *nix one-liner to get a JSON-ish associative array of country abbreviations to botnet counts in that country:

cat ZeroAccessGeoIPs.csv | cut -f1,1 -d\,| sort | uniq -c | sort -n | tr "[:upper:]" "[:lower:]" | while read a b ; do echo "{ \"$b\" : \"$a\" }," ; done > botcounts.js

I found a suitable SVG world map on Wikipedia that had id="ABBREV" country groupings. This is not the most #spiffy map (and, if you dig a bit more than I did, you’ll probably find a better one) but it works for this example and shows you how quickly you can find the bits you need to get a visualization together.

With that data and the SVG file pasted into the same HTML document, it’s a simple matter of generating a gradient with some d3 magic:

color = d3.scale.log().domain([1,47880]).range(["#FFEB38","#F54747"]);

and, then, looping over the associative array while using said color range to fill in the country shapes:

 $.each(botcounts, function(key, value) {
    $('#' + key).css('fill',color(value))
  });
}) ;

we get:

You can view the full, larger example on this separate page where you can do a view-source to see the entire code. I really encourage you to do this as you’ll see that there are just a handful of lines of your own code necessary to make a compelling visualization. Sure, you’ll want to add a legend and some other styling, but the basics can be done in – literally – minutes, leaving customized details to your imagination & creativity.

The entire map could have been done in D3, but I only spent about 5 minutes on the entire exercise (including the one-liner) and am still more comfortable in jQuery than I am in D3. I did this also to show that it’s perfectly fine (as Mainers are wont to say) to do pre-processing and hard-coding when cranking out visualizations. The goal is to communicate something to your audience and there are no hard-and-fast rules governing this process. As with any coding, if you think you’ll be doing this again it is a wise idea to make the solution more generic, but there’s nothing wrong with taking valid shortcuts to get results out as quickly as possible.

Definitely feel invited to share your creations in the comments, especially if you find a better map!

VERIS Community :: JSON vs XML

2012-07-20 – 21:06
Posted in Information Security, Javascript, VERIS
Comments (2)

You may not be aware of the fact that the #spiffy Verizon Biz folk have some VERIS open source components, one of which is the XML schema for the “Vocabulary for Event Recording and Incident Sharing”.

While most Java-backends will readily slurp up and spit back archaic XML data, the modern web is a JSON world and I wanted to take a stab at encoding the sample incident in JSON format since I’m pretty convinced this type of data is definitely a NoSQL candidate and that JSON is the future.

I didn’t run this past the VZB folk prior to the post, but I think I got it right (well, it validates, at least :-) :

```
{
```
```
  "VERIS_community": {
```
```
    "incident": {
```
```
      "incident_uid": "String",
```
```
      "handler_id": "String",
```
```
      "security_compromise": "String",
```

      "related_incidents": { "related_incident_id": "String" },

```
      "summary": "String",
```
```
      "notes": "String",
```
```
      "victim": {
```
```
        "victim_id": "String",
```
```
        "industry": "000",
```

        "employee_count": "25,001 to 50,000",

```
        "location": {
```
```
          "country": "String",
```
```
          "region": "String"
```
```
        },
```
```
        "revenue": {
```
```
          "amount": "0",
```
```
          "iso_currency_code": "USD"
```
```
        },
```
```
        "security_budget": {
```
```
          "amount": "0",
```
```
          "iso_currency_code": "USD"
```
```
        },
```
```
        "notes": "String"
```
```
      },
```
```
      "agent": [
```
```
        {
```
```
          "motive": "String",
```
```
          "role": "String",
```
```
          "notes": "String"
```
```
        },
```
```
        {
```
```
          "type": "External",
```
```
          "motive": "String",
```
```
          "role": "String",
```
```
          "notes": "String",
```
```
          "external_variety": "String",
```
```
          "origins": {
```
```
            "origin": {
```
```
              "country": "String",
```
```
              "region": "String"
```
```
            }
```
```
          },
```
```
          "ips": { "ip": "String" }
```
```
        },
```
```
        {
```
```
          "type": "Internal",
```
```
          "motive": "String",
```
```
          "role": "String",
```
```
          "notes": "String",
```
```
          "internal_variety": "String"
```
```
        },
```
```
        {
```
```
          "type": "Partner",
```
```
          "motive": "String",
```
```
          "role": "String",
```
```
          "notes": "String",
```
```
          "industry": "0000",
```
```
          "origins": {
```
```
            "origin": {
```
```
              "country": "String",
```
```
              "region": "String"
```
```
            }
```
```
          }
```
```
        }
```
```
      ],
```
```
      "action": [
```

        { "notes": "Some notes about a generic action." },

```
        {
```
```
          "type": "Malware",
```
```
          "notes": "String",
```
```
          "malware_function": "String",
```
```
          "malware_vector": "String",
```
```
          "cves": { "cve": "String" },
```

          "names": { "name": "String" },

          "filenames": { "filename": "String" },

          "hash_values": { "hash_value": "String" },

          "outbound_IPs": { "outbound_IP": "String" },

          "outbound_URLs": { "outbound_URL": "String" }

```
        },
```
```
        {
```
```
          "type": "Hacking",
```
```
          "notes": "String",
```
```
          "hacking_method": "String",
```
```
          "hacking_vector": "String",
```
```
          "cves": { "cve": "String" }
```
```
        },
```
```
        {
```
```
          "type": "Social",
```
```
          "notes": "String",
```
```
          "social_tactic": "String",
```
```
          "social_channel": "String",
```
```
          "email": {
```

            "addresses": { "address": "String" },

            "subject_lines": { "subject_line": "String" },

```
            "urls": { "url": "String" }
```
```
          }
```
```
        },
```
```
        {
```
```
          "type": "Misuse",
```

          "notes": "Notes for a misuse action.",

```
          "misuse_variety": "String",
```
```
          "misuse_venue": "String"
```
```
        },
```
```
        {
```
```
          "type": "Physical",
```

          "notes": "Notes for a physical action.",

```
          "physical_variety": "String",
```

          "physical_location": "String",

```
          "physical_access": "String"
```
```
        },
```
```
        {
```
```
          "type": "Error",
```

          "notes": "Notes for a Error action.",

```
          "error_variety": "String",
```
```
          "error_reason": "String"
```
```
        },
```
```
        {
```
```
          "type": "Environmental",
```

          "notes": "Notes for a environmental action.",

          "environmental_variety": "String"

```
        }
```
```
      ],
```
```
      "assets": {
```
```
        "asset_variety": "String",
```
```
        "asset_ownership": "String",
```
```
        "asset_hosting": "String",
```
```
        "asset_management": "String",
```
```
        "os": "String",
```
```
        "notes": "String"
```
```
      },
```
```
      "attribute": [
```
```
        { "notes": "String" },
```
```
        {
```

          "type": "ConfidentialityPossession",

```
          "notes": "String",
```
```
          "data_disclosure": "String",
```
```
          "data": {
```
```
            "data_variety": "String",
```
```
            "amount": "0"
```
```
          },
```
```
          "data_state": "String"
```
```
        },
```
```
        {
```

          "type": "AvailabilityUtility",

```
          "notes": "String",
```

          "availability_utility_variety": "String",

          "availability_utility_duration": "String"

```
        }
```
```
      ],
```
```
      "timeline": {
```

        "timestamp_first_known_action": {

```
          "year": "2001",
```
```
          "month": "--12",
```
```
          "day": "---17",
```
```
          "time": "14:20:00.0Z"
```
```
        },
```

        "timestamp_data_exfiltration": {

```
          "year": "2001",
```
```
          "month": "--12",
```
```
          "day": "---17",
```
```
          "time": "14:20:00.0Z"
```
```
        },
```

        "timestamp_incident_discovery": {

```
          "year": "2001",
```
```
          "month": "--12",
```
```
          "day": "---17",
```
```
          "time": "14:20:00.0Z"
```
```
        },
```
```
        "timestamp_containment": {
```
```
          "year": "2001",
```
```
          "month": "--12",
```
```
          "day": "---17",
```
```
          "time": "14:20:00.0Z"
```
```
        },
```

        "timestamp_initial_compromise": {

```
          "year": "2001",
```
```
          "month": "--12",
```
```
          "day": "---17",
```
```
          "time": "14:20:00.0Z"
```
```
        },
```
```
        "timestamp_investigation": {
```
```
          "year": "2001",
```
```
          "month": "--12",
```
```
          "day": "---17",
```
```
          "time": "14:20:00.0Z"
```
```
        }
```
```
      },
```
```
      "discovery_method": "String",
```
```
      "control_failure": "String",
```
```
      "corrective_action": "String",
```
```
      "loss": {
```
```
        "loss_variety": "String",
```
```
        "loss_amount": {
```
```
          "amount": "0",
```
```
          "iso_currency_code": "USD"
```
```
        }
```
```
      },
```
```
      "impact_rating": "String",
```
```
      "impact_estimate": {
```
```
        "amount": "0",
```
```
        "iso_currency_code": "USD"
```
```
      },
```
```
      "certainty": "String"
```
```
    }
```
```
  }
```
```
}
```

I believe I’d advocate for the “timestamps” to be more timestamp-y in the JSON version (the dashes do not make much sense to me even in the XML version) and any fields with min/max range values to be separated to actual min & max fields. I’m going try to find some cycles to mock up a MongoDB / Node.js sample to show how this JSON format would work. At a minimum, even a rough conversion from XML to JSON when requested by a browser would make it easier for client-side data rendering/manipulation.

If you’re not thinking about using VERIS for documenting incidents or hounding your vendors to enable easier support for it, you should be. If you’re skittish about recording incidents anonymously into the VERIS Community, you should get over it (barring capacity constraints).

Just Like The Federation

[@hrbrmstr starts working in javascript again]
The Internets: What do you think?
@hrbrmstr: It’s vile.
The Internets: I know. It’s so bubbly and cloying and happy.
@hrbrmstr: Just like the Federation.
The Internets: And you know what’s really frightening? If you develop with it enough, you begin to like it.
@hrbrmstr: It’s insidious.
The Internets: Just like the Federation.

(With apologies to ST:DS9)

Slopegraphs in Python – Gratuitous Raphaël Animations

2012-06-12 – 13:21
Posted in Charts & Graphs, Development, Javascript, Programming, Python
Tagged slopegraph, table-chart, tufte
Leave a Comment

UPDATE: It seems my use of <script async> optimization for Raphaël busted the inline slopegraph generation. Will work on tweaking the example posts to wait for Raphaël to load when I get some time.

So, I had to alter this to start after a user interaction. It loaded fine as a static, local page but seems to get a bit wonky embedded in a complex page. I also see some artifacts in Chrome but not in Safari. Still, not a bad foray into basic animation.

Animate Slopegraph