Speaker: Chris Eng \/ Veracode<\/p>\n
Every major infosec company publishes quarterly\/yearly summary reports. Some based on survey, some based on real captured data.<\/p>\n
Recognizing the Narrative<\/strong><\/p>\n Every fancy looking infosec metrics report is a marketing vehicle; each has different perspectives; no consistency, but you can figure out the framing by looking at the exec summary or ToC; other times it may require real digging. Need to understand “what they are selling”. The text in the report is there to back up the narrative.<\/p>\n \u00a0<\/p>\n Veracode Report Narrative<\/em><\/p>\n (goal: use Veracode to analyze third party apps :-)<\/p>\n \u00a0<\/p>\n Trustwave Report Narrative<\/em><\/p>\n (goal: “we can help…we are good at this stuff”)<\/em><\/p>\n WhiteHat Report<\/em><\/p>\n (differs in goal from previous WH reports)<\/p>\n \u00a0<\/p>\n Bottom line: try to understand the framing<\/strong> &\u00a0goal<\/strong> when reviewing the narrative<\/p>\n \u00a0<\/p>\n Using Stats Responsibly<\/strong><\/p>\n Sample distribution review\/discussion<\/p>\n normal distribution curves can still vary, but overall shape remains the same (std deviations, & avg)<\/p>\n bimodal distribution (two peaks)…may miss if you report only on averages<\/p>\n [game: Guess the Report Jeopardy! used primarily to show the pervasiveness of the use of averages]<\/p>\n [side-talk: discussion about different distributions by different sources]<\/p>\n \u00a0<\/p>\n (\/me: this is very interesting)<\/p>\n Would a table of # of flaws per 1K lines of code per language be enough?<\/p>\n Would adding 1st quartike, median and 3rd quartile provide more insight?<\/p>\n Will this help understand the anomalies? Will it help prioritize?<\/p>\n How do we ensure normalized data for comparison?<\/p>\n [side-talk: what’s a “line of code”…same problem in app bug analysis]<\/p>\n [side-talk: Truth in stats: “What’s the question? What matters?”]<\/p>\n Can you overdo it? Yes.<\/p>\n \u00a0<\/p>\n [game: continued]<\/p>\n Power analysis can be use to determine the statisticaly significant sample size required to ensure the prob of error is acceptable.<\/p>\n\n
\n
<\/em><\/p>\n\n