Assessing the security of devices by measuring how many difficult things the programmers tried to do

The Cyber Independent Testing Lab is a security measurement company founded by Mudge Zadko (previously), late of the Cult of the Dead Cow and l0pht Heavy Industries and the NSA's Tailored Access Operations Group; it has a unique method for assessing the security of devices derived from methods developed by Mudge at the NSA.

Rather than parsing through sourcecode (static analysis) or attempting to disrupt the operations of running code (dynamic analysis), CIT uses "binary analysis," combing through the compiled firmware of target devices and looking for signs that the programmers who created that firmware made use of libraries and techniques that are hard to implement correctly, and whose incorrect implementation results in serious security vulnerabilities. In other words, they're not looking at whether the code is secure: they're looking at how hard it would be to make the code secure, and assuming that programmers who chose the hardest-to-secure methods probably made exploitable errors.

In August, CIT released an important report on IoT devices, extracting the firmware for these devices from the updates on the manufacturers' websites and conducting longitudinal analyses of these firmwares to see how secure they were and whether they trended towards better or worse security. They analyzed 22 manufacturers' products -- 1,294 in all -- spanning 4,956 firmware versions spread across 3,333,411 binaries.

You can probably guess where this is heading: over a 15-year dataset, every vendor's security practices worsened over time; updates were more likely to introduce insecure techniques, rather than hardening devices. Read the rest