Automating defense versus offense

I just took a look at the Cyber Grand Challenge, a DARPA sponsored event that will showcase systems that can play CTF (capture the flag) autonomously.

This event scares me.

Developing automated attackers and automated defenders might appear to be a way develop techniques to automatically harden software.  Let the bots slug it out in the simplified, safe environment of the challenge and then, once they’ve proven themselves, throw them loose in the real world to defend real systems (or, at least, adapt their techniques to build practical defenses).

I am certain it won’t work out this way.

The attack techniques developed will generalize and will be very good at finding flaws in real systems.  The defensive techniques, however, will not generalize to the real world.  Thus the outcome of this challenge will be even better ways to attack systems and little improvement in protecting systems.

This difference will occur because in the real world defenses have to work while protecting normal system functioning.  The hard part about defense is not stopping the attacks, it is stopping the attacks while keeping your systems up and your users happy.  (After all, you can always stop the attack by just turning your system off.)  Sure, CTF competitions feature services that have to be kept running; these services are nothing like real-world services though, even when they are running the same binaries, simply because they aren’t being required to provide real services to real users.

Simulating real-world behavior accurately is equivalent to building a detector for anomalous behavior.  If you know what makes it “real”, you know what doesn’t belong.  It thus is not easy to do.  Past efforts in computer security to simulate realistic computer behavior for testing purposes have failed miserably (e.g., see John McHugh’s critique of the late 1990’s DARPA intrusion detection evaluations).

The Cyber Grand Challenge makes no effort to simulate a realistic environment; in fact, it was designed to emphasize reproducibility and determinism, two qualities production systems almost never have.  In this sort of environment it is easy to detect attacks and it is easy to verify that a response strategy does not harm the main defense.

The attackers are playing a game that is very close to what real-world attackers face.  The defenders, however, are facing a much simplified challenge that leaves out all of the really hard aspects of the problem.  Note this even goes for software patching, as the hard part of patching is making sure you didn’t miss any corner cases.  When legitimate traffic has no corner cases, you can get away with being a lot sloppier.

On the attack side clearly things are working when you have systems that  can find vulnerabilities that weren’t inserted intentionally (slide 37).  I didn’t see any, and I don’t expect to see any novel defenses, at least none that would ever work in practice.

Attacking is easy, defending is hard.  Automating defense is fundamentally different from automating attacks.  Only when we accept the true nature of this difference will we be able to change the balance of power in computer security.


The Passing of a Pioneer

Today I learned that John Holland passed away.  John was my “grand advisor”, as he was the Ph.D. advisor to Stephanie Forrest, my Ph.D. advisor.  Thus while I had only met John briefly, his work has profoundly influenced my own.

What most impresses me about John’s work is his clear dissatisfaction with his past work.  He developed genetic algorithms and could have spent his entire career on them; yet he went on to develop other major systems such as learning classifier systems and Echo.  John understood that the models of biology that he gave to computer science only captured small fragments of the richness of living systems; thus, while others have spent their careers elaborating on his past work, he kept working to capture more of that richness.  He knew how far we had to go.

The world is poorer for losing his future insights.

Exceptional Intelligence

This morning I was reading an article about Larry Page’s evolution at Google and it made me reflect on the kinds of smarts that Google and others are embedding in the devices that surround us.

Whether it is Microsoft’s Clippy’s failed attempts at being helpful or Siri‘s inability to simple variations to queries that it otherwise would understand, most attempts at computational intelligence tend to work reasonably well within narrow domains and perform very badly outside of them. If the boundaries between expertise and incompetence are clear, the tools can be a joy to use. When the boundary is fuzzy, we become frustrated with the technology’s limits and often stop using it altogether. If you can ask about the weather in ten ways but Siri can only understand three, and you can’t easily remember which is the “right” way to ask about the weather…well, why not just go and tap on the weather app and get the right results 100% of the time?

This rigidity of expectations – only being able to handle inputs that fit within a narrow range – points to the true limitation of current “intelligent” technology. In everyday life we associate intelligence not with getting common things right but with graceful, creative handling of the exceptional. The handling of the exceptional, however, is where most approaches to artificial intelligence break down. This limitation influences the core design of virtually every computational artifact that we build. Learning how to transcend it is, I think, the core problem of computer science.