Simple fuzzers like the ones developed by Barton Miller are often referred to as “Black Box” fuzzers. They have no knowledge about the inner workings of the programs, and only generate inputs and check to see if the program crashed. The main advantage with this approach is that it can test almost any program with minimal effort – the disadvantage is that randomly generating the input that triggers a bug can take a prohibitive amount of time when you only want to test one piece of software.
“Smarter”, more modern fuzzers analyse some parts of the target program in order to more effectively search the input space for bugs. Their inputs are often semi-random, and use metrics such as code coverage from the program being tested to modify the next set of inputs.
This is referred to as “Grey Box Fuzzing”, since it relies on being able to gather some information on the program being tested. Compared to black box fuzzers, they are able to find bugs that are more hidden and harder to trigger. One popular fuzzer using this technique is American Fuzzy Lop, which has been used to find bugs and security vulnerabilities in for example OpenSSL, OpenSSH, Android, iOS and Linux.
While a fuzzer will identify a bug for you and give you the input that caused a crash, it does not automatically debug the issue for you. Often times, a fuzzer will find multiple inputs that trigger the same bug. Identifying the issue, understanding its potential impact and fixing it will still take manual effort and engineering resources.
It’s also important to note that while Fuzz testing is automatic, it is a complement and not a replacement to other testing techniques. A fuzzer can not determine if a program behaves as it should, as it is typically only able to identify more serious failures that result in crashes. High degree of test coverage with unit tests and integration tests will always be vital.
Employing fuzzy testing as part of an automatic testing pipeline is a great way to secure your application, and find bugs and issues before the bad guys do.
Kristian Alvarez, a software engineer passionated about security and DevOps.