Java Fuzzing with Jazzer compared to Symflower

Fuzzing is a testing technique where random values are generated as inputs to find unexpected behavior such as crashes and security issues. Previously we looked at the new Golang release 1.18 which includes native fuzzing and our Core Technology blog series contained a post that compares symbolic execution to different other testing techniques, including fuzzing. Today, we will dive into the Java world and check out the most popular Java fuzzing solution: Jazzer.

We will explore how Jazzer is used to automatically generate malicious inputs for Java programs, and how it compares to Symflower, which can automatically generate unit tests to uncover bugs and errors in your code. With the help of Jazzer, many bugs - some of them even in the OpenJDK - were found already. Also, as of March 2021, Jazzer is officially part of OSS-Fuzz, Google's cloud fuzzing engine. It should be noted that Jazzer is a pure "bug detection" utility that finds reproducers for errors in user code. Symflower can do the same, but provides additional functionalities to boost developer productivity, like generating high coverage unit tests and providing test templates for the software developer or tester.

Tool Setup

We will now go through the process of installing both Symflower and Jazzer so you can easily follow along on your local machine. First, we create a new folder fuzz where we can place everything we need.

For Symflower, we can just head over to get.symflower.com to install either the CLI version or an IDE plugin. No matter which version we choose, we just need to run the installer and are immediately good to go.

To install Jazzer, we need to download the executable and its dependencies from the GitHub releases page. We then extract jazzer, jazzer_agent_deploy.jar and jazzer_api_deploy.jar into our fuzz folder to have them at the ready. For a permanent installation, we'd need to move these files to where our PATH and Java can pick up on them.

Example 1: Validating a username

Let's look at a very simple example first, to get a feel for how the different tools are used. We assume that we have some user database or login system and want to make sure that the usernames consist of only lowercase letters. From this snippet we can see that we just loop over all characters in the input string and throw an IllegalArgumentException when we encounter an invalid character.

// fuzz/example1/Strings.javapublic class Strings {    public static void validateName(String in) throws IllegalArgumentException {        for (int i = 0; i < in.length(); i++) {            char c = in.charAt(i);            if (c < 'a' || c > 'z') {                throw new IllegalArgumentException("name must consist of letters");            }        }    }}

Jazzer has an "autofuzz" feature that automatically scans the code one wants to analyze and properly matches the input data types to the program input. In this case we will only tell it that we want to analyze our validateName function and it will automatically figure out that it has to generate random strings as inputs. One downside of Jazzer's "autofuzz" mode, however, is that it only detects unexpected Exceptions from the user code. Here though, we actually expect that there are problems with invalid names since we added a throws IllegalArgumentException to the method body.

This tells Jazzer that IllegalArgumentExceptions are fine for us, so no test cases fulfilling the condition c < 'a' || c > 'z' will be generated. As the behavior of only looking for unexpected exceptions cannot be deactivated for Jazzer, we must add a little hack to actually provoke full coverage of the function.

((Object)null).hashCode();// throw new IllegalArgumentException("name must consist of letters");

To analyze this version with Jazzer then, we first need to compile the code with javac fuzz/example1/Strings.java, then we can execute the fuzzer by running ./jazzer --cp=$PWD/fuzz/example1 --autofuzz=Strings::validateName. We will see that two files Crash-<hash> are immediately generated, and the fuzzer keeps going, generating more and more random input strings.

public class Crash_b6334b553dfdbd68c65c21aa59b75ae8dafc02ca {  public static void main(String[] args) throws Throwable {    Strings.validateName(".");  }}public class Crash_da39a3ee5e6b4b0d3255bfef95601890afd80709 {  public static void main(String[] args) throws Throwable {    Strings.validateName((java.lang.String) null);  }}

This shows that Jazzer successfully generated an input that provokes the exception, and additionally found the case where a "null" string is used. After half a minute, the fuzzer gives up to generate more values and terminates with an "out of memory" message. Many more random strings were constructed but none resulted in additional interesting behavior.

With Symflower, we can just run the tool without any additional modifications necessary. This is done by either executing the symflower command in the console in our "fuzz/example1" directory, or - if we have an IDE plugin installed - triggering Symflower right from your IDE. The automatic unit test generation takes roughly a few seconds and results in a file StringsSymflowerTest.java.

@Test // (expected = NullPointerException.class)public void validateName1() throws IllegalArgumentException {    String in = null;    Strings.validateName(in);}@Testpublic void validateName2() throws IllegalArgumentException {    String in = "";    Strings.validateName(in);}@Test // (expected = IllegalArgumentException.class)public void validateName3() throws IllegalArgumentException {    String in = "\u0001";    Strings.validateName(in);}@Testpublic void validateName4() throws IllegalArgumentException {    String in = "z";    Strings.validateName(in);}@Test // (expected = IllegalArgumentException.class)public void validateName5() throws IllegalArgumentException {    String in = "\u00ef";    Strings.validateName(in);}

The Symflower result also includes the cases where a non-letter and a "null" string is present. Though, since these are unit tests, we also have cases that don't provoke any error but simply exercise our whole example code. While Jazzer found the errors too, Symflower additionally computed a complete test suite for our example with additional information on the outcomes.

Jazzer found cases for the problems very fast. However, it continued trying to find additional ones, even though there are none. Symflower on the other hand finished fast with this example. However, it could be faster. We are working hard on improving the performance of Symflower, so if you find some example that takes a while to process, please let us know through our public issue tracker.

Example 2: Modeling a simple train station

Now, we present a more complex scenario where we model a train station and different types of trains. We check if the station can accept a request to have a train stop, and depending on the type of the train, some conditions have to apply for the train to be allowed to stop at the station.

public class Train {}public class FreightTrain extends Train {    int weight;}public class PassengerTrain extends Train {    int length;}public class Station {    int weightLimit;    int lengthLimit;    public void accept(Train t) throws IllegalArgumentException {        if (t instanceof PassengerTrain) {            PassengerTrain pt = (PassengerTrain)t;            if (lengthLimit > pt.length) {                throw new IllegalArgumentException("train cannot safely stop within station borders");            }        } else if (t instanceof FreightTrain) {            FreightTrain ft = (FreightTrain)t;            if (weightLimit > ft.weight) {                throw new IllegalArgumentException("train cannot safely use the station tracks");            }        } else {            throw new IllegalArgumentException("unknown train type");        }    }}

As before, we need to manually provoke a "NullPointerException" for Jazzer to pick up on our unwanted states.

((Object)null).hashCode();// throw new IllegalArgumentException("<error message>");

We note that for Jazzer's "autofuzz" functionality, it is necessary that all of these classes are public, so they need to be placed into their own respective files in fuzz/example2. Then, we must again compile our code with javac fuzz/example2/*.java before we can start the fuzzer with ./jazzer --cp=$PWD/fuzz/example2 --autofuzz=Station::accept. The fuzzer starts generating values, but won't find the more interesting critical inputs any time soon. However, we see that the simple "null" case is present:

public class Crash_adc83b19e793491b1c6ea0fd8b46cd9f32e592fc {  public static void main(String[] args) throws Throwable {    (((java.util.function.Supplier<Station>) (() -> {Station autofuzzVariable0 = new Station(); return autofuzzVariable0;})).get()).accept((Train) null);  }}

In the meantime (in another terminal window) we can check how Symflower performs. We just run symflower in the fuzz/example2 directory, or - if we're working with an IDE plugin - trigger the analysis from within the IDE. Within seconds, we obtain the following test cases.

@Test // (expected = IllegalArgumentException.class)public void accept1() throws IllegalArgumentException {    Station s = new Station();    Train t = null;    s.accept(t);}@Testpublic void accept2() throws IllegalArgumentException {    Station s = new Station();    Train t = new FreightTrain();    s.accept(t);}@Testpublic void accept3() throws IllegalArgumentException {    Station s = new Station();    Train t = new PassengerTrain();    s.accept(t);}@Test // (expected = IllegalArgumentException.class)public void accept4() throws IllegalArgumentException {    Station s = new Station();    s.lengthLimit = 1;    Train t = new PassengerTrain();    s.accept(t);}@Test // (expected = IllegalArgumentException.class)public void accept5() throws IllegalArgumentException {    Station s = new Station();    s.weightLimit = 1;    Train t = new FreightTrain();    s.accept(t);}

We can see that Symflower found both interesting cases, a freight train that is too heavy and a passenger train that's too long, but also generated tests for the remaining cases.

Jazzer however, seems to struggle with the custom data types we introduced. We need to aid Jazzer with these data types by writing a custom test driver fuzz/example2/Fuzzer.java. For this, we must pick which train type we want to check, so we choose the PassengerTrain. If we want to check both cases, we would need to add another test driver. But one case should suffice to demonstrate the process.

import com.code_intelligence.jazzer.api.FuzzedDataProvider;public class Fuzzer {    public static void fuzzerTestOneInput(FuzzedDataProvider data) {        Station s = new Station();        s.lengthLimit = data.consumeInt();        PassengerTrain pt = new PassengerTrain();        pt.length = data.consumeInt();        s.accept(pt);    }}

To compile the code now, we need to add the Jazzer API javac -cp ".:./jazzer_api_deploy.jar" fuzz/example2/*.java. We don't use the "autofuzz" feature anymore, but directly point Jazzer to the test driver using ./jazzer --cp=$PWD/fuzz/example2/ --target_class=Fuzzer. This time we instantly obtain a crashing input, but only for the "PassengerTrain" case:

import java.lang.reflect.InvocationTargetException;import java.lang.reflect.Method;public class Crash_adc83b19e793491b1c6ea0fd8b46cd9f32e592fc {    static final String base64Bytes = String.join("", "rO0ABXNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleHAAAAACdwQAAAACc3IAEWphdmEubGFuZy5JbnRlZ2VyEuKgpPeBhzgCAAFJAAV2YWx1ZXhyABBqYXZhLmxhbmcuTnVtYmVyhqyVHQuU4IsCAAB4cAAAAApzcQB+AAIAAAAAeA==");    public static void main(String[] args) throws Throwable {        ClassLoader.getSystemClassLoader().setDefaultAssertionStatus(true);        try {            Method fuzzerInitialize = Fuzzer.class.getMethod("fuzzerInitialize");            fuzzerInitialize.invoke(null);        } catch (NoSuchMethodException ignored) {            try {                Method fuzzerInitialize = Fuzzer.class.getMethod("fuzzerInitialize", String[].class);                fuzzerInitialize.invoke(null, (Object) args);            } catch (NoSuchMethodException ignored1) {            } catch (IllegalAccessException | InvocationTargetException e) {                e.printStackTrace();                System.exit(1);            }        } catch (IllegalAccessException | InvocationTargetException e) {            e.printStackTrace();            System.exit(1);        }        com.code_intelligence.jazzer.api.CannedFuzzedDataProvider input = new com.code_intelligence.jazzer.api.CannedFuzzedDataProvider(base64Bytes);        Fuzzer.fuzzerTestOneInput(input);    }}

The test case is quite complex, since Jazzer models both input values as a random string. This string is then converted to the two input integers for the length limit and train length, but we cannot be sure which actual integer values are chosen here.

So as we see, there are some caveats when working with Jazzer and custom data types, requiring manual work to supply the fuzzer with the correct test drivers, and being able to understand the generated test values. Symflower on the other hand generated a complete test suite fast and presents easy to understand test cases. However, we are still not done: if you find code that does not work for you or has results that could be done better, please let us know through our public issue tracker.

By the way, there is an additional bug in the examples above. Were you able to spot it? If so, how? By looking at the code or the test case?

Wrap Up

Symflower is an easy-to-apply, out-of-the-box solution that generates unit test suites but also increases developer productivity through functionalities like test templates. Jazzer, on the other hand, specializes on efficiently generating input values to trigger bugs and errors in user code. Both of these tools are free and focus on improving your development experience - so why not just use both of them?

We are working hard on providing the best overall solution for generating inputs for bug detection and unit testing. And, we're improving Symflower every day so feel free to check it out and report your experience back to us - we're always happy to receive your feedback!

To be notified of any new blog posts regarding coding, testing, features of Symflower or the memes, feel free to subscribe to our newsletter. You can also follow us on social media and - if this post helped you - share it via Twitter, LinkedIn or Facebook.