FuzzBench: clang-cov-test report

warning
Please consider this as a preliminary report to demonstrate the capabilities of FuzzBench. While we have tried our best, we have not confirmed that we configured everything correctly. We are hoping to work together with the community to validate results and improve the set of fuzzers, benchmarks, and their configurations in the future. See FAQ for more details.

experiment summary

We show two different aggregate (cross-benchmark) rankings of fuzzers. The first is based on the average of per-benchmarks scores, where the score represents the percentage of the highest reached median coverage on a given benchmark (higher value is better). The second ranking shows the average rank of fuzzers, after we rank them on each benchmark according to their median reached covereges (lower value is better).
By avg. score
average normalized score
fuzzer
honggfuzz 97.44
aflplusplus_optimal 97.06
entropic 94.53
lafintel 92.36
aflcc 92.19
aflsmart 90.43
afl 90.25
libfuzzer 89.78
aflplusplus 89.73
aflfast 88.30
fastcgs_lm 87.33
mopt 87.05
fairfuzz 83.29
eclipser 75.43
manul 65.65
By avg. rank
average rank
fuzzer
honggfuzz 4.34
aflplusplus_optimal 4.45
entropic 5.39
aflcc 5.76
afl 6.26
aflsmart 6.29
aflplusplus 6.42
fastcgs_lm 7.74
mopt 7.95
libfuzzer 8.61
lafintel 8.66
aflfast 9.42
fairfuzz 10.34
eclipser 13.37
manul 14.50
  • Critical difference diagram
    The diagram visualizes the average rank of fuzzers (second ranking above) while showing the significance of the differences as well. What is considered a "critical difference" (CD) is based on the Friedman/Nemenyi post-hoc test. See more in the documentation.
    Note: If a fuzzer does not support all benchmarks, its ranking as shown in this diagram can be lower than it should be. So please check the list of supported benchmarks for the fuzzer(s) of your interest. The list could be specified in the fuzzer's README.md like this.
  • Median coverages on each benchmark
    fuzzer afl aflcc aflfast aflplusplus aflplusplus_optimal aflsmart eclipser entropic fairfuzz fastcgs_lm honggfuzz lafintel libfuzzer manul mopt
    benchmark
    curl_curl_fuzzer_http 17466.0 17859.5 17325.0 17490.0 17753.5 17474.0 15568.0 17489.0 16286.0 17463.5 17546.0 17350.0 16363.0 13434.0 17431.5
    freetype2-2017 18730.0 21997.0 18482.0 18419.0 22178.0 18551.5 17823.0 20270.0 18610.0 18677.0 24051.0 18292.0 16780.0 15725.0 18672.0
    harfbuzz-1.3.2 8301.0 8358.0 8231.0 8461.5 8450.0 8297.0 7539.0 8620.5 7886.0 8312.5 8550.0 8249.0 8242.0 5294.0 8302.5
    jsoncpp_jsoncpp_fuzzer 639.0 639.0 639.0 639.0 639.0 639.0 621.0 640.0 639.0 639.0 641.0 639.0 641.0 504.5 639.0
    lcms-2017-03-21 3164.0 2490.0 2536.0 2833.5 3538.0 3149.0 1479.0 2771.5 3078.5 2643.0 2735.0 2952.5 3357.0 1299.5 2496.0
    libjpeg-turbo-07-2017 3792.0 3782.0 3760.0 3758.0 3786.5 3762.0 2942.0 3796.0 3060.0 3782.0 3774.0 3754.0 3752.5 3008.0 3780.0
    libpcap_fuzz_both 101.0 4362.0 101.0 101.0 4207.0 83.0 2550.0 4268.0 101.0 98.0 4386.5 3844.0 3604.0 83.0 101.0
    libpng-1.2.56 1944.0 1627.5 1942.0 1943.0 1943.0 2126.0 1501.0 2018.0 1942.0 1509.0 2156.5 2003.0 1950.0 1503.0 1510.0
    libxml2-v2.9.2 12911.5 10534.0 12397.0 12732.0 13056.0 12547.0 5476.0 12491.0 9908.5 10956.5 12955.0 12100.0 12480.5 4551.0 10816.0
    mbedtls_fuzz_dtlsclient 8252.0 6560.0 8096.5 8265.0 8296.5 8299.0 7667.0 8192.5 8313.0 8266.5 8269.0 8171.0 7693.0 7496.0 8284.5
    openssl_x509 13849.0 13851.0 13847.0 13851.0 13849.0 13849.0 13780.0 13850.0 13777.0 13849.0 13800.5 13847.0 13822.0 13768.0 13849.0
    openthread-2019-12-23 5838.0 5876.0 5760.5 5850.0 5841.0 5854.5 5692.0 5838.0 4143.0 5831.0 5881.0 5266.0 5292.0 4140.0 5838.5
    proj4-2017-08-14 5390.0 5414.0 5163.0 5413.0 5805.0 5345.0 666.0 5559.0 5137.0 4913.0 7734.0 5305.0 5819.0 1609.0 5006.0
    re2-2014-12-09 3497.0 3510.0 3498.0 3497.0 3510.0 3499.0 3125.0 3515.0 3497.0 3494.0 3519.0 3494.0 3554.0 1684.0 3493.0
    sqlite3_ossfuzz 35129.0 32859.5 34483.5 35195.5 35145.5 34968.0 16033.0 30863.5 23680.0 35357.0 28561.0 28039.0 22965.0 21137.0 34835.0
    systemd_fuzz-link-parser 640.0 640.0 640.0 640.0 640.0 640.0 637.0 639.0 633.0 640.0 639.0 640.0 588.0 629.5 640.0
    vorbis-2017-12-11 2167.0 2178.0 2166.0 2165.0 2168.0 2168.5 2030.0 2159.5 2165.0 2169.0 2165.0 2166.0 1830.0 1992.0 2169.0
    woff2-2016-05-06 1845.5 1851.0 1816.0 1851.0 1850.0 1851.0 1483.0 NaN 1654.0 1856.0 1898.5 1850.0 1681.0 1640.0 1860.0
    zlib_zlib_uncompress_fuzzer 964.0 965.0 942.0 963.5 963.0 963.0 941.0 964.0 963.5 961.0 965.0 964.0 966.0 773.5 960.0

curl_curl_fuzzer_http summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
error
The following fuzzers do not have enough samples: entropic, manul, fairfuzz, fastcgs_lm.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflcc 82800 18.0 17859.388889 71.462157 17678.0 17824.75 17859.5 17925.00 17953.0
    aflplusplus_optimal 82800 18.0 17759.944444 50.827820 17636.0 17737.75 17753.5 17799.75 17845.0
    honggfuzz 82800 16.0 17504.750000 241.201023 16671.0 17492.25 17546.0 17605.00 17762.0
    aflplusplus 82800 19.0 17483.421053 70.906289 17372.0 17438.00 17490.0 17502.50 17659.0
    entropic 82800 15.0 17487.666667 44.169587 17420.0 17450.50 17489.0 17510.00 17575.0
    aflsmart 82800 18.0 17489.222222 82.014744 17356.0 17450.75 17474.0 17495.50 17660.0
    afl 82800 17.0 17483.705882 68.325292 17401.0 17424.00 17466.0 17529.00 17601.0
    fastcgs_lm 82800 14.0 17469.785714 76.712429 17349.0 17434.25 17463.5 17503.00 17664.0
    mopt 82800 18.0 17433.777778 79.430736 17337.0 17388.75 17431.5 17453.00 17636.0
    lafintel 82800 17.0 17345.000000 118.348004 16968.0 17297.00 17350.0 17410.00 17525.0
    aflfast 82800 16.0 17329.812500 103.605803 17161.0 17283.25 17325.0 17383.00 17552.0
    libfuzzer 82800 20.0 16315.100000 503.585868 15526.0 15998.75 16363.0 16516.75 17184.0
    fairfuzz 82800 15.0 16220.400000 688.919630 13938.0 16158.00 16286.0 16491.00 17075.0
    eclipser 82800 18.0 15554.833333 120.282633 15304.0 15487.25 15568.0 15613.00 15829.0
    manul 82800 15.0 13447.466667 64.690325 13395.0 13399.00 13434.0 13473.00 13641.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

freetype2-2017 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
error
The following fuzzers do not have enough samples: mopt, aflplusplus_optimal, eclipser.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 82800 19.0 23951.526316 958.549330 22394.0 23216.00 24051.0 24485.50 26243.0
    aflplusplus_optimal 82800 14.0 22004.142857 945.083786 20047.0 22024.75 22178.0 22680.50 23137.0
    aflcc 82800 17.0 22124.411765 471.243841 21300.0 21855.00 21997.0 22393.00 22951.0
    entropic 82800 19.0 20599.105263 861.665112 19689.0 20101.50 20270.0 20503.00 22396.0
    afl 82800 17.0 18719.470588 117.000597 18414.0 18662.00 18730.0 18787.00 18874.0
    fastcgs_lm 82800 17.0 18652.235294 114.332590 18351.0 18659.00 18677.0 18706.00 18852.0
    mopt 82800 15.0 18668.466667 96.098808 18466.0 18645.00 18672.0 18710.00 18838.0
    fairfuzz 82800 17.0 18642.529412 130.697704 18427.0 18546.00 18610.0 18758.00 18835.0
    aflsmart 82800 16.0 18539.937500 129.648997 18239.0 18502.00 18551.5 18630.50 18750.0
    aflfast 82800 18.0 18430.555556 197.485434 18088.0 18270.75 18482.0 18563.25 18734.0
    aflplusplus 82800 17.0 18441.705882 116.507921 18245.0 18358.00 18419.0 18522.00 18643.0
    lafintel 82800 17.0 18404.058824 538.052213 17757.0 18103.00 18292.0 18542.00 20168.0
    eclipser 82800 13.0 17784.230769 416.261367 17159.0 17565.00 17823.0 18054.00 18455.0
    libfuzzer 82800 20.0 16923.400000 1190.390489 13869.0 16397.25 16780.0 18105.50 18410.0
    manul 82800 19.0 15644.526316 314.265169 15018.0 15506.00 15725.0 15855.00 16047.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

harfbuzz-1.3.2 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
error
The following fuzzers do not have enough samples: libfuzzer, fastcgs_lm, aflcc, afl.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    entropic 82800 16.0 8688.875000 166.814018 8547.0 8599.00 8620.5 8662.25 9043.0
    honggfuzz 82800 19.0 8546.157895 52.880645 8376.0 8535.50 8550.0 8567.50 8657.0
    aflplusplus 82800 16.0 8466.125000 104.392129 8301.0 8436.75 8461.5 8492.50 8698.0
    aflplusplus_optimal 82800 19.0 8397.263158 107.152976 8264.0 8290.50 8450.0 8489.50 8543.0
    aflcc 82800 14.0 8406.928571 108.773911 8316.0 8331.50 8358.0 8492.75 8590.0
    fastcgs_lm 82800 14.0 8310.142857 26.363895 8275.0 8289.00 8312.5 8325.25 8367.0
    mopt 82800 18.0 8333.333333 95.583656 8220.0 8260.50 8302.5 8387.00 8504.0
    afl 82800 14.0 8311.000000 64.846564 8248.0 8280.00 8301.0 8321.00 8518.0
    aflsmart 82800 17.0 8308.823529 49.268696 8253.0 8277.00 8297.0 8317.00 8426.0
    lafintel 82800 17.0 8280.411765 81.114162 8189.0 8235.00 8249.0 8291.00 8471.0
    libfuzzer 82800 15.0 8251.866667 67.939959 8137.0 8213.50 8242.0 8290.50 8412.0
    aflfast 82800 17.0 8243.235294 51.499429 8178.0 8208.00 8231.0 8256.00 8390.0
    fairfuzz 82800 17.0 7858.823529 228.290176 7445.0 7684.00 7886.0 7956.00 8294.0
    eclipser 82800 16.0 7548.000000 58.113682 7433.0 7514.75 7539.0 7559.75 7674.0
    manul 82800 16.0 5296.500000 48.485049 5226.0 5273.25 5294.0 5307.00 5400.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

jsoncpp_jsoncpp_fuzzer summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
error
The following fuzzers do not have enough samples: afl, fairfuzz, honggfuzz, fastcgs_lm.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 82800 15.0 641.0000 0.000000 641.0 641.00 641.0 641.00 641.0
    libfuzzer 82800 16.0 641.0000 0.000000 641.0 641.00 641.0 641.00 641.0
    entropic 82800 16.0 639.9375 0.250000 639.0 640.00 640.0 640.00 640.0
    afl 82800 15.0 639.0000 0.000000 639.0 639.00 639.0 639.00 639.0
    aflcc 82800 19.0 639.0000 0.000000 639.0 639.00 639.0 639.00 639.0
    aflfast 82800 20.0 638.9500 0.223607 638.0 639.00 639.0 639.00 639.0
    aflplusplus 82800 16.0 639.0000 0.000000 639.0 639.00 639.0 639.00 639.0
    aflplusplus_optimal 82800 18.0 639.0000 0.000000 639.0 639.00 639.0 639.00 639.0
    aflsmart 82800 17.0 639.0000 0.000000 639.0 639.00 639.0 639.00 639.0
    fairfuzz 82800 15.0 639.0000 0.000000 639.0 639.00 639.0 639.00 639.0
    fastcgs_lm 82800 13.0 639.0000 0.000000 639.0 639.00 639.0 639.00 639.0
    lafintel 82800 16.0 639.0000 0.000000 639.0 639.00 639.0 639.00 639.0
    mopt 82800 16.0 639.0000 0.000000 639.0 639.00 639.0 639.00 639.0
    eclipser 82800 16.0 618.0000 12.280065 585.0 613.00 621.0 627.00 629.0
    manul 82800 20.0 510.3000 22.450179 478.0 499.25 504.5 522.75 577.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

lcms-2017-03-21 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
error
The following fuzzers do not have enough samples: libfuzzer, entropic.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_optimal 82800 16.0 3498.375000 212.901816 2852.0 3428.00 3538.0 3582.25 3811.0
    libfuzzer 82800 15.0 3335.666667 147.452494 3105.0 3205.00 3357.0 3443.50 3606.0
    afl 82800 17.0 3186.352941 288.437589 2773.0 2945.00 3164.0 3420.00 3754.0
    aflsmart 82800 17.0 3109.470588 154.665008 2723.0 3012.00 3149.0 3160.00 3369.0
    fairfuzz 82800 18.0 3058.388889 168.915272 2765.0 2947.75 3078.5 3120.50 3431.0
    lafintel 82800 20.0 2960.900000 323.345652 2563.0 2627.50 2952.5 3113.75 3611.0
    aflplusplus 82800 18.0 2976.444444 222.464597 2766.0 2786.25 2833.5 3139.75 3407.0
    entropic 82800 14.0 2997.785714 468.039640 2502.0 2570.50 2771.5 3454.75 3685.0
    honggfuzz 82800 19.0 2881.947368 436.397305 2488.0 2571.50 2735.0 3109.50 3771.0
    fastcgs_lm 82800 17.0 2573.294118 91.265249 2457.0 2469.00 2643.0 2647.00 2673.0
    aflfast 82800 18.0 2222.833333 620.972979 1223.0 1676.75 2536.0 2576.25 2905.0
    mopt 82800 18.0 2549.277778 92.187026 2435.0 2471.00 2496.0 2641.50 2684.0
    aflcc 82800 18.0 2516.555556 57.877378 2471.0 2480.25 2490.0 2516.50 2641.0
    eclipser 82800 17.0 1886.117647 574.622037 1362.0 1383.00 1479.0 2442.00 2740.0
    manul 82800 18.0 1401.055556 288.672514 1271.0 1299.00 1299.5 1312.75 2201.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libjpeg-turbo-07-2017 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
error
The following fuzzers do not have enough samples: aflsmart, aflplusplus, eclipser, mopt.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    entropic 82800 20.0 3784.900000 22.680620 3753.0 3757.50 3796.0 3800.0 3818.0
    afl 82800 19.0 3772.421053 73.721937 3474.0 3777.00 3792.0 3800.5 3807.0
    aflplusplus_optimal 82800 16.0 3715.062500 148.681525 3454.0 3706.25 3786.5 3795.0 3839.0
    aflcc 82800 16.0 3736.000000 113.196584 3448.0 3752.25 3782.0 3801.0 3803.0
    fastcgs_lm 82800 17.0 3770.647059 86.898606 3443.0 3776.00 3782.0 3812.0 3822.0
    mopt 82800 13.0 3759.461538 86.750231 3474.0 3774.00 3780.0 3793.0 3803.0
    honggfuzz 82800 19.0 3728.421053 120.083913 3439.0 3756.00 3774.0 3793.0 3803.0
    aflsmart 82800 15.0 3720.600000 117.220062 3433.0 3752.50 3762.0 3778.5 3799.0
    aflfast 82800 19.0 3663.578947 165.884939 3353.0 3459.00 3760.0 3782.0 3803.0
    aflplusplus 82800 15.0 3678.733333 149.053905 3428.0 3602.50 3758.0 3768.5 3782.0
    lafintel 82800 16.0 3628.375000 162.136105 3417.0 3450.00 3754.0 3757.5 3800.0
    libfuzzer 82800 18.0 3541.500000 337.889862 2412.0 3424.00 3752.5 3754.0 3756.0
    fairfuzz 82800 17.0 3000.352941 302.365660 2466.0 2872.00 3060.0 3317.0 3347.0
    manul 82800 19.0 2934.789474 156.791290 2550.0 2959.50 3008.0 3017.0 3052.0
    eclipser 82800 15.0 2938.533333 92.072849 2818.0 2857.00 2942.0 3006.0 3118.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libpcap_fuzz_both summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
error
The following fuzzers do not have enough samples: entropic, aflplusplus.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 82800 16.0 4333.937500 235.019565 3816.0 4306.75 4386.5 4505.75 4604.0
    aflcc 82800 18.0 4319.000000 229.712096 3988.0 4111.25 4362.0 4442.00 4846.0
    entropic 82800 15.0 4158.666667 280.501761 3567.0 3968.00 4268.0 4389.00 4497.0
    aflplusplus_optimal 82800 17.0 4204.352941 255.785296 3683.0 4162.00 4207.0 4389.00 4559.0
    lafintel 82800 16.0 3852.062500 295.115902 3429.0 3584.25 3844.0 4086.25 4295.0
    libfuzzer 82800 19.0 3604.736842 148.309302 3373.0 3506.00 3604.0 3676.50 3954.0
    eclipser 82800 17.0 2675.941176 275.371220 2343.0 2481.00 2550.0 2979.00 3249.0
    afl 82800 19.0 98.894737 6.341177 79.0 101.00 101.0 101.00 101.0
    aflfast 82800 17.0 98.882353 5.977900 83.0 101.00 101.0 101.00 101.0
    aflplusplus 82800 15.0 150.266667 200.866859 83.0 101.00 101.0 101.00 876.0
    fairfuzz 82800 17.0 119.647059 95.755510 83.0 101.00 101.0 101.00 490.0
    mopt 82800 16.0 289.750000 783.849603 83.0 83.00 101.0 101.00 3229.0
    fastcgs_lm 82800 17.0 91.941176 7.309844 83.0 83.00 98.0 98.00 100.0
    aflsmart 82800 16.0 84.125000 4.500000 83.0 83.00 83.0 83.00 101.0
    manul 82800 17.0 170.058824 163.349882 79.0 83.00 83.0 89.00 470.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libpng-1.2.56 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
error
The following fuzzers do not have enough samples: eclipser, lafintel, libfuzzer, honggfuzz, aflplusplus_optimal, afl.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 82800 14.0 2161.000000 30.581543 2111.0 2151.50 2156.5 2188.50 2199.0
    aflsmart 82800 18.0 2122.277778 28.872551 2075.0 2107.75 2126.0 2132.50 2179.0
    entropic 82800 18.0 2027.388889 24.966186 2002.0 2014.25 2018.0 2024.25 2089.0
    lafintel 82800 15.0 2003.933333 3.863135 2000.0 2000.50 2003.0 2008.00 2010.0
    libfuzzer 82800 15.0 1971.133333 27.619524 1942.0 1946.00 1950.0 1999.50 2002.0
    afl 82800 13.0 1943.923077 1.497862 1942.0 1943.00 1944.0 1945.00 1947.0
    aflplusplus 82800 18.0 1943.666667 1.814970 1942.0 1942.00 1943.0 1945.00 1948.0
    aflplusplus_optimal 82800 13.0 1942.923077 1.656379 1941.0 1942.00 1943.0 1943.00 1947.0
    aflfast 82800 20.0 1942.550000 2.282081 1938.0 1941.00 1942.0 1943.25 1949.0
    fairfuzz 82800 18.0 1942.444444 1.789585 1940.0 1941.00 1942.0 1944.00 1947.0
    aflcc 82800 16.0 1631.750000 42.979840 1562.0 1597.25 1627.5 1643.50 1733.0
    mopt 82800 18.0 1510.833333 2.995094 1509.0 1510.00 1510.0 1510.00 1519.0
    fastcgs_lm 82800 17.0 1509.705882 2.519045 1508.0 1509.00 1509.0 1510.00 1519.0
    manul 82800 19.0 1521.526316 27.661382 1500.0 1502.00 1503.0 1542.00 1574.0
    eclipser 82800 15.0 1503.266667 58.178870 1373.0 1468.50 1501.0 1539.50 1603.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libxml2-v2.9.2 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_optimal 82800 17.0 13044.294118 199.202712 12579.0 13001.00 13056.0 13129.00 13439.0
    honggfuzz 82800 17.0 12907.764706 201.933321 12408.0 12813.00 12955.0 13027.00 13270.0
    afl 82800 18.0 12770.222222 325.651664 11924.0 12555.25 12911.5 12982.75 13129.0
    aflplusplus 82800 18.0 12562.888889 384.501035 11886.0 12239.75 12732.0 12872.75 13012.0
    aflsmart 82800 17.0 12649.235294 290.663149 12188.0 12439.00 12547.0 12958.00 13126.0
    entropic 82800 17.0 12415.705882 395.759517 11689.0 12078.00 12491.0 12648.00 13146.0
    libfuzzer 82800 18.0 12320.333333 395.091948 11817.0 11911.25 12480.5 12623.50 13018.0
    aflfast 82800 16.0 12369.625000 307.333234 11668.0 12309.50 12397.0 12481.75 12799.0
    lafintel 82800 19.0 12009.421053 422.377308 11397.0 11592.50 12100.0 12377.50 12630.0
    fastcgs_lm 82800 20.0 10749.750000 1020.148847 6695.0 10636.50 10956.5 11180.75 11698.0
    mopt 82800 18.0 10540.611111 1083.460857 6598.0 10292.75 10816.0 11001.25 11585.0
    aflcc 82800 17.0 10631.764706 873.188234 8716.0 10138.00 10534.0 11029.00 12153.0
    fairfuzz 82800 18.0 10217.833333 613.063882 9670.0 9810.00 9908.5 10389.00 11560.0
    eclipser 82800 16.0 5447.687500 87.032346 5295.0 5367.75 5476.0 5516.75 5577.0
    manul 82800 18.0 4560.666667 261.555978 4182.0 4344.50 4551.0 4698.00 5111.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

mbedtls_fuzz_dtlsclient summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
error
The following fuzzers do not have enough samples: lafintel, fairfuzz, afl.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    fairfuzz 82800 15.0 8319.333333 44.234225 8212.0 8295.50 8313.0 8350.50 8387.0
    aflsmart 82800 17.0 8644.764706 1023.947297 8175.0 8273.00 8299.0 8341.00 11500.0
    aflplusplus_optimal 82800 18.0 8673.833333 958.725272 8230.0 8283.25 8296.5 8316.00 11492.0
    mopt 82800 16.0 8282.812500 60.130102 8186.0 8233.50 8284.5 8329.50 8366.0
    honggfuzz 82800 17.0 8391.529412 553.115395 8208.0 8246.00 8269.0 8275.00 10536.0
    fastcgs_lm 82800 18.0 8611.722222 999.281613 8216.0 8242.50 8266.5 8314.25 11547.0
    aflplusplus 82800 17.0 9106.882353 1352.076130 8212.0 8252.00 8265.0 10670.00 11553.0
    afl 82800 14.0 8258.571429 47.888836 8199.0 8222.75 8252.0 8269.50 8357.0
    entropic 82800 18.0 8189.777778 45.908548 8086.0 8169.00 8192.5 8213.75 8304.0
    lafintel 82800 15.0 8171.666667 26.534793 8134.0 8154.00 8171.0 8189.00 8229.0
    aflfast 82800 16.0 8099.125000 32.612625 8039.0 8082.50 8096.5 8112.75 8177.0
    libfuzzer 82800 20.0 7848.000000 237.709464 7560.0 7635.50 7693.0 8087.50 8171.0
    eclipser 82800 16.0 7683.187500 47.628029 7598.0 7656.75 7667.0 7720.25 7796.0
    manul 82800 17.0 7492.117647 40.595385 7404.0 7474.00 7496.0 7515.00 7572.0
    aflcc 82800 17.0 6560.294118 1.104802 6557.0 6560.00 6560.0 6561.00 6562.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

openssl_x509 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
error
The following fuzzers do not have enough samples: aflfast, eclipser, libfuzzer, manul.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflcc 82800 17.0 13851.000000 6.791539 13827.0 13851.0 13851.0 13853.0 13859.0
    aflplusplus 82800 16.0 13850.750000 2.516611 13847.0 13849.0 13851.0 13851.0 13859.0
    entropic 82800 19.0 13845.578947 9.923685 13826.0 13849.0 13850.0 13851.0 13852.0
    afl 82800 18.0 13849.722222 2.108960 13849.0 13849.0 13849.0 13849.0 13856.0
    aflplusplus_optimal 82800 17.0 13850.470588 2.294495 13849.0 13849.0 13849.0 13851.0 13855.0
    aflsmart 82800 16.0 13849.500000 2.000000 13849.0 13849.0 13849.0 13849.0 13857.0
    fastcgs_lm 82800 20.0 13848.400000 5.154456 13827.0 13849.0 13849.0 13851.0 13851.0
    mopt 82800 17.0 13849.000000 0.000000 13849.0 13849.0 13849.0 13849.0 13849.0
    aflfast 82800 15.0 13841.333333 10.688222 13821.0 13835.5 13847.0 13848.5 13852.0
    lafintel 82800 19.0 13840.157895 11.270725 13815.0 13832.5 13847.0 13849.0 13850.0
    libfuzzer 82800 15.0 13822.333333 9.476789 13813.0 13814.0 13822.0 13823.0 13847.0
    honggfuzz 82800 16.0 13806.562500 10.732932 13800.0 13800.0 13800.5 13807.5 13826.0
    eclipser 82800 15.0 13780.466667 13.324878 13757.0 13771.5 13780.0 13792.5 13800.0
    fairfuzz 82800 17.0 13785.058824 32.512441 13751.0 13753.0 13777.0 13797.0 13849.0
    manul 82800 15.0 13764.866667 8.262364 13752.0 13755.5 13768.0 13772.0 13774.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

openthread-2019-12-23 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
error
The following fuzzers do not have enough samples: honggfuzz, eclipser, aflplusplus_optimal, libfuzzer, aflplusplus, aflfast.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 82800 15.0 5868.600000 39.187826 5763.0 5852.00 5881.0 5887.00 5938.0
    aflcc 82800 17.0 5869.647059 18.644507 5827.0 5854.00 5876.0 5881.00 5897.0
    aflsmart 82800 16.0 5810.625000 157.417015 5247.0 5823.00 5854.5 5881.25 5913.0
    aflplusplus 82800 15.0 5849.933333 22.904980 5798.0 5842.00 5850.0 5860.00 5899.0
    aflplusplus_optimal 82800 15.0 5850.600000 36.116478 5767.0 5835.00 5841.0 5861.50 5916.0
    mopt 82800 16.0 5830.937500 48.449931 5730.0 5820.50 5838.5 5868.50 5892.0
    afl 82800 17.0 5772.529412 186.777648 5268.0 5823.00 5838.0 5860.00 5886.0
    entropic 82800 19.0 5688.210526 256.126397 5236.0 5518.00 5838.0 5845.50 5891.0
    fastcgs_lm 82800 19.0 5793.894737 139.312556 5247.0 5824.00 5831.0 5849.50 5874.0
    aflfast 82800 14.0 5571.785714 284.390895 5237.0 5256.25 5760.5 5811.75 5856.0
    eclipser 82800 15.0 5597.533333 226.549858 5237.0 5450.00 5692.0 5749.50 5822.0
    libfuzzer 82800 15.0 5286.933333 16.201705 5246.0 5289.50 5292.0 5294.50 5304.0
    lafintel 82800 17.0 5335.705882 167.696439 5243.0 5260.00 5266.0 5270.00 5805.0
    fairfuzz 82800 18.0 4570.222222 563.543469 4143.0 4143.00 4143.0 5174.50 5665.0
    manul 82800 17.0 4140.000000 0.000000 4140.0 4140.00 4140.0 4140.00 4140.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

proj4-2017-08-14 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
error
The following fuzzers do not have enough samples: afl, aflplusplus_optimal, manul, aflplusplus.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 82800 17.0 7726.411765 74.526052 7553.0 7687.00 7734.0 7768.00 7844.0
    libfuzzer 82800 16.0 5780.750000 173.885211 5436.0 5711.50 5819.0 5892.00 6072.0
    aflplusplus_optimal 82800 15.0 5779.733333 215.821840 5380.0 5653.50 5805.0 5856.50 6224.0
    entropic 82800 18.0 5533.111111 182.414166 5248.0 5356.50 5559.0 5673.75 5854.0
    aflcc 82800 19.0 5428.736842 179.917216 5159.0 5272.50 5414.0 5544.00 5827.0
    aflplusplus 82800 14.0 5468.071429 152.952212 5288.0 5346.50 5413.0 5592.25 5751.0
    afl 82800 15.0 5395.866667 128.336537 5248.0 5285.00 5390.0 5470.50 5720.0
    aflsmart 82800 19.0 5379.157895 118.371479 5215.0 5304.50 5345.0 5484.00 5588.0
    lafintel 82800 17.0 5340.000000 108.923138 5189.0 5268.00 5305.0 5426.00 5593.0
    aflfast 82800 16.0 5161.000000 139.633329 4940.0 5064.50 5163.0 5202.25 5498.0
    fairfuzz 82800 19.0 5109.684211 147.935441 4777.0 5081.00 5137.0 5215.50 5256.0
    mopt 82800 17.0 4961.117647 160.278618 4541.0 4861.00 5006.0 5066.00 5143.0
    fastcgs_lm 82800 17.0 4927.470588 188.318918 4654.0 4749.00 4913.0 5078.00 5223.0
    manul 82800 14.0 1457.000000 331.215245 895.0 1295.25 1609.0 1663.75 1906.0
    eclipser 82800 17.0 750.411765 353.089411 447.0 538.00 666.0 743.00 1799.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

re2-2014-12-09 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
error
The following fuzzers do not have enough samples: eclipser, aflfast, fastcgs_lm, entropic, afl.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libfuzzer 82800 18.0 3547.944444 16.275106 3508.0 3553.25 3554.0 3555.00 3560.0
    honggfuzz 82800 17.0 3526.764706 15.662572 3516.0 3518.00 3519.0 3521.00 3557.0
    entropic 82800 13.0 3511.153846 13.037421 3469.0 3513.00 3515.0 3516.00 3519.0
    aflcc 82800 17.0 3508.529412 6.911201 3491.0 3504.00 3510.0 3513.00 3518.0
    aflplusplus_optimal 82800 18.0 3508.444444 16.610494 3454.0 3508.25 3510.0 3511.00 3549.0
    aflsmart 82800 18.0 3498.166667 2.307277 3492.0 3497.00 3499.0 3499.00 3502.0
    aflfast 82800 14.0 3475.214286 49.938699 3360.0 3493.25 3498.0 3499.00 3506.0
    afl 82800 11.0 3495.727273 3.770700 3485.0 3496.00 3497.0 3497.00 3499.0
    aflplusplus 82800 18.0 3479.111111 50.106814 3318.0 3493.25 3497.0 3499.50 3508.0
    fairfuzz 82800 16.0 3488.562500 27.121870 3397.0 3495.00 3497.0 3499.25 3505.0
    fastcgs_lm 82800 14.0 3489.214286 16.957769 3431.0 3491.00 3494.0 3495.75 3498.0
    lafintel 82800 19.0 3494.842105 4.645944 3487.0 3491.50 3494.0 3498.00 3505.0
    mopt 82800 18.0 3492.611111 2.252812 3488.0 3491.25 3493.0 3494.00 3497.0
    eclipser 82800 15.0 3120.866667 14.106061 3081.0 3114.50 3125.0 3129.00 3137.0
    manul 82800 17.0 1690.176471 43.682427 1614.0 1662.00 1684.0 1721.00 1762.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

sqlite3_ossfuzz summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    fastcgs_lm 82800 17.0 35315.117647 355.678029 34649.0 35049.00 35357.0 35546.00 35943.0
    aflplusplus 82800 16.0 35198.375000 200.526598 34727.0 35134.25 35195.5 35376.75 35446.0
    aflplusplus_optimal 82800 18.0 35165.666667 362.258276 34518.0 34894.25 35145.5 35385.75 35861.0
    afl 82800 17.0 35119.529412 361.310136 34427.0 34980.00 35129.0 35356.00 35725.0
    aflsmart 82800 15.0 35084.533333 346.612766 34628.0 34834.50 34968.0 35318.00 35733.0
    mopt 82800 17.0 34821.294118 308.962531 34236.0 34598.00 34835.0 34988.00 35370.0
    aflfast 82800 16.0 34434.562500 404.022106 33367.0 34353.50 34483.5 34633.50 35062.0
    aflcc 82800 18.0 32858.833333 470.635771 31847.0 32616.50 32859.5 33150.75 33628.0
    entropic 82800 18.0 30694.333333 790.160669 29156.0 30416.25 30863.5 31192.75 31947.0
    honggfuzz 82800 16.0 28530.437500 755.988357 27291.0 28234.75 28561.0 28980.25 30158.0
    lafintel 82800 18.0 27829.611111 957.052284 25315.0 27198.50 28039.0 28440.25 29331.0
    fairfuzz 82800 17.0 25281.176471 3159.692474 22020.0 22433.00 23680.0 27687.00 32114.0
    libfuzzer 82800 16.0 22923.125000 743.984845 21612.0 22385.00 22965.0 23416.25 24171.0
    manul 82800 17.0 21276.823529 458.464998 20456.0 20959.00 21137.0 21572.00 22226.0
    eclipser 82800 15.0 16823.466667 1373.558770 16027.0 16029.00 16033.0 17325.50 19545.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

vorbis-2017-12-11 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
error
The following fuzzers do not have enough samples: aflplusplus_optimal, libfuzzer.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflcc 82800 17.0 2177.470588 1.280510 2174.0 2177.00 2178.0 2178.00 2180.0
    fastcgs_lm 82800 18.0 2171.166667 4.706066 2166.0 2167.25 2169.0 2175.75 2181.0
    mopt 82800 16.0 2170.000000 4.802777 2164.0 2166.00 2169.0 2172.25 2182.0
    aflsmart 82800 18.0 2170.166667 4.805022 2163.0 2167.25 2168.5 2172.00 2184.0
    aflplusplus_optimal 82800 15.0 2170.533333 5.617151 2165.0 2167.00 2168.0 2172.00 2184.0
    afl 82800 18.0 2167.833333 3.698171 2162.0 2166.00 2167.0 2168.75 2177.0
    aflfast 82800 20.0 2166.750000 4.722566 2160.0 2164.00 2166.0 2169.25 2180.0
    lafintel 82800 17.0 2163.176471 12.665086 2117.0 2165.00 2166.0 2169.00 2171.0
    aflplusplus 82800 16.0 2165.937500 3.732180 2159.0 2164.00 2165.0 2168.50 2173.0
    fairfuzz 82800 16.0 2149.812500 25.166694 2097.0 2143.50 2165.0 2166.25 2169.0
    honggfuzz 82800 16.0 2168.250000 22.251592 2148.0 2163.50 2165.0 2166.25 2249.0
    entropic 82800 18.0 2159.777778 3.387790 2155.0 2158.25 2159.5 2160.00 2171.0
    eclipser 82800 16.0 1992.625000 93.427958 1739.0 1964.75 2030.0 2054.00 2075.0
    manul 82800 17.0 1992.176471 16.819763 1963.0 1985.00 1992.0 1999.00 2025.0
    libfuzzer 82800 15.0 1959.266667 164.624450 1828.0 1829.50 1830.0 2150.50 2162.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

woff2-2016-05-06 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
error
The following fuzzers do not have enough samples: aflplusplus_optimal, lafintel, libfuzzer, aflplusplus, afl, honggfuzz, eclipser.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 82800 14.0 1898.571429 18.768690 1869.0 1884.00 1898.5 1916.75 1922.0
    mopt 82800 19.0 1850.157895 28.101529 1788.0 1825.00 1860.0 1873.50 1883.0
    fastcgs_lm 82800 17.0 1850.588235 19.339270 1826.0 1833.00 1856.0 1860.00 1886.0
    aflcc 82800 19.0 1855.473684 14.785009 1820.0 1849.00 1851.0 1867.00 1880.0
    aflplusplus 82800 15.0 1848.266667 21.295428 1801.0 1835.00 1851.0 1860.50 1888.0
    aflsmart 82800 16.0 1847.562500 19.294105 1810.0 1836.50 1851.0 1862.25 1873.0
    aflplusplus_optimal 82800 15.0 1855.200000 17.238661 1832.0 1842.50 1850.0 1873.50 1879.0
    lafintel 82800 15.0 1851.066667 20.630998 1823.0 1841.00 1850.0 1853.50 1911.0
    afl 82800 14.0 1849.428571 18.690657 1812.0 1840.75 1845.5 1862.00 1882.0
    aflfast 82800 19.0 1808.368421 31.499930 1703.0 1800.50 1816.0 1821.50 1853.0
    libfuzzer 82800 15.0 1694.666667 41.951786 1656.0 1664.50 1681.0 1713.50 1779.0
    fairfuzz 82800 18.0 1646.222222 32.191624 1588.0 1619.50 1654.0 1665.75 1707.0
    manul 82800 17.0 1631.470588 21.969062 1587.0 1626.00 1640.0 1645.00 1666.0
    eclipser 82800 11.0 1471.000000 44.857552 1383.0 1482.00 1483.0 1490.00 1515.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

zlib_zlib_uncompress_fuzzer summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
error
The following fuzzers do not have enough samples: afl.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libfuzzer 82800 18.0 974.277778 10.231471 964.0 965.25 966.0 985.00 986.0
    aflcc 82800 16.0 970.312500 10.568625 960.0 962.25 965.0 983.00 984.0
    honggfuzz 82800 15.0 968.466667 8.822590 961.0 964.00 965.0 965.50 987.0
    afl 82800 14.0 968.357143 10.081263 960.0 960.75 964.0 978.25 984.0
    entropic 82800 17.0 971.000000 10.362191 960.0 963.00 964.0 983.00 984.0
    lafintel 82800 18.0 965.444444 9.894872 943.0 961.00 964.0 965.00 985.0
    aflplusplus 82800 18.0 963.277778 6.771963 946.0 961.50 963.5 964.75 984.0
    fairfuzz 82800 16.0 964.375000 5.451300 960.0 963.00 963.5 964.00 984.0
    aflplusplus_optimal 82800 17.0 962.294118 7.472991 942.0 960.00 963.0 964.00 983.0
    aflsmart 82800 17.0 961.411765 8.646233 942.0 960.00 963.0 964.00 983.0
    fastcgs_lm 82800 17.0 962.882353 5.710542 960.0 960.00 961.0 963.00 984.0
    mopt 82800 17.0 961.705882 6.816934 946.0 960.00 960.0 963.00 983.0
    aflfast 82800 16.0 948.500000 9.252027 938.0 942.00 942.0 960.00 960.0
    eclipser 82800 17.0 941.235294 11.871865 924.0 940.00 941.0 951.00 958.0
    manul 82800 18.0 781.444444 15.919733 763.0 770.25 773.5 791.75 814.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

experiment data

You can download the raw data for this report here.

Check out the documentation on how to create customized reports using this data. Also see some example Colab notebooks for doing custom analysis on the data here.

The experiment was conducted using this FuzzBench commit: a36b6e50b631a8287601e5822985db9caf31845b