5.6 KiB

Raw Blame History

Bisecting

bisect.sh is a script to bisect regressions in Cppcheck utilizing git-bisect.

To learn more about bisecting please refer to https://git-scm.com/docs/git-bisect.

Command

./bisect.sh <hash-good> <hash-bad> "<cppcheck-options>" "[expected]"

hash-good the latest known good commit hash or tag
hash-bad the earliest known bad commit hash or tag
cppcheck-options the options for the Cppcheck invocation
expected (optional) a string that is expected in the output; when provided it will be used instead of the exitcode

If possible use main as the function to test with, since it won't emit an unusedFunction warning.

Bisecting result regressions

Results regressions are being bisected based on the --error-exitcode= result.

If nothing is found the result will be 0 and it is treated as a good commit.
If a finding occurs the result will be 1 which is treated as a bad commit.
If a crash occurs it is treated as a bad commit.

You can also bisect based on expected output via the expected parameter.

If the given string is found in the output it is treated as a good commit.
If the given string is not found in the output it is treated as a bad commit.
If a crash occurs it is treated as a bad commit.

False positive

Provide a code sample which will trigger a single(!) false positive only. Trying to bisect multiple issues at the same time will most likely result in an incorrect result (see below).

// cppcheck-suppress unusedFunction
static void f()
{
    <code triggering FP>
}

./bisect.sh <hash-good> <hash-bad> "<cppcheck-options>"

After the bisecting check the output to make sure that only expected false positive and no additional finding was reported for the bad commits. Any other finding will also cause the commit to be marked as bad leading to an incorrect result.

False negative

Via suppression

Provide a code sample which will trigger a unmatchedSuppression.

// cppcheck-suppress unusedFunction
static void f()
{
    // cppcheck-suppress unreadVariable
    int i;
}

./bisect.sh <hash-good> <hash-bad> "<cppcheck-options>"

Via output

static void f()
{
    int i;
}

Provide the expected error ID (unreadVariable) as the expected parameter.

./bisect.sh <hash-good> <hash-bad> "<cppcheck-options>" "unreadVariable"

Bisecting scan time regressions

It is also possible to bisect for a regression in scan time.

This is done by determining the time it took for the "good" commit to finish and setting a timeout twice that size for the calls to determine the "bad" commit.

To bisect these kinds of regressions you currently need to adjust the bisect.sh script and set the hang variable to appropriate value:
1 - find the commit which started the hang
2 - find the commit which resolved the hang

General notes

As we are currently using the process exitcode to pass the elapsed time to the script it will not work properly with vey long runtime (>= 255 seconds) as it will overflow.

In case the run-time before the regression was very short (<= 1 second) you might need to adjust the elapsed_time variable in bisect.sh to a higher value to avoid potential false positives. This might also be necessary to determine one of multiple regressions in the commit range.

After the bisect finished you should take a look at the output and make sure the elpased time of the respective commit looks as expected.

daca@home notes

We use daca@home to track differences in scan time. An overview of regressions in scan time can be found at http://cppcheck1.osuosl.org:8000/time_gt.html.

If the overall scan time regressed, then you need to specify the whole folder.

If a timeout (potential hang) was introduced, then you can simply specify the file from error: Internal error: Child process crashed with signal 15 [cppcheckError].

Notes

Bisecting daca@home issues

You need to download the archive as specified by the second line in the output and extract it.

Use the following data as respective parameters:

hash-good the latest tagged release - the second value from the cppcheck: line
hash-bad the commit hash from the head-info: line
cppcheck-options the cppcheck-options: line and the path to the folder/file to scan

Known compilation issues:

2.5 and before can only be built with GCC<=10 because of missing includes caused by cleanups within the standard headers. You need to specify CXX=g++-10.
1.88 and 1.89 cannot be compiled:

make: python: No such file or directory

RESOLVED: a hot-patch is applied before compilation.

1.39 to 1.49 (possibly more versions - 1.54 and up work) cannot be compiled:

lib/mathlib.cpp:70:42: error: invalid conversion from ‘char’ to ‘char**’ [-fpermissive]
   70 |         return std::strtoul(str.c_str(), '\0', 16);
      |                                          ^~~~
      |                                          |
      |                                          char

some commits between 2.0 and 2.2 cannot be compiled:

cli/cppcheckexecutor.cpp:333:22: error: size of array ‘mytstack’ is not an integral constant-expression
  333 | static char mytstack[MYSTACKSIZE]= {0}; // alternative stack for signal handler
  |                      ^~~~~~~~~~~