cppcheck/tools/bisect/README.md

# Bisecting

`bisect.sh` is a script to bisect regressions in Cppcheck utilizing `git-bisect`.

To learn more about bisecting please refer to https://git-scm.com/docs/git-bisect. 

## Command

```
./bisect.sh <hash-good> <hash-bad> "<cppcheck-options>" "[expected]"
```

`hash-good` the latest known good commit hash or tag<br/>
`hash-bad` the earliest known bad commit hash or tag<br/>
`cppcheck-options` the options for the Cppcheck invocation<br/>
`expected` (optional) a string that is expected in the output; when provided it will be used instead of the exitcode

If possible use `main` as the function to test with, since it won't emit an `unusedFunction` warning.

## Bisecting result regressions

Results regressions are being bisected based on the `--error-exitcode=` result.

If nothing is found the result will be `0` and it is treated as a _good_ commit.<br/>
If a finding occurs the result will be `1` which is treated as a _bad_ commit.<br/>
If a crash occurs it is treated as a _bad_ commit.

You can also bisect based on expected output via the `expected` parameter.

If the given string is found in the output it is treated as a _good_ commit.<br/>
If the given string is _not_ found in the output it is treated as a _bad_ commit.<br/>
If a crash occurs it is treated as a _bad_ commit.

### False positive

Provide a code sample which will trigger a single(!) false positive only. Trying to bisect multiple issues at the same time will most likely result in an incorrect result (see below).

```cpp
// cppcheck-suppress unusedFunction
static void f()
{
    <code triggering FP>
}
```

```
./bisect.sh <hash-good> <hash-bad> "<cppcheck-options>"
```

After the bisecting check the output to make sure that only expected false positive and no additional finding was reported for the _bad_ commits. Any other finding will also cause the commit to be marked as _bad_ leading to an incorrect result.  

### False negative

#### Via suppression

Provide a code sample which will trigger a `unmatchedSuppression`.

```cpp
// cppcheck-suppress unusedFunction
static void f()
{
    // cppcheck-suppress unreadVariable
    int i;
}
```

```
./bisect.sh <hash-good> <hash-bad> "<cppcheck-options>"
```

#### Via output

```cpp
static void f()
{
    int i;
}
```

Provide the expected error ID (`unreadVariable`) as the `expected` parameter.

```
./bisect.sh <hash-good> <hash-bad> "<cppcheck-options>" "unreadVariable"
```

## Bisecting scan time regressions

It is also possible to bisect for a regression in scan time.

This is done by determining the time it took for the "good" commit to finish and setting a timeout twice that size for the calls to determine the "bad" commit.

To bisect these kinds of regressions you currently need to adjust the `bisect.sh` script and set the `hang` variable to appropriate value:<br/>
`1` - find the commit which started the hang<br/>
`2` - find the commit which resolved the hang<br/>

### General notes

As we are currently using the process exitcode to pass the elapsed time to the script it will not work properly with vey long runtime (>= 255 seconds) as it will overflow.

In case the run-time before the regression was very short (<= 1 second) you might need to adjust the `elapsed_time` variable in `bisect.sh` to a higher value to avoid potential false positives.
This might also be necessary to determine one of multiple regressions in the commit range. 

After the bisect finished you should take a look at the output and make sure the elpased time of the respective commit looks as expected.

### daca@home notes

We use daca@home to track differences in scan time. An overview of regressions in scan time can be found at http://cppcheck1.osuosl.org:8000/time_gt.html.

If the overall scan time regressed, then you need to specify the whole folder.

If a timeout (potential hang) was introduced, then you can simply specify the file from `error: Internal error: Child process crashed with signal 15 [cppcheckError]`.

## Notes

### Bisecting daca@home issues

You need to download the archive as specified by the second line in the output and extract it.

Use the following data as respective parameters:

`hash-good` the latest tagged release - the second value from the `cppcheck:` line<br/>
`hash-bad` the commit hash from the `head-info:` line<br/>
`cppcheck-options` the `cppcheck-options:` line and the path to the folder/file to scan<br/>

### Known compilation issues:

- 2.5 and before can only be built with GCC<=10 because of missing includes caused by cleanups within the standard headers. You need to specify `CXX=g++-10`.
- 1.88 and 1.89 cannot be compiled:
```
make: python: No such file or directory
```
RESOLVED: a hot-patch is applied before compilation.
- 1.39 to 1.49 (possibly more versions - 1.54 and up work) cannot be compiled:
```
lib/mathlib.cpp:70:42: error: invalid conversion from ‘char’ to ‘char**’ [-fpermissive]
   70 |         return std::strtoul(str.c_str(), '\0', 16);
      |                                          ^~~~
      |                                          |
      |                                          char
```
- some commits between 2.0 and 2.2 cannot be compiled:
```
cli/cppcheckexecutor.cpp:333:22: error: size of array ‘mytstack’ is not an integral constant-expression
  333 | static char mytstack[MYSTACKSIZE]= {0}; // alternative stack for signal handler
  |                      ^~~~~~~~~~~
```
RESOLVED: a hot-patch is applied before compilation.
- some commits between 1.54 and 1.55 cannot be compiled:
```
lib/preprocessor.cpp:2103:5: error: ‘errorLogger’ was not declared in this scope; did you mean ‘_errorLogger’?
2103 |     errorLogger->reportInfo(errmsg);
|     ^~~~~~~~~~~
|     _errorLogger
```