155 lines
5.6 KiB
Markdown
155 lines
5.6 KiB
Markdown
# Bisecting
|
||
|
||
`bisect.sh` is a script to bisect regressions in Cppcheck utilizing `git-bisect`.
|
||
|
||
To learn more about bisecting please refer to https://git-scm.com/docs/git-bisect.
|
||
|
||
## Command
|
||
|
||
```
|
||
./bisect.sh <hash-good> <hash-bad> "<cppcheck-options>" "[expected]"
|
||
```
|
||
|
||
`hash-good` the latest known good commit hash or tag<br/>
|
||
`hash-bad` the earliest known bad commit hash or tag<br/>
|
||
`cppcheck-options` the options for the Cppcheck invocation<br/>
|
||
`expected` (optional) a string that is expected in the output; when provided it will be used instead of the exitcode
|
||
|
||
If possible use `main` as the function to test with, since it won't emit an `unusedFunction` warning.
|
||
|
||
## Bisecting result regressions
|
||
|
||
Results regressions are being bisected based on the `--error-exitcode=` result.
|
||
|
||
If nothing is found the result will be `0` and it is treated as a _good_ commit.<br/>
|
||
If a finding occurs the result will be `1` which is treated as a _bad_ commit.<br/>
|
||
If a crash occurs it is treated as a _bad_ commit.
|
||
|
||
You can also bisect based on expected output via the `expected` parameter.
|
||
|
||
If the given string is found in the output it is treated as a _good_ commit.<br/>
|
||
If the given string is _not_ found in the output it is treated as a _bad_ commit.<br/>
|
||
If a crash occurs it is treated as a _bad_ commit.
|
||
|
||
### False positive
|
||
|
||
Provide a code sample which will trigger a single(!) false positive only. Trying to bisect multiple issues at the same time will most likely result in an incorrect result (see below).
|
||
|
||
```cpp
|
||
// cppcheck-suppress unusedFunction
|
||
static void f()
|
||
{
|
||
<code triggering FP>
|
||
}
|
||
```
|
||
|
||
```
|
||
./bisect.sh <hash-good> <hash-bad> "<cppcheck-options>"
|
||
```
|
||
|
||
After the bisecting check the output to make sure that only expected false positive and no additional finding was reported for the _bad_ commits. Any other finding will also cause the commit to be marked as _bad_ leading to an incorrect result.
|
||
|
||
### False negative
|
||
|
||
#### Via suppression
|
||
|
||
Provide a code sample which will trigger a `unmatchedSuppression`.
|
||
|
||
```cpp
|
||
// cppcheck-suppress unusedFunction
|
||
static void f()
|
||
{
|
||
// cppcheck-suppress unreadVariable
|
||
int i;
|
||
}
|
||
```
|
||
|
||
```
|
||
./bisect.sh <hash-good> <hash-bad> "<cppcheck-options>"
|
||
```
|
||
|
||
#### Via output
|
||
|
||
```cpp
|
||
static void f()
|
||
{
|
||
int i;
|
||
}
|
||
```
|
||
|
||
Provide the expected error ID (`unreadVariable`) as the `expected` parameter.
|
||
|
||
```
|
||
./bisect.sh <hash-good> <hash-bad> "<cppcheck-options>" "unreadVariable"
|
||
```
|
||
|
||
## Bisecting scan time regressions
|
||
|
||
It is also possible to bisect for a regression in scan time.
|
||
|
||
This is done by determining the time it took for the "good" commit to finish and setting a timeout twice that size for the calls to determine the "bad" commit.
|
||
|
||
To bisect these kinds of regressions you currently need to adjust the `bisect.sh` script and set the `hang` variable to appropriate value:<br/>
|
||
`1` - find the commit which started the hang<br/>
|
||
`2` - find the commit which resolved the hang<br/>
|
||
|
||
### General notes
|
||
|
||
As we are currently using the process exitcode to pass the elapsed time to the script it will not work properly with vey long runtime (>= 255 seconds) as it will overflow.
|
||
|
||
In case the run-time before the regression was very short (<= 1 second) you might need to adjust the `elapsed_time` variable in `bisect.sh` to a higher value to avoid potential false positives.
|
||
This might also be necessary to determine one of multiple regressions in the commit range.
|
||
|
||
After the bisect finished you should take a look at the output and make sure the elpased time of the respective commit looks as expected.
|
||
|
||
### daca@home notes
|
||
|
||
We use daca@home to track differences in scan time. An overview of regressions in scan time can be found at http://cppcheck1.osuosl.org:8000/time_gt.html.
|
||
|
||
If the overall scan time regressed, then you need to specify the whole folder.
|
||
|
||
If a timeout (potential hang) was introduced, then you can simply specify the file from `error: Internal error: Child process crashed with signal 15 [cppcheckError]`.
|
||
|
||
## Notes
|
||
|
||
### Bisecting daca@home issues
|
||
|
||
You need to download the archive as specified by the second line in the output and extract it.
|
||
|
||
Use the following data as respective parameters:
|
||
|
||
`hash-good` the latest tagged release - the second value from the `cppcheck:` line<br/>
|
||
`hash-bad` the commit hash from the `head-info:` line<br/>
|
||
`cppcheck-options` the `cppcheck-options:` line and the path to the folder/file to scan<br/>
|
||
|
||
### Known compilation issues:
|
||
|
||
- 2.5 and before can only be built with GCC<=10 because of missing includes caused by cleanups within the standard headers. You need to specify `CXX=g++-10`.
|
||
- 1.88 and 1.89 cannot be compiled:
|
||
```
|
||
make: python: No such file or directory
|
||
```
|
||
RESOLVED: a hot-patch is applied before compilation.
|
||
- 1.39 to 1.49 (possibly more versions - 1.54 and up work) cannot be compiled:
|
||
```
|
||
lib/mathlib.cpp:70:42: error: invalid conversion from ‘char’ to ‘char**’ [-fpermissive]
|
||
70 | return std::strtoul(str.c_str(), '\0', 16);
|
||
| ^~~~
|
||
| |
|
||
| char
|
||
```
|
||
- some commits between 2.0 and 2.2 cannot be compiled:
|
||
```
|
||
cli/cppcheckexecutor.cpp:333:22: error: size of array ‘mytstack’ is not an integral constant-expression
|
||
333 | static char mytstack[MYSTACKSIZE]= {0}; // alternative stack for signal handler
|
||
| ^~~~~~~~~~~
|
||
```
|
||
RESOLVED: a hot-patch is applied before compilation.
|
||
- some commits between 1.54 and 1.55 cannot be compiled:
|
||
```
|
||
lib/preprocessor.cpp:2103:5: error: ‘errorLogger’ was not declared in this scope; did you mean ‘_errorLogger’?
|
||
2103 | errorLogger->reportInfo(errmsg);
|
||
| ^~~~~~~~~~~
|
||
| _errorLogger
|
||
```
|