Packages now can contain something like:
```
head results:
Checking temp/openvdb/Platform.cc: __GNUC__=1...
[New Thread 7892.0x91c]
```
"New Thread 7892.0x91c" was wrongly identified as messageId in the HEAD
report.
This commit adds code to skip lines that start with `[` or where the
messageId contains at least one space.
* donate-cpu.py: made exitcodes > 0 negative so they will be detected a crash / changed the ThreadExecutor error to -222
* donate-cpu.py: unconditionally upload results and info now that errors are properly handled - will also properly clear the result/info in case there are no more messages
* donate-cpu.py: bumped version
* donate-cpu.py: added stdout to output in case of exitcode != 0
* donate-cpu.py: do not scan packages with no relevant files
* donate-cpu.py: bumped version
If an upload fails, the reason (exception text) is now printed.
Fix: If the last retry failed do not wait until continuing.
Remove some obsolete "fast" code in the uploadResults() function.
Tested with Python 2.7.16 and Python 3.6.8.
Since the directory for the results does no longer exist on the server,
the server currently crashes every time older clients try to upload
experimental fast results via "write-fast" command.
Now this command is just ignored so the server is instantly ready
again after a "write-fast" command.
* donate-cpu.server.py: increased "Package" column width for latest report and small cleanup
* donate-cpu.server.py: added date and time to crash report
* donate-cpu.server.py: simplified strDateTime()
* donate-cpu.server.py: add stale report to show results which are older than 30 days
* donate-cpu-server.py: added version and some logging
* threadexecutor.cpp: streamlined error messages
* donate-cpu.py: detect additional signals and exitcode != 0 as crash as well and (ab)use elapsedTime to make the errorcode visible in the output / also detect ThreadExecutor issues
* donate-cpu.py: bumped version
* donate-cpu.py: fixed detection of ThreadExecutor errors
* Get stack traces for daca@home crashes
If a command in daca@home crashes, execute it again within gdb to get a stack trace.
* donate-cpu.py: added "gdb" to checkRequirements()
* donate-cpu.py: handle wget failures
* donate-cpu.py: added --no-upload option to disable all uploads
* donate-cpu.py: set max_packages to 1 if --package is provided to avoid endless processing of the same package
* donate-cpu.py: no longer treat missing sources as a crash
* donate-cpu.py: fixed wget "http://: Invalid host name." error caused by empty argument in subprocess.call()
* donate-cpu.py: added --no-upload to --help
* donate-cpu.py: detect crashes when using -j1
* donate-cpu.py: added -g to compiler flags
* donate-cpu.py: fixed gdb call and stacktrace printing / always pass "-j1" to gdb call so the exception will actually occur in the application
* donate-cpu.py: removed left-over --verbose from wget call
* donate-cpu.py: removed unnecessary break
* donate-cpu.py: only use gdb for crash in head run / actually provide the stack trace for the output
* donate-cpu.py: include the last checked file with the stack trace
* donate-cpu.py: removed unnecessary wget() call and a sleep in it / also inverted some logic
* donate-cpu.py: small hasInclude() optimization
* donate-cpu.py: bumped version number
* donate-cpu.py: detect start of gdb output when Cygwin is used
The Cygwin output looks like this:
Thread 1 "cppcheck" received signal SIGSEGV, Segmentation fault.
Co-Authored-By: firewave <firewave@users.noreply.github.com>
The official documentation recommends to include the Python C API via
`#include "Python.h"`:
https://docs.python.org/3/c-api/intro.html
And many projects do it exactly this way, that is why the client script
often does not detect the usage of the Python C API.
The client script will exit after the specified number of packages
have been processed. 0 means infinitely.
Useful for example to regularly quit the script, check for updates to
the client and start it again. Or as an alternative to the `--stop-time`
argument.
The function `iteritems()` of `dict`s is deprecated. The recommended
alternative is to use `items()`, this function also works with Python 2.
The next issue is that lambdas can no longer unpack tuple parameters
in Python 3. It would be possible to use some workaround and still use
a lambda, but using `operator.itemgetter(1)` instead is faster and the
recommended method in such a case.
The syntax is now compatible with Python 2 and 3 but the server script
still does not work with Python 3. For example `socket.recv()` returns
`bytes` in Python 3 and `str` in Python 2. Currently `str` is expected
so it does not work with Python 3.
In my tests there were about 1500 additional packages
available as bz2 on the server.
For some packages a newer version is now used if it is
only available as .tar.bz2 archive.
The donate-cpu.py client is tested to work with .tar.bz2
files under Python 2.7.15 and 3.6.8.
Python 3 directly decodes the text when it is read(). If there is any
invalid UTF-8 character in the text an exception is thrown (IIRC it is
UnicodeDecodeError). Opening the file with `error='ignore'` avoids
throwing an exception and just ignores the invalid character. Since
this is only possible since Python version 3 there must be extra code
for older versions.
The test script has been enhanced. It now also uses a package which
contains a file with at least one invalid UTF-8 character.
Now also found:
- Includes directly at the beginning of a file
- Indented includes
- Includes where there is no white-space between
"include" and header name
The function libcerror_error_set() is currently the function for which
daca@home most often reports a missing configuration (more than 80000
times).
Official repository of libcerror: https://github.com/libyal/libcerror
The library configuration has been tested with the library libvhdi:
ftp://ftp.se.debian.org/debian/pool/main/libv/libvhdi/libvhdi_20181227.orig.tar.gz
This detects more includes / headers. For example includes like
"# include <gtk/gtk.h>" with a space before "include" as it is used in
the package http://cppcheck.osuosl.org:8000/gbatnav are now also
detected.
The regex search also searches all includes for one library in one go
instead of one include per loop.
Tested with several packages to make sure libraries that were detected
before are still detected.
If "head" and "OLD_VERSION" both crash there are no messages and the
variable `results_exist" is set to False. But still the results must be uploaded
for the crashes to be saved also.
Tested with the package http://cppcheck.osuosl.org:8000/double-conversion
Ignore normal results (not fast or info) where the diff was made against the wrong OLD_VERSION. This avoids unwanted results when some client still analyzes an old package but the OLD_VERSION in the server script is changed for example.
Results missing the Cppcheck version info completely are also ignored.
Tested locally with correct and wrong version numbers.
Only enable the library option for those configurations if the
corresponding .cfg file exists to not crash Cppcheck if an
older version than 1.87 is used as the "old" version.
Two logging handler are added. One just prints all output with at least INFO severity to the console. The other only prints ERROR severity and above to a rotating file. The file size is limited to 100 kB. Since one backup file is used that results in a maximum of 200 kB disk usage.
The log file is saved in the directory where the server script is.
Hopefully this way some issues can be found more easily.
Tested locally.
The version number is added to the result output so issues could be more easily reproduced and in the future if it is necessary to handle any specific version of the client by the server in a special way (for example because it is erroneous) this is possible now.
Tested locally.
Searching for `#include <SDL.h>` does not always work. Sometimes `#include "SDL.h"` is used, but for now lets see for how many packages it works this way.
The checkLibraryFunction report is currently about 15 MB in size, lists more than 140000 functions and takes a long time to be generated.
Limiting the functions that are shown should save bandwidth, time and other resources on the server (and client that downloads the report).
Tested locally.
Add check to only use boost library if boost.cfg exists since it is very new.
If the boost.cfg does not exist Cppcheck would exit with an error and we
would not get the desired results.
For now the boost library configuration detection is commented out. It
should be enabled after the release of Cppcheck 1.87.
Tested locally.
Indirectly the scripts (at first the server test script) also document what needs to be done to setup a (local) server.
The productive client script can be used by everyone who wants to support daca@home, not only for testing. But i still think it is good to have it under /test to not lose too much clarity / lucidity in the /tools directory.
Optimize checking of different includes for the same library by using a `list` instead of calling `hasInclude()` several times.
Add includes for gtk and qt library detection that were found missing when looking at several daca@home packages.
* donate-cpu-server.py: Fix that results of package "crash" can not be viewed
Rename crash report page from "crash" to "crash.html" so the url ending with "crash" shows the results of package "crash".
* Also rename "diff" and "head" reports to *.html
In case there would be such packages in the future this avoids conflicts.
Related trac ticket: https://trac.cppcheck.net/ticket/8947
Enable information messages and "--check-library" in the Cppcheck
parameters.
Store the information messages and the rest of the messages in different
variables and upload them separately.
The server stores the information messages in a sub-directory similarly
to the normal issue messages in one file per package.
Reports for "checkLibraryFunction" and "checkLibraryNoReturn" message ids
are generated by the server now.
With this new option it is possible to limit the download rate / bandwidth that wget uses to download packages.
The value is verified by running wget with it and check if there is a parse error (exit code 2). If the value is not valid an error message is shown and the script exits with an error code.
Column width can now be easily changed via the `column_width` array.
Increased column width for the packages column so it is more likely that the date is not dropped.
Increased the column width for the number of results since there are packages which have about 100000 results.
Changes:
- Fixed compiler warnings
- Add "*.*" to file open menu so files without suffix can be opened
- Make it work under Windows
- Make wget() and unpackArchive() methods so they can access the UI
- wget() and unpackArchive() use the new method runProcess() now that also does some error handling
- Errors are shown in the status bar for easier debugging / usage
- Add readme.txt
- Let travis build the triage tool
Changes:
- Rename "time" page to "time.html" to avoid conflict with "time" package
- Output content as HTML instead of plain text
- Format head row of preformatted table bold
- Show factor for total time now too
Tested locally
Format the time output so the columns have the same width (as long as a text is not too long).
Add showing the factor of the time difference so very suspicious differences can be seen instantly.
Made the function body more PEP 8 conform by using lowercase variable names with underscores as delimiters.
Tested locally.
* Clean up redundant actions in travis jobs.
* Drop validatePlatforms from Makefile target checkcfg
* Print out CPU count. Adjust parameters for parallel jobs to 2 - the current result.
Instead of searching for a specific year (which could change) the script searches now for the current and last year to find the date and time information in the results.
Similar to the '--test' option for the donate-cpu-server.py script that lets the server listen locally on port 8001 the client now can also work against such a server by using the same option '--test'.
With Python3 f.read() directly returns a string object that has no
decode() function. As a workaround AttributeError exceptions during
calling the decode() function are ignored and the data read from the file
is left unchanged.
With Python2 calling the decode() function is necessary and still done.
Avoids git printing a warning about redirecting to https every time
the clone command is issued. The warning was:
warning: redirecting to https://github.com/danmar/cppcheck.git/
newer versions of git use a variable lenght proportional to the
repository size (9 for cppcheck)
remove old chomp helper function and make copying the revision
smarter to hopefully cover for edge case that needed it
Use python tarfile instead of tar to extract the packages.
Only extract source files of interest.
Skip dangerous files that could overwrite files outside the temp folder.
Fixes https://trac.cppcheck.net/ticket/8716
Daniel Marjamäki
IOBYTE
Oliver Stöneberg
versat
Bo Rydberg
orbitcowboy
rikardfalkeborn
amai2012
orbitcowboy
practicalswift
amai2012
Armin Müller
Christian Franke
Carlo Marcelo Arenas Belon
LE GARREC Vincent