Commit Graph

155 Commits

Author SHA1 Message Date
IOBYTE 2b6d599f1e donate-cpu.py: show size of results sent to server (#1784) 2019-04-05 14:11:03 +02:00
Oliver Stöneberg 6e6cdae20e donate-cpu.py: improved error handling when fetching cppcheck version (#1780)
During a local scan it did not get any cppcheck versions causing the script to fail with an exception when creating the diff.
2019-04-04 13:09:35 +02:00
Oliver Stöneberg 8c26d4ee6a donate-cpu.py: more error detection improvements / properly update data on server (#1768)
* donate-cpu.py: made exitcodes > 0 negative so they will be detected a crash / changed the ThreadExecutor error to -222

* donate-cpu.py: unconditionally upload results and info now that errors are properly handled - will also properly clear the result/info in case there are no more messages

* donate-cpu.py: bumped version

* donate-cpu.py: added stdout to output in case of exitcode != 0

* donate-cpu.py: do not scan packages with no relevant files

* donate-cpu.py: bumped version
2019-04-03 09:27:04 +02:00
versat 6b9a25869e donate-cpu.py: Improve upload error handling, remove old "fast" code.
If an upload fails, the reason (exception text) is now printed.
Fix: If the last retry failed do not wait until continuing.
Remove some obsolete "fast" code in the uploadResults() function.

Tested with Python 2.7.16 and Python 3.6.8.
2019-04-02 10:23:57 +02:00
versat cdb449a1b8 donate-cpu.py: Only some cosmetic / coding style fixes. 2019-03-29 09:51:56 +01:00
Oliver Stöneberg 8d7d93aea7 donate-cpu.py: improved error detection (#1766)
* threadexecutor.cpp: streamlined error messages

* donate-cpu.py: detect additional signals and exitcode != 0 as crash as well and (ab)use elapsedTime to make the errorcode visible in the output / also detect ThreadExecutor issues

* donate-cpu.py: bumped version

* donate-cpu.py: fixed detection of ThreadExecutor errors
2019-03-29 06:53:40 +01:00
Oliver Stöneberg 7d383d1684 donate-cpu.py: add stack traces for daca@home crashes / bugfixes (#1764)
* Get stack traces for daca@home crashes

If a command in daca@home crashes, execute it again within gdb to get a stack trace.

* donate-cpu.py: added "gdb" to checkRequirements()

* donate-cpu.py: handle wget failures

* donate-cpu.py: added --no-upload option to disable all uploads

* donate-cpu.py: set max_packages to 1 if --package is provided to avoid endless processing of the same package

* donate-cpu.py: no longer treat missing sources as a crash

* donate-cpu.py: fixed wget "http://: Invalid host name." error caused by empty argument in subprocess.call()

* donate-cpu.py: added --no-upload to --help

* donate-cpu.py: detect crashes when using -j1

* donate-cpu.py: added -g to compiler flags

* donate-cpu.py: fixed gdb call and stacktrace printing / always pass "-j1" to gdb call so the exception will actually occur in the application

* donate-cpu.py: removed left-over --verbose from wget call

* donate-cpu.py: removed unnecessary break

* donate-cpu.py: only use gdb for crash in head run / actually provide the stack trace for the output

* donate-cpu.py: include the last checked file with the stack trace

* donate-cpu.py: removed unnecessary wget() call and a sleep in it / also inverted some logic

* donate-cpu.py: small hasInclude() optimization

* donate-cpu.py: bumped version number

* donate-cpu.py: detect start of gdb output when Cygwin is used

The Cygwin output looks like this:

Thread 1 "cppcheck" received signal SIGSEGV, Segmentation fault.

Co-Authored-By: firewave <firewave@users.noreply.github.com>
2019-03-28 15:49:20 +01:00
Daniel Marjamäki c7155a8e08 Removed '--experimental-fast' flag 2019-03-17 08:19:56 +01:00
versat 4e051ef865 donate-cpu.py: Improve Python library detection.
The official documentation recommends to include the Python C API via
`#include "Python.h"`:
https://docs.python.org/3/c-api/intro.html
And many projects do it exactly this way, that is why the client script
often does not detect the usage of the Python C API.
2019-03-15 16:16:38 +01:00
versat 49f2fd6d4a donate-cpu.py: Detect and use ruby library. 2019-03-15 15:34:06 +01:00
Sebastian 1230357146
Library: Add initial configuration with tests for SQLite3 library. (#1737) 2019-03-13 13:57:40 +01:00
orbitcowboy b9623eb4a6 donate-cpu.py: Do not used windows.cfg in daca@home for now. 2019-03-08 17:30:39 +01:00
orbitcowboy 267074964d donate-cpu.py: Auto detect <windows.h> and activate windows.cfg. 2019-03-07 00:59:57 +01:00
orbitcowboy fee5639fc8 donate-cpu.py: Enable microsoft_sal library in case <sal.h> is found by scanPackage(). 2019-03-05 18:18:24 +01:00
versat 0498b9ec34 opgengl.cfg: Introduce OpenGL library configuration file.
Prepare donate-cpu.py to detect OpenGL library usage but comment it
out for now.
2019-03-04 14:10:06 +01:00
orbitcowboy d4511d3a22 donate-cpu.py: Auto detect cppunit configuration after next release. 2019-03-03 17:05:40 +01:00
orbitcowboy d1b42434f3 donate-cpu.py: Auto detect cppunit configuration after next release. 2019-03-03 17:04:03 +01:00
Sebastian 1877476f61
donate-cpu.py: Add new argument `--max-packages=N`. (#1718)
The client script will exit after the specified number of packages
have been processed. 0 means infinitely.
Useful for example to regularly quit the script, check for updates to
the client and start it again. Or as an alternative to the `--stop-time`
argument.
2019-03-01 22:34:50 +01:00
orbitcowboy 2a8f42d0f4 donate-cpu.py: Also extract 'h++'-files. 2019-02-28 22:41:21 +01:00
Sebastian 8c32cfe853
donate-cpu.py: Fix crash during decoding under Python 3. (#1710)
Python 3 directly decodes the text when it is read(). If there is any
invalid UTF-8 character in the text an exception is thrown (IIRC it is
UnicodeDecodeError). Opening the file with `error='ignore'` avoids
throwing an exception and just ignores the invalid character. Since
this is only possible since Python version 3 there must be extra code
for older versions.
The test script has been enhanced. It now also uses a package which
contains a file with at least one invalid UTF-8 character.
2019-02-27 20:01:38 +01:00
Sebastian 117eed2255
donate-cpu.py: Also extract .qml files. (#1708)
.qml files are used when the Qt library is loaded.
2019-02-27 14:59:19 +01:00
versat bcd7830fcb donate-cpu.py: Detect and use googletest library 2019-02-23 18:47:19 +01:00
Sebastian 9379266a52
donate-cpu.py: Improve regex to find more includes. (#1688)
Now also found:
- Includes directly at the beginning of a file
- Indented includes
- Includes where there is no white-space between
  "include" and header name
2019-02-23 17:13:35 +01:00
orbitcowboy 52c7a1b1c2 donate-cpu.py: Auto detect TinyXML2 cfg. 2019-02-22 17:51:47 +01:00
Sebastian 70ea6bfb9b
libcerror.cfg: Add libcerror configuration file. (#1684)
The function libcerror_error_set() is currently the function for which
daca@home most often reports a missing configuration (more than 80000
times).
Official repository of libcerror: https://github.com/libyal/libcerror
The library configuration has been tested with the library libvhdi:
ftp://ftp.se.debian.org/debian/pool/main/libv/libvhdi/libvhdi_20181227.orig.tar.gz
2019-02-22 12:52:07 +01:00
Sebastian e27a44a0ea
donate-cpu.py: Improve header detection by using regex. (#1679)
This detects more includes / headers. For example includes like
"# include <gtk/gtk.h>" with a space before "include" as it is used in
the package http://cppcheck.osuosl.org:8000/gbatnav are now also
detected.
The regex search also searches all includes for one library in one go
instead of one include per loop.
Tested with several packages to make sure libraries that were detected
before are still detected.
2019-02-21 08:14:47 +01:00
versat eeedb44e92 donate-cpu.py: Fix regression: Also upload results if only crashes happened
If "head" and "OLD_VERSION" both crash there are no messages and the
variable `results_exist" is set to False. But still the results must be uploaded
for the crashes to be saved also.
Tested with the package http://cppcheck.osuosl.org:8000/double-conversion
2019-02-19 13:29:32 +01:00
rikardfalkeborn 1c1778e4df donate_cpu: Refactor library checks (#1675) 2019-02-19 11:50:39 +01:00
versat a41e663cd1 Donate CPU: Optimize `list` and `dict` initialization
Initialization with `[]` and `{}` is faster than with `list()` and `dict()`.
Details:
https://stackoverflow.com/questions/30216000/why-is-faster-than-list
2019-02-12 09:47:57 +01:00
versat cf00e0383d donate-cpu.py: Improve Qt library detection
Tested locally
2019-02-11 15:38:28 +01:00
Sebastian 110248c8d8
donate-cpu.py: Optimize string concatenation (use list) (#1657)
Using a list and join it in the end is really much faster when there are several hundred or thousands of strings.
2019-02-10 08:57:35 +01:00
Daniel Marjamäki 78ea6d71ac donate-cpu: run cppcheck with --experimental-fast also 2019-02-09 22:05:12 +01:00
versat bdfc20a482 donate-cpu.py: Enable "python" and "boost" library detection
Only enable the library option for those configurations if the
corresponding .cfg file exists to not crash Cppcheck if an
older version than 1.87 is used as the "old" version.
2019-02-09 18:11:09 +01:00
Sebastian a997d0b387
donate-cpu.py: Introduce version number for client script (#1654)
The version number is added to the result output so issues could be more easily reproduced and in the future if it is necessary to handle any specific version of the client by the server in a special way (for example because it is erroneous) this is possible now.
2019-02-09 09:27:54 +01:00
versat ed4038b1df donate-cpu.py: Minor cleanup: Move `--library=gnu` to `libraries` variable 2019-02-08 11:19:51 +01:00
versat 06d16622e9 donate-cpu.py: Use SDL library configuration when applicable
Tested locally.
Searching for `#include <SDL.h>` does not always work. Sometimes `#include "SDL.h"` is used, but for now lets see for how many packages it works this way.
2019-02-05 13:38:42 +01:00
amai2012 d24c27c9f8 donate-cpu: Postpone usage of python.cfg after release 1.87 2019-02-04 13:07:57 +01:00
versat dcff026da0 donate-cpu.py: Simplify check if line is not empty
In Python an empty string returns False in a conditional check. No need to calculate lenght or compare it with an empty string.
2019-02-04 11:25:29 +01:00
Sebastian 360ba33e4c
donate-cpu.py: Use Boost library configuration if appropriate (#1587)
Add check to only use boost library if boost.cfg exists since it is very new.
If the boost.cfg does not exist Cppcheck would exit with an error and we
would not get the desired results.
For now the boost library configuration detection is commented out. It
should be enabled after the release of Cppcheck 1.87.
Tested locally.
2019-02-04 11:05:14 +01:00
amai2012 c9d77db39a Change pre-defined macro from __GCC__ to __GNUC__ 2019-02-03 21:56:48 +01:00
amai2012 74f3e4cce2 Fix typo 2019-02-03 21:27:47 +01:00
amai2012 89a5eb36ad Add initial configuration file for Pytho C API 2019-02-03 21:08:42 +01:00
amai2012 8078890b1b Detect X11 and Motif library in daca 2019-01-25 21:47:16 +01:00
Sebastian 1066ef87e5 donate-cpu.py: Enhance library configuration detection (#1625)
Optimize checking of different includes for the same library by using a `list` instead of calling `hasInclude()` several times.
Add includes for gtk and qt library detection that were found missing when looking at several daca@home packages.
2019-01-25 21:41:28 +01:00
Sebastian b486eb502a
donate-cpu.py: Always use gnu.cfg for daca analysis (#1622) 2019-01-24 19:23:18 +01:00
Sebastian a410cea59a
Donate CPU: Collect information messages and provide checkLibrary reports (#1610)
Related trac ticket: https://trac.cppcheck.net/ticket/8947
Enable information messages and "--check-library" in the Cppcheck
parameters.
Store the information messages and the rest of the messages in different
variables and upload them separately.
The server stores the information messages in a sub-directory similarly
to the normal issue messages in one file per package.
Reports for "checkLibraryFunction" and "checkLibraryNoReturn" message ids
are generated by the server now.
2019-01-22 15:27:13 +01:00
versat dfaba30290 donate-cpu.py: Fix possible usage of none type variable
"bandwidth_limit" is None by default and therefore can not be used for string
concatenation in this case.
2019-01-19 12:45:08 +01:00
Sebastian e3db99a16e
donate-cpu.py: Add option for limiting bandwidth (#1606)
With this new option it is possible to limit the download rate / bandwidth that wget uses to download packages.
The value is verified by running wget with it and check if there is a parse error (exit code 2). If the value is not valid an error message is shown and the script exits with an error code.
2019-01-18 21:36:03 +01:00
versat be14517bcd donate-cpu.py: Fix too few return values in scanPackage()
In case a crash is detected the scanPackage() function returns too few values (three instead of four).
This likely leads to an exception then.
2019-01-17 14:58:15 +01:00
orbitcowboy c3005f08ab donate-cpu.py: Comment out boost library detection. After next release it shall be enabled again. 2019-01-13 20:37:41 +01:00
orbitcowboy fc2557d43d donate-cpu.py: Added detection of boost-library. 2019-01-13 17:55:28 +01:00
Sebastian bae4040060
donate-cpu.py: Save some system / platform information (#1590)
Save some system information which could help debugging the donate-cpu script and Cppcheck.
2019-01-11 20:07:11 +01:00
amai2012 d3a10d3675 #8769 Don't check .cl files 2019-01-10 17:02:57 +01:00
Sebastian f673d552cb
donate-cpu.py: Add Cppcheck cmd line options to the output (#1581)
It is interesting what libraries, how many jobs, ... were used during analysis.
Tested locally.
2019-01-09 18:38:00 +01:00
Sebastian 2b008b473a
donate-cpu.py: Fix some PEP 8 and other warnings (#1579)
There should not be functional changes.
Tested locally.
2019-01-09 16:01:53 +01:00
orbitcowboy 126bd2bb5d
donate-cpu.py: Improved detection of wxWidgets code in scanPackage. (#1580) 2019-01-09 15:59:48 +01:00
Sebastian c8fde7b963
donate-cpu.py: Add '--test' parameter for testing against local server. (#1559)
Similar to the '--test' option for the donate-cpu-server.py script that lets the server listen locally on port 8001 the client now can also work against such a server by using the same option '--test'.
2019-01-03 13:42:33 +01:00
Sebastian 56d502bf37 donate-cpu.py: Detect and use zlib and gtk library config; fix --help output. (#1534)
* donate-cpu.py: Fix --help output (add --package description)
* donate-cpu.py: Use zlib and gtk library if appropriate.
2018-12-20 21:15:42 +01:00
versat 7ff0968853 donate-cpu.py: Fix indentation 2018-12-20 14:26:00 +01:00
Daniel Marjamäki 49e500d6bf donate-cpu: use 1.86 as base version 2018-12-08 11:39:44 +01:00
Sebastian fe6f4193fd donate-cpu.py: Fix Python3 compatibility problem with f.read().decode() (#1507)
With Python3 f.read() directly returns a string object that has no
decode() function. As a workaround AttributeError exceptions during
calling the decode() function are ignored and the data read from the file
is left unchanged.
With Python2 calling the decode() function is necessary and still done.
2018-12-07 06:51:07 +01:00
Sebastian a4dd423a18 donate-cpu.py: Align multi line output in diff (trac #8868) (#1508)
This is a very simple/stupid fix that just inserts 5 spaces for every line except the first.
2018-12-06 17:36:22 +01:00
Daniel Marjamäki 895a374c9a donate-cpu: fix exception in hasInclude when file is link 2018-12-05 10:25:35 +01:00
Daniel Marjamäki 0249e408d3 donate-cpu: Try to fix exception in hasInclude 2018-12-03 18:15:07 +01:00
Daniel Marjamäki 768eb79653 donate-cpu: Try to fix UnicodeEncodeError in sendAll 2018-12-03 09:02:05 +01:00
Daniel Marjamäki 14bcdd1c3a donate-cpu: fix syntax error 2018-12-01 20:05:50 +01:00
Daniel Marjamäki e5010da931 Donate CPU: include wxwidgets/qt if we detect that the project uses those headers 2018-12-01 20:02:13 +01:00
Daniel Marjamäki ffef44af4c donate-cpu: better detection of crashes 2018-12-01 13:25:48 +01:00
Daniel Marjamäki 86f984020b donate-cpu: save all warnings reported by head 2018-11-29 21:28:49 +01:00
Daniel Marjamäki 5532d1f877 donate-cpu: Add GetCppcheckVersions command 2018-11-29 21:19:45 +01:00
Daniel Marjamäki 15d0aa503b donate-cpu: only retry uploading package if it did not work 2018-11-28 19:30:14 +01:00
Daniel Marjamäki e501b6a460 donate-cpu: try to handle exceptions when uploading better 2018-11-27 12:05:21 +01:00
Daniel Marjamäki a872b3f57a donate-cpu: take care of unhandled exception when unpacking archive 2018-11-22 08:44:24 +01:00
Daniel Marjamäki cba87f913e donate-cpu: try to catch socket.error better 2018-11-20 11:18:18 +01:00
Daniel Marjamäki 4d8a85ec4d donate-cpu: add --package argument 2018-11-19 10:04:53 +01:00
Daniel Marjamäki 5eddcb0827 donate-cpu: use --template=daca2 so we can see some more details in the warnings 2018-11-18 16:26:56 +01:00
amai2012 6b05f8e867 Don't echo file names being extracted from archive 2018-11-17 18:29:12 +01:00
Daniel Marjamäki d95d884bba donate-cpu: try to handle socket error better when getting package 2018-11-14 21:41:16 +01:00
Daniel Marjamäki 2d61ca8883 donate-cpu: try to handle OSError when unpacking a tar file 2018-11-12 14:45:39 +01:00
rikardfalkeborn c9f768a915 donate_cpu: use https url (#1429)
Avoids git printing a warning about redirecting to https every time
the clone command is issued. The warning was:

    warning: redirecting to https://github.com/danmar/cppcheck.git/
2018-10-17 06:14:06 +02:00
Daniel Marjamäki 441d50df25 donate-cpu: use 1.85 as base version 2018-10-14 16:31:13 +02:00
Sebastian 996334eead
Donate CPU: Only extract relevant source files from archives #8716 (#1379)
Use python tarfile instead of tar to extract the packages.
Only extract source files of interest.
Skip dangerous files that could overwrite files outside the temp folder.
Fixes https://trac.cppcheck.net/ticket/8716
2018-09-15 18:56:46 +02:00
Daniel Marjamäki 91a8807810 Donate CPU: include inconclusive results 2018-09-09 13:16:39 +02:00
Sebastian 7fdd039bee
donate-cpu.py: Fix crash when wget destination file does not exist. (#1368)
When os.remove() tried to remove a file that did not exist (which is the case when the script is started for the first time or the working directory has been cleared) a FileNotFoundError was issued and the script just crashed.
2018-09-07 15:59:59 +02:00
versat 1d52875d4f donate-cpu.py: Fix parameter name in comments/help text. 2018-09-06 13:15:54 +02:00
Daniel Marjamäki 79840add38 Donate CPU: Use shutil.rmtree instead of 'rm' 2018-09-01 17:04:34 +02:00
Daniel Marjamäki 173c737a5c Fixed #8718 (Donate CPU: Optionally allow specifying a custom workfolder) 2018-08-31 14:29:15 +02:00
Daniel Marjamäki c1f436f7c0 Donate CPU: Try to handle crashes better 2018-08-29 22:07:48 +02:00
Daniel Marjamäki 4ef59754a4 Donate CPU: Added a '--stop-time' argument 2018-08-29 11:08:56 +02:00
Daniel Marjamäki 50ded8d789 Donate CPU: Include 'elapsed time' in the output 2018-08-29 06:51:33 +02:00
Daniel Marjamäki eadb4226e9 Donate CPU: Try to fix utf-8 decode problem. Just skip invalid chars, we're not very interested in those. 2018-08-28 12:02:28 +02:00
Daniel Marjamäki 629639f6ae Donate CPU: Updated scripts 2018-08-26 16:47:20 +02:00
Daniel Marjamäki 6ec6f70f2d Donate CPU: Concentrate on the diff 2018-08-26 16:23:42 +02:00
Daniel Marjamäki b2f0bd315d Donate CPU: Diff results in the client 2018-08-26 13:42:01 +02:00
Daniel Marjamäki 829a373416 Python cleanup 2018-08-25 20:43:20 +02:00
Daniel Marjamäki c7880e2c8d Donate CPU: Made client python3 compatible 2018-08-25 20:34:43 +02:00
Daniel Marjamäki 7049249460 Donate CPU: Tried to make scripts more robust 2018-08-25 18:38:51 +02:00
Daniel Marjamäki 33b93cdd3a Donate CPU: send data in chunks 2018-08-25 10:25:05 +02:00
Daniel Marjamäki 045ef9a715 Donate CPU: More updates. Run both 1.84 and head. 2018-08-24 21:07:50 +02:00
Daniel Marjamäki 28d1682a54 Donate CPU: Some fixes in donate-cpu.py 2018-08-24 18:49:11 +02:00