Commit Graph

339 Commits

Author SHA1 Message Date
Daniel Marjamäki d9a9743b8a Revert "Calculate token scopes in advance rather than as the tokenlist is iterated (#1882)"
This reverts commit 0d7836f3a0.
2019-06-30 09:12:18 +02:00
rebnridgway 0d7836f3a0 Calculate token scopes in advance rather than as the tokenlist is iterated (#1882)
* Added scopeinfo member to token class

Moved ScopeInfo2 declaration here as well because that's where it needs to be now.

* Added scopeinfo accessors and declaration to class

* Add new method for calculating scopes

This replaces the methods in the TemplateSimplifier which calculate the current scope as the token list is iterated.  The old method required checking if the scope had changed for every token multiple times (for multiple iterations), which was surprisingly costly.  Calculating scopes in advance like this decreases runtime on a worst-case file by around thirty percent.
ScopeInfo objects are disposed of when the TemplateSimplification is done as they are not used later.

* Add calculateScopes method to header

* Removed code that calculated current scope

This has been replaced by code that calculates the scopes up front and stores them with each token, which is much faster.

* Fixed compile errors from extra parentheses

* Added missing code to fix memory leak

* Added code to actually clean up ScopeInfo structs

* Tidy up a dodgy for loop

* Convert argument to const ref

* Calculate missing scopes

As the templatesimplificator expands templates and does multiple passes it needs to make sure all scopes are calculated.

* Remove copying the scope to the next token

This is now done properly when scopes are calculated.

* Remove call to calculateScopes

This is now done by the TemplateSimplifier.

* Recalculate scopes for every pass of simplifyTemplates

* Add code to calculate extra scopes as they are added

I thought that this might be useful for calculating scopes when Tokens are created, but as there are several ways of creating Tokens that don't guarantee that they are placed in a list it is easier to just calculate scopes when you know you have a list and when you know you're adding to a list.

* Fix several bugs and poorly designed code

Remove the global scopes collection, and clean them up instead by iterating through the tokenlist to find them.  This means scopes can be calculated by functions in the Token class as well as in the Tokenizer class without leaking the scope object.
Fix a couple of bugs in the calculateScopes method and make it more efficient.

* Remove unnecessary calls to calculateScopes

* Move brace to correct position

Calculating scopes during insertToken only needs to happen if we created a new Token.

* Handle 'using namespace' declarations separately

This fixes a bug caused by a statement matching 'struct B < 0 > ;'

* Fix argument name mismatch

* Actually use newScopeInfo when inserting Token

* Switch to using shared_ptr to hold scopeInfos

This means ScopeInfo2 objects get properly cleaned up when they are no longer needed.

* Change ScopeInfo member to be a shared_ptr

* Update code to use shared_ptr

* Add missing include for shared_ptr

* Remove unnecessary cleanup code

This has been replaced by shared_ptr for ScopeInfo2 objects
2019-06-29 14:34:22 +02:00
Daniel Marjamäki beea7fa8d2 Token::index(): Created Token member that indicates the Token position in the token list. It can be used to quickly check if tok1 precedes tok2. 2019-06-21 22:16:23 +02:00
orbitcowboy 821903e2e1 Running astyle [ci skip] 2019-06-02 13:29:20 +02:00
Paul Fultz II 6ae7be0f53 Fix FP with lifetime containers (#1865) 2019-06-02 10:14:48 +02:00
adamjrichter 47ce998e6e lib: Separate statements of the form "assert(a && b)" into "assert(a)" and "assert(b)", for more precise diagnostics. (#1825) 2019-05-07 10:28:31 +02:00
Paul Fultz II 091f4bcf8d Add check for unnecessary search before insertion
This will warn for cases where searching in an associative container happens before insertion, like this:

```cpp
void f1(std::set<unsigned>& s, unsigned x) {
    if (s.find(x) == s.end()) {
        s.insert(x);
    }
}

void f2(std::map<unsigned, unsigned>& m, unsigned x) {
    if (m.find(x) == m.end()) {
        m.emplace(x, 1);
    } else {
        m[x] = 1;
    }
}
```

In the case of the map it could be written as `m[x] = 1` as it will create the key if it doesnt exist, so the extra search is not necessary.

I have this marked as `performance` as it is mostly concerning performance, but there could be a copy-paste error possibly, although I dont think thats common.
2019-05-02 11:04:23 +02:00
Rikard Falkeborn c7d7f8738c Optimize astStringVerbose() for large arrays (#1815)
Change the astStringVerbose() recursion to extend a string instead of
returning one. This has the benefit that for tokens where the recursion
runs deep (typically large arrays), the time savings can be substantial
(see comments on benchmarks further down).

The reason is that previously, for each token, the astString of its
operands was constructed, and then appended to this tokens astString.
This led to a lot of unnecessary string copying (and with that
allocations). Instead, by passing the string by reference, the number
of temporary strings is greatly reduced.

Another way of seeing it is that previously, the string was constructed
from end to beginning, but now it is constructed from the beginning to
end. There was no notable speedup by preallocating the entire string
using string::reserve() (at least not on Linux).

To benchmark, the changes and master were tested on Linux using the
commands:

	make
	time cppcheck --debug --verbose $file >/dev/null

i.e., the cppcheck binary was compiled with the settings in the
Makefile. Printing the output to screen or file will of course take
longer time.

In Trac ticket #8355 which triggered this change, an example file from the
Wine repository was attached. Running the above cppcheck on master took
24 minutes and with the changes in this commmit, took 22 seconds.

Another test made was on lib/tokenlist.cpp in the cppcheck repo, which is
more "normal" file. On that file there was no measurable time difference.

A synthetic benchmark was generated to illustrate the effects on dumping
the ast for arrays of different sizes. The generate code looked as
follows:

	const int array[] = {...};

with different number of elements. The results are as follows (times are
in seconds):

	N	master optimized
	10	0.1    0.1
	100	0.1    0.1
	1000	2.8    0.7
	2000	19     1.8
	3000	53     3.8
	5000	350    10
	10000	3215   38

As we can see, for small arrays, there is no time difference, but for
large arrays the time savings are substantial.
2019-04-30 13:35:48 +02:00
Daniel Marjamäki b1b5b27b4e addons: write column in error message(s) 2019-04-13 10:22:13 +02:00
Rikard Falkeborn 794f65bac1 Handle prefixed strings and characters in Token (#1742)
This makes it possible to call getStrLength() and similar functions
before the tokenizer is called.
2019-03-18 06:18:25 +01:00
Daniel Marjamäki 92f4113b59 Array index: Checking array index out of bounds for dynamic buffers 2019-03-17 13:09:15 +01:00
Daniel Marjamäki df8cfe2fc6 Travis: Fix Cppcheck warning 2019-03-10 12:24:41 +01:00
Rikard Falkeborn 6a3dd9a185 Handle concatenated string and char literals
This handles concatenated strings and characters from simplecpp.
Previously, L'c' would be preprocessed to the tokens  "L" and "'c'".
cppcheck would then remove the "L" token and set "'c'" to be a wide
character literal. Now, it needs to remove the prefix instead.

When doing this, add handling of utf32 encoded literals (U) and UTF-8
encoded literals (u8).
2019-03-10 10:38:50 +01:00
Daniel Marjamäki bd7790fd8c Update copyright year 2019-02-09 07:24:06 +01:00
Daniel Marjamäki a26287f491 Fix Token::findExpressionStartEndTokens() regression 2019-01-20 13:23:19 +01:00
Daniel Marjamäki 8da4e31c42 Fixed #8941 (False Positive: Variable 'f' is assigned a value that is never used.) 2019-01-20 13:20:23 +01:00
Daniel Marjamäki b97b3b7ef8 astyle formatting
[ci skip]
2018-12-21 13:54:59 +01:00
rebnridgway 431d068339 Several fairly significant optimisations (#1518)
* Code changes for Token::mImpl optimisation

* Added new TokenImpl optimisation

Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation.  In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%.

* Several optimisations

Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it.
Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations.
Moved template removal to the end of the list as this reduces redundant iteration and saves time.

* Added tok argument to simplifyCalculations

This means callers can avoid unnecessary work if they know which tokens have already been simplified.  Passing nullptr indicates the original behaviour (starting from the front of the list).

* Removed mention of member from another change

* Re-added and optimised some code deleted in error

Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches.  Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators.

* Changed mTemplateInstantiations to a vector

This is an optimisation that makes repeated linear searches of this collection significantly faster. 
Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized.

* A cleaner optimisation to removing template tokens

This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified.

* Reverted vector to list

Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members.

* Added member for template simplifier pointer

This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches.

* Turned loop inside out

This means we only have to iterate through the std::list once.  std::list is very expensive to iterate through.

* Latest code from danmar and fixed optimisations

In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list.  Previous optimisations to this weren't very effective and broke some edge cases.

* Added changes from danmar

Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently.

* Tokens can have many templateSimplifierPointers

* templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
Daniel Marjamäki 37416010ef Unused value: Fix false positive (ast, {}) 2018-12-14 18:56:09 +01:00
Daniel Marjamäki 39c04b650d astyle formatting
[ci skip]
2018-11-14 19:10:52 +01:00
Paul Fultz II 54453c5802 Fix FP when copying pointer to string (#1479) 2018-11-14 06:59:25 +01:00
Paul Fultz II e839001f3d Fix warnings (#1478) 2018-11-14 06:14:04 +01:00
IOBYTE 17a8a4898d Fix another template simplifier namespace bug. (#1467)
* Fix another template simplifier namespace bug.

* Add missing forward declarations caused by token bug.
2018-11-07 21:25:42 +01:00
Daniel Marjamäki 58b21e3071 Refactoring: Reuse Token::expressionString() logic for finding right-most leaf in tree 2018-10-18 12:09:55 +02:00
IOBYTE 0a30768b59 Fixed #8693 (Template specialization: Constructor detected as normal … (#1418)
* Fixed #8693 (Template specialization: Constructor detected as normal function (functionStatic error))

Refactor template simplifier to remove the existing full specialization
function expandSpecialized and allow full specializations to use the
existing function expandTemplate.  The function expandTemplate was
modified to either expand the template like it originally did by copying
it or to modify the template in place.  Both instantiated and
uninstantiated full specializations are modified in place.  This also
fixes #8692 and probably other related tickets as well.

The function simplifyTemplates now tries twice to simplify templates so
more templates can be simplified.  We should try as many times as
necessary to find all possible templates.  We can't do that now because
uninstantiated templates are left unchanged.  It is relatively straight
forward to have the new code also expand in place uninstantiated
templates with their symbolic types but namespaces are not handled
properly (ticket #8671) and it would introduce regressions.

* Fix travis warnings.
2018-10-14 16:57:07 +02:00
Daniel Marjamäki 81f54f7094 Fixed #8681 (ValueFlow: Container size) 2018-08-10 11:29:16 +02:00
Paul Fultz II 0d35a96594 Improve checking of mismatch iterators (#1293) 2018-07-26 22:00:48 +02:00
Daniel Marjamäki 7c4820e047 Rename Library::isargvalid() 2018-07-15 23:05:48 +02:00
rikardfalkeborn 491ee577c6 Support floats in valid config (#1297)
* Add tests for invalid ranges

* Refactor loadLibErrors

This reduces the amount of code slightly and will simplify adding
more tests.

* Handle empty valid field

Before this change, the sequence <valid></valid> in a config file would
result in a segmentation fault. Now an empty field results in the error
message:

cppcheck: Failed to load library configuration file 'mycfg.cfg'. Bad attribute value '""'

* Add support for valid for floating point arguments

Previously, it was not possible to add valid ranges to floating point
arguments since it only handled integers. This made ranges not work well
for floating point arguments since arguments were cast to integers
before the ranges were handled.

Fix this by using doubles instead of integers if the argument is a float.
Add some tests for this and make sure errors are printed with enough
precision (somewhat arbitrarily chosen).

Note that it is still only possible to add integer ranges (i.e. -1:1).

* Add support for floats in configuration valid range

Now that it is possible to handle decimal arguments, there is no reason
to not allow non-integer ranges. Take care to not allow broken
configurations.

* Move check to within if-clause

* Move asin{,f,l} and acos{,f,l} input checks to config file
2018-07-15 22:47:56 +02:00
Daniel Marjamäki ad4ce84cf7 Rename private member variables 2018-06-17 17:20:16 +02:00
Daniel Marjamäki ac28ae5788 Renamed Token::_str 2018-06-16 23:03:15 +02:00
Daniel Marjamäki 237314fb51 Rename _values to mValues 2018-06-16 21:45:53 +02:00
Daniel Marjamäki 0452c5a4e4 Renamed _bits to mBits 2018-06-16 21:43:44 +02:00
Daniel Marjamäki d468eca772 Renamed _valuetype to mValueType 2018-06-16 21:42:40 +02:00
Daniel Marjamäki 82169b6247 Renamed _originalName to mOriginalName 2018-06-16 21:40:25 +02:00
Daniel Marjamäki 759097301b Rename _link to mLink 2018-06-16 20:30:09 +02:00
Daniel Marjamäki e8a00a38ac Rename _scope to mScope 2018-06-16 20:29:17 +02:00
Daniel Marjamäki 964841ba25 Rename _function to mFunction 2018-06-16 20:26:43 +02:00
Daniel Marjamäki 78974e9267 Rename _type to mType 2018-06-16 20:25:54 +02:00
Daniel Marjamäki 31f137f24d Renamed _fileIndex, _linenr, _col 2018-06-16 16:43:54 +02:00
Daniel Marjamäki a6e4663445 Renamed _progressValue to mProgressValue 2018-06-16 16:41:25 +02:00
Daniel Marjamäki aeea2e7427 Renamed _tokType to mTokType 2018-06-16 16:40:02 +02:00
Daniel Marjamäki 8e2b18a75a Rename _varId to mVarId 2018-06-16 16:38:50 +02:00
Daniel Marjamäki cc521bef99 renamed _tokensFrontBack to mTokensFrontBack 2018-06-16 16:22:35 +02:00
Daniel Marjamäki 26d58ddbb9 rename _astOperand1, _astOperand2, _astParent 2018-06-16 16:18:50 +02:00
Daniel Marjamäki 435aa70c3d rename _previous and _next 2018-06-16 16:16:55 +02:00
Daniel Marjamäki 363f4ca939 rename _flags to mFlags 2018-06-16 16:14:34 +02:00
amai2012 db7d267c02 Refactoring: Add _ prefix to Token member variables (#1280) 2018-06-09 08:14:24 +02:00
orbitcowboy de66eedd54 Improved const correctness of local variables. 2018-05-29 17:33:43 +02:00
rebnridgway 42a65c5160 Fix crash bug #8579 (#1238)
* Added declaration for deletePrevious function

* Added definition for deletePrevious function

* Fixed crash from deleteThis invalidating pointers

The crash was caused by deleteThis() invalidating the pointer to a constant variable usage.  This happened when a usage followed an assignment.  This fixes bug #8579.

* Added tokensFront to match tokensBack

This means deletePrevious can set the list's front if necessary.

* Initialised tokensFront in appropriate places

* Switched to using default Token constructor

* Switched to using Token default constructor

* Switched to using default constructor for Token

* Added missing argument to Token constructor

* Changed to use default constructor for Tokens

* Switched to using default constructor for Tokens

* Switched to using default constructor for Token

* Added new test for deleting front Token

Also made sure to use the correct constructor for Token in other tests.

* Syntax error

* Replaced tokensFront and tokensBack with a struct

This decreases the size of the Token class for performance purposes.

* Replaced tokensFront and tokensBack with a struct

* Added tokensFrontBack to destructor

* Reworked to use TokensBackFront struct

Also ran astyle.

* Reworked to use TokenList's TokensFrontBack member

* Reworked to use TokensFrontBack struct

* Reworked to use TokensFrontBack struct

* Reworked to work with TokensFrontBack struct

* Removed unnecessary scope operator

* Added missing parentheses

* Fixed syntax error

* Removed unnecessary constructor

* Default constructor now 0-initialises everything

This is safer for not using a temporary TokensFrontBack object, and doesn't use delegating constructors which aren't supported yet.

* Fixed unsafe null check

* Added missing explicit keyword

* Fixing stylistic nits

Removed default constructor as it has been superseded by the single-argument constructor with a default argument value.
Renamed listEnds to tokensFrontBack.
Fixed if statement that was supposed to be adding safety but would actually cause a crash if tokensFrontBack was null.

* Fixing stylistic nits

Removed default constructor and replaced it with a single-argument constructor with a default value.

* Fixing stylistic nits

Renamed _listEnds to _tokensFrontBack.

* Fixing stylistic nits

Renamed _listEnds to _tokensFrontBack.
2018-05-25 07:15:05 +02:00