cppcheck/lib/token.cpp

2549 lines
81 KiB
C++
Raw Normal View History

2008-12-18 22:28:57 +01:00
/*
* Cppcheck - A tool for static C/C++ code analysis
2023-01-28 10:16:34 +01:00
* Copyright (C) 2007-2023 Cppcheck team.
2008-12-18 22:28:57 +01:00
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
2008-12-18 22:28:57 +01:00
*/
#include "token.h"
2017-05-27 04:33:47 +02:00
#include "astutils.h"
#include "errortypes.h"
2017-05-27 04:33:47 +02:00
#include "library.h"
#include "settings.h"
#include "symboldatabase.h"
#include "tokenlist.h"
#include "utils.h"
2021-06-14 07:39:01 +02:00
#include "tokenrange.h"
#include "valueflow.h"
2017-05-27 04:33:47 +02:00
#include <algorithm>
#include <cassert>
2017-05-27 04:33:47 +02:00
#include <cctype>
#include <climits>
#include <cstdio>
#include <cstdint>
2008-12-18 22:28:57 +01:00
#include <cstring>
#include <functional>
2008-12-18 22:28:57 +01:00
#include <iostream>
#include <iterator>
#include <map>
2017-05-27 04:33:47 +02:00
#include <set>
#include <sstream> // IWYU pragma: keep
#include <stack>
#include <unordered_set>
2017-05-27 04:33:47 +02:00
#include <utility>
namespace {
struct less {
template<class T, class U>
bool operator()(const T &x, const U &y) const {
return x < y;
}
};
}
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
const std::list<ValueFlow::Value> TokenImpl::mEmptyValueList;
Fix crash bug #8579 (#1238) * Added declaration for deletePrevious function * Added definition for deletePrevious function * Fixed crash from deleteThis invalidating pointers The crash was caused by deleteThis() invalidating the pointer to a constant variable usage. This happened when a usage followed an assignment. This fixes bug #8579. * Added tokensFront to match tokensBack This means deletePrevious can set the list's front if necessary. * Initialised tokensFront in appropriate places * Switched to using default Token constructor * Switched to using Token default constructor * Switched to using default constructor for Token * Added missing argument to Token constructor * Changed to use default constructor for Tokens * Switched to using default constructor for Tokens * Switched to using default constructor for Token * Added new test for deleting front Token Also made sure to use the correct constructor for Token in other tests. * Syntax error * Replaced tokensFront and tokensBack with a struct This decreases the size of the Token class for performance purposes. * Replaced tokensFront and tokensBack with a struct * Added tokensFrontBack to destructor * Reworked to use TokensBackFront struct Also ran astyle. * Reworked to use TokenList's TokensFrontBack member * Reworked to use TokensFrontBack struct * Reworked to use TokensFrontBack struct * Reworked to work with TokensFrontBack struct * Removed unnecessary scope operator * Added missing parentheses * Fixed syntax error * Removed unnecessary constructor * Default constructor now 0-initialises everything This is safer for not using a temporary TokensFrontBack object, and doesn't use delegating constructors which aren't supported yet. * Fixed unsafe null check * Added missing explicit keyword * Fixing stylistic nits Removed default constructor as it has been superseded by the single-argument constructor with a default argument value. Renamed listEnds to tokensFrontBack. Fixed if statement that was supposed to be adding safety but would actually cause a crash if tokensFrontBack was null. * Fixing stylistic nits Removed default constructor and replaced it with a single-argument constructor with a default value. * Fixing stylistic nits Renamed _listEnds to _tokensFrontBack. * Fixing stylistic nits Renamed _listEnds to _tokensFrontBack.
2018-05-25 07:15:05 +02:00
Token::Token(TokensFrontBack *tokensFrontBack) :
mTokensFrontBack(tokensFrontBack),
2018-06-16 16:16:55 +02:00
mNext(nullptr),
mPrevious(nullptr),
2018-06-16 20:30:09 +02:00
mLink(nullptr),
2018-06-16 16:40:02 +02:00
mTokType(eNone),
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
mFlags(0)
2008-12-18 22:28:57 +01:00
{
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
mImpl = new TokenImpl();
2008-12-18 22:28:57 +01:00
}
Token::~Token()
2008-12-18 22:28:57 +01:00
{
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
delete mImpl;
2008-12-18 22:28:57 +01:00
}
2021-06-14 07:39:01 +02:00
/*
2021-08-07 20:51:18 +02:00
* Get a TokenRange which starts at this token and contains every token following it in order up to but not including 't'
* e.g. for the sequence of tokens A B C D E, C.until(E) would yield the Range C D
* note t can be nullptr to iterate all the way to the end.
*/
// cppcheck-suppress unusedFunction // only used in testtokenrange.cpp
2021-06-14 07:39:01 +02:00
ConstTokenRange Token::until(const Token* t) const
{
return ConstTokenRange(this, t);
}
static const std::unordered_set<std::string> controlFlowKeywords = {
"goto",
"do",
"if",
"else",
"for",
"while",
"switch",
"case",
"break",
"continue",
"return"
};
// Another list of keywords
static const std::unordered_set<std::string> baseKeywords = {
"asm",
"auto",
"break",
"case",
"const",
"continue",
"default",
"do",
"else",
"enum",
"extern",
"for",
"goto",
"if",
"inline",
"register",
"restrict",
"return",
"sizeof",
"static",
"struct",
"switch",
"typedef",
"union",
"volatile",
"while",
"void"
};
void Token::update_property_info()
2008-12-18 22:28:57 +01:00
{
2018-06-16 23:03:15 +02:00
setFlag(fIsControlFlowKeyword, controlFlowKeywords.find(mStr) != controlFlowKeywords.end());
2018-06-16 23:03:15 +02:00
if (!mStr.empty()) {
if (mStr == "true" || mStr == "false")
2017-10-15 01:27:47 +02:00
tokType(eBoolean);
else if (isStringLiteral(mStr))
tokType(eString);
else if (isCharLiteral(mStr))
tokType(eChar);
2018-06-16 23:03:15 +02:00
else if (std::isalpha((unsigned char)mStr[0]) || mStr[0] == '_' || mStr[0] == '$') { // Name
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (mImpl->mVarId)
2017-10-15 01:27:47 +02:00
tokType(eVariable);
else if (mTokensFrontBack && mTokensFrontBack->list && mTokensFrontBack->list->isKeyword(mStr))
tokType(eKeyword);
else if (baseKeywords.count(mStr) > 0)
tokType(eKeyword);
2018-06-16 16:40:02 +02:00
else if (mTokType != eVariable && mTokType != eFunction && mTokType != eType && mTokType != eKeyword)
2017-10-15 01:27:47 +02:00
tokType(eName);
2018-06-16 23:03:15 +02:00
} else if (std::isdigit((unsigned char)mStr[0]) || (mStr.length() > 1 && mStr[0] == '-' && std::isdigit((unsigned char)mStr[1])))
2017-10-15 01:27:47 +02:00
tokType(eNumber);
2018-06-16 23:03:15 +02:00
else if (mStr == "=" || mStr == "<<=" || mStr == ">>=" ||
(mStr.size() == 2U && mStr[1] == '=' && std::strchr("+-*/%&^|", mStr[0])))
2017-10-15 01:27:47 +02:00
tokType(eAssignmentOp);
2018-06-16 23:03:15 +02:00
else if (mStr.size() == 1 && mStr.find_first_of(",[]()?:") != std::string::npos)
2017-10-15 01:27:47 +02:00
tokType(eExtendedOp);
2018-06-16 23:03:15 +02:00
else if (mStr=="<<" || mStr==">>" || (mStr.size()==1 && mStr.find_first_of("+-*/%") != std::string::npos))
2017-10-15 01:27:47 +02:00
tokType(eArithmeticalOp);
2018-06-16 23:03:15 +02:00
else if (mStr.size() == 1 && mStr.find_first_of("&|^~") != std::string::npos)
2017-10-15 01:27:47 +02:00
tokType(eBitOp);
2018-06-16 23:03:15 +02:00
else if (mStr.size() <= 2 &&
(mStr == "&&" ||
mStr == "||" ||
mStr == "!"))
2017-10-15 01:27:47 +02:00
tokType(eLogicalOp);
2018-06-16 23:03:15 +02:00
else if (mStr.size() <= 2 && !mLink &&
(mStr == "==" ||
mStr == "!=" ||
2021-08-07 20:51:18 +02:00
mStr == "<" ||
2018-06-16 23:03:15 +02:00
mStr == "<=" ||
2021-08-07 20:51:18 +02:00
mStr == ">" ||
2018-06-16 23:03:15 +02:00
mStr == ">="))
2017-10-15 01:27:47 +02:00
tokType(eComparisonOp);
2021-04-22 19:15:22 +02:00
else if (mStr == "<=>")
tokType(eComparisonOp);
2018-06-16 23:03:15 +02:00
else if (mStr.size() == 2 &&
(mStr == "++" ||
mStr == "--"))
2017-10-15 01:27:47 +02:00
tokType(eIncDecOp);
2018-06-16 23:03:15 +02:00
else if (mStr.size() == 1 && (mStr.find_first_of("{}") != std::string::npos || (mLink && mStr.find_first_of("<>") != std::string::npos)))
2017-10-15 01:27:47 +02:00
tokType(eBracket);
else if (mStr == "...")
tokType(eEllipsis);
else
2017-10-15 01:27:47 +02:00
tokType(eOther);
} else {
2017-10-15 01:27:47 +02:00
tokType(eNone);
}
update_property_char_string_literal();
update_property_isStandardType();
}
static const std::unordered_set<std::string> stdTypes = { "bool"
2021-01-22 21:47:24 +01:00
, "_Bool"
, "char"
, "double"
, "float"
, "int"
, "long"
, "short"
, "size_t"
, "void"
, "wchar_t"
2021-08-07 20:51:18 +02:00
};
void Token::update_property_isStandardType()
{
isStandardType(false);
2018-06-16 23:03:15 +02:00
if (mStr.size() < 3)
return;
2018-06-16 23:03:15 +02:00
if (stdTypes.find(mStr)!=stdTypes.end()) {
isStandardType(true);
2017-10-15 01:27:47 +02:00
tokType(eType);
}
}
void Token::update_property_char_string_literal()
{
if (mTokType != Token::eString && mTokType != Token::eChar)
return;
isLong(((mTokType == Token::eString) && isPrefixStringCharLiteral(mStr, '"', "L")) ||
((mTokType == Token::eChar) && isPrefixStringCharLiteral(mStr, '\'', "L")));
}
bool Token::isUpperCaseName() const
{
if (!isName())
return false;
for (const char i : mStr) {
if (std::islower(i))
return false;
}
return true;
}
void Token::concatStr(std::string const& b)
{
mStr.pop_back();
mStr.append(getStringLiteral(b) + "\"");
if (isCChar() && isStringLiteral(b) && b[0] != '"') {
mStr.insert(0, b.substr(0, b.find('"')));
}
update_property_info();
}
std::string Token::strValue() const
{
2018-06-16 16:40:02 +02:00
assert(mTokType == eString);
std::string ret(getStringLiteral(mStr));
std::string::size_type pos = 0U;
while ((pos = ret.find('\\', pos)) != std::string::npos) {
ret.erase(pos,1U);
if (ret[pos] >= 'a') {
if (ret[pos] == 'n')
ret[pos] = '\n';
else if (ret[pos] == 'r')
ret[pos] = '\r';
else if (ret[pos] == 't')
ret[pos] = '\t';
}
if (ret[pos] == '0')
return ret.substr(0,pos);
pos++;
}
return ret;
}
void Token::deleteNext(nonneg int count)
2008-12-18 22:28:57 +01:00
{
while (mNext && count > 0) {
2018-06-16 16:16:55 +02:00
Token *n = mNext;
// #8154 we are about to be unknown -> destroy the link to us
2018-06-16 20:30:09 +02:00
if (n->mLink && n->mLink->mLink == n)
n->mLink->link(nullptr);
2018-06-16 16:16:55 +02:00
mNext = n->next();
delete n;
--count;
}
2018-06-16 16:16:55 +02:00
if (mNext)
mNext->previous(this);
else if (mTokensFrontBack)
mTokensFrontBack->back = this;
Fix crash bug #8579 (#1238) * Added declaration for deletePrevious function * Added definition for deletePrevious function * Fixed crash from deleteThis invalidating pointers The crash was caused by deleteThis() invalidating the pointer to a constant variable usage. This happened when a usage followed an assignment. This fixes bug #8579. * Added tokensFront to match tokensBack This means deletePrevious can set the list's front if necessary. * Initialised tokensFront in appropriate places * Switched to using default Token constructor * Switched to using Token default constructor * Switched to using default constructor for Token * Added missing argument to Token constructor * Changed to use default constructor for Tokens * Switched to using default constructor for Tokens * Switched to using default constructor for Token * Added new test for deleting front Token Also made sure to use the correct constructor for Token in other tests. * Syntax error * Replaced tokensFront and tokensBack with a struct This decreases the size of the Token class for performance purposes. * Replaced tokensFront and tokensBack with a struct * Added tokensFrontBack to destructor * Reworked to use TokensBackFront struct Also ran astyle. * Reworked to use TokenList's TokensFrontBack member * Reworked to use TokensFrontBack struct * Reworked to use TokensFrontBack struct * Reworked to work with TokensFrontBack struct * Removed unnecessary scope operator * Added missing parentheses * Fixed syntax error * Removed unnecessary constructor * Default constructor now 0-initialises everything This is safer for not using a temporary TokensFrontBack object, and doesn't use delegating constructors which aren't supported yet. * Fixed unsafe null check * Added missing explicit keyword * Fixing stylistic nits Removed default constructor as it has been superseded by the single-argument constructor with a default argument value. Renamed listEnds to tokensFrontBack. Fixed if statement that was supposed to be adding safety but would actually cause a crash if tokensFrontBack was null. * Fixing stylistic nits Removed default constructor and replaced it with a single-argument constructor with a default value. * Fixing stylistic nits Renamed _listEnds to _tokensFrontBack. * Fixing stylistic nits Renamed _listEnds to _tokensFrontBack.
2018-05-25 07:15:05 +02:00
}
void Token::deletePrevious(nonneg int count)
Fix crash bug #8579 (#1238) * Added declaration for deletePrevious function * Added definition for deletePrevious function * Fixed crash from deleteThis invalidating pointers The crash was caused by deleteThis() invalidating the pointer to a constant variable usage. This happened when a usage followed an assignment. This fixes bug #8579. * Added tokensFront to match tokensBack This means deletePrevious can set the list's front if necessary. * Initialised tokensFront in appropriate places * Switched to using default Token constructor * Switched to using Token default constructor * Switched to using default constructor for Token * Added missing argument to Token constructor * Changed to use default constructor for Tokens * Switched to using default constructor for Tokens * Switched to using default constructor for Token * Added new test for deleting front Token Also made sure to use the correct constructor for Token in other tests. * Syntax error * Replaced tokensFront and tokensBack with a struct This decreases the size of the Token class for performance purposes. * Replaced tokensFront and tokensBack with a struct * Added tokensFrontBack to destructor * Reworked to use TokensBackFront struct Also ran astyle. * Reworked to use TokenList's TokensFrontBack member * Reworked to use TokensFrontBack struct * Reworked to use TokensFrontBack struct * Reworked to work with TokensFrontBack struct * Removed unnecessary scope operator * Added missing parentheses * Fixed syntax error * Removed unnecessary constructor * Default constructor now 0-initialises everything This is safer for not using a temporary TokensFrontBack object, and doesn't use delegating constructors which aren't supported yet. * Fixed unsafe null check * Added missing explicit keyword * Fixing stylistic nits Removed default constructor as it has been superseded by the single-argument constructor with a default argument value. Renamed listEnds to tokensFrontBack. Fixed if statement that was supposed to be adding safety but would actually cause a crash if tokensFrontBack was null. * Fixing stylistic nits Removed default constructor and replaced it with a single-argument constructor with a default value. * Fixing stylistic nits Renamed _listEnds to _tokensFrontBack. * Fixing stylistic nits Renamed _listEnds to _tokensFrontBack.
2018-05-25 07:15:05 +02:00
{
while (mPrevious && count > 0) {
2018-06-16 16:16:55 +02:00
Token *p = mPrevious;
Fix crash bug #8579 (#1238) * Added declaration for deletePrevious function * Added definition for deletePrevious function * Fixed crash from deleteThis invalidating pointers The crash was caused by deleteThis() invalidating the pointer to a constant variable usage. This happened when a usage followed an assignment. This fixes bug #8579. * Added tokensFront to match tokensBack This means deletePrevious can set the list's front if necessary. * Initialised tokensFront in appropriate places * Switched to using default Token constructor * Switched to using Token default constructor * Switched to using default constructor for Token * Added missing argument to Token constructor * Changed to use default constructor for Tokens * Switched to using default constructor for Tokens * Switched to using default constructor for Token * Added new test for deleting front Token Also made sure to use the correct constructor for Token in other tests. * Syntax error * Replaced tokensFront and tokensBack with a struct This decreases the size of the Token class for performance purposes. * Replaced tokensFront and tokensBack with a struct * Added tokensFrontBack to destructor * Reworked to use TokensBackFront struct Also ran astyle. * Reworked to use TokenList's TokensFrontBack member * Reworked to use TokensFrontBack struct * Reworked to use TokensFrontBack struct * Reworked to work with TokensFrontBack struct * Removed unnecessary scope operator * Added missing parentheses * Fixed syntax error * Removed unnecessary constructor * Default constructor now 0-initialises everything This is safer for not using a temporary TokensFrontBack object, and doesn't use delegating constructors which aren't supported yet. * Fixed unsafe null check * Added missing explicit keyword * Fixing stylistic nits Removed default constructor as it has been superseded by the single-argument constructor with a default argument value. Renamed listEnds to tokensFrontBack. Fixed if statement that was supposed to be adding safety but would actually cause a crash if tokensFrontBack was null. * Fixing stylistic nits Removed default constructor and replaced it with a single-argument constructor with a default value. * Fixing stylistic nits Renamed _listEnds to _tokensFrontBack. * Fixing stylistic nits Renamed _listEnds to _tokensFrontBack.
2018-05-25 07:15:05 +02:00
// #8154 we are about to be unknown -> destroy the link to us
2018-06-16 20:30:09 +02:00
if (p->mLink && p->mLink->mLink == p)
p->mLink->link(nullptr);
Fix crash bug #8579 (#1238) * Added declaration for deletePrevious function * Added definition for deletePrevious function * Fixed crash from deleteThis invalidating pointers The crash was caused by deleteThis() invalidating the pointer to a constant variable usage. This happened when a usage followed an assignment. This fixes bug #8579. * Added tokensFront to match tokensBack This means deletePrevious can set the list's front if necessary. * Initialised tokensFront in appropriate places * Switched to using default Token constructor * Switched to using Token default constructor * Switched to using default constructor for Token * Added missing argument to Token constructor * Changed to use default constructor for Tokens * Switched to using default constructor for Tokens * Switched to using default constructor for Token * Added new test for deleting front Token Also made sure to use the correct constructor for Token in other tests. * Syntax error * Replaced tokensFront and tokensBack with a struct This decreases the size of the Token class for performance purposes. * Replaced tokensFront and tokensBack with a struct * Added tokensFrontBack to destructor * Reworked to use TokensBackFront struct Also ran astyle. * Reworked to use TokenList's TokensFrontBack member * Reworked to use TokensFrontBack struct * Reworked to use TokensFrontBack struct * Reworked to work with TokensFrontBack struct * Removed unnecessary scope operator * Added missing parentheses * Fixed syntax error * Removed unnecessary constructor * Default constructor now 0-initialises everything This is safer for not using a temporary TokensFrontBack object, and doesn't use delegating constructors which aren't supported yet. * Fixed unsafe null check * Added missing explicit keyword * Fixing stylistic nits Removed default constructor as it has been superseded by the single-argument constructor with a default argument value. Renamed listEnds to tokensFrontBack. Fixed if statement that was supposed to be adding safety but would actually cause a crash if tokensFrontBack was null. * Fixing stylistic nits Removed default constructor and replaced it with a single-argument constructor with a default value. * Fixing stylistic nits Renamed _listEnds to _tokensFrontBack. * Fixing stylistic nits Renamed _listEnds to _tokensFrontBack.
2018-05-25 07:15:05 +02:00
2018-06-16 16:16:55 +02:00
mPrevious = p->previous();
Fix crash bug #8579 (#1238) * Added declaration for deletePrevious function * Added definition for deletePrevious function * Fixed crash from deleteThis invalidating pointers The crash was caused by deleteThis() invalidating the pointer to a constant variable usage. This happened when a usage followed an assignment. This fixes bug #8579. * Added tokensFront to match tokensBack This means deletePrevious can set the list's front if necessary. * Initialised tokensFront in appropriate places * Switched to using default Token constructor * Switched to using Token default constructor * Switched to using default constructor for Token * Added missing argument to Token constructor * Changed to use default constructor for Tokens * Switched to using default constructor for Tokens * Switched to using default constructor for Token * Added new test for deleting front Token Also made sure to use the correct constructor for Token in other tests. * Syntax error * Replaced tokensFront and tokensBack with a struct This decreases the size of the Token class for performance purposes. * Replaced tokensFront and tokensBack with a struct * Added tokensFrontBack to destructor * Reworked to use TokensBackFront struct Also ran astyle. * Reworked to use TokenList's TokensFrontBack member * Reworked to use TokensFrontBack struct * Reworked to use TokensFrontBack struct * Reworked to work with TokensFrontBack struct * Removed unnecessary scope operator * Added missing parentheses * Fixed syntax error * Removed unnecessary constructor * Default constructor now 0-initialises everything This is safer for not using a temporary TokensFrontBack object, and doesn't use delegating constructors which aren't supported yet. * Fixed unsafe null check * Added missing explicit keyword * Fixing stylistic nits Removed default constructor as it has been superseded by the single-argument constructor with a default argument value. Renamed listEnds to tokensFrontBack. Fixed if statement that was supposed to be adding safety but would actually cause a crash if tokensFrontBack was null. * Fixing stylistic nits Removed default constructor and replaced it with a single-argument constructor with a default value. * Fixing stylistic nits Renamed _listEnds to _tokensFrontBack. * Fixing stylistic nits Renamed _listEnds to _tokensFrontBack.
2018-05-25 07:15:05 +02:00
delete p;
--count;
Fix crash bug #8579 (#1238) * Added declaration for deletePrevious function * Added definition for deletePrevious function * Fixed crash from deleteThis invalidating pointers The crash was caused by deleteThis() invalidating the pointer to a constant variable usage. This happened when a usage followed an assignment. This fixes bug #8579. * Added tokensFront to match tokensBack This means deletePrevious can set the list's front if necessary. * Initialised tokensFront in appropriate places * Switched to using default Token constructor * Switched to using Token default constructor * Switched to using default constructor for Token * Added missing argument to Token constructor * Changed to use default constructor for Tokens * Switched to using default constructor for Tokens * Switched to using default constructor for Token * Added new test for deleting front Token Also made sure to use the correct constructor for Token in other tests. * Syntax error * Replaced tokensFront and tokensBack with a struct This decreases the size of the Token class for performance purposes. * Replaced tokensFront and tokensBack with a struct * Added tokensFrontBack to destructor * Reworked to use TokensBackFront struct Also ran astyle. * Reworked to use TokenList's TokensFrontBack member * Reworked to use TokensFrontBack struct * Reworked to use TokensFrontBack struct * Reworked to work with TokensFrontBack struct * Removed unnecessary scope operator * Added missing parentheses * Fixed syntax error * Removed unnecessary constructor * Default constructor now 0-initialises everything This is safer for not using a temporary TokensFrontBack object, and doesn't use delegating constructors which aren't supported yet. * Fixed unsafe null check * Added missing explicit keyword * Fixing stylistic nits Removed default constructor as it has been superseded by the single-argument constructor with a default argument value. Renamed listEnds to tokensFrontBack. Fixed if statement that was supposed to be adding safety but would actually cause a crash if tokensFrontBack was null. * Fixing stylistic nits Removed default constructor and replaced it with a single-argument constructor with a default value. * Fixing stylistic nits Renamed _listEnds to _tokensFrontBack. * Fixing stylistic nits Renamed _listEnds to _tokensFrontBack.
2018-05-25 07:15:05 +02:00
}
2018-06-16 16:16:55 +02:00
if (mPrevious)
mPrevious->next(this);
else if (mTokensFrontBack)
mTokensFrontBack->front = this;
2008-12-18 22:28:57 +01:00
}
void Token::swapWithNext()
{
2018-06-16 16:16:55 +02:00
if (mNext) {
2018-06-16 23:03:15 +02:00
std::swap(mStr, mNext->mStr);
2018-06-16 16:40:02 +02:00
std::swap(mTokType, mNext->mTokType);
2018-06-16 16:16:55 +02:00
std::swap(mFlags, mNext->mFlags);
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
std::swap(mImpl, mNext->mImpl);
if (mImpl->mTemplateSimplifierPointers)
for (auto *templateSimplifierPointer : *mImpl->mTemplateSimplifierPointers) {
templateSimplifierPointer->token(this);
}
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (mNext->mImpl->mTemplateSimplifierPointers)
for (auto *templateSimplifierPointer : *mNext->mImpl->mTemplateSimplifierPointers) {
templateSimplifierPointer->token(mNext);
}
2018-06-16 20:30:09 +02:00
if (mNext->mLink)
mNext->mLink->mLink = this;
if (this->mLink)
this->mLink->mLink = mNext;
std::swap(mLink, mNext->mLink);
}
}
2017-08-25 23:30:04 +02:00
void Token::takeData(Token *fromToken)
{
2018-06-16 23:03:15 +02:00
mStr = fromToken->mStr;
2018-06-16 16:40:02 +02:00
tokType(fromToken->mTokType);
2018-06-16 16:14:34 +02:00
mFlags = fromToken->mFlags;
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
delete mImpl;
mImpl = fromToken->mImpl;
fromToken->mImpl = nullptr;
if (mImpl->mTemplateSimplifierPointers)
for (auto *templateSimplifierPointer : *mImpl->mTemplateSimplifierPointers) {
templateSimplifierPointer->token(this);
}
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
mLink = fromToken->mLink;
2018-06-16 20:30:09 +02:00
if (mLink)
mLink->link(this);
2017-08-25 23:30:04 +02:00
}
void Token::deleteThis()
{
2018-06-16 16:16:55 +02:00
if (mNext) { // Copy next to this and delete next
takeData(mNext);
mNext->link(nullptr); // mark as unlinked
deleteNext();
} else if (mPrevious) { // Copy previous to this and delete previous
2018-06-16 16:16:55 +02:00
takeData(mPrevious);
mPrevious->link(nullptr);
deletePrevious();
2011-10-13 20:53:06 +02:00
} else {
// We are the last token in the list, we can't delete
// ourselves, so just make us empty
str(";");
}
}
void Token::replace(Token *replaceThis, Token *start, Token *end)
{
// Fix the whole in the old location of start and end
if (start->previous())
start->previous()->next(end->next());
if (end->next())
end->next()->previous(start->previous());
// Move start and end to their new location
if (replaceThis->previous())
replaceThis->previous()->next(start);
if (replaceThis->next())
replaceThis->next()->previous(end);
start->previous(replaceThis->previous());
end->next(replaceThis->next());
if (end->mTokensFrontBack && end->mTokensFrontBack->back == end) {
while (end->next())
end = end->next();
end->mTokensFrontBack->back = end;
}
// Update mProgressValue, fileIndex and linenr
2012-01-22 00:02:55 +01:00
for (Token *tok = start; tok != end->next(); tok = tok->next())
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
tok->mImpl->mProgressValue = replaceThis->mImpl->mProgressValue;
// Delete old token, which is replaced
delete replaceThis;
}
const Token *Token::tokAt(int index) const
2008-12-18 22:28:57 +01:00
{
const Token *tok = this;
while (index > 0 && tok) {
tok = tok->next();
--index;
}
while (index < 0 && tok) {
tok = tok->previous();
++index;
2008-12-18 22:28:57 +01:00
}
return tok;
}
const Token *Token::linkAt(int index) const
{
const Token *tok = this->tokAt(index);
if (!tok) {
throw InternalError(this, "Internal error. Token::linkAt called with index outside the tokens range.");
}
return tok->link();
}
const std::string &Token::strAt(int index) const
2008-12-18 22:28:57 +01:00
{
const Token *tok = this->tokAt(index);
2018-06-16 23:03:15 +02:00
return tok ? tok->mStr : emptyString;
2008-12-18 22:28:57 +01:00
}
static int multiComparePercent(const Token *tok, const char*& haystack, nonneg int varid)
{
++haystack;
// Compare only the first character of the string for optimization reasons
switch (haystack[0]) {
case '\0':
case ' ':
case '|':
//simple '%' character
haystack += 1;
if (tok->isArithmeticalOp() && tok->str() == "%")
return 1;
break;
case 'v':
if (haystack[3] == '%') { // %var%
haystack += 4;
if (tok->varId() != 0)
return 1;
} else { // %varid%
if (varid == 0) {
throw InternalError(tok, "Internal error. Token::Match called with varid 0. Please report this to Cppcheck developers");
}
haystack += 6;
if (tok->varId() == varid)
return 1;
}
break;
case 't':
// Type (%type%)
{
haystack += 5;
if (tok->isName() && tok->varId() == 0 && (tok->str() != "delete" || !tok->isKeyword())) // HACK: this is legacy behaviour, it should return false for all keywords, except types
return 1;
}
break;
case 'a':
// Accept any token (%any%) or assign (%assign%)
{
if (haystack[3] == '%') { // %any%
haystack += 4;
return 1;
} else { // %assign%
haystack += 7;
if (tok->isAssignmentOp())
return 1;
}
}
break;
case 'n':
// Number (%num%) or name (%name%)
{
if (haystack[4] == '%') { // %name%
haystack += 5;
if (tok->isName())
return 1;
} else {
haystack += 4;
if (tok->isNumber())
return 1;
}
}
break;
case 'c': {
haystack += 1;
// Character (%char%)
if (haystack[0] == 'h') {
haystack += 4;
if (tok->tokType() == Token::eChar)
return 1;
}
// Const operator (%cop%)
else if (haystack[1] == 'p') {
haystack += 3;
if (tok->isConstOp())
return 1;
}
// Comparison operator (%comp%)
else {
haystack += 4;
if (tok->isComparisonOp())
return 1;
}
}
break;
case 's':
// String (%str%)
{
haystack += 4;
if (tok->tokType() == Token::eString)
return 1;
}
break;
case 'b':
// Bool (%bool%)
{
haystack += 5;
if (tok->isBoolean())
return 1;
}
break;
case 'o': {
++haystack;
if (haystack[1] == '%') {
// Op (%op%)
if (haystack[0] == 'p') {
haystack += 2;
if (tok->isOp())
return 1;
}
// Or (%or%)
else {
haystack += 2;
if (tok->tokType() == Token::eBitOp && tok->str() == "|")
return 1;
}
}
// Oror (%oror%)
else {
haystack += 4;
if (tok->tokType() == Token::eLogicalOp && tok->str() == "||")
return 1;
}
}
break;
default:
//unknown %cmd%, abort
2015-03-29 21:05:18 +02:00
throw InternalError(tok, "Unexpected command");
}
if (*haystack == '|')
haystack += 1;
else
return -1;
return 0xFFFF;
}
2019-07-14 17:31:26 +02:00
int Token::multiCompare(const Token *tok, const char *haystack, nonneg int varid)
2008-12-18 22:28:57 +01:00
{
const char *needle = tok->str().c_str();
const char *needlePointer = needle;
2012-03-25 11:51:59 +02:00
for (;;) {
if (needlePointer == needle && haystack[0] == '%' && haystack[1] != '|' && haystack[1] != '\0' && haystack[1] != ' ') {
2018-04-04 21:51:31 +02:00
const int ret = multiComparePercent(tok, haystack, varid);
if (ret < 2)
return ret;
2011-10-13 20:53:06 +02:00
} else if (*haystack == '|') {
if (*needlePointer == 0) {
// If needle is at the end, we have a match.
2008-12-18 22:28:57 +01:00
return 1;
}
2008-12-18 22:28:57 +01:00
needlePointer = needle;
++haystack;
} else if (*needlePointer == *haystack) {
if (*needlePointer == '\0')
return 1;
++needlePointer;
++haystack;
2011-10-13 20:53:06 +02:00
} else if (*haystack == ' ' || *haystack == '\0') {
if (needlePointer == needle)
return 0;
break;
2008-12-18 22:28:57 +01:00
}
// If haystack and needle don't share the same character,
// find next '|' character.
2011-10-13 20:53:06 +02:00
else {
needlePointer = needle;
2008-12-18 22:28:57 +01:00
2011-10-13 20:53:06 +02:00
do {
++haystack;
2008-12-18 22:28:57 +01:00
if (*haystack == ' ' || *haystack == '\0') {
return -1;
}
if (*haystack == '|') {
break;
}
} while (true);
2008-12-18 22:28:57 +01:00
++haystack;
}
}
if (*needlePointer == '\0')
return 1;
2008-12-18 22:28:57 +01:00
return -1;
}
bool Token::simpleMatch(const Token *tok, const char pattern[], size_t pattern_len)
{
if (!tok)
return false; // shortcut
const char *current = pattern;
const char *end = pattern + pattern_len;
const char *next = static_cast<const char*>(std::memchr(pattern, ' ', pattern_len));
if (!next)
next = end;
2011-10-13 20:53:06 +02:00
while (*current) {
2018-04-04 21:51:31 +02:00
const std::size_t length = next - current;
if (!tok || length != tok->mStr.length() || std::strncmp(current, tok->mStr.c_str(), length) != 0)
return false;
current = next;
2011-10-13 20:53:06 +02:00
if (*next) {
2012-12-27 11:51:12 +01:00
next = std::strchr(++current, ' ');
if (!next)
next = end;
}
tok = tok->next();
}
return true;
}
bool Token::firstWordEquals(const char *str, const char *word)
{
2011-10-13 20:53:06 +02:00
for (;;) {
if (*str != *word) {
return (*str == ' ' && *word == 0);
2011-10-13 20:53:06 +02:00
} else if (*str == 0)
break;
++str;
++word;
}
return true;
}
const char *Token::chrInFirstWord(const char *str, char c)
{
2011-10-13 20:53:06 +02:00
for (;;) {
if (*str == ' ' || *str == 0)
return nullptr;
if (*str == c)
return str;
++str;
}
}
bool Token::Match(const Token *tok, const char pattern[], nonneg int varid)
2008-12-18 22:28:57 +01:00
{
if (!(*pattern))
return true;
2008-12-18 22:28:57 +01:00
const char *p = pattern;
while (true) {
2008-12-18 22:28:57 +01:00
// Skip spaces in pattern..
while (*p == ' ')
2009-01-01 23:22:28 +01:00
++p;
2008-12-18 22:28:57 +01:00
// No token => Success!
if (*p == '\0')
break;
2008-12-18 22:28:57 +01:00
2011-10-13 20:53:06 +02:00
if (!tok) {
// If we have no tokens, pattern "!!else" should return true
2012-11-25 15:13:41 +01:00
if (p[0] == '!' && p[1] == '!' && p[2] != '\0') {
2010-09-20 20:15:07 +02:00
while (*p && *p != ' ')
++p;
continue;
}
return false;
}
2008-12-18 22:28:57 +01:00
// [.. => search for a one-character token..
if (p[0] == '[' && chrInFirstWord(p, ']')) {
if (tok->str().length() != 1)
return false;
2011-07-15 19:02:16 +02:00
const char *temp = p+1;
bool chrFound = false;
2019-07-13 16:06:24 +02:00
int count = 0;
2011-10-13 20:53:06 +02:00
while (*temp && *temp != ' ') {
if (*temp == ']') {
++count;
}
else if (*temp == tok->str()[0]) {
chrFound = true;
break;
}
++temp;
}
if (count > 1 && tok->str()[0] == ']')
chrFound = true;
if (!chrFound)
2008-12-18 22:28:57 +01:00
return false;
p = temp;
2008-12-18 22:28:57 +01:00
}
// Parse "not" options. Token can be anything except the given one
else if (p[0] == '!' && p[1] == '!' && p[2] != '\0') {
p += 2;
if (firstWordEquals(p, tok->str().c_str()))
return false;
}
2008-12-18 22:28:57 +01:00
// Parse multi options, such as void|int|char (accept token which is one of these 3)
else {
2018-04-04 21:51:31 +02:00
const int res = multiCompare(tok, p, varid);
2011-10-13 20:53:06 +02:00
if (res == 0) {
2008-12-18 22:28:57 +01:00
// Empty alternative matches, use the same token on next round
2010-09-20 20:15:07 +02:00
while (*p && *p != ' ')
++p;
2008-12-18 22:28:57 +01:00
continue;
}
if (res == -1) {
2008-12-18 22:28:57 +01:00
// No match
return false;
}
}
// using strchr() for the other instances leads to a performance decrease
if (!(p = strchr(p, ' ')))
break;
2008-12-18 22:28:57 +01:00
tok = tok->next();
}
// The end of the pattern has been reached and nothing wrong has been found
return true;
}
nonneg int Token::getStrLength(const Token *tok)
{
2014-02-15 08:05:54 +01:00
assert(tok != nullptr);
2018-06-16 16:40:02 +02:00
assert(tok->mTokType == eString);
2019-07-13 16:06:24 +02:00
int len = 0;
const std::string str(getStringLiteral(tok->str()));
std::string::const_iterator it = str.cbegin();
const std::string::const_iterator end = str.cend();
while (it != end) {
if (*it == '\\') {
++it;
// string ends at '\0'
if (*it == '0')
return len;
}
if (*it == '\0')
return len;
++it;
++len;
}
return len;
}
nonneg int Token::getStrArraySize(const Token *tok)
{
assert(tok != nullptr);
assert(tok->tokType() == eString);
const std::string str(getStringLiteral(tok->str()));
2019-07-13 16:06:24 +02:00
int sizeofstring = 1;
for (int i = 0; i < (int)str.size(); i++) {
if (str[i] == '\\')
++i;
++sizeofstring;
}
return sizeofstring;
}
nonneg int Token::getStrSize(const Token *tok, const Settings *settings)
{
assert(tok != nullptr && tok->tokType() == eString);
nonneg int sizeofType = 1;
if (tok->valueType()) {
ValueType vt(*tok->valueType());
vt.pointer = 0;
sizeofType = ValueFlow::getSizeOf(vt, settings);
}
return getStrArraySize(tok) * sizeofType;
}
void Token::move(Token *srcStart, Token *srcEnd, Token *newLocation)
{
/**[newLocation] -> b -> c -> [srcStart] -> [srcEnd] -> f */
// Fix the gap, which tokens to be moved will leave
srcStart->previous()->next(srcEnd->next());
srcEnd->next()->previous(srcStart->previous());
// Fix the tokens to be moved
srcEnd->next(newLocation->next());
srcStart->previous(newLocation);
// Fix the tokens at newLocation
newLocation->next()->previous(srcEnd);
newLocation->next(srcStart);
// Update _progressValue
for (Token *tok = srcStart; tok != srcEnd->next(); tok = tok->next())
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
tok->mImpl->mProgressValue = newLocation->mImpl->mProgressValue;
}
const Token* Token::nextArgument() const
2011-10-23 11:23:48 +02:00
{
for (const Token* tok = this; tok; tok = tok->next()) {
if (tok->str() == ",")
return tok->next();
else if (tok->link() && Token::Match(tok, "(|{|[|<"))
tok = tok->link();
else if (Token::Match(tok, ")|;"))
return nullptr;
2011-10-23 11:23:48 +02:00
}
return nullptr;
2011-10-23 11:23:48 +02:00
}
const Token* Token::nextArgumentBeforeCreateLinks2() const
{
for (const Token* tok = this; tok; tok = tok->next()) {
if (tok->str() == ",")
return tok->next();
else if (tok->link() && Token::Match(tok, "(|{|["))
tok = tok->link();
else if (tok->str() == "<") {
const Token* temp = tok->findClosingBracket();
if (temp)
tok = temp;
} else if (Token::Match(tok, ")|;"))
return nullptr;
}
return nullptr;
}
const Token* Token::nextTemplateArgument() const
{
for (const Token* tok = this; tok; tok = tok->next()) {
if (tok->str() == ",")
return tok->next();
else if (tok->link() && Token::Match(tok, "(|{|[|<"))
tok = tok->link();
else if (Token::Match(tok, ">|;"))
return nullptr;
}
return nullptr;
}
static bool isOperator(const Token *tok)
{
if (tok->link())
tok = tok->link();
// TODO handle multi token operators
return tok->strAt(-1) == "operator";
}
2013-07-31 10:30:20 +02:00
const Token * Token::findClosingBracket() const
{
2018-06-16 23:03:15 +02:00
if (mStr != "<")
2018-01-01 12:22:04 +01:00
return nullptr;
2020-11-30 19:26:15 +01:00
if (!mPrevious)
return nullptr;
if (!(mPrevious->isName() ||
Token::Match(mPrevious->previous(), "operator %op% <") ||
Token::Match(mPrevious->tokAt(-2), "operator [([] [)]] <")))
return nullptr;
const Token *closing = nullptr;
const bool templateParameter(strAt(-1) == "template");
std::set<std::string> templateParameters;
2013-07-31 10:30:20 +02:00
2020-11-29 12:56:13 +01:00
bool isDecl = true;
for (const Token *prev = previous(); prev; prev = prev->previous()) {
if (prev->str() == "=")
isDecl = false;
if (Token::simpleMatch(prev, "template <"))
isDecl = true;
if (Token::Match(prev, "[;{}]"))
break;
}
2018-01-01 12:22:04 +01:00
unsigned int depth = 0;
for (closing = this; closing != nullptr; closing = closing->next()) {
if (Token::Match(closing, "{|[|(")) {
closing = closing->link();
if (!closing)
return nullptr; // #6803
} else if (Token::Match(closing, "}|]|)|;"))
return nullptr;
// we can make some guesses for template parameters
else if (closing->str() == "<" && closing->previous() &&
(closing->previous()->isName() || isOperator(closing->previous())) &&
(templateParameter ? templateParameters.find(closing->strAt(-1)) == templateParameters.end() : true))
2018-01-01 12:22:04 +01:00
++depth;
else if (closing->str() == ">") {
if (--depth == 0)
return closing;
} else if (closing->str() == ">>" || closing->str() == ">>=") {
2020-11-29 12:56:13 +01:00
if (!isDecl && depth == 1)
continue;
2018-01-01 12:22:04 +01:00
if (depth <= 2)
return closing;
depth -= 2;
}
// save named template parameter
else if (templateParameter && depth == 1 && closing->str() == "," &&
closing->previous()->isName() && !Match(closing->previous(), "class|typename|."))
templateParameters.insert(closing->strAt(-1));
}
2013-07-31 10:30:20 +02:00
return closing;
}
Token * Token::findClosingBracket()
{
// return value of const function
return const_cast<Token*>(const_cast<const Token*>(this)->findClosingBracket());
}
const Token * Token::findOpeningBracket() const
{
if (mStr != ">")
return nullptr;
const Token *opening = nullptr;
unsigned int depth = 0;
for (opening = this; opening != nullptr; opening = opening->previous()) {
if (Token::Match(opening, "}|]|)")) {
opening = opening->link();
if (!opening)
return nullptr;
} else if (Token::Match(opening, "{|{|(|;"))
return nullptr;
else if (opening->str() == ">")
++depth;
else if (opening->str() == "<") {
if (--depth == 0)
return opening;
}
}
return opening;
}
Token * Token::findOpeningBracket()
{
// return value of const function
return const_cast<Token*>(const_cast<const Token*>(this)->findOpeningBracket());
}
2008-12-18 22:28:57 +01:00
//---------------------------------------------------------------------------
const Token *Token::findsimplematch(const Token * const startTok, const char pattern[], size_t pattern_len)
{
for (const Token* tok = startTok; tok; tok = tok->next()) {
if (Token::simpleMatch(tok, pattern, pattern_len))
return tok;
}
return nullptr;
}
const Token *Token::findsimplematch(const Token * const startTok, const char pattern[], size_t pattern_len, const Token * const end)
{
for (const Token* tok = startTok; tok && tok != end; tok = tok->next()) {
if (Token::simpleMatch(tok, pattern, pattern_len))
return tok;
}
return nullptr;
}
const Token *Token::findmatch(const Token * const startTok, const char pattern[], const nonneg int varId)
{
for (const Token* tok = startTok; tok; tok = tok->next()) {
if (Token::Match(tok, pattern, varId))
return tok;
}
return nullptr;
}
const Token *Token::findmatch(const Token * const startTok, const char pattern[], const Token * const end, const nonneg int varId)
2010-07-26 16:46:37 +02:00
{
for (const Token* tok = startTok; tok && tok != end; tok = tok->next()) {
2010-07-26 16:46:37 +02:00
if (Token::Match(tok, pattern, varId))
return tok;
}
return nullptr;
2010-07-26 16:46:37 +02:00
}
2019-07-05 14:00:59 +02:00
void Token::function(const Function *f)
{
mImpl->mFunction = f;
if (f) {
if (f->isLambda())
tokType(eLambda);
else
tokType(eFunction);
} else if (mTokType == eFunction)
tokType(eName);
}
Token* Token::insertToken(const std::string& tokenStr, const std::string& originalNameStr, bool prepend)
{
Token *newToken;
2018-06-16 23:03:15 +02:00
if (mStr.empty())
newToken = this;
else
newToken = new Token(mTokensFrontBack);
newToken->str(tokenStr);
if (!originalNameStr.empty())
newToken->originalName(originalNameStr);
if (newToken != this) {
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
newToken->mImpl->mLineNumber = mImpl->mLineNumber;
newToken->mImpl->mFileIndex = mImpl->mFileIndex;
newToken->mImpl->mProgressValue = mImpl->mProgressValue;
if (prepend) {
if (this->previous()) {
newToken->previous(this->previous());
newToken->previous()->next(newToken);
} else if (mTokensFrontBack) {
mTokensFrontBack->front = newToken;
}
this->previous(newToken);
newToken->next(this);
} else {
if (this->next()) {
newToken->next(this->next());
newToken->next()->previous(newToken);
} else if (mTokensFrontBack) {
mTokensFrontBack->back = newToken;
}
this->next(newToken);
newToken->previous(this);
}
if (mImpl->mScopeInfo) {
// If the brace is immediately closed there is no point opening a new scope for it
if (newToken->str() == "{") {
std::string nextScopeNameAddition;
// This might be the opening of a member function
Token *tok1 = newToken;
while (Token::Match(tok1->previous(), "const|volatile|final|override|&|&&|noexcept"))
tok1 = tok1->previous();
2020-05-20 18:03:03 +02:00
if (tok1->previous() && tok1->strAt(-1) == ")") {
tok1 = tok1->linkAt(-1);
if (Token::Match(tok1->previous(), "throw|noexcept")) {
tok1 = tok1->previous();
while (Token::Match(tok1->previous(), "const|volatile|final|override|&|&&|noexcept"))
tok1 = tok1->previous();
if (tok1->strAt(-1) != ")")
return newToken;
} else if (Token::Match(newToken->tokAt(-2), ":|, %name%")) {
tok1 = tok1->tokAt(-2);
if (tok1->strAt(-1) != ")")
return newToken;
}
if (tok1->strAt(-1) == ">")
tok1 = tok1->previous()->findOpeningBracket();
if (tok1 && Token::Match(tok1->tokAt(-3), "%name% :: %name%")) {
tok1 = tok1->tokAt(-2);
std::string scope = tok1->strAt(-1);
while (Token::Match(tok1->tokAt(-2), ":: %name%")) {
scope = tok1->strAt(-3) + " :: " + scope;
tok1 = tok1->tokAt(-2);
}
nextScopeNameAddition += scope;
}
}
// Or it might be a namespace/class/struct
if (Token::Match(newToken->previous(), "%name%|>")) {
Token* nameTok = newToken->previous();
while (nameTok && !Token::Match(nameTok, "namespace|class|struct|union %name% {|::|:|<")) {
nameTok = nameTok->previous();
}
if (nameTok) {
for (nameTok = nameTok->next(); nameTok && !Token::Match(nameTok, "{|:|<"); nameTok = nameTok->next()) {
nextScopeNameAddition.append(nameTok->str());
nextScopeNameAddition.append(" ");
}
if (!nextScopeNameAddition.empty())
nextScopeNameAddition.pop_back();
}
}
// New scope is opening, record it here
std::shared_ptr<ScopeInfo2> newScopeInfo = std::make_shared<ScopeInfo2>(mImpl->mScopeInfo->name, nullptr, mImpl->mScopeInfo->usingNamespaces);
if (!newScopeInfo->name.empty() && !nextScopeNameAddition.empty()) newScopeInfo->name.append(" :: ");
newScopeInfo->name.append(nextScopeNameAddition);
nextScopeNameAddition = "";
newToken->scopeInfo(newScopeInfo);
} else if (newToken->str() == "}") {
Token* matchingTok = newToken->previous();
int depth = 0;
while (matchingTok && (depth != 0 || !Token::simpleMatch(matchingTok, "{"))) {
if (Token::simpleMatch(matchingTok, "}")) depth++;
if (Token::simpleMatch(matchingTok, "{")) depth--;
matchingTok = matchingTok->previous();
}
if (matchingTok && matchingTok->previous()) {
newToken->mImpl->mScopeInfo = matchingTok->previous()->scopeInfo();
}
} else {
if (prepend && newToken->previous()) {
newToken->mImpl->mScopeInfo = newToken->previous()->scopeInfo();
} else {
newToken->mImpl->mScopeInfo = mImpl->mScopeInfo;
}
if (newToken->str() == ";") {
const Token* statementStart;
for (statementStart = newToken; statementStart->previous() && !Token::Match(statementStart->previous(), ";|{"); statementStart = statementStart->previous());
if (Token::Match(statementStart, "using namespace %name% ::|;")) {
const Token * tok1 = statementStart->tokAt(2);
std::string nameSpace;
while (tok1 && tok1->str() != ";") {
if (!nameSpace.empty())
nameSpace += " ";
nameSpace += tok1->str();
tok1 = tok1->next();
}
mImpl->mScopeInfo->usingNamespaces.insert(nameSpace);
}
}
}
}
}
return newToken;
2008-12-18 22:28:57 +01:00
}
void Token::eraseTokens(Token *begin, const Token *end)
2008-12-18 22:28:57 +01:00
{
if (!begin || begin == end)
2008-12-18 22:28:57 +01:00
return;
2011-10-13 20:53:06 +02:00
while (begin->next() && begin->next() != end) {
2008-12-18 22:28:57 +01:00
begin->deleteNext();
}
}
void Token::createMutualLinks(Token *begin, Token *end)
{
2014-02-15 08:05:54 +01:00
assert(begin != nullptr);
assert(end != nullptr);
assert(begin != end);
begin->link(end);
end->link(begin);
}
void Token::printOut(const char *title) const
2008-12-18 22:28:57 +01:00
{
if (title && title[0])
std::cout << "\n### " << title << " ###\n";
std::cout << stringifyList(stringifyOptions::forPrintOut(), nullptr, nullptr) << std::endl;
}
void Token::printOut(const char *title, const std::vector<std::string> &fileNames) const
{
if (title && title[0])
std::cout << "\n### " << title << " ###\n";
std::cout << stringifyList(stringifyOptions::forPrintOut(), &fileNames, nullptr) << std::endl;
2008-12-18 22:28:57 +01:00
}
// cppcheck-suppress unusedFunction - used for debugging
2020-05-23 11:30:54 +02:00
void Token::printLines(int lines) const
{
const Token *end = this;
while (end && end->linenr() < lines + linenr())
end = end->next();
2020-08-20 18:21:29 +02:00
std::cout << stringifyList(stringifyOptions::forDebugExprId(), nullptr, end) << std::endl;
2020-05-23 11:30:54 +02:00
}
std::string Token::stringify(const stringifyOptions& options) const
{
std::string ret;
2020-08-20 18:21:29 +02:00
if (options.attributes) {
if (isUnsigned())
ret += "unsigned ";
else if (isSigned())
ret += "signed ";
if (isComplex())
ret += "_Complex ";
if (isLong()) {
if (!(mTokType == eString || mTokType == eChar))
ret += "long ";
}
}
2020-08-20 18:21:29 +02:00
if (options.macro && isExpandedMacro())
ret += '$';
2018-06-16 23:03:15 +02:00
if (isName() && mStr.find(' ') != std::string::npos) {
for (const char i : mStr) {
if (i != ' ')
ret += i;
}
2018-06-16 23:03:15 +02:00
} else if (mStr[0] != '\"' || mStr.find('\0') == std::string::npos)
ret += mStr;
else {
for (const char i : mStr) {
if (i == '\0')
ret += "\\0";
else
ret += i;
}
}
if (options.varid && mImpl->mVarId != 0) {
ret += '@';
ret += (options.idtype ? "var" : "");
ret += std::to_string(mImpl->mVarId);
} else if (options.exprid && mImpl->mExprId != 0) {
ret += '@';
ret += (options.idtype ? "expr" : "");
ret += std::to_string(mImpl->mExprId);
}
return ret;
}
std::string Token::stringify(bool varid, bool attributes, bool macro) const
2020-08-20 18:21:29 +02:00
{
stringifyOptions options;
options.varid = varid;
options.attributes = attributes;
options.macro = macro;
return stringify(options);
2020-08-20 18:21:29 +02:00
}
std::string Token::stringifyList(const stringifyOptions& options, const std::vector<std::string>* fileNames, const Token* end) const
{
if (this == end)
return "";
std::string ret;
2020-08-20 18:21:29 +02:00
unsigned int lineNumber = mImpl->mLineNumber - (options.linenumbers ? 1U : 0U);
unsigned int fileIndex = options.files ? ~0U : mImpl->mFileIndex;
std::map<int, unsigned int> lineNumbers;
for (const Token *tok = this; tok != end; tok = tok->next()) {
assert(tok && "end precedes token");
if (!tok)
return ret;
bool fileChange = false;
2019-07-17 16:28:47 +02:00
if (tok->mImpl->mFileIndex != fileIndex) {
if (fileIndex != ~0U) {
lineNumbers[fileIndex] = tok->mImpl->mFileIndex;
}
2019-07-17 16:28:47 +02:00
fileIndex = tok->mImpl->mFileIndex;
2020-08-20 18:21:29 +02:00
if (options.files) {
ret += "\n\n##file ";
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (fileNames && fileNames->size() > tok->mImpl->mFileIndex)
ret += fileNames->at(tok->mImpl->mFileIndex);
else
ret += std::to_string(fileIndex);
ret += '\n';
}
2019-07-17 16:28:47 +02:00
lineNumber = lineNumbers[fileIndex];
fileChange = true;
}
2020-08-20 18:21:29 +02:00
if (options.linebreaks && (lineNumber != tok->linenr() || fileChange)) {
2019-07-17 16:28:47 +02:00
if (lineNumber+4 < tok->linenr() && fileIndex == tok->mImpl->mFileIndex) {
ret += '\n';
ret += std::to_string(lineNumber+1);
ret += ":\n|\n";
ret += std::to_string(tok->linenr()-1);
ret += ":\n";
ret += std::to_string(tok->linenr());
ret += ": ";
2020-08-20 18:21:29 +02:00
} else if (this == tok && options.linenumbers) {
ret += std::to_string(tok->linenr());
ret += ": ";
} else if (lineNumber > tok->linenr()) {
lineNumber = tok->linenr();
ret += '\n';
if (options.linenumbers) {
ret += std::to_string(lineNumber);
ret += ':';
ret += ' ';
}
} else {
while (lineNumber < tok->linenr()) {
++lineNumber;
ret += '\n';
2020-08-20 18:21:29 +02:00
if (options.linenumbers) {
ret += std::to_string(lineNumber);
ret += ':';
if (lineNumber == tok->linenr())
ret += ' ';
}
}
}
2010-04-09 21:40:37 +02:00
lineNumber = tok->linenr();
}
ret += tok->stringify(options); // print token
if (tok->next() != end && (!options.linebreaks || (tok->next()->linenr() == tok->linenr() && tok->next()->fileIndex() == tok->fileIndex())))
ret += ' ';
}
2020-08-20 18:21:29 +02:00
if (options.linebreaks && (options.files || options.linenumbers))
ret += '\n';
return ret;
}
2020-08-20 18:21:29 +02:00
std::string Token::stringifyList(bool varid, bool attributes, bool linenumbers, bool linebreaks, bool files, const std::vector<std::string>* fileNames, const Token* end) const
{
stringifyOptions options;
options.varid = varid;
options.attributes = attributes;
options.macro = attributes;
options.linenumbers = linenumbers;
options.linebreaks = linebreaks;
options.files = files;
return stringifyList(options, fileNames, end);
}
std::string Token::stringifyList(const Token* end, bool attributes) const
{
return stringifyList(false, attributes, false, false, false, nullptr, end);
}
std::string Token::stringifyList(bool varid) const
{
return stringifyList(varid, false, true, true, true, nullptr, nullptr);
}
void Token::astParent(Token* tok)
{
const Token* tok2 = tok;
while (tok2) {
if (this == tok2)
throw InternalError(this, "Internal error. AST cyclic dependency.");
tok2 = tok2->astParent();
}
// Clear children to avoid nodes referenced twice
if (this->astParent()) {
Token* parent = this->astParent();
if (parent->astOperand1() == this)
parent->mImpl->mAstOperand1 = nullptr;
if (parent->astOperand2() == this)
parent->mImpl->mAstOperand2 = nullptr;
}
mImpl->mAstParent = tok;
}
void Token::astOperand1(Token *tok)
{
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (mImpl->mAstOperand1)
mImpl->mAstOperand1->astParent(nullptr);
// goto parent operator
if (tok) {
tok = tok->astTop();
tok->astParent(this);
}
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
mImpl->mAstOperand1 = tok;
}
void Token::astOperand2(Token *tok)
{
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (mImpl->mAstOperand2)
mImpl->mAstOperand2->astParent(nullptr);
// goto parent operator
if (tok) {
tok = tok->astTop();
tok->astParent(this);
}
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
mImpl->mAstOperand2 = tok;
}
static const Token* goToLeftParenthesis(const Token* start, const Token* end)
{
// move start to lpar in such expression: '(*it).x'
int par = 0;
for (const Token *tok = start; tok && tok != end; tok = tok->next()) {
if (tok->str() == "(")
++par;
else if (tok->str() == ")") {
if (par == 0)
start = tok->link();
else
--par;
}
}
return start;
}
static const Token* goToRightParenthesis(const Token* start, const Token* end)
{
// move end to rpar in such expression: '2>(x+1)'
int par = 0;
for (const Token *tok = end; tok && tok != start; tok = tok->previous()) {
if (tok->str() == ")")
++par;
else if (tok->str() == "(") {
if (par == 0)
end = tok->link();
else
--par;
}
}
return end;
}
std::pair<const Token *, const Token *> Token::findExpressionStartEndTokens() const
{
const Token * const top = this;
// find start node in AST tree
const Token *start = top;
while (start->astOperand1() && precedes(start->astOperand1(), start))
start = start->astOperand1();
// find end node in AST tree
const Token *end = top;
while (end->astOperand1() && (end->astOperand2() || end->isUnaryPreOp())) {
// lambda..
if (end->str() == "[") {
const Token *lambdaEnd = findLambdaEndToken(end);
if (lambdaEnd) {
end = lambdaEnd;
break;
}
}
2020-03-08 16:45:51 +01:00
if (Token::Match(end,"(|[|{") &&
!(Token::Match(end, "( %type%") && !end->astOperand2())) {
end = end->link();
break;
}
end = end->astOperand2() ? end->astOperand2() : end->astOperand1();
}
// skip parentheses
start = goToLeftParenthesis(start, end);
end = goToRightParenthesis(start, end);
if (Token::simpleMatch(end, "{"))
end = end->link();
return std::pair<const Token *, const Token *>(start,end);
}
bool Token::isCalculation() const
{
if (!Token::Match(this, "%cop%|++|--"))
return false;
if (Token::Match(this, "*|&")) {
// dereference or address-of?
if (!this->astOperand2())
return false;
if (this->astOperand2()->str() == "[")
return false;
// type specification?
std::stack<const Token *> operands;
operands.push(this);
while (!operands.empty()) {
const Token *op = operands.top();
operands.pop();
if (op->isNumber() || op->varId() > 0)
return true;
if (op->astOperand1())
operands.push(op->astOperand1());
if (op->astOperand2())
operands.push(op->astOperand2());
else if (Token::Match(op, "*|&"))
return false;
}
// type specification => return false
return false;
}
return true;
}
bool Token::isUnaryPreOp() const
{
if (!astOperand1() || astOperand2())
return false;
if (this->tokType() != Token::eIncDecOp)
return true;
2018-06-16 16:16:55 +02:00
const Token *tokbefore = mPrevious;
const Token *tokafter = mNext;
2015-10-26 13:29:47 +01:00
for (int distance = 1; distance < 10 && tokbefore; distance++) {
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (tokbefore == mImpl->mAstOperand1)
return false;
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (tokafter == mImpl->mAstOperand1)
return true;
2018-06-16 16:16:55 +02:00
tokbefore = tokbefore->mPrevious;
tokafter = tokafter->mPrevious;
}
return false; // <- guess
}
static std::string stringFromTokenRange(const Token* start, const Token* end)
{
std::string ret;
if (end)
2017-07-21 09:17:25 +02:00
end = end->next();
for (const Token *tok = start; tok && tok != end; tok = tok->next()) {
2017-07-21 09:17:25 +02:00
if (tok->isUnsigned())
ret += "unsigned ";
if (tok->isLong() && !tok->isLiteral())
ret += "long ";
if (tok->tokType() == Token::eString) {
for (const unsigned char c: tok->str()) {
if (c == '\n')
ret += "\\n";
else if (c == '\r')
ret += "\\r";
else if (c == '\t')
ret += "\\t";
else if (c >= ' ' && c <= 126)
ret += c;
else {
char str[10];
sprintf(str, "\\x%02x", c);
ret += str;
}
}
} else if (tok->originalName().empty() || tok->isUnsigned() || tok->isLong()) {
ret += tok->str();
} else
ret += tok->originalName();
if (Token::Match(tok, "%name%|%num% %name%|%num%"))
ret += ' ';
}
return ret;
}
std::string Token::expressionString() const
{
const auto tokens = findExpressionStartEndTokens();
return stringFromTokenRange(tokens.first, tokens.second);
}
static void astStringXml(const Token *tok, nonneg int indent, std::ostream &out)
{
const std::string strindent(indent, ' ');
2014-07-14 15:51:45 +02:00
out << strindent << "<token str=\"" << tok->str() << '\"';
if (tok->varId())
2014-07-14 15:51:45 +02:00
out << " varId=\"" << MathLib::toString(tok->varId()) << '\"';
if (tok->variable())
out << " variable=\"" << tok->variable() << '\"';
if (tok->function())
out << " function=\"" << tok->function() << '\"';
if (!tok->values().empty())
out << " values=\"" << &tok->values() << '\"';
2014-07-14 15:51:45 +02:00
if (!tok->astOperand1() && !tok->astOperand2()) {
2014-07-14 15:51:45 +02:00
out << "/>" << std::endl;
}
2014-07-14 15:51:45 +02:00
else {
out << '>' << std::endl;
if (tok->astOperand1())
astStringXml(tok->astOperand1(), indent+2U, out);
if (tok->astOperand2())
astStringXml(tok->astOperand2(), indent+2U, out);
out << strindent << "</token>" << std::endl;
}
}
2020-10-31 18:57:48 +01:00
void Token::printAst(bool verbose, bool xml, const std::vector<std::string> &fileNames, std::ostream &out) const
{
if (!xml)
out << "\n\n##AST" << std::endl;
2015-07-21 11:54:11 +02:00
std::set<const Token *> printed;
for (const Token *tok = this; tok; tok = tok->next()) {
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (!tok->mImpl->mAstParent && tok->mImpl->mAstOperand1) {
if (printed.find(tok) != printed.end())
2015-07-21 11:54:11 +02:00
continue;
printed.insert(tok);
if (xml) {
out << "<ast scope=\"" << tok->scope() << "\" fileIndex=\"" << tok->fileIndex() << "\" linenr=\"" << tok->linenr()
<< "\" column=\"" << tok->column() << "\">" << std::endl;
2015-07-21 11:54:11 +02:00
astStringXml(tok, 2U, out);
2014-07-14 15:51:45 +02:00
out << "</ast>" << std::endl;
} else if (verbose)
2020-10-31 18:57:48 +01:00
out << "[" << fileNames[tok->fileIndex()] << ":" << tok->linenr() << "]" << std::endl << tok->astStringVerbose() << std::endl;
else
2015-07-21 11:54:11 +02:00
out << tok->astString(" ") << std::endl;
2014-01-18 09:58:32 +01:00
if (tok->str() == "(")
tok = tok->link();
}
}
}
2014-01-18 09:58:32 +01:00
static void indent(std::string &str, const nonneg int indent1, const nonneg int indent2)
{
2019-07-13 16:06:24 +02:00
for (int i = 0; i < indent1; ++i)
Optimize astStringVerbose() for large arrays (#1815) Change the astStringVerbose() recursion to extend a string instead of returning one. This has the benefit that for tokens where the recursion runs deep (typically large arrays), the time savings can be substantial (see comments on benchmarks further down). The reason is that previously, for each token, the astString of its operands was constructed, and then appended to this tokens astString. This led to a lot of unnecessary string copying (and with that allocations). Instead, by passing the string by reference, the number of temporary strings is greatly reduced. Another way of seeing it is that previously, the string was constructed from end to beginning, but now it is constructed from the beginning to end. There was no notable speedup by preallocating the entire string using string::reserve() (at least not on Linux). To benchmark, the changes and master were tested on Linux using the commands: make time cppcheck --debug --verbose $file >/dev/null i.e., the cppcheck binary was compiled with the settings in the Makefile. Printing the output to screen or file will of course take longer time. In Trac ticket #8355 which triggered this change, an example file from the Wine repository was attached. Running the above cppcheck on master took 24 minutes and with the changes in this commmit, took 22 seconds. Another test made was on lib/tokenlist.cpp in the cppcheck repo, which is more "normal" file. On that file there was no measurable time difference. A synthetic benchmark was generated to illustrate the effects on dumping the ast for arrays of different sizes. The generate code looked as follows: const int array[] = {...}; with different number of elements. The results are as follows (times are in seconds): N master optimized 10 0.1 0.1 100 0.1 0.1 1000 2.8 0.7 2000 19 1.8 3000 53 3.8 5000 350 10 10000 3215 38 As we can see, for small arrays, there is no time difference, but for large arrays the time savings are substantial.
2019-04-30 13:35:48 +02:00
str += ' ';
2019-07-13 16:06:24 +02:00
for (int i = indent1; i < indent2; i += 2)
Optimize astStringVerbose() for large arrays (#1815) Change the astStringVerbose() recursion to extend a string instead of returning one. This has the benefit that for tokens where the recursion runs deep (typically large arrays), the time savings can be substantial (see comments on benchmarks further down). The reason is that previously, for each token, the astString of its operands was constructed, and then appended to this tokens astString. This led to a lot of unnecessary string copying (and with that allocations). Instead, by passing the string by reference, the number of temporary strings is greatly reduced. Another way of seeing it is that previously, the string was constructed from end to beginning, but now it is constructed from the beginning to end. There was no notable speedup by preallocating the entire string using string::reserve() (at least not on Linux). To benchmark, the changes and master were tested on Linux using the commands: make time cppcheck --debug --verbose $file >/dev/null i.e., the cppcheck binary was compiled with the settings in the Makefile. Printing the output to screen or file will of course take longer time. In Trac ticket #8355 which triggered this change, an example file from the Wine repository was attached. Running the above cppcheck on master took 24 minutes and with the changes in this commmit, took 22 seconds. Another test made was on lib/tokenlist.cpp in the cppcheck repo, which is more "normal" file. On that file there was no measurable time difference. A synthetic benchmark was generated to illustrate the effects on dumping the ast for arrays of different sizes. The generate code looked as follows: const int array[] = {...}; with different number of elements. The results are as follows (times are in seconds): N master optimized 10 0.1 0.1 100 0.1 0.1 1000 2.8 0.7 2000 19 1.8 3000 53 3.8 5000 350 10 10000 3215 38 As we can see, for small arrays, there is no time difference, but for large arrays the time savings are substantial.
2019-04-30 13:35:48 +02:00
str += "| ";
}
void Token::astStringVerboseRecursive(std::string& ret, const nonneg int indent1, const nonneg int indent2) const
{
if (isExpandedMacro())
2016-01-01 13:54:07 +01:00
ret += '$';
2018-06-16 23:03:15 +02:00
ret += mStr;
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (mImpl->mValueType)
ret += " \'" + mImpl->mValueType->str() + '\'';
if (function()) {
std::ostringstream ostr;
ostr << std::hex << function();
ret += " f:" + ostr.str();
}
2016-01-01 13:54:07 +01:00
ret += '\n';
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (mImpl->mAstOperand1) {
2019-07-13 16:06:24 +02:00
int i1 = indent1, i2 = indent2 + 2;
Optimize astStringVerbose() for large arrays (#1815) Change the astStringVerbose() recursion to extend a string instead of returning one. This has the benefit that for tokens where the recursion runs deep (typically large arrays), the time savings can be substantial (see comments on benchmarks further down). The reason is that previously, for each token, the astString of its operands was constructed, and then appended to this tokens astString. This led to a lot of unnecessary string copying (and with that allocations). Instead, by passing the string by reference, the number of temporary strings is greatly reduced. Another way of seeing it is that previously, the string was constructed from end to beginning, but now it is constructed from the beginning to end. There was no notable speedup by preallocating the entire string using string::reserve() (at least not on Linux). To benchmark, the changes and master were tested on Linux using the commands: make time cppcheck --debug --verbose $file >/dev/null i.e., the cppcheck binary was compiled with the settings in the Makefile. Printing the output to screen or file will of course take longer time. In Trac ticket #8355 which triggered this change, an example file from the Wine repository was attached. Running the above cppcheck on master took 24 minutes and with the changes in this commmit, took 22 seconds. Another test made was on lib/tokenlist.cpp in the cppcheck repo, which is more "normal" file. On that file there was no measurable time difference. A synthetic benchmark was generated to illustrate the effects on dumping the ast for arrays of different sizes. The generate code looked as follows: const int array[] = {...}; with different number of elements. The results are as follows (times are in seconds): N master optimized 10 0.1 0.1 100 0.1 0.1 1000 2.8 0.7 2000 19 1.8 3000 53 3.8 5000 350 10 10000 3215 38 As we can see, for small arrays, there is no time difference, but for large arrays the time savings are substantial.
2019-04-30 13:35:48 +02:00
if (indent1 == indent2 && !mImpl->mAstOperand2)
i1 += 2;
Optimize astStringVerbose() for large arrays (#1815) Change the astStringVerbose() recursion to extend a string instead of returning one. This has the benefit that for tokens where the recursion runs deep (typically large arrays), the time savings can be substantial (see comments on benchmarks further down). The reason is that previously, for each token, the astString of its operands was constructed, and then appended to this tokens astString. This led to a lot of unnecessary string copying (and with that allocations). Instead, by passing the string by reference, the number of temporary strings is greatly reduced. Another way of seeing it is that previously, the string was constructed from end to beginning, but now it is constructed from the beginning to end. There was no notable speedup by preallocating the entire string using string::reserve() (at least not on Linux). To benchmark, the changes and master were tested on Linux using the commands: make time cppcheck --debug --verbose $file >/dev/null i.e., the cppcheck binary was compiled with the settings in the Makefile. Printing the output to screen or file will of course take longer time. In Trac ticket #8355 which triggered this change, an example file from the Wine repository was attached. Running the above cppcheck on master took 24 minutes and with the changes in this commmit, took 22 seconds. Another test made was on lib/tokenlist.cpp in the cppcheck repo, which is more "normal" file. On that file there was no measurable time difference. A synthetic benchmark was generated to illustrate the effects on dumping the ast for arrays of different sizes. The generate code looked as follows: const int array[] = {...}; with different number of elements. The results are as follows (times are in seconds): N master optimized 10 0.1 0.1 100 0.1 0.1 1000 2.8 0.7 2000 19 1.8 3000 53 3.8 5000 350 10 10000 3215 38 As we can see, for small arrays, there is no time difference, but for large arrays the time savings are substantial.
2019-04-30 13:35:48 +02:00
indent(ret, indent1, indent2);
ret += mImpl->mAstOperand2 ? "|-" : "`-";
mImpl->mAstOperand1->astStringVerboseRecursive(ret, i1, i2);
}
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (mImpl->mAstOperand2) {
2019-07-13 16:06:24 +02:00
int i1 = indent1, i2 = indent2 + 2;
Optimize astStringVerbose() for large arrays (#1815) Change the astStringVerbose() recursion to extend a string instead of returning one. This has the benefit that for tokens where the recursion runs deep (typically large arrays), the time savings can be substantial (see comments on benchmarks further down). The reason is that previously, for each token, the astString of its operands was constructed, and then appended to this tokens astString. This led to a lot of unnecessary string copying (and with that allocations). Instead, by passing the string by reference, the number of temporary strings is greatly reduced. Another way of seeing it is that previously, the string was constructed from end to beginning, but now it is constructed from the beginning to end. There was no notable speedup by preallocating the entire string using string::reserve() (at least not on Linux). To benchmark, the changes and master were tested on Linux using the commands: make time cppcheck --debug --verbose $file >/dev/null i.e., the cppcheck binary was compiled with the settings in the Makefile. Printing the output to screen or file will of course take longer time. In Trac ticket #8355 which triggered this change, an example file from the Wine repository was attached. Running the above cppcheck on master took 24 minutes and with the changes in this commmit, took 22 seconds. Another test made was on lib/tokenlist.cpp in the cppcheck repo, which is more "normal" file. On that file there was no measurable time difference. A synthetic benchmark was generated to illustrate the effects on dumping the ast for arrays of different sizes. The generate code looked as follows: const int array[] = {...}; with different number of elements. The results are as follows (times are in seconds): N master optimized 10 0.1 0.1 100 0.1 0.1 1000 2.8 0.7 2000 19 1.8 3000 53 3.8 5000 350 10 10000 3215 38 As we can see, for small arrays, there is no time difference, but for large arrays the time savings are substantial.
2019-04-30 13:35:48 +02:00
if (indent1 == indent2)
i1 += 2;
Optimize astStringVerbose() for large arrays (#1815) Change the astStringVerbose() recursion to extend a string instead of returning one. This has the benefit that for tokens where the recursion runs deep (typically large arrays), the time savings can be substantial (see comments on benchmarks further down). The reason is that previously, for each token, the astString of its operands was constructed, and then appended to this tokens astString. This led to a lot of unnecessary string copying (and with that allocations). Instead, by passing the string by reference, the number of temporary strings is greatly reduced. Another way of seeing it is that previously, the string was constructed from end to beginning, but now it is constructed from the beginning to end. There was no notable speedup by preallocating the entire string using string::reserve() (at least not on Linux). To benchmark, the changes and master were tested on Linux using the commands: make time cppcheck --debug --verbose $file >/dev/null i.e., the cppcheck binary was compiled with the settings in the Makefile. Printing the output to screen or file will of course take longer time. In Trac ticket #8355 which triggered this change, an example file from the Wine repository was attached. Running the above cppcheck on master took 24 minutes and with the changes in this commmit, took 22 seconds. Another test made was on lib/tokenlist.cpp in the cppcheck repo, which is more "normal" file. On that file there was no measurable time difference. A synthetic benchmark was generated to illustrate the effects on dumping the ast for arrays of different sizes. The generate code looked as follows: const int array[] = {...}; with different number of elements. The results are as follows (times are in seconds): N master optimized 10 0.1 0.1 100 0.1 0.1 1000 2.8 0.7 2000 19 1.8 3000 53 3.8 5000 350 10 10000 3215 38 As we can see, for small arrays, there is no time difference, but for large arrays the time savings are substantial.
2019-04-30 13:35:48 +02:00
indent(ret, indent1, indent2);
ret += "`-";
mImpl->mAstOperand2->astStringVerboseRecursive(ret, i1, i2);
}
Optimize astStringVerbose() for large arrays (#1815) Change the astStringVerbose() recursion to extend a string instead of returning one. This has the benefit that for tokens where the recursion runs deep (typically large arrays), the time savings can be substantial (see comments on benchmarks further down). The reason is that previously, for each token, the astString of its operands was constructed, and then appended to this tokens astString. This led to a lot of unnecessary string copying (and with that allocations). Instead, by passing the string by reference, the number of temporary strings is greatly reduced. Another way of seeing it is that previously, the string was constructed from end to beginning, but now it is constructed from the beginning to end. There was no notable speedup by preallocating the entire string using string::reserve() (at least not on Linux). To benchmark, the changes and master were tested on Linux using the commands: make time cppcheck --debug --verbose $file >/dev/null i.e., the cppcheck binary was compiled with the settings in the Makefile. Printing the output to screen or file will of course take longer time. In Trac ticket #8355 which triggered this change, an example file from the Wine repository was attached. Running the above cppcheck on master took 24 minutes and with the changes in this commmit, took 22 seconds. Another test made was on lib/tokenlist.cpp in the cppcheck repo, which is more "normal" file. On that file there was no measurable time difference. A synthetic benchmark was generated to illustrate the effects on dumping the ast for arrays of different sizes. The generate code looked as follows: const int array[] = {...}; with different number of elements. The results are as follows (times are in seconds): N master optimized 10 0.1 0.1 100 0.1 0.1 1000 2.8 0.7 2000 19 1.8 3000 53 3.8 5000 350 10 10000 3215 38 As we can see, for small arrays, there is no time difference, but for large arrays the time savings are substantial.
2019-04-30 13:35:48 +02:00
}
std::string Token::astStringVerbose() const
{
std::string ret;
astStringVerboseRecursive(ret);
return ret;
}
2020-05-18 19:53:35 +02:00
std::string Token::astStringZ3() const
{
if (!astOperand1())
return str();
if (!astOperand2())
return "(" + str() + " " + astOperand1()->astStringZ3() + ")";
return "(" + str() + " " + astOperand1()->astStringZ3() + " " + astOperand2()->astStringZ3() + ")";
}
2014-07-14 15:51:45 +02:00
void Token::printValueFlow(bool xml, std::ostream &out) const
2014-01-18 09:58:32 +01:00
{
2019-07-13 16:06:24 +02:00
int line = 0;
2014-07-14 15:51:45 +02:00
if (xml)
out << " <valueflow>" << std::endl;
else
out << "\n\n##Value flow" << std::endl;
2014-01-18 09:58:32 +01:00
for (const Token *tok = this; tok; tok = tok->next()) {
const auto* const values = tok->mImpl->mValues;
if (!values)
2014-01-18 09:58:32 +01:00
continue;
if (values->empty()) // Values might be removed by removeContradictions
continue;
2014-07-14 15:51:45 +02:00
if (xml)
out << " <values id=\"" << values << "\">" << std::endl;
2014-07-14 15:51:45 +02:00
else if (line != tok->linenr())
out << "Line " << tok->linenr() << std::endl;
2014-01-18 09:58:32 +01:00
line = tok->linenr();
if (!xml) {
ValueFlow::Value::ValueKind valueKind = values->front().valueKind;
const bool same = std::all_of(values->begin(), values->end(), [&](const ValueFlow::Value& value) {
return value.valueKind == valueKind;
});
out << " " << tok->str() << " ";
if (same) {
switch (valueKind) {
case ValueFlow::Value::ValueKind::Impossible:
case ValueFlow::Value::ValueKind::Known:
out << "always ";
break;
case ValueFlow::Value::ValueKind::Inconclusive:
2021-01-28 22:19:37 +01:00
out << "inconclusive ";
break;
case ValueFlow::Value::ValueKind::Possible:
out << "possible ";
break;
}
}
if (values->size() > 1U)
out << '{';
}
for (const ValueFlow::Value& value : *values) {
2014-07-14 15:51:45 +02:00
if (xml) {
out << " <value ";
switch (value.valueType) {
case ValueFlow::Value::ValueType::INT:
if (tok->valueType() && tok->valueType()->sign == ValueType::UNSIGNED)
out << "intvalue=\"" << (MathLib::biguint)value.intvalue << '\"';
else
out << "intvalue=\"" << value.intvalue << '\"';
2016-11-13 22:59:56 +01:00
break;
case ValueFlow::Value::ValueType::TOK:
out << "tokvalue=\"" << value.tokvalue << '\"';
2016-11-13 22:59:56 +01:00
break;
case ValueFlow::Value::ValueType::FLOAT:
out << "floatvalue=\"" << value.floatValue << '\"';
2016-11-13 22:59:56 +01:00
break;
case ValueFlow::Value::ValueType::MOVED:
out << "movedvalue=\"" << ValueFlow::Value::toString(value.moveKind) << '\"';
break;
case ValueFlow::Value::ValueType::UNINIT:
out << "uninit=\"1\"";
break;
case ValueFlow::Value::ValueType::BUFFER_SIZE:
out << "buffer-size=\"" << value.intvalue << "\"";
break;
case ValueFlow::Value::ValueType::CONTAINER_SIZE:
out << "container-size=\"" << value.intvalue << '\"';
break;
case ValueFlow::Value::ValueType::ITERATOR_START:
out << "iterator-start=\"" << value.intvalue << '\"';
break;
case ValueFlow::Value::ValueType::ITERATOR_END:
out << "iterator-end=\"" << value.intvalue << '\"';
break;
case ValueFlow::Value::ValueType::LIFETIME:
2018-11-14 06:14:04 +01:00
out << "lifetime=\"" << value.tokvalue << '\"';
out << " lifetime-scope=\"" << ValueFlow::Value::toString(value.lifetimeScope) << "\"";
out << " lifetime-kind=\"" << ValueFlow::Value::toString(value.lifetimeKind) << "\"";
2018-11-14 06:14:04 +01:00
break;
case ValueFlow::Value::ValueType::SYMBOLIC:
out << "symbolic=\"" << value.tokvalue << '\"';
out << " symbolic-delta=\"" << value.intvalue << '\"';
break;
2016-11-13 22:59:56 +01:00
}
out << " bound=\"" << ValueFlow::Value::toString(value.bound) << "\"";
if (value.condition)
out << " condition-line=\"" << value.condition->linenr() << '\"';
if (value.isKnown())
out << " known=\"true\"";
else if (value.isPossible())
out << " possible=\"true\"";
else if (value.isImpossible())
out << " impossible=\"true\"";
else if (value.isInconclusive())
out << " inconclusive=\"true\"";
out << " path=\"" << value.path << "\"";
2014-07-14 15:51:45 +02:00
out << "/>" << std::endl;
}
else {
if (&value != &values->front())
out << ",";
out << value.toString();
2014-07-14 15:51:45 +02:00
}
2014-01-18 09:58:32 +01:00
}
2014-07-14 15:51:45 +02:00
if (xml)
out << " </values>" << std::endl;
else if (values->size() > 1U)
out << '}' << std::endl;
2014-07-14 15:51:45 +02:00
else
out << std::endl;
2014-01-18 09:58:32 +01:00
}
2014-07-14 15:51:45 +02:00
if (xml)
out << " </valueflow>" << std::endl;
2014-01-18 09:58:32 +01:00
}
const ValueFlow::Value * Token::getValueLE(const MathLib::bigint val, const Settings *settings) const
{
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (!mImpl->mValues)
return nullptr;
return ValueFlow::findValue(*mImpl->mValues, settings, [&](const ValueFlow::Value& v) {
return !v.isImpossible() && v.isIntValue() && v.intvalue <= val;
});
}
const ValueFlow::Value * Token::getValueGE(const MathLib::bigint val, const Settings *settings) const
{
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (!mImpl->mValues)
return nullptr;
return ValueFlow::findValue(*mImpl->mValues, settings, [&](const ValueFlow::Value& v) {
return !v.isImpossible() && v.isIntValue() && v.intvalue >= val;
});
}
const ValueFlow::Value * Token::getInvalidValue(const Token *ftok, nonneg int argnr, const Settings *settings) const
{
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (!mImpl->mValues || !settings)
return nullptr;
const ValueFlow::Value *ret = nullptr;
std::list<ValueFlow::Value>::const_iterator it;
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
for (it = mImpl->mValues->begin(); it != mImpl->mValues->end(); ++it) {
if (it->isImpossible())
continue;
2018-07-15 23:05:48 +02:00
if ((it->isIntValue() && !settings->library.isIntArgValid(ftok, argnr, it->intvalue)) ||
(it->isFloatValue() && !settings->library.isFloatArgValid(ftok, argnr, it->floatValue))) {
if (!ret || ret->isInconclusive() || (ret->condition && !it->isInconclusive()))
ret = &(*it);
if (!ret->isInconclusive() && !ret->condition)
break;
}
}
if (ret) {
if (ret->isInconclusive() && !settings->certainty.isEnabled(Certainty::inconclusive))
return nullptr;
if (ret->condition && !settings->severity.isEnabled(Severity::warning))
return nullptr;
}
return ret;
}
const Token *Token::getValueTokenMinStrSize(const Settings *settings, MathLib::bigint* path) const
{
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (!mImpl->mValues)
return nullptr;
const Token *ret = nullptr;
2019-07-13 16:06:24 +02:00
int minsize = INT_MAX;
std::list<ValueFlow::Value>::const_iterator it;
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
for (it = mImpl->mValues->begin(); it != mImpl->mValues->end(); ++it) {
2016-11-13 22:33:39 +01:00
if (it->isTokValue() && it->tokvalue && it->tokvalue->tokType() == Token::eString) {
const int size = getStrSize(it->tokvalue, settings);
if (!ret || size < minsize) {
minsize = size;
ret = it->tokvalue;
if (path)
*path = it->path;
}
}
}
return ret;
}
const Token *Token::getValueTokenMaxStrLength() const
{
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (!mImpl->mValues)
return nullptr;
const Token *ret = nullptr;
2019-07-13 16:06:24 +02:00
int maxlength = 0;
std::list<ValueFlow::Value>::const_iterator it;
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
for (it = mImpl->mValues->begin(); it != mImpl->mValues->end(); ++it) {
2016-11-13 22:33:39 +01:00
if (it->isTokValue() && it->tokvalue && it->tokvalue->tokType() == Token::eString) {
2019-07-13 16:06:24 +02:00
const int length = getStrLength(it->tokvalue);
if (!ret || length > maxlength) {
maxlength = length;
ret = it->tokvalue;
}
}
}
return ret;
}
static bool isAdjacent(const ValueFlow::Value& x, const ValueFlow::Value& y)
{
if (x.bound != ValueFlow::Value::Bound::Point && x.bound == y.bound)
return true;
if (x.valueType == ValueFlow::Value::ValueType::FLOAT)
return false;
return std::abs(x.intvalue - y.intvalue) == 1;
}
static bool removePointValue(std::list<ValueFlow::Value>& values, ValueFlow::Value& x)
{
const bool isPoint = x.bound == ValueFlow::Value::Bound::Point;
if (!isPoint)
x.decreaseRange();
else
values.remove(x);
return isPoint;
}
static bool removeContradiction(std::list<ValueFlow::Value>& values)
{
bool result = false;
for (ValueFlow::Value& x : values) {
if (x.isNonValue())
continue;
for (ValueFlow::Value& y : values) {
if (y.isNonValue())
continue;
if (x == y)
continue;
if (x.valueType != y.valueType)
continue;
if (x.isImpossible() == y.isImpossible())
continue;
if (x.isSymbolicValue() && !ValueFlow::Value::sameToken(x.tokvalue, y.tokvalue))
continue;
if (!x.equalValue(y)) {
auto compare = [](const ValueFlow::Value& x, const ValueFlow::Value& y) {
return x.compareValue(y, less{});
};
const ValueFlow::Value& maxValue = std::max(x, y, compare);
const ValueFlow::Value& minValue = std::min(x, y, compare);
// TODO: Adjust non-points instead of removing them
if (maxValue.isImpossible() && maxValue.bound == ValueFlow::Value::Bound::Upper) {
values.remove(minValue);
return true;
}
if (minValue.isImpossible() && minValue.bound == ValueFlow::Value::Bound::Lower) {
values.remove(maxValue);
return true;
}
continue;
}
const bool removex = !x.isImpossible() || y.isKnown();
const bool removey = !y.isImpossible() || x.isKnown();
if (x.bound == y.bound) {
if (removex)
values.remove(x);
if (removey)
values.remove(y);
return true;
} else {
result = removex || removey;
bool bail = false;
if (removex && removePointValue(values, x))
bail = true;
if (removey && removePointValue(values, y))
bail = true;
if (bail)
return true;
}
}
}
return result;
}
using ValueIterator = std::list<ValueFlow::Value>::iterator;
2021-08-07 20:51:18 +02:00
template<class Iterator>
static ValueIterator removeAdjacentValues(std::list<ValueFlow::Value>& values, ValueIterator x, Iterator start, Iterator last)
{
if (!isAdjacent(*x, **start))
return std::next(x);
2021-04-04 18:20:32 +02:00
auto it = std::adjacent_find(start, last, [](ValueIterator x, ValueIterator y) {
return !isAdjacent(*x, *y);
});
if (it == last)
it--;
(*it)->bound = x->bound;
2021-04-04 18:20:32 +02:00
std::for_each(start, it, [&](ValueIterator y) {
values.erase(y);
});
return values.erase(x);
}
static void mergeAdjacent(std::list<ValueFlow::Value>& values)
{
for (auto x = values.begin(); x != values.end();) {
if (x->isNonValue()) {
x++;
continue;
}
if (x->bound == ValueFlow::Value::Bound::Point) {
x++;
continue;
}
std::vector<ValueIterator> adjValues;
for (auto y = values.begin(); y != values.end(); y++) {
if (x == y)
continue;
if (y->isNonValue())
continue;
if (x->valueType != y->valueType)
continue;
if (x->valueKind != y->valueKind)
continue;
if (x->isSymbolicValue() && !ValueFlow::Value::sameToken(x->tokvalue, y->tokvalue))
continue;
if (x->bound != y->bound) {
if (y->bound != ValueFlow::Value::Bound::Point && isAdjacent(*x, *y)) {
adjValues.clear();
break;
}
// No adjacent points for floating points
if (x->valueType == ValueFlow::Value::ValueType::FLOAT)
continue;
if (y->bound != ValueFlow::Value::Bound::Point)
continue;
}
if (x->bound == ValueFlow::Value::Bound::Lower && !y->compareValue(*x, less{}))
continue;
if (x->bound == ValueFlow::Value::Bound::Upper && !x->compareValue(*y, less{}))
continue;
adjValues.push_back(y);
}
if (adjValues.empty()) {
x++;
continue;
}
std::sort(adjValues.begin(), adjValues.end(), [&values](ValueIterator xx, ValueIterator yy) {
(void)values;
assert(xx != values.end() && yy != values.end());
return xx->compareValue(*yy, less{});
});
if (x->bound == ValueFlow::Value::Bound::Lower)
x = removeAdjacentValues(values, x, adjValues.rbegin(), adjValues.rend());
else if (x->bound == ValueFlow::Value::Bound::Upper)
x = removeAdjacentValues(values, x, adjValues.begin(), adjValues.end());
}
}
static void removeOverlaps(std::list<ValueFlow::Value>& values)
{
for (ValueFlow::Value& x : values) {
if (x.isNonValue())
continue;
values.remove_if([&](ValueFlow::Value& y) {
if (y.isNonValue())
return false;
if (&x == &y)
return false;
if (x.valueType != y.valueType)
return false;
if (x.valueKind != y.valueKind)
return false;
2022-09-27 06:48:06 +02:00
// TODO: Remove points covered in a lower or upper bound
// TODO: Remove lower or upper bound already covered by a lower and upper bound
if (!x.equalValue(y))
return false;
if (x.bound != y.bound)
return false;
return true;
});
}
mergeAdjacent(values);
}
// Removing contradictions is an NP-hard problem. Instead we run multiple
// passes to try to catch most contradictions
static void removeContradictions(std::list<ValueFlow::Value>& values)
{
removeOverlaps(values);
for (int i = 0; i < 4; i++) {
if (!removeContradiction(values))
return;
removeOverlaps(values);
}
}
2021-08-22 18:06:54 +02:00
static bool sameValueType(const ValueFlow::Value& x, const ValueFlow::Value& y)
{
if (x.valueType != y.valueType)
return false;
// Symbolic are the same type if they share the same tokvalue
if (x.isSymbolicValue())
return x.tokvalue->exprId() == 0 || x.tokvalue->exprId() == y.tokvalue->exprId();
return true;
}
bool Token::addValue(const ValueFlow::Value &value)
{
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (value.isKnown() && mImpl->mValues) {
// Clear all other values of the same type since value is known
mImpl->mValues->remove_if([&](const ValueFlow::Value& x) {
return sameValueType(x, value);
});
}
// Don't add a value if its already known
if (!value.isKnown() && mImpl->mValues &&
std::any_of(mImpl->mValues->begin(), mImpl->mValues->end(), [&](const ValueFlow::Value& x) {
return x.isKnown() && sameValueType(x, value) && !x.equalValue(value);
}))
return false;
// assert(value.isKnown() || !mImpl->mValues || std::none_of(mImpl->mValues->begin(), mImpl->mValues->end(),
// [&](const ValueFlow::Value& x) {
// return x.isKnown() && sameValueType(x, value);
// }));
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (mImpl->mValues) {
// Don't handle more than 10 values for performance reasons
// TODO: add setting?
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (mImpl->mValues->size() >= 10U)
return false;
// if value already exists, don't add it again
std::list<ValueFlow::Value>::iterator it;
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
for (it = mImpl->mValues->begin(); it != mImpl->mValues->end(); ++it) {
// different types => continue
if (it->valueType != value.valueType)
continue;
if (it->isImpossible() != value.isImpossible())
continue;
// different value => continue
if (!it->equalValue(value))
continue;
if ((value.isTokValue() || value.isLifetimeValue()) && (it->tokvalue != value.tokvalue) && (it->tokvalue->str() != value.tokvalue->str()))
continue;
// same value, but old value is inconclusive so replace it
if (it->isInconclusive() && !value.isInconclusive() && !value.isImpossible()) {
*it = value;
if (it->varId == 0)
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
it->varId = mImpl->mVarId;
break;
}
// Same value already exists, don't add new value
return false;
}
// Add value
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (it == mImpl->mValues->end()) {
ValueFlow::Value v(value);
if (v.varId == 0)
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
v.varId = mImpl->mVarId;
2018-11-14 19:10:52 +01:00
if (v.isKnown() && v.isIntValue())
mImpl->mValues->push_front(std::move(v));
else
mImpl->mValues->push_back(std::move(v));
}
} else {
ValueFlow::Value v(value);
if (v.varId == 0)
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
v.varId = mImpl->mVarId;
mImpl->mValues = new std::list<ValueFlow::Value>(1, v);
}
removeContradictions(*mImpl->mValues);
return true;
}
void Token::assignProgressValues(Token *tok)
{
2019-07-13 16:06:24 +02:00
int total_count = 0;
for (Token *tok2 = tok; tok2; tok2 = tok2->next())
++total_count;
2019-07-13 16:06:24 +02:00
int count = 0;
for (Token *tok2 = tok; tok2; tok2 = tok2->next())
2021-08-07 20:51:18 +02:00
tok2->mImpl->mProgressValue = count++ *100 / total_count;
}
void Token::assignIndexes()
{
2019-07-13 16:06:24 +02:00
int index = (mPrevious ? mPrevious->mImpl->mIndex : 0) + 1;
for (Token *tok = this; tok; tok = tok->next())
tok->mImpl->mIndex = index++;
}
void Token::setValueType(ValueType *vt)
{
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (vt != mImpl->mValueType) {
delete mImpl->mValueType;
mImpl->mValueType = vt;
}
}
void Token::type(const ::Type *t)
{
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
mImpl->mType = t;
if (t) {
2017-10-15 01:27:47 +02:00
tokType(eType);
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
isEnumType(mImpl->mType->isEnumType());
2018-06-16 16:40:02 +02:00
} else if (mTokType == eType)
2017-10-15 01:27:47 +02:00
tokType(eName);
}
const ::Type* Token::typeOf(const Token* tok, const Token** typeTok)
{
if (!tok)
return nullptr;
if (typeTok != nullptr)
*typeTok = tok;
const Token* lhsVarTok{};
if (tok->type()) {
return tok->type();
} else if (tok->variable()) {
return tok->variable()->type();
} else if (tok->function()) {
return tok->function()->retType;
} else if (Token::simpleMatch(tok, "return")) {
const Scope *scope = tok->scope();
if (!scope)
return nullptr;
const Function *function = scope->function;
if (!function)
return nullptr;
return function->retType;
} else if (Token::Match(tok->previous(), "%type%|= (|{")) {
return typeOf(tok->previous(), typeTok);
} else if (Token::simpleMatch(tok, "=") && (lhsVarTok = getLHSVariableToken(tok)) != tok->next()) {
return Token::typeOf(lhsVarTok, typeTok);
} else if (Token::simpleMatch(tok, ".")) {
return Token::typeOf(tok->astOperand2(), typeTok);
} else if (Token::simpleMatch(tok, "[")) {
return Token::typeOf(tok->astOperand1(), typeTok);
} else if (Token::simpleMatch(tok, "{")) {
int argnr;
const Token* ftok = getTokenArgumentFunction(tok, argnr);
if (argnr < 0)
return nullptr;
if (!ftok)
return nullptr;
if (ftok == tok)
return nullptr;
std::vector<const Variable*> vars = getArgumentVars(ftok, argnr);
if (vars.empty())
return nullptr;
if (std::all_of(
vars.cbegin(), vars.cend(), [&](const Variable* var) {
2021-08-07 20:51:18 +02:00
return var->type() == vars.front()->type();
2021-06-03 07:35:50 +02:00
}))
2021-08-07 20:51:18 +02:00
return vars.front()->type();
}
return nullptr;
}
2022-12-07 09:38:36 +01:00
std::pair<const Token*, const Token*> Token::typeDecl(const Token* tok, bool pointedToType)
{
2020-06-18 00:06:06 +02:00
if (!tok)
return {};
else if (tok->type()) {
return {tok, tok->next()};
} else if (tok->variable()) {
const Variable *var = tok->variable();
if (!var->typeStartToken() || !var->typeEndToken())
return {};
2022-12-18 16:54:58 +01:00
std::pair<const Token*, const Token*> result;
2020-06-18 00:06:06 +02:00
if (Token::simpleMatch(var->typeStartToken(), "auto")) {
const Token * tok2 = var->declEndToken();
if (Token::Match(tok2, "; %varid% =", var->declarationId()))
tok2 = tok2->tokAt(2);
if (Token::simpleMatch(tok2, "=") && Token::Match(tok2->astOperand2(), "!!=") && tok != tok2->astOperand2()) {
tok2 = tok2->astOperand2();
std::pair<const Token*, const Token*> r = typeDecl(tok2);
2020-06-18 00:06:06 +02:00
if (r.first)
return r;
2022-12-07 09:38:36 +01:00
if (pointedToType && tok2->astOperand1() && Token::simpleMatch(tok2, "new")) {
if (Token::simpleMatch(tok2->astOperand1(), "("))
return { tok2->next(), tok2->astOperand1() };
const Token* declEnd = nextAfterAstRightmostLeaf(tok2->astOperand1());
if (Token::simpleMatch(declEnd, "<") && declEnd->link())
declEnd = declEnd->link()->next();
return { tok2->next(), declEnd };
}
2022-12-18 16:54:58 +01:00
const Token *typeBeg{}, *typeEnd{};
if (tok2->str() == "::" && Token::simpleMatch(tok2->astOperand2(), "{")) { // empty initlist
typeBeg = previousBeforeAstLeftmostLeaf(tok2);
typeEnd = tok2->astOperand2();
}
else if (tok2->str() == "{") {
typeBeg = previousBeforeAstLeftmostLeaf(tok2);
typeEnd = tok2;
}
if (typeBeg)
result = { typeBeg->next(), typeEnd }; // handle smart pointers/iterators first
2020-06-18 00:06:06 +02:00
}
if (astIsRangeBasedForDecl(var->nameToken()) && astIsContainer(var->nameToken()->astParent()->astOperand2())) { // range-based for
const ValueType* vt = var->nameToken()->astParent()->astOperand2()->valueType();
if (vt && vt->containerTypeToken)
return { vt->containerTypeToken, vt->containerTypeToken->linkAt(-1) };
}
2020-06-18 00:06:06 +02:00
}
if (pointedToType && astIsSmartPointer(var->nameToken())) {
const ValueType* vt = var->valueType();
if (vt && vt->smartPointerTypeToken)
return { vt->smartPointerTypeToken, vt->smartPointerTypeToken->linkAt(-1) };
}
if (pointedToType && astIsIterator(var->nameToken())) {
const ValueType* vt = var->valueType();
if (vt && vt->containerTypeToken)
return { vt->containerTypeToken, vt->containerTypeToken->linkAt(-1) };
}
if (result.first)
return result;
return {var->typeStartToken(), var->typeEndToken()->next()};
} else if (Token::simpleMatch(tok, "return")) {
const Scope* scope = tok->scope();
if (!scope)
return {};
const Function* function = scope->function;
if (!function)
return {};
return { function->retDef, function->returnDefEnd() };
2022-06-09 18:25:17 +02:00
} else if (tok->previous() && tok->previous()->function()) {
const Function *function = tok->previous()->function();
return {function->retDef, function->returnDefEnd()};
} else if (Token::simpleMatch(tok, "=")) {
return Token::typeDecl(tok->astOperand1());
} else if (Token::simpleMatch(tok, ".")) {
return Token::typeDecl(tok->astOperand2());
} else {
const ::Type * t = typeOf(tok);
if (!t || !t->classDef)
return {};
return {t->classDef->next(), t->classDef->tokAt(2)};
}
}
std::string Token::typeStr(const Token* tok)
{
if (tok->valueType()) {
const ValueType * vt = tok->valueType();
std::string ret = vt->str();
if (!ret.empty())
2019-06-02 13:29:20 +02:00
return ret;
}
std::pair<const Token*, const Token*> r = Token::typeDecl(tok);
if (!r.first || !r.second)
return "";
return r.first->stringifyList(r.second, false);
}
void Token::scopeInfo(std::shared_ptr<ScopeInfo2> newScopeInfo)
{
mImpl->mScopeInfo = std::move(newScopeInfo);
}
std::shared_ptr<ScopeInfo2> Token::scopeInfo() const
{
return mImpl->mScopeInfo;
}
bool Token::hasKnownIntValue() const
{
if (!mImpl->mValues)
return false;
return std::any_of(mImpl->mValues->begin(), mImpl->mValues->end(), [](const ValueFlow::Value& value) {
return value.isKnown() && value.isIntValue();
});
}
bool Token::hasKnownValue() const
{
return mImpl->mValues && std::any_of(mImpl->mValues->begin(), mImpl->mValues->end(), std::mem_fn(&ValueFlow::Value::isKnown));
}
bool Token::hasKnownValue(ValueFlow::Value::ValueType t) const
{
return mImpl->mValues &&
2021-08-07 20:51:18 +02:00
std::any_of(mImpl->mValues->begin(), mImpl->mValues->end(), [&](const ValueFlow::Value& value) {
2021-07-26 16:32:00 +02:00
return value.isKnown() && value.valueType == t;
});
}
bool Token::hasKnownSymbolicValue(const Token* tok) const
{
if (tok->exprId() == 0)
return false;
return mImpl->mValues &&
std::any_of(mImpl->mValues->begin(), mImpl->mValues->end(), [&](const ValueFlow::Value& value) {
return value.isKnown() && value.isSymbolicValue() && value.tokvalue &&
value.tokvalue->exprId() == tok->exprId();
});
}
const ValueFlow::Value* Token::getKnownValue(ValueFlow::Value::ValueType t) const
{
if (!mImpl->mValues)
return nullptr;
auto it = std::find_if(mImpl->mValues->begin(), mImpl->mValues->end(), [&](const ValueFlow::Value& value) {
return value.isKnown() && value.valueType == t;
});
return it == mImpl->mValues->end() ? nullptr : &*it;
}
const ValueFlow::Value* Token::getValue(const MathLib::bigint val) const
{
if (!mImpl->mValues)
return nullptr;
const auto it = std::find_if(mImpl->mValues->begin(), mImpl->mValues->end(), [=](const ValueFlow::Value& value) {
return value.isIntValue() && !value.isImpossible() && value.intvalue == val;
});
return it == mImpl->mValues->end() ? nullptr : &*it;
}
const ValueFlow::Value* Token::getMaxValue(bool condition, MathLib::bigint path) const
{
if (!mImpl->mValues)
return nullptr;
const ValueFlow::Value* ret = nullptr;
for (const ValueFlow::Value& value : *mImpl->mValues) {
if (!value.isIntValue())
continue;
if (value.isImpossible())
continue;
if (path > -0 && value.path != 0 && value.path != path)
continue;
if ((!ret || value.intvalue > ret->intvalue) &&
((value.condition != nullptr) == condition))
ret = &value;
}
return ret;
}
const ValueFlow::Value* Token::getMovedValue() const
{
if (!mImpl->mValues)
return nullptr;
const auto it = std::find_if(mImpl->mValues->begin(), mImpl->mValues->end(), [](const ValueFlow::Value& value) {
return value.isMovedValue() && !value.isImpossible() &&
2021-08-07 20:51:18 +02:00
value.moveKind != ValueFlow::Value::MoveKind::NonMovedVariable;
});
return it == mImpl->mValues->end() ? nullptr : &*it;
}
// cppcheck-suppress unusedFunction
const ValueFlow::Value* Token::getContainerSizeValue(const MathLib::bigint val) const
{
if (!mImpl->mValues)
return nullptr;
const auto it = std::find_if(mImpl->mValues->begin(), mImpl->mValues->end(), [=](const ValueFlow::Value& value) {
return value.isContainerSizeValue() && !value.isImpossible() && value.intvalue == val;
});
return it == mImpl->mValues->end() ? nullptr : &*it;
}
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
TokenImpl::~TokenImpl()
{
delete mOriginalName;
delete mValueType;
delete mValues;
2022-07-18 16:20:17 +02:00
if (mTemplateSimplifierPointers) {
for (auto *templateSimplifierPointer : *mTemplateSimplifierPointers) {
templateSimplifierPointer->token(nullptr);
}
2022-07-18 16:20:17 +02:00
}
delete mTemplateSimplifierPointers;
while (mCppcheckAttributes) {
struct CppcheckAttributes *c = mCppcheckAttributes;
mCppcheckAttributes = mCppcheckAttributes->next;
delete c;
}
}
void TokenImpl::setCppcheckAttribute(TokenImpl::CppcheckAttributes::Type type, MathLib::bigint value)
{
struct CppcheckAttributes *attr = mCppcheckAttributes;
while (attr && attr->type != type)
attr = attr->next;
if (attr)
attr->value = value;
else {
attr = new CppcheckAttributes;
attr->type = type;
attr->value = value;
attr->next = mCppcheckAttributes;
mCppcheckAttributes = attr;
}
}
bool TokenImpl::getCppcheckAttribute(TokenImpl::CppcheckAttributes::Type type, MathLib::bigint *value) const
{
struct CppcheckAttributes *attr = mCppcheckAttributes;
while (attr && attr->type != type)
attr = attr->next;
if (attr)
*value = attr->value;
return attr != nullptr;
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
}
Token* findTypeEnd(Token* tok)
{
while (Token::Match(tok, "%name%|.|::|*|&|&&|<|(|template|decltype|sizeof")) {
if (Token::Match(tok, "(|<"))
tok = tok->link();
if (!tok)
return nullptr;
tok = tok->next();
}
return tok;
}
const Token* findTypeEnd(const Token* tok) {
return findTypeEnd(const_cast<Token*>(tok));
}
Token* findLambdaEndScope(Token* tok)
{
if (!Token::simpleMatch(tok, "["))
return nullptr;
tok = tok->link();
if (!Token::Match(tok, "] (|{"))
return nullptr;
tok = tok->linkAt(1);
if (Token::simpleMatch(tok, "}"))
return tok;
if (Token::simpleMatch(tok, ") {"))
return tok->linkAt(1);
if (!Token::simpleMatch(tok, ")"))
return nullptr;
tok = tok->next();
#11134 Fix broken AST with (designated) initializers (#4550) * Make control flow a bit easier, and more similar to previous code Made similar to around line 790 * In a cpp11init, always parse only the corresponding } (#11134) - _always_, because in some cases this was omitted (around line 790) or too strict (around line 860) - _only_, and not following tokens which happen to be } as well (around line 1030) * Fix unit tests: AST was incorrect, now is fixed auto var{ {{},{}}, {} }; Old AST: ``` { |-var `-{ `-, |-, | |-{ | `-{ `-{ ``` New AST: ``` { |-var `-, |-{ | `-, | | |-{ | | `-{ `-{ ``` Compare the same example, but with `X{}` instead of just `{}`: `auto var{ a{b{},c{}}, d{} };` ``` { |-var `-, |-{ | |-a | `-, | | |-{ | | | `-b | | `-{ | | | `-c `-{ `-d ``` This structure is similar to that of the new AST, not the old AST * Fix unit tests: another AST was incorrect, now is fixed Code: `auto var{{1,a::b{2,3}}, {4,a::b{5,6}}};` Old AST: ``` { |-var `-{ `-, |-, | |-1 'signed int' | `-{ | | |-:: | | | |-a | | | `-b | | `-, | | | |-2 'signed int' | | | `-3 'signed int' `-{ `-, |-4 'signed int' `-{ |-:: | |-a | `-b `-, |-5 'signed int' `-6 'signed int' ``` New AST: ``` { |-var `-, |-{ | `-, | | |-1 'signed int' | | `-{ | | | |-:: | | | | |-a | | | | `-b | | | `-, | | | | |-2 'signed int' | | | | `-3 'signed int' `-{ `-, |-4 'signed int' `-{ |-:: | |-a | `-b `-, |-5 'signed int' `-6 'signed int' ``` * Fix unit tests: missing ; after class, resulting in incorrectly being marked as cpp11init Because of the missing `;` after the class declaration, it was marked as a cpp11init block. Which it isn't, and which now throws an exception * Fix cpp11init to let unit tests pass again The following unit tests failed on the newly introduced throws, because the code for these tests incorrectly marked some tokens as cpp11init: TestVarID::varid_cpp11initialization TestTokenizer::checkRefQualifiers * Fix typo * Improve check for void trailing return type Observation: the only function body _not_ containing a semicolon, is a void function: something like auto make_zero(ini& i) -> void { while(--i > 0) {} } Non-void function? Then it must return a value, and thus contain a semicolon, which is checked for a few lines later. * Fix cpp11init with templated trailing return type In the following example, vector was marked as cpp11init due to the mismatch of `%any% {` auto f() -> std::vector<int> { return {}; } I made the assumption that whenever "%any% {" matches, endtok must be set too. If this assumtion doesn't hold (so "%any% {" matches, but endtok == nullptr), then the for-loop would search all the way to the end of stream. Which I guess was not the intention. * Remove comments Co-authored-by: Gerbo Engels <gerbo.engels@ortec-finance.com>
2022-10-19 07:25:15 +02:00
while (Token::Match(tok, "mutable|constexpr|consteval|noexcept|.")) {
if (Token::simpleMatch(tok, "noexcept ("))
tok = tok->linkAt(1);
if (Token::simpleMatch(tok, ".")) {
tok = findTypeEnd(tok);
break;
}
tok = tok->next();
}
if (Token::simpleMatch(tok, "{"))
return tok->link();
return nullptr;
}
const Token* findLambdaEndScope(const Token* tok) {
return findLambdaEndScope(const_cast<Token*>(tok));
}