cppcheck/lib/token.cpp

2038 lines
63 KiB
C++
Raw Normal View History

2008-12-18 22:28:57 +01:00
/*
* Cppcheck - A tool for static C/C++ code analysis
2019-02-09 07:24:06 +01:00
* Copyright (C) 2007-2019 Cppcheck team.
2008-12-18 22:28:57 +01:00
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
2008-12-18 22:28:57 +01:00
*/
#include "token.h"
2017-05-27 04:33:47 +02:00
#include "astutils.h"
#include "errorlogger.h"
2017-05-27 04:33:47 +02:00
#include "library.h"
#include "settings.h"
#include "symboldatabase.h"
#include "utils.h"
2017-05-27 04:33:47 +02:00
#include <cassert>
2017-05-27 04:33:47 +02:00
#include <cctype>
2008-12-18 22:28:57 +01:00
#include <cstring>
#include <iostream>
#include <map>
2017-05-27 04:33:47 +02:00
#include <set>
#include <stack>
2017-05-27 04:33:47 +02:00
#include <utility>
static const std::string literal_prefix[4] = {"u8", "u", "U", "L"};
static bool isStringCharLiteral(const std::string &str, char q)
{
if (!endsWith(str, q))
return false;
if (str[0] == q && str.length() > 1)
return true;
for (const std::string & p: literal_prefix) {
2019-03-10 12:24:28 +01:00
if ((str.length() + 1) > p.length() && (str.compare(0, p.size() + 1, (p + q)) == 0))
return true;
}
return false;
}
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
const std::list<ValueFlow::Value> TokenImpl::mEmptyValueList;
Fix crash bug #8579 (#1238) * Added declaration for deletePrevious function * Added definition for deletePrevious function * Fixed crash from deleteThis invalidating pointers The crash was caused by deleteThis() invalidating the pointer to a constant variable usage. This happened when a usage followed an assignment. This fixes bug #8579. * Added tokensFront to match tokensBack This means deletePrevious can set the list's front if necessary. * Initialised tokensFront in appropriate places * Switched to using default Token constructor * Switched to using Token default constructor * Switched to using default constructor for Token * Added missing argument to Token constructor * Changed to use default constructor for Tokens * Switched to using default constructor for Tokens * Switched to using default constructor for Token * Added new test for deleting front Token Also made sure to use the correct constructor for Token in other tests. * Syntax error * Replaced tokensFront and tokensBack with a struct This decreases the size of the Token class for performance purposes. * Replaced tokensFront and tokensBack with a struct * Added tokensFrontBack to destructor * Reworked to use TokensBackFront struct Also ran astyle. * Reworked to use TokenList's TokensFrontBack member * Reworked to use TokensFrontBack struct * Reworked to use TokensFrontBack struct * Reworked to work with TokensFrontBack struct * Removed unnecessary scope operator * Added missing parentheses * Fixed syntax error * Removed unnecessary constructor * Default constructor now 0-initialises everything This is safer for not using a temporary TokensFrontBack object, and doesn't use delegating constructors which aren't supported yet. * Fixed unsafe null check * Added missing explicit keyword * Fixing stylistic nits Removed default constructor as it has been superseded by the single-argument constructor with a default argument value. Renamed listEnds to tokensFrontBack. Fixed if statement that was supposed to be adding safety but would actually cause a crash if tokensFrontBack was null. * Fixing stylistic nits Removed default constructor and replaced it with a single-argument constructor with a default value. * Fixing stylistic nits Renamed _listEnds to _tokensFrontBack. * Fixing stylistic nits Renamed _listEnds to _tokensFrontBack.
2018-05-25 07:15:05 +02:00
Token::Token(TokensFrontBack *tokensFrontBack) :
mTokensFrontBack(tokensFrontBack),
2018-06-16 16:16:55 +02:00
mNext(nullptr),
mPrevious(nullptr),
2018-06-16 20:30:09 +02:00
mLink(nullptr),
2018-06-16 16:40:02 +02:00
mTokType(eNone),
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
mFlags(0)
2008-12-18 22:28:57 +01:00
{
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
mImpl = new TokenImpl();
2008-12-18 22:28:57 +01:00
}
Token::~Token()
2008-12-18 22:28:57 +01:00
{
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
delete mImpl;
2008-12-18 22:28:57 +01:00
}
static const std::set<std::string> controlFlowKeywords = {
"goto",
"do",
"if",
"else",
"for",
"while",
"switch",
"case",
"break",
"continue",
"return"
};
void Token::update_property_info()
2008-12-18 22:28:57 +01:00
{
2018-06-16 23:03:15 +02:00
setFlag(fIsControlFlowKeyword, controlFlowKeywords.find(mStr) != controlFlowKeywords.end());
2018-06-16 23:03:15 +02:00
if (!mStr.empty()) {
if (mStr == "true" || mStr == "false")
2017-10-15 01:27:47 +02:00
tokType(eBoolean);
else if (isStringCharLiteral(mStr, '\"'))
tokType(eString);
else if (isStringCharLiteral(mStr, '\''))
tokType(eChar);
2018-06-16 23:03:15 +02:00
else if (std::isalpha((unsigned char)mStr[0]) || mStr[0] == '_' || mStr[0] == '$') { // Name
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (mImpl->mVarId)
2017-10-15 01:27:47 +02:00
tokType(eVariable);
2018-06-16 16:40:02 +02:00
else if (mTokType != eVariable && mTokType != eFunction && mTokType != eType && mTokType != eKeyword)
2017-10-15 01:27:47 +02:00
tokType(eName);
2018-06-16 23:03:15 +02:00
} else if (std::isdigit((unsigned char)mStr[0]) || (mStr.length() > 1 && mStr[0] == '-' && std::isdigit((unsigned char)mStr[1])))
2017-10-15 01:27:47 +02:00
tokType(eNumber);
2018-06-16 23:03:15 +02:00
else if (mStr == "=" || mStr == "<<=" || mStr == ">>=" ||
(mStr.size() == 2U && mStr[1] == '=' && std::strchr("+-*/%&^|", mStr[0])))
2017-10-15 01:27:47 +02:00
tokType(eAssignmentOp);
2018-06-16 23:03:15 +02:00
else if (mStr.size() == 1 && mStr.find_first_of(",[]()?:") != std::string::npos)
2017-10-15 01:27:47 +02:00
tokType(eExtendedOp);
2018-06-16 23:03:15 +02:00
else if (mStr=="<<" || mStr==">>" || (mStr.size()==1 && mStr.find_first_of("+-*/%") != std::string::npos))
2017-10-15 01:27:47 +02:00
tokType(eArithmeticalOp);
2018-06-16 23:03:15 +02:00
else if (mStr.size() == 1 && mStr.find_first_of("&|^~") != std::string::npos)
2017-10-15 01:27:47 +02:00
tokType(eBitOp);
2018-06-16 23:03:15 +02:00
else if (mStr.size() <= 2 &&
(mStr == "&&" ||
mStr == "||" ||
mStr == "!"))
2017-10-15 01:27:47 +02:00
tokType(eLogicalOp);
2018-06-16 23:03:15 +02:00
else if (mStr.size() <= 2 && !mLink &&
(mStr == "==" ||
mStr == "!=" ||
mStr == "<" ||
mStr == "<=" ||
mStr == ">" ||
mStr == ">="))
2017-10-15 01:27:47 +02:00
tokType(eComparisonOp);
2018-06-16 23:03:15 +02:00
else if (mStr.size() == 2 &&
(mStr == "++" ||
mStr == "--"))
2017-10-15 01:27:47 +02:00
tokType(eIncDecOp);
2018-06-16 23:03:15 +02:00
else if (mStr.size() == 1 && (mStr.find_first_of("{}") != std::string::npos || (mLink && mStr.find_first_of("<>") != std::string::npos)))
2017-10-15 01:27:47 +02:00
tokType(eBracket);
else
2017-10-15 01:27:47 +02:00
tokType(eOther);
} else {
2017-10-15 01:27:47 +02:00
tokType(eNone);
}
update_property_char_string_literal();
update_property_isStandardType();
}
static const std::set<std::string> stdTypes = { "bool"
, "_Bool"
, "char"
, "double"
, "float"
, "int"
, "long"
, "short"
, "size_t"
, "void"
, "wchar_t"
};
void Token::update_property_isStandardType()
{
isStandardType(false);
2018-06-16 23:03:15 +02:00
if (mStr.size() < 3)
return;
2018-06-16 23:03:15 +02:00
if (stdTypes.find(mStr)!=stdTypes.end()) {
isStandardType(true);
2017-10-15 01:27:47 +02:00
tokType(eType);
}
}
void Token::update_property_char_string_literal()
{
if (!(mTokType == Token::eString || mTokType == Token::eChar)) // Token has already been updated
return;
for (const std::string & p : literal_prefix) {
if (((mTokType == Token::eString) && mStr.compare(0, p.size() + 1, p + "\"") == 0) ||
((mTokType == Token::eChar) && (mStr.compare(0, p.size() + 1, p + "\'") == 0))) {
mStr = mStr.substr(p.size());
isLong(p != "u8");
break;
}
}
}
bool Token::isUpperCaseName() const
{
if (!isName())
return false;
2018-06-16 23:03:15 +02:00
for (size_t i = 0; i < mStr.length(); ++i) {
if (std::islower(mStr[i]))
return false;
}
return true;
}
void Token::concatStr(std::string const& b)
{
2018-06-16 23:03:15 +02:00
mStr.erase(mStr.length() - 1);
mStr.append(b.begin() + 1, b.end());
update_property_info();
}
std::string Token::strValue() const
{
2018-06-16 16:40:02 +02:00
assert(mTokType == eString);
2018-06-16 23:03:15 +02:00
std::string ret(mStr.substr(1, mStr.length() - 2));
std::string::size_type pos = 0U;
while ((pos = ret.find('\\', pos)) != std::string::npos) {
ret.erase(pos,1U);
if (ret[pos] >= 'a') {
if (ret[pos] == 'n')
ret[pos] = '\n';
else if (ret[pos] == 'r')
ret[pos] = '\r';
else if (ret[pos] == 't')
ret[pos] = '\t';
}
if (ret[pos] == '0')
return ret.substr(0,pos);
pos++;
}
return ret;
}
void Token::deleteNext(nonneg int count)
2008-12-18 22:28:57 +01:00
{
while (mNext && count > 0) {
2018-06-16 16:16:55 +02:00
Token *n = mNext;
// #8154 we are about to be unknown -> destroy the link to us
2018-06-16 20:30:09 +02:00
if (n->mLink && n->mLink->mLink == n)
n->mLink->link(nullptr);
2018-06-16 16:16:55 +02:00
mNext = n->next();
delete n;
--count;
}
2018-06-16 16:16:55 +02:00
if (mNext)
mNext->previous(this);
else if (mTokensFrontBack)
mTokensFrontBack->back = this;
Fix crash bug #8579 (#1238) * Added declaration for deletePrevious function * Added definition for deletePrevious function * Fixed crash from deleteThis invalidating pointers The crash was caused by deleteThis() invalidating the pointer to a constant variable usage. This happened when a usage followed an assignment. This fixes bug #8579. * Added tokensFront to match tokensBack This means deletePrevious can set the list's front if necessary. * Initialised tokensFront in appropriate places * Switched to using default Token constructor * Switched to using Token default constructor * Switched to using default constructor for Token * Added missing argument to Token constructor * Changed to use default constructor for Tokens * Switched to using default constructor for Tokens * Switched to using default constructor for Token * Added new test for deleting front Token Also made sure to use the correct constructor for Token in other tests. * Syntax error * Replaced tokensFront and tokensBack with a struct This decreases the size of the Token class for performance purposes. * Replaced tokensFront and tokensBack with a struct * Added tokensFrontBack to destructor * Reworked to use TokensBackFront struct Also ran astyle. * Reworked to use TokenList's TokensFrontBack member * Reworked to use TokensFrontBack struct * Reworked to use TokensFrontBack struct * Reworked to work with TokensFrontBack struct * Removed unnecessary scope operator * Added missing parentheses * Fixed syntax error * Removed unnecessary constructor * Default constructor now 0-initialises everything This is safer for not using a temporary TokensFrontBack object, and doesn't use delegating constructors which aren't supported yet. * Fixed unsafe null check * Added missing explicit keyword * Fixing stylistic nits Removed default constructor as it has been superseded by the single-argument constructor with a default argument value. Renamed listEnds to tokensFrontBack. Fixed if statement that was supposed to be adding safety but would actually cause a crash if tokensFrontBack was null. * Fixing stylistic nits Removed default constructor and replaced it with a single-argument constructor with a default value. * Fixing stylistic nits Renamed _listEnds to _tokensFrontBack. * Fixing stylistic nits Renamed _listEnds to _tokensFrontBack.
2018-05-25 07:15:05 +02:00
}
void Token::deletePrevious(nonneg int count)
Fix crash bug #8579 (#1238) * Added declaration for deletePrevious function * Added definition for deletePrevious function * Fixed crash from deleteThis invalidating pointers The crash was caused by deleteThis() invalidating the pointer to a constant variable usage. This happened when a usage followed an assignment. This fixes bug #8579. * Added tokensFront to match tokensBack This means deletePrevious can set the list's front if necessary. * Initialised tokensFront in appropriate places * Switched to using default Token constructor * Switched to using Token default constructor * Switched to using default constructor for Token * Added missing argument to Token constructor * Changed to use default constructor for Tokens * Switched to using default constructor for Tokens * Switched to using default constructor for Token * Added new test for deleting front Token Also made sure to use the correct constructor for Token in other tests. * Syntax error * Replaced tokensFront and tokensBack with a struct This decreases the size of the Token class for performance purposes. * Replaced tokensFront and tokensBack with a struct * Added tokensFrontBack to destructor * Reworked to use TokensBackFront struct Also ran astyle. * Reworked to use TokenList's TokensFrontBack member * Reworked to use TokensFrontBack struct * Reworked to use TokensFrontBack struct * Reworked to work with TokensFrontBack struct * Removed unnecessary scope operator * Added missing parentheses * Fixed syntax error * Removed unnecessary constructor * Default constructor now 0-initialises everything This is safer for not using a temporary TokensFrontBack object, and doesn't use delegating constructors which aren't supported yet. * Fixed unsafe null check * Added missing explicit keyword * Fixing stylistic nits Removed default constructor as it has been superseded by the single-argument constructor with a default argument value. Renamed listEnds to tokensFrontBack. Fixed if statement that was supposed to be adding safety but would actually cause a crash if tokensFrontBack was null. * Fixing stylistic nits Removed default constructor and replaced it with a single-argument constructor with a default value. * Fixing stylistic nits Renamed _listEnds to _tokensFrontBack. * Fixing stylistic nits Renamed _listEnds to _tokensFrontBack.
2018-05-25 07:15:05 +02:00
{
while (mPrevious && count > 0) {
2018-06-16 16:16:55 +02:00
Token *p = mPrevious;
Fix crash bug #8579 (#1238) * Added declaration for deletePrevious function * Added definition for deletePrevious function * Fixed crash from deleteThis invalidating pointers The crash was caused by deleteThis() invalidating the pointer to a constant variable usage. This happened when a usage followed an assignment. This fixes bug #8579. * Added tokensFront to match tokensBack This means deletePrevious can set the list's front if necessary. * Initialised tokensFront in appropriate places * Switched to using default Token constructor * Switched to using Token default constructor * Switched to using default constructor for Token * Added missing argument to Token constructor * Changed to use default constructor for Tokens * Switched to using default constructor for Tokens * Switched to using default constructor for Token * Added new test for deleting front Token Also made sure to use the correct constructor for Token in other tests. * Syntax error * Replaced tokensFront and tokensBack with a struct This decreases the size of the Token class for performance purposes. * Replaced tokensFront and tokensBack with a struct * Added tokensFrontBack to destructor * Reworked to use TokensBackFront struct Also ran astyle. * Reworked to use TokenList's TokensFrontBack member * Reworked to use TokensFrontBack struct * Reworked to use TokensFrontBack struct * Reworked to work with TokensFrontBack struct * Removed unnecessary scope operator * Added missing parentheses * Fixed syntax error * Removed unnecessary constructor * Default constructor now 0-initialises everything This is safer for not using a temporary TokensFrontBack object, and doesn't use delegating constructors which aren't supported yet. * Fixed unsafe null check * Added missing explicit keyword * Fixing stylistic nits Removed default constructor as it has been superseded by the single-argument constructor with a default argument value. Renamed listEnds to tokensFrontBack. Fixed if statement that was supposed to be adding safety but would actually cause a crash if tokensFrontBack was null. * Fixing stylistic nits Removed default constructor and replaced it with a single-argument constructor with a default value. * Fixing stylistic nits Renamed _listEnds to _tokensFrontBack. * Fixing stylistic nits Renamed _listEnds to _tokensFrontBack.
2018-05-25 07:15:05 +02:00
// #8154 we are about to be unknown -> destroy the link to us
2018-06-16 20:30:09 +02:00
if (p->mLink && p->mLink->mLink == p)
p->mLink->link(nullptr);
Fix crash bug #8579 (#1238) * Added declaration for deletePrevious function * Added definition for deletePrevious function * Fixed crash from deleteThis invalidating pointers The crash was caused by deleteThis() invalidating the pointer to a constant variable usage. This happened when a usage followed an assignment. This fixes bug #8579. * Added tokensFront to match tokensBack This means deletePrevious can set the list's front if necessary. * Initialised tokensFront in appropriate places * Switched to using default Token constructor * Switched to using Token default constructor * Switched to using default constructor for Token * Added missing argument to Token constructor * Changed to use default constructor for Tokens * Switched to using default constructor for Tokens * Switched to using default constructor for Token * Added new test for deleting front Token Also made sure to use the correct constructor for Token in other tests. * Syntax error * Replaced tokensFront and tokensBack with a struct This decreases the size of the Token class for performance purposes. * Replaced tokensFront and tokensBack with a struct * Added tokensFrontBack to destructor * Reworked to use TokensBackFront struct Also ran astyle. * Reworked to use TokenList's TokensFrontBack member * Reworked to use TokensFrontBack struct * Reworked to use TokensFrontBack struct * Reworked to work with TokensFrontBack struct * Removed unnecessary scope operator * Added missing parentheses * Fixed syntax error * Removed unnecessary constructor * Default constructor now 0-initialises everything This is safer for not using a temporary TokensFrontBack object, and doesn't use delegating constructors which aren't supported yet. * Fixed unsafe null check * Added missing explicit keyword * Fixing stylistic nits Removed default constructor as it has been superseded by the single-argument constructor with a default argument value. Renamed listEnds to tokensFrontBack. Fixed if statement that was supposed to be adding safety but would actually cause a crash if tokensFrontBack was null. * Fixing stylistic nits Removed default constructor and replaced it with a single-argument constructor with a default value. * Fixing stylistic nits Renamed _listEnds to _tokensFrontBack. * Fixing stylistic nits Renamed _listEnds to _tokensFrontBack.
2018-05-25 07:15:05 +02:00
2018-06-16 16:16:55 +02:00
mPrevious = p->previous();
Fix crash bug #8579 (#1238) * Added declaration for deletePrevious function * Added definition for deletePrevious function * Fixed crash from deleteThis invalidating pointers The crash was caused by deleteThis() invalidating the pointer to a constant variable usage. This happened when a usage followed an assignment. This fixes bug #8579. * Added tokensFront to match tokensBack This means deletePrevious can set the list's front if necessary. * Initialised tokensFront in appropriate places * Switched to using default Token constructor * Switched to using Token default constructor * Switched to using default constructor for Token * Added missing argument to Token constructor * Changed to use default constructor for Tokens * Switched to using default constructor for Tokens * Switched to using default constructor for Token * Added new test for deleting front Token Also made sure to use the correct constructor for Token in other tests. * Syntax error * Replaced tokensFront and tokensBack with a struct This decreases the size of the Token class for performance purposes. * Replaced tokensFront and tokensBack with a struct * Added tokensFrontBack to destructor * Reworked to use TokensBackFront struct Also ran astyle. * Reworked to use TokenList's TokensFrontBack member * Reworked to use TokensFrontBack struct * Reworked to use TokensFrontBack struct * Reworked to work with TokensFrontBack struct * Removed unnecessary scope operator * Added missing parentheses * Fixed syntax error * Removed unnecessary constructor * Default constructor now 0-initialises everything This is safer for not using a temporary TokensFrontBack object, and doesn't use delegating constructors which aren't supported yet. * Fixed unsafe null check * Added missing explicit keyword * Fixing stylistic nits Removed default constructor as it has been superseded by the single-argument constructor with a default argument value. Renamed listEnds to tokensFrontBack. Fixed if statement that was supposed to be adding safety but would actually cause a crash if tokensFrontBack was null. * Fixing stylistic nits Removed default constructor and replaced it with a single-argument constructor with a default value. * Fixing stylistic nits Renamed _listEnds to _tokensFrontBack. * Fixing stylistic nits Renamed _listEnds to _tokensFrontBack.
2018-05-25 07:15:05 +02:00
delete p;
--count;
Fix crash bug #8579 (#1238) * Added declaration for deletePrevious function * Added definition for deletePrevious function * Fixed crash from deleteThis invalidating pointers The crash was caused by deleteThis() invalidating the pointer to a constant variable usage. This happened when a usage followed an assignment. This fixes bug #8579. * Added tokensFront to match tokensBack This means deletePrevious can set the list's front if necessary. * Initialised tokensFront in appropriate places * Switched to using default Token constructor * Switched to using Token default constructor * Switched to using default constructor for Token * Added missing argument to Token constructor * Changed to use default constructor for Tokens * Switched to using default constructor for Tokens * Switched to using default constructor for Token * Added new test for deleting front Token Also made sure to use the correct constructor for Token in other tests. * Syntax error * Replaced tokensFront and tokensBack with a struct This decreases the size of the Token class for performance purposes. * Replaced tokensFront and tokensBack with a struct * Added tokensFrontBack to destructor * Reworked to use TokensBackFront struct Also ran astyle. * Reworked to use TokenList's TokensFrontBack member * Reworked to use TokensFrontBack struct * Reworked to use TokensFrontBack struct * Reworked to work with TokensFrontBack struct * Removed unnecessary scope operator * Added missing parentheses * Fixed syntax error * Removed unnecessary constructor * Default constructor now 0-initialises everything This is safer for not using a temporary TokensFrontBack object, and doesn't use delegating constructors which aren't supported yet. * Fixed unsafe null check * Added missing explicit keyword * Fixing stylistic nits Removed default constructor as it has been superseded by the single-argument constructor with a default argument value. Renamed listEnds to tokensFrontBack. Fixed if statement that was supposed to be adding safety but would actually cause a crash if tokensFrontBack was null. * Fixing stylistic nits Removed default constructor and replaced it with a single-argument constructor with a default value. * Fixing stylistic nits Renamed _listEnds to _tokensFrontBack. * Fixing stylistic nits Renamed _listEnds to _tokensFrontBack.
2018-05-25 07:15:05 +02:00
}
2018-06-16 16:16:55 +02:00
if (mPrevious)
mPrevious->next(this);
else if (mTokensFrontBack)
mTokensFrontBack->front = this;
2008-12-18 22:28:57 +01:00
}
void Token::swapWithNext()
{
2018-06-16 16:16:55 +02:00
if (mNext) {
2018-06-16 23:03:15 +02:00
std::swap(mStr, mNext->mStr);
2018-06-16 16:40:02 +02:00
std::swap(mTokType, mNext->mTokType);
2018-06-16 16:16:55 +02:00
std::swap(mFlags, mNext->mFlags);
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
std::swap(mImpl, mNext->mImpl);
2018-12-21 13:54:59 +01:00
for (auto templateSimplifierPointer : mImpl->mTemplateSimplifierPointers) {
templateSimplifierPointer->token(this);
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
}
2018-12-21 13:54:59 +01:00
for (auto templateSimplifierPointer : mNext->mImpl->mTemplateSimplifierPointers) {
templateSimplifierPointer->token(mNext);
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
}
2018-06-16 20:30:09 +02:00
if (mNext->mLink)
mNext->mLink->mLink = this;
if (this->mLink)
this->mLink->mLink = mNext;
std::swap(mLink, mNext->mLink);
}
}
2017-08-25 23:30:04 +02:00
void Token::takeData(Token *fromToken)
{
2018-06-16 23:03:15 +02:00
mStr = fromToken->mStr;
2018-06-16 16:40:02 +02:00
tokType(fromToken->mTokType);
2018-06-16 16:14:34 +02:00
mFlags = fromToken->mFlags;
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
delete mImpl;
mImpl = fromToken->mImpl;
fromToken->mImpl = nullptr;
2018-12-21 13:54:59 +01:00
for (auto templateSimplifierPointer : mImpl->mTemplateSimplifierPointers) {
templateSimplifierPointer->token(this);
2017-08-25 23:30:04 +02:00
}
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
mLink = fromToken->mLink;
2018-06-16 20:30:09 +02:00
if (mLink)
mLink->link(this);
2017-08-25 23:30:04 +02:00
}
void Token::deleteThis()
{
2018-06-16 16:16:55 +02:00
if (mNext) { // Copy next to this and delete next
takeData(mNext);
mNext->link(nullptr); // mark as unlinked
deleteNext();
2018-06-16 16:16:55 +02:00
} else if (mPrevious && mPrevious->mPrevious) { // Copy previous to this and delete previous
takeData(mPrevious);
2018-06-16 16:16:55 +02:00
Token* toDelete = mPrevious;
mPrevious = mPrevious->mPrevious;
mPrevious->mNext = this;
delete toDelete;
2011-10-13 20:53:06 +02:00
} else {
// We are the last token in the list, we can't delete
// ourselves, so just make us empty
str("");
}
}
void Token::replace(Token *replaceThis, Token *start, Token *end)
{
// Fix the whole in the old location of start and end
if (start->previous())
start->previous()->next(end->next());
if (end->next())
end->next()->previous(start->previous());
// Move start and end to their new location
if (replaceThis->previous())
replaceThis->previous()->next(start);
if (replaceThis->next())
replaceThis->next()->previous(end);
start->previous(replaceThis->previous());
end->next(replaceThis->next());
if (end->mTokensFrontBack && end->mTokensFrontBack->back == end) {
while (end->next())
end = end->next();
end->mTokensFrontBack->back = end;
}
// Update mProgressValue, fileIndex and linenr
2012-01-22 00:02:55 +01:00
for (Token *tok = start; tok != end->next(); tok = tok->next())
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
tok->mImpl->mProgressValue = replaceThis->mImpl->mProgressValue;
// Delete old token, which is replaced
delete replaceThis;
}
const Token *Token::tokAt(int index) const
2008-12-18 22:28:57 +01:00
{
const Token *tok = this;
while (index > 0 && tok) {
tok = tok->next();
--index;
}
while (index < 0 && tok) {
tok = tok->previous();
++index;
2008-12-18 22:28:57 +01:00
}
return tok;
}
const Token *Token::linkAt(int index) const
{
const Token *tok = this->tokAt(index);
if (!tok) {
throw InternalError(this, "Internal error. Token::linkAt called with index outside the tokens range.");
}
return tok->link();
}
const std::string &Token::strAt(int index) const
2008-12-18 22:28:57 +01:00
{
const Token *tok = this->tokAt(index);
2018-06-16 23:03:15 +02:00
return tok ? tok->mStr : emptyString;
2008-12-18 22:28:57 +01:00
}
static int multiComparePercent(const Token *tok, const char*& haystack, nonneg int varid)
{
++haystack;
// Compare only the first character of the string for optimization reasons
switch (haystack[0]) {
case '\0':
case ' ':
case '|':
//simple '%' character
haystack += 1;
if (tok->isArithmeticalOp() && tok->str() == "%")
return 1;
break;
case 'v':
if (haystack[3] == '%') { // %var%
haystack += 4;
if (tok->varId() != 0)
return 1;
} else { // %varid%
if (varid == 0) {
throw InternalError(tok, "Internal error. Token::Match called with varid 0. Please report this to Cppcheck developers");
}
haystack += 6;
if (tok->varId() == varid)
return 1;
}
break;
case 't':
// Type (%type%)
{
haystack += 5;
if (tok->isName() && tok->varId() == 0 && !tok->isKeyword())
return 1;
}
break;
case 'a':
// Accept any token (%any%) or assign (%assign%)
{
if (haystack[3] == '%') { // %any%
haystack += 4;
return 1;
} else { // %assign%
haystack += 7;
if (tok->isAssignmentOp())
return 1;
}
}
break;
case 'n':
// Number (%num%) or name (%name%)
{
if (haystack[4] == '%') { // %name%
haystack += 5;
if (tok->isName())
return 1;
} else {
haystack += 4;
if (tok->isNumber())
return 1;
}
}
break;
case 'c': {
haystack += 1;
// Character (%char%)
if (haystack[0] == 'h') {
haystack += 4;
if (tok->tokType() == Token::eChar)
return 1;
}
// Const operator (%cop%)
else if (haystack[1] == 'p') {
haystack += 3;
if (tok->isConstOp())
return 1;
}
// Comparison operator (%comp%)
else {
haystack += 4;
if (tok->isComparisonOp())
return 1;
}
}
break;
case 's':
// String (%str%)
{
haystack += 4;
if (tok->tokType() == Token::eString)
return 1;
}
break;
case 'b':
// Bool (%bool%)
{
haystack += 5;
if (tok->isBoolean())
return 1;
}
break;
case 'o': {
++haystack;
if (haystack[1] == '%') {
// Op (%op%)
if (haystack[0] == 'p') {
haystack += 2;
if (tok->isOp())
return 1;
}
// Or (%or%)
else {
haystack += 2;
if (tok->tokType() == Token::eBitOp && tok->str() == "|")
return 1;
}
}
// Oror (%oror%)
else {
haystack += 4;
if (tok->tokType() == Token::eLogicalOp && tok->str() == "||")
return 1;
}
}
break;
default:
//unknown %cmd%, abort
2015-03-29 21:05:18 +02:00
throw InternalError(tok, "Unexpected command");
}
if (*haystack == '|')
haystack += 1;
else
return -1;
return 0xFFFF;
}
2019-07-14 17:31:26 +02:00
int Token::multiCompare(const Token *tok, const char *haystack, nonneg int varid)
2008-12-18 22:28:57 +01:00
{
const char *needle = tok->str().c_str();
const char *needlePointer = needle;
2012-03-25 11:51:59 +02:00
for (;;) {
if (needlePointer == needle && haystack[0] == '%' && haystack[1] != '|' && haystack[1] != '\0' && haystack[1] != ' ') {
2018-04-04 21:51:31 +02:00
const int ret = multiComparePercent(tok, haystack, varid);
if (ret < 2)
return ret;
2011-10-13 20:53:06 +02:00
} else if (*haystack == '|') {
if (*needlePointer == 0) {
// If needle is at the end, we have a match.
2008-12-18 22:28:57 +01:00
return 1;
}
2008-12-18 22:28:57 +01:00
needlePointer = needle;
++haystack;
} else if (*needlePointer == *haystack) {
if (*needlePointer == '\0')
return 1;
++needlePointer;
++haystack;
2011-10-13 20:53:06 +02:00
} else if (*haystack == ' ' || *haystack == '\0') {
if (needlePointer == needle)
return 0;
break;
2008-12-18 22:28:57 +01:00
}
// If haystack and needle don't share the same character,
// find next '|' character.
2011-10-13 20:53:06 +02:00
else {
needlePointer = needle;
2008-12-18 22:28:57 +01:00
2011-10-13 20:53:06 +02:00
do {
++haystack;
2011-10-13 20:53:06 +02:00
} while (*haystack != ' ' && *haystack != '|' && *haystack);
2008-12-18 22:28:57 +01:00
2011-10-13 20:53:06 +02:00
if (*haystack == ' ' || *haystack == '\0') {
return -1;
}
2008-12-18 22:28:57 +01:00
++haystack;
}
}
if (*needlePointer == '\0')
return 1;
2008-12-18 22:28:57 +01:00
return -1;
}
bool Token::simpleMatch(const Token *tok, const char pattern[])
{
if (!tok)
return false; // shortcut
const char *current = pattern;
const char *next = std::strchr(pattern, ' ');
if (!next)
2012-12-27 11:51:12 +01:00
next = pattern + std::strlen(pattern);
2011-10-13 20:53:06 +02:00
while (*current) {
2018-04-04 21:51:31 +02:00
const std::size_t length = next - current;
2018-06-16 23:03:15 +02:00
if (!tok || length != tok->mStr.length() || std::strncmp(current, tok->mStr.c_str(), length))
return false;
current = next;
2011-10-13 20:53:06 +02:00
if (*next) {
2012-12-27 11:51:12 +01:00
next = std::strchr(++current, ' ');
if (!next)
2012-12-27 11:51:12 +01:00
next = current + std::strlen(current);
}
tok = tok->next();
}
return true;
}
bool Token::firstWordEquals(const char *str, const char *word)
{
2011-10-13 20:53:06 +02:00
for (;;) {
if (*str != *word) {
return (*str == ' ' && *word == 0);
2011-10-13 20:53:06 +02:00
} else if (*str == 0)
break;
++str;
++word;
}
return true;
}
const char *Token::chrInFirstWord(const char *str, char c)
{
2011-10-13 20:53:06 +02:00
for (;;) {
if (*str == ' ' || *str == 0)
return nullptr;
if (*str == c)
return str;
++str;
}
}
bool Token::Match(const Token *tok, const char pattern[], nonneg int varid)
2008-12-18 22:28:57 +01:00
{
const char *p = pattern;
2011-10-13 20:53:06 +02:00
while (*p) {
2008-12-18 22:28:57 +01:00
// Skip spaces in pattern..
while (*p == ' ')
2009-01-01 23:22:28 +01:00
++p;
2008-12-18 22:28:57 +01:00
// No token => Success!
if (*p == '\0')
break;
2008-12-18 22:28:57 +01:00
2011-10-13 20:53:06 +02:00
if (!tok) {
// If we have no tokens, pattern "!!else" should return true
2012-11-25 15:13:41 +01:00
if (p[0] == '!' && p[1] == '!' && p[2] != '\0') {
2010-09-20 20:15:07 +02:00
while (*p && *p != ' ')
++p;
continue;
2011-10-13 20:53:06 +02:00
} else
return false;
}
2008-12-18 22:28:57 +01:00
// [.. => search for a one-character token..
if (p[0] == '[' && chrInFirstWord(p, ']')) {
if (tok->str().length() != 1)
return false;
2011-07-15 19:02:16 +02:00
const char *temp = p+1;
bool chrFound = false;
2019-07-13 16:06:24 +02:00
int count = 0;
2011-10-13 20:53:06 +02:00
while (*temp && *temp != ' ') {
if (*temp == ']') {
++count;
}
else if (*temp == tok->str()[0]) {
chrFound = true;
break;
}
++temp;
}
if (count > 1 && tok->str()[0] == ']')
chrFound = true;
if (!chrFound)
2008-12-18 22:28:57 +01:00
return false;
p = temp;
while (*p && *p != ' ')
++p;
2008-12-18 22:28:57 +01:00
}
// Parse "not" options. Token can be anything except the given one
else if (p[0] == '!' && p[1] == '!' && p[2] != '\0') {
p += 2;
if (firstWordEquals(p, tok->str().c_str()))
return false;
while (*p && *p != ' ')
++p;
}
2008-12-18 22:28:57 +01:00
// Parse multi options, such as void|int|char (accept token which is one of these 3)
else {
2018-04-04 21:51:31 +02:00
const int res = multiCompare(tok, p, varid);
2011-10-13 20:53:06 +02:00
if (res == 0) {
2008-12-18 22:28:57 +01:00
// Empty alternative matches, use the same token on next round
2010-09-20 20:15:07 +02:00
while (*p && *p != ' ')
++p;
2008-12-18 22:28:57 +01:00
continue;
2011-10-13 20:53:06 +02:00
} else if (res == -1) {
2008-12-18 22:28:57 +01:00
// No match
return false;
}
}
2010-09-20 20:15:07 +02:00
while (*p && *p != ' ')
++p;
2008-12-18 22:28:57 +01:00
tok = tok->next();
}
// The end of the pattern has been reached and nothing wrong has been found
return true;
}
nonneg int Token::getStrLength(const Token *tok)
{
2014-02-15 08:05:54 +01:00
assert(tok != nullptr);
2018-06-16 16:40:02 +02:00
assert(tok->mTokType == eString);
2019-07-13 16:06:24 +02:00
int len = 0;
std::string::const_iterator it = tok->str().begin() + 1U;
const std::string::const_iterator end = tok->str().end() - 1U;
while (it != end) {
if (*it == '\\') {
++it;
// string ends at '\0'
if (*it == '0')
return len;
}
if (*it == '\0')
return len;
++it;
++len;
}
return len;
}
nonneg int Token::getStrSize(const Token *tok)
{
assert(tok != nullptr);
assert(tok->tokType() == eString);
const std::string &str = tok->str();
2019-07-13 16:06:24 +02:00
int sizeofstring = 1;
for (int i = 1; i < (int)str.size() - 1; i++) {
if (str[i] == '\\')
++i;
++sizeofstring;
}
return sizeofstring;
}
2019-07-13 16:06:24 +02:00
std::string Token::getCharAt(const Token *tok, MathLib::bigint index)
2013-01-13 20:52:38 +01:00
{
2014-02-15 08:05:54 +01:00
assert(tok != nullptr);
2013-01-13 20:52:38 +01:00
std::string::const_iterator it = tok->str().begin() + 1U;
const std::string::const_iterator end = tok->str().end() - 1U;
2013-01-13 20:52:38 +01:00
while (it != end) {
2013-01-13 20:52:38 +01:00
if (index == 0) {
if (*it == '\0')
return "\\0";
std::string ret(1, *it);
if (*it == '\\') {
++it;
ret += *it;
2013-01-13 20:52:38 +01:00
}
return ret;
}
if (*it == '\\')
++it;
++it;
2013-01-13 20:52:38 +01:00
--index;
}
assert(index == 0);
return "\\0";
}
void Token::move(Token *srcStart, Token *srcEnd, Token *newLocation)
{
/**[newLocation] -> b -> c -> [srcStart] -> [srcEnd] -> f */
// Fix the gap, which tokens to be moved will leave
srcStart->previous()->next(srcEnd->next());
srcEnd->next()->previous(srcStart->previous());
// Fix the tokens to be moved
srcEnd->next(newLocation->next());
srcStart->previous(newLocation);
// Fix the tokens at newLocation
newLocation->next()->previous(srcEnd);
newLocation->next(srcStart);
// Update _progressValue
for (Token *tok = srcStart; tok != srcEnd->next(); tok = tok->next())
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
tok->mImpl->mProgressValue = newLocation->mImpl->mProgressValue;
}
Token* Token::nextArgument() const
2011-10-23 11:23:48 +02:00
{
for (const Token* tok = this; tok; tok = tok->next()) {
if (tok->str() == ",")
return tok->next();
else if (tok->link() && Token::Match(tok, "(|{|[|<"))
tok = tok->link();
else if (Token::Match(tok, ")|;"))
return nullptr;
2011-10-23 11:23:48 +02:00
}
return nullptr;
2011-10-23 11:23:48 +02:00
}
Token* Token::nextArgumentBeforeCreateLinks2() const
{
for (const Token* tok = this; tok; tok = tok->next()) {
if (tok->str() == ",")
return tok->next();
else if (tok->link() && Token::Match(tok, "(|{|["))
tok = tok->link();
else if (tok->str() == "<") {
const Token* temp = tok->findClosingBracket();
if (temp)
tok = temp;
} else if (Token::Match(tok, ")|;"))
return nullptr;
}
return nullptr;
}
Token* Token::nextTemplateArgument() const
{
for (const Token* tok = this; tok; tok = tok->next()) {
if (tok->str() == ",")
return tok->next();
else if (tok->link() && Token::Match(tok, "(|{|[|<"))
tok = tok->link();
else if (Token::Match(tok, ">|;"))
return nullptr;
}
return nullptr;
}
2013-07-31 10:30:20 +02:00
const Token * Token::findClosingBracket() const
{
2018-06-16 23:03:15 +02:00
if (mStr != "<")
2018-01-01 12:22:04 +01:00
return nullptr;
const Token *closing = nullptr;
const bool templateParameter(strAt(-1) == "template");
std::set<std::string> templateParameters;
2013-07-31 10:30:20 +02:00
2018-01-01 12:22:04 +01:00
unsigned int depth = 0;
for (closing = this; closing != nullptr; closing = closing->next()) {
if (Token::Match(closing, "{|[|(")) {
closing = closing->link();
if (!closing)
return nullptr; // #6803
} else if (Token::Match(closing, "}|]|)|;"))
return nullptr;
// we can make some guesses for template parameters
else if (closing->str() == "<" &&
(!templateParameter || (closing->previous() && closing->previous()->isName() &&
templateParameters.find(closing->strAt(-1)) == templateParameters.end())))
2018-01-01 12:22:04 +01:00
++depth;
else if (closing->str() == ">") {
if (--depth == 0)
return closing;
} else if (closing->str() == ">>") {
if (depth <= 2)
return closing;
depth -= 2;
}
// save named template parameter
else if (templateParameter && depth == 1 && closing->str() == "," &&
closing->previous()->isName() && !Match(closing->previous(), "class|typename|."))
templateParameters.insert(closing->strAt(-1));
}
2013-07-31 10:30:20 +02:00
return closing;
}
Token * Token::findClosingBracket()
{
// return value of const function
return const_cast<Token*>(const_cast<const Token*>(this)->findClosingBracket());
}
const Token * Token::findOpeningBracket() const
{
if (mStr != ">")
return nullptr;
const Token *opening = nullptr;
unsigned int depth = 0;
for (opening = this; opening != nullptr; opening = opening->previous()) {
if (Token::Match(opening, "}|]|)")) {
opening = opening->link();
if (!opening)
return nullptr;
} else if (Token::Match(opening, "{|{|(|;"))
return nullptr;
else if (opening->str() == ">")
++depth;
else if (opening->str() == "<") {
if (--depth == 0)
return opening;
}
}
return opening;
}
Token * Token::findOpeningBracket()
{
// return value of const function
return const_cast<Token*>(const_cast<const Token*>(this)->findOpeningBracket());
}
2008-12-18 22:28:57 +01:00
//---------------------------------------------------------------------------
const Token *Token::findsimplematch(const Token * const startTok, const char pattern[])
{
for (const Token* tok = startTok; tok; tok = tok->next()) {
if (Token::simpleMatch(tok, pattern))
return tok;
}
return nullptr;
}
const Token *Token::findsimplematch(const Token * const startTok, const char pattern[], const Token * const end)
{
for (const Token* tok = startTok; tok && tok != end; tok = tok->next()) {
if (Token::simpleMatch(tok, pattern))
return tok;
}
return nullptr;
}
const Token *Token::findmatch(const Token * const startTok, const char pattern[], const nonneg int varId)
{
for (const Token* tok = startTok; tok; tok = tok->next()) {
if (Token::Match(tok, pattern, varId))
return tok;
}
return nullptr;
}
const Token *Token::findmatch(const Token * const startTok, const char pattern[], const Token * const end, const nonneg int varId)
2010-07-26 16:46:37 +02:00
{
for (const Token* tok = startTok; tok && tok != end; tok = tok->next()) {
2010-07-26 16:46:37 +02:00
if (Token::Match(tok, pattern, varId))
return tok;
}
return nullptr;
2010-07-26 16:46:37 +02:00
}
2019-07-05 14:00:59 +02:00
void Token::function(const Function *f)
{
mImpl->mFunction = f;
if (f) {
if (f->isLambda())
tokType(eLambda);
else
tokType(eFunction);
} else if (mTokType == eFunction)
tokType(eName);
}
void Token::insertToken(const std::string &tokenStr, const std::string &originalNameStr, bool prepend)
{
Token *newToken;
2018-06-16 23:03:15 +02:00
if (mStr.empty())
newToken = this;
else
newToken = new Token(mTokensFrontBack);
newToken->str(tokenStr);
if (!originalNameStr.empty())
newToken->originalName(originalNameStr);
if (newToken != this) {
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
newToken->mImpl->mLineNumber = mImpl->mLineNumber;
newToken->mImpl->mFileIndex = mImpl->mFileIndex;
newToken->mImpl->mProgressValue = mImpl->mProgressValue;
if (prepend) {
if (this->previous()) {
newToken->previous(this->previous());
newToken->previous()->next(newToken);
} else if (mTokensFrontBack) {
mTokensFrontBack->front = newToken;
}
this->previous(newToken);
newToken->next(this);
} else {
if (this->next()) {
newToken->next(this->next());
newToken->next()->previous(newToken);
} else if (mTokensFrontBack) {
mTokensFrontBack->back = newToken;
}
this->next(newToken);
newToken->previous(this);
}
if (mImpl->mScopeInfo) {
// If the brace is immediately closed there is no point opening a new scope for it
if (tokenStr == "{") {
std::string nextScopeNameAddition = "";
// This might be the opening of a member function
Token *tok1 = newToken;
while (Token::Match(tok1->previous(), "const|volatile|final|override|&|&&|noexcept"))
tok1 = tok1->previous();
if (tok1 && tok1->previous() && tok1->strAt(-1) == ")") {
tok1 = tok1->linkAt(-1);
if (Token::Match(tok1->previous(), "throw|noexcept")) {
tok1 = tok1->previous();
while (Token::Match(tok1->previous(), "const|volatile|final|override|&|&&|noexcept"))
tok1 = tok1->previous();
if (tok1->strAt(-1) != ")")
return;
} else if (Token::Match(newToken->tokAt(-2), ":|, %name%")) {
tok1 = tok1->tokAt(-2);
if (tok1->strAt(-1) != ")")
return;
}
if (tok1->strAt(-1) == ">")
tok1 = tok1->previous()->findOpeningBracket();
if (tok1 && Token::Match(tok1->tokAt(-3), "%name% :: %name%")) {
tok1 = tok1->tokAt(-2);
std::string scope = tok1->strAt(-1);
while (Token::Match(tok1->tokAt(-2), ":: %name%")) {
scope = tok1->strAt(-3) + " :: " + scope;
tok1 = tok1->tokAt(-2);
}
if (!nextScopeNameAddition.empty() && !scope.empty()) nextScopeNameAddition += " :: ";
nextScopeNameAddition += scope;
}
}
// Or it might be a namespace/class/struct
if (Token::Match(newToken->previous(), "%name%|>")) {
Token* nameTok = newToken->previous();
while (nameTok && !Token::Match(nameTok, "namespace|class|struct|union %name% {|::|:|<")) {
nameTok = nameTok->previous();
}
if (nameTok) {
for (nameTok = nameTok->next(); nameTok && !Token::Match(nameTok, "{|:|<"); nameTok = nameTok->next()) {
nextScopeNameAddition.append(nameTok->str());
nextScopeNameAddition.append(" ");
}
if (nextScopeNameAddition.length() > 0) nextScopeNameAddition = nextScopeNameAddition.substr(0, nextScopeNameAddition.length() - 1);
}
}
// New scope is opening, record it here
std::shared_ptr<ScopeInfo2> newScopeInfo = std::make_shared<ScopeInfo2>(mImpl->mScopeInfo->name, nullptr, mImpl->mScopeInfo->usingNamespaces);
if (!newScopeInfo->name.empty() && !nextScopeNameAddition.empty()) newScopeInfo->name.append(" :: ");
newScopeInfo->name.append(nextScopeNameAddition);
nextScopeNameAddition = "";
newToken->scopeInfo(newScopeInfo);
} else if (tokenStr == "}") {
Token* matchingTok = newToken->previous();
int depth = 0;
while (matchingTok && (depth != 0 || !Token::simpleMatch(matchingTok, "{"))) {
if (Token::simpleMatch(matchingTok, "}")) depth++;
if (Token::simpleMatch(matchingTok, "{")) depth--;
matchingTok = matchingTok->previous();
}
if (matchingTok && matchingTok->previous()) {
newToken->mImpl->mScopeInfo = matchingTok->previous()->scopeInfo();
}
} else {
if (prepend && newToken->previous()) {
newToken->mImpl->mScopeInfo = newToken->previous()->scopeInfo();
} else {
newToken->mImpl->mScopeInfo = mImpl->mScopeInfo;
}
if (tokenStr == ";") {
const Token* statementStart;
for (statementStart = newToken; statementStart->previous() && !Token::Match(statementStart->previous(), ";|{"); statementStart = statementStart->previous());
if (Token::Match(statementStart, "using namespace %name% ::|;")) {
const Token * tok1 = statementStart->tokAt(2);
std::string nameSpace;
while (tok1 && tok1->str() != ";") {
if (!nameSpace.empty())
nameSpace += " ";
nameSpace += tok1->str();
tok1 = tok1->next();
}
mImpl->mScopeInfo->usingNamespaces.insert(nameSpace);
}
}
}
}
}
2008-12-18 22:28:57 +01:00
}
void Token::eraseTokens(Token *begin, const Token *end)
2008-12-18 22:28:57 +01:00
{
if (!begin || begin == end)
2008-12-18 22:28:57 +01:00
return;
2011-10-13 20:53:06 +02:00
while (begin->next() && begin->next() != end) {
2008-12-18 22:28:57 +01:00
begin->deleteNext();
}
}
void Token::createMutualLinks(Token *begin, Token *end)
{
2014-02-15 08:05:54 +01:00
assert(begin != nullptr);
assert(end != nullptr);
assert(begin != end);
begin->link(end);
end->link(begin);
}
void Token::printOut(const char *title) const
2008-12-18 22:28:57 +01:00
{
if (title && title[0])
std::cout << "\n### " << title << " ###\n";
std::cout << stringifyList(true, true, true, true, true, nullptr, nullptr) << std::endl;
}
void Token::printOut(const char *title, const std::vector<std::string> &fileNames) const
{
if (title && title[0])
std::cout << "\n### " << title << " ###\n";
std::cout << stringifyList(true, true, true, true, true, &fileNames, nullptr) << std::endl;
2008-12-18 22:28:57 +01:00
}
void Token::stringify(std::ostream& os, bool varid, bool attributes, bool macro) const
{
if (attributes) {
if (isUnsigned())
os << "unsigned ";
else if (isSigned())
os << "signed ";
if (isComplex())
os << "_Complex ";
if (isLong()) {
2018-06-16 16:40:02 +02:00
if (mTokType == eString || mTokType == eChar)
os << "L";
else
os << "long ";
}
}
if (macro && isExpandedMacro())
os << "$";
2018-06-16 23:03:15 +02:00
if (isName() && mStr.find(' ') != std::string::npos) {
for (std::size_t i = 0U; i < mStr.size(); ++i) {
if (mStr[i] != ' ')
os << mStr[i];
}
2018-06-16 23:03:15 +02:00
} else if (mStr[0] != '\"' || mStr.find('\0') == std::string::npos)
os << mStr;
else {
2018-06-16 23:03:15 +02:00
for (std::size_t i = 0U; i < mStr.size(); ++i) {
if (mStr[i] == '\0')
os << "\\0";
else
2018-06-16 23:03:15 +02:00
os << mStr[i];
}
}
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (varid && mImpl->mVarId != 0)
os << '@' << mImpl->mVarId;
}
std::string Token::stringifyList(bool varid, bool attributes, bool linenumbers, bool linebreaks, bool files, const std::vector<std::string>* fileNames, const Token* end) const
{
if (this == end)
return "";
std::ostringstream ret;
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
unsigned int lineNumber = mImpl->mLineNumber - (linenumbers ? 1U : 0U);
2019-07-17 16:28:47 +02:00
unsigned int fileIndex = files ? ~0U : mImpl->mFileIndex;
std::map<int, unsigned int> lineNumbers;
for (const Token *tok = this; tok != end; tok = tok->next()) {
bool fileChange = false;
2019-07-17 16:28:47 +02:00
if (tok->mImpl->mFileIndex != fileIndex) {
if (fileIndex != ~0U) {
lineNumbers[fileIndex] = tok->mImpl->mFileIndex;
}
2019-07-17 16:28:47 +02:00
fileIndex = tok->mImpl->mFileIndex;
if (files) {
ret << "\n\n##file ";
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (fileNames && fileNames->size() > tok->mImpl->mFileIndex)
ret << fileNames->at(tok->mImpl->mFileIndex);
else
2019-07-17 16:28:47 +02:00
ret << fileIndex;
2016-07-18 10:42:03 +02:00
ret << '\n';
}
2019-07-17 16:28:47 +02:00
lineNumber = lineNumbers[fileIndex];
fileChange = true;
}
if (linebreaks && (lineNumber != tok->linenr() || fileChange)) {
2019-07-17 16:28:47 +02:00
if (lineNumber+4 < tok->linenr() && fileIndex == tok->mImpl->mFileIndex) {
ret << '\n' << lineNumber+1 << ":\n|\n";
ret << tok->linenr()-1 << ":\n";
ret << tok->linenr() << ": ";
2016-07-18 10:42:03 +02:00
} else if (this == tok && linenumbers) {
ret << tok->linenr() << ": ";
} else {
while (lineNumber < tok->linenr()) {
++lineNumber;
ret << '\n';
if (linenumbers) {
ret << lineNumber << ':';
if (lineNumber == tok->linenr())
ret << ' ';
}
}
}
2010-04-09 21:40:37 +02:00
lineNumber = tok->linenr();
}
tok->stringify(ret, varid, attributes, attributes); // print token
if (tok->next() != end && (!linebreaks || (tok->next()->linenr() <= tok->linenr() && tok->next()->fileIndex() == tok->fileIndex())))
ret << ' ';
}
2016-07-18 10:42:03 +02:00
if (linebreaks && (files || linenumbers))
ret << '\n';
return ret.str();
}
std::string Token::stringifyList(const Token* end, bool attributes) const
{
return stringifyList(false, attributes, false, false, false, nullptr, end);
}
std::string Token::stringifyList(bool varid) const
{
return stringifyList(varid, false, true, true, true, nullptr, nullptr);
}
void Token::astOperand1(Token *tok)
{
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (mImpl->mAstOperand1)
mImpl->mAstOperand1->mImpl->mAstParent = nullptr;
// goto parent operator
if (tok) {
std::set<Token*> visitedParents;
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
while (tok->mImpl->mAstParent) {
if (!visitedParents.insert(tok->mImpl->mAstParent).second) // #6838/#6726/#8352 avoid hang on garbage code
throw InternalError(this, "Internal error. Token::astOperand1() cyclic dependency.");
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
tok = tok->mImpl->mAstParent;
}
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
tok->mImpl->mAstParent = this;
}
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
mImpl->mAstOperand1 = tok;
}
void Token::astOperand2(Token *tok)
{
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (mImpl->mAstOperand2)
mImpl->mAstOperand2->mImpl->mAstParent = nullptr;
// goto parent operator
if (tok) {
std::set<Token*> visitedParents;
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
while (tok->mImpl->mAstParent) {
//std::cout << tok << " -> " << tok->mAstParent ;
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (!visitedParents.insert(tok->mImpl->mAstParent).second) // #6838/#6726 avoid hang on garbage code
throw InternalError(this, "Internal error. Token::astOperand2() cyclic dependency.");
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
tok = tok->mImpl->mAstParent;
}
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
tok->mImpl->mAstParent = this;
}
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
mImpl->mAstOperand2 = tok;
}
static const Token* goToLeftParenthesis(const Token* start, const Token* end)
{
// move start to lpar in such expression: '(*it).x'
int par = 0;
for (const Token *tok = start; tok && tok != end; tok = tok->next()) {
if (tok->str() == "(")
++par;
else if (tok->str() == ")") {
if (par == 0)
start = tok->link();
else
--par;
}
}
return start;
}
static const Token* goToRightParenthesis(const Token* start, const Token* end)
{
// move end to rpar in such expression: '2>(x+1)'
int par = 0;
for (const Token *tok = end; tok && tok != start; tok = tok->previous()) {
if (tok->str() == ")")
++par;
else if (tok->str() == "(") {
if (par == 0)
end = tok->link();
else
--par;
}
}
return end;
}
std::pair<const Token *, const Token *> Token::findExpressionStartEndTokens() const
{
const Token * const top = this;
// find start node in AST tree
const Token *start = top;
while (start->astOperand1() &&
(start->astOperand2() || !start->isUnaryPreOp() || Token::simpleMatch(start, "( )") || start->str() == "{"))
start = start->astOperand1();
// find end node in AST tree
const Token *end = top;
while (end->astOperand1() && (end->astOperand2() || end->isUnaryPreOp())) {
// lambda..
if (end->str() == "[") {
const Token *lambdaEnd = findLambdaEndToken(end);
if (lambdaEnd) {
end = lambdaEnd;
break;
}
}
if (Token::Match(end,"(|[") &&
!(Token::Match(end, "( %type%") && !end->astOperand2())) {
end = end->link();
break;
}
end = end->astOperand2() ? end->astOperand2() : end->astOperand1();
}
// skip parentheses
start = goToLeftParenthesis(start, end);
end = goToRightParenthesis(start, end);
if (Token::simpleMatch(end, "{"))
end = end->link();
return std::pair<const Token *, const Token *>(start,end);
}
bool Token::isCalculation() const
{
if (!Token::Match(this, "%cop%|++|--"))
return false;
if (Token::Match(this, "*|&")) {
// dereference or address-of?
if (!this->astOperand2())
return false;
if (this->astOperand2()->str() == "[")
return false;
// type specification?
std::stack<const Token *> operands;
operands.push(this);
while (!operands.empty()) {
const Token *op = operands.top();
operands.pop();
if (op->isNumber() || op->varId() > 0)
return true;
if (op->astOperand1())
operands.push(op->astOperand1());
if (op->astOperand2())
operands.push(op->astOperand2());
else if (Token::Match(op, "*|&"))
return false;
}
// type specification => return false
return false;
}
return true;
}
bool Token::isUnaryPreOp() const
{
if (!astOperand1() || astOperand2())
return false;
if (!Token::Match(this, "++|--"))
return true;
2018-06-16 16:16:55 +02:00
const Token *tokbefore = mPrevious;
const Token *tokafter = mNext;
2015-10-26 13:29:47 +01:00
for (int distance = 1; distance < 10 && tokbefore; distance++) {
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (tokbefore == mImpl->mAstOperand1)
return false;
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (tokafter == mImpl->mAstOperand1)
return true;
2018-06-16 16:16:55 +02:00
tokbefore = tokbefore->mPrevious;
tokafter = tokafter->mPrevious;
}
return false; // <- guess
}
static std::string stringFromTokenRange(const Token* start, const Token* end)
{
std::ostringstream ret;
if (end)
2017-07-21 09:17:25 +02:00
end = end->next();
for (const Token *tok = start; tok && tok != end; tok = tok->next()) {
2017-07-21 09:17:25 +02:00
if (tok->isUnsigned())
ret << "unsigned ";
if (tok->isLong())
ret << (tok->isLiteral() ? "L" : "long ");
if (tok->originalName().empty() || tok->isUnsigned() || tok->isLong()) {
ret << tok->str();
} else
ret << tok->originalName();
if (Token::Match(tok, "%name%|%num% %name%|%num%"))
ret << ' ';
}
return ret.str();
}
std::string Token::expressionString() const
{
const auto tokens = findExpressionStartEndTokens();
return stringFromTokenRange(tokens.first, tokens.second);
}
static void astStringXml(const Token *tok, nonneg int indent, std::ostream &out)
{
const std::string strindent(indent, ' ');
2014-07-14 15:51:45 +02:00
out << strindent << "<token str=\"" << tok->str() << '\"';
if (tok->varId() > 0U)
out << " varId=\"" << MathLib::toString(tok->varId()) << '\"';
if (tok->variable())
out << " variable=\"" << tok->variable() << '\"';
if (tok->function())
out << " function=\"" << tok->function() << '\"';
if (!tok->values().empty())
out << " values=\"" << &tok->values() << '\"';
2014-07-14 15:51:45 +02:00
if (!tok->astOperand1() && !tok->astOperand2()) {
2014-07-14 15:51:45 +02:00
out << "/>" << std::endl;
}
2014-07-14 15:51:45 +02:00
else {
out << '>' << std::endl;
if (tok->astOperand1())
astStringXml(tok->astOperand1(), indent+2U, out);
if (tok->astOperand2())
astStringXml(tok->astOperand2(), indent+2U, out);
out << strindent << "</token>" << std::endl;
}
}
2014-07-14 15:51:45 +02:00
void Token::printAst(bool verbose, bool xml, std::ostream &out) const
{
2015-07-21 11:54:11 +02:00
std::set<const Token *> printed;
for (const Token *tok = this; tok; tok = tok->next()) {
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (!tok->mImpl->mAstParent && tok->mImpl->mAstOperand1) {
2015-07-21 11:54:11 +02:00
if (printed.empty() && !xml)
2014-07-14 15:51:45 +02:00
out << "\n\n##AST" << std::endl;
2015-07-21 11:54:11 +02:00
else if (printed.find(tok) != printed.end())
continue;
printed.insert(tok);
if (xml) {
out << "<ast scope=\"" << tok->scope() << "\" fileIndex=\"" << tok->fileIndex() << "\" linenr=\"" << tok->linenr()
<< "\" col=\"" << tok->col() << "\">" << std::endl;
2015-07-21 11:54:11 +02:00
astStringXml(tok, 2U, out);
2014-07-14 15:51:45 +02:00
out << "</ast>" << std::endl;
} else if (verbose)
Optimize astStringVerbose() for large arrays (#1815) Change the astStringVerbose() recursion to extend a string instead of returning one. This has the benefit that for tokens where the recursion runs deep (typically large arrays), the time savings can be substantial (see comments on benchmarks further down). The reason is that previously, for each token, the astString of its operands was constructed, and then appended to this tokens astString. This led to a lot of unnecessary string copying (and with that allocations). Instead, by passing the string by reference, the number of temporary strings is greatly reduced. Another way of seeing it is that previously, the string was constructed from end to beginning, but now it is constructed from the beginning to end. There was no notable speedup by preallocating the entire string using string::reserve() (at least not on Linux). To benchmark, the changes and master were tested on Linux using the commands: make time cppcheck --debug --verbose $file >/dev/null i.e., the cppcheck binary was compiled with the settings in the Makefile. Printing the output to screen or file will of course take longer time. In Trac ticket #8355 which triggered this change, an example file from the Wine repository was attached. Running the above cppcheck on master took 24 minutes and with the changes in this commmit, took 22 seconds. Another test made was on lib/tokenlist.cpp in the cppcheck repo, which is more "normal" file. On that file there was no measurable time difference. A synthetic benchmark was generated to illustrate the effects on dumping the ast for arrays of different sizes. The generate code looked as follows: const int array[] = {...}; with different number of elements. The results are as follows (times are in seconds): N master optimized 10 0.1 0.1 100 0.1 0.1 1000 2.8 0.7 2000 19 1.8 3000 53 3.8 5000 350 10 10000 3215 38 As we can see, for small arrays, there is no time difference, but for large arrays the time savings are substantial.
2019-04-30 13:35:48 +02:00
out << tok->astStringVerbose() << std::endl;
else
2015-07-21 11:54:11 +02:00
out << tok->astString(" ") << std::endl;
2014-01-18 09:58:32 +01:00
if (tok->str() == "(")
tok = tok->link();
}
}
}
2014-01-18 09:58:32 +01:00
static void indent(std::string &str, const nonneg int indent1, const nonneg int indent2)
{
2019-07-13 16:06:24 +02:00
for (int i = 0; i < indent1; ++i)
Optimize astStringVerbose() for large arrays (#1815) Change the astStringVerbose() recursion to extend a string instead of returning one. This has the benefit that for tokens where the recursion runs deep (typically large arrays), the time savings can be substantial (see comments on benchmarks further down). The reason is that previously, for each token, the astString of its operands was constructed, and then appended to this tokens astString. This led to a lot of unnecessary string copying (and with that allocations). Instead, by passing the string by reference, the number of temporary strings is greatly reduced. Another way of seeing it is that previously, the string was constructed from end to beginning, but now it is constructed from the beginning to end. There was no notable speedup by preallocating the entire string using string::reserve() (at least not on Linux). To benchmark, the changes and master were tested on Linux using the commands: make time cppcheck --debug --verbose $file >/dev/null i.e., the cppcheck binary was compiled with the settings in the Makefile. Printing the output to screen or file will of course take longer time. In Trac ticket #8355 which triggered this change, an example file from the Wine repository was attached. Running the above cppcheck on master took 24 minutes and with the changes in this commmit, took 22 seconds. Another test made was on lib/tokenlist.cpp in the cppcheck repo, which is more "normal" file. On that file there was no measurable time difference. A synthetic benchmark was generated to illustrate the effects on dumping the ast for arrays of different sizes. The generate code looked as follows: const int array[] = {...}; with different number of elements. The results are as follows (times are in seconds): N master optimized 10 0.1 0.1 100 0.1 0.1 1000 2.8 0.7 2000 19 1.8 3000 53 3.8 5000 350 10 10000 3215 38 As we can see, for small arrays, there is no time difference, but for large arrays the time savings are substantial.
2019-04-30 13:35:48 +02:00
str += ' ';
2019-07-13 16:06:24 +02:00
for (int i = indent1; i < indent2; i += 2)
Optimize astStringVerbose() for large arrays (#1815) Change the astStringVerbose() recursion to extend a string instead of returning one. This has the benefit that for tokens where the recursion runs deep (typically large arrays), the time savings can be substantial (see comments on benchmarks further down). The reason is that previously, for each token, the astString of its operands was constructed, and then appended to this tokens astString. This led to a lot of unnecessary string copying (and with that allocations). Instead, by passing the string by reference, the number of temporary strings is greatly reduced. Another way of seeing it is that previously, the string was constructed from end to beginning, but now it is constructed from the beginning to end. There was no notable speedup by preallocating the entire string using string::reserve() (at least not on Linux). To benchmark, the changes and master were tested on Linux using the commands: make time cppcheck --debug --verbose $file >/dev/null i.e., the cppcheck binary was compiled with the settings in the Makefile. Printing the output to screen or file will of course take longer time. In Trac ticket #8355 which triggered this change, an example file from the Wine repository was attached. Running the above cppcheck on master took 24 minutes and with the changes in this commmit, took 22 seconds. Another test made was on lib/tokenlist.cpp in the cppcheck repo, which is more "normal" file. On that file there was no measurable time difference. A synthetic benchmark was generated to illustrate the effects on dumping the ast for arrays of different sizes. The generate code looked as follows: const int array[] = {...}; with different number of elements. The results are as follows (times are in seconds): N master optimized 10 0.1 0.1 100 0.1 0.1 1000 2.8 0.7 2000 19 1.8 3000 53 3.8 5000 350 10 10000 3215 38 As we can see, for small arrays, there is no time difference, but for large arrays the time savings are substantial.
2019-04-30 13:35:48 +02:00
str += "| ";
}
void Token::astStringVerboseRecursive(std::string& ret, const nonneg int indent1, const nonneg int indent2) const
{
if (isExpandedMacro())
2016-01-01 13:54:07 +01:00
ret += '$';
2018-06-16 23:03:15 +02:00
ret += mStr;
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (mImpl->mValueType)
ret += " \'" + mImpl->mValueType->str() + '\'';
2016-01-01 13:54:07 +01:00
ret += '\n';
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (mImpl->mAstOperand1) {
2019-07-13 16:06:24 +02:00
int i1 = indent1, i2 = indent2 + 2;
Optimize astStringVerbose() for large arrays (#1815) Change the astStringVerbose() recursion to extend a string instead of returning one. This has the benefit that for tokens where the recursion runs deep (typically large arrays), the time savings can be substantial (see comments on benchmarks further down). The reason is that previously, for each token, the astString of its operands was constructed, and then appended to this tokens astString. This led to a lot of unnecessary string copying (and with that allocations). Instead, by passing the string by reference, the number of temporary strings is greatly reduced. Another way of seeing it is that previously, the string was constructed from end to beginning, but now it is constructed from the beginning to end. There was no notable speedup by preallocating the entire string using string::reserve() (at least not on Linux). To benchmark, the changes and master were tested on Linux using the commands: make time cppcheck --debug --verbose $file >/dev/null i.e., the cppcheck binary was compiled with the settings in the Makefile. Printing the output to screen or file will of course take longer time. In Trac ticket #8355 which triggered this change, an example file from the Wine repository was attached. Running the above cppcheck on master took 24 minutes and with the changes in this commmit, took 22 seconds. Another test made was on lib/tokenlist.cpp in the cppcheck repo, which is more "normal" file. On that file there was no measurable time difference. A synthetic benchmark was generated to illustrate the effects on dumping the ast for arrays of different sizes. The generate code looked as follows: const int array[] = {...}; with different number of elements. The results are as follows (times are in seconds): N master optimized 10 0.1 0.1 100 0.1 0.1 1000 2.8 0.7 2000 19 1.8 3000 53 3.8 5000 350 10 10000 3215 38 As we can see, for small arrays, there is no time difference, but for large arrays the time savings are substantial.
2019-04-30 13:35:48 +02:00
if (indent1 == indent2 && !mImpl->mAstOperand2)
i1 += 2;
Optimize astStringVerbose() for large arrays (#1815) Change the astStringVerbose() recursion to extend a string instead of returning one. This has the benefit that for tokens where the recursion runs deep (typically large arrays), the time savings can be substantial (see comments on benchmarks further down). The reason is that previously, for each token, the astString of its operands was constructed, and then appended to this tokens astString. This led to a lot of unnecessary string copying (and with that allocations). Instead, by passing the string by reference, the number of temporary strings is greatly reduced. Another way of seeing it is that previously, the string was constructed from end to beginning, but now it is constructed from the beginning to end. There was no notable speedup by preallocating the entire string using string::reserve() (at least not on Linux). To benchmark, the changes and master were tested on Linux using the commands: make time cppcheck --debug --verbose $file >/dev/null i.e., the cppcheck binary was compiled with the settings in the Makefile. Printing the output to screen or file will of course take longer time. In Trac ticket #8355 which triggered this change, an example file from the Wine repository was attached. Running the above cppcheck on master took 24 minutes and with the changes in this commmit, took 22 seconds. Another test made was on lib/tokenlist.cpp in the cppcheck repo, which is more "normal" file. On that file there was no measurable time difference. A synthetic benchmark was generated to illustrate the effects on dumping the ast for arrays of different sizes. The generate code looked as follows: const int array[] = {...}; with different number of elements. The results are as follows (times are in seconds): N master optimized 10 0.1 0.1 100 0.1 0.1 1000 2.8 0.7 2000 19 1.8 3000 53 3.8 5000 350 10 10000 3215 38 As we can see, for small arrays, there is no time difference, but for large arrays the time savings are substantial.
2019-04-30 13:35:48 +02:00
indent(ret, indent1, indent2);
ret += mImpl->mAstOperand2 ? "|-" : "`-";
mImpl->mAstOperand1->astStringVerboseRecursive(ret, i1, i2);
}
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (mImpl->mAstOperand2) {
2019-07-13 16:06:24 +02:00
int i1 = indent1, i2 = indent2 + 2;
Optimize astStringVerbose() for large arrays (#1815) Change the astStringVerbose() recursion to extend a string instead of returning one. This has the benefit that for tokens where the recursion runs deep (typically large arrays), the time savings can be substantial (see comments on benchmarks further down). The reason is that previously, for each token, the astString of its operands was constructed, and then appended to this tokens astString. This led to a lot of unnecessary string copying (and with that allocations). Instead, by passing the string by reference, the number of temporary strings is greatly reduced. Another way of seeing it is that previously, the string was constructed from end to beginning, but now it is constructed from the beginning to end. There was no notable speedup by preallocating the entire string using string::reserve() (at least not on Linux). To benchmark, the changes and master were tested on Linux using the commands: make time cppcheck --debug --verbose $file >/dev/null i.e., the cppcheck binary was compiled with the settings in the Makefile. Printing the output to screen or file will of course take longer time. In Trac ticket #8355 which triggered this change, an example file from the Wine repository was attached. Running the above cppcheck on master took 24 minutes and with the changes in this commmit, took 22 seconds. Another test made was on lib/tokenlist.cpp in the cppcheck repo, which is more "normal" file. On that file there was no measurable time difference. A synthetic benchmark was generated to illustrate the effects on dumping the ast for arrays of different sizes. The generate code looked as follows: const int array[] = {...}; with different number of elements. The results are as follows (times are in seconds): N master optimized 10 0.1 0.1 100 0.1 0.1 1000 2.8 0.7 2000 19 1.8 3000 53 3.8 5000 350 10 10000 3215 38 As we can see, for small arrays, there is no time difference, but for large arrays the time savings are substantial.
2019-04-30 13:35:48 +02:00
if (indent1 == indent2)
i1 += 2;
Optimize astStringVerbose() for large arrays (#1815) Change the astStringVerbose() recursion to extend a string instead of returning one. This has the benefit that for tokens where the recursion runs deep (typically large arrays), the time savings can be substantial (see comments on benchmarks further down). The reason is that previously, for each token, the astString of its operands was constructed, and then appended to this tokens astString. This led to a lot of unnecessary string copying (and with that allocations). Instead, by passing the string by reference, the number of temporary strings is greatly reduced. Another way of seeing it is that previously, the string was constructed from end to beginning, but now it is constructed from the beginning to end. There was no notable speedup by preallocating the entire string using string::reserve() (at least not on Linux). To benchmark, the changes and master were tested on Linux using the commands: make time cppcheck --debug --verbose $file >/dev/null i.e., the cppcheck binary was compiled with the settings in the Makefile. Printing the output to screen or file will of course take longer time. In Trac ticket #8355 which triggered this change, an example file from the Wine repository was attached. Running the above cppcheck on master took 24 minutes and with the changes in this commmit, took 22 seconds. Another test made was on lib/tokenlist.cpp in the cppcheck repo, which is more "normal" file. On that file there was no measurable time difference. A synthetic benchmark was generated to illustrate the effects on dumping the ast for arrays of different sizes. The generate code looked as follows: const int array[] = {...}; with different number of elements. The results are as follows (times are in seconds): N master optimized 10 0.1 0.1 100 0.1 0.1 1000 2.8 0.7 2000 19 1.8 3000 53 3.8 5000 350 10 10000 3215 38 As we can see, for small arrays, there is no time difference, but for large arrays the time savings are substantial.
2019-04-30 13:35:48 +02:00
indent(ret, indent1, indent2);
ret += "`-";
mImpl->mAstOperand2->astStringVerboseRecursive(ret, i1, i2);
}
Optimize astStringVerbose() for large arrays (#1815) Change the astStringVerbose() recursion to extend a string instead of returning one. This has the benefit that for tokens where the recursion runs deep (typically large arrays), the time savings can be substantial (see comments on benchmarks further down). The reason is that previously, for each token, the astString of its operands was constructed, and then appended to this tokens astString. This led to a lot of unnecessary string copying (and with that allocations). Instead, by passing the string by reference, the number of temporary strings is greatly reduced. Another way of seeing it is that previously, the string was constructed from end to beginning, but now it is constructed from the beginning to end. There was no notable speedup by preallocating the entire string using string::reserve() (at least not on Linux). To benchmark, the changes and master were tested on Linux using the commands: make time cppcheck --debug --verbose $file >/dev/null i.e., the cppcheck binary was compiled with the settings in the Makefile. Printing the output to screen or file will of course take longer time. In Trac ticket #8355 which triggered this change, an example file from the Wine repository was attached. Running the above cppcheck on master took 24 minutes and with the changes in this commmit, took 22 seconds. Another test made was on lib/tokenlist.cpp in the cppcheck repo, which is more "normal" file. On that file there was no measurable time difference. A synthetic benchmark was generated to illustrate the effects on dumping the ast for arrays of different sizes. The generate code looked as follows: const int array[] = {...}; with different number of elements. The results are as follows (times are in seconds): N master optimized 10 0.1 0.1 100 0.1 0.1 1000 2.8 0.7 2000 19 1.8 3000 53 3.8 5000 350 10 10000 3215 38 As we can see, for small arrays, there is no time difference, but for large arrays the time savings are substantial.
2019-04-30 13:35:48 +02:00
}
std::string Token::astStringVerbose() const
{
std::string ret;
astStringVerboseRecursive(ret);
return ret;
}
2014-07-14 15:51:45 +02:00
void Token::printValueFlow(bool xml, std::ostream &out) const
2014-01-18 09:58:32 +01:00
{
2019-07-13 16:06:24 +02:00
int line = 0;
2014-07-14 15:51:45 +02:00
if (xml)
out << " <valueflow>" << std::endl;
else
out << "\n\n##Value flow" << std::endl;
2014-01-18 09:58:32 +01:00
for (const Token *tok = this; tok; tok = tok->next()) {
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (!tok->mImpl->mValues)
2014-01-18 09:58:32 +01:00
continue;
2014-07-14 15:51:45 +02:00
if (xml)
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
out << " <values id=\"" << tok->mImpl->mValues << "\">" << std::endl;
2014-07-14 15:51:45 +02:00
else if (line != tok->linenr())
out << "Line " << tok->linenr() << std::endl;
2014-01-18 09:58:32 +01:00
line = tok->linenr();
if (!xml) {
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
out << " " << tok->str() << (tok->mImpl->mValues->front().isKnown() ? " always " : " possible ");
if (tok->mImpl->mValues->size() > 1U)
out << '{';
}
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
for (const ValueFlow::Value &value : *tok->mImpl->mValues) {
2014-07-14 15:51:45 +02:00
if (xml) {
out << " <value ";
switch (value.valueType) {
2016-11-13 22:59:56 +01:00
case ValueFlow::Value::INT:
if (tok->valueType() && tok->valueType()->sign == ValueType::UNSIGNED)
out << "intvalue=\"" << (MathLib::biguint)value.intvalue << '\"';
else
out << "intvalue=\"" << value.intvalue << '\"';
2016-11-13 22:59:56 +01:00
break;
case ValueFlow::Value::TOK:
out << "tokvalue=\"" << value.tokvalue << '\"';
2016-11-13 22:59:56 +01:00
break;
case ValueFlow::Value::FLOAT:
out << "floatvalue=\"" << value.floatValue << '\"';
2016-11-13 22:59:56 +01:00
break;
case ValueFlow::Value::MOVED:
out << "movedvalue=\"" << ValueFlow::Value::toString(value.moveKind) << '\"';
break;
case ValueFlow::Value::UNINIT:
out << "uninit=\"1\"";
break;
case ValueFlow::Value::BUFFER_SIZE:
out << "buffer-size=\"" << value.intvalue << "\"";
break;
case ValueFlow::Value::CONTAINER_SIZE:
out << "container-size=\"" << value.intvalue << '\"';
break;
2018-11-14 06:14:04 +01:00
case ValueFlow::Value::LIFETIME:
out << "lifetime=\"" << value.tokvalue << '\"';
break;
2016-11-13 22:59:56 +01:00
}
if (value.condition)
out << " condition-line=\"" << value.condition->linenr() << '\"';
if (value.isKnown())
out << " known=\"true\"";
else if (value.isPossible())
out << " possible=\"true\"";
else if (value.isInconclusive())
out << " inconclusive=\"true\"";
2014-07-14 15:51:45 +02:00
out << "/>" << std::endl;
}
else {
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (&value != &tok->mImpl->mValues->front())
out << ",";
switch (value.valueType) {
2016-11-13 22:59:56 +01:00
case ValueFlow::Value::INT:
if (tok->valueType() && tok->valueType()->sign == ValueType::UNSIGNED)
out << (MathLib::biguint)value.intvalue;
else
out << value.intvalue;
2016-11-13 22:59:56 +01:00
break;
case ValueFlow::Value::TOK:
out << value.tokvalue->str();
2016-11-13 22:59:56 +01:00
break;
case ValueFlow::Value::FLOAT:
out << value.floatValue;
2016-11-13 22:59:56 +01:00
break;
case ValueFlow::Value::MOVED:
out << ValueFlow::Value::toString(value.moveKind);
break;
case ValueFlow::Value::UNINIT:
out << "Uninit";
break;
case ValueFlow::Value::BUFFER_SIZE:
case ValueFlow::Value::CONTAINER_SIZE:
out << "size=" << value.intvalue;
break;
2018-11-14 06:14:04 +01:00
case ValueFlow::Value::LIFETIME:
out << "lifetime=" << value.tokvalue->str();
break;
2016-11-13 22:59:56 +01:00
}
2014-07-14 15:51:45 +02:00
}
2014-01-18 09:58:32 +01:00
}
2014-07-14 15:51:45 +02:00
if (xml)
out << " </values>" << std::endl;
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
else if (tok->mImpl->mValues->size() > 1U)
out << '}' << std::endl;
2014-07-14 15:51:45 +02:00
else
out << std::endl;
2014-01-18 09:58:32 +01:00
}
2014-07-14 15:51:45 +02:00
if (xml)
out << " </valueflow>" << std::endl;
2014-01-18 09:58:32 +01:00
}
const ValueFlow::Value * Token::getValueLE(const MathLib::bigint val, const Settings *settings) const
{
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (!mImpl->mValues)
return nullptr;
const ValueFlow::Value *ret = nullptr;
std::list<ValueFlow::Value>::const_iterator it;
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
for (it = mImpl->mValues->begin(); it != mImpl->mValues->end(); ++it) {
2016-11-13 22:33:39 +01:00
if (it->isIntValue() && it->intvalue <= val) {
if (!ret || ret->isInconclusive() || (ret->condition && !it->isInconclusive()))
ret = &(*it);
if (!ret->isInconclusive() && !ret->condition)
break;
}
}
if (settings && ret) {
if (ret->isInconclusive() && !settings->inconclusive)
return nullptr;
if (ret->condition && !settings->isEnabled(Settings::WARNING))
return nullptr;
}
return ret;
}
const ValueFlow::Value * Token::getValueGE(const MathLib::bigint val, const Settings *settings) const
{
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (!mImpl->mValues)
return nullptr;
const ValueFlow::Value *ret = nullptr;
std::list<ValueFlow::Value>::const_iterator it;
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
for (it = mImpl->mValues->begin(); it != mImpl->mValues->end(); ++it) {
2016-11-13 22:33:39 +01:00
if (it->isIntValue() && it->intvalue >= val) {
if (!ret || ret->isInconclusive() || (ret->condition && !it->isInconclusive()))
ret = &(*it);
if (!ret->isInconclusive() && !ret->condition)
break;
}
}
if (settings && ret) {
if (ret->isInconclusive() && !settings->inconclusive)
return nullptr;
if (ret->condition && !settings->isEnabled(Settings::WARNING))
return nullptr;
}
return ret;
}
const ValueFlow::Value * Token::getInvalidValue(const Token *ftok, nonneg int argnr, const Settings *settings) const
{
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (!mImpl->mValues || !settings)
return nullptr;
const ValueFlow::Value *ret = nullptr;
std::list<ValueFlow::Value>::const_iterator it;
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
for (it = mImpl->mValues->begin(); it != mImpl->mValues->end(); ++it) {
2018-07-15 23:05:48 +02:00
if ((it->isIntValue() && !settings->library.isIntArgValid(ftok, argnr, it->intvalue)) ||
(it->isFloatValue() && !settings->library.isFloatArgValid(ftok, argnr, it->floatValue))) {
if (!ret || ret->isInconclusive() || (ret->condition && !it->isInconclusive()))
ret = &(*it);
if (!ret->isInconclusive() && !ret->condition)
break;
}
}
if (ret) {
if (ret->isInconclusive() && !settings->inconclusive)
return nullptr;
if (ret->condition && !settings->isEnabled(Settings::WARNING))
return nullptr;
}
return ret;
}
const Token *Token::getValueTokenMinStrSize() const
{
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (!mImpl->mValues)
return nullptr;
const Token *ret = nullptr;
2019-07-13 16:06:24 +02:00
int minsize = INT_MAX;
std::list<ValueFlow::Value>::const_iterator it;
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
for (it = mImpl->mValues->begin(); it != mImpl->mValues->end(); ++it) {
2016-11-13 22:33:39 +01:00
if (it->isTokValue() && it->tokvalue && it->tokvalue->tokType() == Token::eString) {
2019-07-13 16:06:24 +02:00
const int size = getStrSize(it->tokvalue);
if (!ret || size < minsize) {
minsize = size;
ret = it->tokvalue;
}
}
}
return ret;
}
const Token *Token::getValueTokenMaxStrLength() const
{
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (!mImpl->mValues)
return nullptr;
const Token *ret = nullptr;
2019-07-13 16:06:24 +02:00
int maxlength = 0;
std::list<ValueFlow::Value>::const_iterator it;
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
for (it = mImpl->mValues->begin(); it != mImpl->mValues->end(); ++it) {
2016-11-13 22:33:39 +01:00
if (it->isTokValue() && it->tokvalue && it->tokvalue->tokType() == Token::eString) {
2019-07-13 16:06:24 +02:00
const int length = getStrLength(it->tokvalue);
if (!ret || length > maxlength) {
maxlength = length;
ret = it->tokvalue;
}
}
}
return ret;
}
static const Scope *getfunctionscope(const Scope *s)
{
while (s && s->type != Scope::eFunction)
s = s->nestedIn;
return s;
}
const Token *Token::getValueTokenDeadPointer() const
{
const Scope * const functionscope = getfunctionscope(this->scope());
std::list<ValueFlow::Value>::const_iterator it;
for (it = values().begin(); it != values().end(); ++it) {
// Is this a pointer alias?
2016-11-13 22:33:39 +01:00
if (!it->isTokValue() || (it->tokvalue && it->tokvalue->str() != "&"))
continue;
// Get variable
const Token *vartok = it->tokvalue->astOperand1();
if (!vartok || !vartok->isName() || !vartok->variable())
continue;
const Variable * const var = vartok->variable();
if (var->isStatic() || var->isReference())
continue;
if (!var->scope())
return nullptr; // #6804
2015-03-11 20:25:27 +01:00
if (var->scope()->type == Scope::eUnion && var->scope()->nestedIn == this->scope())
continue;
// variable must be in same function (not in subfunction)
if (functionscope != getfunctionscope(var->scope()))
continue;
// Is variable defined in this scope or upper scope?
const Scope *s = this->scope();
while ((s != nullptr) && (s != var->scope()))
s = s->nestedIn;
if (!s)
return it->tokvalue;
}
return nullptr;
}
bool Token::addValue(const ValueFlow::Value &value)
{
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (value.isKnown() && mImpl->mValues) {
// Clear all other values of the same type since value is known
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
mImpl->mValues->remove_if([&](const ValueFlow::Value & x) {
return x.valueType == value.valueType;
});
}
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (mImpl->mValues) {
// Don't handle more than 10 values for performance reasons
// TODO: add setting?
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (mImpl->mValues->size() >= 10U)
return false;
// if value already exists, don't add it again
std::list<ValueFlow::Value>::iterator it;
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
for (it = mImpl->mValues->begin(); it != mImpl->mValues->end(); ++it) {
// different types => continue
if (it->valueType != value.valueType)
continue;
// different value => continue
bool differentValue = true;
switch (it->valueType) {
case ValueFlow::Value::ValueType::INT:
case ValueFlow::Value::ValueType::CONTAINER_SIZE:
case ValueFlow::Value::ValueType::BUFFER_SIZE:
differentValue = (it->intvalue != value.intvalue);
break;
case ValueFlow::Value::ValueType::TOK:
case ValueFlow::Value::ValueType::LIFETIME:
differentValue = (it->tokvalue != value.tokvalue);
break;
case ValueFlow::Value::ValueType::FLOAT:
// TODO: Write some better comparison
differentValue = (it->floatValue > value.floatValue || it->floatValue < value.floatValue);
break;
case ValueFlow::Value::ValueType::MOVED:
differentValue = (it->moveKind != value.moveKind);
break;
case ValueFlow::Value::ValueType::UNINIT:
differentValue = false;
break;
}
if (differentValue)
continue;
if ((value.isTokValue() || value.isLifetimeValue()) && (it->tokvalue != value.tokvalue) && (it->tokvalue->str() != value.tokvalue->str()))
continue;
// same value, but old value is inconclusive so replace it
if (it->isInconclusive() && !value.isInconclusive()) {
*it = value;
if (it->varId == 0)
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
it->varId = mImpl->mVarId;
break;
}
// Same value already exists, don't add new value
return false;
}
// Add value
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (it == mImpl->mValues->end()) {
ValueFlow::Value v(value);
if (v.varId == 0)
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
v.varId = mImpl->mVarId;
2018-11-14 19:10:52 +01:00
if (v.isKnown() && v.isIntValue())
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
mImpl->mValues->push_front(v);
else
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
mImpl->mValues->push_back(v);
}
} else {
ValueFlow::Value v(value);
if (v.varId == 0)
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
v.varId = mImpl->mVarId;
mImpl->mValues = new std::list<ValueFlow::Value>(1, v);
}
return true;
}
void Token::assignProgressValues(Token *tok)
{
2019-07-13 16:06:24 +02:00
int total_count = 0;
for (Token *tok2 = tok; tok2; tok2 = tok2->next())
++total_count;
2019-07-13 16:06:24 +02:00
int count = 0;
for (Token *tok2 = tok; tok2; tok2 = tok2->next())
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
tok2->mImpl->mProgressValue = count++ * 100 / total_count;
}
void Token::assignIndexes()
{
2019-07-13 16:06:24 +02:00
int index = (mPrevious ? mPrevious->mImpl->mIndex : 0) + 1;
for (Token *tok = this; tok; tok = tok->next())
tok->mImpl->mIndex = index++;
}
void Token::setValueType(ValueType *vt)
{
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
if (vt != mImpl->mValueType) {
delete mImpl->mValueType;
mImpl->mValueType = vt;
}
}
void Token::type(const ::Type *t)
{
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
mImpl->mType = t;
if (t) {
2017-10-15 01:27:47 +02:00
tokType(eType);
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
isEnumType(mImpl->mType->isEnumType());
2018-06-16 16:40:02 +02:00
} else if (mTokType == eType)
2017-10-15 01:27:47 +02:00
tokType(eName);
}
const ::Type *Token::typeOf(const Token *tok)
{
if (Token::simpleMatch(tok, "return")) {
const Scope *scope = tok->scope();
if (!scope)
return nullptr;
const Function *function = scope->function;
if (!function)
return nullptr;
return function->retType;
} else if (Token::Match(tok, "%type%")) {
return tok->type();
} else if (Token::Match(tok, "%var%")) {
const Variable *var = tok->variable();
if (!var)
return nullptr;
return var->type();
} else if (Token::Match(tok, "%name%")) {
const Function *function = tok->function();
if (!function)
return nullptr;
return function->retType;
} else if (Token::simpleMatch(tok, "=")) {
return Token::typeOf(tok->astOperand1());
} else if (Token::simpleMatch(tok, ".")) {
return Token::typeOf(tok->astOperand2());
}
return nullptr;
}
std::pair<const Token*, const Token*> Token::typeDecl(const Token * tok)
{
if (Token::simpleMatch(tok, "return")) {
const Scope *scope = tok->scope();
if (!scope)
return {};
const Function *function = scope->function;
if (!function)
return {};
return {function->retDef, function->returnDefEnd()};
} else if (Token::Match(tok, "%type%")) {
return {tok, tok->next()};
} else if (Token::Match(tok, "%var%")) {
const Variable *var = tok->variable();
if (!var)
return {};
if (!var->typeStartToken() || !var->typeEndToken())
return {};
return {var->typeStartToken(), var->typeEndToken()->next()};
} else if (Token::Match(tok, "%name%")) {
const Function *function = tok->function();
if (!function)
return {};
return {function->retDef, function->returnDefEnd()};
} else if (Token::simpleMatch(tok, "=")) {
return Token::typeDecl(tok->astOperand1());
} else if (Token::simpleMatch(tok, ".")) {
return Token::typeDecl(tok->astOperand2());
} else {
const ::Type * t = typeOf(tok);
if (!t || !t->classDef)
return {};
return {t->classDef->next(), t->classDef->tokAt(2)};
}
}
std::string Token::typeStr(const Token* tok)
{
if (tok->valueType()) {
const ValueType * vt = tok->valueType();
std::string ret = vt->str();
if (!ret.empty())
2019-06-02 13:29:20 +02:00
return ret;
}
std::pair<const Token*, const Token*> r = Token::typeDecl(tok);
if (!r.first || !r.second)
return "";
return r.first->stringifyList(r.second, false);
}
void Token::scopeInfo(std::shared_ptr<ScopeInfo2> newScopeInfo)
{
mImpl->mScopeInfo = newScopeInfo;
}
std::shared_ptr<ScopeInfo2> Token::scopeInfo() const
{
return mImpl->mScopeInfo;
}
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
TokenImpl::~TokenImpl()
{
delete mOriginalName;
delete mValueType;
delete mValues;
2018-12-21 13:54:59 +01:00
for (auto templateSimplifierPointer : mTemplateSimplifierPointers) {
templateSimplifierPointer->token(nullptr);
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
}
while (mCppcheckAttributes) {
struct CppcheckAttributes *c = mCppcheckAttributes;
mCppcheckAttributes = mCppcheckAttributes->next;
delete c;
}
}
void TokenImpl::setCppcheckAttribute(TokenImpl::CppcheckAttributes::Type type, MathLib::bigint value)
{
struct CppcheckAttributes *attr = mCppcheckAttributes;
while (attr && attr->type != type)
attr = attr->next;
if (attr)
attr->value = value;
else {
attr = new CppcheckAttributes;
attr->type = type;
attr->value = value;
attr->next = mCppcheckAttributes;
mCppcheckAttributes = attr;
}
}
bool TokenImpl::getCppcheckAttribute(TokenImpl::CppcheckAttributes::Type type, MathLib::bigint *value) const
{
struct CppcheckAttributes *attr = mCppcheckAttributes;
while (attr && attr->type != type)
attr = attr->next;
if (attr)
*value = attr->value;
return attr != nullptr;
Several fairly significant optimisations (#1518) * Code changes for Token::mImpl optimisation * Added new TokenImpl optimisation Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation. In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%. * Several optimisations Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it. Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations. Moved template removal to the end of the list as this reduces redundant iteration and saves time. * Added tok argument to simplifyCalculations This means callers can avoid unnecessary work if they know which tokens have already been simplified. Passing nullptr indicates the original behaviour (starting from the front of the list). * Removed mention of member from another change * Re-added and optimised some code deleted in error Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches. Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators. * Changed mTemplateInstantiations to a vector This is an optimisation that makes repeated linear searches of this collection significantly faster. Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized. * A cleaner optimisation to removing template tokens This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified. * Reverted vector to list Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members. * Added member for template simplifier pointer This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches. * Turned loop inside out This means we only have to iterate through the std::list once. std::list is very expensive to iterate through. * Latest code from danmar and fixed optimisations In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list. Previous optimisations to this weren't very effective and broke some edge cases. * Added changes from danmar Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently. * Tokens can have many templateSimplifierPointers * templateSimplifierPointers must be kept in sync
2018-12-21 13:51:45 +01:00
}