walkero
/
cppcheck
1
0
Fork 0
Code Issues Pull Requests Projects Releases 1 Wiki Activity

Several fairly significant optimisations (#1518) 

* Code changes for Token::mImpl optimisation

* Added new TokenImpl optimisation

Moving members to the TokenImpl struct reduces the size of the Token class, which is a fairly significant optimisation.  In my testing on Windows with 32-bit Release-PCRE, this change reduced the size of the Token class from 108 bits to 52 bits and reduced run-time of my test case by around 20%.

* Several optimisations

Deleted some code that ran very slowly and did nothing, as there is no need to change a Token's string to null if you are about to delete it.
Added a frontToken to simplifyCalculations to reduce the amount of work it has to do on already-simplified calculations.
Moved template removal to the end of the list as this reduces redundant iteration and saves time.

* Added tok argument to simplifyCalculations

This means callers can avoid unnecessary work if they know which tokens have already been simplified.  Passing nullptr indicates the original behaviour (starting from the front of the list).

* Removed mention of member from another change

* Re-added and optimised some code deleted in error

Changing mTemplateInstantiations to a vector avoids the high cost of doing repeated linear searches.  Changing how the code iterates through the array was necessary because the vector can be resized at several points during the loop, which breaks existing references and iterators.

* Changed mTemplateInstantiations to a vector

This is an optimisation that makes repeated linear searches of this collection significantly faster. 
Also added a copy constructor to TokenAndName so code can make copies of these objects to keep a reference if a vector gets resized.

* A cleaner optimisation to removing template tokens

This reverts the previous change to made mInstantiatedTemplates a vector and the iterator changes to support this, and makes mTypesUsedInTemplateInstantiation so the eraseTokens logic can be unified.

* Reverted vector to list

Also made mTypesUsedInTemplateInstantiation a vector of TokenAndName objects so it can share the same logic as the other members.

* Added member for template simplifier pointer

This can be used more efficiently than marking Tokens with a flag and then searching through all templates to find the one that matches.

* Turned loop inside out

This means we only have to iterate through the std::list once.  std::list is very expensive to iterate through.

* Latest code from danmar and fixed optimisations

In particular I have optimised simplifying template instantiation names as this was incredibly slow because of the number of times it had to iterate through the template instantiation list.  Previous optimisations to this weren't very effective and broke some edge cases.

* Added changes from danmar

Made mExplicitInstantiationsToDelete a vector of TokenAndName to be consistent with the rest of the members, which are cleaned up very efficiently.

* Tokens can have many templateSimplifierPointers

* templateSimplifierPointers must be kept in sync
This commit is contained in:
rebnridgway 2018-12-21 12:51:45 +00:00 committed by Daniel Marjamäki
parent e7760c0db1
commit 431d068339
4 changed files with 274 additions and 272 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

88
lib/templatesimplifier.cpp
View File

@ -56,7 +56,21 @@ namespace {
TemplateSimplifier::TokenAndName::TokenAndName(Token *tok, const std::string &s, const std::string &n, const Token *nt) :
token(tok), scope(s), name(n), nameToken(nt)
{
token->hasTemplateSimplifierPointer(true);
if (token)
token->templateSimplifierPointer(this);
}
TemplateSimplifier::TokenAndName::TokenAndName(const TokenAndName& otherTok) :
token(otherTok.token), scope(otherTok.scope), name(otherTok.name), nameToken(otherTok.nameToken)
{
if (token)
token->templateSimplifierPointer(this);
}
TemplateSimplifier::TokenAndName::~TokenAndName()
{
if (token)
token->templateSimplifierPointers().erase(this);
}
TemplateSimplifier::TemplateSimplifier(TokenList &tokenlist, const Settings *settings, ErrorLogger *errorLogger)
@ -343,38 +357,6 @@ void TemplateSimplifier::eraseTokens(Token *begin, const Token *end)
return;
while (begin->next() && begin->next() != end) {
// check if token has a pointer to it
if (begin->next()->hasTemplateSimplifierPointer()) {
// check if the token is in a list
const std::list<TokenAndName>::iterator it = std::find_if(mTemplateInstantiations.begin(),
mTemplateInstantiations.end(),
FindToken(begin->next()));
if (it != mTemplateInstantiations.end()) {
// remove the pointer flag and prevent the deleted pointer from being used
it->token->hasTemplateSimplifierPointer(false);
it->token = nullptr;
}
const std::list<TokenAndName>::iterator it2 = std::find_if(mTemplateDeclarations.begin(),
mTemplateDeclarations.end(),
FindToken(begin->next()));
if (it2 != mTemplateDeclarations.end()) {
// remove the pointer flag and prevent the deleted pointer from being used
it2->token->hasTemplateSimplifierPointer(false);
it2->token = nullptr;
}
for (size_t i = 0; i < mTypesUsedInTemplateInstantiation.size(); ++i) {
if (mTypesUsedInTemplateInstantiation[i] == begin->next()) {
mTypesUsedInTemplateInstantiation[i]->hasTemplateSimplifierPointer(false);
mTypesUsedInTemplateInstantiation[i] = nullptr;
}
}
for (size_t i = 0; i < mExplicitInstantiationsToDelete.size(); ++i) {
if (mExplicitInstantiationsToDelete[i] == begin->next()) {
mExplicitInstantiationsToDelete[i]->hasTemplateSimplifierPointer(false);
mExplicitInstantiationsToDelete[i] = nullptr;
}
}
}
begin->deleteNext();
}
}
@ -1060,7 +1042,7 @@ void TemplateSimplifier::expandTemplate(
if (itype < typeParametersInDeclaration.size()) {
typeindentlevel = 0;
for (const Token *typetok = mTypesUsedInTemplateInstantiation[itype];
for (const Token *typetok = mTypesUsedInTemplateInstantiation[itype].token;
typetok && (typeindentlevel > 0 || !Token::Match(typetok, ",|>"));
typetok = typetok->next()) {
if (Token::simpleMatch(typetok, ". . .")) {
@ -1133,8 +1115,7 @@ void TemplateSimplifier::expandTemplate(
if (Token::Match(start, "template !!<")) {
if (start->strAt(-1) == "extern")
start = start->previous();
start->hasTemplateSimplifierPointer(true);
mExplicitInstantiationsToDelete.push_back(start);
mExplicitInstantiationsToDelete.emplace_back(start, "", "", nullptr);
}
}
@ -1254,7 +1235,7 @@ void TemplateSimplifier::expandTemplate(
// replace type with given type..
if (itype < typeParametersInDeclaration.size()) {
unsigned int typeindentlevel = 0;
for (const Token *typetok = mTypesUsedInTemplateInstantiation[itype];
for (const Token *typetok = mTypesUsedInTemplateInstantiation[itype].token;
typetok && (typeindentlevel>0 || !Token::Match(typetok, ",|>"));
typetok = typetok->next()) {
if (Token::simpleMatch(typetok, ". . .")) {
@ -1500,10 +1481,14 @@ bool TemplateSimplifier::simplifyNumericCalculations(Token *tok)
// TODO: This is not the correct class for simplifyCalculations(), so it
// should be moved away.
bool TemplateSimplifier::simplifyCalculations()
bool TemplateSimplifier::simplifyCalculations(Token* frontToken)
{
bool ret = false;
for (Token *tok = mTokenList.front(); tok; tok = tok->next()) {
if (!frontToken)
{
frontToken = mTokenList.front();
}
for (Token *tok = frontToken; tok; tok = tok->next()) {
// Remove parentheses around variable..
// keep parentheses here: dynamic_cast<Fred *>(p);
// keep parentheses here: A operator * (int);
@ -1754,8 +1739,7 @@ std::string TemplateSimplifier::getNewName(
else if (indentlevel > 0 && Token::Match(tok3, "> [,>]"))
--indentlevel;
if (indentlevel == 0 && Token::Match(tok3->previous(), "[<,]")) {
tok3->hasTemplateSimplifierPointer(true);
mTypesUsedInTemplateInstantiation.push_back(tok3);
mTypesUsedInTemplateInstantiation.emplace_back(tok3, "", "", nullptr);
}
const bool constconst = tok3->str() == "const" && tok3->strAt(1) == "const";
if (!constconst) {
@ -1830,7 +1814,7 @@ bool TemplateSimplifier::simplifyTemplateInstantiations(
for (const TokenAndName &instantiation : mTemplateInstantiations) {
if (numberOfTemplateInstantiations != mTemplateInstantiations.size()) {
numberOfTemplateInstantiations = mTemplateInstantiations.size();
simplifyCalculations();
simplifyCalculations(instantiation.token);
++recursiveCount;
if (recursiveCount > 100) {
// bail out..
@ -1912,7 +1896,7 @@ bool TemplateSimplifier::simplifyTemplateInstantiations(
// TODO: remove the specialized check and handle all uninstantiated templates someday.
if (it == mTemplateInstantiations.end() && specialized) {
simplifyCalculations();
simplifyCalculations(templateDeclaration.token);
Token * tok2 = const_cast<Token *>(tok->tokAt(namepos));
if (mErrorLogger && !mTokenList.getFiles().empty())
@ -2027,7 +2011,7 @@ void TemplateSimplifier::replaceTemplateUsage(
// match parameters
Token * tok2 = nameTok->tokAt(2);
unsigned int typeCountInInstantiation = 1U; // There is always at least one type
const Token *typetok = (!mTypesUsedInTemplateInstantiation.empty()) ? mTypesUsedInTemplateInstantiation[0] : nullptr;
const Token *typetok = (!mTypesUsedInTemplateInstantiation.empty()) ? mTypesUsedInTemplateInstantiation[0].token : nullptr;
unsigned int indentlevel2 = 0; // indentlevel for tokgt
while (tok2 && (indentlevel2 > 0 || tok2->str() != ">")) {
if (tok2->str() == "<" && templateParameters(tok2) > 0)
@ -2046,7 +2030,7 @@ void TemplateSimplifier::replaceTemplateUsage(
typetok = typetok->next();
} else {
if (typeCountInInstantiation < mTypesUsedInTemplateInstantiation.size())
typetok = mTypesUsedInTemplateInstantiation[typeCountInInstantiation++];
typetok = mTypesUsedInTemplateInstantiation[typeCountInInstantiation++].token;
else
typetok = nullptr;
}
@ -2062,18 +2046,18 @@ void TemplateSimplifier::replaceTemplateUsage(
if (tok2->str() == ">" && typeCountInInstantiation == mTypesUsedInTemplateInstantiation.size()) {
const Token * const nameTok1 = nameTok;
nameTok->str(newName);
for (std::list<TokenAndName>::iterator it = mTemplateInstantiations.begin(); it != mTemplateInstantiations.end(); ++it) {
if (it->token == nameTok1)
it->token = nameTok;
}
for (Token *tok = nameTok1->next(); tok != tok2; tok = tok->next()) {
if (tok->isName()) {
if (tok->isName() && !tok->templateSimplifierPointers().empty()) {
std::list<TokenAndName>::iterator ti;
for (ti = mTemplateInstantiations.begin(); ti != mTemplateInstantiations.end();) {
if (ti->token == tok)
{
mTemplateInstantiations.erase(ti++);
else
break;
} else {
++ti;
}
}
}
}
@ -2263,7 +2247,7 @@ void TemplateSimplifier::simplifyTemplates(
// remove explicit instantiations
for (size_t j = 0; j < mExplicitInstantiationsToDelete.size(); ++j) {
Token * start = mExplicitInstantiationsToDelete[j];
Token * start = mExplicitInstantiationsToDelete[j].token;
if (start) {
Token * end = start->next();
while (end && end->str() != ";")

9
lib/templatesimplifier.h
View File

@ -70,6 +70,9 @@ public:
*/
struct TokenAndName {
TokenAndName(Token *tok, const std::string &s, const std::string &n, const Token *nt);
TokenAndName(const TokenAndName& otherTok);
~TokenAndName();
bool operator == (const TokenAndName & rhs) const {
return token == rhs.token && scope == rhs.scope && name == rhs.name && nameToken == rhs.nameToken;
}
@ -120,7 +123,7 @@ public:
* @return true if modifications to token-list are done.
* false if no modifications are done.
*/
bool simplifyCalculations();
bool simplifyCalculations(Token* tok = nullptr);
private:
/**
@ -269,8 +272,8 @@ private:
std::list<TokenAndName> mTemplateInstantiations;
std::list<TokenAndName> mInstantiatedTemplates;
std::list<TokenAndName> mMemberFunctionsToDelete;
std::vector<Token *> mExplicitInstantiationsToDelete;
std::vector<Token *> mTypesUsedInTemplateInstantiation;
std::vector<TokenAndName> mExplicitInstantiationsToDelete;
std::vector<TokenAndName> mTypesUsedInTemplateInstantiation;
};
/// @}

226
lib/token.cpp
View File

@ -33,37 +33,22 @@
#include <stack>
#include <utility>
const std::list<ValueFlow::Value> Token::mEmptyValueList;
const std::list<ValueFlow::Value> TokenImpl::mEmptyValueList;
Token::Token(TokensFrontBack *tokensFrontBack) :
mTokensFrontBack(tokensFrontBack),
mNext(nullptr),
mPrevious(nullptr),
mLink(nullptr),
mScope(nullptr),
mFunction(nullptr), // Initialize whole union
mVarId(0),
mFileIndex(0),
mLineNumber(0),
mColumn(0),
mProgressValue(0),
mTokType(eNone),
mFlags(0),
mBits(0),
mAstOperand1(nullptr),
mAstOperand2(nullptr),
mAstParent(nullptr),
mOriginalName(nullptr),
mValueType(nullptr),
mValues(nullptr)
mFlags(0)
{
mImpl = new TokenImpl();
}
Token::~Token()
{
delete mOriginalName;
delete mValueType;
delete mValues;
delete mImpl;
}
static const std::set<std::string> controlFlowKeywords = {
@ -88,7 +73,7 @@ void Token::update_property_info()
if (mStr == "true" || mStr == "false")
tokType(eBoolean);
else if (std::isalpha((unsigned char)mStr[0]) || mStr[0] == '_' || mStr[0] == '$') { // Name
if (mVarId)
if (mImpl->mVarId)
tokType(eVariable);
else if (mTokType != eVariable && mTokType != eFunction && mTokType != eType && mTokType != eKeyword)
tokType(eName);
@ -249,20 +234,21 @@ void Token::swapWithNext()
std::swap(mStr, mNext->mStr);
std::swap(mTokType, mNext->mTokType);
std::swap(mFlags, mNext->mFlags);
std::swap(mVarId, mNext->mVarId);
std::swap(mFileIndex, mNext->mFileIndex);
std::swap(mLineNumber, mNext->mLineNumber);
std::swap(mImpl, mNext->mImpl);
for (auto templateSimplifierPointer : mImpl->mTemplateSimplifierPointers)
{
templateSimplifierPointer->token = this;
}
for (auto templateSimplifierPointer : mNext->mImpl->mTemplateSimplifierPointers)
{
templateSimplifierPointer->token = mNext;
}
if (mNext->mLink)
mNext->mLink->mLink = this;
if (this->mLink)
this->mLink->mLink = mNext;
std::swap(mLink, mNext->mLink);
std::swap(mScope, mNext->mScope);
std::swap(mFunction, mNext->mFunction);
std::swap(mOriginalName, mNext->mOriginalName);
std::swap(mValues, mNext->mValues);
std::swap(mValueType, mNext->mValueType);
std::swap(mProgressValue, mNext->mProgressValue);
}
}
@ -271,23 +257,14 @@ void Token::takeData(Token *fromToken)
mStr = fromToken->mStr;
tokType(fromToken->mTokType);
mFlags = fromToken->mFlags;
mVarId = fromToken->mVarId;
mFileIndex = fromToken->mFileIndex;
mLineNumber = fromToken->mLineNumber;
mLink = fromToken->mLink;
mScope = fromToken->mScope;
mFunction = fromToken->mFunction;
if (fromToken->mOriginalName) {
delete mOriginalName;
mOriginalName = fromToken->mOriginalName;
fromToken->mOriginalName = nullptr;
delete mImpl;
mImpl = fromToken->mImpl;
fromToken->mImpl = nullptr;
for (auto templateSimplifierPointer : mImpl->mTemplateSimplifierPointers)
{
templateSimplifierPointer->token = this;
}
delete mValues;
mValues = fromToken->mValues;
fromToken->mValues = nullptr;
delete mValueType;
mValueType = fromToken->mValueType;
fromToken->mValueType = nullptr;
mLink = fromToken->mLink;
if (mLink)
mLink->link(this);
}
@ -340,7 +317,7 @@ void Token::replace(Token *replaceThis, Token *start, Token *end)
// Update mProgressValue, fileIndex and linenr
for (Token *tok = start; tok != end->next(); tok = tok->next())
tok->mProgressValue = replaceThis->mProgressValue;
tok->mImpl->mProgressValue = replaceThis->mImpl->mProgressValue;
// Delete old token, which is replaced
delete replaceThis;
@ -796,7 +773,7 @@ void Token::move(Token *srcStart, Token *srcEnd, Token *newLocation)
// Update _progressValue
for (Token *tok = srcStart; tok != srcEnd->next(); tok = tok->next())
tok->mProgressValue = newLocation->mProgressValue;
tok->mImpl->mProgressValue = newLocation->mImpl->mProgressValue;
}
Token* Token::nextArgument() const
@ -960,9 +937,9 @@ void Token::insertToken(const std::string &tokenStr, const std::string &original
newToken->originalName(originalNameStr);
if (newToken != this) {
newToken->mLineNumber = mLineNumber;
newToken->mFileIndex = mFileIndex;
newToken->mProgressValue = mProgressValue;
newToken->mImpl->mLineNumber = mImpl->mLineNumber;
newToken->mImpl->mFileIndex = mImpl->mFileIndex;
newToken->mImpl->mProgressValue = mImpl->mProgressValue;
if (prepend) {
if (this->previous()) {
@ -1052,8 +1029,8 @@ void Token::stringify(std::ostream& os, bool varid, bool attributes, bool macro)
os << mStr[i];
}
}
if (varid && mVarId != 0)
os << '@' << mVarId;
if (varid && mImpl->mVarId != 0)
os << '@' << mImpl->mVarId;
}
std::string Token::stringifyList(bool varid, bool attributes, bool linenumbers, bool linebreaks, bool files, const std::vector<std::string>* fileNames, const Token* end) const
@ -1063,21 +1040,21 @@ std::string Token::stringifyList(bool varid, bool attributes, bool linenumbers,
std::ostringstream ret;
unsigned int lineNumber = mLineNumber - (linenumbers ? 1U : 0U);
unsigned int fileInd = files ? ~0U : mFileIndex;
unsigned int lineNumber = mImpl->mLineNumber - (linenumbers ? 1U : 0U);
unsigned int fileInd = files ? ~0U : mImpl->mFileIndex;
std::map<int, unsigned int> lineNumbers;
for (const Token *tok = this; tok != end; tok = tok->next()) {
bool fileChange = false;
if (tok->mFileIndex != fileInd) {
if (tok->mImpl->mFileIndex != fileInd) {
if (fileInd != ~0U) {
lineNumbers[fileInd] = tok->mFileIndex;
lineNumbers[fileInd] = tok->mImpl->mFileIndex;
}
fileInd = tok->mFileIndex;
fileInd = tok->mImpl->mFileIndex;
if (files) {
ret << "\n\n##file ";
if (fileNames && fileNames->size() > tok->mFileIndex)
ret << fileNames->at(tok->mFileIndex);
if (fileNames && fileNames->size() > tok->mImpl->mFileIndex)
ret << fileNames->at(tok->mImpl->mFileIndex);
else
ret << fileInd;
ret << '\n';
@ -1088,7 +1065,7 @@ std::string Token::stringifyList(bool varid, bool attributes, bool linenumbers,
}
if (linebreaks && (lineNumber != tok->linenr() || fileChange)) {
if (lineNumber+4 < tok->linenr() && fileInd == tok->mFileIndex) {
if (lineNumber+4 < tok->linenr() && fileInd == tok->mImpl->mFileIndex) {
ret << '\n' << lineNumber+1 << ":\n|\n";
ret << tok->linenr()-1 << ":\n";
ret << tok->linenr() << ": ";
@ -1129,37 +1106,37 @@ std::string Token::stringifyList(bool varid) const
void Token::astOperand1(Token *tok)
{
if (mAstOperand1)
mAstOperand1->mAstParent = nullptr;
if (mImpl->mAstOperand1)
mImpl->mAstOperand1->mImpl->mAstParent = nullptr;
// goto parent operator
if (tok) {
std::set<Token*> visitedParents;
while (tok->mAstParent) {
if (!visitedParents.insert(tok->mAstParent).second) // #6838/#6726/#8352 avoid hang on garbage code
while (tok->mImpl->mAstParent) {
if (!visitedParents.insert(tok->mImpl->mAstParent).second) // #6838/#6726/#8352 avoid hang on garbage code
throw InternalError(this, "Internal error. Token::astOperand1() cyclic dependency.");
tok = tok->mAstParent;
tok = tok->mImpl->mAstParent;
}
tok->mAstParent = this;
tok->mImpl->mAstParent = this;
}
mAstOperand1 = tok;
mImpl->mAstOperand1 = tok;
}
void Token::astOperand2(Token *tok)
{
if (mAstOperand2)
mAstOperand2->mAstParent = nullptr;
if (mImpl->mAstOperand2)
mImpl->mAstOperand2->mImpl->mAstParent = nullptr;
// goto parent operator
if (tok) {
std::set<Token*> visitedParents;
while (tok->mAstParent) {
while (tok->mImpl->mAstParent) {
//std::cout << tok << " -> " << tok->mAstParent ;
if (!visitedParents.insert(tok->mAstParent).second) // #6838/#6726 avoid hang on garbage code
if (!visitedParents.insert(tok->mImpl->mAstParent).second) // #6838/#6726 avoid hang on garbage code
throw InternalError(this, "Internal error. Token::astOperand2() cyclic dependency.");
tok = tok->mAstParent;
tok = tok->mImpl->mAstParent;
}
tok->mAstParent = this;
tok->mImpl->mAstParent = this;
}
mAstOperand2 = tok;
mImpl->mAstOperand2 = tok;
}
static const Token* goToLeftParenthesis(const Token* start, const Token* end)
@ -1265,9 +1242,9 @@ bool Token::isUnaryPreOp() const
const Token *tokbefore = mPrevious;
const Token *tokafter = mNext;
for (int distance = 1; distance < 10 && tokbefore; distance++) {
if (tokbefore == mAstOperand1)
if (tokbefore == mImpl->mAstOperand1)
return false;
if (tokafter == mAstOperand1)
if (tokafter == mImpl->mAstOperand1)
return true;
tokbefore = tokbefore->mPrevious;
tokafter = tokafter->mPrevious;
@ -1333,7 +1310,7 @@ void Token::printAst(bool verbose, bool xml, std::ostream &out) const
{
std::set<const Token *> printed;
for (const Token *tok = this; tok; tok = tok->next()) {
if (!tok->mAstParent && tok->mAstOperand1) {
if (!tok->mImpl->mAstParent && tok->mImpl->mAstOperand1) {
if (printed.empty() && !xml)
out << "\n\n##AST" << std::endl;
else if (printed.find(tok) != printed.end())
@ -1369,21 +1346,21 @@ std::string Token::astStringVerbose(const unsigned int indent1, const unsigned i
if (isExpandedMacro())
ret += '$';
ret += mStr;
if (mValueType)
ret += " \'" + mValueType->str() + '\'';
if (mImpl->mValueType)
ret += " \'" + mImpl->mValueType->str() + '\'';
ret += '\n';
if (mAstOperand1) {
if (mImpl->mAstOperand1) {
unsigned int i1 = indent1, i2 = indent2 + 2;
if (indent1==indent2 && !mAstOperand2)
if (indent1==indent2 && !mImpl->mAstOperand2)
i1 += 2;
ret += indent(indent1,indent2) + (mAstOperand2 ? "|-" : "`-") + mAstOperand1->astStringVerbose(i1,i2);
ret += indent(indent1,indent2) + (mImpl->mAstOperand2 ? "|-" : "`-") + mImpl->mAstOperand1->astStringVerbose(i1,i2);
}
if (mAstOperand2) {
if (mImpl->mAstOperand2) {
unsigned int i1 = indent1, i2 = indent2 + 2;
if (indent1==indent2)
i1 += 2;
ret += indent(indent1,indent2) + "`-" + mAstOperand2->astStringVerbose(i1,i2);
ret += indent(indent1,indent2) + "`-" + mImpl->mAstOperand2->astStringVerbose(i1,i2);
}
return ret;
}
@ -1397,19 +1374,19 @@ void Token::printValueFlow(bool xml, std::ostream &out) const
else
out << "\n\n##Value flow" << std::endl;
for (const Token *tok = this; tok; tok = tok->next()) {
if (!tok->mValues)
if (!tok->mImpl->mValues)
continue;
if (xml)
out << " <values id=\"" << tok->mValues << "\">" << std::endl;
out << " <values id=\"" << tok->mImpl->mValues << "\">" << std::endl;
else if (line != tok->linenr())
out << "Line " << tok->linenr() << std::endl;
line = tok->linenr();
if (!xml) {
out << " " << tok->str() << (tok->mValues->front().isKnown() ? " always " : " possible ");
if (tok->mValues->size() > 1U)
out << " " << tok->str() << (tok->mImpl->mValues->front().isKnown() ? " always " : " possible ");
if (tok->mImpl->mValues->size() > 1U)
out << '{';
}
for (const ValueFlow::Value &value : *tok->mValues) {
for (const ValueFlow::Value &value : *tok->mImpl->mValues) {
if (xml) {
out << " <value ";
switch (value.valueType) {
@ -1450,7 +1427,7 @@ void Token::printValueFlow(bool xml, std::ostream &out) const
}
else {
if (&value != &tok->mValues->front())
if (&value != &tok->mImpl->mValues->front())
out << ",";
switch (value.valueType) {
case ValueFlow::Value::INT:
@ -1482,7 +1459,7 @@ void Token::printValueFlow(bool xml, std::ostream &out) const
}
if (xml)
out << " </values>" << std::endl;
else if (tok->mValues->size() > 1U)
else if (tok->mImpl->mValues->size() > 1U)
out << '}' << std::endl;
else
out << std::endl;
@ -1493,11 +1470,11 @@ void Token::printValueFlow(bool xml, std::ostream &out) const
const ValueFlow::Value * Token::getValueLE(const MathLib::bigint val, const Settings *settings) const
{
if (!mValues)
if (!mImpl->mValues)
return nullptr;
const ValueFlow::Value *ret = nullptr;
std::list<ValueFlow::Value>::const_iterator it;
for (it = mValues->begin(); it != mValues->end(); ++it) {
for (it = mImpl->mValues->begin(); it != mImpl->mValues->end(); ++it) {
if (it->isIntValue() && it->intvalue <= val) {
if (!ret || ret->isInconclusive() || (ret->condition && !it->isInconclusive()))
ret = &(*it);
@ -1516,11 +1493,11 @@ const ValueFlow::Value * Token::getValueLE(const MathLib::bigint val, const Sett
const ValueFlow::Value * Token::getValueGE(const MathLib::bigint val, const Settings *settings) const
{
if (!mValues)
if (!mImpl->mValues)
return nullptr;
const ValueFlow::Value *ret = nullptr;
std::list<ValueFlow::Value>::const_iterator it;
for (it = mValues->begin(); it != mValues->end(); ++it) {
for (it = mImpl->mValues->begin(); it != mImpl->mValues->end(); ++it) {
if (it->isIntValue() && it->intvalue >= val) {
if (!ret || ret->isInconclusive() || (ret->condition && !it->isInconclusive()))
ret = &(*it);
@ -1539,11 +1516,11 @@ const ValueFlow::Value * Token::getValueGE(const MathLib::bigint val, const Sett
const ValueFlow::Value * Token::getInvalidValue(const Token *ftok, unsigned int argnr, const Settings *settings) const
{
if (!mValues || !settings)
if (!mImpl->mValues || !settings)
return nullptr;
const ValueFlow::Value *ret = nullptr;
std::list<ValueFlow::Value>::const_iterator it;
for (it = mValues->begin(); it != mValues->end(); ++it) {
for (it = mImpl->mValues->begin(); it != mImpl->mValues->end(); ++it) {
if ((it->isIntValue() && !settings->library.isIntArgValid(ftok, argnr, it->intvalue)) ||
(it->isFloatValue() && !settings->library.isFloatArgValid(ftok, argnr, it->floatValue))) {
if (!ret || ret->isInconclusive() || (ret->condition && !it->isInconclusive()))
@ -1563,12 +1540,12 @@ const ValueFlow::Value * Token::getInvalidValue(const Token *ftok, unsigned int
const Token *Token::getValueTokenMinStrSize() const
{
if (!mValues)
if (!mImpl->mValues)
return nullptr;
const Token *ret = nullptr;
std::size_t minsize = ~0U;
std::list<ValueFlow::Value>::const_iterator it;
for (it = mValues->begin(); it != mValues->end(); ++it) {
for (it = mImpl->mValues->begin(); it != mImpl->mValues->end(); ++it) {
if (it->isTokValue() && it->tokvalue && it->tokvalue->tokType() == Token::eString) {
const std::size_t size = getStrSize(it->tokvalue);
if (!ret || size < minsize) {
@ -1582,12 +1559,12 @@ const Token *Token::getValueTokenMinStrSize() const
const Token *Token::getValueTokenMaxStrLength() const
{
if (!mValues)
if (!mImpl->mValues)
return nullptr;
const Token *ret = nullptr;
std::size_t maxlength = 0U;
std::list<ValueFlow::Value>::const_iterator it;
for (it = mValues->begin(); it != mValues->end(); ++it) {
for (it = mImpl->mValues->begin(); it != mImpl->mValues->end(); ++it) {
if (it->isTokValue() && it->tokvalue && it->tokvalue->tokType() == Token::eString) {
const std::size_t length = getStrLength(it->tokvalue);
if (!ret || length > maxlength) {
@ -1641,22 +1618,22 @@ const Token *Token::getValueTokenDeadPointer() const
bool Token::addValue(const ValueFlow::Value &value)
{
if (value.isKnown() && mValues) {
if (value.isKnown() && mImpl->mValues) {
// Clear all other values of the same type since value is known
mValues->remove_if([&](const ValueFlow::Value & x) {
mImpl->mValues->remove_if([&](const ValueFlow::Value & x) {
return x.valueType == value.valueType;
});
}
if (mValues) {
if (mImpl->mValues) {
// Don't handle more than 10 values for performance reasons
// TODO: add setting?
if (mValues->size() >= 10U)
if (mImpl->mValues->size() >= 10U)
return false;
// if value already exists, don't add it again
std::list<ValueFlow::Value>::iterator it;
for (it = mValues->begin(); it != mValues->end(); ++it) {
for (it = mImpl->mValues->begin(); it != mImpl->mValues->end(); ++it) {
// different intvalue => continue
if (it->intvalue != value.intvalue)
continue;
@ -1671,7 +1648,7 @@ bool Token::addValue(const ValueFlow::Value &value)
if (it->isInconclusive() && !value.isInconclusive()) {
*it = value;
if (it->varId == 0)
it->varId = mVarId;
it->varId = mImpl->mVarId;
break;
}
@ -1680,20 +1657,20 @@ bool Token::addValue(const ValueFlow::Value &value)
}
// Add value
if (it == mValues->end()) {
if (it == mImpl->mValues->end()) {
ValueFlow::Value v(value);
if (v.varId == 0)
v.varId = mVarId;
v.varId = mImpl->mVarId;
if (v.isKnown() && v.isIntValue())
mValues->push_front(v);
mImpl->mValues->push_front(v);
else
mValues->push_back(v);
mImpl->mValues->push_back(v);
}
} else {
ValueFlow::Value v(value);
if (v.varId == 0)
v.varId = mVarId;
mValues = new std::list<ValueFlow::Value>(1, v);
v.varId = mImpl->mVarId;
mImpl->mValues = new std::list<ValueFlow::Value>(1, v);
}
return true;
@ -1706,24 +1683,35 @@ void Token::assignProgressValues(Token *tok)
++total_count;
unsigned int count = 0;
for (Token *tok2 = tok; tok2; tok2 = tok2->next())
tok2->mProgressValue = count++ * 100 / total_count;
tok2->mImpl->mProgressValue = count++ * 100 / total_count;
}
void Token::setValueType(ValueType *vt)
{
if (vt != mValueType) {
delete mValueType;
mValueType = vt;
if (vt != mImpl->mValueType) {
delete mImpl->mValueType;
mImpl->mValueType = vt;
}
}
void Token::type(const ::Type *t)
{
mType = t;
mImpl->mType = t;
if (t) {
tokType(eType);
isEnumType(mType->isEnumType());
isEnumType(mImpl->mType->isEnumType());
} else if (mTokType == eType)
tokType(eName);
}
TokenImpl::~TokenImpl()
{
delete mOriginalName;
delete mValueType;
delete mValues;
for (auto templateSimplifierPointer : mTemplateSimplifierPointers)
{
templateSimplifierPointer->token = nullptr;
}
}

223
lib/token.h
View File

@ -24,6 +24,7 @@
#include "config.h"
#include "mathlib.h"
#include "valueflow.h"
#include "templatesimplifier.h"
#include <algorithm>
#include <cstddef>
@ -49,6 +50,69 @@ struct TokensFrontBack {
Token *back;
};
struct TokenImpl {
unsigned int mVarId;
unsigned int mFileIndex;
unsigned int mLineNumber;
unsigned int mColumn;
// AST..
Token *mAstOperand1;
Token *mAstOperand2;
Token *mAstParent;
// symbol database information
const Scope *mScope;
union {
const Function *mFunction;
const Variable *mVariable;
const ::Type* mType;
const Enumerator *mEnumerator;
};
/**
* A value from 0-100 that provides a rough idea about where in the token
* list this token is located.
*/
unsigned int mProgressValue;
// original name like size_t
std::string* mOriginalName;
// ValueType
ValueType *mValueType;
// ValueFlow
std::list<ValueFlow::Value>* mValues;
static const std::list<ValueFlow::Value> mEmptyValueList;
/** Bitfield bit count. */
unsigned char mBits;
// Pointer to a template in the template simplifier
std::set<TemplateSimplifier::TokenAndName*> mTemplateSimplifierPointers;
TokenImpl()
: mVarId(0)
, mFileIndex(0)
, mLineNumber(0)
, mColumn(0)
, mAstOperand1(nullptr)
, mAstOperand2(nullptr)
, mAstParent(nullptr)
, mScope(nullptr)
, mFunction(nullptr) // Initialize whole union
, mProgressValue(0)
, mOriginalName(nullptr)
, mValueType(nullptr)
, mValues(nullptr)
, mBits(0)
, mTemplateSimplifierPointers()
{}
~TokenImpl();
};
/// @addtogroup Core
/// @{
@ -86,7 +150,7 @@ public:
template<typename T>
void str(T&& s) {
mStr = s;
mVarId = 0;
mImpl->mVarId = 0;
update_property_info();
}
@ -236,7 +300,7 @@ public:
static std::string getCharAt(const Token *tok, std::size_t index);
const ValueType *valueType() const {
return mValueType;
return mImpl->mValueType;
}
void setValueType(ValueType *vt);
@ -244,7 +308,7 @@ public:
const Token *top = this;
while (top && !Token::Match(top->astParent(), ",|("))
top = top->astParent();
return top ? top->mValueType : nullptr;
return top ? top->mImpl->mValueType : nullptr;
}
Token::Type tokType() const {
@ -456,19 +520,19 @@ public:
}
bool isBitfield() const {
return mBits > 0;
return mImpl->mBits > 0;
}
unsigned char bits() const {
return mBits;
return mImpl->mBits;
}
bool hasTemplateSimplifierPointer() const {
return getFlag(fHasTemplateSimplifierPointer);
std::set<TemplateSimplifier::TokenAndName*> &templateSimplifierPointers() const {
return mImpl->mTemplateSimplifierPointers;
}
void hasTemplateSimplifierPointer(const bool value) {
setFlag(fHasTemplateSimplifierPointer, value);
void templateSimplifierPointer(TemplateSimplifier::TokenAndName* tokenAndName) {
mImpl->mTemplateSimplifierPointers.emplace(tokenAndName);
}
void setBits(const unsigned char b) {
mBits = b;
mImpl->mBits = b;
}
/**
@ -529,24 +593,24 @@ public:
static int multiCompare(const Token *tok, const char *haystack, unsigned int varid);
unsigned int fileIndex() const {
return mFileIndex;
return mImpl->mFileIndex;
}
void fileIndex(unsigned int indexOfFile) {
mFileIndex = indexOfFile;
mImpl->mFileIndex = indexOfFile;
}
unsigned int linenr() const {
return mLineNumber;
return mImpl->mLineNumber;
}
void linenr(unsigned int lineNumber) {
mLineNumber = lineNumber;
mImpl->mLineNumber = lineNumber;
}
unsigned int col() const {
return mColumn;
return mImpl->mColumn;
}
void col(unsigned int c) {
mColumn = c;
mImpl->mColumn = c;
}
Token *next() const {
@ -579,10 +643,10 @@ public:
unsigned int varId() const {
return mVarId;
return mImpl->mVarId;
}
void varId(unsigned int id) {
mVarId = id;
mImpl->mVarId = id;
if (id != 0) {
tokType(eVariable);
isStandardType(false);
@ -681,14 +745,14 @@ public:
* @param s Scope to be associated
*/
void scope(const Scope *s) {
mScope = s;
mImpl->mScope = s;
}
/**
* @return a pointer to the scope containing this token.
*/
const Scope *scope() const {
return mScope;
return mImpl->mScope;
}
/**
@ -696,7 +760,7 @@ public:
* @param f Function to be associated
*/
void function(const Function *f) {
mFunction = f;
mImpl->mFunction = f;
if (f)
tokType(eFunction);
else if (mTokType == eFunction)
@ -707,7 +771,7 @@ public:
* @return a pointer to the Function associated with this token.
*/
const Function *function() const {
return mTokType == eFunction ? mFunction : nullptr;
return mTokType == eFunction ? mImpl->mFunction : nullptr;
}
/**
@ -715,8 +779,8 @@ public:
* @param v Variable to be associated
*/
void variable(const Variable *v) {
mVariable = v;
if (v || mVarId)
mImpl->mVariable = v;
if (v || mImpl->mVarId)
tokType(eVariable);
else if (mTokType == eVariable)
tokType(eName);
@ -726,7 +790,7 @@ public:
* @return a pointer to the variable associated with this token.
*/
const Variable *variable() const {
return mTokType == eVariable ? mVariable : nullptr;
return mTokType == eVariable ? mImpl->mVariable : nullptr;
}
/**
@ -739,14 +803,14 @@ public:
* @return a pointer to the type associated with this token.
*/
const ::Type *type() const {
return mTokType == eType ? mType : nullptr;
return mTokType == eType ? mImpl->mType : nullptr;
}
/**
* @return a pointer to the Enumerator associated with this token.
*/
const Enumerator *enumerator() const {
return mTokType == eEnumerator ? mEnumerator : nullptr;
return mTokType == eEnumerator ? mImpl->mEnumerator : nullptr;
}
/**
@ -754,7 +818,7 @@ public:
* @param e Enumerator to be associated
*/
void enumerator(const Enumerator *e) {
mEnumerator = e;
mImpl->mEnumerator = e;
if (e)
tokType(eEnumerator);
else if (mTokType == eEnumerator)
@ -785,7 +849,7 @@ public:
/** Get progressValue */
unsigned int progressValue() const {
return mProgressValue;
return mImpl->mProgressValue;
}
/** Calculate progress values for all tokens */
@ -827,11 +891,11 @@ public:
* @return the original name.
*/
const std::string & originalName() const {
return mOriginalName ? *mOriginalName : emptyString;
return mImpl->mOriginalName ? *mImpl->mOriginalName : emptyString;
}
const std::list<ValueFlow::Value>& values() const {
return mValues ? *mValues : mEmptyValueList;
return mImpl->mValues ? *mImpl->mValues : mImpl->mEmptyValueList;
}
/**
@ -839,24 +903,24 @@ public:
*/
template<typename T>
void originalName(T&& name) {
if (!mOriginalName)
mOriginalName = new std::string(name);
if (!mImpl->mOriginalName)
mImpl->mOriginalName = new std::string(name);
else
*mOriginalName = name;
*mImpl->mOriginalName = name;
}
bool hasKnownIntValue() const {
return hasKnownValue() && std::any_of(mValues->begin(), mValues->end(), std::mem_fn(&ValueFlow::Value::isIntValue));
return hasKnownValue() && std::any_of(mImpl->mValues->begin(), mImpl->mValues->end(), std::mem_fn(&ValueFlow::Value::isIntValue));
}
bool hasKnownValue() const {
return mValues && std::any_of(mValues->begin(), mValues->end(), std::mem_fn(&ValueFlow::Value::isKnown));
return mImpl->mValues && std::any_of(mImpl->mValues->begin(), mImpl->mValues->end(), std::mem_fn(&ValueFlow::Value::isKnown));
}
const ValueFlow::Value * getValue(const MathLib::bigint val) const {
if (!mValues)
if (!mImpl->mValues)
return nullptr;
for (const ValueFlow::Value &value : *mValues) {
for (const ValueFlow::Value &value : *mImpl->mValues) {
if (value.isIntValue() && value.intvalue == val)
return &value;
}
@ -864,10 +928,10 @@ public:
}
const ValueFlow::Value * getMaxValue(bool condition) const {
if (!mValues)
if (!mImpl->mValues)
return nullptr;
const ValueFlow::Value *ret = nullptr;
for (const ValueFlow::Value &value : *mValues) {
for (const ValueFlow::Value &value : *mImpl->mValues) {
if (!value.isIntValue())
continue;
if ((!ret || value.intvalue > ret->intvalue) &&
@ -878,9 +942,9 @@ public:
}
const ValueFlow::Value * getMovedValue() const {
if (!mValues)
if (!mImpl->mValues)
return nullptr;
for (const ValueFlow::Value &value : *mValues) {
for (const ValueFlow::Value &value : *mImpl->mValues) {
if (value.isMovedValue() && value.moveKind != ValueFlow::Value::NonMovedVariable)
return &value;
}
@ -893,9 +957,9 @@ public:
const ValueFlow::Value * getInvalidValue(const Token *ftok, unsigned int argnr, const Settings *settings) const;
const ValueFlow::Value * getContainerSizeValue(const MathLib::bigint val) const {
if (!mValues)
if (!mImpl->mValues)
return nullptr;
for (const ValueFlow::Value &value : *mValues) {
for (const ValueFlow::Value &value : *mImpl->mValues) {
if (value.isContainerSizeValue() && value.intvalue == val)
return &value;
}
@ -942,28 +1006,6 @@ private:
Token *mPrevious;
Token *mLink;
// symbol database information
const Scope *mScope;
union {
const Function *mFunction;
const Variable *mVariable;
const ::Type* mType;
const Enumerator *mEnumerator;
};
unsigned int mVarId;
unsigned int mFileIndex;
unsigned int mLineNumber;
unsigned int mColumn;
/**
* A value from 0-100 that provides a rough idea about where in the token
* list this token is located.
*/
unsigned int mProgressValue;
Token::Type mTokType;
enum {
fIsUnsigned = (1 << 0),
fIsSigned = (1 << 1),
@ -989,12 +1031,15 @@ private:
fIsLiteral = (1 << 21),
fIsTemplateArg = (1 << 22),
fIsAttributeNodiscard = (1 << 23), // __attribute__ ((warn_unused_result)), [[nodiscard]]
fHasTemplateSimplifierPointer = (1 << 24), // used by template simplifier to indicate it has a pointer to this token
fAtAddress = (1 << 25), // @ 0x4000
fAtAddress = (1 << 24), // @ 0x4000
};
unsigned int mFlags;
Token::Type mTokType;
TokenImpl *mImpl;
/**
* Get specified flag state.
* @param flag_ flag to get state of
@ -1020,41 +1065,23 @@ private:
/** Update internal property cache about isStandardType() */
void update_property_isStandardType();
/** Bitfield bit count. */
unsigned char mBits;
// AST..
Token *mAstOperand1;
Token *mAstOperand2;
Token *mAstParent;
// original name like size_t
std::string* mOriginalName;
// ValueType
ValueType *mValueType;
// ValueFlow
std::list<ValueFlow::Value>* mValues;
static const std::list<ValueFlow::Value> mEmptyValueList;
public:
void astOperand1(Token *tok);
void astOperand2(Token *tok);
const Token * astOperand1() const {
return mAstOperand1;
return mImpl->mAstOperand1;
}
const Token * astOperand2() const {
return mAstOperand2;
return mImpl->mAstOperand2;
}
const Token * astParent() const {
return mAstParent;
return mImpl->mAstParent;
}
const Token *astTop() const {
const Token *ret = this;
while (ret->mAstParent)
ret = ret->mAstParent;
while (ret->mImpl->mAstParent)
ret = ret->mImpl->mAstParent;
return ret;
}
@ -1070,20 +1097,20 @@ public:
bool isCalculation() const;
void clearAst() {
mAstOperand1 = mAstOperand2 = mAstParent = nullptr;
mImpl->mAstOperand1 = mImpl->mAstOperand2 = mImpl->mAstParent = nullptr;
}
void clearValueFlow() {
delete mValues;
mValues = nullptr;
delete mImpl->mValues;
mImpl->mValues = nullptr;
}
std::string astString(const char *sep = "") const {
std::string ret;
if (mAstOperand1)
ret = mAstOperand1->astString(sep);
if (mAstOperand2)
ret += mAstOperand2->astString(sep);
if (mImpl->mAstOperand1)
ret = mImpl->mAstOperand1->astString(sep);
if (mImpl->mAstOperand2)
ret += mImpl->mAstOperand2->astString(sep);
return ret + sep + mStr;
}