Research Summary: SuperDetector: A Framework for Performance Detection on Vulnerabilities of Smart Contracts

[Harvesto]

TLDR

• The researcher introduces eight common vulnerabilities and categorizes them based on call-related and call-irrelated vulnerabilities.
• They proposed SuperDetector, a framework for detecting both vulnerabilities by combining the different tools and static analysis technology.
• The research shows that SuperDetector can analyze the performance of each tool and that vulnerability coverage and detection accuracy on call-related vulnerabilities is much better than that on call-irrelated vulnerabilities.

Core Research Question

What tool can accurately detect smart contracts while comparing the performance of each tool used?

Citation

Dai, M., Yang, Z., & Guo, J. (2022). SuperDetector: A framework for performance detection on vulnerabilities of smart contracts. Journal of Physics. Conference Series, 2289(1), 012010. doi:10.1088/1742-6596/2289/1/012010

Background

• Smart Contract Vulnerabilities: Security issues in smart contracts that can be exploited. There are many different types. In this paper, the researchers focused purely on CallVUL and CallIRVUL bugs which combined have resulted in millions of dollars of losses in DeFi protocols.
• Call-related and Call-Irrelated Vulnerabilities (CallVUL and CallIRVUL): CallVUL and CallIRVUL are the causes of eight typical vulnerabilities of smart contracts. Caller functions ensure the flexible development of smart contracts. Furthermore, they support the transfer of Ether (ETH) between smart contracts. This characteristic and relationship between multiple calls make them vulnerable. In addition, factors like mining and Ethereum Virtual Machine’s (EVM) execution cause call-irrelated vulnerabilities (CallIRVUL).
• Static Analysis Technology: Static analysis is a technique for analyzing computer programs without executing the code. The process gives the user a better understanding of the code structure and ensures the code meets industry standards. Static analysis is usually performed by an automated tool controlled by a human researcher.
• Re-Entrancy (RE): Re-entrancy attacks are well-known due to the famous 2016 DAO hack on the Ethereum blockchain. It is one of the most destructive attacks in Solidity smart contracts and occurs when a call function makes an external call to another. Its other causes are:
  • No gas restriction
  • Transferring before recording
  • Using call() function to transfer money
• Unhandled Exception (UE): An operation in Solidity used to send Ether that reports the return state with booleans. If the return value is unchecked, the contract will continue executing the instructions leading to inconsistencies. It is caused by:
  • Using call() or send() function to call
  • Never throwing an exception when call() or send() fails
• Dangerous Tx.origin (DT): Tx.origin is a global transaction variable in Solidity which reports the address of the account that sent the transaction. Contracts that use tx.origin to authorize transactions are dangerous as they are vulnerable to phishing attacks. The vulnerability is due to obtained permissions of the caller smart contract through the tx.origin variable.
• Dangerous Delegatecall (DD): This vulnerability happens when a ContractA calls ContractB using delegatecall. The transaction tells ContractB to run ContractA’s code inside its context (storage, msg.sender, msg.value, msg.data, etc). However, the called smart contract can affect the current state of the caller smart contract using delegatecall() function.
• Integer Bugs (IB): They are difficult unavoidable smart contract vulnerabilities due to characteristics and bugs in EVM and the Solidity programming language. Integer bugs occur while converting the type of integer.
• Transaction Ordering Dependence (TOD): This vulnerability is a race-conditioned attack. With enough computing resources, a malicious user can reorder the transactions in the mempool to get whatever result they want. TOD occurs when the execution order of transactions in the same contract causes differences in the final state of the smart contract.
• Timestamp Dependence (TD): Block timestamps execute call and fund transfer functions. This vulnerability is related to how easily miners can easily control block timestamps. A malicious actor can manipulate the timestamp of the transactions because of the execution logic, which allows miners to post a timestamp within 15 seconds of the block being validated to favor themselves.
• Dangerous Selfdestruct (DS): Selfdestruct(address) functions send all ether stored to the specified address. An attacker can create a contract with a selfdestruct() function, send ether to it, and create another called selfdestruct(target). This action forces the first smart contract to send ether to the target smart contract address.
• Oyente: This is an execution and vulnerability detection tool that works directly with Ethereum virtual machine (EVM) byte code.
• Osiris: This is an analysis tool to detect integer bugs in Ethereum smart contracts. Osiris is based on Oyente.
• Slither: Known as the Solidity source analyzer, Slither has a variety of vulnerability detectors and provides an easy-to-use API for writing custom analyses.
• Securify: This is a security scanner for Ethereum smart contracts supported by the Ethereum Foundation and ChainSecurity. The core research behind Securify was conducted at the ICE Center at ETH Zurich.
• Smartcheck: This is a static analysis tool just like those mentioned above for discovering vulnerabilities and other code issues in Ethereum smart contracts written in Solidity.
• Mythril: Mythril detects security vulnerabilities in smart contracts built for Ethereum, Hedera, Quorum, Vechain, Roostock, Tron and other EVM-compatible blockchains.
• Transaction Variables/Hash: Each transaction made in the blockchain has data slots that store the values of the transaction. These variables vanish when the transaction is complete or canceled and specific only to the single transaction.
• Average Execution Time: This is the amount of time it takes each tool to compile the smart contract, convert the intermediate language, analyze and produce output.
• Usability: The probability that the selected tool can detect the smart contract vulnerabilities well.
• Solidity Abstract Syntax Tree: A concept in Solidity that represents a given code in a syntax tree. The tree helps developers view the code better.
• EVM Bytecode: Bytecode is a low-level programming language that is compiled from Solidity. EVM is the reason Ethereum smart contracts can run on almost any computer.
• Solidity XML Parse Tree: XML stands for eXtensible Markup Language. XML documents are structured hierarchically like trees. The Solidity XML Parse Tree is a software library containing solidity and XML-based contracts.

Summary

• Smart contracts are the core of transactions on the Ethereum blockchain. However, they possess vulnerabilities that can be exploited, leading to losses.
• This research focuses on the vulnerabilities and a framework for effectively detecting them.
• The authors analyzed vulnerability coverage, usability, detection accuracy, and execution time of the tools.
• The authors analyzed two smart contract sets (a total of 1133) provided by the HuangGai project recognized by experts.
• Six open source tools were selected in the SuperDetector Framework to detect CallVUL and CallIRVUL vulnerabilities. Oyente, Osiris, Slither, Securify, Smartcheck, and Mythril were the tools used.
• These tools use solidity and cover two or more kinds of vulnerabilities.
• SuperDetector has two main functions - vulnerability detection and computing each tool’s performance on the two kinds of vulnerabilities.
• To start the experiment, the detection tools and caller functions are selected in the framework.
• After that, the selected tools execute the inputted instructions.
• According to the experiment result and analysis, each tool can detect the two kinds of smart contract vulnerabilities, but there are differences.
• The research reveals that Mythril has 100% vulnerability coverage, Smartcheck only covers three vulnerabilities, while Osiris, Securify, and Slither perform well by detecting more vulnerabilities.
• According to the detection results, Securify’s usability is about 67%, Smartcheck is about 33%, and the other three tools are over 94%. Also, Securify and Mythril take longer to analyze the smart contract.
• Finally, the analysis shows that optimal vulnerability detection, coverage, and accuracy are not guaranteed if a single tool is used.
• Therefore, the authors design a selector in SuperDetector that matches each tool with the best-suited vulnerability. This will increase the vulnerability coverage and detection accuracy for developers.

Method

• The framework is unique as it uses different tools to detect bugs and compares each tool’s performance against specific vulnerabilities.
• Next, we list the eight vulnerabilities for the analysis and classify them according to their causes - CallVUL and CallIRVUL. Caller functions cause call-related vulnerabilities, while Call-Irrelated bugs come from mining and network activity in the Ethereum Virtual Machine.
• SuperDetector computes the tools’ performance (vulnerability coverage, detection accuracy) while they detect vulnerabilities. It has five task layers, and the core parts are input conversion, detection, and output conversion.
• The framework regards the tools as plugins to concurrently use them all. The detection process is elaborated as follows:
  • The criteria for selecting the detection tools are based on research reports and previous experiments.
  • Based on the data, eight open source tools are selected. These tools must be able to analyze the two kinds of smart contract vulnerabilities, accept Solidity, and cover at least three of the eight focal vulnerabilities.
  • After selecting the tools and functions, the framework converts the instructions, and the process starts. For example, suppose a vulnerability is detected, SuperDetector sorts and converts the output according to the selected function. And if an exception occurs during the process, SuperDetector returns to the tool selection stage.
    
    

Results

• Each tool detects both vulnerabilities (CallVUL and CallIRVUL), but the number of vulnerabilities they cover varies.
  
  
  • Due to the variety in vulnerability coverage, it is recommended to combine multiple tools to increase the number of vulnerabilities detected.
• For detection accuracy of the tools, the table below demonstrates the accuracy of four tools on CallVUL and CallIRVUL.
  
  
  • The three colors indicate the proportion of hitting, missing to detect the corresponding vulnerability, and failed detection, while the number of smart contracts is displayed on the right. We can see that the accuracy of each tool on the vulnerabilities varies.
• The result also shows that Mythril, with 100% coverage, has only 80% accuracy for three vulnerabilities, while Slither’s hits RE and UE of CallVUL more than DS of CallIRVUL.
• This proves that the tools’ accuracy on CallVUL is higher than its counterpart. It was therefore speculated that this is because the former is closely related to caller functions which the tools can analyze faster. Call-Irrelated Vulnerabilities are complex, making them difficult to detect easily and quickly.
• The authors use data from the same table for usability and average execution time. They also consider the table below, which provides more information about the tools’ execution.
  
  • In the table above, we can see each tool’s usability percentage and average execution time. We can also see that Securify and Mythril take longer to execute the instructions. This is because both tools are based on EVM bytecode (which is slower to compute), while Slither and Smartcheck compile syntax and XML tree codes.
• Because of their format, both can be compiled faster than Securify and Mythril.
• Finally, a single tool cannot accurately detect all the vulnerabilities at a time. Therefore, the researchers design a selector in SuperDetector that matches the tool with the highest accuracy to the corresponding vulnerability.

Discussion and Key Takeaways

• This work provides a solution to one of the biggest threats to smart contract security–the inaccuracy in detecting all exploitable bugs and vulnerabilities.
• The solution is a breakthrough since previous approaches and tools have failed to cover the two kinds of vulnerabilities. Before the research, developers utilized the tools separately, making the detection process tedious.
• The paper also provides insights into SuperDetector, how it works, and each tool’s performance during the experiment.
• SuperDetector provides developers with more options as they can run their code in the framework and detect vulnerabilities. Hence, developers can easily and better detect smart contract vulnerabilities in their Dapps that could be exploited.

Implications and Follow-ups

• The inaccuracy of current detection methods has pushed the researchers to pursue and create an alternative detection method, SuperDetector, which combines all the detection tools.
• SuperDetector may positively affect smart contract scalability and execution once more developers discover, start using it and get results.
• The steady adoption and use of the framework may lead to the detection of more detailed vulnerabilities.

Applicability

• DeFi/NFT hacks are regular in crypto. Most of the hacks are due to vulnerabilities in smart contracts. With SuperDetector, developers can detect bugs better in their Dapps and secure the protocols against potential asset-draining attacks.

[Chrisarch]

This is an amazing research piece @Harvesto.

There have been numerous vulnerabilities on smart contracts and it feels accomplishing seeing this means of performance detection in your research summery using the SuperDetector.

The SuperDector which uses combination of different tools as illustrated herein has the ability for vulnerability coverage and detection accuracy.

@Harvesto

SuperDetector computes the tools’ performance (vulnerability coverage, detection accuracy) while they detect vulnerabilities.

Of all these tools, the researcher showed that the Mythril has 100% vulnerability coverage as shown in Table 2 and also herein

@Harvesto

The research reveals that Mythril has 100% vulnerability coverage

While Slither which is one of the tools with >94% detection accuracy,

@Harvesto

According to the detection results, Securify’s usability is about 67%, Smartcheck is about 33%, and the other three tools are over 94%.

has higher usability and lower execution time as illustrated in Table 3.

Since the SuperDetector is built around vulnerability coverage and detection accuracy while detecting attacks, why didn’t the researcher incorporate only Mythril and Slither since they are more efficient in their respective functions?

[Harvesto]

Hey @Chrisarch

Glad to read your comment on the summary.

Regarding your closing question, it may be because the researchers wanted a broader analysis to provide better context on why/how the tools work and how Superdetector is superior/detects more vulnerabilities.

That’s my opinion, but it may be because of any various reasons.

[Ulysses]

Clear writing, as always, @Harvesto.

Excited to see that smart contract vulnerability is getting enough attention as it should. This is evident in various papers published on the Forum.

Having read through this summary, I strongly believe that if this tool can deliver on its promises, it would create a tremendous impact.

By classifying the vulnerabilities into CallVUL and CallIRVUL, do the researchers mean that all smart contract vulnerabilities would always fall into either group? Or, was this done for convenience of the research?

@Harvesto

The framework regards the tools as plugins to concurrently use them all.

If the tools are regarded as plugins and will be used concurrently, is there any anticipated interference in performance? Did the research highlight that, or the authors are confident it would work perfectly okay?

[Yeoriton56]

Hey @Harvesto delighted to read you work!

Just have a question. Are there motivations for selecting the six tools used in SuperDectector? Because, I believe, there are many more tools out there.

[Harvesto]

Always a delight to see your comment, @Ulysses.

I also share that excitement, as the niche seemed abandoned for a while.

Concerning your question, from personal observation, the authors used the CallVUL and CallIRVUL classifications for the reasons you mentioned.

All smart contracts possess caller functions which are:

• The instructions they execute once “called” - the transaction is approved - and allows the smart contract to interact with other contracts

Examples of caller functions include NFT standards, EIPs, transfer of ether tokens in the blockchain, storing/managing funds, etc.

For more context, call functions contain the hash of transactions that will eventually be recorded on the blockchain. Ethereum transaction hashes are sadly predictable - meaning you can get the hash/ID of a transaction before it is executed.

This feature and each smart contract’s relationship with other smart contracts while being executed is what opens the code to hacks and exploits. Such vulnerabilities are the CallVUL types, while those caused by external blockchain factors are classified as CallIRVUL types.

I say YES on whether the smart contracts were classified for the convenience of the research.

There are 33 known types of smart contracts, but the researchers selected eight recurring vulnerabilities. Then, they used the CallVUL and CallIRLVUL classifications for better context and understanding.

Without that distinction, the paper would not read as well as it did, and some context would be lost.

Wouldn’t you agree?

[Harvesto]

Hi @Yeoriton56

I put a lot of thought into the summary. Thank you for reading through.

Yes, there are certain motivations behind the use of the six tools. As of August 2022, there are 86 known security analysis tools. That’s a lot of tools, and it would be cumbersome trying all of them to decide which ones best suit the research.

The tools used in the Superdetector were selected based on criteria the researchers set:

• They use the same analysis technology which all six do - static analysis technology.
• They can “see” and analyze the two kinds of vulnerabilities - CallVUL and CallIRLVUL.
• They accept Solidity as input
• They can cover at least three or more of the eight vulnerabilities focused on in the paper

Hope that answers your question.

[Yeoriton56]

Thanks for your reply @Harvesto . It did answer my question.

[Harvesto]

Glad to know my reply answered your question.

[Ulysses]

@Harvesto

For more context, call functions contain the hash of transactions that will eventually be recorded on the blockchain. Ethereum transaction hashes are sadly predictable - meaning you can get the hash/ID of a transaction before it is executed.

This caught my attention.There should, probably, be some way to around this. It’s like shooting oneself on the leg.

Maybe, in some way, zero-knowledge-proof can be employed to tackle this.

[Harvesto]

Like shooting oneself in the foot, indeed.

Yes, ZK Rollups could help keep hashes private. I am just wondering whether there is any possibility of a practical application in Ethereum in this context. I can only think of Zcash as a blockchain that readily solves the hash/transaction privacy problem.

[Ulysses]

I agree too, but the limitation of Zcash is that it is not a turing complete blockchain. Therefore, it wouldn’t be able to support smart contracts.

Sadly, this brings us back to square one.

[Lisayanky]

Good job on your summary. Your ability to clearly explain the material was truly enlightening.

[WaterLily]

It’s really gratifying to see this method of capacity analysis in this study report utilising SuperDetector due to the discovery of multiple bugs on smart contracts.

This SuperDector, who utilizes a mix of several techniques does have the potential of risk detection as well as based method.

The issue of Zcash would be that the crypto does not have all the necessary turns. Smart contracts couldn’t then be backed by it.

[Harvesto]

Thank you!

I’m glad you understood the subject matter.

[Harvesto]

Yes… SuperDetector’s ability to combine the tools mentioned is commendable because it ensures more vulnerabilities are detected. So far, it makes me rest assured that more vulnerabilities can be found faster and better. Although, as a side note, using tools does not determine vulnerability detection success.

As for Zcash, I look forward to seeing its privacy mechanisms implemented in a blockchain like Ethereum.

[DIVINE]

Hello @Harvesto, the paper is an interesting one, my little contribution on the subject matter and based on my understanding & experience, I feel to guarantee the security of smart contracts, it is required that one can only analyze all the options for its execution. When executing smart contracts in Turing-complete languages, you need to be sure that the computer program does not contain bugs, which is almost impossible. Therefore, when working or creating smart contracts, you will have to audit them.

Vulnerabilities in Ethereum smart contracts can have catastrophic consequences. Even though protocols like Aave are managed by professionals and regularly checked, security vulnerabilities still pose the risk of a hacker attack with the loss of crypto assets for huge amounts, thereby negatively affecting investor confidence in the protocol and subsequently causing financial losses for users/companies and price volatility.

These vulnerabilities stem from the complexity of Ethereum’s native smart contract language and its accounting system, which, unlike Bitcoin’s UTXO system, is much more flexible and thus more susceptible to additional vulnerabilities and attack vectors.

Since Solidity and other smart contract languages are new and extremely complex, it would be incorrect to blame these vulnerabilities on developers.

[Harvesto]

@DIVINE

it would be incorrect to blame these vulnerabilities on developers.

It has never been the fault of developers and protocol owners. Blaming them for hacks and exploits would be one-sided. Yes, they should have done due diligence and testing, but as you remarked, Solidity and other contract languages are very novel and as such, different parameters are unknown.

There is no one-for-all solution to Solidity-related attack vectors other than developers iterating forward and learning more about the code they work with.

Bitcoin’s UTXO makes it more resistant to such headaches but the blockchain is Turing-incomplete by design which removes the functionality of smart contracts. Of course, developers have found a way around it with Stacks which means the blockchain can now support smart contracts. Let’s see how that goes.

It is important to note that there are more exploits to come… What this space need is more smart contract security analysts, engineers, etc. Plus a solution/mechanism that can stop or counter hacks as they happen/in real-time.