Research Summary: A Survey of Verification, Validation, & Testing Solutions for Smart Contracts

[LTTOguns]

TLDR

• Developing and deploying smart contracts comes with high stakes. Therefore, VV&T is a particularly important step in smart contract development lifecycles.
• The researchers surveyed and categorized VV&T solutions into four - public test networks, security analysis tools, blockchain emulators, and blockchain simulators.
• This paper elaborated on these four categories and further analyzed them by their key benefits and drawbacks.

Core Research Question

How do we categorize verification, validation, and testing (VV&T) solutions, and what would the benefits analysis of each solution be?

Citation

Benabbou, C., & Gürcan, Ö. (2021). A Survey of Verification, Validation, and Testing Solutions for Smart Contracts. In 2021 Third International Conference on Blockchain Computing and Applications (BCCA) (pp. 57-64). IEEE. Available at: https://arxiv.org/ftp/arxiv/papers/2112/2112.03426.pdf

Background

• Verification: Ensuring the software artifacts means its specifications.
• Validation: Ensuring the product fulfills stakeholders’ (e.g. users’) needs.
• Testing: The process of examining software to eliminate errors.
• Smart Contract Vulnerabilities: Exploitable smart contracts weaknesses. The authors categorized them into three:
  • Vulnerabilities at the source code level
  • Vulnerabilities that occur when smart contracts interact with contracts or other entities
  • Vulnerabilities caused by the blockchain system
    

Summary

• Effective implementation and testing are vital, given that deployed smart contracts are definitive
• Smart contracts development typically goes through the following lifecycle
  • Iterating between “Analysis and design” and “Implementation and testing”
  • Deployment
  • Execution
  • Termination
• The study considers the need for a standardized process and best practices for testing smart contracts.
• This is one of the first surveys to cover all existing VV&T solutions for smart contracts.

Method

• The authors group VV&T solutions into 4, with each type possessing distinct properties.
• They compared these solutions by reviewing the different tools that apply to different blockchains, languages, and vulnerabilities and the effectiveness of VV&T if the tools were used together. They considered 10 public test networks, 20 security analysis tools, 10 blockchain emulators, and 1 blockchain simulator.
• They finally presented open questions and challenges in the field as potential directions for future research.

Results

• 4 types of VV&T Solutions:
  • Public Test Networks (“testnet”): Networks publicly available to developers and testers to mimic, assess, edit, or upgrade their smart contracts and protocols before deploying them on the Mainnet.
  • Security Analysis Tools: Programs used to evaluate and question an entire network framework using specific parameters to determine issues and vulnerabilities.
  • Blockchain Emulators: A virtual environment to imitate the blockchain network that can be reproduced locally. Emulates the complete environment of the main network features.
  • Blockchain Simulators: A virtual environment to imitate the blockchain network that can be reproduced locally. Simulates only desired settings of the main network, which is configured by the testing developer.
• An overview of VV&T solutions:
  
  
  • Target Blockchain (Bitcoin, Ethereum, Hyperledger Fabric, Tendermint/Cosmos, Others)
    • Test networks: Each public test network is dedicated to a specific blockchain test network.
    • Static analysis tools: Most security analysis tools support only the Ethereum blockchain.
    • Emulators: Also typically dedicated to a specific blockchain
    • Simulators: There is only 1 simulator on the market: Gauntlet. It supports Ethereum and Tendermint/Cosmos.
  • Smart Contract and Test Languages
    • Test networks: In the same testnet, the supported language used for writing smart contracts and that of writing tests can be different (e.g. Solidity for writing smart contracts and Javascript for writing tests), but there are also testnets that offer more options.
    • Static analysis tools: Most security analysis tools support Solidity as the smart contract language.
    • Emulators: Dedicated languages for target blockchains.
    • Simulators: Python is supported on Gauntlet.
  • Vulnerabilities
    • Static analysis tools: They offer pre-defined vulnerabilities that could be further improved with plug-ins. However, thy can not detect vulnerabilities in the blockchain system scope.
    • Test networks, Simulators: Manual VV&T solutions available.
    • Emulators: Manual & Automated VV&T solutions available.
• The authors also compared the parameter configurations available in different solutions.
  
  

Discussion and Key Takeaways

• There are more security analysis tools than any of the other three types of VV&T solutions.
• Among all the surveyed blockchains, Ethereum has the largest number of VV&T tools available.
• In contrast, the authors noted that Bitcoin, despite being the most well-known crypto asset, has only one VV&T solution (a public test network).

Implications and Follow-ups

• The study highlights community volunteering as an essential target for smart contract projects. The communal engagement would enable volunteers to conduct more tests and, therefore, a higher probability of detecting various vulnerabilities.
• On the other hand, the study reveals confidential issues for businesses where information such as sensitive data and confidential algorithms being exchanged on public test networks would pose a risk. Consequently, community testing would not be suitable in this instance.
• Simulators appear to be the most effective option however, the simulators currently lack the capabilities to be explored on various blockchains, which restricts their efficacy.
• The study proposes a hybrid or mixed-use of VV&T solutions to ensure the reduction of vulnerable patterns in smart contracts.

Applicability

• Smart contracts play an integral part in executing transactions on blockchain networks, and testing is critical; consequently, they need to test functionally and methodically. The consideration is mostly for easier testing of smart contracts for developers and testers.
• The study gives an insight into the number of VV&T solutions available, the categorizations, and issues and vulnerabilities proliferating different networks.
• The solution adopted is a standard model for designing, executing, and recommending a combination of proven VV&T solutions that allow developers and testers to push the boundaries of the discovery of unusual vulnerabilities, thereby producing more secure smart contracts.

[Stallonaking]

The blockchain is a system that functions independently of a third party and is transparent and safe for storing and sending information.
Ethereum developed the idea of smart contracts, or immutable code on the blockchain that is automatically executed after certain circumstances are satisfied between parties who may not necessarily trust each other, in order to make the blockchain a general-purpose solution. Blockchain technology has quickly risen to popularity thanks to smart contracts, with applications in many different fields. By automating claims when specific events occur, they can enhance the insurance process and improve supply chains. Applications for business, trade, and governance continue to expand. Best practices must be followed while putting this code into effect because of the vast range of smart contract adoption.
since clever

[Desotech]

Kudos to you dear, your writeup is really educative.

[Mansion]

Thank you very much for this insightful summary. Although, I have a question.

Are there factors to consider for selecting application security Testing Tools ?

[Mansion]

@LTTOguns

Sorry I didn’t mention you on my post above.

[DIVINE]

Hello @LTTOguns, nice work, I understand that this paper tries demonstrate the need for Smart contract to be tested in an effective and
systematic manner before it can be carried out. It true that Smart contracts benefit from a widespread interest because of their huge potential, I think the reason for testing is because smart contracts are immutable, you won’t be able to fix them after releasing them. Many smart contracts handle high-value transactions. Even if you create a smart contract as a small passion project, you may have people using it to trade millions of dollars worth of assets. Minor flaws in your code could result in large sums of cryptocurrency or valuable NFTs being lost or stolen. Smart-contract testing can highlight vulnerabilities and help you avoid significant losses. But these variety of testing solutions available make it
confusing and difficult as to where to start working with them.

I think in order to improve the VV&T process,
setting up an effective method that compose between different VV&T solutions could be a better and significant approach to
secure smart contracts.

[Hybridspark]

Solutions for verification and validation of blockchain-based software applications have started receiving equal attention as blockchain technology gains traction in business and society. Prior to deployment, it is crucial to conduct systematic Verification and Validation to check that BC-Apps meet all of their functional and non-functional requirements. Blockchain-based software development is still a developing research field, therefore best practices and tools for the Verification and Validation of BC-Apps are not yet sufficiently developed. Research tries to overcome the issues of designing BC-Apps by providing testing approaches and tools. We offer a thorough analysis of Verification and Validation methods for BC-Apps. We specifically synthesize Verification and Validation tools and approaches addressing various components at various stages of the BC-App utilizing a layered approach.

[Humphery]

Hello @LTTOguns These solutions can be categorized in a number of different ways, depending on the specific context and needs of the project.

Some common categories for VV&T solutions include:

1. Static vs. dynamic: Static VV&T methods involve analyzing the product without executing it, while dynamic methods involve executing the product and observing its behavior.

2. White-box vs. black-box: White-box VV&T methods involve analyzing the internal structure and design of the product, while black-box methods involve testing the product from the perspective of an external user or system.

3. Manual vs. automated: Manual VV&T methods involve human testers performing the testing, while automated methods involve using software or tools to perform the testing.

The benefits of each VV&T solution will depend on the specific characteristics of the product and the goals of the testing. Some common benefits of VV&T solutions include:

• Finding and fixing defects before the product is released
• Ensuring that the product meets specified requirements
• Improving the quality and reliability of the product
• Reducing the risk of costly failures or defects in the field
• Improving customer satisfaction and confidence in the product

I hope this helps! Let me know if you have any other questions.