Research Summary: DevOps for Ethereum Blockchain Smart Contracts

TLDR

  • A DevOps approach and appropriate tools could remedy the difficulty of developing smart contracts at scale.
  • Before this paper was released, no structured guidance or detailed solutions for applying DevOps to blockchain-based software development existed.
  • The processes and tools of continuous integration (CI) and continuous deployment/delivery (CD) for smart contracts are illustrated with an analysis of the difference between DevOps for general projects versus blockchain-based applications.

Core Research Question

How can DevOps approach improve blockchain-based software development? What DevOps tools are available and how do they relate to blockchains?

Citation

Maximilian Wohrer and Uwe Zdun, “DevOps for Ethereum Blockchain Smart Contracts,” University of Vienna, Faculty of Computer Science, Research Group Software Architecture, Vienna, Austria, 2021. Available: http://eprints.cs.univie.ac.at/7141/1/document.pdf

Background

  • DevOps: Refers to the combination of software development (Dev) and operations (Ops). It focuses on fast-paced product development and delivery. Teams with a DevOps culture cooperate by using technology stacks and tooling to replace manual operations with automatic processes. This helps them independently accomplish tasks that typically require help from other teams. There are several benefits of DevOps, including rapid development and delivery, scalability, and improved collaboration, reliability, and security.
  • Continuous integration (CI): The goal of CI is to optimize the quality of software production by allowing a faster and more automated process of detecting and fixing errors in code.
  • CI flow: The process of continuous integration, which involves developers changing and testing the code with a local copy and then submitting a merge request, followed by a series of automated tests and eventually integration into a shared code repository.
  • Continuous deployment/delivery (CD): CD refers to a DevOps practice where deployment pipelines deliver updated configurations and code to environments. The core concepts of blockchain solutions are the same as non-blockchain solutions, i.e., involving a virtual machine, container, or serverless function, or an infrastructure as code (IaC) approach for provisioning these environments.

Summary

  • Previous works have mentioned that DevOps could be a useful approach to facilitate the process of blockchain-based software development, but there is still a lack of detailed discussion about its breakdown and applications. This is why the authors tried to build the first systematic approach for it in this paper.
  • Considering the topic of this research mainly concerns field applications and there are many practical knowledge reports provided by practitioners, the authors decided to take a pattern derivation approach which is built by applying grounded theory (GT) techniques.
  • CI/CD processes are the main parts of DevOps practice. The authors divide them into stages to discuss their findings, including various tools that support these stages.
  • The CI/CD process starts with CI, which aims to lift the speed and quality of code updating and testing from multiple developers with automatic processes in the build pipeline.
  • This is followed by CD, which allows deploying updated code and configuration to one or several host environments at any time.

Method

  • Using the pattern derivation approach, the authors define a pattern as the conceptual equivalent of best practices, where GT techniques are used to discover and codify the patterns.
  • The authors collect and combine gray literature and typical CI/CD configuration files for smart contracts from Github and the major search engines.
  • To identify candidate patterns, labels and optional memos were used to analyze, explain, and establish conceptual relationships among materials. The authors conducted constant iterative pattern comparisons until they reached theoretical saturation: i.e., until adding new sources no longer revealed new insights.

Results

  • CI stage: Several established CI solutions have provided sufficient tools to build and test smart contracts. They have also been used in practice. CI and tools for blockchain-based applications encompass the phases of Coding, Building, and Testing.

  • Code phase: This phase mainly focuses on conducting core development tasks within integrated development environments (IDEs) supported by proper plugins and frameworks. The general design principle of software coding, simplifying the code, is especially applicable for smart contracts. They should be written to focus on a single task or capability. The number of on-chain transactions also needs to be minimized. For general concerns, production-tested library contracts, or standardized contract implementations should be used. Contracts should be developed and tested by locking pragmas with a fixed compiler version. Also, all public contract interfaces should be fully annotated with specially tagged comments in the NatSpec format.

  • Build phase: This phase mainly focuses on including all steps for generating artifacts for execution from source code. Two compilers, solc, and solc-js (both use the same compiler source code) are used by Solidity to generate the artifacts to interact with the EVM bytecode. To interface with the Solidity compiler, the JSON-input-output interface is recommended. A preprocessor before compilation may be required in some situations. Raw source files are better kept in a separate directory and then the preprocessor is run to output the code to the source directory of the pipeline before compiling the source.

  • An overview of all possible test types, though discussion of them is beyond the scope of this paper. (Fig. 1)

  • Test phase: This phase mainly focuses on automatically deploying the code in a test environment to conduct a series of automated tests. In this phase, it is useful to separate software’s separable components and test purposes. Testing environment, Test Data, Unit and Integration Tests, Static/Dynamic Analysis, and Reports are respectively discussed below.

  • Testing Environment: For testing, smart contracts need to be deployed in a blockchain environment. A (temporary) local (in-memory) blockchain for testing and development purposes that simulates the characteristics of a real blockchain network is preferred.

  • Test Data: The testing blockchain can be equipped in one of three ways. The first is an empty blockchain without any transaction history; the second is a blockchain filled synthetically; the third is a fork of a production blockchain.

  • Unit and Integration Tests: To check the correctness of blockchain software, unit tests are the most common technique, followed by manual code reviews. Unit tests should cover all contract methods. Each test case should be executable in isolation without relying on the state that is imposed by other test cases. Integration tests may involve interactions between complex scenarios with multiple calls between different dependencies of a single contract or across multiple contracts, oracles, and front-end client applications. Solidity and JavaScript/Typescript are two options for writing automated test code. Also, some frameworks resolve the cumbersome mock situation.

  • Figure 2 illustrates the structure for Solidity and JavaScript-based testing. Where an API is used, the logic implementation can be tested externally (including the transmission path) or directly internally.

  • Static/Dynamic Analysis: Static analysis is aimed at uncovering potential vulnerabilities in the code with heuristics, without actually executing it. A linter tool is typically one of the first applied measures. VCS hooks can be used to conduct execution before each commit. Some advanced tools extend the static analysis and are commonly used to detect security vulnerabilities, such as dynamic analysis. There are some special test tools for smart contracts as well, such as fuzz testing and (code) mutation testing.

  • Reports: An important metric is test coverage. There are code coverage tools for Solidity that are run as a separate CI job, as well as gas reporter tools. It is better to combine different analysis tools to uncover as many existing errors as possible. Another important metric is tracking gas consumption. Tools are useful to generate gas reports for smart contracts code iteration.

  • CD stage: CD and tools for blockchain-based applications encompass the phases of Releasing, Deploying, Operating, and monitoring.

  • Release phase: A manual approval process that allows only a few key individuals to authorize a release for production is preferred. For crucial impact smart contracts, at least two independent security audits by at least two organizations should be implemented before the deployment. Provision of shared central archiving that collects and stores all artifacts generated for the deployment is also suggested.

  • Deploy phase: In this phase, updated and tested code pushes release builds into a production environment. Several tools to automatically facilitate the process in various contexts are discussed in the paper. These contexts include IaC, Smart Contract Deployment, Upgradeable Smart Contracts, and Testnet.

  • Operate phase: For blockchain software that is built on permissionless blockchains, there are tools that are useful for automatically conducting public verification of the deployed contract in the CD process.

  • Monitor phase: This phase uses various metrics or events to detect erroneous or suspicious behavior of smart contracts. There are several ways to monitor this behavior, such as Blockchain Explorer, operating one’s own blockchain explorer, running a dedicated blockchain node, and implementing a smart contract activity tracker that sends JSON RPC requests to scan data. Service providers and monitoring solutions also serve these tasks.

  • To demonstrate a holistic DevOps approach, the authors build a smart contract sample project (Maxwoe/sc-devops) based on the Hardhat development framework and implement a GitLab CI/CD pipeline.

Discussion and Key Takeaways

  • Due to the decentralized and immutable nature of blockchains, there are some peculiarities of DevOps for blockchain-based software.
  • In the design of smart contracts, the basic principles include reducing complexity, focusing on a single task or capability, minimizing the number/size of on-chain transactions/writes, and the dependencies required for testing.
  • A greater focus on testing is recommended, particularly for the use of static and dynamic code analysis. When testing is conducted, understanding interfaces and communication points (much like API) is crucial.
  • As the deployment of smart contracts is currently a limited and delicate undertaking in which developers want to have tight control, the most often observed situation is manually triggered deployment. Therefore, CD here refers to continuous delivery rather than continuous deployment. Measures for continuous deployment are not commonly used at the moment.

Implications and Follow-ups

  • This is the first structured approach and breakdown of specifics regarding a DevOps approach to blockchain-based applications.
  • The authors describe DevOps in the context of Ethereum and Solidity, but point out that the concepts and basic practices presented should be easily transferable to other platforms.
  • The biggest challenge is to specify the necessary test and deployment requirements and to select and orchestrate the appropriate tools.
  • Devising testing strategies that combine the various techniques and tools meaningfully to integrate DevOps with blockchain-based applications could be the focus of future works.
  • Other future works could include the implementation of continuous deployment for smart contracts.
  • The authors built a sample project (Maxwoe/sc-devops) to demonstrate a holistic DevOps approach for smart contracts.

Applicability

  • DevOps principles and tools can be applied to blockchain-based applications. Doing so will help the industry improve its development and operations, making the production process faster, more reliable, more collaborative, and more controlled.
11 Likes

Thank you for posting this summary. From the summary, it seems like there is good reason to adopt a DevOps approach and tooling. Having worked on this summary, what do you think are some good steps for people to take when choosing tooling?

I’m also curious about the mention of Ethereum in the paper and title. Are the authors focusing on Ethereum because they happened to be using smart contracts as their research focus, or is there something inherent to Ethereum networks that are more advantageous for a CI/CD approach?

5 Likes

I think we can choose tools by the steps as follows:
[Step 1] Decide the stage and phase: As the paper mentioned, there are two stages(CI & CD) and several phases. It provides corresponding tools for each phase. So the first step should be finding out the right point we are standing.
[Step 2] According to our development features, see the tools that the paper advised.

About your second question, I think both are the reasons. The paper focused on discussing smart contract’s characteristics in some paragraphs. Also, when this paper was written, the ecosystem of the Ethereum networks was relatively completed. There were more tools for the Ethereum networks for the authors to conduct their Grounded Theory (GT) techniques to finish the research.

4 Likes

@Astrid_CH Thank you for this well-structured research summary.

Are there any similar principles for non-blockchain vs. blockchain-based software?

4 Likes

Remaining the code and structure as simple as possible is a general principle. Though this principle also applies to non-blockchain-based software, it is particularly important for blockchain-based software as the cost of deploying on blockchain again may be huge, keeping code simple could reduce mistakes, so that to diminish the cost amount and the possibility of deploying again.

5 Likes

An excellent summary, @Astrid_CH. I am curious what existing tools or frameworks could enable software engineers to implement DevOps with ETH blockchain. If not, what tools or ideas have been proposed in the community to help the developers exercise DevOps with ETH Blockchain?

4 Likes

Some tools or communities are mentioned in the paper:

  1. CI: Jenkis, Travis CI, Gitlab CI/CD, GitHub Actions provide means to build and test smart contracts. Smart contract development frameworks are also essential in a CI approach, such as Truffle, Hardhat, Embark, Brownie, Waffle.
  1. Code: Remix IDE. Production-tested library contracts (e.g., OpenZeppelin) and standardized contract implementations (e.g., ERC-20) should be used for general concerns.
  2. Build: there are two compilers, solc and solc-js. The recommended way to interface with the Solidity compiler is the JSON-input-output interface. In some cases, a preprocessor (e.g., solpp) may be necessary to use before compilation.
  3. Test
    a) Testing Environment: a (temporary) local (in-memory) blockchain for testing and development purposes (e.g., Ganache) is used to simulate the characteristics of a real blockchain network.
    c) Unit and Integration Tests: Toos like truffleassertions and OpenZeppelin Test Helpers are used to check the expected execution of reverts and event emitting. For writing automated tests for Ethereum, two main options are Solidity and JavaScript/Typescript. For JavaScript tests, established testing utilities such as the Mocha testing framework paired with Chai as an assertion library are used to test smart contracts asynchronously. Some frameworks have also addressed the cumbersome mock situation of Solidity tests and created mocks dynamically within the test code (e.g., Waffle, MockContract). Some frameworks also allow executing tests in parallels, such as OpenZeppelin and Truffle.
    d) Static/Dynamic Analysis: some more advanced tools beyond a linter that extend on static analysis and are commonly used to detect security vulnerabili-ties (e.g., MythX, Securify, SmartCheck, Slither, Manticore, Mythril).
    e) Reports: soliditycoverage and sol-coverage are code coverage tools for Solidity. On the other hand, gas reporter tools (e.g., eth-gas-reporter) are helpful for examining cost.
  1. CD:
  1. Release: at least two from different organizations (e.g., Consensys, OpenZeppelin) are recommended for conducting independent smart contracts security audits.
  2. Deploy:
    a) IaC: Terraform, Ansible, Puppet, and Chef are some IaC utilities on the market to automate the provisioning process, but they did not come across in the research.
    b) Smart Contract Deployment: Truffle Migrations, Hardhat Ignition can be used to automate the necessary steps for linking contracts to other contracts and populating contracts with initial data.
    c) Upgradeable Smart Contracts: the EIP-2535 formulates a standard for building modular smart contract systems that can be extended in production. OpenZeppelin’s Upgrades Plugins can be integrated into existing development environments and workflows.
    d) Testnet: Goerli(PoA) with a block time of 15 seconds. Rinkeby(PoA) and Kovan(PoA) with fewer client support and a block time of 15 respectively 4 seconds. Ropsten(PoW) ,a block time of under 30 seconds.
  3. Operate: truffle-plugin-verify and hardhat-etherscan allow the public to audit and verify the code.
  4. Monitor: For simple checking purposes, a block explorer (e.g., Etherscan, Etherchain, Blockchair) can be used. BlockScout, and Expedition can be used to operate one’s own blockchain explorer. There are also implementations for private EVM-based networks (e.g. Ethernal). Some service providers (e.g. Tenderly, OpenZeppelin Sentinel, Parsiq) run a dedicated blockchain node and implement smart contract activity trackers that provide easy monitoring approaches.
5 Likes

Hello @Astrid_CH

This is an excellent summary because it speaks directly to the development of Dapps using traditional DevOps principles. In the research questions, it was mentioned how the DevOps approach can improve blockchain-based software development. Although the process flow for DevOps software development was highlighted, there was no actual comparable mention in the analysis. Is it a kind of enhancement process? Or do I miss it?

Furthermore, I believe the research focuses more on the process and tooling requirements for DevOps in building applications, do clarify this thought.

Finally, the team set up for DevOps is usually based on organizational preference due to size and scale. What is your take on standardization across industries?

5 Likes

Excellent summary at @Astrid_CH. Glad that DevOps practices and tools are now being deployed in Ethereum Blockchain development.
The role played by DevOps tools is crucial.The majority of the deployment process can be automated using the latest DevOps practices, which also provide us complete control over our services and an infrastructure that is easy to scale.
Assume that developing a complex infrastructure for Ethereum blockchain makes it difficult to manage because we are essentially building a black box, making it difficult and time-consuming to repair errors.
I believe that the following DevOps tools like Ansible, Terraform, Prometheus and Grafana, Docker will help to speed up Ethereum Blockchain development.

1 Like

Exciting times ahead!

Thank you for this interesting summary @Astrid_CH

A developing trend in blockchain technology is smart contracts. And it’s fascinating how the challenge of creating smart contracts at scale could be overcome by a DevOps approach and the right tools. DevOps teams can learn a lot from blockchain technology, particularly the usage of smart contracts.

But does DevOps already utilize Blockchain?

Good job on summary, interesting research
Buh Is Ethereum the best Blockchain for smart contracts

1 Like

what an excellent summary @Astrid_CH thanks for this research

I’m happy that Blockchain solutions benefit from the DevOps methodology as it enables rapid iterations or continuous Integration and continuous deployment. It enables circular development and implementation while maintaining the integrity of your distributed ledger.

I think a DevOps team may configure the computing environment, test use cases, and shorten time to market by weeks or months by using purposes-built DevOps tools such as:

  • Terraform — Create AWS instances and manage configurations
  • Ansible — Configure environments and execute scripts
  • Docker — Launch blockchain nodes
  • Prometheus and cAdvisor — Monitor the activity of network nodes
  • Grafana — Visualize Prometheus data

Hi Astrid, thanks for doing this research and I also have been trying to wrap my mind the idea of a DevOps practice around Smart Contract and Web3 development.

What I have come up with so far is I think a DevOps methodology would work seamlessly for Web3(Front End) applications as it works in Web2.

I think the biggest issue in DevOps would be the security issues that would arise in deploying a smart contract the same way we deploy a Web3/Web2 application which is always Frontend.

The question that has alluded me from not implement or recommending DevOps for Smart Contract is the secret key or private key custodian process.

In most DevOps environment all secrets will be integrated in some sort of secret manager and the build server would pull that key(secrets) from the secret manager(ie Hashicorp Vault, AWS Secret Manager).

The question that alludes me is would that work for smart contract when the secret manager administrator would now have access to the secret key and without that how would we deploy the smart contract.

In any event I think it is a great idea for a single developer but if we have funds to protect I think it would cause more financial losses.

Thanks again for the research.