TLDR
- Finding a reliable blockchain performance benchmark is difficult because each blockchain paper uses its own standards for benchmarking. In addition, there are no existing blockchain benchmarking tools for multilayer blockchains.
- The concept of a “parameter space” is defined to bring a consistent set of standards for benchmarking blockchains. A parameter space provides a comprehensive view of a blockchain testing environment, including both hardware and software parameters.
- Parameter Space Exploration, the design proposed in the paper, is released in open-source code and tunable using the configuration file provided.
Citation
Tsai, D. Y., Sie, M. F., Harding, S. A., Wu, J., Liao, S. W. (2021, May). Testbed Design and Performance Analysis for Multilayer Blockchains. In 2021 IEEE International Conference on Blockchain and Cryptocurrency (ICBC)(accepted). IEEE. Testbed Design and Performance Analysis for Multilayer Blockchains | IEEE Conference Publication | IEEE Xplore
Background
- Multilayer blockchain: This type of blockchain is designed to bypass the limitations of a conventional single-layer blockchain in terms of throughput, scalability, security, and other attributes. Multilayer blockchains are typically composed of a layer-one blockchain (the main chain) and a layer-two solution (the side chain).
- Permissioned blockchain: This type of blockchain uses an access control management system that allows only for verified nodes to access the blockchain network.
- Fault Tolerance: This term refers to when each participating node in a blockchain network may continue validating transactions without interruption even when one or more nodes fail (“fail” can include crashing, losing connection, etc.). One such example would be PBFT tolerance, in which 3F+1 nodes may tolerate at most F faults.
- Byzantine Fault Tolerance (BFT): BFT-based systems are able to resist the class of failures derived from the Byzantine Generals’ Problem; the network may continue to operate normally even when some of the nodes fail or act maliciously.
- Practical Byzantine Fault Tolerance (PBFT): These systems have higher efficiency than BFT systems and reduce consensus complexity from exponential to polynomial. With F malicious nodes in a network of N total nodes, consensus may be successfully reached under the following condition: F ≤ (N-1)/3.
- Crash Fault Tolerance (CFT): CFT-based systems allow for nodes to crash or disconnect from the network while still properly maintaining the system. Some examples include Paxos and RAFT. CFT-based consensus is less secure than BFT-based consensus, but it has reduced network complexity.
- Evm-lite: This lightweight Ethereum node has a consensus architecture that can be combined with Tendermint to make a complete blockchain system.
- Quorum: This permissioned blockchain allows for private, fast transactions between nodes. Quorum uses RAFT and Istanbul algorithms to achieve consensus, and it uses Tessara and Constellation to encrypt and share node data.
- Tendermint: This middleware provides an improvement from the PBFT algorithm. In a network of N nodes, at most F error nodes can be tolerated, where N = 3F+1. There are three types of voting in Tendermint: prevote, pre-commit, and commit. When a block is committed by the entire network, the block must have been signed and broadcasted by more than two-thirds of the verification nodes in the entire network.
- Stellar: Stellar is a platform that connects three entities: users, banks, and payment systems. Stellar implements a system that allows for low currency exchange fees, allowing for low-cost cross-border transactions.
Summary
- Other works almost exclusively focus on comparing efficacy of different consensus algorithms, but not entire blockchain systems; in addition, many blockchain performance analysis papers focus on the analysis of a single system rather than side-to-side comparison. Other performance benchmarks also do not use the concept of a parameter space. In the work “Blockbench”, the authors provide a comprehensive performance analysis of private, single-layer blockchains.
- There are two key elements in this paper: (1) the proposed Parameter Space Exploration (PSE) mechanic and (2) candidate multilayer blockchain systems that PSE tests on.
- Multilayer blockchain systems are designed specifically to increase throughput while maintaining the system’s integrity.
Method
- A parameter space, as defined by the authors, is composed of “Parameters” - the blockchain system parameters, and the “Hardware” - the hardware on which the blockchain system runs.
- A parameter space can consist of block size, block time, transaction size, and other important parameters together with the hardware that the system - whether a blockchain or a consensus engine - runs on.
- Parameter Space Exploration (PSE) leverages the concept of a parameter space, defining a consistent testing environment for comparison among multiple parameter spaces. PSE’s architecture is illustrated in the figure below; PSE measures performance of blockchain systems with the interface layer and a config file for parameters. Users define the interface layer and may modify system parameters via the config file.
- Two parameter spaces could have different consensus mechanisms or levels of decentralization. The authors assert that accurate and coherent benchmarks may be obtained through their proposed architecture.
- Metrics used in experimentation include throughput, latency, and input rate from the user’s perspective. Throughput is taken as the average TPS (transactions per second) of all nodes in the network. Latency is taken as the average time between proposal and confirmation among all transactions.
- In the experimentation, a dedicated node is set up to send transactions, simulating real-world transactions.
Results
- Based on Figure 4, the authors conclude that at lower transaction frequency, Tendermint and IBFT have similar performance, and at higher transaction frequency, Tendermint’s performance is stable while IBFT’s performance decreases. For these reasons and the below observations, the authors choose Tendermint on evm-lite and Quorum-Raft for the side chain.
- Stellar-4’s throughput increases significantly when transaction rate is between 100 and 400 TPS, and as it surpasses 400, latency begins increasing.
- Raft, a CFT-based blockchain, attains the highest performance with throughput of 600 and transaction rate of 1200.
- Stellar, a BFT-based blockchain, attains peak performance at 350 TPS, 10 second latency, and input rate 400.
- Tendermint, a BFT-based blockchain, attains peak performance at 470 TPS, 5 second latency, and input rate 500.
- IBFT, a BFT-based blockchain, attains peak performance at 500 TPS, 5 second latency, and input rate 500.
- Based on Figure 5 (a), the authors find that as the number of nodes increases and as the network reaches max capacity, the throughput of Tendermint on evm-lite decreases.
- Based on Figure 5 (b), the authors find that Tendermint’s latency increases as its throughput decreases.
- The paper proposes a set of design choices with both power efficiency and high performance: Quorum-Raft and Tendermint-evmlite.
- After comparison among throughput, latency, and scalability, the authors provide the following conclusion: with no malicious nodes in the system, Quorum-Raft is the best choice because it achieves the highest performance and is the most stable; with malicious nodes in the system, Tendermint-evmlite is the best choice because Tendermint achieves the highest performance among the three proposed BFT-based consensus algorithms.
Discussion and Key Takeaways
- Parameter space: The authors define the concept of a parameter space that accounts for hardware and software used in blockchain benchmarking.
- Parameter Space Exploration: This is a system design built on the concept of a parameter space designed to benchmark multilayer blockchains more accurately.
- Using Parameter Space Exploration, we may find design choices that adhere to the application scenario designed by the authors: a selection among different systems that have both high performance and high power efficiency. The selection criteria defined also takes into account security requirements.
- In future work, the authors plan to detail the tradeoffs between Raft and other Ethereum blockchains. They also plan to create multiple different configurable parameter spaces.
Implications & Follow-ups
- The work most similar to Parameter Space Exploration is “Blockbench” which provides a systematic framework for performance analysis for private blockchains.
- Both Parameter Space Exploration and Blockbench can be used to identify and solve performance bottlenecks by measuring latency, throughput, and scalability.
- Both papers provide the framework with which to make performance comparisons while showing these frameworks in action via their experiments.
- The blockchains that are chosen by these papers are not the only blockchains you can compare using their frameworks; Parameter Space Exploration’s open source code and configuration file can be edited to compare a different set of blockchains.
Applicability
- The consensus algorithm chosen in a blockchain system is a crucial determining factor for its power consumption.
- This paper provides a detailed discussion on a multilayer blockchain’s performance, power consumption, and security.
- Multilayer blockchain systems are becoming ever more relevant in both industry and academia due to their strong potential in targeting the problems posed by the CAP Theorem. The proposed Parameter Space Exploration can be used to compare different multilayer blockchain systems and performance of their respective consensus algorithms through an adjustable configuration file.