Mini-post: The Art of Auditing

CTA: In these threads, we attempt to further the discussion of a key problem in this category and evolve our understanding of the domain space where research work has not yet answered the specific problem or question being considered. These posts are living documents, and it is our hope that the community will continue to contribute to their structure and content.

Key Problem / Topic Area

This post discusses the basic approach to a security audit, using Ethereum smart contracts as an example.

Specific Question or Problem Statement

How should one perform code audits?

Background

  • An audit is an independent assessment of a project, seen through the lens of a particular concern. Examples of these concerns include security, compliance, economics, etc.
  • In the crypto space, audits have largely focused on security, with their primary goal being to assess correctness (with respect to a specification), as well as to identify potential vulnerabilities in the implemented code. These are known as code audits.
  • A code audit assesses the security quality of a target code, typically as a means to convey trust to the community at large.
  • As in any other business-to-business relationship, an audit has an agreed timeline and scope, and its findings point to specific improvements in the code, ranked according to a severity scale.
  • NOTE: If you are interested in improving the understanding of what is an audit, please visit the ā€œWhat is an auditā€ thread. Most of the definitions herein presented stem from that post.

Approach / Methodology

Essentially, a code audit relies on two things: automation and manual review of the target code.

Automation concerns the use of push-button scanners that automatically attempt to discover pre-defined bug types. The latter includes arithmetic overflow/underflow, reentrancy cases, assertion failures, transaction order dependencies, etc. For auditing of Ethereum smart contracts, two main tools are widely used: Slither and Mythril. These tools simulate the execution of the target smart contract, attempting to find execution paths matching specific issues. In the realm of formal verification and testing, the underlying mechanism used in such simulations is known as symbolic execution.

Slither and Mythril (and symbolic execution tools alike) can only identify a set of generic issues that are independent of the contract logic and business rules. Most of these tools also report a significant number of false positives (which get easier and faster to spot the more experienced an auditor becomes). Hence, findings are limited in scope. Human reviews generally aim to inspect the code in a line-by-line fashion, attempting to find issues dependent on the contract implementation and underlying business requirements. The quality of such reviews directly reflects the experience brought by the audit team.

Unfortunately, the road to learning how to do a quality manual code audit is not straight. This is due to the fact that a generic and deterministic audit script (proprietary or not) does not exist; manual code inspection and the creation of an underlying mental model of the code and its related issues depends on human abstraction, which in turn depends on subjectivity and training. Hence, outputs vary from auditor to auditor.

The best approach to becoming an auditor is to learn by doing. For training purposes, I recommend the following resources:

From those, an aspiring auditor should be able to memorize the major issues in smart contract development.

As a general yet non-deterministic guideline, an auditor should at the very least be able to perform two steps when manually auditing a smart contract:

  1. Given a specification (e.g., white paper, internal docs, external docs), an auditor should verify that the code is doing what the specification states. The goal is to check whether the code is in line with the business requirements. This is an intensive and tedious task. Essentially one reads the code, understands it, and makes sure it meets the given specification.

  2. From the previous step, the auditor should have been able to glean a mental model of the target code. The next step requires more insight. It involves investigating how a user might put the pieces of a contract together in a way that the original developers never intended. Here an auditor needs to have (or develop) an attackerā€™s mindset. This comes with experience. A good way to bootstrap that mindset is to read post mortems and replay the attack itself. Having built a mental model of the target code, one starts investigating potential threats, by means of asking questions like:

    • Is it possible for any actor to put the contract in an unusable state?
    • Is there a sequence of transactions that could cause the smart contract to misbehave?
    • Are funds secure?
    • Are there underlying assumptions made by the contract that may not hold?
    • Are there interactions with external contracts? If so, are they trusted? If not, how could they exploit the current code?
    • etc.

Conclusions / Key Takeaways

While there is no single prescriptive approach, there are a few basic strategies one can take to start a smart contract audit. Additionally, in the context of having general strategies, it becomes unlikely that any two auditors will find the same set of issues.

CTA: Future Work / RFP

It will be useful to continue to add to the list of auditing tools and strategies to ensure that auditors build upon each othersā€™ experiences.

6 Likes

We tend to hear about audits and auditors after a major failure. How often do projects fail their audits? What happens then?

5 Likes

@jmcgirk Thanks for your interest in this post. To answer your question properly, could you please elaborate what you mean by ā€œprojects fail their auditsā€?

4 Likes

I meant something along the lines of ā€˜how often do projects have major security issues that slip past auditors unnoticedā€™ but please feel free to interpret it how you like ā€“ Iā€™m trying to understand what role auditors play in the environment

5 Likes

That is a great question, and currently, I am unaware there is any data available assessing the effectiveness of audits in actually preventing hacks. But in all honesty, such data can be misleading. For instance, can one conclude that an un-hacked audited project is indeed secure? The answer is a plain no, as there are always unknown unknowns.

To assess the effectiveness of an audit in actually preventing attacks, one must first be aware of the scope of the audit, and that is often not done. As discussed in ā€œWhat is an audit postā€, there are different audit types, each focusing on a particular concern. Examples of the latter include: code audits, economic audits, legal audits, etc.

Generally, audits in the crypto space are limited to code audits, but I dare to say that the latest hacks we have seen in the DeFi space are a mix of economic issues (e.g, the attacker manipulates a coin price in a given exchange to buy it cheaper elsewhere), as well as flaws in the code itself (e.g., use of unreliable oracles, missing slippage limits, etc).

If we continue to limit audits to code audits only, while disregarding the composabilty of the comprising components and the resulting economics, hacks will continue to make big headlines. To conclude, code audits can only go as far as their scope of analysis. If the project at hand gets hacked, that is not to say that its audit failed. An audit is NOT a statement that a project is safe and free from bugs. An audit reports what has been found to be wrong with the best effort possible given the agreed scope and timeline.

8 Likes

This is a really astute point!

There would remain a critical need for code audits, but as an industry, what would be needed to go beyond code audits as the accepted standard? It seems like economic or law audits would introduce subjectivity or probability that might be interpreted as being less concrete than a code audit.

6 Likes

Thereā€™s an entire movement/practice geared towards better understanding the game theoretical/economic dynamics and pressures associated with the creation of novel economic systems, and providing economic audits called ā€œToken Engineering.ā€ Its a fairly novel area of study and collaboration and is pretty fascinating. There are a number of folks and organizations working in the space, and its something that Iā€™ve personally stayed interested in learning more about. The Token Engineering community are a decentralized group of contributors, and folks like Lisa Tan are working actively to contribute to education on the subject

Iā€™ve been wondering what other resources are available to folks to better understand typical smart contract exploits, the Consensys Dilligence Smart Contract Security Best Practices repo is a great one, they also have a Smart Contract Security Newsletter, primarily (exclusively?) penned by @root, thereā€™s also Open Zeppelinā€™s List of Ethereum Smart Contracts Post-Mortems and the less academic but very informative https://www.rekt.news/.

What others are out there?

4 Likes

@zube.paul Since we are largely an unregulated industry, the accepted standard shall be whatever users end up accepting prior to locking their money in a given crypto product. Right now no one launches a product without a code audit, but economic audits are catching up and to some extent so is formal analysis. But ultimately, the market pressure sets the tone, and no one will go above and beyond until the market forces and underlying risks say so.

However, I do acknowledge that the answer above is somewhat shallow. To mitigate that and give you a more objective reply, I believe the standard should be along the following lines:

  • layer 1 products, i.e., any implementation of the a given blockchain network (consensus layer, p2p layer, storage layer, etc) should have different types of audits depending on the component under analysis. For instance:
    - consensus layer: economic (generally based on game theory analysis) + code audits + some level of formal verification
    - p2p layer: code audit + simulations (stress testing some scenarios)
    - storage layer: code audit
    Example of products that did an excellent job here include Ethereum 2, Cardano, and Casper Labs (note: this is not an exhaustive list).
  • as for smart contracts (layer 2 audits), I would recommend:
    - critical parts: formal analysis
    - code audits for every other part
    - if the product at hand depends on economic factors brought by external components, then have an economic audit as well
    Compound and Aave are great products close to fully meeting this criteria.

Hope that helps.

3 Likes

@Eric Thanks for bringing in such a rich set of references (note that some are already part to our list of notable works). The Token Engineering resource seems particularly interesting. Thanks for sharing that.

For code audits, you pretty much covered all the main ones. I would only add a few more:

For economic audits, I highly recommend looking at the work that Gauntlet is doing. They do list some use cases that are similar to code audit reports, but from a financial and economic perspective. As an example, consider looking at their report for the Compound protocol. Other reports are also publicly available.

Hope that helps :)

4 Likes

Pretty interesting thread and ideas here. Here are some random thoughts:

Regarding the separation of ā€œcode auditsā€ and ā€œeconomic auditsā€, as you mentioned with DeFi composability (DeFi Legos), itā€™s becoming challenging to separate those two, as many of the recent hacks use some kind of price manipulation that falls outside the code audit but is expected from auditors to find.

Indeed this is an unregulated space but as an auditor, and no clear guideline on what is the full extend of an audit. One approach is to try to get developers more security-aware which some resources has been posted in this thread, as well as Token Interaction Checklist. That being said, my opinion is that one of the main challenges auditors face is time-boxed audits. To expand on this, how would you scope a project and give an estimate sufficient time for the audit? As an auditor you may want to spend more time reviewing the code, but as the developer team, you want to pay less and launch fast.
As an experiment, go through the public audit reports and notice how many of them mention ā€œtime constraintsā€ or ā€œbest effortā€. Itā€™s more than you think. Sometimes itā€™s mainly due to Curse of Knowledge, meaning the developers think their code is simple enough to be reviewed in one week, however, it takes at least a few days to understand the overall system, not to mention doing proper code audit (and on top of that economic audits).


I took over Smart Contract Security Newsletter from @maurelian and ran it for 2020, however mostly due to packed schedule, too many DeFi hacks to cover, and having similar newsletter out there, there hasnā€™t been any new issues in 2021. For those missing the newsletter I suggest checking out Blockchain Threat Intelligence.

Another resource we maintained for a while but hasnā€™t been updated recently is Blockchain Security Database, trying to create a human & machine readable (JSON) database of audit reports for researchers to use. Itā€™s open source on Github for anyone that is interested to keep this database alive.

5 Likes

@root For the Blockchain Security Database, I see the full list of json files in blockchainSecurityDB/projects at master Ā· ConsenSys/blockchainSecurityDB Ā· GitHub. Is it the case that all these files have the same structure? If so, where can I find a schema for them? By the way, I love this initiative of having a machine readable and open database of audits.

3 Likes

they all follow the below structure, and the html is generated using a script:

{
  "project": "Project Name",
  "project_url": "Project URL",
  "description": "Description of your project",
  "bounty": "Bounty program URL",
  "bounty_max": "Max bounty payout",
  "security_contact": "Security contact email",
  "audits": [
    {
      "repos": [
        {
          "url": "URL of the repo pertaining to audit"
        }
      ],
      "title": "Audit title",
      "date": "mm/yy",
      "auditor": "Name of auditor",
      "url": "Audit URL",
      "effort": "Person-weeks effort of audit"
    }
  ]
}

Would love to see the database resuscitate :)

6 Likes

Oh, I thought the schema would define a format for the findings as well, which is the most important part of any audit. Without that, one cannot easily aggregate data of the findings, nor query them. We, as a community, should start thinking of a schema that could be used across companies, and ideally, have a public (and queryable) database of findings. Thoughts?

5 Likes

That was part of the initial goal. However lack of standardization in the audit report made it impossible to automate. I think it is still manageable to do but requires collaboration between audit firms and manual work to input the findings in the defined schema.

If we can create that schema here, I commit to spend sometime to populate the findings from ConsenSys Diligenceā€™s public audit reports.

5 Likes

@root I was wondering if first we should consult the various market players and see if there is interest in defining a common schema. Alternatively, we could define our own and then try to create scripts to automate the export task. Not an easy task, specially when handling pdfs. I would still lean towards the first option. Thoughts?

5 Likes

@Astrid_CH @Twan this thread might also be of some interest to you ā€“ itā€™s a nice overview of auditing in the crypto world. It would be interesting to hear about how it might overlap with auditing and compliance in the traditional, centralized banking systems.

4 Likes

I really appreciate this detailed explanation. Pardon my ignorance, but are there any opportunities for blockchain auditing for those who have a non-technical background? Iā€™m completely new to the (blockchain, coding, crypto, etc.) space. My background is in title examination for real estate underwriting, and I recently started exploring IT Auditioning. Blockchain technology is really intriguing to me, so Iā€™m just researching how to get in the door.

4 Likes

Hi Dorrii, it seems that you have prior experience on legal documentations writing.
When it comes to auditing, broadly speaking thereā€™s two form of it in the IT industry.

  • Audits for ISO/ERP
    e.g., internal IT systems, regulations, compliances, for publicly traded companies.

Of course, something more technical such as risk assessment, penetration testing and maybe red team exercises will be part of this compliances auditing.
But, most of the time, itā€™s more of a ā€œif you have this in place, tick itā€ kind of auditing.

This kind of auditing is less common in the blockchain realm. You can still see that in some defi projects, they have to follow rules like the imfamous travel rule. AFAIK some services like the Perpetual Protocol banned user from the US completely, in order to comply with this rule.

Anyway, although this seems off-topic, you can still understand from the example that this kind of audit is hugely different from the code audit.

I myself have no idea what kind of companies do these for the blockchain industry. Maybe try the big 4 (accounting firm)?

  • Audits for code

These are almost all covered in the original post.
For starters, you can try to play some CTFs(capture the flag) for beginners aiming on smart contract security.

Or you can even try to rediscover vulnerbilities in some famous incident, such the parity multisig wallet, or the dao incidentā€¦, by following the guides/tutorials in the original post. Try to come up with your insights before looking the detailed write-up posts.

Generally, after you accquired some of them, you can try hackathon by Quantstamp or other famous auditing company in the Blockchain realm, to see if theyā€™re interested in your progress. AFAIK, Quantstamp is willing to train junior level people for contract auditing.

Disclaimer: Iā€™m not affiliated with Quantstamp in any way.

6 Likes

Thank you, Jerry. Your response is very much appreciated!

2 Likes

@lnrdpss very well said. The trend in technology will definitely impact auditing systems. The key aspect is that the auditors need to be abreast with the trend and develop measures to review systems as they continue to evolve.

2 Likes