CTA: In these threads we attempt to inventory key areas of exploration that may exist in a particular discipline. These posts are living documents and it is our hope that the community will contribute to the list.
CTA: We need help identifying Key Problems in the Auditing and Security space. Please help us! Here’s what we think a Key Problem looks like:
- Provides direction for individual research efforts and projects
- Is a broadly applicable question or problem statement
- Requires many coordinated research efforts to answer or solve
What makes a good audit?
Audit thoroughness varies. As auditing companies compete with one another, some are likely to dominate the market and set the standard by simply eliminating competition. Ideally, however, setting the standard for a good audit should be a collaborative effort. From a crypto community stand-point, a good audit requires different companies to talk to one another, as well as to software engineers in the crypto space. Discussions that could drive such an effort include detailed insights on auditing practices such as internal methodologies, threat models, checklists, etc. Researchers, in turn, could help evaluate existing practices within the audit space.
Here are some avenues that the research community could begin exploring:
- What audit methodologies produce better findings, and under what circumstances?
- How comprehensive are the threat models in crypto?
- What is the breadth of existing audit checklists?
- How do methodologies differ across the board, and how do we evaluate them?
Last, but not least, researchers could propose their own auditing frameworks and seek their validation within the crypto community.
What security KPIs (Key Performance Indicators) should crypto projects use?
Currently, projects rely on audit reports as a go-pass for launch. These audits report auditors’ findings when they read a target code line-by-line. Essentially, an audit relies on a single KPI: the number of code issues found, weighted by how severe they are. However, other security indicators do exist and could be used to assess the security in crypto projects during their entire lifecycle (e.g., number of individuals with privileged roles within a contract, the project’s truck factor, amount of unaudited code, number of external dependencies, test suite quality, etc). However, evidence on what KPIs effectively work in assessing how secure crypto projects are is an open question.
Here are some avenues that the research community could begin exploring:
- What KPIs should one use when assessing the security of crypto projects? Do they have any predictive power?
- What threshold values should one rely on when accepting or rejecting a given KPI?
- Which KPIs do different stakeholders seek? Do they change per domain application?