Panel Summary: Aligning Incentives for Research and Review @ EthDenver 2022
TLDR
- Decentralized Science (DeSci) is a growing community within web3 of practitioners, academics, and independent researchers that use web3 products and principles to engage in decentralized research and peer-review. DeSci takes place outside traditional institutions that gatekeep the research and publication process.
- DeSci organizations and DAOs, including but not limited to OpSci, Foresight Institute, Protocol Labs, and The Smart Contract Research Forum, have been creating an ecosystem of researchers that are focused on developing consensus around a decentralized peer review process. SCRF brought these groups together (seemingly for the first time in public) to discuss the philosophical principles that have to provide the foundations for conducting science in a decentralized manner.
- The current research and publication process is riddled with funding from organizations that insert bias into research. On the publication side, the paywall of journals provides no profits back to researchers. The reputational system of citations and increasing your publication count further incentivize a publication process that does not benefit researchers.
- Funding, publication, reputation, and peer review are all prime scientific processes for innovation. DeSci is a new ecosystem within web3 that has the potential to grow over the next year to solve these problems.
About the Panel
Panelists:
- Allison Duettmann - President & CEO @ Foresight Institute
- Darren Zhu - VP of Engineering @ TigerConnect
- Evan Miyazono - Research @ Protocol Labs
- Jack Yin - Core Analytics Contributor @ Index Coop
- Dr. Kelly Page - Ethnographer, Curator & CEO @ LWYL Studio, Founding Team, GR-DAO
Main Questions Asked (not exhaustive of the conversation that occurred):
- What are the problems DeSci is looking to tackle?
- How can we bootstrap reputation while not being penalized for having a poor reputation?
- What are the reasons to not have full transparency in research funding?
- What are your thoughts on reviewers being pseudo-anonymous today and tomorrow?
- How important is structuring the best experiment possible compared to execution and meta-level research?
Personal Takeaways:
- Research can be done outside of the academic institution and without a pedigree. Independent researchers and science simply need agreed-upon frameworks and processes for engaging in research and science. This can ensure the standards for science are met while addressing the issue of inequitable systems and gate keepers.
- The right set of tools, web3 or not, could enable research democratized for entire groups of people without advanced degrees (but that still have the skills, intelligence, and determination to engage in research).
- The DeSci ecosystem should take a look at anyone with a college degree in the STEAM fields. They offer a proposition to engage in research and with a community of their peers that is open and accessible.
Questions & Discussion Summary:
What are the problems DeSci is looking to tackle?
- DeSci is seeking to address:
- How do we provide a peer review process as a community?
- How can an open system for peer review account for reputation?
- How do we create a credentialing system or curation mechanism for the audience we want? All the things that help accrue reputation. Reputation is a big part of peer review work and will be important in web3 DeSci.
- Can we capture the conversations and ideas that happen in conferences and off journals? These are intrinsically important to keep the process of science and reputation alive.
- There are problems with reputation in the current peer-review process:
- The process of journals is random and blind
- Researchers can cite themselves to increase citation indexes
- Reputation is essential, but itâs often invisible
- Who is doing the peer review process is unknown
- Peer review is dominated by Western, English-speaking, and male scientists
- Bootstrapping reputation is quite tricky, and the current peer review process is almost the only way towards establishing reputation
How can we bootstrap reputation while not being penalized for having a poor reputation?
- Verifying your work is everything. Everything seems connected, and if you donât already have these connections or are part of these institutions, we cant quickly validate you as a person.
- So how do we solve this? Some ideas might include:
- Double-blind review process
- Double pseudonymity
- Decentralized identity and reputation
- OpSci is currently working on building a DID for Science
- But even before new innovations, creating a more open science community with transparent onboarding processes can help.
- A model that worked well for Foresight Institute was to start small with an invitation and then invite others
- Over time, Foresight added open applications and gradual onboarding. Foresight currently has onboarding calls with all applicants and has prospecting members interact as though theyâre on the teamâonboarding person by person. Starting small then growing upward.
- Foresight noted that diversity is challenging. But, the issue is more an inclusion problem and less of a diversity one.
- Funding has often created inclusion issues, as who is funding your project influences the direction your community goes in and of the work itself.
- âI look first at who funded it even before the resultsâ - Dr. Kelly Page
- On the topic of making science more an open system, in the history of Ph.D. work, you couldnât always get a Ph.D. in your 20. Before, you had a lifetime of research, and then in your 70s, your peers came together and looked at your qualifications and contributions, and then you were given a Ph.D.
- âIâm a big believer in âproof of workââ - Dr. Kelly Page
What are the reasons to not have full transparency in research funding?
- Donors want to donate privately, and donors care about their personal safety
- If theyâre anonymous, can they dictate anything about the research? If you want to know where the power is and what the power values follow the money.
- It is important to look at the interests behind the funding and where you publish, and how you publish. Some researchers get paid to ghost for research. This is an ethical thing.
- You need to know which lenses are on the research if you truly want to decentralize it - Dr. Kelly Page.
- So really, look at your funding streams to make sure you have multiple.
- There are areas of funding that are less globally agreed upon to be socially acceptable. Like genetic links to intelligence will always be fraught. Or funding psychedelics research. You canât fund these without the social implications.
- If you are funding everything publicly, you canât spend all your weirdness points at once. If youâre backing one weird horse publicly might not be socially acceptable to do it in more places.
- Beyond funding transparency, organizations could use bounties to encourage review. There are healthy ones in engineering that provide strong incentives. So this can help us think about more extensive mechanisms and incentives for researchers and why people might even want to provide peer review.
- In web3, the idea of being anonymized is accepted and routine, so the places where funding can come from publically do have a pull on research. But we can design a system that is anonymous and still accounts for the bias of public funding.
- âScience funding is very zero-sumâ - Evan Miyazono
What are your thoughts on reviewers being pseudo-anonymous today and tomorrow?
- Major pros and cons of both
- You might not want people to see all your ideas, but there is value in freedom of information and transparency.
- People having the option for self-selecting identity is essential, whether a pseudo or real name, but it falls down either way when you donât have manners. Peer reviewers might steal work or can be brutal.
- Whether with full or partial anonymity, researchers need to bring integrity back into the peer review process and define what integrity looks like together.
- Furthermore, a positive reputation works in pseudonymity too
- If someone wasnât cordial, that could still show on a pseudo
- So many biases can be detached from the pseudo, which is both a positive and a negative
- And if that feeds into a bigger reputation model, then it comes back down to the work that we do and tokenizing it so we can get paid
- Because research is contextual: methodology, optimization.
- The research excellence framework (REF) published by the UK evaluates the impact of research.
- And on a human level, what does impact mean.
- Humans are always subjective, so research can never indeed be objective
- Finally, Dr. Kelly Page asked to look at the whole publication system and how it benefits the researchers. These giant media companies take advantage of institutions and researchers across the world. The media companies exist in both anonymous and pseudo-anonymous roles in research.
How important is structuring the best experiment possible compared to execution and meta-level research?
- 10 hours at the wet bench saves you 1 hour in literature review
- âWe assume our ideas are novel and theyâre notâ - Darren Zhu.
- Metascience 2019 and 2021 were good at bringing people together and asking how do we make science better and faster, and cheaper. You can over-prepare for an experiment so letâs see some MVPs or use an 80/20 rule to optimize the process.
- When we are biased toward practitioners and doing things, we can connect research and people to actual practical results. Sometimes itâs not clear how to do this, so whatâs the goal. So researchers have to drive towards change and impact and measure that. Then weâve achieved something.
- We often look at the ideas that came before us and what we even mean by innovation and ideation. âTo have true impact, you have to execute.â
- Additionally, itâs essential to look at meta-analysis, and what came before us, so we were not destined to recreate the wheel.
- Meta-research can be excellent training too.
- Sometimes in academia, you donât know how close you are actually to having a startup.
- Foresight Institute is often taking academics and pulling them into the real world so they can drive change as quickly as possible.
- There is a difference between science and innovation, and we want to get to the stuff that affects change in the world.
- âPulling more people into the land where they can affect changeâ - Allison Deuttmann
Links to Notable Topics Referenced:
- Ethnography of The DAO
- Aaron Shwartzâs activism toward open-source knowledge
- Metaculus for voting
- Good Judgement Project. Briar scores are used to estimate the accuracy of peopleâs predictive capacity.
- Double Click. Anonymous profiling systems that eventually became Google Add Words