Open Peer Review Proposal | Request for Feedback

Open Peer Review Proposal | Request for Feedback

Peer review has historically been a mechanism within the scientific pipeline for filtering out low-quality research and providing feedback for improvement. Rigorous review has traditionally only been available to those striving for publication in a journal. However, many independent researchers, especially in Web3, do not have publication as a goal. This post outlines a proposal for an open peer review pilot to be implemented at the Smart Contract Research Forum (SCRF) that provides rigorous review. As part of this, we want to understand what crypto-native peer review would look like. We begin by paying our reviewers and will seek to explore how emerging technology could benefit the peer review process.

Feedback on the process outlined below is requested and appreciated in order to help improve.

The creation of this process involved consideration of the sources collected in this knowledge repository.

Our goals with this pilot are to:

  • Increase the avenues to feedback for independent researchers

  • Explore the effects of financially incentivizing public peer review

  • Collect and share data from the process

  • Identify how to improve our design

  • Grow our international community of academics, researchers, developers, and industry experts

Our process:

In the spirit of creating an OPEN peer review system, the author’s research and the peer reviews for that research will be publicly available on the SCRF Forum.

Our process is divided roughly into three steps:

  • Screen & Select

  • Post & Review

  • Synthesize Results

Screen and Select:

In this step, we identify the researchers and corresponding reviewers who will be the core participants in our open review.

Researchers: We will source web3 researchers from our network (and by our we really mean @eleventh19’s) who are working on problems in web3 such as DAO Governance, cryptoeconomics, and blockchain who have research ready for review.

Reviewers: For each paper, we will recruit at least three reviewers who are knowledgeable about the subject area. We will tell the reviewer the author’s name so that they can self-report whether or not they have any conflicts of interest.

Meta-reviewer: For each paper, we will recruit one meta-reviewer who is responsible for evaluating the quality of the reviews. Evaluating the quality of reviews is important so that we have a metric to determine how different variables influence the quality of a review.

Post and Review:

In this step authors share their research, reviewers share feedback, and authors respond.

Authors will visit smartcontractresearch.org and make an account if they do not already have one. They will create a new topic in the ‘Peer Review’ category that includes a post with their research to be reviewed. Research can be posted directly on the forum as text, PDF, or as a link to where the research can be read.

Reviewers are requested to follow the instructions in Appendix A, sign their review, and post it on the forum during a narrowly-defined window of time to ensure authors get feedback in a timely manner.

Authors are requested to respond, rate the review, and follow the instructions in Appendix B

Meta-reviewers are requested to respond, rate the review, and follow the instructions in Appendix C.

Templates to use will be provided based on the items in the Appendix.

We select reviewers to ensure that the authors receive quality feedback from multiple sources with deep insight. However, in the spirit of open peer review, we strongly encourage anyone to leave reviews on the papers.

Synthesize Results:

In the final step, we will aggregate reviewer ratings into an overall view of the paper and conduct interviews with all contributors. The purpose of the interview is to understand the contributors’ perspectives on the process we used, how the public nature affected them, and how much the financial or reputational incentive motivated them.

Lastly, we will payout reviewers and meta-reviewers. In this first round, we will pay out $500 for their contribution to making better science. $500 was chosen as a baseline partly in order to be in accordance with the 450-movement, adjusting for inflation of course.

To Conclude:

This is the first round of open peer review. We plan to experiment with many variables (incentive structure, review format, the anonymity of authors & reviewers, etc.) in order to find a peer review process that works best for all parties and leads to the highest quality research. Throughout our efforts, we hope to provide peer review for independent researchers who would not normally have it, provide financial rewards to peer reviewers for their time, and to collect data that will help all peer review designers create an optimal process for their objectives. We look forward to working openly with the community throughout this process!

Acknowledgment: We’d like to thank Dr.Nihar Shah for his time, advice, questions, and an excellent resource surveying peer review.

This process is still a draft and we would sincerely appreciate your feedback to help us improve all parts of this process. Please leave any feedback about this process on this post.

Appendix A: Instructions for Reviewers

Thank you for reviewing a research paper as part of the Smart Contract Research Forum’s Open Peer Review Project! We appreciate your contribution to the scientific discourse of web3 research. We humbly request from all reviewers a moment of your time:

Please familiarize yourself with these guidelines before writing your review:

  • Please remember that we are not asking for an accept or reject decision as we are not a journal publication or a conference proceeding. We are instead creating a forum for scientific discussion and improvement of research.
  • Please provide feedback on the scientific content of the paper.
    • As a peer reviewer, your role is not to proofread or check for typographic errors but to understand the author’s scientific intent and any potential missteps.
    • When you find flaws, please help the author overcome them by suggesting how to fix them or by suggesting alternative approaches.
  • Please be empathetic and professional. As part of fostering a learning environment, we want to avoid any rudeness or arguments that attack the person rather than the position of the paper.
  • Please keep in mind that peer review is not just beneficial for the author, but it is beneficial for you as well! Reviewers report that peer reviewing helps them:
  1. advance their own research by stimulating new ideas and becoming aware of new methods

  2. stay informed with the latest research in their field before their peers

  3. reduce misinformation from being disseminated into the world

  • Please remember that your reviews may be relied upon not just by the author of the paper but by readers of it in attempting to critically assess it.
  • Bear in mind, that your reviews and your name will be publicly available to anyone reading the Smart Contract Research Forum. You can add your reviews to your CV since anyone can access them. We encourage you to take the time to make a review you will be proud of.
  • Please share your review within two weeks after the research has been posted.

Please include the following in your review:

  • Disclosures of any competing or conflicting interests including but not limited to: organization affiliations, competing research theories, prior beliefs on the topic, funding sources, knowledge of the author(s), etc.
  • A brief summary of the paper, its objective, and findings.
  • Suggested revisions to improve the legitimacy and credibility of the substance of the paper.
  • Strengths of the paper.
  • Answers to the following multiple-choice questions. When answering these questions, copy and paste the questions into your review and delete the choices you do not select. Please provide a brief explanation of your selection.
    • Please rate this paper on scientific correctness (does the research have an identifiable question or objective? Are sound methods and/or statistics used? are the conclusions logically based on the results?).
      • I am confident that the paper is technically sound, and I have carefully checked the details.
      • The paper appears to be technically sound, but I have not carefully checked the details.
      • The paper has minor, easily fixable, technical flaws that do not impact the validity of the main results.
      • The paper has major technical flaws.
    • Please rate this paper on organization and clarity of language (is the writing easy to follow? are concepts well-explained? is the document well-organized?),
      • The paper is well-organized and clearly written.
      • The paper is well organized but the presentation could be improved.
      • The paper is somewhat clear, but some important details are missing or unclear.
      • The paper is unclear and very hard to understand.
    • Please rate this paper on openness and reproducibility (is the data available? is the code? are the methods explained clearly enough to be replicable?)
      • Key resources (e.g., proofs, code, data) are available and key details (e.g., proof sketches, experimental setup) are comprehensively described for competent researchers to confidently and easily reproduce the main results.
      • Key resources (e.g., proofs, code, data) are available and key details (e.g., proofs, experimental setup) are sufficiently well-described for competent researchers to confidently reproduce the main results.
      • Key resources (e.g., proofs, code, data) are unavailable but key details (e.g., proof sketches, experimental setup) are sufficiently well-described for an expert to confidently reproduce the main results.
      • Key details (e.g., proof sketches, experimental setup) are incomplete/unclear, or key resources (e.g., proofs, code, data) are unavailable.
    • How confident are you in your understanding of this paper? Why?
      • I have checked all points of the paper carefully and was sure that I knew exactly what the author meant in their writing. I am certain I did not miss any aspects that could otherwise have impacted my evaluation.
      • I checked the important points carefully and felt pretty sure I understood what the author meant in their writing. It is unlikely, though conceivable, that I misunderstood some aspects that could have impacted the quality of my evaluation.
      • There’s a chance I misunderstood some aspects of the paper that I do not have the depth of knowledge required to confirm what the author writes is accurate. Moreover, I may not have carefully checked some of the details, e.g., proof of a theorem, experimental design, or statistical validity of conclusions.
      • I am able to defend my evaluation of some aspects of the paper, but it is quite likely that I missed or did not understand some key details.
      • This paper was very confusing to me. My evaluation is an educated guess.
  • Any other questions or comments about the research which was not captured here?
  • Please share any thoughts on how we can improve the peer review process in this forum post.
Appendix B: Instructions for Authors Responding to Reviews

Thank you for evaluating reviews of research papers at the Smart Contract Research Forum! We appreciate your contribution to the scientific discourse of web3 research. When responding to reviewers we humbly request that you do the following:

Please familiarize yourself with these guidelines before writing your response:

  • Please remember that reviewers are attempting to help authors improve research and have dedicated a significant amount of time to doing so.
  • Please respond to each reviewer individually and make it clear which revisions you are implementing and where in the paper you have done so.
  • Please be empathetic and professional. As part of fostering a learning environment, we want to avoid any rudeness or arguments that attack the person rather than the position of the paper.
  • Bear in mind, that your reviews and your name will be publicly available to anyone reading the Smart Contract Research Forum. You can add your reviews to your CV since anyone can access them. We encourage you to take time to make a review you will be proud of.

Please include the following in your response:

  • Disclosures of any competing or conflicting interests including but not limited to: organization affiliations, competing research theories, prior beliefs on the topic, funding sources, knowledge of the author(s), etc.
  • A response to suggested revisions and for each suggestion an explanation of if the revision was implemented and how
  • Answers to the following multiple-choice questions with a brief explanation of your ratings for each:
    • Please rate this review on the level of understanding it showed of the research
      • This review showed a thorough understanding of the research objective, methods, and conclusions
      • This review mostly understands the research objective, methods, and conclusions but shows that it may have missed some small details
      • This review partly understands the research objective, methods, and conclusions but misses significant details
      • This review does not understand the research objective, methods, and/or conclusion
    • Please rate this review on tone
      • This review makes use of appropriate language and is written with complete professionalism and respect,
      • This review is written with professionalism and respect but is occasionally condescending
      • This review is written with some professionalism and respect but makes use of inappropriate language or is condescending
      • This review is written with little professionalism and respect
    • Please rate this review on helpfulness (did it help you strengthen your paper?)
      • This review provided clear areas to strengthen the substance of the paper and easy-to-understand descriptions of how to improve
      • This review provided clear areas to strengthen the substance of the paper and descriptions of how to improve
      • This review provided areas to strengthen the paper and/or descriptions of how to improve
      • This review did not provide substantive or clear areas to strengthen the paper
  • Any other questions or comments about the review which was not captured here?
  • Any thoughts on how we can improve the peer review process? Please share them in this forum post or in our final interview/survey
Appendix C: Instructions for Meta-reviewers

Thank you for evaluating reviews of research papers at the Smart Contract Research Forum! We appreciate your contribution to the scientific discourse of web3 research. When responding to reviewers we humbly request that you do the following:

Please familiarize yourself with these guidelines before writing your response:

  • Please remember that reviewers are attempting to help authors improve research and have dedicated a significant amount of time to doing so.
  • Please be empathetic and professional. As part of fostering a learning environment, we want to avoid any rudeness or arguments that attack the person rather than the position of the paper.
  • Bear in mind, that your reviews and your name will be publicly available to anyone reading the Smart Contract Research Forum. You can add your reviews to your CV since anyone can access them. We encourage you to take the time to make a review you will be proud of.

Please include the following in your response:

  • Disclosures of any competing or conflicting interests including but not limited to: organization affiliations, competing research theories, prior beliefs on the topic, funding sources, knowledge of the author(s), etc.
  • Answers to the following multiple-choice questions with a brief explanation of your ratings for each:
    • Please rate this review on the level of understanding it showed of the research
      • This review showed a thorough understanding of the research objective, methods, and conclusions
      • This review mostly understands the research objective, methods, and conclusions but shows that it may have missed some small details
      • This review partly understands the research objective, methods, and conclusions but misses significant details
      • This review does not understand the research objective, methods, and/or conclusion
    • Please rate this review on tone
      • This review makes use of appropriate language and is written with complete professionalism and respect,
      • This review is written with professionalism and respect but is occasionally condescending
      • This review is written with some professionalism and respect but makes use of inappropriate language or is condescending
      • This review is written with little professionalism and respect
    • Please rate this review on helpfulness (did it help you strengthen your paper?)
      • This review provided clear areas to strengthen the substance of the paper and easy-to-understand descriptions of how to improve
      • This review provided clear areas to strengthen the substance of the paper and descriptions of how to improve
      • This review provided areas to strengthen the paper and/or descriptions of how to improve
      • This review did not provide substantive or clear areas to strengthen the paper
  • Any other questions or comments about the review which was not captured here?
  • Any thoughts on how we can improve the peer review process? Please share them in this forum post or in our final interview/survey
3 Likes