Implementing SourceCred on the Forum

Meta
1

The goal of this post is to get community input on the implementation of SourceCred on the forum. This tool allows a community to identify and incentivize online behaviors through reward mechanisms. As a community dedicated to connecting industry and academia in the web3 space, it seems appropriate that we implement web3 tools that allow us to accomplish our mission of facilitating and rewarding long-tail discussion and thoughtful contributions in the web3 research space.

The implementation of SourceCred is not just so that SCRF can have a web3 tool, however. This fits into our overall engagement strategy of elevating thoughtful, researched, and insightful contributions to the forum. The type of on-forum engagement that we seek to encourage takes effort from researchers and other members of our community. There are obviously hourly and grant incentives that are in place to recognize this work, but SourceCred helps us additionally identify and value content that makes an impact on how people think about the content that they read on the forum.

The cred system described below rewards thinking that may have been missed initially, but people come back to months later with new perspectives and appreciation. This is the result of SourceCred’s retroactively updating system that reflects new information and engagement with content. For example, someone might write a great forum post that goes overlooked initially. At first, it earns little Cred, but once it gets discovered, the Cred score will retroactively increase. This frees participants to focus on doing great work, without needing to worry that all of it gets appreciated right away. Additionally, this tool can be configured to reward SCRF cultural norms, such as visiting the forum often, engaging heavily with certain content categories, or other norms that we can discuss in this post.

SCRF plans to implement SourceCred in mid-March of 2022, and we need to make some community decisions to help accomplish this implementation.

  • Cred generating nodes
  • Weighting nodes and edges
  • Initial pool of DAI to allocate
  • Eligibility for allocation

Cred Generating Nodes

When enabled, SourceCred produces user cred scores based on their contributions and actions. These are called nodes. Each discrete Node represents you and every contribution you make.

These contributions are automatically tracked and measured for its value and impact to the community. The value is denoted as a Cred score, which is how much Cred you have at the moment. Cred is not transferable as it is meant to be something that stays with you, like a reputation. In other words, it isn’t thought of as an asset to be bought, sold, or traded.

SourceCred uses a modified PageRank algorithm to determine the cred score of each node.

For example, posting this thread might generate +1 cred and an additional +1 cred for each like and/or response. Those are decisions we get to make. I would like us to discuss what those actions on the forum might include. The engagement team did a little brainstorming and generated the following:

  • Research Summary post
  • Discussion Post post
  • Likes to posts or comments
  • Having people reply to your post or comment
  • Replying to a post or comment

Obviously, there are potentials for gamification here. SourceCred discusses this in their documentation. We can also modify these nodes or their weights as we observe actual user behavior.

Weighting Nodes and Edges

Using this post as an example, I would potentially have +1 cred minted. That does not have to be the case. It could mint +30 or +.0001. When discussing the nodes that generate cred, it is also valuable for us to consider how heavily weighted that behavior should be. Long term, aligning our values for high-quality contributions and engagement best practices with what SourceCred rewards are the aims of SCRF’s implementation.

The SourceCred Docs define Edge Weights: Edge weights determine how Cred flows once it is in the graph. Suppose we have a Discourse post with exactly two edges out: one edge to the author of the post, and one edge to a person mentioned by the post. Who should receive more Cred: the author, or the person mentioned?

Community’s weights determine how much Cred each type of contribution earns. Cred allocations are distributed not just to the individual taking action but to all the supporting nodes and their respective stakeholders. Cred is calculated and distributed in what SourceCred refers to as the Contribution Graph which is the network of nodes for every contribution and participant in a project, and how they’re connected to each other. In other words, the result is that every contribution earns Cred if it was connected to, or depended on by, other things that earned Cred. \

The choices we make about how nodes are weighted is not set in stone. SourceCred weights can be discussed by the community and adjusted throughout the life of the instance.

Initial Pool of DAI to Allocate

For those that have been reading the SourceCred documentation, you’ll notice that they use the Grain Token as their exchangeable token. The Grain Token is given to members of the community by an amount that is determined by each user’s individual Cred Score. When Grain is distributed it recalculates Cred for all the users. Once that calculation finishes the Grain distribution is sent out. This keeps Grain distributions as close to the moment as it can be. The distribution policy and schedule for Grain is something we have control over.

Every project that uses SourceCred uses its own independent Grain token. Our community then can decide what the Grain token is worth. The value can be defined as monetary or simply as internet points. Typically, Grain is offered as having monetary value by exchange with a stable coin such as DAI. When paired to a stable coin the value of the Grain token is fixed to $1 each.

SCRF’s plan is to allocate DAI based on monthly cred scores instead. As a community, we need to determine what our initial pool of DAI should be that gets allocated. Perhaps a poll would capture this better, but maybe we start somewhere between $5 and $5,000.

Eligibility for Allocation

Similar to the Comment of the Month initiative, there is an open question regarding the eligibility of participants. Many of the people participating on the forum are also compensated for their time. As a community, we should determine whether we want those already compensated for contributions to be eligible for cred based contributions as well.

There is a lot for us to discuss as a community, and I am looking forward to working on this as a group.

20 Likes
2

One big question for the community to consider – should salaried and hourly workers be eligible for SourceCred? Salaried folk are motivated by incentives as much as anyone else, but they already have a stake in SCRF, so maybe they shouldn’t be eligible for these grants. What does everyone think?

7 Likes
3

I have been thinking about this, and I think since SCRF employees already get compensated, there would only be negative effects if we are then competing against the community for these grants. On the one hand, it seems unfair for SCRF employees not to be eligible if the funds are available to the public in general. On the other hand, people getting compensated twice for the same work is not necessarily the best use of funds for SCRF.

I feel like SCRF employees facilitating the community members having a better chance at writing a comment that will receive a grant or giving them examples through their posts is as far as SCRF employees should be compensated with the sourcecred pool. I don’t want community members to feel like they are competing against us for funds, when the point of the funds is to reward the best content and compensate contributors that aren’t already being compensated.

Maybe if there is a short-term compensation type vs. a long-term compensation type then it might make sense to enable employees to get long-term cred compensation whereas only community members qualify for the “short-term” rewards. In that, it would be more like royalty payments for IP rather than an up-front payment for labor. If we separate compensation into “labor” vs. “royalty” then I think royalties make much more sense for employees to qualify vs. labor which they’re already being paid by SCRF.

7 Likes
4

I’m very excited for SCRF to use SourceCred. I’d be interested to track this as a case study to publish on the forum later too. We will learn a lot about work here.

I’m not sure adding SourceCred for compensated employees would have negative effects. Their time, I assume, should be taken up by other activities and tasks to support the forum rather than find areas to be competitive with other contributors.

Regardless, there is a weighting metric in SourceCred that fits here perfectly.

"SourceCred currently has three policies for how a project distributes Grain [native token]:

  • IMMEDIATE splits Grain evenly based on how much Cred each participant earned in the last week. (This policy ignores all Cred from previous weeks, and is intended to give fast rewards to active participants).
  • BALANCED splits Grain based both on lifetime Cred and on lifetime Grain earnings. BALANCED tries to ensure that everyone in the project receives a total Grain payment which is consistent with their total Cred score. For example, suppose that a contributor used to have a low Cred score, and as such received a small amount of Grain. However, the community recently changed its weights, or added a new plugin, such that the contributor now has a larger amount of Cred. The balanced policy sees that this contributor is underpaid, so it will pay them extra to “catch them up” to others in the project. Conversely, contributors might be “overpaid” and they’ll receive less Grain until the payouts have been equalized.
  • RECENT splits Grain-based on recent Cred using an exponential decay to prioritize more recent cred. The recentWeeklyDecayRate parameter determines to what degree you want to focus on recent contributions. If recentWeeklyDecayRate is set to 0.5 (i.e. 50% discount), as in the above example, the policy will count 100% of the Cred generated in the last week, 50% of the Cred generated in the week before, 25% from the week before that, 12.5% from the week before that, and so on."

I would be interested to know if we can set up cred based on the role as well. That would help solve for knowing who’s already compensated and who is not (If you are X role, your available cred is % less).

13 Likes
5

I suspect we are able to weigh cred by role. This is something that @brian.alexakis might have a little more insight to contribute as he has been taking a more technical look at SourceCred. Right now I don’t think we have implemented much in terms of roles on the forum and we are only starting to do so over in chat. Obviously, this is something that can become more sophisticated over time as we find need to create more roles internally.

Our initial implementation does not commit us to those nodes or weights forever. Right now I’m thinking that we try out a small allocation, let’s say $500.00 and see how our cred and allocation actually work based on the behavior we have already seen? Maybe I should just make a poll for level of initial funding?

That might also give us a sense of how competitive cred actually is. Thanks to @valeriespina for laying out the distribution models. My inclination would be BALANCED or RECENT because I think those two most address some of the concerns @Larry_Bates articulated. I think right now I am more drawn toward BALANCED because I think it best incentivizes what is currently on my wishlist:

  • Habitual use/visitation of the forum
  • Thoughtful commenting that replies to a thread and potentially connects to other threads
  • Providing insights that is recognized through likes and replies
  • Engaging others in conversation by way of tagging

This should align with the Comment of the Month general criteria as we are trying to incentivize similar things. I think SourceCred could be used to focus on not just the quality of content, however, but also how well one takes that content and generates discussion and debate around that content. In some ways, this is what SCRF provides to the web3 space in a unique and compelling way.

What are other initial wish list items that people have? We should also start working toward a discussion on weighting for these nodes as well.

7 Likes
6

After looking over some of the documentation, implementing this into Discord and Github may be a useful means of tracking activity from ideation to publication on the forum.

Discord | SourceCred
Github | SourceCred

It comes to mind in the context of ideas that occur in Discord may translate into publications in the forum, and tracking that throughput may be beneficial in understanding how to make it more likely that a Discord chat results in a forum publication that can then be monitored through the Github pipeline. Additionally, this recognizes that some of the individuals involved in the Github pipeline may be a crucial part of the process that never gets recognized on the forum publication side.

In other words, a prolific reviewer may never publish a comment or topic on the forum while making a significant contribution to the content creation process. The potential to treat each zone of SCRF as its own weighted region in which activity can result in Cred makes it easier to define weights and allocations especially when SCRF employees have the capacity to earn Cred without competing with community members directly.

8 Likes
7

Another potential worry with SourceCred, we should be careful what we’re maximizing for. If it’s engagement we’re after, we definitely need to maximize for the right kind of engagement, otherwise, it becomes a race for the bottom. Maybe scrambling names would help or some other kind of pseudoanonymity?

5 Likes
8

As someone who uses SourceCred (SC) in their DAO I want to point out a few things:

1) SC is dynamic and malleable.

Don’t expect to have a perfect strategy and implement it with SourcCred right away. It is only after the first calculation that you can start making any sense of how value is generated and it is only after a few runs of adjusting weights and fine-tuning parameters that you will be able to get SourceCred to conform to your standards. But the standards themselves can change according to SourceCred. It’s a synergistic relationship.

2) SC is heavy and complex.

It requires some expertise in handling the whole algorithm and maintaining its operation. Simply put, the fact that it is an AI algorithm doesn’t mean that it gets people out of the way. It is the other way round, SC makes human input from everyone involved indispensable!

Luckily though, SCRF already has a DAI reserve and doesn’t need to mint new tokens for paying contributors which would require even more expertise.

3) SC will radically transform the way people interact with the forum.

As with any mechanism that works on the principle of incentive design, SC can deeply affect not only how people are compensated or the frequency and quality of posting but the way people think and feel about forum interactions. This means that a lot of deliberation is required on where and how SC will be implemented. Incentives can enable certain behaviors at the expense of others. Gaming the system will become an issue.

Here’s a hilarious and well-thought out comedic site created by a member of Metagame that seeks to provide a cautionary tale about the “gaming” dynamics that SC gives rise to:
https://www.sourcecredjunkies.info/sourcecredaddictionward
(Keep in mind, the stories and characters are all fictional, but the phenomenon is quite real!)

4) SouceCred might also change the compensation structure of SCRF in general.

When people can be compensated directly through their interactions and output, this may raise questions about which parts of the compensations processes are needed and what can be automated, which may disrupt some well-established well-oiled processes that are already in place. But this also means that new possibilities can open up for the reward of contributions that usual fall under the radar or are not even thought of normally as contributions.

But these are not reasons to shy away from using SouceCred. It is a very powerful tool which means that the benefits it can provide depend on how wisely you make use of it.

Here’s also another text from Metagame about using SourceCred that some of you may find interesting.

9 Likes
9

Thank you for your thoughts. I’m sure they’ll be a big help to Paul et al. Do you have any experience with other web3 tooling platforms? I’d be curious about how something like Coordinape might differ in execution and its effects on a community?

3 Likes
10

When this subject was first discussed, I was against SCRF staff “being eligible” for SourceCred compensation on the same grounds as @Larry_Bates:

But our thinking about SourceCred is quickly evolving. Larry’s thoughts about “long term royalty” payments and rewards for Discord and Github contributions clearly have merit.

However, IMO they still need to be considered inside the context of existing compensation. People contributing to SCRF with no ongoing agreement about their reward (including people who produce summaries on a flat-fee basis) should come first.

5 Likes
11

@Hermes_Corp’s experience with SourceCred adds new value to this discussion. All four of his points seem to deliver the same message to SCRF: SourceCred is complex, and we’re going to make mistakes and get a lot of it wrong at first. We should make it clear to the community that we understand this.

My instinct is to allocate a reasonable but not excessive amount of DAI to the first iteration of our SourceCred experiment, and openly accept the fact that it is an experiment. Months down the road, we’ll have hard-won experience and know a lot more.

8 Likes
12

Considering the forum has built in metrics like “read time” or “last activity” there seems to be some basic levels that could be determined for an account to need to qualify so that it does not simply become attempts at gaming the DAI pool.

For example:
10 Days visited
1 hour read time
30 mins recent read time
5 topics viewed
5 posts read

Now, these specific metrics are arbitrary; but in that context there does seem to be a point where we could say “this person needs to read or visit more before they can start voting on who receives Cred”.

I don’t think there should be a huge hurdle, but I do believe that making it so that there has to be some sort of activity on an account BEFORE they are able to vote on cred would start to eliminate that low-level bot manipulation. If the goal is to encourage engagement, we don’t want to water down the litmus for participation and by proxy encourage poor quality engagement.

On the other hand, the litmus for participation is cultural and not set in stone. Based on Hermes_Corp’s observations and experiences; my inclinations are in line with that approach.

This is why I lean towards longer-term royalties in general, as short-term compensation is much easier to game.

I am not sure “making quick money” is the type of culture that encourages quality participation or quality output and is more concerned with “making something happen quickly”.

I would infer that making rewards payout over a longer vesting period relative to an immediate payout would be more inclined to encourage the type of quality participation that SCRF is looking for instead of the general engagement that can be easily purchased via bots or individuals seeking to complete bounties with no regard for the quality at completion.

7 Likes
13

I tend to agree with these conclusions.

6 Likes
14

This is a welcome idea. I frequently visit the forum to read and acquire knowledge and I am always impressed at the high level of conversation being shared on the forum.
I think implementing SourceCred on the forum will encourage active participation as readers will not only read the conversations but also contribute to the conversation discussed.

Questions: won’t there be a tradeoff between the quality of discussions shared to the volume of posts?
Won’t there be risks of bots’ activities, thereby reducing the quality of the forum’s output? Will SourceCred take into consideration these tradeoffs and potential risks?
How do readers that are not SCRF staff monitor their activities?

8 Likes
15

The opportunities for shaping culture and discussions around what the forum values and needs (from an engagement and content standpoint) is what excites me most about SourceCred. @Larry_Bates I like the idea of a distribution policy that involves allocating across areas/regions of SCRF a la github, forum and discord. It could be an opportunity to highlight/discuss how different types of participation have similar…or different weights.

7 Likes
16

In thinking about this further, I don’t think “identity” necessarily has to be tied to a person’s academic or professional history if this is executed correctly. In this, “identity” itself could be a standalone metric that as a weight multiplies a person’s reputation by 2. This would mean that if a regular account with no history would have a weight of “1” as a default, then some sort of identity verification process would multiply that by 2 to get a total weight of 2. This has a logic that will become more clear as I move to the next section.

If identity has been separated as its own metric, then what’s left is the need to prove either “graduation from a given university” or “work experience at a given company”. At first assessment, a LinkedIn profile may seem to be a sufficient placeholder. However, there is no real validation mechanism within LinkedIn to “prove” the information presented in a profile is valid. This is where “proof of graduation” or “proof of work experience” could be means of an academic “anti-curriculum vitae fraud” or “anti-resume fraud” detection which doesn’t necessarily currently exist. There are automatic plagiarism detection algorithms, but as far as I am aware there is no mainstream tool which contains profiles of “validated” graduates or professionals.

In this framework, then it becomes easier to start to classify types of work that could be validated without exposing the validated identity to the public. The highest level classes would be “academic” or “professional”. Within these classes, we can start to break down arbitrary weights that roughly represent the amount of work that is associated with each classification. Here is an example:
Academic Classes -
Undergrad Enrolled= X 50
Graduate = X 250
Master’s = X 500
Doctor = X 1000

Professional Classes
1 Year Junior Position = X 100
3 - 5 Years experience in a given field = X 500
1 Year management = 250
3-5 years Executive = 1000

One of the reasons there is a need for some level of proving of work/education history to receive “cred” is that if someone has a history of academic or professional work outside of “SCRF” then they should be able to leverage that into weighted influence if they are willing to go through an agreed upon standard. Over time, a person who has no outside history in academia would eventually be able to post enough on SCRF to get as much weight as someone who comes into the conversation with a doctorate; but that would only occur if a person engaged in a manner that the forum rewarded with upvotes. In this same vein, over time the weight of the algorithm would not be disproportionately leaning towards academics or professionals over a forum poster with really quality engagement. The algorithm should treat them as equals if their output has equal validity or value to the forum.

All that said, if someone wants to “Prove their work history” or “Prove their education history” without divulging their full identity; cred seems to be the weighting mechanism that will make that possible. I am thinking further that if someone wanted to have “multiple accounts” their cred should not theoretically be the same on both accounts. In other words, we can never stop people from having multiple accounts; but we CAN identify individual work histories and individual graduation histories as being associated with an account or not. Just to give an example: if I wanted to validate my work and academic history then I could submit proof of the universities from which I graduated to then prevent someone else from assuming my academic identity within the forum.

Once an academic history or work history has been submitted for proof, ultimately they could be cross-referenced for attempted academic/work history theft or fraud. This tool build could be used to not only track the activity across the forum; but it could then be used to keep track of potential peer reviewers, collaborators, editors, practitioners, or anyone else within the space that is willing to validate their experience.

6 Likes
17

I really appreciate the thoughtfulness put into this thread so far and the ideas people are bringing up.

Could @Larry_Bates clarify the class system a little?

Cred can be generated through responsive mechanisms, so I’m not sure why embedding a class system increases/decreases gamification. I generally agree there would be some benefit for being able to meaningfully signal expertise, but I"m not sure I agree that it becomes a weight.
For example, imagine that an Executive comments “great insight” on a post whereas a Junion Position writes out a thoughtful response to a summary or discussion post. I think it might even be acceptable if that thoughtful response ends up being wrong because that contribution to the thread ultimately moved it forward and was acknowledge by users as meaningful enough to interact with. The “great insight” comment might garner some likes or off topic discussion, but the non-expert comment moved it forward. Am I misunderstanding your class recommendation though? I want to make sure to understand it correctly.

5 Likes
18

The issue becomes: I have posted a summary about an article, and then the article writer came into the thread and commented. That person’s first comment ever made on the forum was in the thread. If we implement cred where the weight of someone’s expertise isn’t brought into the equation, we could end up in a situation where my comments on a summary have more weight than the person that wrote the summary because their previous history is not able to be recognized by our cred system. That could result in some embarrassing situations where the article writers are being outweighed in conversation by someone who is not necessarily even a specialist but has more weight because they’ve been on SCRF longer.

Further, as someone who has experience in both academia and in the private sector; I believe we have to be able to find some way to give equal weight to people in a conversation if they have one type of experience vs. another. A doctorate doesn’t really have an equivalent in the private sector concerning the trajectory of how to achieve that title even though many doctors practice in the private sector. This is where trying to classify experience within that “class” may not need to be necessary for the sake of terminology, but being able to somehow recognize someone’s work or academic history seems to be very much necessary so we don’t end up having an article writer outweighed on their own work.

That specific situation can be avoided if we weight experience accurately in a way that seems feasible and agreed upon as a mean of validating that history. I think it would benefit from not being solely a SCRF repository in the long run, and the more research/private sector hubs that kept these validation repositories the less likely academic/work histories will be fraudulently used one way or another while giving those work/academic histories weight within the forum.

I’m not married to any specific terminology or specific structure, but the post was meant to recognize that we have the potential to create a process to validate work/academic history so that “identity” does not inherently mean “work/academic history” since non-professionals and non-academics should still be able to use their identity validation as a means of getting more weight. In that construct, the identity as its own metric makes it so that an anonymous doctor has more weight than a regular user with no registered credentials while also having half the weight of a doctor that offers their identity.

Obviously the weights are arbitrary, but the concepts around “why” and “where” we are establishing metrics are what I was trying to create a conversation around. Obviously the “first implementation” can’t attend to all these issues, but I think the realistic implementation has some hurdles to clear concerning reconciling encouraging engagement while also not punishing academics or professionals with experience for not having posted on SCRF frequently.

To reiterate for clarity: having had experience in the private sector and academia; this framing was originally an attempt at trying to quantify both sides of their experience without giving preferential weight to either side. The practical thought was that if a conversation came down to an established theoretical doctor having a conversation with an established CEO or other executive level, their weights should be roughly equal based on what it takes to be in either position. This was all in the context of having had the writer of an article on which I did a summary make a comment on the summary with the realization that my one year of experience on the forum shouldn’t outweigh them having written the article. These thoughts came after asking the Research Hub team some questions about their protocol and how they prevent an individual from creating bot accounts and overweighting a conversation.

5 Likes
19

I’ll give another example. Let’s assume a well-known developer named “Mitalik” shows up and they want to make a comment on a thread. In the process, “Mitalik” wants to further the conversation without distracting from the thread while also asserting experience. Allowing “Mitalik” to verify his experience as a developer with some mechanism without forcing him to divulge his identity publicly could create an environment in which experts felt comfortable sharing their expertise without fear of being shouted down while also not necessarily having to dox themselves publicly to prove they have experience.

It’s not to say this is a perfect solution. The issue is that there are ways to make this system really inviting towards academics or professionals that frequently put their work behind anonymity or paywalls so that they do not invite antagonism by proxy of working on a particular subject. Looking across academia, everyone does not equally want to be publicly associated with their works or have their works be widely known as not to invite the byproduct hostility that often comes with becoming widely known.

5 Likes
20

This is not much of a problem practically. A solution would be making an “author” role in Discord and each time the author makes a comment, they are automatically tagged by a bot in a higher weighted, message-restricted Discord channel. Fortunately there are numerous ways to hack SourceCred into submission.

But my thought on this is: isn’t it important for people who are long-time contributors in SCRF to earn more from the system?

Reputation systems such as SC are ways to incentivize and reward commitment and long-term engagement and what matters most in the system is how consistent and how valuable people’s contributions are to SCRF rather than what people’s history is. The SC algorithm doesn’t need to be any more biased toward paper authors than anyone else if they don’t also happen to be active contributors in the forum. At least this is what happens in practice with SourceCred.

Is this reasonable? What are your thoughts?

4 Likes