Extracting expert knowledge, schools of moderation, and StackExchange

A conversation with Discrete Lizard

How about capturing the knowledge hidden in the minds and practices of experts, putting it into a database so that novices could enter questions and have them answered immediately?

This is the goal that the StackExchange community is working towards. And as far as computer science is concerned, it has in many regards been reached. Few are the moments when a programmer cannot enter a question into google’s search bar and have it serve up an answer from one of StackExchange’s several Q & A websites on programming and computer science.

An etching by Erik Desmazière from the 1997 edition of Borges’ The Library of Babel.

How did the StackExchange community manage to do that? How was it possible for a group of strangers to gather online and extract the expertise out of each other’s heads, making it searchable for novices?

Understanding this (and the limitations of the approach) is crucial as we seek to accelerate the transfer of expert knowledge in other domains.

Can the transfer of all types of expertise be automated through similar processes? Is it, for example, possible to use community curation to scale tacit knowledge, or do we need other approaches?

In researching these questions, I have been chatting with Discrete Lizard. Discrete Lizard, who goes by the name Martijn Struijs in meatspace, is a doctoral candidate at Eindhoven University of Technology and one of the senior moderators on StackExchange’s Computer Science website. 

What follows has been lightly edited to make me look smarter. 

Henrik: At the heart of what StackExhange does lies the work of its moderators. Being a moderator – someone who gardens the conversation between large groups of strangers – is a fairly new social role historically. So let me start with the naive 5-year-old's question: what does a moderator even do? What takes up most of your time – and what are the hard problems?

Discrete Lizard: Most of the time is spent watching what is happening, and thinking about whether you should do something. Often, people will ask you to have a look, which you should do. Sometimes, they also ask basic organizational questions. These are easy to answer immediately. Other times, people have more complex concerns. These take more time.

One of the more important things you should do is gently explain the rules to new users.

StackExchange has a reputation for being hostile to newcomers, mostly as a result of newcomers having expectations that we are not going to meet, such as "you have to answer my question, no matter what".

The hardest tasks are when prolific users make understandable, honest, but unreasonable demands in public. You essentially need to say "no" in the nicest possible way. Failure risks losing an important member of the community.

Now, all of the above is something all users that are not elected to be moderators can already do. In the SE model, they also should. The community should moderate itself, most of the time. 

Of course, there are always exceptional cases. These require the privileges of elected moderators.

Henrik: What are these privileges?

Discrete Lizard: The most important one is a distinct symbol, the diamond, that is carried around as a badge. The diamond indicates you are an elected moderator, and that you should be listened to.

Another is access to and control of information. We're permitted to look at private contact details (e.g. email address) for the purposes of moderation. We mainly use this to inspect sockpuppetry or ban evaders. We can also delete pretty much any comment, such that only high-reputation users can view it. Of course, moderators can view any deleted content.

We also have the banhammer, or rather, the suspension option. Now, we often first give a warning. Depending on the nature of the offense and the history, we usually follow an escalation pattern of a temporary ban from 7 days, 30 days, then a year. If you continue the behaviour that got you banned, you will get suspended again, possibly indefinitely, or even have your account destroyed.

Schools of moderation

Henrik: I assume there are different schools of moderation? If that is true, how would you classify the most important distinctions between them?

Discrete Lizard: I know not all of them, but I can classify what I know. I will list them in order of increasing sophistication.

The first school is: no moderation at all. Increasingly rare nowadays. It means the conversation is truly private. Perhaps the only way is face to face and via snail mail.

The second is minimal oversight, usually only observation and reporting to the authorities. Most "DM" or other private messaging solutions on the internet are of this form.

The third is minimal moderation, such as a requirement to be legal in the relevant country, and some simple rules to prevent the worst kind of abuse. Note that this is the first level where the "mod" themselves can censor. This is the model under which 4chan and similar places operate. The advantages are that there can be almost unrestrained creativity, the disadvantages are the said creativity can at times lead to bad results.

Henrik: When would you recommend this style of moderation?

Discrete Lizard: I think the model is most suited to small communities focused on niche creative interests. The 4chan boards /vp/ (Pokemon), /jp/ (japanese culture), /vg/ (video game "generals", long-running repeated threads about specific games/genres/series, no general discussion) are some communities I hold dear. Some discord servers also use this model, and this is perhaps better than the 4chan model, because it is a bit more private.

The fourth model of moderation is much more sophisticated and has been applied very frequently. The previous models weren't really concerned with how we actually get people to listen to the moderators. This model does, and the method is purely based on the social authority of the moderator. That is, the moderator must cultivate and maintain a social standing within the community that gives their voice authority.

Henrik: That is a model that makes intuitive sense to me. Relying on natural authority. What is the weakness in that model?

Discrete Lizard: The main weakness is that social authority has a lot of costs. Time investment on behalf of the moderator. Social cost on the community, as it would be required to test the "loyalty" towards the moderator at times, perhaps through unfair accusations. 

The moderator also may see it reasonable to cultivate a "either with me or against me" stance.

In general, I think this mode has all the cachets and disadvantages of tribalism.

Note that there is also some variation within this model. You can have a single mods reign of terror, or a chain of command of electable moderators and moderator assistants, each with their own roles. As long as they derive their legitimacy purely from their social standing, they still fall within this model, no matter the other organisation.

Henrik: This makes me think a bit about natural authority in general. When teaching, I tend to prefer situations where students submit to me out of social standing, rather than compulsion – I do not teach in schools, only open access places like libraries and such. I guess a part of what is good about that is that I do have the moral authority to be a tyrant when people are free to walk out. But I sense we haven’t come to the school of moderation that you subscribe to yet. What is that?

Discrete Lizard: Yes. The model I follow tries to deal with the weaknesses of models that rely on social standing.

It is the model employed on StackExchange and also by Scott Alexander on SSC, to some extent. Both Scott, and Shog9, a long-time and well-respected Community Manager at SE, use the garden as a metaphor for their idea of community and moderation. (You should read Scott's post on moderation. As well as the general primer to moderation on StackExchange.) 

1. Your authority as a moderator does not derive from your social standing but instead is granted by a formal institution. Elections are a popular (ha!) mechanism, but a company hiring people based on their skills and experience is another common option. 

2. You try to teach the community to uphold the rules themselves and only intervene (in a censoring way) when necessary. This means that often, you are interacting with the community in a non-mod fashion. Note that the only reason this is possible at all is that your moderator status does not depend on your social status! (Otherwise, even friendly disagreement with a moderator risks punishment for "getting on their bad side". In this model, the latter is explicitly bad behaviour of a mod.)

I don't want to get too political, but I think one of the important differences between EU and US politics is that the US works on a model 4 type of governance (the president is mainly in a competition for social power) versus model 5 governance (the prime minister is mainly chosen by the party as the person most capable to achieve the points in the party program, which is what people vote for).

Henrik: What lies beyond model 5?

Discrete Lizard: That’s an interesting question. I think StackExchange is approaching something fundamentally beyond model 5 with their website, and that perhaps a newer project, Codidact, is even closer still. 

The idea here is: If all models have their advantages when applied correctly, why not try to use them all? SE has many channels, questions, answers, edits, comments, chats, and all of them receive a different type of moderation. 

Consider a scouting camp. What is more efficient, having a single person command everyone, or having a whole hierarchical network? The answer is: we don’t know! It depends on the task and the degree of autonomy required of each agent. Maybe a searching expedition would be suitable for the single leader star model, but a trading expedition would benefit from more management structures.

Also, the idea of formally distributing moderation through the community via a mostly automatic trust measure is interesting. 

StackExchange from the inside

Henrik: For most people, StackExchange is just this magic thing that appears whenever you google a programming problem. There’s even a joke that no one has ever visited the start page. It is just a seemingly endless repository of knowledge that springs from… well, can you tell me a bit about that. How does StackExchange look from the inside? 

Discrete Lizard: It is a type of community. A strange one, but nevertheless a community. The main things that keep the community together beyond asking and answering questions are the secondary channel of "meta" (for serious discussion about the site) and the tertiary channel of "chat" (for any other discussion), as well as the diamond moderators who lead the community. 

Of course, not all moderators are heavily involved in the moderator community, mostly based in the TL chatroom or on the main Meta discussion board. Most aren't, in fact. The same goes for the site: most visitors are just coming from Google for a quick answer. This is fine, the site was designed for this kind of interaction from the start. But these visitors aren't really part of the community. The frequent answerers and questioners, who have acquired some reputation, are. This is why they are permitted to perform certain moderation and curation tasks.

Allowing trusted users to edit posts by other users is fairly unique to StackExchange.

Henrik: What are the biggest bottlenecks in your work as a moderator?

Discrete Lizard: Mostly, dealing with the fact that our users do not vote enough (for various reasons, mostly because CS is so broad, most users do not feel they can evaluate most questions and answers), so people do not have the privileges they deserve, in my opinion. It also doesn't help that most of our users with enough reputation to close vote are no longer active. I guess the main reason for that is that we kind of exhausted the standard CS curriculum, and all the programming goes to StackOverflow.

Henrik: Votes being a limiting factor. That’s interesting. If you could solve all the bottlenecks and limitations, what would success look like for StackExchange? Not from a business perspective, but from the perspective of the community. 

Discrete Lizard: The most successful state is to basically facilitate all the structured, merit-based, original research-based question and answer communication, whether private or public. Let me unpack that. By structured I mean, that, unlike Reddit, we don’t do loose discussions. And the communication is merit-based, rather than social-reputation-based, like Quora. And unlike Wikipedia, it focuses on original research. Of course, if StackExchange is to extend this to dominate more topic areas than programming, as they are trying, they would not only need to scale the servers, but also hire a lot more community managers.

This is what the community is organically working towards, by tending the garden. But it doesn’t always match up with the corporate goals, which are partly shaped by the interest of venture capitalist investors. Corporate has tried many inorganic things and failed hard at them, while the community works on organic growth that actually works.

Henrik: Is the company responsive to needs from the grassroots? I get a sense from the links you share that they've failed a bit when it comes to community management; how are they doing with tooling? Is there a sense of the company and community growing more or less aligned?

Discrete Lizard: Well, in terms of management, the people directly working with the moderators and other users, the community managers (or the CM team, I think they call themselves nowadays) always understood what the community needed. Unfortunately, this team was understaffed, the internal corporate structure of roughly 5 years back grew too fast for its own good, which is why this particular problem was blocked by management instead of being dealt with. Upper management always had a sort of idealistic streak that was at odds with actually managing the people. 

Consider this post by one of the founders. This was very poorly received by most regulars, because meta simply was not the place for political soapboxing. It is where we discuss issues with the website. Of course, the founder was of the opinion that anything he found was important to his company and the world at large was on topic in meta. But he forgot that this is an international site, and that many countries have already solved this issue, and others haven’t even considered it. None of these users understand the fuss!

There was also dissatisfaction with SE Inc. after they demoted a respected moderator without warning on a religious holiday where she wasn't permitted to touch a computer. By this, they effectively threatened all moderators with the same – for shooting down one of the best of us is a dire warning. There was behaviour from her that warranted some action, but the case was completely mismanaged, stalled endlessly, and ended in a demotion only because the persons affected by her behaviour couldn't take it anymore. (The latter is not officially confirmed, but one of the people involved pretty much noted this was the case.)

These kinds of situations are deeply unfortunate. You can see the community management team enacting the will of their boss, but failing to justify their position coherently because they can't, they don't agree that this action is a good idea, but yeah, they don't call the shots. 

But I have to say there has been some improvement as of late.

Creating the position "VP of community" and giving the job to someone that has been involved in Wikipedia and Reddit was a good move. He knows what the value of the community is, and has listened properly to our concerns. Still, simple blunders continue to be made at parts of the company beyond his influence, like unannounced "tiny changes" that drastically change the user experience. 

The future of knowledge moderation

Henrik: Does the tooling you have access to allow you to garden to your satisfaction as the site grows and matures? 

Discrete Lizard: Currently, I think that I have good tools that are even being improved as we speak. However, the system to give tools to the other users has been working sub-optimally on my site. This will be hard to fix. The company doesn't like making exceptions for reputation thresholds on privileges. It has this idea that StackOverflow (StackExchange's most popular forum) is the one and only model, and everything else should conform to it. This seems silly and disconnected from getting on the floor and looking around you.

I think of SE Inc similarly to my university board: disconnected from how research and education is actually done, and only concerned with the output, especially the societal relevance of the output. They foolishly believe they can ask everything top-down without even asking how it actually is done at the bottom.

Newer projects, like Codidact and topanswers (created by dissatisfied SE users, including said exiled mod, and my veteran colleague and primary mentor, Gilles) have more interesting tooling choices.

Henrik: Tell me about Codidact. What are the most interesting differences from StackExchange, tooling- and culture-wise?

Discrete Lizard: Well, Codidact is still too small to do Q & A in a way that is comparable to SE. Still, from browsing their discussions, it seems they are both 1. taking it as a maxim that different topics and communities have different needs, and that the system should be flexible in this regard, and not pretend adaptations are modifications to the "one true form" of a Q&A site, and 2. are very idealistic, yet still experienced in how Q & A works, and craft the tools to support the construction of their community as they wish.

Some of them (among them several ex-moderators on SE) believe SE Inc. could no longer be trusted to carry out the mission of "creating and curating a library of questions and answers", jumped ship, and joined Codidact once it had a basic prototype.

While I agree with their ideas, the fact is that the community I care for is still on SE. Therefore, I remain on this platform. This is less than ideal, but this is how platforms operate as a business. 

Henrik: It feels like a common dynamic with these platforms, where the company and the users grow increasingly misaligned as the platform has established strong network effects and shifts into extractive mode. A lot of people are hoping for different (web3) models that push more value toward the users, but I’m not sure how realistic it is to actually pull that off in a healthy way.

Discrete Lizard: My sense is that the only way would be to legally force platforms to be replaced by open protocols. I'd like the EU or US in particular to push this legislation one day. 

Henrik: It will be interesting to see how our collective attempts to extract and distribute expert knowledge evolves. Thank you so much for taking the time to share your thoughts.