Check out SO Podcast 117 to hear Kristina talk about the mentorship experiment.
First, if you haven’t read the original announcement post about the Stack Overflow Mentorship Research Project, you should go and do that now. If you don’t, the rest of this probably won’t make a ton of sense.
From the beginning of August to the beginning of September - 33 days, to be precise - we ran a small pilot of a mentorship program:
- 63 mentors from the community volunteered to be our partners.
- Over the course of the experiment, over 500 new users entered our mentorship chat.
- 271 of those users had conversations with our mentors.
- Comparing those 271 users’ questions with questions from all eligible users who did not enter or participate in mentorship, their question score (means upvotes + downvotes) increased by 50%.
IMPLEMENTATION DETAILS
Here’s how the system worked at first:
We began with one private room for mentors, and one room for the actual mentorship chat to happen. We limited the rate at which eligible mentees could enter the chat so that we weren’t overwhelmed. When someone came into mentorship chat from the Ask a Question page, we sent a notification to the private chat so that someone could “claim” that question, pop in, and help:
Because of your feedback, we started the project with a “draft post” feature implemented. When mentees entered the chat, their draft question was copied over. They can edit the draft from there, and then copy it back over to the Ask a Question page when complete. I don’t say this lightly: our experiment would have been a failure if we hadn’t had this from the beginning. Thank you. Here's what the draft post feature looked like.
- My draft is posted in the chat.
- I can go edit my draft by clicking the "edit" button.
ITERATION
Soooooo now I’ll go through all of the problems with our initial approach, and talk through the solutions we implemented. The point here is that this experiment was very much a learning process.
Problem: Overcrowding: As you all correctly guessed, trying to put a bunch of mentorship conversations into one room was a bit of an issue. Multiple conversations going on at once was confusing for everyone - especially for people new to the site.
Solution: Multiple rooms: We decided to keep one main private mentor room, but split out the mentorship rooms into four so that conversations, if they did overlap, wouldn’t overlap frequently, because new mentees were cycling through rooms.
Problem: Uneven traffic: Even though we rate-limited by percentage of overall traffic, our traffic gets a lot of spikes. There was one Friday night in which one mentor was online and he was mentoring five people at once. Whoops. O_O
Solution: Smarter rate-limiting: Jeremy spent a bunch of time fixing the rate at which people were permitted in chat so that it scaled based on the number of people available to mentor instead of as a function of overall traffic. That way, a single mentor can’t get overwhelmed with too many mentees.
Problem: Entry points: We originally placed the mentorship chat entry point in the sidebar of the Ask a Question page. This was good for only capturing people who had high intention for mentorship, but mentees frequently entered chat with just a title and a line or two, which made it harder for mentors to help them fix up the question.
Solution: Change them: We moved the prompt from the sidebar to an overlay that pops up once you’ve finished your question. It asks if they want some help with fixing up the question before they post. That way, the question is already finished, and mentors can help much more effectively.
Problem: Mentor education: While every mentor that we chose was extremely capable and effective, there were lots of different approaches and opinions floating around, and a good amount of uncertainty about how, exactly, they should help.
Solution: FAQs: We quickly realized that we needed an FAQ document for mentors. We collaboratively created guidelines for everything from homework questions to fixing typos in code. This was a good reference for mentors when they were unsure about how to approach an issue.
Problem: Delays in response/non-responses: The system wasn’t perfect - sometimes, a mentor would be idling in chat, a mentee would come in… and no-one would notice. Other times, a mentee would come in, and the mentor would spend 5 minutes giving feedback on their question… only to realize that the mentee had left and posted their question anyway.
Solution: Kristinabot: We created an auto-response in chat that welcomed mentees, told them a bit about the chat, and asked them to talk a bit about what issues they have. This way, mentees had something to do while waiting, and if they never responded to the bot, mentors wouldn’t have to spend their time talking to someone who wasn’t there.
RESULTS!
So, now you have a clearer picture of how this worked and what issues we faced… what about its effectiveness? As I mentioned above, question score means increased by 50%. Practically, this means that questions from mentored users had fewer net downvotes than those that were not mentored.
Another way of measuring it is by looking at individual question quality. We used a rating system where “good” questions have positive ratings (or neutral with an accepted answer), “neutral” questions have no interactions, and “bad” questions have negative ratings. Within that system, mentored questions had a much higher share of “good” questions, and way fewer “bad” ones. Good questions increased from 18% of all questions asked by that population to 25%. Bad questions decreased from 30% of questions to 25%.
All of these results are statistically significant, with p < .05.
NEXT STEPS
We got a lot of positive (and constructive!) feedback from mentors, mentees, and from others at SO. This wild experiment exceeded our expectations, for sure. Going forward, there are a couple of ways we want to continue this work:
Continue to experiment with the concept of a draft post. We think that this would enable ad hoc versions of mentoring particularly for smaller communities. It could even replace the “sandbox” that some communities have hacked together.
For larger communities like SO, we’re going to explore ideas for a scalable mentorship program - though no promises, here, on any front. We’re also unsure about how chat fits in here long-term, so we can’t guarantee any changes or updates to chat. This will be prioritized in the context of other ideas for improving question quality.
Relatedly, if you didn’t see it, Joe posted an update about the DAG team’s priorities yesterday that outlines other relevant work that they’re planning and that they've done.
THANKS
We have a LOT of thanks to hand out:
To our mentors: THANK YOU SO MUCH. If it wasn’t for your patience, your dedication, your wonderful and useful feedback, and your Skynet jokes, this experiment would have been a total flop. I’m speaking for all of us at SO when I say that we really appreciate your time and your passion.
To our mentees (Are you reading this? Say hi!): thank you for being willing to try something new, for patiently working through SO’s rule intricacies, and for filling out the survey afterwards.
To y’all on Meta: Thank you for the encouragement and tough-but-actionable feedback on our first post. We were able to implement a couple of critical features in this project, including draft editing and room cycling, because of your great ideas and feedback.
To the members of the DAG team (specifically Jeremy, Joe, and Pawel): I dunno if I should be thanking you or if you should be co-thanking everyone else with me, but thanks for going along with this idea and, you know, building and designing it. Double-thanks to Jeremy for spending way too many hours making rate-limiting work.
To my fellow researchers (specifically Denae and Chris): I guess y’all should also be co-thanking everyone with me, but thanks for putting in so much work and time to make all this happen. It wouldn’t have happened without you.
FAQ (I get that you haven’t asked any questions yet, I’m just guessing)
Q: So wait… do you think this was successful or not?
A: Yes! By a couple of measures. First, we learned a ton. We learned about the kinds of things that people need help with when they’re first starting (MCVE! Formatting!) and we learned about the logistics of how to pull off a mentorship chat like this. It also proved successful, stats-wise. People who were mentored had overall better questions than people who weren’t. It makes sense to mention here that people who opt in to chatting might be predisposed to asking better questions overall, but anecdotally, most of the questions were in need of some fixing up.
Q: You said that over 500 people entered but only 271 had conversations? What gives?
A: A lot of people entered chat who maybe didn’t realize what they were doing, or who got there, realized they weren’t going to get answers, and left. This is an issue with the way we implemented this, and we’d need to fix it before we launched something more broadly.
Q: How did you calculate your results?
A: We looked at all of the questions asked by users in the same population (3 or fewer questions, <16 rep) in the same time frame, and compared them with our sample of mentored users. We compared their mean question scores - the difference was statistically significant.
Q: Okay, but how many people really wanted their question to be answered in chat?
A: There were some! But many times, when our patient and steadfast mentors explained that they wouldn’t answer the question and instead could help with fixing the question itself, mentees were excited about it and worked to fix up their questions.
Relatedly: I’ll be talking about this project on the Stack Overflow podcast that’s coming out this Monday, so you can tune in if you’re super interested and want to know more.
I’ll also be hanging out here for a while (with Joe and Jeremy!) to answer questions, respond to feedback, and talk with you all.
There was one Friday night in which one mentor was online and he was mentoring five people at once.
Sounds like a badge opportunity ;-)