175

Check out SO Podcast 117 to hear Kristina talk about the mentorship experiment.

First, if you haven’t read the original announcement post about the Stack Overflow Mentorship Research Project, you should go and do that now. If you don’t, the rest of this probably won’t make a ton of sense.

From the beginning of August to the beginning of September - 33 days, to be precise - we ran a small pilot of a mentorship program:

  • 63 mentors from the community volunteered to be our partners.
  • Over the course of the experiment, over 500 new users entered our mentorship chat.
  • 271 of those users had conversations with our mentors.
  • Comparing those 271 users’ questions with questions from all eligible users who did not enter or participate in mentorship, their question score (means upvotes + downvotes) increased by 50%.

IMPLEMENTATION DETAILS

Here’s how the system worked at first:

We began with one private room for mentors, and one room for the actual mentorship chat to happen. We limited the rate at which eligible mentees could enter the chat so that we weren’t overwhelmed. When someone came into mentorship chat from the Ask a Question page, we sent a notification to the private chat so that someone could “claim” that question, pop in, and help:

[notification in private mentor chat]

Because of your feedback, we started the project with a “draft post” feature implemented. When mentees entered the chat, their draft question was copied over. They can edit the draft from there, and then copy it back over to the Ask a Question page when complete. I don’t say this lightly: our experiment would have been a failure if we hadn’t had this from the beginning. Thank you. Here's what the draft post feature looked like.

  1. My draft is posted in the chat.

photo of draft post feature

  1. I can go edit my draft by clicking the "edit" button.

photo of post in process of being edited

ITERATION

Soooooo now I’ll go through all of the problems with our initial approach, and talk through the solutions we implemented. The point here is that this experiment was very much a learning process.

Problem: Overcrowding: As you all correctly guessed, trying to put a bunch of mentorship conversations into one room was a bit of an issue. Multiple conversations going on at once was confusing for everyone - especially for people new to the site.

Solution: Multiple rooms: We decided to keep one main private mentor room, but split out the mentorship rooms into four so that conversations, if they did overlap, wouldn’t overlap frequently, because new mentees were cycling through rooms.

Problem: Uneven traffic: Even though we rate-limited by percentage of overall traffic, our traffic gets a lot of spikes. There was one Friday night in which one mentor was online and he was mentoring five people at once. Whoops. O_O

Solution: Smarter rate-limiting: Jeremy spent a bunch of time fixing the rate at which people were permitted in chat so that it scaled based on the number of people available to mentor instead of as a function of overall traffic. That way, a single mentor can’t get overwhelmed with too many mentees.

Problem: Entry points: We originally placed the mentorship chat entry point in the sidebar of the Ask a Question page. This was good for only capturing people who had high intention for mentorship, but mentees frequently entered chat with just a title and a line or two, which made it harder for mentors to help them fix up the question.

Solution: Change them: We moved the prompt from the sidebar to an overlay that pops up once you’ve finished your question. It asks if they want some help with fixing up the question before they post. That way, the question is already finished, and mentors can help much more effectively.

photo of updated entry point

Problem: Mentor education: While every mentor that we chose was extremely capable and effective, there were lots of different approaches and opinions floating around, and a good amount of uncertainty about how, exactly, they should help.

Solution: FAQs: We quickly realized that we needed an FAQ document for mentors. We collaboratively created guidelines for everything from homework questions to fixing typos in code. This was a good reference for mentors when they were unsure about how to approach an issue.

Problem: Delays in response/non-responses: The system wasn’t perfect - sometimes, a mentor would be idling in chat, a mentee would come in… and no-one would notice. Other times, a mentee would come in, and the mentor would spend 5 minutes giving feedback on their question… only to realize that the mentee had left and posted their question anyway.

Solution: Kristinabot: We created an auto-response in chat that welcomed mentees, told them a bit about the chat, and asked them to talk a bit about what issues they have. This way, mentees had something to do while waiting, and if they never responded to the bot, mentors wouldn’t have to spend their time talking to someone who wasn’t there.

photo of Kristinabot in action

RESULTS!

So, now you have a clearer picture of how this worked and what issues we faced… what about its effectiveness? As I mentioned above, question score means increased by 50%. Practically, this means that questions from mentored users had fewer net downvotes than those that were not mentored.

Another way of measuring it is by looking at individual question quality. We used a rating system where “good” questions have positive ratings (or neutral with an accepted answer), “neutral” questions have no interactions, and “bad” questions have negative ratings. Within that system, mentored questions had a much higher share of “good” questions, and way fewer “bad” ones. Good questions increased from 18% of all questions asked by that population to 25%. Bad questions decreased from 30% of questions to 25%.

All of these results are statistically significant, with p < .05.

NEXT STEPS

We got a lot of positive (and constructive!) feedback from mentors, mentees, and from others at SO. This wild experiment exceeded our expectations, for sure. Going forward, there are a couple of ways we want to continue this work:

  • Continue to experiment with the concept of a draft post. We think that this would enable ad hoc versions of mentoring particularly for smaller communities. It could even replace the “sandbox” that some communities have hacked together.

  • For larger communities like SO, we’re going to explore ideas for a scalable mentorship program - though no promises, here, on any front. We’re also unsure about how chat fits in here long-term, so we can’t guarantee any changes or updates to chat. This will be prioritized in the context of other ideas for improving question quality.

Relatedly, if you didn’t see it, Joe posted an update about the DAG team’s priorities yesterday that outlines other relevant work that they’re planning and that they've done.

THANKS

We have a LOT of thanks to hand out:

  • To our mentors: THANK YOU SO MUCH. If it wasn’t for your patience, your dedication, your wonderful and useful feedback, and your Skynet jokes, this experiment would have been a total flop. I’m speaking for all of us at SO when I say that we really appreciate your time and your passion.

  • To our mentees (Are you reading this? Say hi!): thank you for being willing to try something new, for patiently working through SO’s rule intricacies, and for filling out the survey afterwards.

  • To y’all on Meta: Thank you for the encouragement and tough-but-actionable feedback on our first post. We were able to implement a couple of critical features in this project, including draft editing and room cycling, because of your great ideas and feedback.

  • To the members of the DAG team (specifically Jeremy, Joe, and Pawel): I dunno if I should be thanking you or if you should be co-thanking everyone else with me, but thanks for going along with this idea and, you know, building and designing it. Double-thanks to Jeremy for spending way too many hours making rate-limiting work.

  • To my fellow researchers (specifically Denae and Chris): I guess y’all should also be co-thanking everyone with me, but thanks for putting in so much work and time to make all this happen. It wouldn’t have happened without you.

FAQ (I get that you haven’t asked any questions yet, I’m just guessing)

Q: So wait… do you think this was successful or not?

A: Yes! By a couple of measures. First, we learned a ton. We learned about the kinds of things that people need help with when they’re first starting (MCVE! Formatting!) and we learned about the logistics of how to pull off a mentorship chat like this. It also proved successful, stats-wise. People who were mentored had overall better questions than people who weren’t. It makes sense to mention here that people who opt in to chatting might be predisposed to asking better questions overall, but anecdotally, most of the questions were in need of some fixing up.

Q: You said that over 500 people entered but only 271 had conversations? What gives?

A: A lot of people entered chat who maybe didn’t realize what they were doing, or who got there, realized they weren’t going to get answers, and left. This is an issue with the way we implemented this, and we’d need to fix it before we launched something more broadly.

Q: How did you calculate your results?

A: We looked at all of the questions asked by users in the same population (3 or fewer questions, <16 rep) in the same time frame, and compared them with our sample of mentored users. We compared their mean question scores - the difference was statistically significant.

Q: Okay, but how many people really wanted their question to be answered in chat?

A: There were some! But many times, when our patient and steadfast mentors explained that they wouldn’t answer the question and instead could help with fixing the question itself, mentees were excited about it and worked to fix up their questions.

Relatedly: I’ll be talking about this project on the Stack Overflow podcast that’s coming out this Monday, so you can tune in if you’re super interested and want to know more.

I’ll also be hanging out here for a while (with Joe and Jeremy!) to answer questions, respond to feedback, and talk with you all.

50
  • 29
    Another question to maybe add to the FAQ: Do we consider the net result of 271 mentees scoring 50% higher on their questions worth the amount of effort spent by mentors getting them there? Do we have a way of measuring the number of mentor minutes per mentee question score gained? Commented Sep 28, 2017 at 13:51
  • 11
    Were these questions more likely to be answered, and have an answer accepted? Commented Sep 28, 2017 at 13:52
  • 53
    What percentage of people that went to the room and actually said anything ended up actually posting a question? How common was it for the person to realize that their question simply couldn't be improved enough to be acceptable, and ended up not posting it?
    – Servy
    Commented Sep 28, 2017 at 13:57
  • 2
    @Servy We didn't measure that, aside from seeing very broadly in our data that people who entered mentorship were less likely to ultimately ask a question. I think that's a good thing - sometimes mentors explained why opinion-based questions didn't have a place on SO, for example.
    – kristinalustig StaffMod
    Commented Sep 28, 2017 at 13:58
  • 2
    @battery.cord Our metric for whether a question is "good" or not takes into account if it has an accepted answer, or if it's upvoted. So by that bar, yes. But I don't have that particular analysis right now. If I have some time later in the day I'll give it a look.
    – kristinalustig StaffMod
    Commented Sep 28, 2017 at 14:01
  • 7
    @gameweld The point of my question is to determine how much of the increase in average score is a result of improvable questions being improved, and how much of it was a result of bad questions just not being asked in the first place, and thus not dragging down the mean. Presumably there was some amount of both, but it would be nice to have some idea of the degree of each.
    – Servy
    Commented Sep 28, 2017 at 14:02
  • 2
    @NisargShah Indeed. It's unavoidable for it to happen some amount of the time. But there's a big difference between 5% of users ending up not asking their question and 25%. I was hoping for some [rough] idea of degree that the mentors acted simply as triage, vs when they acted as H&I.
    – Servy
    Commented Sep 28, 2017 at 14:04
  • 8
    One of the problems really for me was that whenever I was mentoring, I was bound to help the one person with their post, and wait for them to respond and edit the draft, instead of just me leaving a comment and moving forward. So in the end, instead of me commenting, deduping, voting on 25-50 posts in the same time period, I was waiting for one to come up with a new draft - that then never came up - and the OP didn't end up posting the question at all. Commented Sep 28, 2017 at 14:10
  • 2
    @Servy I just looked into the data, and it seems like the actual reduction in questions asked for mentored users was small (3-4%, I'm eyeballing).
    – kristinalustig StaffMod
    Commented Sep 28, 2017 at 14:17
  • 24
    One key thing I don't see factored into the results of how successful this experiment was is how the mentors felt about it and whether it's something they'd want to participate in again as it stands or when adjustments are made. If people aren't going to volunteer to mentor or feel it doesn't work for them and not do so... the whole thing just won't work to start with.
    – Jon Clements Mod
    Commented Sep 28, 2017 at 14:18
  • 4
    @AnttiHaapala Hello! Yes, something we should definitely think about going forward is the time tradeoff - many mentors would likely be spending their Stack Overflow time on something more efficient like deduping or editing if they weren't in mentorship chat. That's good feedback.
    – kristinalustig StaffMod
    Commented Sep 28, 2017 at 14:19
  • 14
    "Has accepted answer" is not a measure of a question's quality. Askers accept junk answers to poor questions all the time.
    – jscs
    Commented Sep 28, 2017 at 17:07
  • 3
    "As I mentioned above, question score means increased by 50%." From +2 to +3? Please give also absolute numbers when presenting relative numbers. ;) Commented Sep 28, 2017 at 20:04
  • 18
    I was the one mentoring 5 people at once, and I just want to say that it was totally manageable. Once I gave some suggestions, the mentee would take a few minutes to fix their draft, so I had plenty of time to help other mentees. I probably couldn't have done more than 5 people at once, but I just wanted to add that multi-tasking is very feasible when mentoring. Commented Sep 28, 2017 at 20:18
  • 7
    There was one Friday night in which one mentor was online and he was mentoring five people at once. Sounds like a badge opportunity ;-)
    – TylerH
    Commented Sep 29, 2017 at 18:00

9 Answers 9

88

As a mentor myself I was quite happy with the experiment and I do believe that it may work, however I have a comment in the quantitative results of the experiment.

Certainly I do remember that we did help some bad question to become good ones, so that is great! However, how many of the upvotes given to those questions where given by non-mentors? I did upvote questions when I though that the user made a legitimate effort in improving them that I would have no exposure to if I wasn't a mentor, so I think that perhaps if my votes count towards the "hey this worked" conclusion, the metric may be skewed.

Certainly not all upvotes would come from mentors, and certainly some of us may have also downvoted questions that were bad that mentees refused to fix, but I believe that the real metric should not take into account the people involved in the experiment, as we were a significant percentage of the people involved in the experiment (63 mentors vs 271 mentees/questions). The influence we might have had in the final result is not neglectable, I believe. Actually, it is very possible that our net influence in the final number is small, but we should not make assumptions.

TL;DR: I believe you should re-calculate that 50% figure without the votes from people involved in the experiment.


Clarification

I did not upvote every mentee question, nor every question by my mentees. I did upvote following my general upvote criteria. But I would not have had exposure to those questions if I was not a mentor

18
  • 11
    That's a really good point. I'm going to look into that: I wonder how common it was for mentors to upvote their mentee's questions, or if it was something that you and a couple others did only?
    – kristinalustig StaffMod
    Commented Sep 28, 2017 at 14:34
  • 20
    Heh. Automatic upvotes because the question showed research effort: they participated in a mentorship chat. Maybe not the best idea.
    – Cody Gray Mod
    Commented Sep 28, 2017 at 14:35
  • 2
    @KristinaLustig Speaking as a mentor, I did that a couple of times, in those times where I thought there was a really solid question coming out of the mentorship. Likewise, when people insisted on posting their bad question despite my advice, I also did downvote / closevote as appropriate.
    – Magisch
    Commented Sep 28, 2017 at 14:39
  • 16
    I'm honestly not sure that human beings feeling empathy for each other and showing support with an upvote is a bad thing. Especially if this is saved for people who showed an effort to improve their question. But then I'm a soppy, sentimental kinda person.
    – Joe Friend StaffMod
    Commented Sep 28, 2017 at 14:43
  • 12
    @CodyGray To be more clear: I did not upvote every mentee question, nor my mentees questions. I did upvote following my general upvote criteria. But I would not have had exposure to those questions if I was not a mentor Commented Sep 28, 2017 at 14:45
  • 23
    @JoeFriend That's... not how these sites are supposed to work. Let's not turn upvotes into likes, ok?
    – yannis
    Commented Sep 28, 2017 at 14:57
  • 5
    @yannis My point is that creating human connections where we learn to care about the person we're helping isn't so bad. Also, it was said with a bit of levity. Why so serious?
    – Joe Friend StaffMod
    Commented Sep 28, 2017 at 14:59
  • 36
    @JoeFriend Sure, but Ander's point is that it's possible that the mentor questions didn't have a higher score because they were a better question, but just because mentors upvoted the questions even though they had an on average equal quality. If that's were the case, then the mentorship didn't improve quality, it just changed the measurement of it. Of course, that's a hypothesis that would need to be tested in the data before acting on it, naturally.
    – Servy
    Commented Sep 28, 2017 at 15:00
  • @Servy Fair enough.
    – Joe Friend StaffMod
    Commented Sep 28, 2017 at 15:00
  • 3
    @Servy yes and no. While that could have happened, my point is that we were 63 people having increased exposure to questions. Perhaps just the higher exposure to the questions made us upvote more. I never dwell in JAVA lands, but I needed to do several times during mentorship, and some people did actually write a good question. But yeah, ultimately I think it should be at least tested. Commented Sep 28, 2017 at 15:02
  • 4
    @AnderBiguri That's what I was trying to say. If I implied that you were upvoting bad quesitons, then I didn't mean that. I understood your point as saying you were upvoting good questions that you otherwise wouldn't have, due to the increased attention.
    – Servy
    Commented Sep 28, 2017 at 15:10
  • 3
    "But yeah, ultimately I think it should be at least tested." Not should. I would think it must be tested if you want to know if mentoring did really help. Commented Sep 28, 2017 at 20:13
  • @AnderBiguri 63 people?, maybe 2-3 checked the question when it entered, but after posted I can't remember that I went often to questions that I did not mentor myself (maybe a few times when mentor complained in chat that OP did not listen and went on to post off-topic question, to review it on site and take appropriate actions), but true for the question you mentored yourself, if OP improved question and something nice and interesting came out, after editing on site I up-voted a few. Commented Sep 29, 2017 at 15:19
  • 1
    @PetterFriberg 63 people is just the number of mentors. But yes, some upvotes must have been from mentors, perhaps some downvotes also. Commented Sep 29, 2017 at 15:21
  • 1
    @AnderBiguri well if they ever think of rerunning it then ^
    – barlop
    Commented May 15, 2020 at 14:43
37

I found it to be refreshing and rewarding to participate in helping new users to ask better questions.

A full implementation of this could benefit from:

  • An automated system that only shows someone entering mentorship when they've already replied

  • A small timeout to prevent people from entering mentorship and then immediately posting their question anyways

  • true scalability based on active mentors and current question frequency.

  • the ability to mark yourself as "available to mentor" even when you're currently doing 1. Many mentees take a while to respond, so I could easily see myself handling 2-3 at a time if I wanted to.

1
  • 23
    A better system for "I'm available" would definitely be nice. That was hard, especially if you had other things going on when mentoring.
    – enderland
    Commented Sep 28, 2017 at 15:26
36

I also participated in this project, and I will submit my two cents about the project and what can be interesting for the future.

Collaborative editing

I had fun, but it was also time consuming, specially teaching each OP to format a post (instructions in chat, wait for result, repeat). My temporary solution after a while was to only concentrate to get all information in the post (and leave out formatting), then once it was posted I went to edit it (format code, etc.). It would have been nice if we had collaborative editing.

Keep the OP in chat

Keeping the OP in chat (have them respond) was a challenge as you pointed out in your post. Some thought needs to go into this to not waste both the mentor's and OP's time. Personally I had a simple strategy asking first simple questions like "How are you"?, to see if I could get a response and also to establish a minimum relationship avoiding that the OP gave up (darn all this stuff I need to improve) on me in middle of the mentor session. However I think that some hard interface should be put in place that encourage the mentee to stay in chat.

Avoiding off-topic or blunt duplicates

In your post, you don't speak about how many posts entered mentor chat, but were actually never posted. My personal experience was that probably all questions in my domain (Java) were never posted. The reason often was that I could easily find a good duplicate, indicate a typo in yet another scanner problem, etc. In all these cases the OP seemed happy to not post (compared to all the mess in comments on heavily down-voted posts).

As I see it, this was a major positive result by the mentor chat (both for community users and OPs. Jon and others have more time to answer a good question). I am aware that this can also be dangerous; chat becomes a help-center to teach how to search the web, but currently I prefer to indicate how to search, instead of involving multiple community users to review and execute moderation actions on the post.

We also avoid a fair amount of resource requests, super user, etc. questions even if it was harder to convince the OP and keep them in a good mood.

Opt-in to mentor only in certain tags

This is related to the previous point, but also it's simpler (and for some, more fun) to mentor when you have domain knowledge. The final result of the mentor session will normally be better, and I think it will be easier to get mentors to participate if they can do it on tags that they like; they feel more secure and often more motivated.

Conclusion

Probably it's no surprise that a new user with some mentoring can post (or not post off-topic) a question better than without. The challenge is to find mentors (of the 63 volunteers probably only a handful was really active), that are willing to invest time and have fun.

I had fun, thank you Kristina.

4
  • 33
    Opt-in to mentor only in certain tags Is something I would really like. being a C++ person there is only so much help I can give a JS person. Commented Sep 28, 2017 at 16:15
  • 6
    I had fun, but it was also time consuming - This. So much this. I commonly found myself one of two mentors (I was assigned to week 3) and would be fielding up to 3 mentees at the same time while also at my day job. The end result was that I could only help for 20 minutes at a time 2-3 times a day. Scalability will be important if this goes forward as this modal.
    – amflare
    Commented Sep 29, 2017 at 3:50
  • Who is "Jon" (not a rhetorical question)? Commented Oct 4, 2017 at 17:37
  • 2
    @Peter just another Doe without an h that I use to indicate someone that pass great answers, similar to using Peter to indicate someone that do great edits. Commented Oct 4, 2017 at 19:06
28

I took part as a mentor. Overall there was only one main difference which I saw in the mentees versus in the wild so to speak. Attentiveness.

In chat, if we asked for an update or an inclusion, the mentees would very often make an attempt at doing that for the question to be well received.

In SO main, if you ask an OP to make an update or an inclusion, the odds are pretty low that it will happen.

To me, this was the main distinction. If nothing else, this process allowed for new users to experience how attentive they need to be in order to produce a well received question without actually having a question pulverized on the main site.

It was also interesting to see the reasoning behind some very, very, very low quality questions which would normally not see the OP return. For the most part, a request for terminology seemed to be behind these posts. I think many times when a user asks how to foo the bar, we assume we need to make it for them, when often this assumes some malice. New users don't tend to ask for implementations so much as terminology or a broad view of how to accomplish the task, but it certainly reads as "build me the next facebook app" to most veterans here.

I am not sure what reconciliation process is available there, but it would be nice to sort it out so we don't constantly feel like people are asking us to do free work in the form of building or researching; and so new users don't constantly taken aback for asking about identifying process or technology.

One aspect of the workflow that sparked my interest was the ability for this to be turned into a queue. There is a preview, a chat room the user is in, and an entry point. I would imagine that there should be a way to make a queue for these mentee requests as the basics of the tooling are already built. This could greatly help build community and increase question quality.

I think the staff running this experiment did an excellent job. Jeremy did very well implementing the feature sets, Kristina was very available for communications and kept everyone in the loop, and Joe provided insights and feedback into the overall process.

9
  • 4
    Requests for terminology are obviously OK (though usually a duplicate) but a broad view on how to accomplish a task is running perilously close to a "too broad" question (meta.stackoverflow.com/questions/284236/…) If we are going to be OK with those questions that should probably be discussed on Meta seperately. Commented Sep 29, 2017 at 0:37
  • 7
    "New users don't tend to ask for implementations so much as terminology" This is a very interesting insight. Definitely worth some study as to how that can be addressed so that the intent is made clearer, and the questions answered in a long-term helpful way.
    – jscs
    Commented Sep 29, 2017 at 12:17
  • Overall there was only one main difference which I saw in the mentees versus in the wild so to speak. Attentiveness. Well, this is likely because those that don't bother to answers to update in main sites are the 239 ones that didn't bother to go in the chat room.
    – Walfrat
    Commented Sep 29, 2017 at 12:19
  • 3
    @Walfrat maybe true some, but I don't think it is that simple, you have a better relation with OP in chat, easier to exchange and express ideas, 1-1 communication, chat message less crude and more understandable for new user then a comment "You need to post [mcve]" + OP has not yet received tons of dv yet, hence their mod is better ;) Commented Sep 29, 2017 at 13:37
  • 2
    Two years ago, I failed to convince the community here on Meta that if a question too broad to be answered in detail is asked, an OP is typically okay with an equally broad answer (i.e. at the same level of detail). And now, I see clear evidence supporting this. Commented Oct 3, 2017 at 15:08
  • 2
    To be honest @ivan_pozdeev, I don't necessarily disagree with that. However, the complicated part is walking the line between questions that have acceptably broad answers and questions which are truly too broad. Figuring out how to make such a distinction could be worthwhile.
    – Travis J
    Commented Oct 3, 2017 at 18:47
  • I mentioned "at the same level of detail". So, "truly broad" questions are those for which an answer at the same level of detail is useless/impractical. "How do I build a web site?" - "You get a web server, then attach site-specific logic to it." (...Now, this doesn't sound so useless actually...) Commented Oct 3, 2017 at 19:13
  • If they are not asking about terminology in their question then they probably are not asking about terminology. Maybe they are confused about terminology, and should be asking about that or told about that, but that's another matter. It doesn't mean they are asking about that
    – barlop
    Commented May 15, 2020 at 12:55
  • @barlop - I think you are confused.
    – Travis J
    Commented May 15, 2020 at 15:12
23

As a mentor myself, I really enjoyed participating in this mentorship project, and I liked to guide people in all ways whether the question was blatantly off-topic or it had serious problems that needed to be fixed or it was just a fair enough question and just needed to be double-checked before posting.

Adding to the other answers that are already posted, I would like to recommend adding most if not all of these features if there will be any future updates to this project:

  • Not use 4 chat rooms only: A better approach but more costly to implement is to create separate chat rooms for each new mentee, so that the mentee doesn't get overwhelmed with the last mentorship session. Being able to see old sessions is not a good experience for the mentees. The chat should always be kept new and clean, so that no confusion occurs. I know that creating a new different room for each mentee is not completely feasible, but it would be more than great if that can ever get implemented.

  • Add status for messages such as "delivered" and "seen": To solve the problem of having mentees ignoring the messages we send that we sometimes waste our time to write, it would be a very good idea if that feature can be added to know whether the mentee is available to be mentored or not.

  • Add the possibility to claim questions with certain tags only that are defined by the mentors themselves: That will not only help the mentors' mentorship process but also will give a much better experience to the mentees themselves as the question will now ask about a certain tag that the mentor is very well-aware of and would like to help improve this question according to his/her area of expertise/interest.

    Minor related feature: Adding the ability to claim any kind of tags in general is also a good addition to this feature if the mentor prefers to do so.

  • Add the possibility to mark oneself as available for mentorship: Adding something simple like a checkbox will already get this feature easily implemented. Sometimes the mentor likes to read or even chat in the Private Mentor Chat room without wanting to help any mentee at the same time. I find adding this feature will solve the problem of having few mentors available online, as during the experiment we needed to leave the Private Mentor Chat room when we are not available or just currently doesn't want to mentor someone, so that the system doesn't list us as available.

  • Add a button to all pop-up questions in the Private Mentor Chat room that acts as a claiming method: Generally during the experiment, we needed to say something like "I've got [mentee's name]", so that other online mentors know that someone already got this and took care of it. Ideally, if we can add this button, thereby when pressed, some text gets appeared at the right instead of the starboard, that shows something like "[mentee's name] is currently assigned to [mentor's name]", that will give an easier and better experience to the mentors themselves.

  • Add a button for ending a mentorship session: During the experiment, the mentor was being marked as available to mentor if the last message the mentor posted in the mentee's chat room was less than 10 minutes ago. This was surely a good temporary solution, but adding this feature will surely be a better experience. Sometimes the mentor doesn't need a break between every mentorship session. ;)

  • Add the ability to edit the question with the mentee through a collaborative editing window: Through my experiences, being kept to tell the mentees to fix that and that and that, then post another draft, that did surely got boring at some point especially when trying to teach the mentee how to format his/her code. Adding this feature will save not only the mentor's time but also the mentee's time. I understand that that might already get fixed by other SO members by just editing the question when posted. However, the main idea of this project is to teach new users how to ask a new good question, so that in future questions, the mentee already has some quite knowledge on how to write a good answerable question as well as some knowledge in how to format the code. So by saving time, editing the question with the mentee collaboratively and discussing the changes with him/her through chat is a whole another level of mentoring new users.

  • Gain reputation for mentoring a mentee just as having a suggested edit that got accepted: Later on, after being finished with the mentoring session, the mentor should get some reputation as a bonus reward for mentoring someone and helping the mentee to improve his/her overall post. However the reputation should only be gained through certain ways to prevent any fraud. Currently I don't have an exact idea on how to recognize a mentorship session as successful to get the reputation points. (Maybe if the post got some up-votes after some period of time as a result of mentoring?)

    Edit: After having a second thought of gaining reputation and seeing the stuff that went around Documentation, there have been some nonconstructive documentation. Gaining reputation might not be a good idea because sometimes there might be mentors who would just like to edit mentee's questions quickly without concern only for the sake of "reputation". So it's definitely not a good idea. It should be a volunteering work.

  • Remove the starboard for the mentorship rooms: Thinking about the starboard, I think it's useless for mentorship rooms, especially if we are going to create a new room for each new mentee. Through my experiences, no message got starred either by a mentor or a mentee. A better feature to compensate that right side panel is to divide the screen into two halves, at the left side the chat as it is now, and at the right side the collaborative editing window.

At the end, all these points were just my thoughts and suggestions. If you have a different opinion or have any comment on any of the above ideas, it is surely welcome. I would like to know your opinion as well. Anyway, I really feel this project will become successful and will have a promising future only if some more features get implemented.

TL;DR: The mentorship chat rooms should have a different layout unlike the normal chat rooms, to be able to give unique features only made for mentorship purposes, and remove all the unnecessary unused stuff such as the starboard.

3
  • 10
    I agree with most point but I'm not sure about the reputation, reputation often mess up things (as documents), I would prefer no reputation and keep only mentors who do it for fun and the "better" cause of SO. Anyway, Thanks Hussein you really made a dent in mentoring lots of new user! Commented Sep 28, 2017 at 16:42
  • 1
    Thanks @PetterFriberg. It has definitely been good days to all of us. Yeah, I actually agree with you now after having a second thought. After seeing the stuff that went around Documentation, there have been some nonconstructive documentation. Gaining reputation might not be a good idea because sometimes there might be mentors who would just like to edit mentee's questions only for the sake of reputation. You're definitely right. +1 Commented Sep 28, 2017 at 16:58
  • 1
    "create separate chat rooms for each new mentee" - +1 for that, but this problem seems to require a redesign - something that would allow mentors to see multiple rooms at once (on the same page), and either be able to reply directly from there or seamlessly travel to and from any given room (e.g. rooms are pop-ups). Commented Oct 8, 2017 at 0:15
8

I liked this experiment overall. Primarily from the fact that most of the mentees who entered the chat were better off as compared to their peers (users with 3 or fewer questions, <16 rep),

  • They received fewer downvotes - as the mentors often ensured MCVE

  • Their questions were easier to understand - as either the mentees themselves improved the formatting as per mentors' suggestions, or the mentors fixed grammatical issues after the question was posted.

I have a few suggestions for the future iterations,

  1. Ability to Skip

    During the experiment, if I was the only person in the chat, I'd be assigned a mentee by default. It doesn't give me the opportunity to decide whether or not I can help someone. It might be a helpful feature for future phases of the project.

  2. Reducing the time required from mentors

    On one of the occasions, I had to spend roughly an hour with a mentee - who struggled to understand basic details such as where the edit button was.

    I suspect, it would get annoying for anyone if the mentees take too much of their time. Also, there is no polite way of saying, "You are taking too much of my time"!

    The mentees are definitely better off, but can we reduce the time the mentors have to invest? Perhaps as Antti Haapala mentions, leaving a set of comments, and moving on?

  3. Allowing Collaboration on the question draft

    As Antti Haapala mentions, all we could do is suggest improvements, but could not edit the draft ourselves. The reason was to let the mentees learn by improving the questions themselves, i.e. if the mentors just edited the questions, the menteed may not learn much themselves.

    I think this point should be revisited. The mentors could save some of their time if only they could fix minor formatting issues in the draft themselves rather that educating the mentees. I believe most mentees would still learn from that.

Conclusion

I liked the experiment purely because the mentees almost always benefited from the interactions. If I had to, I would participate in the project without any of the above mentioned improvements. But I would really appreciate if it took me lesser time per question.

Another thing that comes to my mind is today's announcement about Question templates (among other things). I feel they could be more effective in improving question quality in comparison to Chat! I understand it is just an idea at the moment, but I like that better than mentorship.

8

I think the real proof will be in the pudding. Will the mentees produce better questions long term, becoming good contributers to the site? I'd like to see a 6+ month follow up to see how this mentorship has impacted both these users future questions/answers, and general behaviour on the site.

1

As one of the mentors, I enjoyed the experiment thoroughly, and although most important points have already been made by various others mentors, I do have a couple of suggestions myself, which may also prove helpful for any future experiment (I hope there are!)

  • A automated template for asking questions

    This has been brought up before, and I personally feel that this was something that was needed, because I (and I'm sure several other mentors as well) found myself reminding almost every mentee that they should add their code (if any) as well as mention whatever research they had done, or conversely, ask them to research their problem a bit first.

    Since this was something that was repeated for almost every mentee, this is something that could be (and in my opinion, should be) mentioned to the mentee (perhaps via Kristina Bot?)

  • Reminders for Mentors

    At the beginning of the experiment, after the selection process was over, all of us had received an email letting us know that we had been selected as mentors for the experiment, as well as mentioning the weeks that we were assigned to mentor during, with a helpful (and needed) reminder that we were free to mentor during the other weeks as well or as Kristina put it:

    You’re of course allowed/encouraged to hang out there whenever you’d like, we just want to try to avoid everyone piling on in the first three days and then leaving forever. :)

    However, it did happen. :(

    The day that I received the aforementioned email with a link to the private chat, I saw numerous mentors coming into the chat, and this continued into the first week of mentoring as well. There were infact times when there were mentors sitting idly in chat (including myself) with no. of mentees > no. of mentors. However, after the first couple of days of the experiment, the no. of mentors dropping in to mentor dropped noticeably, and the absence was even more pronounced as the days followed.

    I might be gravely wrong here, but I suspect that this was due to the fact that people who'd been assigned the second/ third week simply forgot about it, while juggling between work and home. A reminder email to the mentors before the beginning of their assigned week might've taken care of this. (I did not receive one, so I'm assuming neither did anyone else)

    This absence of mentors would in turn lead to times when available mentors were juggling between >1 mentees. This was rectified by the updated rate limiting, however, the temporary halt to the experiment was only mentioned in the private chat. During the second week of mentoring, there were several mentors (who'd been assigned the second week of the experiment to mentor in) who came into the private chat and only then found out that the experiment had been halted temporarily.

    In addition, the dates for when the experiment would be brought back online were again olny mentioned in the private chat, leading to several mentors not finding out about the same and consequently not being available for mentoring. Again, a broadcast email regarding the same might've solved this. (Again, I did not receive one, so I'm assuming neither did anyone else)

Having said that, the experiment was great, and from my experience, I feel that it achieved (in varying capacity) its goals (mentioned below, quoted from the email):

the goal of this project is threefold: (1) to increase question quality overall, (2) to improve the experience for new question askers, and (3) to learn how to scale these benefits to the whole community.

The quality of the questions posted were certainly better after the mentee had been guided by the assigned mentor. The mentorship also meant that duplicates were resolved in the chat itself.

All in all, this was a good experiment, and because it was an experiment, it was obviously not perfect and there is a lot of room for improvements/ additonal features (all the points mentioned by the other mentors above) to be added to the mentorship program if it is to be introduced as an integral part of the SO community, and I look forward to seeing this experiment growing to become one.

2
  • 2
    Heya - good feedback! I did actually send out broadcast emails... but I suppose not all of them arrived intact. :( I'm sorry about that!
    – kristinalustig StaffMod
    Commented Sep 29, 2017 at 20:20
  • @KristinaLustig, ah just my bad luck I suppose ): Commented Oct 1, 2017 at 15:39
1

Congratulations to the team and volunteers on a prototypical experiment.

There are ample users on Stack Overflow who participate (and answer) because of the fun and hunt of reputation and rewards and not primarily because of helping others or making the content qualitative or better for future readers per se. The reputation and badges are fun and simply exist because they are loved. Moreover there are some users who want reputation and a good Q&A profile for mentioning in resumes and/or developer jobs.

I am skeptical therefore of how mentors would be rewarded and how the whole structure of the number game will be included in this new type of participation. And if it won't, then I doubt that mentors will be as numerous as expected. (I could be wrong.)

Having said that, I still believe that SO is a strict Q&A site and Not a Forum, and by moving in a direction of using chat seems to break this strictness. Even though the previous post said that mentors won't answer the questions, but will help new users to improve their question quality, this might be a goof up, as a person looking for answers won't be so patient as to learn 'how to ask' when there are other alternatives out there in the huge digital world which can answer him without extra and unwanted forced teachings.

It is even a little strange how SO is putting more money and resources in just to teach 'how to ask', instead of other high-priority tasks (according to me) like making high quality answered questions more discoverable (you can't rely only on search engines, can you?), or making developer jobs more convenient for freelancers or part-timers; the list is long.

Well, let's hope that after lots of community involvement, this feature does not sunset like "Documentation".

3
  • 1
    I understand your point of view and have a lot of empathy. But the up-and-coming generation of SO users just aren't limited to a strict Q&A perspective in their expectations on how they interact with online sites. SO is going to have to accomodate this fact and atleast be able to change with the times, or it will go away. Not quickly, but assuredly.
    – ouflak
    Commented Oct 3, 2017 at 7:39
  • Thanks. True, Up and coming generation of SO users aren't limited to strict Q&A perspective, but at least what has kept current generation here is the fact that Q&A here are well moderated and are under constant surveillance that a low quality and even a duplicate post is cleaned up and tracked. Not only moderation, but tracking duplicates, indulging rep game and badges--everything will be seriously affected if chats were used primarily (like other forums)-leading to destruct the very core SO principles, quality, moderation and cleanups currently the site has. I fear that. Commented Oct 3, 2017 at 7:52
  • 1
    " ...putting more money and resources in just to teach 'how to ask' ..." I don't feel either way like they should or they should not put money and resources there but I feel like it is one of the things that could really improve. It's one of the important points, but yes, there are other important points too. Commented Oct 3, 2017 at 17:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .