81

We’d like to share some results from our experimentation with two earlier releases: the first, a tool to generate titles for your post, and the second, a tool to help users format their questions’ bodies. We’re sharing both sets of results at the same time because we think the insights they offer are similar to each other, and taken together they can better communicate the way we’re thinking about proceeding from here.

Not to bury the lede: We won’t be proceeding to release these features on the platform in their current iteration. By the numbers, these features did seem to improve the experience of the people who used them; however, they didn’t present a sufficient benefit to on-site outcomes to justify releasing them in their current forms. The content formatting assistant also experienced some issues during its launch, such as hallucinations, and inadvertently fixing the bugs with users’ code while editing their questions, which were not desired behaviors. We may still look to test incorporating features like these into the site at a later point since the users who used these tools did appreciate them. However, before we do so, we’ll have to take more time to develop and test appropriate safeguards, and flesh out exactly how we hope users will use these tools on the platform.


With that said, let’s start with the title-drafting assistant’s results. We tested four variants over the course of this experiment:

  1. The ask page as it currently is, with no changes.
  2. The ask page as it currently is, except we move the ‘title’ field to the bottom of the question page, after a user writes their question body.
  3. The ask page with title-drafting assistance.
  4. The ask page with title-drafting assistance, except the ‘title’ field is moved to a step after the question body is written, similar to variant #2.

For more information about the variants, you can check out the original post on the topic.

We measured a few key values from this experiment, which I’ll summarize briefly.

  1. The ‘question completion rate’ represents the percent of the time a user begins to write a question, and then actually posts one. While it’s hard to tell what the maximum reasonable value here is, we hoped that the question completion rate would see a slight boost, or at least be neutrally affected, by AI assistance.
  2. The ‘time to question completion’ represents the amount of time a user spends drafting a question before posting it. It takes users a considerable amount of time to write a question on Stack Overflow – an average of around 11 minutes. (The distribution is long-tailed, with some users taking significant amounts of time to write questions, but the vast majority of questions come in close to 11 minutes.)
  3. The ‘question success rate’ represents the percent of the time a question receives one of the following: 2+ question score, 2+ total answer score, an accepted answer, or at least two non-deleted answers. While there are other ways to define question success, we generally accept this definition as representing the improvement or degradation in the question asking experience on the site.
  4. Edits to the question title – this one’s fairly straightforward.
  5. The ‘satisfaction rate’ is the percentage of people who gave positive feedback for the suggested title on the ask page. (Users had the option of giving ‘thumbs up’ or ‘thumbs down’ feedback.)

Without further ado…

Baseline (#1) Variant #2 Variant #3 Variant #4
Question completion rate 48% 47% 46% 48%
Time to complete question 10 mins 36 secs 10 mins 30 secs 10 mins 12 secs 10 mins 36 secs
Question success rate 22% 23% 23% 23%
Edits to the question title 10% 10% 8% 7%
Satisfaction rate n/a n/a 88% 94%

So, here’s our subjective evaluation. Moving the title to the bottom of the question page by itself doesn’t seem to have an impact on the questions’ success rates. However, generating AI-assisted titles one step after the user writes their question body is better than leaving the title in the position it’s currently in. Otherwise, we do see a small reduction in the amount of time it takes users to write their questions, but it’s not by an overwhelming amount. Question success has an analogous story: title-drafting assistance didn’t seem to harm the user experience surrounding question asking, but neither was the improvement significant enough for us to pursue further as-is in this iteration.


We ran essentially the same measures for the content formatting experiment. However, the test variants were naturally different. We tested both on the Ask Wizard, and on our normal ask page (“AskV2”).

  1. The Ask Wizard page as it currently is, with no changes.
  2. The Ask Wizard page with AI support, and with the question review step removed.
  3. The Ask Wizard page without AI support, but with the question review step removed.
  4. The AskV2 page as it currently is, with no changes.
  5. The AskV2 page with AI draft-editing assistance.

Before you read too much into this table, please keep in mind – due to issues with the experiment, it was only on for a short time. Normally we’d want to leave an experiment like this running for at least a week or two, in order to capture a better understanding of what normal user activity should look like. So I’d recommend tempering expectations a bit for how strongly these results can be interpreted.

Ask Wizard - Baseline Ask Wizard Var. 2 Ask Wizard Var. 3 AskV2 - Baseline AskV2 - AI
Question completion rate 46% 51% 49% 70% 69%
Question success rate 15% 16% 18% 28% 29%
Edits to the question body 46% 36% 46% 30% 36%
AI suggestion acceptance 83% 83% 69%

There was a noticeable improvement in question success rate for newer users who were using the Ask Wizard, though not an overwhelming one. For users who are more experienced, we saw no meaningful change in question completion rate. Newer users also saw the most benefit from the AI content formatter, though again the margin here is very small. The most significant impact seems to be on the question body edit rate, which - for new users only - seems to have fallen around 10%. Though, the degree to which this is due to user engagement differences on these questions, or a genuine reduction in need for edits, was not directly tested.

Overall, newer users liked the content formatter’s suggestions significantly more than established ones. This isn’t too surprising, as new users are often the ones who need the most help massaging their questions to form.


So, where are we going from here? Well, both experiments have concluded, and we’re not planning to release these as features to the site as-is.

On the positive side, the results we see here show that these features are at least safe for us to play around with, which is a good baseline result – we don’t expect further experimentation with these features to have immediate, catastrophic effects. This means that there’s room for us to play around with design, implementation, and workflows without risking severe effects in production. (Yes, it is clear to us that if we want to publish final features, we need better safeguards to proceed.)

On the downside, these results aren’t as good as we were hoping for. While there does seem to be a reduction in curator workload as a result of these features, we were hoping that they would have more of an impact on the questions’ overall success rates, and therefore give a greater proportion of the users who come to our sites a good experience. While folks did overwhelmingly appreciate having AI suggestions on hand (even experienced askers liked it on the balance), this alone doesn’t necessarily make the overall user experience better if it doesn’t lead to better on-site outcomes.

The net result is that we’re going to shelve the public release of these features for a bit while we work on better proposals for how to integrate it into user workflows. And, if we believe our ideas are promising, then as usual, we’ll come back here to share more details and ask for your feedback.

20
  • 3
    "Bury the lede": US - Well that explains why I'd never heard it before. Commented Jul 11, 2023 at 17:05
  • 2
    @Nickistired it's not a common phrase here either. Commented Jul 11, 2023 at 18:36
  • 19
    It's almost exclusively a journalism phrase--if you aren't reading a lot of journals/newspapers you probably aren't familiar with the saying. Use also seems generational--older generations who got much of their news from newspapers tend to be more familiar with it than younger generations who never grew up with traditional print media.
    – TylerH
    Commented Jul 11, 2023 at 18:41
  • 2
    "Moving the title to the bottom of the question page by itself doesn’t seem to have an impact on the questions’ success rates." Well, sure, except for the impact your table shows it had: a 1% increase in success rate. At over 3,000 new questions per day, that's 30 questions (or, put another way, potentially 30 people) on SO that could succeed where they otherwise might not. It's small, to be sure, but it's meaningful. It's even more impactful if you consider that there appeared to be no negative impacts or trends from the change.
    – TylerH
    Commented Jul 11, 2023 at 18:43
  • 3
    It would be great if all the data for this test (at least the first one, or both individually) was released in a dump for review so people could comb through it, for any questions that were ultimately posted (to preserve privacy of anyone who didn't actually submit their question), showing both the initial draft and the AI suggestions, along with what the user ultimately chose. This way the entire community (including those who didn't participate in the test directly) could determine at a much deeper level whether there are any problems or shortcomings in the test, and offer feedback.
    – TylerH
    Commented Jul 11, 2023 at 18:55
  • 35
    @TylerH, a 1% increase in the success rate is almost certainly within the margin of error.
    – Mark
    Commented Jul 12, 2023 at 4:13
  • 6
    You keep saying 'AI', but I do hope you realize that there is not yet, and may never be, any true Artificial Intelligence ever developed. Such a thing is speculative science fiction at the moment. As we make more progress towards our philosophical and scientific understanding of what 'intelligence' really is, we learn that we are much farther away from even being able to feasibly attempt such a feat in the distant future than we were before. It might not even be possible. Just please keep that in mind when using the term 'AI'. There is a BIG difference between a chat bot LLM and 'AI'.
    – ouflak
    Commented Jul 12, 2023 at 8:51
  • 5
    I'm misunderstanding something. The "Ask Wizard Var. 3" column has a figure for "AI suggestion acceptance", but isn't that column "The Ask Wizard page without AI support, but with the question review step removed."?
    – tgdavies
    Commented Jul 12, 2023 at 10:01
  • 11
    @ouflak Please don't confuse the area of general intelligence with the wider computer science field of Artificial Intelligence, LLMs are most certainly a form of AI, but they are not a form of general intelligence. Commented Jul 12, 2023 at 12:36
  • 3
    @MarkRotteveel, It would appear, from all that I am seeing from Stack Exchange, that I am not the one with that confusion. So far, it appears that they genuinely believe that LLM's are true Artificial Intelligence. LLM's simulate an aspect what we consider to be artificial intelligence (that goes into philosophy... so will leave it), but if they truly are, as you put it, 'a form of AI', then they are at a minimum sentient. ChatBot LLM's are not.
    – ouflak
    Commented Jul 12, 2023 at 13:22
  • 11
    Arguments about terminology don't change what the described things fundamentally are. For our purposes, we understand what the staff are talking about when they say "AI"; and if the company is deluded about what those products can actually do, that's their problem (although I don't mind gloating). Commented Jul 13, 2023 at 3:28
  • 10
    About 30% of the time I try to ask a question on StackOverflow, the Ask Question page has popped up a helpful-enough duplicate while I was writing that I haven't had to ask the question, which duplicate I hadn't been able to find before while searching. I would count that as a success -- I got an answer to my question without even having to ask it -- but it counts as a failed Question Completion by your metrics. Commented Jul 14, 2023 at 6:44
  • 11
    @user207421 No, “lede” is not a “mis-spelling”. It’s a word created ~70 years ago (to be distinct from “lead” the metal used in typesetting). You don’t have to like it, but it’s firmly established. proofed.com/writing-tips/…
    – nobody
    Commented Jul 16, 2023 at 13:45
  • 2
    @TylerH We reported confidence intervals internally. These posts are written for a general audience, though. I'd have taken more effort to put our CI in context if the results had held more meaning for the site - as it stands, most of these values are stat. significant, but do not mean much for on-site actions regardless.
    – Slate StaffMod
    Commented Jul 17, 2023 at 15:56
  • 4
    I'm finding it humorous just how many pings I've gotten since creating this post about the word 'lede'. At this point I'm forced to admit that it might not be as common knowledge as I took it for...
    – Slate StaffMod
    Commented Jul 22, 2023 at 3:52

6 Answers 6

75

One big problem with the way these tests are set up is that they never test the quality of the titles or edits directly. All the actual metrics are more indirect measures of success.

I worry that at a later test, if by chance some metrics look good SE might go ahead even despite serious quality issues simply because they're not systematically measured. Of course you might get a lot of feedback on that on meta, but I'm not that confident that people shouting on meta would be a reliable way to get that feedback to the right point and taken seriously.

There are also fundamental issues with the current generation of generative AI tools, even with guardrails there isn't really a good way to prevent bad quality output in these cases. Unless you systematically evaluate the actual quality of the suggestions, your data will not show you this. I don't see a way other than having actual humans evaluate a random selection of suggestions collected during the test to evaluate how well it works. This is a lot of work, but if you skip that you risk introducing bad tools to the sites.

On the positive side, the results we see here show that these features are at least safe for us to play around with, which is a good baseline result – we don’t expect further experimentation with these features to have immediate, catastrophic effects.

I disagree here, you actually don't know if there were any catastrophic effects. It's entirely possible the suggestions feature completely butchered a question by a new user that didn't know enough to decline the suggestion. This is not something the test evaluated at all, so you simply don't know if it failed spectacularly in some cases. You can only measure if it had failed on such a grand scale that the indirect metrics shifted enormously, but the bar for this kind of feature should be much higher than "it didn't immediately destroy the site".

2
  • 23
    i mean, when the purpose of a feature is to support new marketing copy, not destroying the site is probably all that matters.
    – Kevin B
    Commented Jul 11, 2023 at 18:23
  • 2
    In particular, "question completion rate" is a strange metric. If a question is duplicate, we want that to be low because the asker found the answer without having to finish the question. Commented Jul 21, 2023 at 20:19
33

I was really hoping to see this study produce some sort of metric regarding the accuracy of the title generator. Here's a suggestion for how to evaluate title quality for future experiments.

Present the user with the standard "Ask Question" UI that requires the user to manually input a title. Have your title generator create a set of titles. Do not display these anywhere on the page, but attach them to the question in a hidden field. After the experiment is over, compare the user-supplied title against the bot-generated titles. This doesn't alter the UI in any visible or functional way, so you could collect this data for as long as you want without any impact on the user experience.

You can measure success in several ways. I imagined something like the current review queue, where experienced users can click through a sequence of questions. The page could display the bot-generated titles and ask the user to rate how similar each one is to the human-supplied title. Or, the page could display the question plus the bot- and human-generated titles and ask the user which one fits the question best. Those are unfortunately manual processes but since the expected consumer of the output will be a human, I don't see any way to measure quality without a human in the loop.

For best results, limit this test to existing users whose site history suggests that they are likely to write an accurate, high-quality title. You could even run this test against existing questions, which might give you the best apples-to-apples comparison. Find questions whose title was edited by someone other than the author, then run the title generator on them. Compare the original title against both the bot-generated titles and the human-edited title. Were the bot suggestions as good as the human editor?

The ‘question completion rate’ represents the percent of the time a user begins to write a question, and then actually posts one.

I'll caution you against paying too much attention to this particular metric. I can't tell you how many times I've solved a problem simply by trying to explain it to somebody. It forces you to think about the problem from a different angle, and is often enough to clear up the misunderstanding that was causing the problem. People sometimes call this rubber duck debugging.

When a user starts writing a question but doesn't post it, a lot of times it's because they answered their own question in the process. That should be counted as a success, since what they came here for was an answer. My own personal question completion rate on SO is probably under 15% for this very reason.

An automatic question drafting assistant would mean that the question asker did less of the work. That means less opportunity for the rubber-duck effect to solve the problem. For that reason, I'd expect the bot-assisted ask wizard to see an overall increase in the number of questions that end up surviving and getting posted.

This metric also appears to ignore cases where (for example) the asker viewed one of the "similar questions" that the wizard displays, finds their answer there, and subsequently abandons the question. Those questions should be filtered out of the data set since the goal of that feature is to eliminate the need to post a question in the first place.

What you're attempting to measure here is a lot more complicated than your current model appears to take into account, and there are a number of aspects which can't really be measured (e.g., number of people who answered their own question prior to posting). For those reasons, I suggest not using this metric in the future.

2
  • 4
    This is indeed a far better experiment on a scientific point of view. But the risk with your proposal is that the conclusion will be that the automatic title does not improve the overall quality. As I believe that the whole adventure is only to prove that the AI generated thing is marvelous, any test that could say the opposite has no chance to be used... Commented Jul 19, 2023 at 21:08
  • I wonder which works better in terms of answer metrics: a title that accurately reflects the boring and routine nature of the question, or a clickbait title that makes it sound more interesting than it is? Commented Jul 25, 2023 at 0:27
21

The complete lack of any kind of qualitative analysis is sorely disappointing. A relatively charitable reading is that those who conducted the experiment don’t understand that quantitative metrics like the number of edits are at best a crude proxy for quality. A less charitable one is that the whole point of the exercise is a vain pursuit of ‘engaugement’ (treating users as a series of gauges that you can tune to maximize advertising income), without regard for what actually brings people here.

Here is one study, of titles specifically, that could have been done instead. Take, say, 2000 questions per bucket (one for each variant of the UI, and obviously one in a control group). Write down all titles for them: provided by the asker, edited in by other users, suggested by an LLM, possibly even (or perhaps not) draft titles. Classify titles manually into a number of categories, for example:

  • completely uninformative (‘hi folx can yous guyses help me plz’)
  • very vague with minimal context (‘problem with python’)
  • misleading (‘FileNotFoundError after adding two numbers’)
  • almost good (‘opening a file raises an error’)
  • decent, maybe too specific (‘trying to open C:\Users\anna\file.txt raises FileNotFoundError’)
  • perfect (‘FileNotFoundError when a path contains backslashes’)

Only then perform any analysis. Observe whether each title is coherent with the question body. Observe whether LLM titles fall into a higher-quality category than askers’ own. Observe whether putting the title box at the end prompts askers to come up with better titles on their own. While you’re at it, observe whether edits from the community actually move titles into a higher-quality category, or if they’re constrained to typo fixes. And of course, count the frequency of each category of title in each bucket. This is the kind of analysis that would tell us something.

The number of edits alone tells us nothing. Some edits are trivial spelling fixes, some remove obvious problems without substantially improving quality. And even if a title is decent, it may need to be edited. Titles are hard! Sometimes a good title for a question can only be found in retrospect, when the solution to the problem is known, because only then there is certainty about which circumstances were relevant, and which were not.

The only positive I see from this post is that the LLM tools from this experiment will not be made generally available, so at least we know things will not get any worse than they are now. Even that, though, is damning with faint praise.

6
  • 1
    "titles…even drafts". Ick. Not unless the user pushes a button to knowingly submit that. All too often I'm accidentally pasting sensitive info into my drafts.
    – Laurel
    Commented Jul 16, 2023 at 17:58
  • Into a question body I’d understand, but into the title field? I usually write titles from scratch on the spot, and if I don’t, then it’s usually because I copy-pasted the question body too (which I have already cleared of sensitive info).
    – dumbass
    Commented Jul 16, 2023 at 18:18
  • 1
    It just feels really icky. What about copying error messages (which often contain file paths with your name)? In my particular case, my keyboard is wonky and I'm easily distracted. When a similar conversation happened on the main meta, others also brought up GDPR concerns.
    – Laurel
    Commented Jul 16, 2023 at 18:28
  • Alright, the study might still be useful even without considering draft titles. It would lose information on whether various kinds of nudging the asker towards better titles work, but it’s not the central point of this hypothetical study anyway.
    – dumbass
    Commented Jul 16, 2023 at 18:38
  • The rest of it I'm on board with :)
    – Laurel
    Commented Jul 16, 2023 at 18:44
  • engaugement isn't a word. Do you mean engagement? - "3. (uncountable, by extension, about human emotional state) The feeling of being compelled, drawn in, connected to what is happening, interested in what will happen next." Commented Jul 23, 2023 at 12:15
15

Thank you for this detailed report and status update! I appreciate being included and hearing the reasons for your decisions. Your decisions make sense to me.

Based on this data, the title tool (Variant #4) looks like it may have had some benefit. It reduced the percentage of posts whose title was edited from 10% to 7%.

It would have been better if you had directly measured the quality of the title, in a direct and apples-to-apples way, so you can compare the quality of titles currently to titles in the new method. But the indirect evidence listed here suggests that the title tool might well have been beneficial in increasing the quality of titles.

I can understand that the benefit seems small and it might not warrant the effort needed to bring it to production or the downside risks of using AI for this purpose.

2
  • 20
    10% to 7% is likely within the error threshold. Furthermore, given the way that generative AI works, the edit percentage is likely to be highly misleading. That is, generative AI is well-known to create plausible-looking English prose. It's not going to create titles that look like they are in need of editing, so it makes sense that the number of edits is going to go down, compared to poorly-written human titles. The problem, and what makes genAI dangerous, is that it has no idea whether the titles it generates are correct, and neither does the average reviewer who might edit them.
    – Cody Gray Mod
    Commented Jul 13, 2023 at 4:06
  • 2
    @CodyGray-onstrike, good points! I accept your criticism. I didn't think of either of those. Thank you for pointing that out.
    – D.W.
    Commented Jul 13, 2023 at 4:11
12

It's good that the experiments are being done. However, these tests do not address my concerns with allowing and incorporating LLMS/AI on SE. My concerns are based around the failure cases, not the success cases.

My concerns are:

  1. Question/answer quality will superficially improve, but create confusion in the long run by butchering the intent or inventing things. Put the other direction, trust in the average question/answer will decrease, and visitors will need to work harder than before to check the validity of what they find.

  2. SE will be inundated with questions/answers that are more busy-work than valuable discussions and increase moderation costs for no real gain. Moderators will have to work harder than before for this and the first reason above.

  3. There is intrinsic value for individuals to spend the time to craft a good question/answer. Providing a method to create a superficial question/answer that looks good encourages people to never learn those skills.

In the sense that LLMS are used as very advanced spellcheckers, they can be useful. But the boundary between that and the negative outcomes above is very blurry right now. I believe that experiments should focus on measuring these negative outcomes - rather than measuring an increase in engagement metrics - and would need to be measured separately for each implementation. Inverting the aims like this and measuring failures will enable a better conversation about whether any particular implementation is worthwhile.

It is difficult to measure the benefit of AI, since it tends to maximise metrics, often to the detriment of real outcomes. Only really robust results should be trusted. It is heartening to see that hallucinations came up and were part of the decision to not proceed. This goes to show that these concerns are not unfounded, and, hopefully, that they will be taken seriously. Definitely keep doing experiments before any LLMS are included anywhere on SE.

3

..these results aren’t as good as we were hoping for

Just concentrating on this. It's probably important to find out why the results weren't that decisive. I imagine it could be that the content was so low quality to begin with, there wasn't much to work with for your tool, or, the tool wasn't very efficient, or, the content was so high quality to begin with that no improvement was needed.

Maybe you can find a way to differentiate between these cases, especially in conjunction with direct human estimation of the quality of a title or the formatting.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .