76

June 7th 2023 -- The experiment has now concluded, and the title drafting assistant is no longer enabled on Stack Overflow. We will update you soon with the results of the experiment. Thanks all for your feedback along the way.


This experiment is now live, and testing is under way. Also, Philippe (the VP-Community at Stack Exchange) has created a post on Meta Stack Exchange about the launch.


Beginning the week of May 15th, our developer team plans to experiment with an assistive feature to help users generate more descriptive, accurate titles for their questions. This experiment builds on the proposal first introduced on Meta Stack Exchange, where we shared some early draft screenshots of what such a feature could look like, and how it might perform. This experiment will run at most three weeks, but we will likely have all the data we need by the end of the second week. (We may do more rounds of experimentation later, if needed.)

This experiment is designed to provide critical insights into whether such a feature could improve the day-to-day experience of askers, answerers, and searchers using the site, and if so, what forms it should take. (We will also be taking the opportunity to test a couple small layout changes for the Ask Wizard page, including relocating the “Title” input box to a step after a user writes the question body text.) Once this initial experiment concludes, we’ll be opening up a data-informed Meta discussion about what such a feature should look like, and how such a feature can function on the platform in its final implementation.

I know the introduction of these features is a sensitive topic, so I want to give you a bit of a peek behind the curtain at what we’re actually going to test, what we’re hoping to learn from it, and how many users we’ll be running this test with. I also want you to know that we are not rushing this forward into implementation: the design is certainly not set in stone. Community comments will be carefully considered, and your feedback is more than welcome.

How we’re expecting this test to look on the platform

We’re going to be testing four different layouts for the Ask Wizard. All experiments will run on the Ask Wizard page only at this time.

  1. Ask page and review page: No change from the current state.
  2. Ask page and review page: We’ll move the ‘title’ field to the question review step, and ask users to write one after they have written their question body and tags. We won’t offer the title assistant in this variant.
  3. Ask page and review page: We’ll offer suggestions for titles in this variant, during the review step.
  4. Ask page and review page: We’ll move the ‘title’ field to the question review step, and we’ll also offer title suggestions at this step.

Our plans for this experiment

For each layout we are testing, we will collect data for around 9000 questions. We’re principally interested in assessing five core questions:

  1. Do users’ questions perform better when they’ve received title-drafting assistance?
  2. Are titles written with assistance edited more or less often by community members?
  3. Do readers and answerers interact differently with questions that received title-drafting assistance? For example, via reviews or comments.
  4. Do better titles lead to a reduction in users – particularly new users – abandoning their questions? Does it increase the rate at which new users come back to the site in the future?
  5. How many users will accept the titles that are recommended to them?

While we have a couple of experiments we plan to start with, we’re going to be revising this plan as we go, based on information we learn from prior results. This means that we don’t have a pre-prepared list of the exact statistical tests we are going to run, nor the full list of A/B tests ready at the start of this experiment. On the flipside, this means that if you think there is something specific you feel strongly we should test as part of this experiment, please let us know in the answers below, and we may be able to work it into our plans.

Feedback - what we’re looking for, and where to post it

We’re interested in a wide range of feedback on this experiment. However, we’d ask that you hold on to feedback about the appearance and question workflow integration for the time being – we’ll come back to this when we’re ready to begin final design and implementation, once the results of this experiment are in. Lastly, we’ve also already gotten some feedback on Philippe’s Meta Stack Exchange post about this proposal, which I’d encourage you to look through and read, as other users’ thoughts may inform the feedback you’d like to provide here. (Don’t worry – we’ve read the answers there, too.)

For this post...:

  1. Conceptually, are we asking all the questions we need to measure success during the experiment? Are there more facets of site participation that we need to pay attention to as a part of this test?
  2. Given how the feature is going to look and how we believe that people will use it, are there reasons you would be skeptical of the results of the experiment?
  3. Supposing the feature works as designed, do you see any subtle or hidden downsides to implementing it on the platform?
  4. Suppose the feature is successful in its stated aims. What concerns would you have with its implementation?

And if you’ve got other questions, the answers are, of course, open to you as well. Finally, I hope it’s clear by now that this is a genuine request for community input, coming to you early enough to change plans based upon the feedback, and we have a history of modifying experiments or plans based on the outcome of consultations like this one. All that to say this: your voice is important, and we would love to hear what you think. Members of the product team and the community team will be actively monitoring this question for your ideas for the next two weeks. (After that I’ll still be reading, so no worries if you’ve gotta slip some critical feedback in late.)

14
  • 14
    concerning "How we’re expecting this test to look on the platform"'s #2 and #4, @TylerH pay up! :)
    – starball
    Commented May 10, 2023 at 20:09
  • 27
    Given the quality of many (possibly even most) titles, you could randomly sample words from the question and get a better title. Almost anything you folk do is going to be an improvement. Commented May 10, 2023 at 21:03
  • 9
    I really wish the communications and community-consultation process for the Content-Discovery initiative could be like what we're seeing here. I really like what's written in this post in terms of process and engaging the meta community.
    – starball
    Commented May 11, 2023 at 7:33
  • 1
    I think this are a lot of different changes at the same time. In order to gather enough statistics to differentiate meaningfully between each of the possible outcomes, the test should run for quite some time. Commented May 11, 2023 at 8:22
  • @Trilarion what number do you suggest other than their planned ~9000 questions? SO gets ~3.6k questions per day. Not sure how much percentage of that is through the Ask Wizard though.
    – starball
    Commented May 11, 2023 at 8:23
  • 1
    @user They are roughly testing 4 conditions with multiple metrics. In the end, when choosing the best conditions they should make a significance test and if the results aren't yet significant, continue the experiment. I have no idea how long that takes. My gut feeling says it should run for 4-6 weeks. Commented May 11, 2023 at 8:37
  • 3
    @Trilarion Without getting too much into the statistics, we do need to draw the line somewhere -- so it's a function of the magnitude of difference that we care to establish significance for. ~9000 data points per layout is our estimate for how much statistical power we actually need; and, two weeks is our estimate for how long that will take.
    – Slate StaffMod
    Commented May 11, 2023 at 21:35
  • 27
    "I know the introduction of these features is a sensitive topic" - that's why it doesn't say AI anywhere in this description, right? Commented May 12, 2023 at 5:14
  • 7
    @Karl Well... yes and no. If you're curious, it's because I wanted folks to focus on whether the tool does what it needs to do to help the site, and not whether it's made of steel or brass... So, mostly out of concern it would lead to an unbalanced discussion. If I wanted to hide the AI, I wouldn't have x-linked to a post where we're loud and proud about it...
    – Slate StaffMod
    Commented May 15, 2023 at 15:14
  • 4
    Will this also be available when editing other people's posts?
    – The_spider
    Commented May 16, 2023 at 7:02
  • 2
    @The_spider Not during the experiment. If the experiment demonstrates that the feature materially improves the measures we're looking to collect, it's on our list to consider.
    – Slate StaffMod
    Commented May 16, 2023 at 18:58
  • 3
    "We'll move the title...": and now we have another symptom, where users type the title as the first line of their question -- admittedly less disturbing. example.
    – trincot
    Commented May 19, 2023 at 9:45
  • 4
    How much does SO pay to OpenAI for this and how many additional ads will have to be added to every page to cover it? (I am well aware that almost nobody runs their own AI models in 2023 and everyone pays OpenAI to do it for them) Commented May 19, 2023 at 13:18
  • Just a suggestion, titles are summaries for question. If they are auto-generated already, then one could as well have titles for answers (i.e. summaries of answers) auto-generated (but overridable by the answerer). That might help seeing what might be in an answer. Commented May 22, 2023 at 6:26

16 Answers 16

39

Certainly an interesting test to run, and it fits well for the wizard which is where users who already need help end up.

A large issue I have encountered with prompt-created titles is that the generation misses key caveats with technical jargon. This is going to be a problem, perhaps one of nuance more than substance. For example, the generator does not understand the difference between "faster processing" versus "a faster processor". These cases of jargon will be interesting to see play out.

I am curious what the prompt will be, behind the scenes, for generating the titles. Will it request for certain styles, or just a summation? Keep in mind internet titles are different from chapter titles, etc.

Since we are here, I can't help but ask for a quick analysis if possible. As work is being done with title generation, and as there is an entire set of user-verified duplicates on network, would it be possible to run a prompt for generating titles in a hidden field, and seeing if there is correlation between similar or identical titles in existing duplicate closure?

8
  • 3
    "I am curious what the prompt will be" - do we know that they're using an LLM behind the scenes? And if they are... I wonder how much this'll cost to maintain.
    – starball
    Commented May 11, 2023 at 3:18
  • 5
    @user - Yes, they are using an LLM. There was an MSE post on this already
    – Travis J
    Commented May 11, 2023 at 5:49
  • oh sorry. for some reason I didn't pick up on the LLM part. Are LLM and "Generative AI" synonymous? (I don't know much of anything technical about this field)
    – starball
    Commented May 11, 2023 at 6:02
  • 4
    LLMs are a type of Generative AI : a class of methods trained in a certain way with some characteristics, destined to generate text. Generative AI just means AI which generates stuff (eg pictures such as midjourney)
    – Soltius
    Commented May 11, 2023 at 14:40
  • 9
    As for the issue with prompt-created titles -- I think we'll have to wait and see where the pain points are here. We hope it's an edge case, but no way to know until we test. As for your quick analysis request -- it's a good thought, and one I've heard a bit of discussion about internally. Not so quick, though, and probably of more relevance to future projects. I'll see if we can find time to consider it, but probably no full writeup. Thanks for the suggestions.
    – Slate StaffMod
    Commented May 11, 2023 at 15:25
  • 1
    @Soltius: I am not any type of AI expert, but according to the plan meaning of the terms, there's a lot of overlap between generative AI and LLM, but no superset-subset relationship. For example, calculating a distance metric between two inputs has to benefit from a detailed model of the input language, and I would say the term LLM applies. But it isn't generative. ("Calculating a distance metric" has real applications on StackExchange such as finding duplicate questions, and in the larger world such as plagiarism detection and patent searches)
    – Ben Voigt
    Commented May 16, 2023 at 19:12
  • @BenVoigt Good point, but even non-generative language models are pre-trained on generative tasks (basically predicting the next work in a sentence), giving them the "detailed model of the input language" you mention. They can then be adapted to other tasks, which can be non-generative (such as your example). So I'd say that LLMs are a subset of generative AI. But it's just fussing over nomenclature anyway.
    – Soltius
    Commented May 18, 2023 at 17:43
  • @Soltius: Both are going to be very concerned with conditional probability of a certain "next word" given previous text and topic (prompt). But the generative task to output the next word which maximizes the conditional probability and the analysis task of returning the conditional probability given an arbitrary "next word" input are very different, and don't even require the model to encode the same information.
    – Ben Voigt
    Commented May 18, 2023 at 19:17
28

Conceptually, are we asking all the questions we need to measure success during the experiment? Are there more facets of site participation that we need to pay attention to as a part of this test?

An interesting metric would be whether or not the title has any impact on the question's path to an answer. Does it on average get less comments requesting clarification? Are the answers it receives considered more useful on average?

Given how the feature is going to look and how we believe that people will use it, are there reasons you would be skeptical of the results of the experiment? Supposing the feature works as designed, do you see any subtle or hidden downsides to implementing it on the platform?

I would consider the results of "Are titles written with assistance edited more or less often by community members?" to be mostly useless. While a reduction in suggested edits would be nice, obviously grammatically correct titles are going to be edited less. If that turns out to not be the case, there's probably major problems with this system.

Suppose the feature is successful in its stated aims. What concerns would you have with its implementation?

My primary concern with this feature as a whole is we're expecting users who can't solve their own problem to decide on a pre-generated title that rather than describing their problem, is just putting seemingly relevant words together. While, sure, users often create terrible titles, at least when they're created by users we can sometimes understand why they used the title they did and potentially gain context from it.

Is it actually improving something, or is it just creating a new set of problems that we aren't measuring for.

14
  • 4
    I'm not sure how to incorporate this into my answer yet as I'm still formulating the idea, but I'm worried that... if potentially all titles are generated using this system... and all titles are suddenly "interesting" titles... do we lose the ability to differentiate between interesting questions and non interesting questions at a glance?
    – Kevin B
    Commented May 10, 2023 at 21:24
  • Doesn't seem so silly now, does it?
    – Makoto
    Commented May 10, 2023 at 21:24
  • 4
    @Makoto eh, i think even my answer on that question agrees with you, conceptually, I just disagree with your presentation of it.
    – Kevin B
    Commented May 10, 2023 at 21:25
  • @KevinB Judging questions at a glance is exactly the problem. Commented May 10, 2023 at 23:15
  • 2
    @KevinKrumwiede in a magic ideal world where people put in work to narrow down their question and write good titles, I think we should be able to get a good general understanding of the question body from just seeing the title. see also my answer post.
    – starball
    Commented May 11, 2023 at 2:25
  • @KevinKrumwiede I don't see why. "Don't judge a book by its cover", as an idiom, dates back to the time when books were sold without covers; and nobody ever said "don't judge a book by its title".
    – wizzwizz4
    Commented May 11, 2023 at 3:26
  • 8
    "An interesting metric would be whether or not the title has any impact on the question's path to an answer" - that is definitely on our list of things that we will be watching
    – Yaakov Ellis StaffMod
    Commented May 11, 2023 at 7:20
  • 9
    "I would consider the results of "Are titles written with assistance edited more or less often by community members?" to be mostly useless" - the reason why I care about this is that significantly fewer title edits is a quality of life improvement for curators/reviewers.
    – Yaakov Ellis StaffMod
    Commented May 11, 2023 at 7:21
  • 1
    i mean, in terms of reducing edits? sure... though i'd be more interested in whether or not it improves searchability. LIke i said in the answer, reducing edits would certainly be a nice positive, but not necessarily one that I'd want with some of the potential downsides.
    – Kevin B
    Commented May 11, 2023 at 14:55
  • 4
    As for your last point, it's a good thought for us to keep in mind as we start to think about on-site trades. What you're describing will certainly happen sometimes, but how often? I think it will be very hard to predict accurately. And when it does happen, how bad actually is it for the site? This will be even harder to quantify in a trustworthy way, and therefore tricky to make trades about. I will give it some thought, because I think it's important for us to be at least aware of the risk.
    – Slate StaffMod
    Commented May 11, 2023 at 15:20
  • 1
    "While a reduction in suggested edits would be nice, obviously grammatically correct titles are going to be edited less." If the only thing the title assistant does is reduce the amount of work SO reviewers spend making sure a title is grammatically correct, isn't that still a reduction in scut work?
    – Nick ODell
    Commented May 16, 2023 at 19:17
  • 1
    @NickODell Surely, though i just assign that a lower value than the quality of Q&A outcomes. Clearly we have no shortage of editors, given the queue stays full.
    – Kevin B
    Commented May 16, 2023 at 19:33
  • @KevinB But we do have a shortage of edit reviewers, given the queue stays full. If we can reduce the volume of grammar edits, it potentially frees up the edit queue for more valuable edits. Commented May 24, 2023 at 2:08
  • 1
    i mean... there's better solutions to the edit queue than an AI, such as changing the metrics in which users gain the ability to directly edit (which is in the works)
    – Kevin B
    Commented May 24, 2023 at 15:10
24

As someone who has been a member of the Stack Overflow community for nearly a decade and a half, this "feature" fills me with despair, because it's going to deprive me of one of the few ways that remain to distinguish between good and bad questions: namely, by looking at their title1.

This "feature" isn't magically going to make those bad questions good, it's just going to obfuscate the fact that they're bad. That's not an improvement.

Yes, there are some deserving questions that will get more attention as a result of this, but they are going to be vastly outnumbered by the turds that are getting polished AI-generated titles. In short, all you're accomplishing is further obscuration of the quality problem that exists on Stack Overflow.

That plus the "AI" label may help sell the site to investors, but it's not going to fix that underlying, massive quality issue that Stack Exchange Inc. seems to think will just go away if they ignore it long enough. What's especially sad is that LLMs could quite conceivably be a massive help in this regard, but SE Inc.'s continual refusal to allocate development resources to curation will never allow this.

1 To anyone who wants to jump in with the tired old "don't judge a book by its cover", I suggest you find someone who cares about your lazy non-argument.

8
  • 3
  • I think there's a large class of questions that you'll still be able to tell something about. See my answer post- particularly the part with the geode/gem analogy.
    – starball
    Commented May 12, 2023 at 20:38
  • 3
    It won't be nearly as bad as you say. The AI will only be able to write superficially good-sounding titles. They won't actually be good titles, probably won't even relate to the question, and the people who will use the AI to generate their titles won't know the difference, so they'll just accept whatever is offered. This means you & I (and everyone else who uses titles and other metadata as a proxy to judge the quality of a question and thus gauge our interest in it) will still be able to do so unaffected. Now, if the follow-up question is, "Then why do it?", I don't have an answer for that
    – Cody Gray Mod
    Commented May 13, 2023 at 5:50
  • 3
    I wonder what the suggested titles for "Here is my code so far, It's not doing what I want" type of questions will be? A correct title would probably be "I have a problem somewhere". Commented May 13, 2023 at 9:12
  • 1
    The community of researchers training and open-sourcing language models that will likely be used here by SO seems to disagree, selecting the site for their training sets, together with e.g. Project Gutenberg, Wikipedia and (non-copylefted) Github. All it takes is to filter the site data using quality metrics such as upvotes, discarding the rest (which will still be a valid strategy, ignoring the new "AI-polished turds":).
    – mirekphd
    Commented May 13, 2023 at 14:42
  • 2
    @mirekphd That will result in better titles for worse questions, which will only exacerbate the problem.
    – Ian Kemp
    Commented May 15, 2023 at 7:44
  • IanKemp - thanks for putting in this answer, which is the best answer here.
    – Fattie
    Commented May 21, 2023 at 16:12
  • 1
    The process of writing a question (and its title) by itself is an exercise in troubleshooting your problem. If you are able to write it down and clearly explain what it is, you're 90% of the way toward a solution. Now here comes along Mr. AI telling you that what you're writing is unclear and then tries to "help" by twisting your words. All that work you've done trying to express your problem is erased. To a non-native english speaker, the problem is worse because they may be unable to tell the difference between the AI-suggested text and their own. I can't see how this helps anyone.
    – Seth
    Commented May 26, 2023 at 23:11
21

Are titles written with assistance edited more or less often by community members?

I'm glad this is going to be looked at.

But be careful about how much you try to infer from title edits happening when evaluating "success" (know the limitations)

  • No community-made title edit does not necessarily mean the title is as good as it gets (it can just mean nobody cared enough to improve it (yet)).

  • A community-made title edit

    • Does not mean the question asker could have known how to do better given their knowledge and best effort.

      There's always a knowledge gap when you need to ask a question. And that knowledge gap can lead to overly specific / generalizable (to no fault of the asker's) or overly general questions (and their titles).

    • Does not mean whatever technology you're using to suggest titles could have done better.

      I think this follows from the above point. Assuming you generate titles primarily based on the post body, an overly general post body (needing details/clarity) will usually correlate with an overly general title, and so with a generalizable post body. Unless you plan on solving that problem with this same technology (how?), "garbage in, garbage out".

      There is one scenario (that is actually somewhat common) where I'd be happy to see if this is helpful: Questions where the title is too-general / ambiguous, but the body contains information that can be used to disambiguate the title.

Just to make this a little more concrete, here's are some examples:

  • Question asks about a problem with using library X, Y, and Z and TypeScript. Turns out the problem doesn't have anything to do with specifics of library X, Y, or Z, and is really just to do with TypeScript and perhaps dependency-related things with TypeScript in general. (Question can be further generalized -> edit to generalize)

  • Question title asks about a problem with using CMake. Question body elaborates on the problem being related to an attempt to do something with a specific library dependency. Problem turns out to have to do with specifics of that library. (Question title is ambiguous -> disambiguate title)

In both the above cases, it generally takes enough subject-matter-expertise to fill that knowledge gap that motivated the question to know how to disambiguate or generalize the question and its title, which I'm going to guess what you're building is not designed to do?

And why does generality/ambiguity matter?: Because ambiguous titles are annoying to future readers looking through search results. "Oh this looks related. Wait- after reading the full Q&A, it's really not. Darn (wasted time)". And questions with room for generalization will show up for fewer readers looking through search results (they'll never find something that could have helped them).

This is also why I made a related suggestion in my MSE answer- to incorporate community-made title edits into the model's training. Take information about community-made title edits as an opportunity to train something that could possibly have some usefulness with respect to generality problems.

Going deeper into my thoughts here but kind of sidetracking now: My general preference would be that when a question asker doesn't know whether their question/problem can generalize in a certain way, they try to err on the side of being too specific and giving too much contextual info. It's easier for the community of subject-matter-experts to generalize questions (remove unnecessary information) than go through the process of soliciting missing information.

Okay, maybe I should qualify the above statement a bit better. Providing too much context is something I want askers to do after trying to narrow down the cause of their problem as much as possible. And that's a related problem here. I avoid popular programming language tags because there's such a high concentration of highly-application-specific my-code-isn't-working questions that are like geodes: Inside all the application-specific context is probably hiding a generalized question that is more applicable (and therefore valuable) to the masses. We want the gems inside the geodes (unless we already found them before (duplicates)), and in this analogy, the "gems" are actually quite easy to title in a way that is clear and not ambiguous, whereas that's difficult to do with the "geodes" (application-specific questions lacking effort to narrow down the problem). To make this more concrete, I'm talking about questions like "why isn't my sudoku program working? <code dump>". How do you title that in a way that isn't ambiguous (name-clashing with all other peoples' problems that have to do with broken sudoku programs?) TL;DR the limitations and "garbage in, garbage out" problems go even deeper.

And that just brings me back to something I've been thinking for a while and over and over: I think we should do more to promote reading the Help Center pages- the same pages that show good and bad examples of titles and give guidance on how to write good titles, and the same pages that say "Eliminate any issues that aren't relevant to the problem.", and "The more code there is to go through, the less likely people can find your problem. Streamline your example in one of two ways: Restart from scratch. [...], Divide and conquer. [...]." (tag wikis can be useful too, when they've been written with tips for asking questions).

There's also related discussion on this in Are there legitimate "fix my code" questions?.

11
  • 3
    Not only "promote reading the Help Center pages" but also the Tag Wiki of the "main" Tag for their Question... This is where the most info is located that will be specific to asking a good Question in that Tag.
    – chivracq
    Commented May 11, 2023 at 2:09
  • @chivracq very good point! (but mileage may vary. not all tag wikis give such guidance).
    – starball
    Commented May 11, 2023 at 2:10
  • 7
    "Questions where the title is too-general / ambiguous, but the body contains information that can be used to disambiguate the title." - in testing, this is the case where the suggested titles were really obviously a big win. And I can tell you that I did not have to try hard to find examples of this on the site. More relevant title instead of ambiguous for decent question can definitely raise the chances of the question being viewed and answered.
    – Yaakov Ellis StaffMod
    Commented May 11, 2023 at 7:24
  • 1
    @YaakovEllis "And I can tell you that I did not have to try hard to find examples of this on the site." But it's also not hard to find examples of questions that are unclear as posted. Maybe consider having an output option along the lines of "Sorry, it's likely not possible to construct a better title, please go back to your question and improve it before.". Just to minimize the number of cases, where the title doesn't get better. Commented May 11, 2023 at 8:26
  • @Trilarion "better" than what? and would a generative AI or whatever they're using even be able to judge the quality of titles and their fitness for the question body? Not that I don't wish what you're suggesting could be done. I'm just wonder if it really can.
    – starball
    Commented May 11, 2023 at 8:28
  • 1
    @user They will have a training set soon. Question which improved (tm) titles that nevertheless got closed afterwards. There improving the title didn't help. Or they take the existing dataset of closed as unclear questions and train it to classify that it will be closed. Only suggest improved titles if the other model says that this question likely won't be closed. Commented May 11, 2023 at 8:34
  • @Trilarion I see writing representative titles as being orthogonal to writing on-topic questions. My understanding here is that the goal is generating representative titles- not detecting topicality. So I don't understand why closure (failure in on-topicality) should be taken as part of the training feedback here. (unless they wanted to make something that does both)
    – starball
    Commented May 11, 2023 at 8:38
  • 1
    @user My thinking was that without sufficient information in the question it's simply impossible to write a representative title ("Something about programming"). At least the unclear close reason would be an indicator of no good title possible. Commented May 11, 2023 at 8:40
  • 1
    @Trilarion ah good point. Then it should be close reasons that don't have to do with on-topicality and specifically with problems that can interfere with title generation, such as "needs details / clarity".
    – starball
    Commented May 11, 2023 at 8:42
  • "(Question can be further generalized -> edit to generalize)" - are we actually officially supposed to be doing that? Commented May 14, 2023 at 22:08
  • @KarlKnechtel I've done that a couple times where possible and when I'm the only one who's answered the question and the generalized question hasn't been asked (to my knowledge)
    – starball
    Commented May 14, 2023 at 22:21
18

I think there are valid criticisms of the "how many edits" metric for judging the success of this intervention, but other answerers have made those already. I'll focus instead on whether this intervention is likely to add value to Stack Overflow.

Even if successful, this intervention will bring the most improvement to titles on questions where the author wouldn't have written a decent title by themselves. But the knowledge of what information is relevant and important to include in the question title, overlaps significantly with the knowledge of what information is relevant and important to include in the question itself. So even a good result for this experiment will mostly just mean better titles on low-quality questions.

That is, the questions where this intervention makes the most difference to the quality of the title are mostly the questions that will be closed and provide no lasting value to Stack Overflow. Therefore I don't see question titles as a high priority for staff to focus their efforts on.

A better use for generative AI might be to predict what experienced users are likely to ask for in comments, in order to clarify the question. Stock comments like "please include the full error message including the traceback", or "please include your code as formatted text, not an image", could plausibly be made by an AI model during question creation, with the asker required to either address the raised issues or mark them as spurious, before the question can be posted.

5
  • 2
    what you're saying in your second paragraph is very much what I was trying to say in my post in the "Does not mean whatever technology you're using to suggest titles could have done better." bullet point. Your last paragraph triggers my knee-jerk reaction: "let's please promote reading of our Help Center pages!"
    – starball
    Commented May 11, 2023 at 1:42
  • 10
    "A better use for generative AI might be to predict what experienced users are likely to ask for in comments, in order to clarify the question" - I cant get into any more details other than to say that this is current experiment is the first one we are doing, not the last.
    – Yaakov Ellis StaffMod
    Commented May 11, 2023 at 7:29
  • 7
    while it could be neat to have a chatbot probing the asker to improve their question, I also wonder- do we really need something so fancy for that? Why not just a general checklist + tag-specific checklists in the Ask Question UI? Take a look at the issue ticket templates for VS Code and for TypeScript (just to name two examples) and you'll see what I mean. It's nothing complicated, and it's somewhat baffling to me that we haven't picked up on it.
    – starball
    Commented May 11, 2023 at 7:45
  • 1
    "A better use for generative AI..." One could do both. And if better titles without better questions do not improve how a question fares, that will hopefully be evaluated as part of this experiment. Commented May 11, 2023 at 8:29
  • @Trilarion Yes, it's just about priorities. I'm glad to see that something like this is already in the works.
    – kaya3
    Commented May 11, 2023 at 16:07
18

I know it's not the primary topic of this discussion, but I just wanted to point out what to me looks like a significant UX issue with most of these proposals.

On the last three of your example screenshot sets, the box to enter the title is on the second screen (the "review" screen). In essentially every form submittal UI I've ever seen, a "review" screen is where you look over the data you entered and verify it before submitting. The idea that the user should input new data on a "review" screen is unexpected. Occasionally a user might be able to amend an existing entry on this screen instead of having to go back. You're going a step beyond that, though, and making the "review" screen the first and only place where this data can be entered. Once a user gets to a "review" screen, they're not looking for new fields that they have to fill in. Your description at the top of the page even says "do a final review and then post", so the user's already in the mindset of "oh, I just have to give this a once-over and then hit the button". If I was a first time user, I would find this arrangement very confusing and would be unlikely to operate it correctly on my first few attempts.

I also noticed that some of your screenshots appear to show the "Post your Question" button as active while the "Title" field is still blank. This might just be a photoshop error, but please double-check that a question cannot be submitted without the user either entering or selecting a title. That also means don't make one of the auto-generated titles the default, force the user to consciously and explicitly select one. Otherwise, since the user isn't expecting to see brand new fields on a "review" screen, they're likely to just click "Submit" without even noticing the title field is there.

As far as AI-generated titles themselves go, I suggest also keeping track of whether the generated titles (selected or not) happen to match the titles of existing questions. I can foresee a future issue where lots of questions end up with the exact same title even though the question is different, and then that throws off the engines that generate the "related questions" and "possible duplicates" lists. As an extension, it might also be worth tracking how often the AI generates the same title (whether selected or not) for two different input questions. If the same title gets suggested for two questions that end up being marked as duplicates of each other (or of the same third question) then that's probably fine. If, however, we're seeing the same suggested titles for multiple distinct questions, then that's a big hint that the generated titles may not be accurately representing the question.

4
  • yeah I was also wondering about "name collisions". if there is a name collision though and the title suggestions are any good... that's a duplicate smell isn't it?
    – starball
    Commented May 11, 2023 at 20:20
  • 1
    @user It could be, or it could mean the titles are vague and generic. You could have a million non-duplicate questions named "how does jquery work" or "how do I do this in C++" that are technically accurate but aren't actually meaningful.
    – bta
    Commented May 11, 2023 at 20:29
  • 7
    Duplicate titles -- now there's a dang good thought. Similar titles as well. I'll mention this to devs and see what's feasible. At minimum, afaik the system shouldn't allow submission of duplicate titles, so we shouldn't see them posted. It's also a good observation that 'review' containing fresh input is potentially a bit odd, and I'll pass that along (probably as something to return to after the experiment layout). FYI the draft images are not necessarily exactly how it'll look in prod -- we've still got some implementation and polishing to do, so no worries there. Thanks for the thoughts.
    – Slate StaffMod
    Commented May 11, 2023 at 21:12
  • 1
    @Slate I don't want to draw too much attention, just wanted to say that I mentioned the "problem/opportunity" with duplicate titles already in an answer to the previous post on meta.stackexchange. Hopefully already now in this experiment you are not allowing duplicate titles because that would probably not be helpful. Either the AI should not suggest anything if it would suggest a title of a question that is already existing or it should note that and link to that question as potential duplicate question. Commented May 12, 2023 at 8:47
13

tl;dr: I fear that generating "good" titles for poor questions will cause less moderation on said questions.

I like discovering questions from the frontpage. The main or almost sole trigger for interaction is the title. I'm probably not alone in this workflow.

I wonder whether I'll skip questions with "better" (generated) titles. Poor quality titles entice me into clicking them, because their poster probably needs help editing or clarifying (and frankly, often a close-vote). A perfect AI-generated title, which may or may not cover the question being asked therein, may make me skip it for looking either good enough or not within my area of expertise.

I hope your "perform better" and "comments and reviews" metrics account for that. I think that a poor question with a good-looking title is more prone to being ghosted.

6
  • 3
    This should be measured. Number of visits maybe. I could imagine that others have given up on bad titles and only click on good titles. But I fully agree that the title is more or less the only interaction for many potential visitors (even through search engines). One needs to be careful to not only optimize for click through rates though. Commented May 11, 2023 at 8:32
  • 1
    Thanks. I'm more afraid of a "bounce", i.e. "wow, what an interesting title!", followed by a "nope" on the back button because of the contents.
    – CodeCaster
    Commented May 11, 2023 at 8:35
  • 12
    Not sure why this is being downvoted, the quality of a title is a great indicator of the question overall, and artificially generating high quality titles for low quality posts may well reduce the chances of clean-up-inclined users spotting it.
    – DBS
    Commented May 11, 2023 at 8:40
  • 2
    @DBS Maybe people think that there aren't that many cleanup inclined users or that it's a necessary price to pay for this kind of auto cleanup of titles. You gain something, you lose something. A better title doesn't automatically make better content. But I think it's a step in the right direction. Maybe other indicators of potentially bad questions can be found instead (first question, ...). I upvoted this answer but I still think this experiment is good. Commented May 11, 2023 at 10:51
  • "I'm more afraid of a "bounce"" We already have problems like that (which I touch on in my answer post)- due to titles not being good at representing the question (which is their main job) and people just not being good at writing representative titles. And who says a low quality question can be easily given a great title? (I also touch on this in my answer post. In fact, I think it's harder to give bad questions good titles, and that good titles come naturally for good questions)
    – starball
    Commented May 15, 2023 at 4:24
9

Will you save the AI-generated titles with the question so an editor (and/or you) can pick the "better" title after the fact or point out why they weren't a good fit?

4
  • 4
    That isn't something that we are going to be doing yet. Would require a whole separate UI workflow to be built (to highlight the titles after the fact), an infrastructure for how to use the information, and something akin to a review queue for helping users find these. That said, we are considering other ways that the community will be able to get involved to help give more proactive feedback on the quality of responses (which will be discussed when those are more concrete).
    – Yaakov Ellis StaffMod
    Commented May 11, 2023 at 7:36
  • 2
    Alright, but will it be logged somewhere, so you can review what's being suggested, and will SMEs look at the choices and chosen one to determine whether the user understood what they were suggested and picked the most applicable one? I think it would be interesting for you to see, not necessarily for the end user.
    – CodeCaster
    Commented May 11, 2023 at 7:39
  • 6
    That is on a list of potential things to do. Wont be done for this first experiment, though we will be taking the thumbs up/down voting into account when testing variations on models and LLM prompts.
    – Yaakov Ellis StaffMod
    Commented May 11, 2023 at 7:41
  • Related: the edit question dialog should get this suggestion feature too after the experiment concluded, so editors can profit too. Commented May 11, 2023 at 10:54
8

I would like to see this feature implemented using also the content of the question.

The LLM AI could:

  • read the original title and content
  • criticize the content in order to make the OP clarify it, add relevant missing elements (OS, version of the tool/framework used, minimal reproducible example if pertinent, ...)
  • and then reformulate the title based on that enhanced content.
6
  • 1
    see my comment here for my thoughts on this.
    – starball
    Commented May 11, 2023 at 19:09
  • 2
    @user A good old template? Yes, I miss that too.
    – VonC
    Commented May 11, 2023 at 19:16
  • 4
    Issue templates are about as useful as pinned FAQ topics atop forum boards. Inexperienced users don't read them. Just like tag wikis, how-to-ask pages and debugging manuals.
    – CodeCaster
    Commented May 11, 2023 at 22:35
  • 1
    Related musing: I've been playing around with my own version of this (based on generally available LLM chatbots and my own prompts), and I've been completely ignoring the asker's title. I've only been using the content of the question, and that seems to work well. Then again, I've been mainly testing it on questions with really bad titles that contain little to no useful information. It might be worth testing whether taking the original asker's title into account is useful; I suspect it might, at least for the cases where the title contains essential information that's not in the question.
    – Ryan M Mod
    Commented May 12, 2023 at 22:38
  • @RyanM I agree. But in the context of this new feature, I would like the AI to help the user refining their question content, before the same AI suggests a good title.
    – VonC
    Commented May 12, 2023 at 22:50
  • Yeah, that'd be great, although probably not necessary for a first iteration. I'm not sure my musing actually had a concrete point (it was sort of stream-of-consciousness related thoughts), beyond perhaps the fact that useful titles can be extracted even from the verbatim, unimproved posts of people who can't write good titles.
    – Ryan M Mod
    Commented May 12, 2023 at 23:41
7

Just a few questions:

  1. Will we do away with the list of disallowed words in the titles - at least for this test?

  2. Will we get an explicit reason why a title is not allowed, and will that reason always apply to the context of the question?

  3. If a title is not initially allowed, can we override that with a reason such as making a question easier to find using the most likely search words?

  4. If this is strictly for questions in the Ask Wizard, will it be limited to new questions by new users, or just anybody going through the Wizard route?

  5. Has there been any discussion of letting us use a tool like this on older questions that have languished with seemingly immutable title issues for years, but that we haven't been able to edit because of obscure disallowed-word reasons that are usually impossible (or too tiring) to track down?

0
5

* Converting my comment into a post.

It looks like it's possible for the question author to vote on the quality of suggestions. How will this data be used? Will it be used to make the model "better"? Could other users vote on whether the suggestions are better than what OP was going to enter? Because at least for the example given, the suggestions don't improve the title a lot in my opinion.

I guess what I'm trying to say is, this experiment could also collect data to improve the currently trained model. The plans for the experiment seem to imply that the model is fixed and the experiment is about seeing how users respond to it.

5

Warning: this site is widely scraped 1 and its text - including the titles - may be used to train the very models (the core - generic - part of them) that this Title Generator will be using in its future iterations, thus creating a delayed feedback loop... with unpredictable consequences for these models (such as overfitting).

Do it only if you are prepared to train the core models from scratch and do not really care if other people's models will be working correctly in the future, OR (and I would not recommend this motive for humanity's sake) as a guaranteed method of removing your site data from LLMs training datasets such as The Pile and others (scientists training these models will most likely notice the problems, especially overfitting, like they already did with the Twitter data). Even if core LLMs authors selectively remove just the SO question titles alone (noticing they adversely affect their test-set accuracy), this would harm the future accuracy of the Title Generator / Assistant, which will be heavily relying on this data subset...

One workaround would be to tag the AI-generated content with an appropriate tag to facilitate its exclusion from core models training sets.


[1] e.g. as part of The Pile dataset and surely also most of the proprietary ones.

A quick inspection of this dataset shows that SO question titles are indeed concatenated with the question body, so AI-generated content added "just" to some of the titles will currently pollute all SO Q&A data (unless it is scraped more intelligently or not at all) due to the inclusion of some "AI-assisted" (i.e. model-inferred) SO questions into The Pile, thus affecting multiple open-source LLMs that depend on this dataset (such as GPT-J or Dolly 2.0).

Here's a sample SO data record excerpted from The Pile:

"Q: TextView Not centered in app but centered in match_constraint I've created a simple activity design using ConstraintLayout. Whenever I try to center a textView, it does it correctly in the blueprints but never does it in the actual app. Not sure if i am doing something wrong or I'm losing my mind. Here is the image [..] A: You need to add: android:gravity="center" to the TextView. This is the only certain way to center the text inside a TextView object or one of its subclasses. The android:textAlignment is not working in all the cases and as reported by this answer that it has problems in lower API levels. " [source: HuggingFace]

which was scraped from this question (with a title bound to be "fixed" by the Title Assistant): TextView Not centered in app but centered in match_constraint

6
  • 5
    This doesn’t really seem to be SO's problem, does it? All public content is likely "polluted" by AI already (heck, this comment is largely autocompleted by my phone), why should SO be the one to take the bullet of doing things the hard way? Commented May 13, 2023 at 8:52
  • The "polluted" sites are not included in the training sets... the Twitter for example (remember Elon Musk's estimate of the bot-generated content there?:). But if the site data is removed from the training sets, it will harm the accuracy of the Tittle Generator Assistant as well... The Big Boys can watermark their core models output and exclude it very selectively, keeping the human-generated content. But not the output of SO augmentations / fine tuning of their core models... this has to be probably completely excluded.
    – mirekphd
    Commented May 13, 2023 at 8:59
  • 3
    Frankly, I would not expect anything but SO to be the training set in the first place. And for this they know what is generated by themselves. That SO content will be partially AI generated is then someone else's problem. Commented May 13, 2023 at 9:13
  • 2
    The Pile: "The Pile is a large, diverse, open source language modelling data set that consists of many smaller datasets combined together. ". It includes GitHub, Stack Exchange, English Wikipedia, and Ubuntu IRC (presumably both Libera Chat and the disgraced Freenode). Approximately 800 GiB. Commented May 13, 2023 at 14:55
  • 4
    While one can argue about whether deep neural networks trained on copyrighted data produce derivative output, it should be much clearer that aggregations of copyrighted input are required to attribute. What I see when I click the "source: HuggingFace" link in this answer is a clear license violation.
    – Ben Voigt
    Commented May 18, 2023 at 19:41
  • @MisterMiyagi: it can soon become SO's problem... serving unlabeled AI-generated content to the European market can soon get you fined, GDPR-style (search for instance for "european commission wants clear labelling of ai generated content")
    – mirekphd
    Commented Jun 9, 2023 at 14:21
5

There is good and bad question body content (good = useful, important, clear, well-researched, ... ) and there are good and bad question titles. This feature would improve titles based on the question body content, so it will most help those that have good content but didn't manage to create a good title too. One should therefore measure the quality of the title and see if it has improved. I think your evaluation metrics like (number of edits, how often do people return, ...) try to indirectly measure the quality of the title improvement. I wonder if it would be feasible to directly rate the quality of titles instead? (Like: Is this a good title for the question? Yes/No, if no what would be a better one.) It might be too much effort.

And you do not need to give a suggestion for the title every time. If you think the question body content isn't yet clear enough to enable a good title suggestion, don't suggest one (and don't count towards the test results). Depending on how often that occurs, it might improve the efficiency of the suggestion feature a lot.

Also, question body content and question titles evolve after creation. After comments questions can become clearer and only then a really good title can be found. I think that the AI suggestions for good titles should definitely also be part of the edit question dialog, not only at the beginning of the question asking process. Maybe it would even have been better to test this feature directly on the edit dialog instead the question asking wizard because editors might have a better knowledge to judge if suggested titles are appropriate and the question body content might be more mature by then.

Finally, the capacity for duplicate detection should further get investigated. Similar proposed titles are a strong hint for duplicate questions (or for titles not specific enough). Questions with similar proposed titles should be shown early. It's probably already part of the "related questions" section, I guess.

3
  • 1
    "There is good and bad question body content": that is why I suggested for the AI to assist on the content too.
    – VonC
    Commented May 12, 2023 at 13:47
  • 1
    Also related in my answer: "A community-made title edit does not mean whatever technology you're using to suggest titles could have done better.". Also the paragraph which says "err on the side of being too specific and giving too much contextual info"
    – starball
    Commented May 12, 2023 at 19:54
  • I also see good question content being downvoted for their bad titles. Metrics should also look at that.
    – ccprog
    Commented May 18, 2023 at 18:46
3

It will be interesting to see the results of this experiment. If it reduces the rate at which users ask the whole of their question in the title (only using the body of the question for supporting material if at all) that would be nice.

4
  • 1
    Why would a reduction of that be a good thing?
    – starball
    Commented May 12, 2023 at 19:57
  • 3
    @user a reduction of "See the title." questions would be a good thing, but also, there's far easier ways to solve that than AI.
    – Kevin B
    Commented May 12, 2023 at 20:00
  • 1
    I'm assuming the title suggestions are generated from the post body and maybe the tags. What would an AI possibly suggest for a body that says "see the title"? See also meta.stackexchange.com/a/119476/997587. TL;DR rephrasing is useful for the rare questions that warrant little-to-no elaboration.
    – starball
    Commented May 12, 2023 at 20:45
  • I view it as "AI as copyeditor". Commented May 15, 2023 at 11:55
2

Moving the title draft to the review step seems like a bad idea. Note that's not part of the prior Meta Stack Exchange proposal.

In particular, I find it very useful in the current interface to write a summary title, and then check the resulting list of possible related or duplicate questions before proceeding to write the full body of the question. If that goes away, then it would be a significant time-efficiency loss, not a net gain.

3
  • 1
    related: bta's answer. while I can see your point, the first step before even trying to draft a question should be to search. That's what it says in How to Ask, and what it says in the interstitial page, and in How much research effort is expected of Stack Overflow users?
    – starball
    Commented May 13, 2023 at 19:50
  • 1
    @user: And yet we have the current interface that does a search based on the initial title, which I presume was found to have utility for the site and other users. Commented May 13, 2023 at 20:38
  • 2
    I don't like the multi step approach much because it adds overhead. It would be better if all could be done in a single form and the position of the title field (above, below, wherever) could be chosen to one's liking. Commented May 14, 2023 at 11:39
0

Taking a bit of time to have my head on straight really was the right call here, so I took a bit of time to read through this.

It sounds like you're moving boxes around. If that's the case, a valid A/B test would be to see that questions which are written without the AI-generated text do better than ones that are.

Do note that the concern I had brought up before is still relevant; this isn't going to improve the situation where someone who doesn't really have the practice of writing titles - let alone coherent, good titles - just gets handed a string that feels coherent and runs with it.

In either event, this experiment also does bring up a really thorny edge case. Technically, I haven't submitted my content to Stack Overflow, and yet the content will (presumably) be scanned and used to prop up an LLM that then tells me what my title should be. How does this square with the licensing terms we've set beforehand, where one can't reasonably expect that the terms are agreed to before the content is posted?

8
  • 9
    For the license issue: I'm not sure this actually counts as "using your content" as long as the content and the generated title are not stored server-side. For me this is similar to generating a rendered preview of the question, proposing related questions, checking for content issues that prevent a question from posting etc; all of those are processing of the data before submitting the question.
    – Marijn
    Commented May 10, 2023 at 20:11
  • @Marijn: It does. As in, there's a very clear boundary between me putting something into the question box and hitting submit. This blurs that substantially. If I don't consent to or agree to the publication terms but am OK with everything else (which, btw, you can do; you don't have to submit things to the site but you are otherwise bound by other usage terms), then Stack Overflow is, without express consent, taking my content and licensing it.
    – Makoto
    Commented May 10, 2023 at 21:11
  • 8
    Stack Overflow already does server-side stuff before you hit "submit". For example, it tries to find duplicate questions based on your title, which obviously requires sending your title to the servers to do a search of some sort. AFAIK, it's been like that for many years. I don't see how passing your question body through an LLM is much different from passing your title through a search engine. Commented May 11, 2023 at 0:10
  • 4
    @Makoto Machine learning systems aren't like real brains, in that they don't automatically "learn" from data when you use them to operate over that data. I would be quite surprised if Stack Overflow's implementation blurred those lines (though other companies pulling such a stunt wouldn't surprise me in the slightest).
    – wizzwizz4
    Commented May 11, 2023 at 3:36
  • 7
    Yes. Moving boxes around has nothing to do with AI assistance. Do not mix the two of them. People (including yours truly) have been suggesting to move these boxes around for years, long before AI was a buzzword. Also, yeah, only the people who already wrote good titles are going to know what a good title is and whether to accept, reject, or modify an AI-generated title. Everyone else is just going to accept whatever the AI generated. Then it'll be on the small number of us who know how to write a good title to decide whether the AI-generated title is acceptable. So, really, nothing's changed.
    – Cody Gray Mod
    Commented May 11, 2023 at 5:41
  • 6
    We can only test the suggested titles properly once the body has been composed. So anyway, we were going to be moving title to the end. So while we are doing this, we are also going to be testing the variant of moving the title to the end while hiding it in the beginning. In essence, there are two things being tested here, thus 4 variants (including the baseline), so we will be able to isolate the effects of each one. And TylerH - I know you are seeing this.
    – Yaakov Ellis StaffMod
    Commented May 11, 2023 at 7:28
  • Writing the title last is a good exercise, but it does have the impact of delaying the duplicate suggester (unless that is also changing). You have this weird set of circular dependencies (see below) and SO Inc ultimately has to decide which bits are more important to them. Commented May 23, 2023 at 19:25
  • Discover dups before lots of question-writing has taken place. Discover best possible dup target. Suggest good titles. Suggest good tags. Don't put tags in the title suggestion. etc. Commented May 23, 2023 at 19:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .