-81

2023-09-07 -- We're now targeting next week for a first round of invitations. We'll have a more in-depth update to post around that time. Apologies for the delay.


Stack Overflow is planning to release an opt-in alpha test of a major update to the on-site search UI soon, called “OverflowAI Search.” We initially previewed this feature in this Meta Stack Overflow post, and we’re now looking for volunteer reviewers to provide feedback on the feature. We plan to launch the alpha test on around the end of August, and we’ll be soliciting feedback from volunteer reviewers through the end of September.

Important info for you to know

  • We’ll be discussing and gathering your feedback in a private instance of Stack Overflow for Teams.
  • Our plan is to respond to feedback, feature requests, bug reports, and concerns that users may have while navigating the feature. The Teams instance is for us as staff, (Community Managers, Developers, Product Managers, and Designers) to engage with community members about these features as we iterate on them before release.
  • To volunteer to review and discuss this feature, please fill out the following form. (Please spend at most a few minutes on this!)
  • Not all volunteers will be selected to provide this feedback; if you have been selected, you should receive an invite within the next two weeks. We’ll post here when we have notified those who are selected.

If you’d only like to sign up for the alpha test and would not like to register to provide private feedback at this time, you may do so by clicking “Register interest” in the upper-right corner on the Stack Overflow Labs page. Or, you can toggle the alpha waitlist opt-in icon within your account settings.

How do I know if I am qualified to participate?

  • We are seeking participants across a wide range of users with varying levels of experience using the site. If you are in doubt, please fill out the form!
  • You are willing to sign up to use the feature during the test.
  • You are willing to provide open, constructive feedback on the features we present to the group.
  • You have at least two hours to provide feedback during the month of September.
  • You agree not to share content from the reviewer group, such as screenshots, publicly (before the release date). If you are interested in signing up, please do via this form. We are looking forward to receiving your feedback and working together on these features.
11
  • 5
    To clarify, this is only the semantic search from that linked post, and not the AI generated answers?
    – Erik A
    Commented Aug 15, 2023 at 17:40
  • 82
    You agree not to share content from the reviewer group This is exactly the opposite way Stack Overflow was always developed. Public testing MUST be, well, public.
    – Alejandro
    Commented Aug 15, 2023 at 19:05
  • 36
    Hey folks, for clarity's sake, we're still going to be welcoming feedback on Metas as normal after the closed Alpha test and initial feedback round has concluded. This working group isn't intended to circumvent public feedback, but rather, make it possible for community members to provide direct feedback during a very early stage as we prep for initial full release.
    – Bella_Blue StaffMod
    Commented Aug 15, 2023 at 19:25
  • 18
    Have you considered throwing in a small compensation for the at least two hours of feedback per reviewer? Maybe make a donation to a worthy cause of choice from a given set of choices and a given height. Commented Aug 15, 2023 at 23:05
  • 8
    Well, that answers my question why my question for clarification on the success metrics hasn't been answered. Are you planning to share success metrics (such as a drop in question rate due to the new search giving the correct answers) at some point, now or in the future?
    – Adriaan
    Commented Aug 16, 2023 at 5:46
  • 15
    What does having at least two hours for sharing feedback means? Do you expect some kind of one on one conversation, or is this just the minimal amount of time you expect people will devote to share their feedback through Teams.
    – Dalija Prasnikar Mod
    Commented Aug 16, 2023 at 8:28
  • 1
    @DalijaPrasnikar I expect the latter.
    – TylerH
    Commented Aug 16, 2023 at 11:57
  • 5
    @DalijaPrasnikar yes the latter, as in the amount of time that we hope community members dedicate to providing feedback through the Teams instance.
    – Bella_Blue StaffMod
    Commented Aug 18, 2023 at 16:45
  • 7
    I feel like that having this in a separate, locked-away Teams instance isn't exactly transparent. I get the reasoning, so I don't need it brow-beat into me, but it's not the same as using the same platform that we all have access to (e.g. Meta) to provide feedback on this feature, which will have significant, wide-reaching consequences on how users and power users alike can leverage this site. Yes, it makes sense to have some kind of controlled way to gather useful feedback, but I feel that this comes at a significant loss in transparency.
    – Makoto
    Commented Aug 21, 2023 at 15:24
  • 7
    so... it's been 15 days (over the two week window in which we were promised an update about when those who have been selected have been notified). Did I just miss it? Where's that update?
    – starball
    Commented Aug 30, 2023 at 18:52
  • Followup post: OverflowAI Search is now available for alpha testing (September 13, 2023)
    – V2Blast StaffMod
    Commented Sep 13, 2023 at 18:55

6 Answers 6

72

Again, what is so proprietary about providing feedback and developing a public feature that all of this must again be kept behind teams? Developing it in this way hides important context for how we end up with what we get in the end.

I’d love to participate, but I refuse to support continuing to bury feedback/collaboration.

14
  • 16
    We have used working groups on Teams in the past on a variety of features. We most often do so when we expect discussion about that feature to be involved and require focused back-and-forth between community members, Community Managers, and developers. In this case, however, the feature will be in an Alpha state and will not generally available.
    – Bella_Blue StaffMod
    Commented Aug 15, 2023 at 18:06
  • 4
    @Bella_Blue Can you make it publicly accessible, then, so that no registration is required for it accessing the private teams instance with read-only permissions? Commented Aug 15, 2023 at 18:13
  • 8
    @AndreasismovingtoCodidact read-only permissions are not a feature that is available on Teams without a registration. So unfortunately, this is not possible.
    – Bella_Blue StaffMod
    Commented Aug 15, 2023 at 18:49
  • 4
    Is there a reason that focused back-and-forth discussion couldn't take place on a Meta site, like is being done here right now?
    – Sam Hanley
    Commented Aug 15, 2023 at 19:31
  • 25
    For one thing, having a bunch of separate bug reports for, or questions on how to use, a thing that's in private-access beta show up on Meta would be a little noisy. One thing about a team that is nice is that it's silo'd away so that you don't have to sort/sift through other topics. It also allows you to avoid commentary or questions from people who aren't participating from clogging up the site or serving as distractions. cc @SamHanley
    – TylerH
    Commented Aug 15, 2023 at 19:39
  • 35
    That being said, Teams really should support a 'public read' option.
    – TylerH
    Commented Aug 15, 2023 at 19:39
  • 3
    @TylerH Or, has has been suggested countless times before: enhance Meta with functionality that accomplishes all this. Maybe... Meta Collectives! (Obviously not the exact same (because that wouldn't solve the issues Tyler mentioned), but you get the point). Commented Aug 15, 2023 at 19:42
  • @AndreasismovingtoCodidact i mean... a reimagined "collections" would potentially be useful for this purpose. but at the end of the day that's just meta tags.
    – Kevin B
    Commented Aug 15, 2023 at 20:04
  • 2
    "Noisy" versus "transparent" is in the eye of the beholder, that's all I'm saying. There's also chat rooms, which are publicly readable and are where noisy discussion normally gets sent. This just feels like an arbitrarily tiered system of when certain tools do and don't get used as they're supposedly intended.
    – Sam Hanley
    Commented Aug 15, 2023 at 20:04
  • 2
    There is value in having these things happen behind close doors, most notably, you get to control first impressions more directly in that you get a whole second (third?) chance to make a first impression when the "initial" release occurs in addition to the one that occurs to those who participate privately. It's hard to overcome negative sentiment when people write it off entirely due to first impressions. but... that ship has sailed with this product. I think it'd be valuable to use this and future initiatives to build trust.
    – Kevin B
    Commented Aug 15, 2023 at 20:09
  • 2
    @AndreasismovingtoCodidact It sounds like you just described SO Teams.
    – TylerH
    Commented Aug 15, 2023 at 20:32
  • 4
    @SamHanley No, noisy and transparent are not a difference in perspective, they are completely different, orthogonal terms. It is objectively a lot more work to track questions or bug reports about a beta feature if they are posted here vs in a dedicated site. It is also objectively a lot more work to work through them when you have countless people uninvolved in a feature's testing or development interacting with said bug reports. I do not think "use Meta" would really mean more transparency... it just means they will be more likely to be missed by developers.
    – TylerH
    Commented Aug 15, 2023 at 20:35
  • 49
    I don't think it's because it's proprietary. I think they just want to avoid launching a tool that then gets completely destroyed and shown to be actively destructive in public. That'll still happen, because they cannot fix the fundamental problem with LLMs that makes it a problem in the first place, but I suspect they think they can. I suspect it's far more about the optics around the release, and not because it's super secret Commented Aug 15, 2023 at 22:56
  • 13
    Probably because the last attempt of "using AI" (aka building a thin ChatGPT wrapper) was so laughably poor that the feature soon turned into a joke. I guess they don't want the next failed experiment to blow up in their face in public as well?
    – Lundin
    Commented Aug 16, 2023 at 7:01
44

You haven't really addressed questions asked from your initial announcements.

If you intend to use AI summarization, then this experiment will be exercise in futility. AI will hallucinate and I don't see how can you solve that problem.

No amount of Alpha or Beta testing and polishing will suffice, if you are not willing to give up on the part of the feature if it fails.

So the real question is if we volunteer to test and spend time on giving you feedback, how seriously are you prepared to act on that feedback?

15
  • 2
    the part of the feature if it fails It should read when it fails instead.
    – Alejandro
    Commented Aug 15, 2023 at 21:42
  • 5
    OpenAI (ChatGPT) often hallucinates and suggests something you mentioned you already tried or already ruled out but it’s unaware of context. I expect worse results for GenAI because if even Google and Microsoft can’t use something like OpenAI properly then I don’t have any hope for smaller institutions Commented Aug 15, 2023 at 22:10
  • 36
    "So the real question is if we volunteer to test and spend time on giving you feedback, how seriously are you prepared to act on that feedback?" - I suspect the application form exists to weed out users opposing the features, and to prioritise users who don't plan to give that type of feedback. I have no way to prove this, obviously, but the wording of the last question seems pretty suspicious ("What steps do you take to ensure your feedback is constructive and actionable?" in context appears to call negative feedback unconstructive and/or unactionable) Commented Aug 15, 2023 at 23:00
  • @Alejandro I tried to be open minded when phrasing the answer ;)
    – Dalija Prasnikar Mod
    Commented Aug 16, 2023 at 6:20
  • 7
    @Zoe which will work wondrously well. A nice, padded echo room with only positive bobble-heads nicking and applauding whenever asked to. We'll see the feature in public beta, at least, they promised to do so in the negotiation outcome. Otherwise, it'll at some point go live and latest then the public gets to provide the unwanted criticism, if the entire system turns out to be an utter, hallucinating failure.
    – Adriaan
    Commented Aug 16, 2023 at 6:29
  • 5
    @Zoe Everything points in that direction. I haven't seen the application form, but there was plenty of feedback on the introductory post that we can say feature is not only half baked, but the recipe is wrong. And jumping straight to private testing without addressing already given feedback does not sound promising.
    – Dalija Prasnikar Mod
    Commented Aug 16, 2023 at 6:30
  • @Zoe and given the companys insistence on shoehorning AI the network - and subsequent attempts (read: dumpsterfires)
    – Ja Da
    Commented Aug 16, 2023 at 8:56
  • 3
    Candidate filtering is a good thing, @Zoe. They can go ahead with the feature based on feedback from genAI fans, and then in public beta it meets users who actually try to break it. It's a lot more spectacular that way. And "look, genAI hallucinates again" is probably not actionable anyway :P Commented Aug 16, 2023 at 11:38
  • 24
    @Zoe To me, the "actionable and constructive feedback" portion is to encourage feedback to include why you feel a certain way, or what you would propose to change. We'd like to avoid responses in the vein of "I like it" or "I hate it". Also, I heartily encourage skeptics to participate in the alpha feedback cycle. It does us no good to only hear from those excited about these products.
    – Tyler McEntee Mod
    Commented Aug 16, 2023 at 14:05
  • 1
    @TylerMcEntee But still the constructive feedback from skeptics would probably be along the lines of "it's better to scrape it". The presentation doesn't give the impression that this is within the mission parameters. Participants should probably one way or another kind of believe in that feature. Others should simply ignore the volunteer call and wait for later. Commented Aug 16, 2023 at 21:01
  • 5
    @NoDataDumpNoContribution "it would be better to scrap it" is feedback, but it's certainly not constructive. It doesn't say anything that will improve the feature or provide any feedback that can be used to determine it should be scrapped. This kind of feedback is very easy to just brush aside and ignore.
    – Kevin B
    Commented Aug 16, 2023 at 21:04
  • @KevinB I meant that proper include reasons why, but just meant that if one thinks there is nothing to improve then probably not volunteering is the best choice. Commented Aug 17, 2023 at 5:55
  • 3
    @NoDataDumpNoContribution If the feature can cause harm if it doesn't work, then people would still want to give that kind of feedback during testing along with reasons why, even if they knew up front that feature will not work properly.
    – Dalija Prasnikar Mod
    Commented Aug 17, 2023 at 6:06
  • "If you intend to use human summarization, then this experiment will be exercise in futility. Humans will hallucinate and I don't see how can you solve that problem."
    – Ian Boyd
    Commented Sep 1, 2023 at 17:28
  • 3
    @IanBoyd I guess we are doomed then ;)
    – Dalija Prasnikar Mod
    Commented Sep 1, 2023 at 18:03
26

You agree not to share content from the reviewer group, such as screenshots, publicly (before the release date)

Why not? I don't see a good reason to keep such minutiae secret (aside from whether it should be in a Team or on Meta or elsewhere). Who cares if someone sees a screenshot of an alpha or beta feature that may or may not exist by the time of release?

3
  • 13
    Because the feature could later look much different and then there might be much ado about nothing in the meantime. One could easily ask the question the other way around. Why do people want to see everything that's in alpha stage? And where does it end? Commented Aug 15, 2023 at 23:08
  • 5
    @NoDataDumpNoContribution I'm not arguing that they should be making the entire process public and in our face. But a non-disclosure agreement is kind of extreme for something like this.
    – TylerH
    Commented Aug 16, 2023 at 11:55
  • @Cuzy I don't see how sharing screenshots/info will invoke confusion. If someone doesn't understand something about a screenshot of a private beta test they aren't participating in, that should obviously be of no concern because the person is, as just mentioned, not participating in the test.
    – TylerH
    Commented Aug 17, 2023 at 14:15
14

You agree not to share content from the reviewer group, such as screenshots, publicly (before the release date)

What exactly is the "release date"? This is alpha, so is the "release date" beta? The end of September when feedback for this round is no longer being gathered? "Final" launch?

0
12

Just to know, could you at least share a brief description of how this new feature was implemented? I don't need the specifics, I am more interested in details like what technology / frameworks you used and so on.

To be blunt: your previous not-so-great attempt at incorporating AI seemed to be based on ChatGPT, as it was clearly proved by its extreme vulnerability to the same jailbreak attacks ChatGPT struggles with. And all of this was caused by an attempt to forcibly use ChatGPT for some task it couldn't handle very well in the first place.
Do you intend to use ChatGPT or any similar LLM again or did you switch to something that ... was made for this type of task at the very least (semantic search libraries etc)? As you can probably imagine this would be a decisive point for many users in deciding if helping with this new feature test is worth the time spent.

1
  • 7
    In the original post it seems to be pretty clearly using a GPT/GPT-like model. Someone high-up is still desperately trying to push LLMs and generative content into places they don't belong.
    – DBS
    Commented Aug 30, 2023 at 11:05
3

The toggle switch in the settings says

We are gradually enabling this feature to users who opt in. You may not see changes reflected immediately.

If I toggle on, once the feature is enabled for me, will I be able to toggle it and immediately switch between the two? I want to try it, but I also want to switch back to regular search quickly and frequently for practical usage.

And does the time when I toggle it on affect when I will have the feature enabled? Is it first-come-first-serve? How gradual is "gradual"?

2
  • 1
    As soon as the feature is enabled for you, you should be able to toggle back and forth.
    – Bella_Blue StaffMod
    Commented Sep 13, 2023 at 18:53
  • 1
    It will be immediate depending on if the feature is enabled. You should receive an email telling you when you are able to use the new features.
    – Bella_Blue StaffMod
    Commented Sep 13, 2023 at 20:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .