-42

A few weeks ago, Stack Overflow CEO Prashanth Chandrasekar spoke at the WeAreDevelopers conference to announce some exciting features we have been working on. On that day, we also shared a small preview of our new search experience. Today, OverflowAI Search is launching into a limited private alpha.

What is an alpha launch?

An alpha launch refers to the initial release of a new product/feature to a limited number of people. Its main purpose is to collect early feedback, identify bugs and other issues, and ensure that the fundamental features function properly by gradually making improvements before it is scaled to a larger audience during a later release.

In short, this alpha launch is a very early glimpse of this feature. We expect the feature set at times to be unstable, and major changes to the product will be made as we continue to iterate. Also, this alpha launch does not replace later testing, where the broader community will have the opportunity to provide feedback here on Meta.

What problems are we trying to solve with this feature set?

We have shown you a sneak peek of what we hope to achieve with this feature set, recognizing the feedback we’ve received that for many developers on Stack Overflow, finding answers to their questions can be difficult or sometimes take a while. This is a complex issue, so let’s look at each of those challenges individually and explain how we are trying to provide solutions.

Challenge: Difficulty in Finding Relevant Answers

  • We understand that many developers within the Stack Overflow community face the challenge of locating answers to their coding questions. The sheer volume of content and the lack of search relevancy can make it frustrating and time-consuming to identify solutions that truly address their coding issues.

  • Solution: Semantic Search and Improved Relevance

    • To address this, we're introducing AI-powered enhancements that will significantly improve search relevance. Through a hybrid Elasticsearch and semantic search, you'll receive search results that better match your queries, allowing you to quickly find the answers you're looking for to efficiently and effectively solve your problems.

Challenge: Time-Consuming Consumption of Multiple Search Results

  • We've heard consistent feedback about the time it takes to navigate through search results on Stack Overflow. We recognize that people want to spend less time searching and more time coding, so we're committed to streamlining this process.

  • Solution: AI Summarization for Quick Insights

    • Our new AI summarization feature will enhance your interaction with search results. Not only will it pull out the essential information from answers, but it will also credit the contributors who provided those insights. This means you can get to the core of answers swiftly while still having the expertise of community members at your disposal.

Challenge: Precision in Search Query Formulation

  • We're aware that crafting queries that yield accurate results can be challenging. Expressing coding issues precisely can be tough, leading to search outcomes that may not quite align with what you need.

  • Solution: Conversational Search Query Refinement

    • To tackle this challenge, we're introducing conversational AI capabilities that allow you to refine your queries in a natural, conversational manner. This approach ensures that your search results match your actual requirements, reducing frustration and delivering more accurate answers.

Our primary aim is to improve the overall search experience for developers and community members by making it easier for users to surface answers to technical questions. We believe AI can play a significant role in opening up possibilities for a wider range of search experiences.

We will share a deeper dive into the origin and the research behind OverflowAI Search in a Meta post early next week. We'll link to the post once here once it's been posted, so stay tuned for updates.

What we are looking for during this alpha, and how we define success?

During the initial alpha phase, our measures of success will primarily be qualitative (rather than quantitative) due to limited audience size. Over time, with wider user adoption and the progression towards Beta, quantitative measures will help provide a more complete picture of the landscape.

As mentioned above, regarding the challenges we’re trying to solve, we can define success based on several hypotheses which we hope to validate with both qualitative and quantitative measurements.

  • Hypothesis 1: Implementing semantic search will improve search relevancy for coding questions.
  • Hypothesis 2: AI summarization will enhance search efficiency by extracting key insights from answers.
  • Hypothesis 3: Conversational search will lead to more helpful search outcomes for user queries.

Here are the qualitative measures for success:

  • Are you easily able to complete the core tasks of information retrieval, results assessment, and refining results?
  • How do you feel about the product experience? (e.g. satisfaction, value, trust)
  • Which features are most/least valuable, and how are you engaging with them?
  • Do the AI responses provide consistently useful, relevant and correct information based on your queries and the associated sources?
  • Has your usage or perspective of SO changed?

Here are the quantitative measures for success, which can evolve over time as we gather more data:

  • Successful searches including, but not limited to:
    • Attempting to vote up sources
    • Voting on a post referred from OverflowAI Search
    • Clicking “Yes” in the AI feedback question on whether the summary helped you solve your problem
  • Number of searches performed
  • Traffic measured by MAUs (monthly active users)
  • Engagement with the OverflowAI Search features including, but not limited to:
    • Expanded summary
    • Copying code blocks
    • AI feedback submissions

We will also be monitoring page views, questions asked and user flags as counterbalance metrics to ensure we are not over-optimizing on success metrics.

What will this process entail, and what kind of feedback are we looking for?

Fundamentally, the choice to launch this alpha testing round is twofold. First, we want to provide more transparency within the product development process. This is something that's been requested by contributors here, and as a company we're pivoting more toward working publicly to respect those desires.

But more importantly, we want to build these features collaboratively, by inviting community members to give input earlier in the design process. While the alpha stage includes a holistic approach, including many different data points and user research, we want to provide the opportunity for users to help influence the direction of this feature by contributing feedback and helping us identify risks/concerns that might not have been considered yet. We plan for this concentrated feedback collection to span over a four-week period, but will adjust that time based on readiness and the data we have collected.

A look behind the curtain

A key focus of our ongoing efforts is to enhance the relevance of our results. We recognize the importance of user feedback in this specifically, as it provides invaluable insights into OverflowAI Search performance. To offer a more transparent view of the various aspects we’ll be enhancing in the backend, Felippe Rangel, Senior Software Engineer and Tech Lead, describes our options as follows:

The process of summarizing search content in our current system involves six different areas where we can make adjustments to enhance the summarization results. These areas, or “tuning knobs”, each play a crucial role in the process.

The first knob is Semantic search quality. This is the starting point of the process, where we retrieve posts that we believe will contain useful answers based on your query. We continually improve our algorithm and data analysis to retrieve better results.

The second knob is the Question selection strategy. Once we have a number of questions that fit your query, we need to decide on the parameters we use to select amongst those questions. For example, we might prioritize questions that are non-deleted, non-closed, and have an accepted answer.

The third knob is the Answer selection strategy. This involves deciding how we should select answers from the set of questions. Should we consider them per-question or as a whole?

The fourth knob is the Answer ranking strategy. This involves deciding how we rank the selected answers from the most relevant to your query to the least relevant, and how many answers we should summarize. For instance, we might prioritize accepted answers first, then by highest score, etc.

The fifth knob is Prompt engineering. Once we have the answers, we need to decide how we communicate to the LLM how it should summarize those answers into one concise response, possibly with examples.

The final knob is Temperature. This refers to how random the LLM results should be. We generally operate on a very low temperature, but it could make sense to leverage the LLM more here.

All of these tuning knobs are already implemented and functional in our backend. They will be our first attempt at providing better results as we gather more info from your alpha experience.

How can you participate?

Recently we announced a call for volunteer reviewers for this feature set within a private Teams instance, and we are still looking for participants. Please fill out this form to be considered for the working group, and read more about what participation in this group entails.

Alternatively, if you would like to just check out the feature set without providing feedback, you can sign up on our Labs page here. Or, you can toggle the alpha waitlist opt-in icon within your account settings.

How we’re letting users know about these features?

We’ve updated the left sidebar on Stack Overflow pages to allow users to more easily access some OverflowAI Labs features (including OverflowAI Search and Discussions).

enter image description here

You may also see a pop-up on the Search bar on Stack Overflow to ask if you are interested in participating in the alpha launch.

enter image description here

For those who choose to participate, we will be gradually rolling out the OverflowAI Search features to waitlisted users in waves. Look for an email confirming your access to try out the new features, and please provide feedback to help us continually improve.

If you are participating in the alpha and want to go back to the previous search experience, you can always toggle it off in preferences to remove yourself from the alpha.

As always, we look forward to hearing from you as we work together to build OverflowAI Search and will respond to inquiries as soon as possible.

TL;DR Thank you

This was a long post where we outlined a lot of technical specs and big goals for this alpha launch of OverflowAI Search. The TL;DR is that OverflowAI Search is a work in progress but we are excited to work with the community as we shape and refine this feature. Thank you in advance for collaborating with us. This process would not be possible without your passionate feedback, and we are thrilled to see what we can build together.

16
  • 11
    If I've already filled the form and have not been notified, does that mean I have not been "selected"? Considering that you're still looking for volunteers, would you share your selection criteria?
    – M--
    Commented Sep 13, 2023 at 16:12
  • 19
    Tiny future request: make the announcement post a little bit earlier before you turn on the in-site notices (like a day, at least). It is good to give users a bit of time so they are prepared for unexpected announcement windows attached to various parts of pages on main-SO.
    – TylerH
    Commented Sep 13, 2023 at 16:15
  • I suspect, my opt out early on due to receiving teams spam after signing up still has me opted out, even though i reopted in.
    – Kevin B
    Commented Sep 13, 2023 at 16:19
  • 7
    @M-- as we mentioned in the post we are doing a slow rollout to participants. If you did not receive your notification while you have not been chosen for this first wave of participants that doesn't mean that you will not be selected for a later wave.
    – Bella_Blue StaffMod
    Commented Sep 13, 2023 at 16:21
  • 2
    "and we are still looking for participants" - The edit should probably have removed this, as the link says that isn't the case any longer ;-) Commented Sep 13, 2023 at 17:39
  • 7
    @NotTheDr01ds We've reopened the form. Apologies for the lack of clarity. (For context, there are technically two ways to sign up for Alpha -- using the form, and via Stack Overflow Labs. The form doubled as an application to the volunteer reviewer Teams group, so we'd closed just the form as part of Alpha launch. But you're right that it's confusing, so to minimize confusion, I'll just leave the form open for now.)
    – Slate StaffMod
    Commented Sep 13, 2023 at 17:46
  • 2
    I don't see OverflowAI search in my SO sidebar.
    – starball
    Commented Sep 13, 2023 at 18:36
  • 1
    @KevinB: The Discussions link in the sidebar does point to stackoverflow.co/labs/discussions, which contains links pointing to the Discussions tabs within each collective it's enabled on. However, the Search link in the sidebar does point to the feature itself – so I'm guessing it's only visible to those who are in the alpha. (But I'll leave it to Bella or Slate to confirm that.)
    – V2Blast StaffMod
    Commented Sep 13, 2023 at 18:59
  • 4
    Your "qualitative measures for success" are 5 questions. Your "quantitative measures for success" do not say how the measures relate to success.
    – philipxy
    Commented Sep 13, 2023 at 20:02
  • 8
    We don't need this. This is going to be terrible. Please, add an opt-out functionality for those that prefer the predictably broken search instead of the randomly inaccurate-but-always-convincing AI garbo. Also, if you think AI is ready to deal with any of this, just keep this question in mind: codegolf.stackexchange.com/questions/265259/… - it just generates stuff that, at a first glance, looks accurate but isn't. Commented Sep 16, 2023 at 2:39
  • 22
    I google and see the results on the top, and it works. I don't know why everyone is so desperate looking for some way to add an AI bot.
    – Rohan Bari
    Commented Sep 16, 2023 at 3:56
  • 20
    @RohanBari AI is the fad of the year. It has to be added to everything, or you miss the hype train. Commented Sep 16, 2023 at 5:19
  • 3
    Unpopular opinion: more often than not, I reach Stack Overflow from a private browsing, because I'm searching with a real search engine and don't want to pollute my non-private browsing with every searches I may type throughout the day. As such, I always get greeted by this floating ad that is very invasive. I would have preferred if it was a small button that allows you to open the whole thing, so that I can see it when I want instead of having it thrown at my face every single time. But something tells me it would have been too small to be visible by everyone (i.e. what I think you want).
    – Clockwork
    Commented Sep 21, 2023 at 7:26
  • 2
    Has anyone from meta that isn't already on the private team for this gained access to it? seems awfully quiet around here for a feature this controversial being in alpha test.
    – Kevin B
    Commented Sep 25, 2023 at 18:04
  • 1
    @KevinB I got access yesterday and am not in the Team. Commented Sep 27, 2023 at 12:31

16 Answers 16

189

You folks seem to be searching desperately for a way to use a chatbot. What users need is a search that actually finds what they are looking for.

The first step of your chatbot enhanced search is to make the search function return relevant results.

If the search would actually return relevant results, then the chatbot "enhancement" isn't needed. Questions and answers with relevant information are more useful than a chatbot generated summary of the same questions and answers.

Fix the real problem (poor search) instead of tossing some useless chatbot "enhancement" into the mix.

All we need (and want) is for the search to actually work.


Joke:

Q: How do you find information on Stackoverflow using the built-in Stack site search?

A: Ignore it and use google with "site:https://stackoverflow.com".

Seriously:

Fix the lousy site search feature. That's it. That's all.

13
  • 4
    Seeing how many people expect an answer tailored to their question, I do actually see value in an "answer personalisation" even if I would not use it myself. Commented Sep 14, 2023 at 8:29
  • 20
    @MisterMiyagi: People may want a simple "do X" answer. What they need is an explanation of the background so that they understand what they need to do and why. The summaries will throw away that background, possibly adding obfuscating BS in the process - or just, you know, make crap up because it's a chatbot.
    – JRE
    Commented Sep 14, 2023 at 10:02
  • 8
    Your joke is the reality for a lot of community members who are trying to find information on Stack Overflow, and not a really great user experience. While Conversational Search will allow for a more tailored experience, our goal is to also improve relevancy by utilizing semantic search. In short, we hope to improve the overall search experience because we know that is what the community wants. Good joke though, it's kinda the inspiration behind this initiative.
    – Bella_Blue StaffMod
    Commented Sep 14, 2023 at 11:10
  • 99
    @Bella_Blue: That "joke" is my experience in nine years on the electrical engineering stack exchange. I have 1584 answers there. I cannot find things I know I have answered before using the site search. I gave up on it long ago. Google usually returns my own answers within the top few items. This is not semantic anything. I supply keywords that I know I used, and the site search simply cannot locate them. When google coughs up my own answers for me, the keywords I gave are right there, just like I knew they were. The site search doesn't find them - at all.
    – JRE
    Commented Sep 14, 2023 at 12:15
  • 15
    We understand that frustration and know you are not the only one. I get it; you just want a Stack Overflow Search to just deliver the thing you searched for. Simple and to the point. While summarization and conversational search are for users who want to try them out, we didn't forget about the relevancy of traditional search results and are working towards improving those as well.
    – Bella_Blue StaffMod
    Commented Sep 14, 2023 at 12:30
  • 2
    I do wonder exactly what kind of search system is in place currently. There are extremely simple search systems out there that are fast, dirt cheap to run and produces sensible results even if only some of the time.
    – Passer By
    Commented Sep 14, 2023 at 16:53
  • 7
    @Bella_Blue It isn't a joke, it is the reality of the current search implementation. I have never found a relevant search result even when filtered down to only search my own answers that I know exist. As noted by Passer By, search shouldn't be this much of a mystery in this day and age, and conversational search just isn't the answer (Ask Jeeves anyone?) even in this early AI era.
    – pilchard
    Commented Sep 15, 2023 at 0:15
  • 1
    Making a search engine that returns relevant results probably isn't even difficult. Google has a search API, so all that needs to be done to make the built-in site search return relevant results would be (1) parse the search query, (2) send it to Google's API, (3) filter the results to only show questions and answers (not e.g. user profiles) and to match potential search queries that Google can't do (for example score:), and (4) display the results. Commented Sep 15, 2023 at 18:17
  • I use google as searching engine for every website though, because google's entire purpose is searching. Virtually every website has garbage searching options. Commented Sep 25, 2023 at 12:20
  • 3
    @Bella_Blue How about making the site search a thin wrapper on top of Google instead of a thin wrapper on top ChatGPT then? This would be much more to the point and solve the actual problem you claim to be dealing with.
    – Lundin
    Commented Sep 26, 2023 at 13:37
  • 22
    Except... I suspect there is no actual problem to be solved here - I rather suspect you were just told by upper management to do AI-something-something... because: AI. Then you have to go looking for a problem that GenAI solves and none was found. The second best then is to find something that is a problem that GenAI does not solve, then try to solve it with the wrong tool anyway. Because: AI.
    – Lundin
    Commented Sep 26, 2023 at 13:38
  • 1
    @Bella_Blue how long you been working to solve those, 9 years as well? Why not vectorize the Q&As using an LLM and connect the site search to that. Did you consider that before adding extra steps with a chatbot? Commented Oct 8, 2023 at 16:18
  • 2
    I think the problem is actually bigger... back in the day you'd often get decent results because things had been upvoted lots. Now the sheer volume of questions I find there's a lot less relevant stuff, and the important stuff doesn't see enough eyes, to get enough votes to then gravitate towards to top of search results.
    – Ian
    Commented Oct 13, 2023 at 9:28
131

This feature is in violation of the share by license. It is easily correctable.

There is no attribution to code that is provided. It is broadly "sourced" in a hidden section for sources. There needs to be citation for use.

These are verbatim copies of the sourced answers.

Plagiarized code

Hidden and overly broad sources do not count for actual attribution when it is verbatim like this.

Copy sections with no attribution

Each one of the "copy" sections needs to have a citation source in there. It can be small, out of the way, or even just a [1] which references the answer it came from.

9
  • 23
    As an aside, from a common courtesy point of view, generally, in industry, when you verbatim copy someone's work you will include a source code comment to indicate where it came from. That isn't possible here with this raw copy.
    – Travis J
    Commented Sep 13, 2023 at 17:16
  • 29
    As we had stated in the post this is the alpha version of this feature and is not polished or fully realized as to what the final product will look like. That being said, this is something that we want to get right and there are internal talks being had about the exact issue of appropriate attribution. It is good to know we are of the same mind.
    – Bella_Blue StaffMod
    Commented Sep 13, 2023 at 17:22
  • 7
    The company already has a right to redistribute our content under any terms they'd like. You'll find it in the ToS. Commented Sep 13, 2023 at 17:28
  • 51
    @AndreasismovingtoCodidact: Those terms must still abide by CC-by-SA, per CC-by-SA.
    – Makoto
    Commented Sep 13, 2023 at 17:29
  • 34
    No, not "under any terms". Under specific terms, notably "pursuant to Creative Commons licensing terms (CC BY-SA 4.0)". Would you like to know more? stackoverflow.com/legal/terms-of-service/public#licensing
    – Travis J
    Commented Sep 13, 2023 at 18:43
  • 32
    @Bella_Blue with all due respect, compliance with the licensing should be the top priority before this alpha goes any further. SO's already walking an extremely risky line of trustworthiness with everything else that has gone on. Actually & knowingly violating the legal contract with contributors isn't the right move
    – vbnet3d
    Commented Sep 21, 2023 at 21:02
  • 3
    > this is the alpha version of this feature (translation: ethics will be considered in a future release.) Commented Oct 4, 2023 at 11:07
  • 2
    -1 because the US Copyright office has ruled that AI-generated content (if sufficiently different from the original) is not copyrightable. Additionally, small code snippets are generally not copyrightable. You cannot violate copyright for something that's in the public domain. Commented Oct 7, 2023 at 14:18
  • 2
    Irrelevant. For those interested see here: copyright.gov/ai federalregister.gov/documents/2023/03/16/2023-05321/…. That has nothing to do with this though, because this isn't about copyright. It is about an explicit license agreement between Stack Overflow users and Stack Overflow the company. Also, illegally placing something for everyone to view does not suddenly make it free of its copyright or license.
    – Travis J
    Commented Oct 9, 2023 at 8:57
55

Solution: AI Summarization for Quick Insights

What's the license of the machine-generated summaries?

44
  • I don't think "Number of searches performed" is a particularly great way to measure success. What if I try to do more searches because the search results are bad?

  • First, we want to provide more transparency within the product development process

    Great!

  • The second knob is the Question selection strategy. Once we have a number of questions that fit your query, we need to decide on the parameters we use to select amongst those questions. For example, we might prioritize questions that are non-deleted, non-closed, and have an accepted answer.

    Hold on a sec. Why would you ever show content from a deleted question? Why would you even consider such content as a candidate for a search result?

  • The third knob is the Answer selection strategy. This involves deciding how we should select answers from the set of questions. Should we consider them per-question or as a whole?

    What do "per-question" and "as a whole" mean?

  • For instance, we might prioritize accepted answers first, then by highest score, etc.

    Going just by score is probably better on average than prioritizing the accepted answer first. Prioritizing the accepted answer is letting a single person determine what is considered best rather than letting the masses speak with their votes. See also the earlier feedback you got (including the comments there)

  • The fifth knob is Prompt engineering. Once we have the answers, we need to decide how we communicate to the LLM how it should summarize those answers into one concise response, possibly with examples.

    I thought the whole point of this was "trust"? Prashanth keeps talking about trust in his blogposts and in the WAD fireside video and OverflowAI announcement video, and linking that trust to being able to cite to answers. And I thought that the generative part of AI was what undermined trust?

    • Why risk generating nonsense examples that mislead readers and erode that trust?
    • How will you know whether it even makes sense to give examples for a question? Not every Q&A is the type that needs examples.
    • Who will the examples be attributed to, and where will they be shown in the UI? I hope you don't mislead readers into thinking that generated content was written by the people who wrote the posts you feed to the LLM.
  • The final knob is Temperature. This refers to how random the LLM results should be. We generally operate on a very low temperature, but it could make sense to leverage the LLM more here.

    How is increasing the temperature "leveraging the LLM more"? I know people say that higher temperature can make ChatGPT more "creative", but what's to say that's a good thing here? You literally cite that a major pain point is having to search through multiple answers, but won't you be making that worse if you put readers in a situation where the random dice rolls of higher temperature make it so that you can get a potentially infinite number of answers by regenerating responses?

  • Are you going to follow OpenAI's sharing and publication policy this time?

2
  • How will it take into account answers that are wrong, as pointed out in the comments, but positively scored because reasons? I've yet to see an example of this system that provides context to the provided "summary" outside of score and user rep.
    – Kevin B
    Commented Sep 13, 2023 at 18:37
  • 5
    I think you're getting lost in the details a little; the point I took from the section overall was just "it's configurable, and here's some general ideas of how", not necessarily "here are the settings we're using". I think having the control is positive; giving themselves lots of wiggle room to experiment sounds like the best shot this project has.
    – zcoop98
    Commented Sep 13, 2023 at 23:14
42

I'd hate to see randomness introduced into the search. A core feature of the public web is that content and pages can be linked; that includes search results. Sure, the results change over time as the underlying data changes (slowly), but when I link a search I would expect that multiple users visiting that link around the same time will see the same results. Similarly, making search conversational might break this, as the URL needs to carry the context with it.

Please add a permalink or "share" feature to this, which allows others to reproduce the search. This would also benefit discussions about how to improve the search.

1
  • +1. Logs of the back-and-forth between a human user and a bot should be linkable; as should individual parts of the exchange; this would allow users to show what work they did before asking a question, and where confusion remains. Commented Oct 2, 2023 at 19:17
36

As a user with a gold tag badge I am often using search to identify duplicates. I see questions I know have been answered numerous times before, but when looking through search results, I find answers that are all slightly off. They tackle the same basic problem, but zoom in on a different detail, are outdated, or explain the same principle as an answer to a different premise. I, as an expert in my field, can see the similarity, but the asker I am trying to point to a duplicate would surely be confused.

Currently, I often find I am faster just writing the same answer for what feels to be the tenth time, than digging through search results that all are just not good enough.

And that is what I will be looking for when taking part in this testing phase: do I mark questions as duplicates more often?

10
  • 11
    This is an interesting insight. While it's not the original intent of the design to make duplicate question discovery easier, I can actually see why the design we've selected could potentially help quite a lot. Once you've got a chance to try it out, I'd be curious what your observations are here.
    – Slate StaffMod
    Commented Sep 13, 2023 at 20:07
  • 2
    I'll be sharing my experience here.
    – ccprog
    Commented Sep 13, 2023 at 20:45
  • 4
    @Slate the goal of duplicate discovery is kind of there implicitly: askers are supposed to search for existing on-site Q&A before posting. a question that is not asked (posted) because one found the same question by searching is a good thing. that's the asker finding the "duplicate-target" themselves instead of putting the burden on curators (the by-design happy-path)
    – starball
    Commented Sep 13, 2023 at 22:24
  • 3
    +10. This is my primary use case of the search. I'm looking for existing answers that I can point someone to, using terminology from the solution I know, which is typically found in answers and not in questions.
    – Bergi
    Commented Sep 14, 2023 at 1:25
  • I take this mechanism as requiring you to write less of such duplicate rehashes because that is what the AI will be doing* already for askers. Ideally people won't ask (as many) duplicate questions if the system is better at providing them the answer they want. (*That's an "if the Gen-AI actually works well".) Commented Sep 14, 2023 at 8:35
  • 7
    I vote a lot to close as duplicate and never once have I used the in-site search to actually find them, Google is far faster and returns far more relevant results every time. The only exception is when I know I am the author of the dupe and then i generally just resort to scrolling back through my own answers until I find it (harder since the 'redesign' that severely cut the number of answers shown per page).
    – pilchard
    Commented Sep 14, 2023 at 10:00
  • @starball most duplicate questions show very little effort on the part of the asker to find the answers themselves to the point of titles being identical save for one word. Often just pasting the OP's title into Google is enough to yield multiple duplicates (where as pasting into in-site search doesn't, alas). I don't think slightly improved (up from non-functional) in-site search is going to stop this tendency in askers.
    – pilchard
    Commented Sep 14, 2023 at 10:05
  • 1
    @pilchard I think my answer here can speak to what I think about improving that situation, but really I'd go even deeper for the problem of people not searching first: just make the ask question button only show up in the search results page (wishing upon a star).
    – starball
    Commented Sep 14, 2023 at 10:09
  • @Slate "it's not the original intent of the design to make duplicate question discovery easier" Utterly jaw-droppingly head-shakingly baffling. What do you think the site search facility is for? What do you think people are searching for? (Rhetorical.)
    – philipxy
    Commented Oct 1, 2023 at 19:00
  • I can't seem to opt out (at least opt out is not respected) and the search sometimes returns no results when there should be some, see this separate post: meta.stackoverflow.com/questions/427201/… Commented Nov 15, 2023 at 20:42
30

These points may have been already said in previous answers but I want to emphasize them:

  1. I see great value in using semantic search and think it'll greatly enhance and potentially fix the historically broken SO search so thanks for this!

  2. I'm somewhat skeptical of the AI helping in refining the search query but I think it's worth a try.

  3. I'm extremely skeptical of the AI summarization part and still think you haven't addressed the elephant in the room regarding its validity, intrinsic LLM hallucinations.

    I also learned a lot and gained a lot of value as a beginner developer a few years back (and still definitely do) from reading the answers in their entirety myself and even the discussions in the comments of answers and questions. Consequently, I really think abstracting this away is a very bad idea.

  4. Since you are going to do the summarization anyway, please don't prioritize accepted answers. They are the opinion of a single user at a single point of time and in many cases an inexperienced user. Instead prioritize highly upvoted and trending (frequently upvoted recently) answers.

24

There's plenty of info here (and everywhere this is talked about) how it will supposedly help people looking for an answer, but how will it help people looking for questions? Will it get out of my way when I'm looking for a specific well known question rather than trying to spoon-feed me a generated answer I'm not looking for?

I don't typically use SO's search when I'm looking for an answer because SO isn't the only source of answers. There's an entire internet out there with other places answers may exist that this tool can't access. Git repositories with issues to search through. Official docs with documented methods/params/returns and examples. I'd be quite frustrated if new devs only used SO's search to find solutions given how much existing information it can't access.

7
  • 13
    The purpose of this feature is to help users who are looking for answers to their questions. With this release, we are only intending to solve for the use case wherein a user wants to use Stack Overflow's resources to find an answer to their question.
    – Bella_Blue StaffMod
    Commented Sep 13, 2023 at 16:45
  • 6
    That's how i've interpreted it thus far. My concern is it leaves out those who aren't, while still being a solution that will never be more useful than using google or bing instead for that user group.
    – Kevin B
    Commented Sep 13, 2023 at 16:46
  • 11
    While this feature isn’t intended to be the equivalent to Google or Duck Duck Go as it won’t for example let you simultaneously search blogs or github, we believe that users will find value in what OverflowAI was intended to do, namely allowing users to quickly find an answer within Stack Overflow’s repository of information.
    – Bella_Blue StaffMod
    Commented Sep 13, 2023 at 17:12
  • 2
    I'm confused about the relation between your two paragraphs and the general point you're trying to make. My interpretation of your first paragraph is that you're concerned that this is replacing the current SO search, which is valid, but the current search is not useful to begin with so I don't think it's a major problem. My interpretation of your second paragraph is that you're concerned that people will use this instead of googling their question, but I don't see how that's SO's problem. Commented Sep 13, 2023 at 21:43
  • 5
    @TheGuywithTheHat my concern is all of this effort is being poured into a solution that doesn't fix a problem, while there's plenty of other problems (even some with existing solutions, such as SG, in the wind that are on hold due to this!) just continue to get ignored. Yes, site search has long been a major problem, but it's been a problem for power users. The avg user looking for an answer doesn't use it or need it. Just give us the semantic search upgrade and move on with more important improvements.
    – Kevin B
    Commented Sep 13, 2023 at 21:45
  • You know, this distinction actually sounds like something that makes SO search worthwhile. Let's face it, SO search will never be a better Google. If I know that information is on SO (say, because Google barfed out a dozen pages of hits, or because it's a basic topic and should be here at least indirectly) but not in shape (like most content here...) then having a summary of limited scope (just SO quality, not, say, reddit) seems useful. Just marketing it as "search" in the classical sense might be a problem... Commented Sep 14, 2023 at 8:24
  • 4
    @MisterMiyagi my issue is i don't trust that there's a prompt/process that involves an LLM taking 2+ answers and generating a summary without hallucinations. On top of that it leavesout a lot of important context around why the answer is the way it is. Most answers are specific to the question asked, and most highly popular answers have further comments providing even more context. Will it occasionally get things right? sure.
    – Kevin B
    Commented Sep 14, 2023 at 14:32
23

I recently saw the alert inviting me to try the new search experience, and immediately opted out of doing so. For what it's worth, my reasoning for doing so was this:

As a user, I don't care about the mechanism underlying the search functionality I'm using. No other site I use makes user-facing boasts about their search tech, I don't really want to have to know or think about it – I just want a search functionality that works in the conventional, as-expected manner: searching content and presenting it exactly as it exists, surfacing me the content most relevant to my query without editorialization.

The idea of a search function that sounds to me like may not return the same results every time I use it, or might synthesize content in a way that changes the author's intent? That's deeply off-putting, and frankly just not something I want to be bothered spending even one second learning or adjusting to. The popup for this "new experience" feels like a big neon sign hung up by the marketing department that profoundly misses the point of what users expect from a well-established UI norm like a search box, which is quiet, familiar simplicity and predictability – and for me, it's one more reason to just keep using Google as the de facto Stack Overflow search functionality.

3
  • 1
    Google’s Search AI now has Search Generative Experience => AI summaries of long articles
    – QHarr
    Commented Sep 15, 2023 at 14:21
  • 3
    Thanks for the feedback Sam. For users that want to simply browse the search results below, we have made the search summary collapsable. The search results you get for the same search input should not change, but with this new feature we have enhanced the results you get by implementing semantic search which has shown evidence of better results by utilizing natural language instead of keyword based matching. Again I appreciate the feedback shared so we can find ways of improving.
    – J.S Staff
    Commented Sep 18, 2023 at 16:00
  • That's good to hear, and thanks for welcoming the feedback. Like I said, my top-level feedback is just that it feels unusual for a company to be trying to get me excited about the technology underlying their site's search bar. If you've really made the search experience better, that's great for your users - but the louder the callouts about it being AI-powered, the more it gives the impression of being trend-based hype. Honore Doktorr's answer underscores this same point.
    – Sam Hanley
    Commented Sep 18, 2023 at 21:08
21

I have to say that the language you're using in the popup, "New search experience powered by AI" and "leveraging AI to summarize", is offputting and undercuts confidence that stackoverflow knows what it's doing. (Independently of whether using LLM chatbots or what have you will improve search experience, or whether bot summaries cite code properly, etc.) This comes from being keyed in to the current LLM hype cycle, how it's being promoted as "AI", and what that implies vs. what the actual capabilities are.

You're a technical knowledge site whose answers are written by your users, many of whom are sophisticated enough to know that "powered by AI" can mean "we've integrated a LLM somewhere in our site", or "we've integrated what superficially seems like machine intelligence somewhere in our site, but isn't quite there yet so uses human assistance behind the scenes", or even "we want our new feature to catch the eye of gullible investors, so we'll call it ‘AI’". If your users think less of you for the way you're promoting this new search feature, it's going to undercut what gets your users to answer questions here or trust answers as authoritative.

You should be very careful & specific with how you describe these features if you want to maintain trust. As of right now, with the "OverflowAI" messaging, you are not. This comes off as trying to sell snake oil to investors by jamming LLMs into your product. Like other users have mentioned, SO’s search function is not great right now; this messaging seems like waving AI at it in the hopes that'll fix it. It's not an effective way to approach the problem. At this point, I don't trust the company's judgement at all, and I doubt I'm the only one.

2
  • 4
    While many understand the distinction between the different forms of machine learning and "AI", there are even more who don't, but rest assured we are not trying to obfuscate what these tools are capable of. Many of our visitors are new to programming, and we believe they would find these features that allow them to not only find the helpful content in Stack Overflow answers but also apply it to their specific problem useful. That being said, I recognize your concerns and they are noted.
    – Bella_Blue StaffMod
    Commented Sep 19, 2023 at 12:54
  • 3
    I mean, the fact that there's so many that don't is why i think it's so important to get the wording correct. Just blanket calling everything AI because it's the top trending thing right now makes it sound like the solutions you're putting in place that have existed for 10+ years in one form or another are instead things that were invented in the past 8 months. I'd prefer if the two parts of the "OverflowAI Search" were split into two separate talking points; the semantic search, being a solution in and of itself, and then the summarization/chatbot side of the feature.
    – Kevin B
    Commented Sep 19, 2023 at 16:09
8

There seems to be a way to opt in to seeing the alpha, but as the feature goes wider, an option for users to opt out of both:

  • ever being shown potentially incorrect summaries powered by guessing engines of any kind (LLMs or otherwise)
  • their content being used by summarization features, to avoid potentially incorrect summaries of their content being served to novice users, with the original poster's name on it (as per 'CC BY-SA')

would be ideal.

An example that springs to mind of the danger here is an answer like this:

Here is how to do X; {SOME COMMENTED CODE THAT CORRECTLY HANDLES BAD INPUTS}.

But doing X is bad practice because security/performance/etc, ideally you would do Y instead. For more on Y, see [this other question].

-UserName

being summarised to:

to do X; {SHORTENED UNSAFE CODE} -UserName

The summarized version is shorter and direct so it could improve the number-of-searches and copy-button-clicks metrics, but perpetuates bad practice, hides useful information, and makes UserName appear reckless.

At present the alternative available to users who strongly object to guessed summarization is to delete content, and given according to the quote included above "we might prioritize questions that are non-deleted", deleted content is only lower ranked and not excluded, some users might be motivated to edit their content to be nonsensical before leaving the site. This could add moderation tasks, cause confusion for novice users, and so damage the StackExchange brand.

7

Is there a way to permanently dismiss the banner asking me to test that feature? It reappears every time I log in (i.e., every day), even though I closed it several times via the "X" button.

3
  • 5
    We checked this internally and found that as far as we know the dismiss button is working as intended. Are you potentially clearing cookies from Stack Overflow? Because we track the dismissal of the banner in one.
    – Bella_Blue StaffMod
    Commented Sep 15, 2023 at 15:33
  • 6
    @Bella_Blue Yes, my browser deletes all cookies each time I close it, that explains the issue. I was a bit confused as IIRC other banners in the past were gone forever once I dismissed them, across sessions. Thank you for checking!
    – janw
    Commented Sep 15, 2023 at 17:45
  • 3
    @janw We identified an issue and deployed a fix. Please let me know if you're still experiencing issues. If you are logging into stackoverflow.com every day you should no longer see it if you've dismissed it once. Apologies for the inconvenience. Thank you!
    – tanj92 Staff
    Commented Sep 19, 2023 at 20:51
2

I gave it a couple of tries so far.

Can we get a free-response box for feedback even if I say "yes" it was helpful? In one response there were three short sections, the last of which was garbage but the first two appeared reasonable.

9
  • 3
    That's a great suggestion! Just created a ticket for the team to look into adding something like that. In the meantime, if you come across a similar situation, feel free to use the "no" + text box to explain any nuance like you just described. I'm comfortable with any summary that contains garbage to go into the "not helpful" pile, even if parts of it were helpful.
    – emmabee Staff
    Commented Sep 27, 2023 at 16:18
  • 1
    Out of curiosity, were you using the conversational search or the search summarization (accessible via the search bar) experience when you got the only partially helpful summary in question?
    – emmabee Staff
    Commented Sep 27, 2023 at 16:26
  • 1
    @EmmaBee I guess the latter? Where does the conversational search live? Commented Sep 27, 2023 at 17:06
  • 2
    If you have access to the alpha, you should see the conversational search entry point located in the left side navigation under the 'LABS' section, it's just called Search. Can you see that?
    – emmabee Staff
    Commented Sep 27, 2023 at 18:16
  • 1
    Assuming you find it. I have another question: what do you think prevented you from discovering conversational search, before I pointed it out? Curious what you think the main issue is, and what improvements you'd suggest to help others locate it on their own.
    – emmabee Staff
    Commented Sep 27, 2023 at 18:43
  • @EmmaBee I don't look at the left side panel much, especially on mobile (being under the hamburger) as I was when I first saw that I had been given access to the test. Under the search bar version, after the first ask, I saw the "Refine your search" box and thought that might continue a LLM-conversation, but got the Ask a question instead. It might be a bit much, but perhaps two options there? Commented Sep 27, 2023 at 20:04
  • 1
    given we have the option to have the side bar hidden by default, anyone using that mode wouldn't find it if they weren't looking for it or don't often use that menu. (i just use keyboard shortcuts)
    – Kevin B
    Commented Sep 27, 2023 at 20:10
  • 1
    Thanks @BenReiniger for the insight about being on mobile. That's useful feedback! Could you clarify one thing: "I saw the "Refine your search" box and thought that might continue a LLM-conversation, but got the Ask a question instead" - I know the call to action on that flow currently says "Ask" (I realize could be confusing), but are you saying that when you clicked that you landed on the "Ask a question" page? Or just that you thought it would trigger the "Ask a question" page? It should trigger the conversation flow, so want to confirm you didn't run into a bug, or just confusing copy.
    – emmabee Staff
    Commented Sep 27, 2023 at 20:45
  • 2
    Thanks @KevinB - you taught a relatively newbie to Stack something new today! Did not know there's an option to hide the left side bar.
    – emmabee Staff
    Commented Sep 27, 2023 at 20:46
-1

The Featured on Meta panel is currently showing me two entries with the term "AI":

  1. This one which explains an alpha release of "AI" powered search.
  2. Temporary policy: Generative AI (e.g., ChatGPT) is banned which contains:

    All use of generative AI (e.g., ChatGPT1 and other LLMs) is banned when posting content on Stack Overflow.

If "AI" is considered an unreliable source of posted content, why is considered useful to provide summarised search results?

2
  • 7
    Heads-up, that's a fallacy of generalisation: large language models for answer generation are too unreliable for their speed, which is what led to this kind of AI being banned. Semantic search mechanisms, also considered AI, are not banned, and the company is working to evaluate their usefulness through testing.
    – E_net4
    Commented Oct 1, 2023 at 14:36
  • @E_net4 thanks for the clarification. Perhaps I should have made this as a comment on the question, rather than as a (non-)answer. Commented Oct 1, 2023 at 14:52
-1

The total number of upvotes is sometimes incorrect, e.g.:

rrec

-5

while the search on SO is keyword based IMO and returns exact matches in most cases (which can be different from what user is actually trying to search),

can you guys consider the search feature of popular software /docs website where they use https://www.algolia.com/ and shows as (powered by algolia). it is very effective and finds alternate similar words too, and accounts for typos. in some cases it perform better than Google or bing as the content is very site specific.

let me know what you guys think.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .