44

I'm curious as to the extent of the problem of AI generated answers at Stack Overflow. So I extracted the links to 250 consecutive answers via search (is:answer; tab=newest, pagesize=50 [5 pages total]), which I've listed below, and manually inspected all of them [x2].

These answers span a bit over 4 hours. Importantly, they do not include deleted answers (which I don't have the rep to see), so there may be AI-generated content that was deleted before I looked at it.

So how many problematic answers were there (we would expect the number to be particularly high, since this sample was taken during a moderator strike):

There was also this answer which looks a bit weird.

I just thought I'd share this with the community; I believe it's helpful to get more objective data, especially concerning the extent of the problem. I encourage someone who can see all the deleted answers (maybe this has to be Stack Exchange staff) to repeat this experiment. Yes, it takes time, but it's concrete evidence, and it's actually an interesting exercise to figure out what's different between human and AI answers.


  1. Show UIMenu when single-tapping UIBarButtonItem
  2. Cannot open OpenCL Settings in Cytoscape (not an answer)
  3. i have error to recognize folder in node.js
  4. Forcing CPU/ABI to armeabi-v7a on Android
  5. Jpa @Query Annotation problem with using IN clause with java ArrayList (suspected AI-generated)
  6. Flutter webview throws PlatformException(NSInternalInconsistencyException,InstanceManager does not contain an NSURL with identifier:65540, null, null)
  7. Can you declare multiple variable types for function arguments if you have multiple types to declare? (Golang)
  8. Azure function app billing per app when using a shared plan
  9. How to combine a RepeatListener in Android with a smooth image rotation
  10. How does typescript narrow generics?
  11. How to create datagrid combobox in WinUI 3
  12. Trying to compress pdf file stored in AWS S3 bucket using AWS Lambda with python but it showing Error
  13. Process file uploaded with a fetch post request with request.on in node js
  14. Formatting an input by counting characters until a symbol
  15. Error pinging docker server on "terraform apply"
  16. Splitting Columns in Pandas DataFrame by a Specific String
  17. Cloud Firestore - this document has no data
  18. Splitting Columns in Pandas DataFrame by a Specific String
  19. How to pass template parameter of a function to a lambda in the function
  20. Testing MS Team's Tab with different Organization:MS GraphAPI AADSTS65001: user or administrator has not consented to use the application with<app-id>
  21. How to treat ALL C# 8 nullable reference warnings as errors?
  22. How to have field validation call when submitting a field value (not the whole form) - Flutter
  23. Terraform creating S3 Bucket
  24. The Package Conflicts With An Existing Package By The Same Name Error
  25. Post Function is not giving output
  26. how to use npm package with hugo
  27. Airflow Standalone Cannot use relative path:
  28. How to detect changes on an Angular @Input array?
  29. apend method is not working in python code
  30. Endpoint for Azure model
  31. Cannot import name 'available_if' from 'sklearn.utils.metaestimators'
  32. How can I incorporate cluster-robust standard errors into a randomization test using the 'ritest' function?
  33. Script to remove Python comments/docstrings
  34. i am using multiple nestjs servers and i am using nginx as a api-gateway from the front to the the specific backend server but its not working
  35. Error: Failed to create keystore in Android Studio
  36. How to inherit from std::ostream?
  37. Add-Migration fails on WinUI 3
  38. Spring boot deployed in Lambda throws "statusCode": 502 Gateway timeout"
  39. Post Function is not giving output
  40. Obtain max-width using JS in px when it is defined in other units in CSS
  41. Qt configure warning No backend for low level audio found
  42. Recovering one specific future in a future composition
  43. Problem for check if sql request return a value
  44. How do I assign another owner for my Azure Tenant?
  45. QML module not found (QtLocation)
  46. Previous firebase query code no longer works, after npm audit fix
  47. How to change the order in which parameters are arranged in the body of post to a soap api from an azure portal custom connector (deleted before I could read it)
  48. Adding external CSS in an HTML file
  49. How to change the order in which parameters are arranged in the body of post to a soap api from an azure portal custom connector
  50. Fails to read GMAIL inbox with "Invalid credentials (Failure)" when setting up ImapFlow
  51. How to export a declared external value as a named export?
  52. CSS: How to constantly display the bottom of the screen in a overflow window
  53. "GraphQLError: field not found in type: 'query_root'" after merge schema
  54. Mongodb + Atlas: 'bad auth Authentication failed.', code: 8000,
  55. Expanding a symbolic multivariate polynomial with R for usage in POV-Ray
  56. How to completely uninstall django-taggit from my project?
  57. Navbar hides when page hits top
  58. Angular material form field is not updating when changing values in the component
  59. The relationship between interrupts and process scheduling
  60. error [email protected] when deploying nodejs app to render
  61. Why is there nothing like PgBouncer for Mongo?
  62. How do you import an enum into a different namespace in C++?
  63. How to universally enable scrolling on website that disabled scrolling?
  64. Optimizing Matrix Multiplication in Python
  65. How to have more than one case statement for a column in adf?
  66. ImportError: cannot import name 'Annotated' from 'pydantic.typing' when using inflect library in Python
  67. Cannot exclude DataSourceAutoConfiguration for it is not auto-configuation class
  68. Spring possibility to @Value on a Pojo / object
  69. ASP .NET application is trying to use local paths to open a file when I have changed the code to use HTTP paths
  70. Non-greedy regex extraction from JSON with sed
  71. Python psycopg2 not able to connect to postgres server "psycopg2.OperationalError: FATAL: no pg_hba.conf entry for host, no encryption"
  72. Passing multiple cookies to puppeteer
  73. https://stackoverflow.com/questions/76593157/hi-guys-im-making-a-website-and-i-wanted-it-not-to-look-like-this-when-adapted/76593238?r=SearchResults#76593238
  74. How to override User Defined Variable from JSR223 PreProcessor in Jmetr?
  75. How to copy and paste the last row until a certain date?
  76. Filter JSON Data upto 2nd or 3rd Level
  77. "Begin size 0 is not equal to fixed size 5" Error in JMeter while generating HTML
  78. How to get desktop heap free size?
  79. How to list prometheus alerts in a dashboard in grafana?
  80. Could async methods that change the same value be a concurrency problem in Flutter Web?
  81. How to active a user after creating one with LDAP.JS/LDAP.TS?
  82. How to resolve error in 'flutter.gradle' line 1201 that prevents app execution on devices?
  83. State communication between child and parent in React.js not working
  84. Is there an activity monitor for VS Code extensions?
  85. Fastify Response Validation not working as expected
  86. Playing sound using Python
  87. Selenium: How to keep scrolling the page and stop scrolling when the target url is met?
  88. javax.net.ssl.SSLHandshakeException in Android Studio
  89. How to pass an argument to dc_shell in tcl mode?
  90. Why is my PyTorch code 10x slower than the TensorFlow version?
  91. Send call back to previous fragment with data using navigation graph
  92. Terraform creating S3 Bucket
  93. How to reset a postgresql cursor with psycopg2
  94. How can I pass multiple implementations of a generic (which must also conform to a protocol) into my Model struct?
  95. Is there a way to get lifted tabs in daisyUI to use the full width?
  96. How to detect the movement of the sash of a ttk.Panedwindow?
  97. Why does it say module has no attribute text?
  98. WPF TwoWay Binding Not Working on TextBox
  99. Terraform filebase64 on pfx certificate difference vs azurerm_key_vault_certificate.kvtcert.certificate_data_base64
  100. Why is a FetchContent using a different TAG than the one I specified? How can I resolve this?
  101. Can I restrict users who're authenticated anonymously in firebase?
  102. Why do I get overload error while using std::enable_if
  103. Set Django send_mail backend host with Gmail
  104. How to access Service in MAUI Lifecycle Event
  105. useReducer stockpiles component renders
  106. Can I change the default Markdown font color in VS Code Jupyter Notebooks?
  107. Accessing sharepoint list with token unclear
  108. Send header in HTTP post request in flutter
  109. NextAuth.js - Credentials signIn always returns ok=true when redirect=false
  110. C# - System.Data.SqlClient.SqlException: 'Failed to generate SSPI context.' (kind of weirdly written; maybe copy/pasted from somewhere, but not AI)
  111. Scraping saved comments and posts from Reddit
  112. Android BottomSheet bounce animation
  113. Image is not being upload Django
  114. Android biometric - How to detect biometric is lockout? (suspected AI-generated [user has a pattern])
  115. DIscord bot not detecting any messages
  116. Define functions expression - Clarity Smart contract Stacks
  117. Nothing appears when opening an HTML file with Folium
  118. Google Sign-In Not Working in Next.js with Firebase due to Cross-Origin-Opener-Policy Error
  119. https://stackoverflow.com/questions/76593107/langchain-how-to-get-the-pdf-page-number-when-showing-a-result/76593134?r=SearchResults#76593134 (deleted before I could read it)
  120. React vitejs Unexpected string
  121. --recurse ERROR: Insufficient Permission: Request had insufficient authentication scopes
  122. Encrypting a stream with ChaCha20Poly1305
  123. Why do I get overload error while using std::enable_if
  124. How to access query parameters in Next.js 13.4 with the new app router?
  125. Directional network graph in javascript
  126. Cannot set property closed of #<Readable> which has only a getter
  127. How do I write a SQL query to display course_id, course_name and fee collected from each course?
  128. trouble installing apache.logging.log4j and slf4j on the build.sbt of my Scala Spark project
  129. Is there a way to use the sep="" function with an inputed string
  130. ‘Invalid color name error’ when using multiple color palette as a nested list in shiny selectInput for sf plot function
  131. Get all baklink domains of any particular domain using php (spam)
  132. App Tracking Transparency on Flutter was rejected by Apple
  133. How to fetch elements w/ listitem role using Selenium?
  134. javascript: detect scroll end
  135. Is it possible to accept credit card payments on a static site?
  136. C# How to use ISupportExternalScope with customer ILogger/ILoggerProvider and middleware
  137. Slack bot post mesage to usergroup
  138. Always Unauthorized POST request with JWT
  139. How to get a Python Style Traceback in Ironpython3?
  140. Passing CSV columns to a command in bash
  141. Define Non Fungible Token - Stacks Clarity Smart Contract
  142. How does extern "C" affect linking?
  143. Python in VS Code Not Allowing Implicit Line Continuation?
  144. Vuetify v-select prop type check failure
  145. JSONObject(doc.text()) not working in a particular case (spam)
  146. Dealing with an API with a lot of data
  147. I am getting this Error: Field '_controller' should be initialized because its type 'AnimationController' doesn't allow null. How do I instantiate?
  148. expect script failing on exec command
  149. Dropdown menu in nav is overlapping behind content
  150. Web API dynamic parameters for filtering (possible spam; I'm not sure)
  151. I am going through the chainlink functions tutorial and trying to deploy FunctionsConsumer contract and getting an error from undici
  152. Is there a way to import IntelliJ IDEA theme in Visual Studio?
  153. Running two Streamlit functions at the same time (deleted before I could read it)
  154. How to achieve crawling web and waiting for specific event?
  155. Ant Design - Upload file into memory before posting (not an answer)
  156. reading in tab delimited text file with tidyverse readr - columns not parsing
  157. how i can put two action for a html-python from?(Django)
  158. Properly implement a full-screen background image in SwiftUI
  159. Breaking c++ 20 module dependencies
  160. Why is useEffect render-blocking(paint-blocking) in the tooltip example?
  161. Is there an activity monitor for VS Code extensions?
  162. Loop for writing in every sheet without erasing the previous data - EDIT
  163. Pandas is not aggregating a simple data frame correctly. Is it a bug, or my platform?
  164. TypeORMError: Driver not Connected - NestJs (suspected AI-generated)
  165. Exporting data from C# to Excel version error
  166. Converting grouped records (either DataFrame or SQL table) into a JSON array of objects
  167. How can I make VS Code check for errors when referencing properties of JSON imported into JS?
  168. Can we use data directly from RDS or df as a data source for training job in Sagemaker, rather than pulling it from from s3 or EFS?
  169. Rails Enum '0' is not a valid incomeType
  170. importxml formula google spreadhseets does not display required data
  171. How to enforce minimum delay between events from Combine publisher
  172. Radio button not taking the value as expected
  173. MPAndroidChart - hide missing data
  174. Express, error - connectRedis is not a function - Solved the problem by installing connect-redis version 6 instead of 7
  175. How to highlight text using readline?
  176. Insert userID and Password with selenium
  177. More precise sleep (usleep) in Python?
  178. How to get the status of a fetch correctly?
  179. Spring JPA + Hibernate + Postgres - unable to perform "FOR UPDATE OF" query
  180. Send one email with table (specific columns)
  181. XPath expression fails to return correct elements in Selenium (Java)
  182. Recursive function to calculate 25% of the sum of what is returned before
  183. How can I solve the AttributeError: 'Snake' object has no attribute 'distance'
  184. I'm getting error CS1003. How do I fix it?
  185. Recursive function to calculate 25% of the sum of what is returned before
  186. Clarity Smart contract validate principal calling public function
  187. Multer: "ENOENT: no such file or directory"
  188. Pandas is not aggregating a simple data frame correctly. Is it a bug, or my platform?
  189. How to plot squares using lists in python?
  190. what is the major difference GridView and GridView.builder in flutter
  191. Return type of HashMap put() vs compute()
  192. Why my SysLink control appears as plain text?
  193. What is this error and how to fix it?: ValueError: Invalid value supplied 'WktVersion.WKT2_2019'
  194. In my Express Routes When I use '/' end point in my router, it works, however when I change the endpoint to '/login' it doesn't work
  195. "Empty Subscriber" in Swift Combine?
  196. Return type of HashMap put() vs compute()
  197. What's the role of unification in Coq's core type system?
  198. How does memory alloc work in C? // What's wrong w my code?
  199. Report for getting followups outside of start_date parameter
  200. How to loop this css slider animation? (not an answer)
  201. GitHub Actions - Use `on` to filter by source and target branch for PR
  202. Unable to upadate combo box in tkinter python
  203. expect script failing on exec command
  204. How can I make the suggestions widget appear "automatically" for a completion provider in a VS Code extension
  205. running make is failing from boost::bind with _1 and _2 placeholders...?
  206. bridge-nf-call-iptables and bridge-nf-call-ip6tables disabled on WSL2
  207. Nextjs 13 Api Route remains static when calling with cache:no-store
  208. Radio button not taking the value as expected
  209. CORS: "No 'Access-Control-Allow-Origin' header is present on the requested resource.", but pre-flight data shows that it is
  210. How to have field validation call when submitting a field value (not the whole form) - Flutter
  211. The problem of misdiagnosing the max() function in returning the largest feature value of nodes
  212. Enable findAll function to handle > 3 digit currency
  213. https://stackoverflow.com/questions/41225711/wsl-run-linux-from-windows-without-spawning-a-cmd-window/76592901?r=SearchResults#76592901
  214. https://stackoverflow.com/questions/68411057/how-to-run-storage-link-command-on-forge/76592899?r=SearchResults#76592899
  215. How does memory alloc work in C? // What's wrong w my code?
  216. https://stackoverflow.com/questions/76534289/adding-javascript-hotkeys-to-date-controls-for-increased-efficiency-entering-dat/76592896?r=SearchResults#76592896
  217. https://stackoverflow.com/questions/76592808/how-to-give-inside-padding-to-a-text-in-swift-ui/76592895?r=SearchResults#76592895
  218. https://stackoverflow.com/questions/2342324/how-can-i-connect-to-sap-using-sapnwrfc-library/76592894?r=SearchResults#76592894 (not an answer)
  219. https://stackoverflow.com/questions/76592265/why-wont-my-vba-code-loop-through-subfolders-correctly/76592891?r=SearchResults#76592891
  220. https://stackoverflow.com/questions/76592830/how-to-remove-a-limited-number-of-characters-before-a-in-a-string/76592889?r=SearchResults#76592889
  221. https://stackoverflow.com/questions/76592858/how-to-perform-a-series-of-multiplication-with-one-of-the-multiplier-as-the-prod/76592887?r=SearchResults#76592887
  222. https://stackoverflow.com/questions/76582892/using-a-udf-based-on-a-group-of-keys-in-order-to-build-a-new-column-using-spark/76592886?r=SearchResults#76592886
  223. https://stackoverflow.com/questions/76561104/unity-error-could-not-connect-pins-renderstream-only-on-my-logitech-came/76592885?r=SearchResults#76592885 (not an answer)
  224. https://stackoverflow.com/questions/76461561/how-do-i-download-whl-files-more-efficiently/76592884?r=SearchResults#76592884
  225. https://stackoverflow.com/questions/76337229/loading-a-large-csv-file-into-oracle-table-using-peoplecode/76592883?r=SearchResults#76592883
  226. https://stackoverflow.com/questions/76135179/firebase-auth-responds-with-getrecaptchaconfig-is-not-a-function-during-signin/76592882?r=SearchResults#76592882
  227. https://stackoverflow.com/questions/67020476/cannot-connect-to-to-fastapi-with-websocket-in-flutter-403-forbidden-code-100/76592881?r=SearchResults#76592881
  228. https://stackoverflow.com/questions/75865840/clarity-smart-contract-function-loop-if-error/76592880?r=SearchResults#76592880
  229. https://stackoverflow.com/questions/76592397/how-to-regexp-replace-the-first-occurrence-of-decimal-if-theres-more-than-one-i/76592879?r=SearchResults#76592879
  230. https://stackoverflow.com/questions/31461902/is-it-possible-to-extend-a-default-method-implementation-of-a-trait-in-a-struct/76592877?r=SearchResults#76592877
  231. https://stackoverflow.com/questions/76592055/adding-source-urls-in-html-requests/76592876?r=SearchResults#76592876
  232. https://stackoverflow.com/questions/76300671/android-getting-error-namespace-not-specified/76592875?r=SearchResults#76592875
  233. https://stackoverflow.com/questions/76161464/apple-pay-with-stripe-on-web-error-while-domain-verification-from-stripe-apple/76592874?r=SearchResults#76592874
  234. https://stackoverflow.com/questions/73782320/onbackinvokedcallback-is-not-enabled-for-the-application-in-set-androidenableo/76592871?r=SearchResults#76592871
  235. https://stackoverflow.com/questions/76592755/how-to-get-each-sheet-url-and-set-it-in-range-a1-automatically-using-googlesheet/76592870?r=SearchResults#76592870
  236. https://stackoverflow.com/questions/76592800/how-can-i-troubleshoot-react-issues-easier/76592867?r=SearchResults#76592867
  237. https://stackoverflow.com/questions/76592689/sum-and-total-per-row-for-calculated-column-in-power-bi-dax/76592866?r=SearchResults#76592866
  238. https://stackoverflow.com/questions/76589593/cant-match-statusbar-and-view-background-color/76592865?r=SearchResults#76592865
  239. https://stackoverflow.com/questions/76591358/trouble-writing-real-time-segment-events-to-amazon-personalize-using-putevents/76592862?r=SearchResults#76592862
  240. https://stackoverflow.com/questions/76592800/how-can-i-troubleshoot-react-issues-easier/76592855?r=SearchResults#76592855
  241. https://stackoverflow.com/questions/76592803/aws-route-53-wont-validate-my-dns-certificate/76592854?r=SearchResults#76592854
  242. https://stackoverflow.com/questions/76592503/conda-update-fails-due-to-python-version-in-base-environment/76592852?r=SearchResults#76592852
  243. https://stackoverflow.com/questions/76592830/how-to-remove-a-limited-number-of-characters-before-a-in-a-string/76592851?r=SearchResults#76592851
  244. https://stackoverflow.com/questions/76592816/uncaught-typeerror-cannot-read-properties-of-undefined-reading-close/76592846?r=SearchResults#76592846
  245. https://stackoverflow.com/questions/76592830/how-to-remove-a-limited-number-of-characters-before-a-in-a-string/76592845?r=SearchResults#76592845
  246. https://stackoverflow.com/questions/76591822/microsoft-connected-cache-cannot-connect-to-and-perform-tls-handshake-with-iothu/76592840?r=SearchResults#76592840
  247. https://stackoverflow.com/questions/76589922/typegraphql-add-fieldresolver-for-an-array-of-object/76592836?r=SearchResults#76592836
  248. https://stackoverflow.com/questions/3071067/php-how-to-catch-a-division-by-zero/76592833?r=SearchResults#76592833
  249. https://stackoverflow.com/questions/76551297/i-already-installed-32-bit-python-3-9-in-win-10-i-downloaded-it-from-the-python/76592832?r=SearchResults#76592832
  250. https://stackoverflow.com/questions/76592732/group-by-week-date-when-start-of-week-is-monday/76592831?r=SearchResults#76592831
30
  • 6
    The "2 x deleted before I could read it" look very much like AI answers, using distinctive structuring formulations. Judging by the comments, they also failed to answer the question. Jul 1 at 7:25
  • 4
    Now I'm thinking I should have just started https://stackoverflow.com/a/76593376 and decreased the number one by one... Jul 1 at 7:40
  • 12
    #5 is almost certainly ChatGPT-generated. Some signals are phrases like "The error you're encountering", the verboseness, and the absence of missing articles (which is close to 100% for that area; yes, it is controversial, but it is the reality). Jul 1 at 7:50
  • 8
    Just because something looks like ChatGPT doesnt mean it is.. overzealous anti-AI sentiment is running rampant
    – prusswan
    Jul 1 at 7:59
  • 17
    @prusswan Indeed, as you may have noticed people are already taking care here to avoid claiming posts are AI generated when they just have suspicions. Jul 1 at 8:16
  • 41
    @prusswan Indeed, the barrage of direct or thinly veiled accusations against curators is also driving away lots of people. Jul 1 at 8:27
  • 4
    #5 received a 2nd 'Chat-GPT' Answer... + Another 'ChatGPT' Answer by the same User, 1h later... (The 1st Answer (to #5) is also definitely 'ChatGPT'-generated...)
    – chivracq
    Jul 1 at 11:41
  • 62
    @prusswan: Show me an answer that was flagged as chatGPT generated, but actually wasn't. This suggestion that answers are getting identified as chatGPT but really aren't seems bogus. Also, the objection to AI-generated answers does not come from an anti-AI brigade but rather from the folks who make a real effort to curate the site. Jul 1 at 11:45
  • 1
    Judging from this "snapshot", Stack Overflow is still far more signal than noise. Jul 1 at 12:29
  • 12
    @RebeccaJ.Stones "Judging from this "snapshot", Stack Overflow is still far more signal than noise." I agree--the LLM answers here are a small percentage, and I think the overall quality is reasonably good (still). But if you imagine Stack Overflow as curating a Wikipedia-style, high-quality, verified programming resource, it'd be a potentially serious problem if 1 out of every 25 Wikipedia articles you visited was AI generated (and probably inaccurate, based on what we see here), no? I guess we can each decide what threshold is tolerable. For me, it's 0% since I can just ask the LLM directly.
    – ggorlen
    Jul 2 at 0:26
  • 24
    @prusswan: You are confused, the people that you are suggesting should leave actually built this site and gave it value. They should stay and keep fighting the good fight. Also, I noticed you neglected to find a single incorrectly evaluated chatGPT answer, meaning you don't know of any. Jul 2 at 0:48
  • 11
    "Judging from this "snapshot", Stack Overflow is still far more signal than noise." there are AI-generated answers, but there are also just plainly low-quality answers. Those may overlap, but the long-standing quality problem is not only about AI...
    – Andrew T.
    Jul 2 at 4:49
  • 4
    @RebeccaJ.Stones When you have a moment, can you update the "3 x suspected AI" segment? We have at least 7 deleted and virtually uncontestable GPT answers at this point, with 4 more suspected. Thanks.
    – ggorlen
    Jul 2 at 13:56
  • 5
    Some objectives on methodology: 1. The sample size is too small to represent Stack Overflow as a whole, 2. The 4-hour interval introduces a geographical bias. 3. The judgement isn't objective (i.e. based on our own limited experience) so the result can't be objective. Although it must have taken a considerable amount of time to write this question, I can't but doubt how useful this operation is. We won't ever judge all posts, let alone come to an agreed verdict on them. So what's left is a public opinionated discussion on a couple of arbitrary cases. I don't think we gain anything from that. Jul 2 at 20:45
  • 2
    Note that even with the strike, problematic content still gets deleted. So checking for problematic content without the ability to include/view deleted posts is... problematic. A shorter sample window but sampled more often may give more accurate results. Jul 3 at 13:56

1 Answer 1

20

Here's a potential GPT user who is responsible for two of the posts in this batch: https://stackoverflow.com/users/7066658/ali

But most of their other recent answers look GPT-made.


Other virtually certain LLM answers from the batch, all deleted:

Suspicious-looking answers that I'm less certain are AI generated, but may well be:


I didn't count the number of non-LLM spam posts, or the number of low-quality posts in general, or the ratio of correct-to-incorrect answers (this is hard to say) but I would estimate that LLM answers roughly equal the number of spam posts from this batch, about 10 out of 250. Of those LLM answers, maybe two or three are correct (I can improve the guesstimate after SME weigh-in).

I doubt this exercise will impact the debate much either way--both sides can probably see what they want in it. It's hard to draw conclusions from such a small sample, especially since deleted posts weren't considered and random factors like time of day and weekend vs weekday (among other things) weren't accounted for. I took a different angle in this SE meta post which breaks down all of the LLM flags I've raised. The experience I share there matches this in that the obvious LLM answers are low-quality.

While the numbers of LLM posts here may appear low, I don't see any reason not to delete these LLM answers, as most seem useless. The moderation activity that's occurred on the LLM answers in this batch seems to be a clear improvement for the site, just as much as the non-controversial moderation activity that's occurred on the non-LLM spam answers in the batch.

30
  • 3
    stackoverflow.com/questions/76590374/… <-- This one looks like a generated answer as well. It has that ChatGPT feel, but I can't pin-point exactly how. Jul 1 at 19:57
  • 2
    @IsmaelMiguel I don't see that one in the batch posted by OP, so I'm not including it, although it is from the likely-LLM spammer I've identified and does look like ChatGPT (you can see it in the screenshot).
    – ggorlen
    Jul 1 at 20:01
  • 4
    I'd argue this [the second "suspicious-looking answer"] is definitely not AI (it has a punctuation error ("Its"), a capitalization error ("In"), and a typo ("doing do"), and a link). Jul 1 at 23:04
  • 1
    @RebeccaJ.Stones Agreed and removed, thanks. On closer inspection, I'm also removing this answer because the code mixes var and const and uses "null", which I don't think a GPT would do.
    – ggorlen
    Jul 1 at 23:26
  • 2
    @IsmaelMiguel Good call, but the answer also has grammatical problems that an LLM would not emit (has run-on sentences and missing articles like the), so it's definitely not LLM on further inspection. (I had limited time going through the 250, so I only looked at each post for a few seconds).
    – ggorlen
    Jul 1 at 23:34
  • 1
    "I doubt this exercise will impact the debate much either way" - especially this. This is a snapshot of Saturday morning 00:00-05:00 GMT. What do 10 GPT posts prove? 20? 50?
    – CodeCaster
    Jul 2 at 17:53
  • 6
    @CodeCaster it proves the claim that there's less than 10 a day absurd.
    – Kevin B
    Jul 3 at 1:36
  • 1
    @philipxy It's probably GPT, given that pretty much all of the other recent answers posted by that user are almost certainly GPT, but I'm just erring on the side of caution. It could be a GPT answer that is mostly code--I see those from time to time, or maybe the answerer removed some of the text to make it less obvious. There's just not much there to work from. I'll try to clarify my comment.
    – ggorlen
    Jul 3 at 3:13
  • 1
  • 1
    eh, i'd put that one in column not gpt. if there's no history of other gpt posts, and the only thing we have to go on is perfect english and it's a well written/correct answer, no blatant GPTisms, there's no reason to throw that answer into the "maybe should be actioned" group.
    – Kevin B
    Jul 3 at 14:54
  • 1
    @KevinB I mean that one random sample collected over four hours on a weekend day does little to demonstrate the problem or prove lack thereof.
    – CodeCaster
    Jul 3 at 15:36
  • 2
    @KevinB I am not contesting that the metric is wrong, refer to my earlier posted answer. My statistcs are rusty, but I think that from such a small, non-representative sample you cannot extrapolate meaningful conclusions. That's all I'm saying. It doesn't seem like this question asks for discussion. It's unclear to me why this question was posted.
    – CodeCaster
    Jul 3 at 15:44
  • 1
    @CodeCaster "It's unclear to me why this question was posted", likewise, especially since OP isn't even a Stack Overflow user, as far as I can tell, and can't see the deleted answers we've been discussing. OP has a history of pro-LLM posts on meta SE, so I suspect it's part of the agenda to discredit the strike by showing how rare LLM answers are, or how good they are when they are posted. I'm unconvinced by the demonstration. If anything, this only solidifies my perception that LLM answers are being spammed without regard for accuracy and are posing a serious qualitative threat to SO.
    – ggorlen
    Jul 3 at 15:52
  • 2
    @ggorlen I'd caution you in making statements like that– asserting an "agenda to discredit the strike" gets pretty darn close to witch-hunt territory IMO, and I don't think it's very helpful or productive. We don't need to try and separate folks into camps "for" and "against" the strike unless they themselves draw that line in the sand; labeling folks doesn't help the strike. You've also picked someone who is very much public about how nuanced they feel about the strike, so I don't think it makes much sense trying to divine what their agenda is.
    – zcoop98
    Jul 3 at 18:39
  • 1
    @ggorlen You also commented under their stance on that very post, so.... I doubly don't know where the agenda comment is coming from. I don't think the strike is benefitted by folks disparaging the nuance of others' stances on the matter.
    – zcoop98
    Jul 3 at 18:42

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .