Can we see some (more) data? Is there more?
In your linked answer, you say it was a "data-led re-evaluation". You also imply that Meta users might care more about this data, but still don't seem to show much of the data.
You link to this answer to justify that the 5-rep-per-question-upvote decision wasn't haven't the desired effect, but this only seems to show that our question quality isn't that great currently, and there's been a decline over time (which could've been caused by any combination of so many factors). It doesn't in any way show the direct effect of the decision. It might be that the decision helped significantly and the site would've been in a much worse state without it. Also, one of the primary metrics used to define a bad question is it having a negative score - this is definitely questionable to use for evaluating this decision, since upvotes counting more would arguably not make much of a difference to questions that aren't upvoted in the first place. Askers of such questions might be motivated to try to avoid downvotes or bans instead of getting upvotes. I would expect it to make more of a difference to users that consistently get an upvote here or there to their questions, but never too many. They'd be much more motivated to try to ask better questions to increase the number of upvotes they get.
It also seems that you're aware we have too many low-quality questions, and this decision was to try to address that. Yet it seems like you were pretty rash in irrevocably reversing it and thus making possibly by far the biggest change to the site in a long time. I say "irrevocably" because of course you could just undo it, but that would greatly upset the people you were trying to appease in the first place by reverting it. This is especially true because it would take some time to get meaningful data that might tell you reverting it was a terrible idea, at which point everyone would be very much used to the new way things are.
I would agree that asking a good question is (or at least can be) hard, but I would need some evidence to believe people asking good questions care much about reputation, especially compared to people asking low-quality questions. I would also be sceptical that reputation hunters would focus on questions instead of answers (and whether they'd even be able to ask a good question), since it's generally the roll of the dice whether a question will be seen by anyone, and there's certainly an element of luck in terms of how it will be received. Not to mention that you actually need to have a decent question before you can ask it (a question that's just another duplicate is not decent, IMO). Although users might be more inclined to ask unusual questions to try to get to HNQ, which can seemingly be done fairly easily in some cases. You may consider this a good or bad thing. I personally consider such questions to be the scourge of the few sites I frequent, thus an increase in such questions would be quite bad, but maybe that's just me.
What would I like to see (or have seen)?
Analysis of what happened following the initial decision.
Consider:
- Total number of questions
- Total number of answers
- Average number of questions and answers per user
- Scores for questions and answers (the total, average per post and average per user)
- Other measures of quality (views, closure, deletion)
- Increases of average post score and quality for individual users over time (i.e. are subsequent questions asked by users better?)
Consider both what happened immediately, and over the next year or so. Include a detailed analysis of the data. Identify and show points in time where other changes occurred which might also affect this data.
Of course this happened during the early days, so it might not be that meaningful and all of that data might not even exist.
Although I hope to at least see the above analysis of the effect of reverting the decision a few months or a year from now.
Some user study or survey to determine what motivates people to (a) ask questions and (b) take time to improve their question before asking.
My primary motivators to ask or improve a question are getting an answer, extremely high standards, contributing to the body of knowledge that is the internet and what others think of me. Reputation is so far down in my list of concerns, it's not even funny. For answers reputation might be a more significant factor. Although that's just me, and I've just asked 6 or so questions on Stack Overflow. It's not that I know everything, it's just that I can almost always find the answer somewhere online.
Trying to check the correlation between score and quality, instead of using score as a measure of quality.
I would focus on those around 0 score (say +3 to -3), as highly upvoted or highly downvoted posts have a much higher chance to be correlated with the other measures of quality.
Other ways to measure quality would include whether it's closed and deleted and the number of views it gets (especially after the initial spike and those coming from search engines). Although, unfortunately, all of these are at least somewhat linked to score.
This would show you whether people are upvoting low quality posts, which would show that this change is likely to encourage more such posts, i.e. do harm.
Check upvotes on low-quality posts.
This is similar to the above point, but instead of looking at score, look at just upvotes.
Here you can also include score as a measure of quality, and you can also include highly downvoted posts.
This would be a stronger signal of whether people are upvoting "low quality" posts.
Analyse other (non-SE) sites.
I wouldn't quite know whether this is ethical, nor what you'd be willing to share publicly about any such analysis, but how other sites are getting by using certain models (at least based on what can be seen publicly) might be a decent measure of which models work and what one might expect when moving in one direction or the other. Even if there are many other differences between SE and those sites.
Some experiments on smaller sites.
The large number of sites on SE could be a great environment for experimentation.
Rolling this out to one or two smaller sites before doing it network-wide would provide some good data for what effects we can expect, and also make undoing it more viable.
On the other hand, having reputation work differently across different sites on the network might be questionable and confusing. Also, sites might not much appreciate being experimented on.