19

This question is similar to Filter questions by asker's reputation or Ability to ignore questions from low rep users, but does not equate the asker's reputation with the difficulty or level. There are many other questions on meta evoking this phenomenon: trivial questions have taken over.

Note: I speak of difficult or level, and not of quality. A very simple question can nonetheless by of high quality (showing research, good redaction skills, etc...) and a difficult question can be of low quality. I am not interested, here, about quality; it is addressed by other features.


Background:

When Stack Overflow started, the community was somewhat of an elite, and therefore the level of the questions was relatively high. Speaking about the C++ tag, for example, every day would see questions that required digging into the darkest corners of the C++ Standard.

As time went by, and Stack Overflow gained in popularity, the level of questions has dwindled. I believe this is for the better, as it means that the bar of entry is low and therefore we reach (and help) more people.

On the other hand, it also means that as a somewhat experienced developer (in C++), most questions are desperately boring. I am not interested in their answer, and if I were to attempt to actually answer I would probably be overtaken by 2 or 3 other users, so why bother?

And thus I face starvation: difficult questions, to which I could contribute or learn from, are buried in a mass of easy questions and I starve as I attempt to sift through them. And I know that I am not alone (from the echoes on meta).


Feature-request:

It seems that we need a mean (whatever it is) of:

  • ranking the difficulty of a question
  • filtering questions based on this difficulty

I have a couple ideas as to how this could be addressed, and I will propose a straw-man as a starting point to spark the discussion.

Note: it should be noted, by all straw-men, that approaching sparks can prove lethal.


Note: the bounty already help target interesting/difficult questions in general; however setting up a bounty requires reputation already, and the willingness to part with it. I see no reason why the system should penalize people asking difficult questions.

14
  • 3
    I'm thinking a lot about this as well. This trick (as always) is in defining "difficulty".
    – David Fullerton Mod
    Commented May 2, 2014 at 19:41
  • 6
    @DavidFullerton I think the only feasible way to define "difficulty" is to let everyone vote on it.
    – Mysticial
    Commented May 2, 2014 at 19:43
  • 5
    @mysticial The volume is too high for voting -- it has to be somehow algorithmic (even if it's only approximately correct). If people hate paging through bad questions to find the gems, who is going to want to page through bad questions voting on the difficulty level? We'd just be making more work.
    – David Fullerton Mod
    Commented May 2, 2014 at 19:53
  • 5
    @Mysticial - Otherwise, every new user would simply tag their question "expert" so that the experts would see it (like how they use moderator flags to demand that the "experts" answer their question). Even then, there will be arguments about whether or not something is "expert" enough, and I worry that this would create the kind of class system I describe here: meta.stackoverflow.com/a/252810/19679
    – Brad Larson Mod
    Commented May 2, 2014 at 19:54
  • @BradLarson: The very issue I see with a reputation-based system is that there is no reason a new user cannot ask a very difficult question or that an experienced user cannot ask an easy question... in a technology he is just picking up. Commented May 2, 2014 at 19:57
  • 2
    LOL (via flag for mod-expert) @BradLarson, that never crossed my mind as something that happens. Wowwwwww Commented May 2, 2014 at 19:57
  • 1
    @DavidFullerton: it's indeed a concern; I believe though that it could be addressed if we find a mean to have questions bubble up through the difficulty levels. Let is start as "easy" and have the people there throw their hands up and mark it as "medium", etc... Of course, it increases the latency before it reaches "expert" users; but if those experts have abandoned reading the questions this latency is already high... Commented May 2, 2014 at 19:59
  • @DavidFullerton I doubt it will be possible to do this algorithmically based on currently existing information. Certainly the correlation between question upvotes and difficulty is not great. I believe Slashdot has people vote on attributes such as insightful, interesting etc. Why wouldn't collecting multi dimensional ratings work for SO too? Commented May 2, 2014 at 20:47
  • @Martin We don't get anything like enough votes. Nearly 50% of posts have score = 0. We could use some hand labeling to feed a machine learning algorithm, but we can't rely on users ranking every single question (or even just the good ones -- most questions don't get upvoted until they've already been answered)
    – David Fullerton Mod
    Commented May 2, 2014 at 20:51
  • I've posted my thoughts here, since I seem to be going in a different direction than this post: meta.stackoverflow.com/questions/253351/…
    – David Fullerton Mod
    Commented May 2, 2014 at 21:14
  • @DavidFullerton: it's indeed a different axis; of course I would rather not see low-quality questions (let them die), but even questions of a reasonable quality come in at too high a rate that really difficult questions quickly get buried under :x Commented May 3, 2014 at 10:36
  • @BradLarson: It's possible to prevent that everyone-marks-as-expert situation, see my answer.
    – user541686
    Commented Jul 6, 2014 at 20:04
  • 1
    Maybe questions could have an "endorse" button and the question could get a reputation related to the highest reputation of everyone who endorsed it? Commented Mar 8, 2017 at 19:47
  • SF "interesting" queue on the front page is anything but interesting. Maybe 1 in 20 is worth reading for me these days which equals a lot of wasted time and annoyance for me. I often struggle to find anything interesting enough to warrant answering. Mostly the interesting feed contains questions with 0 or less votes going back several hours/days. IMHO interesting is a function of votes, rank of the person who asks the question (i.e. ban idiots already, please), and the votes on the answers (if any). I'm pretty sure even a naive attempt to rank based on that could produce way better ranking. Commented Mar 29, 2017 at 16:40

5 Answers 5

5

I'd like to address this question from another point of view : I often would like to explain why I didn't answer, vote or comment. Actually, in all other cases the time I took to read the question wasn't a waste since I actually did something. So only when I don't answer vote or comment, I feel my impression could be useful for other "helpers". Not just my "+1" to the number of total views.

The list of cases isn't that long (feel free to comment if you have other options to add to the list) :

  • I wasn't able to understand the question (too technical for me)
  • I understood but I don't know the answer
  • I'm able to judge that the question isn't clear (and other users already asked for more details)
  • it's too easy for me, let's leave the answer to less experienced users (not sure about this entry though)

I think it could guide people before picking a question, if they have a statistical indication of "what did people who didn't answer think of this question". It would be independent on reputation (but with a ~20rep threshold maybe ?) and not require many votes to be effective. It could be represented by one point on a small graph (one axe for difficulty and one for clearness).

And then people could have an overview in a glance of :

  • How hard was the question evaluated ?
  • Was it unclear (these votes would be suppressed after an edit by the original poster) ?

N.B. I'm not sure if this 'vote' should be restricted to those who didn't answer/vote/comment, and I excluded these cases since there are already appropriate actions :

  • duplicate
  • "bad" question, no chance to improve it
  • it's already answered correctly
2
  • 5
    The risk with your bullet it's too easy for me, let's leave the answer to less experienced users, is that the quality of the answers will be low too. Often you see low rep users answering other low rep users giving bad advice. The intent may be good, but the answers often aren't. Commented Jul 6, 2014 at 19:16
  • 7
    I'd love an option "Seen this a million times already, but can't be bothered to locate a good duplicate"... It would serve as a hint to the OP that the result can be found by searching.
    – Kerrek SB
    Commented Jul 6, 2014 at 22:02
3

Here's another possible approach. A difficult question is one that remains neither closed nor downvoted nor effectively answered for some time period.

The problem with this approach is defining "effectively answered". Most questions will attract at least one quick but uninformed answer. On the other hand, many properly answered questions have no accepted answer.

I suggest considering a question "effectively answered" if it has either an accepted answer or at least one answer with a positive net vote at least as high as the question's net vote.

1
  • To be honest I see quite a lot of questions ( from beginners mostly) where the answer is provided in comments and agknowledged in comments and the question is never formally answered but just left to disappear. It's not a bad question as such, it's just beginner level / simple.
    – Martin
    Commented Mar 5, 2016 at 19:59
1

Proposal:

Let everyone tag as Easy or Medium (I think it's actually useful for newbies to be able to indicate their questions as such; by default everything would be Medium).

Then let people who have a statistically significant number of upvoted + accepted answers (with higher weight toward accepts rather than upvotes) in a particular tag mark their own questions with that tag as "hard", or (with a higher threshold) as "expert".
Everyone would then be able to change the difficulty of questions belonging to someone who hasn't achieved that high of a rank, but not those of equal rank. (So experts can't change the difficulty of Expert questions, but they can change the difficulty of Hard questions.)

This should work much better than a rep paywall, since it actually rewards expertise.
It should also work better than gold/bronze badges, since those are kind of hard to get and less accurate for people who should arguably still be able to mark question difficulties.

-1

A straw-man proposal...

Ranking the difficulty, accurately

Or exposing the issues in getting this ranking to be accurate.

Whilst we could take a majority vote regarding the difficulty of a question, I doubt the system. Following the Drunning-Kruger effect, I am afraid that difficult questions would soon be judged "easy" by a lot of users (which would not understand the difficulty); or, on the contrary, that a lot of users may tag the questions as "hard" as soon as they do not understand them... even though they would merely be "medium".

Since the level of the users is most likely a pyramid, I would tend toward giving a stronger voice in the ranking process to more experienced users.

However, I would NOT take reputation as a measure of experience. The issue with reputation on a site as diverse as StackOverflow, is that just because I know C++ fairly well, which is where I earned those points, does not mean much as to my competence in Rust or Haskell (FYI: I am n00b).

We could argue that users who managed to answer difficult questions would be best suited; but unfortunately this would not help in jump-starting the system. On the contrary, questions suddenly elected "difficult" in the early days (and which are not) could give unexperienced users the ability to heavily influence what "difficult" means.

On the other hand, tag-badges do represent if not the experience at least the participation of a user in a given area. It's imperfect, much as the reputation a user has would be, but it exists and can be put to use.

The straw-man proposal

I propose to rank questions following the colors of badges:

  • none
  • bronze
  • argent
  • gold

By default, a question should start at none, and users will gradually rank it if necessary.

In order to determine the rank of a question, I would propose a poll-like system: pick what you think is the most suitable rank for this question. Each "rank" thus gets a score, and the current rank is the one with the highest score, rounding down. (*)

Also, in order to give more influence to more experienced users, we leverage tags and badges:

  • the "color" of a would-be answerer is determined depending on his best tag-badge for the list of tags affixed to the question: none, bronze, argent or gold.
  • the number of points a user contribute to the score of a rank depends on its color: 1 for bronze, 4 for argent, 10 for gold.

Users with no tag-badge simply cannot vote (*).

(*) An alternative would be: 1 for none, 10 for bronze, 40 for silver, 100 for gold. Or some other values... that could be fine-tuned.

Note: an issue with the system is that the rank should be re-evaluated whenever the tags are changed (or at least, whenever a tag is removed) as this changes the "color" of the voters.

Filtering

There is the issue of the ivory-tower syndrom; I would suggest that instead of being able to filter "silver-only" or "gold-only" the option would be "none & bronze", "bronze & silver", "silver & gold" where the overlapping between the ranks should allow difficult questions to naturally bubble up.

6
  • 2
    I don't think this can scale. Questions don't even get this many votes, and this would be a completely new action that presumably users would have to be trained on.
    – David Fullerton Mod
    Commented May 2, 2014 at 20:00
  • @DavidFullerton: first of all, thanks for your comment, that straw-man is meant to spark a discussion after all. Indeed scaling is a potential issue, but most questions should not need ranking since, by almost definition, the bulk of questions is trivia. What we need is a system that will allow hard questions to bubble up, and this system does work on this point: after all, the first user to rank the question will immediately determine its rank (by virtue of being the only one) and thus a non-trivial question would quickly appear on bronze/silver/gold lists where "experts" can weigh in. Commented May 2, 2014 at 20:05
  • @DavidFullerton: regarding the training aspect, I may be underestimating it indeed. I would hope that it be possible to design the UI so that this new functionality is obvious (right under the score ?) and that to start with a link could be provided underneath to explain what this new stuff is all about. Also, if we decide to only allow bronze-badge holders (for the question tags) to be able to interact, the population to be trained would be reduced. Commented May 2, 2014 at 20:07
  • I do strongly agree that we need some way to denote difficulty. Not sure if this is the way to go about it though.
    – user541686
    Commented Jul 6, 2014 at 19:47
  • I suppose the issue is how do you coerce users into scoring a post's difficulty? e.g. rep/badges? Commented Aug 9, 2014 at 18:46
  • @AndyHayden: Yes, as always enticing users to make an informed choice (rather than a random click), is always difficult. Commented Aug 10, 2014 at 9:55
-3

I think as a very broad concept this is a very good idea and actually well needed by Stackoverflow due to the sites exponential popularity.

But as others have stated the "ranking" of a question needs valid setting.

So, here's my take on it:

  • When a user (any level) posts a question, they have a 5 level bullet to choose the level of the question, before submitting the question. Example:

Level 1 - beginner

Level 2 - Intermediate

Level 3 - Competent / Professional

Level 4 - Difficult

Level 5 - Expert

Possibly if the user has a reputation below say 50, or 25 (or some value but definitely no more than 50) then the question level must be 1 or 2 (beginner or intermediate).

  • A question can be edited and the "level" of the question can be changed, so that in my area of knowledge, being PHP and MySQL I can easily find qestions I can't be bothered to anwser because they're amateur/beginner level and mark them appropriately, alternatively there are questions about things that are beyond my scope of knowledge and I can adjust their Level appropriately.

  • Over time once introduced this system will be self-leveling as people who edit and maintain posts can mark questions as approximate levels, a question doesn't need to be always in the correct level, in the same way there are a percentage of questions that are answered and marked as solved which do not always have thetechnically correct tags.

Many users are self aware and state in their questions that they are not "professionals" at their chosen programming topic and so will mark their levels as approprate.

  • The level relates to the question NOT to the user.

An important factor to take account of is that this concept is a guide an is not going to be 100% pure correct either at point of release or the future thereafter, but it will serve a very good method of general question categorisation and clearing questions that a user such as the OP dosn't want to care about.

So that people who are pro's at, say, PHP can easily skip out and remove listings for questions marked as Level 1 or Level 2 by adding a checkbox ability on listings to list questions only of levels the user feels comfortable being able to view, read and answer. Users such as the OP can set to view questions of level 4 or Level 5 in their chosen field and I can view questions at Level 2, Level 3 or Level 4.


A simpler concept would be that the question or the user can be self- tagged as Professional or amateur and so differentiating between the two main cores of StackOverflow questions. this tag can of course be edited and changed by edits to the question, as appropriate.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .