82

A number of the Survey 2017 results are displayed as bar graphs, eg:

Job Priorities

potential job factors bar graph

However, there's a problem. The length of the bars are misleading, given the fine print:

the fine print

If the lowest score that could be given was 1, that should be the 'zero' of the graph. This problem really stands out with the 'attitudes to SO' graph:

attitudes to SO bar graph

The bottom bar should in fact be about half the length it is, given that responses below 1 weren't actually possible to give. It should in fact look like this:

better SO bar graph

which makes the overall positivity to SO much more apparent.

2
  • I think there's no issue with the Bar graphs, you are talking with respect to the Bar rated 4.3 But in actual 1.9 is correctly rated if you see in accordance to 5.0 If there would have been a bar greater than 4.3, you could see that 1.9 is correct in length. So assume a 5.0 bar rating and see the difference.
    – Kamesh
    Commented Mar 24, 2017 at 12:31
  • 16
    Likewise, the bar graphs should end at 5, and have a scale showing where they end. Otherwise, a question that got a maximum of, say, 3.7 make it look like everything is at the top; that fine print saying the actual number or that it's a 1-5 scale won't affect things for people glancing through
    – Nic
    Commented Mar 25, 2017 at 11:36

2 Answers 2

16

I agree: bar graphs without scale are bad graphs.

Furthermore:

  • graphs where users can choose between discrete values should display median values;

  • graphs where users can choose between continuous values should display mean values.

A satisfaction level of 4.23 makes no sense because no user could choose that value. The median shows the value which is most "in between" the votes of users. The average suffers from false precision.

On the other hand, if you are measuring people's heights, which is a continuous value, it makes sense to take an average, since all values are possible.

http://www.purplemath.com/modules/meanmode.htm

6
  • 2
    A mean of 4.23 makes as much sense as saying the average family has 2.5 children.
    – Hong Ooi
    Commented Mar 25, 2017 at 16:56
  • 15
    No, mean has value even in discrete cases. The precision is not false. Mean possibly doesn't mean what you think it means, but 1.8, 2.0, 2.5 and 2.9 average children all mean different things. And this problem has almost nothing to do with scale; the bar graphs have their values displayed. In short, I disagree with everything in this answer except "I agree"; the original question makes a great observation. Commented Mar 25, 2017 at 17:17
  • @Yakk say we have 3 choices "happy", "meh" and "sad". Is it better to average 10 "happy" answers and 5 "sad" answers to a "meh" average? Or does a median value of "happy" represent this better? In my opinion, the latter.
    – Sklivvz
    Commented Mar 25, 2017 at 17:26
  • 8
    @Sklivvz: Perhaps, but that has little to do with discreteness vs. continuity. Suppose we allow people to choose any real number in [0.0, 10.0] to represent their level of happiness. If ten people choose values of 9.9 or higher, and five people choose values of 0.1 or lower, would you suddenly say that it does make sense to compute the mean (around 6.7) instead of the median (above 9.9)? I think the real problem is that a bimodal or multimodal distribution can't be meaningfully captured with a single statistic.
    – ruakh
    Commented Mar 25, 2017 at 20:20
  • @ruakh while the last point is true, the discreteness vs. continuity introduces another problem. If we allow people to choose on a continuous scale, the hidden assumption is that any value in the scale is understandable by them, and therefore a result on a continuous scale makes sense. If we allow people to choose on a discrete scale, the hidden assumption is that values in between the discrete steps are not meaningful to people, so reporting a continuous result does not make sense.
    – Sklivvz
    Commented Mar 25, 2017 at 23:09
  • Furthermore, what's the point of making people choose between discrete values if the value we are measuring is continuous, and we are reporting it at the end on a continuous scale? If a continuous result is appropriate, then asking for a discrete value introduces approximation error without reason. Either a value of "3.23" is meaningful to users, and therefore they should be allowed to choose it, or it isn't in which case the reporting needs to be discrete.
    – Sklivvz
    Commented Mar 25, 2017 at 23:11
9

I agree with the OP, the presentation is seriously flawed.

brief explanation first:

  • The measured variable in each case is neither discrete nor continuous -- it's ordinal; yet ordinal variables have characteristics of both discrete and continuous.
  • Ordinal variables are like discrete variables in that they are not numeric (the choice of values {1,2,3,4,5} is pure artifact; {a,b,C,d,e} is an equally valid way to encode an ordinal variable of cardinality 5.
  • But ordinal variables encode information that discrete variables do not -- ranking, in particular; perhaps analogous to transitivity: if 1 < 2 and 2 < 3, then 1 < 3; in this sense they are like continuous variables.
  • Rank isn't a necessary condition for a discrete variable -- eg, for the variable Sex which has two values Male and Female, Male > Female or Female > Male has no useful meaning.
  • Finally, ordinal and discrete are similar in that even simple arithmetic is undefined -- eg, the average of New York and California isn't Nebraska.

But why then, in the cases the OP raised, when the person who created the survey plots, attempted to calculate the mean from each set of responses, was a valid result returned (eg, 4.15)?

Because a human chose to encode the values comprising the variable as sequential integers (1 - 5); this mapping -- i.e., {"strongly disagree" -> 1, "strongly agree" -> 5, ...} -- is arbitrary. In other words, A through E would have sufficed for the mapping just as well as the symbols 1 through 5

But because integers were chosen for the mapping, and integers have a bunch of well-defined arithmetic operations, such as average, minimum, maximum, etc., it's tempting to go to just hog wild.

In this instance, calculating the mean of the mapped responses is full of assumptions, not the least of which is that the values are separated by an equal interval (another is usually referred to as paired-comparison intransitivity, which if I had correctly learned the first time around, I would never have had the privilege of taking Advanced Probability Theory a second time).


Anyway, here's how i'd do it:

choose a representation that removes the dependency on all of these assumptions by avoiding the mapping step altogether (other than as a plot-annotation convenience, eg, "1" means "Strongly Disagree"). This way, i have no contrived symbols that look like integers which trick me into thinking they are integers that I then mistakenly do arithmetic on.

Display the results of each question like so,

enter image description here

a standardized distribution of response counts

Won't a separate bar plot for each question make the viewport too cluttered?

If done right, it'll look very clean, what's more, this is a fairly routine technique to plot data of this sort. A plotting library based on Tufte sparklines would do the job nicely. What's more, displaying these in a single column (e. g., immediately following the question) and standardizing the counts, would allow quick visual comparison among question responses by the reader.

1
  • Yeah, I agree with you that would work wonderfully.
    – Sklivvz
    Commented Mar 26, 2017 at 12:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .