After a bit of trial-and-error, it seems that the user script The Stack Overflow Unofficial Patch (SOUP) is the culprit. Disabling this user script removed the scroll bar.
The page lists under Known bugs for Chrome
- The mse154788 / stats1987 fix in SOUP v1.22+ may sometimes add spurious scroll bars to comments on Chrome. This should hopefully be fixed in SOUP v1.24, but since I can't personally reproduce the bug, and since the underlying cause is uncertain (I suspect a bug in Chrome), it's hard to be sure. If it still happens for you, please let me know!
The used version of SOUP is 1.56.2