7

By ""invalid" JSON" I mean these (or a part thereof):

(Example partly taken from json5.org)

{
  // Single line comment
  # This too
  /*
    Block
    comments
  */
  unquoted: 'and you can quote me on that',
  singleQuotes: 'I can use "double quotes" here',
  lineBreaks: "Look, Mom! \
No \\n's!",
  tripleQuotes: '''
    for the win!
  ''',
  hexadecimal: 0xdec13a7,
  leadingDecimalPoint: .8675309, andTrailing: 8675309.,
  positiveSign: +1,
  infinity: -Infinity, and: NaN,
  trailing: {comma: 'in objects',}, andIn: ['arrays',],
  orSomethingCrazier: like these
  // => "like these"
  noEscapeNeeded: \w+
  // => "\\w+"
  multipleCommas: ["first",,,, "second", {"third": 3,,,, secondKeyOfThird: 4}]
  noCommaBetweenEntries: [and 'unquoted-strings']  // => ["and", "unquoted-strings"]
}

There are a lot of questions asking for a way to "fix" text like this with regex. An user at a recent PHP question found 5 of them (all about PHP):

Some other questions I found by googling:

The list goes on and on.

Most of the answers I read are regex-based; others suggest using superclasses of JSON which support some of the above syntax (I have yet to find a language that works with all of them; not even JS itself): YAML, JSON5, Relaxed JSON, HJSON, etc.

I think a canonical question is needed where readers can find language-oriented answers on how to use implementations of these data formats. Has there already been one? If there is no such question, should we create one?

13
  • 23
    Not sure what a cnonical would be other than "either fix it so you have actual JSON or figure out what (probably ad-hoc) format you have and try to deal with that". I don't believe there is one true and useful way to handle every possible not-JSON that people try to "fix" or "parse" and expect there is some simple magic that does it. There is Parsing "relaxed" JSON without eval but again it's only one very specific type of not-JSON.
    – VLAZ
    Commented Oct 20, 2023 at 21:43
  • 16
    I mean... most of the time the answer is don't.
    – Kevin B
    Commented Oct 20, 2023 at 21:43
  • 1
    @VLAZ Out of those presented in the question, the most common ones are unquoted keys, // and /**/ comments, single-quoted strings, hexadecimal(/binary?) numbers and trailing commas. Would a question consisting of those alone can somewhat suffice as a valid one?
    – InSync
    Commented Oct 20, 2023 at 21:54
  • 5
    For code that can use libraries written in Java, Gson (disclosure: created by my employer) can parse a number of these when set to lenient mode. I think it's a legit question to ask, especially when there is an answer that would parse it.
    – Ryan M Mod
    Commented Oct 20, 2023 at 22:16
  • 1
    I'm about to say that almost all entries you mention are covered by JSON5 "standard" (no support for missing commas, hash comments and triple quotes). It has wide library support (e.g. json5 in python, json5 for TypeScript and Javascript, json5 for Rust, golang, and many others)
    – STerliakov
    Commented Oct 21, 2023 at 21:26
  • What about duplicate keys? Commented Oct 23, 2023 at 18:50
  • @VictorStafusa-BozoNaCadeia I don't know of any JSON-derived format that supports duplicate keys. Do you mean something like TOML's array of tables syntax?
    – InSync
    Commented Oct 23, 2023 at 20:52
  • @InSync I already saw some broken JSONs that featured duplicated keys for objects and the JSON parser which consumed it didn't check for duplicates. E.g. {"some_key": "a", "some_key": "b", "some_key": "c"} - The most common result was that only the last key prevailed and the rest was silently dropped. The correct, of course, would be to throw an error and reject the input instead. Commented Oct 23, 2023 at 21:26
  • @VictorStafusa-BozoNaCadeia the official JSON spec has this to say: "The JSON syntax does not impose any restrictions on the strings used as names, does not require that name strings be unique, and does not assign any significance to the ordering of name/value pairs. These are all semantic considerations that may be defined by JSON processors or in specifications defining specific uses of JSON for data interchange." - chapter 6, page 3 of the document / page 11 of the PDF.
    – VLAZ
    Commented Oct 24, 2023 at 10:55
  • Why would we need this? What problem does it solve? No argument is provided for it, just "I think we need it". It is going to be yet another list of things. You can't even tag it properly, I wouldn't be surprised it would end up with five pages of answers where several answers target the same language but different tools. And eventually duplicate answers will keep rolling in because who is going to check 5 pages of answers to see if theirs is not already in it.
    – Gimby
    Commented Oct 24, 2023 at 11:20
  • @VLAZ "does not require that name strings be unique" - well, then a lot of actual JSON parsers out there are simply broken and have been built on the assumption of just the opposite of that. Some of them emit errors when some name isn't unique. Some of them silently drop repeated keys or keep them somewhere in memory but in a way that it is very cumbersome or even impossible to read. Can't recall of any single parser that offers an API that can access all the duplicate values in a high-level, simple, elegant and straight-forward way (i.e. not depending on very ugly hacks). Commented Oct 24, 2023 at 11:38
  • @VictorStafusa-BozoNaCadeia that's what the JSON data interchange syntax standard says. As far as it is concerned, you can have duplicate keys. It only defines allowable grammar and constructs, not how to interpret those. How the parsers handle this is not defined in ECMA-404. RFC 8259 has notes on interoperability and describes some different ways parsing libraries have differed in handling duplicate or ordered properties. See section 4. Overall, there is no "the correct" way. Throwing an error has not been mandatory.
    – VLAZ
    Commented Oct 24, 2023 at 12:50
  • @Gimby Please post that as an answer for better visibility.
    – InSync
    Commented Oct 24, 2023 at 17:09

2 Answers 2

30

There can be no single canonical.

To understand why, let's first examine what is JSON: it stands for JavaScript Object Notation is a textual data interchange format. It has very simple grammar which is subset of JavaScript Objects (hence the name). The more technical definition can be found here but in short the possible types it handles are strings, numbers, true, false, null, as well as objects and arrays which can contain any other type. Whitespace between tokens is ignored but often used when presented for readability. For example:

{ 
    "a": "hello",
    "b": 42,
    "c": true,
    "d": null,
    "e": ["world", 3.14, false],
    "f": { "foo": "bar" }
}

Although the top level of JSON value can also be any of the simple types - strings ("hello" is valid JSON), numbers (42 is valid JSON), or true, false, null.

With that said, what is invalid JSON. Simply put: anything that is not valid as per the rules that describe JSON. Examples include:

  • unquoted keys: { foo: "bar" }
  • no quotes at all: { foo: bar }
  • using single quotes for strings 'hello'
  • using single quotes for keys: { 'foo': "bar" }
  • trailing commas: { "foo": "bar", } or [ 1, 2, 3, ]
  • double commas: [1, , 2]
  • semicolon used outside values: "hello"; or { "foo": "hello" ; "bar": "world" } or [1;2;3]
  • objects containing no keys: { {"foo": "hello"}, {"bar": "world"} }
  • comments: { /* this is a comment */ "foo": "bar" }
  • arrays with keys: [ 0: "hello", 1: "world" ]
  • and much more
  • and any other combination of the above

A canonical trying to tackle dealing with not-JSON is about as useful as a recipe which calls for not-apples. Defining something by what it is not does not give insight into what it is. There is no single way to "handle" or "parse" or "fix" what is not JSON. Not all of it. Not every possible combination.

Note: some tools do handle some cases that are technically not in the JSON specification. Common ones include having comments or trialling commas or no quotes on keys. Yet, no tool or library can be assumed to conform to these. And different ones might have support for different combinations of these common cases.

With that aside, the only sane way to deal with not-JSON is to fix the source to make it JSON. Or any other common format. Yet, the reason somebody would be searching for how to handle not-JSON is because they could not have a well-defined format.

The other somewhat reasonable way to handle the situation is to derive the grammar of whatever not-well-defined format is, then write a parser for it. Yet the fact that the only apt descriptor for this data is "not-well-defined" is a signal that it is not a universally doable. Very often there is no real way to know what the grammar is, if the source is a closed system.

A canonical that contains either or both of these solutions does not seem very helpful to have. At best it only takes ones step step towards a solution (fix the source or write a parser) but not really going the whole way (how to fix the source, how to write the parser). And I would argue that while these are the correct approaches, a truly correct answer would be out of scope for the site.

Thus at best, there could be a Q&A for "what steps should I take to deal with not-JSON" but that certainly should not be a universal dupe target for any and all not-JSON questions.

11
  • 5
    There could, however, be language- or library-specific canonicals.
    – Ryan M Mod
    Commented Oct 21, 2023 at 2:36
  • 11
    @RyanM that would be true. There could be a canonical that tackles one or more of the "deviations". Although, I think discoverability of "what exact not-JSON I have" might become an issue
    – VLAZ
    Commented Oct 21, 2023 at 8:33
  • 3
    We should IMO have a canonical that a) lays out the sort of information presented here; b) explains how to check whether the input is valid JSON (both with tools and by visual inspection); c) mentions JSON-like and related formats that the data might actually have (for example, people very often want to throw a normal JSON parser at JSONL). Or possibly split that across questions, but I don't know how that would look. My (unified) attempt was stackoverflow.com/questions/75188362 but it didn't go very well. Commented Oct 21, 2023 at 12:36
  • I like the "recipe for not-apples" analogy. A single canonical q would be closed as too broad because the problem space is too large to have a single generic one-size-fits-all q&a -- even restricting the problem space to a single language. Commented Oct 21, 2023 at 17:28
  • “the only sane way to deal with not-JSON is to fix the source to make it JSON” – or just parse it from first principles: define a formal grammar, write a tokenizer and parser.
    – dumbass
    Commented Oct 22, 2023 at 13:17
  • 1
    @user3840170 "The other somewhat reasonable way to handle the situation is to derive the grammar of whatever not-well-defined format is, then write a parser for it"
    – VLAZ
    Commented Oct 22, 2023 at 13:21
  • Ah, missed that while skimming.
    – dumbass
    Commented Oct 22, 2023 at 13:34
  • 1
    The problem with "... derive the grammar of whatever not-well-defined format is..." is that said grammar is often ambiguous, in theory if not in practice. Parsers for ambiguous grammars are difficult.
    – Stephen C
    Commented Oct 23, 2023 at 9:31
  • 1
    @StephenC "Very often there is no real way to know what the grammar is, if the source is a closed system." although even if it's not a closed system it's probably hard. But might be reverse engineered easier. Still wouldn't recommend this approach for unknown ad-hoc formats.
    – VLAZ
    Commented Oct 23, 2023 at 9:34
  • It is worse than that. An unknown grammar can be reverse engineered / inferred from examples. The problem arises when some "valid" input for the grammar can be parsed in two (or more) different ways with different meanings. Then you can't tell which is the correct. Even determining that there >are< multiple parses for the same input makes your parser a lot more complicated.
    – Stephen C
    Commented Oct 23, 2023 at 9:38
  • 3
    The classic example is when the software that generates the JSON is not escaping quotes properly. Then you don't know for sure where strings start and end.
    – Stephen C
    Commented Oct 23, 2023 at 9:42
6

The problem is not that there is no canonical. The problem is that there can be no canonical, because invalid JSON is invalid for an infinite number of different reasons.

A canonical would also be a waste of time because the people who are asking why a JSON parser can't handle their invalid JSON, are invariably the same people who don't understand how JSON works, nor do they care to understand it. Telling them to use regex or build a custom parser is therefore about as useful to them, and as likely to be successful, as teaching a fish to dance.

2
  • "the people who are asking why a JSON parser can't handle their invalid JSON, are invariably the same people who don't understand how JSON works, nor do they care to understand it." - Ouch, cynical. Sure, some of these people are help vampires, but others do want to learn how JSON works, especially once they realize it's relevant to their issue. Commented Oct 23, 2023 at 20:09
  • @KalindaPride Unfortunately the ones who are willing to learn are a tiny minority, and we simply don't have the numbers of curators needed to service that.
    – Ian Kemp
    Commented Oct 24, 2023 at 16:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .