15

What I want to do:

I wish to create a question:

Title: Print all lines between two patterns, inclusive (in sed, AWK or Perl)

Question: it'll look much like this one, but ask about the inclusive, rather than exclusive, case.

Why am I asking here?

Because when I created that question above it caused a massive controversy.

What is the problem I am trying to solve:

As with the other question, I found it way too hard to find an answer to this trivial, extremely common, sed/AWK question. Type in search terms, "sed print lines between two patterns inclusive" and you may see what I mean. Part of the problem is there are so many similar hits, as a result of so many badly asked and badly worded questions, that it's hard to find the actual answer. And for people like me who prefer to find answers on Stack Overflow, I wasted some time.

(Frankly, at this time, Stack Overflow is a very poor resource for finding one-liners in sed/AWK. We could make it way better with some cleanup.)

Has the question been asked before:

I would argue, no. The answer is at Stack Overflow, but it's underneath a different question, i.e.:

That answer isn't an actual answer to the question asked at that link. So I didn't, and probably wouldn't next time, notice it there. The question there, despite 31 upvotes, is also not great, i.e. no examples of expected outputs.

A whole lot of similar questions exist, but AFAICT, not this exact question.

What is the answer I want to document:

sed -n '/PATTERN1/,/PATTERN2/p'

I found it here, admittedly the second highest ranked post on Google. I didn't go there, initially, however, because in my experience, people's blog posts on sed/AWK tend to be badly organised and are lists of their own personal favourite one-liners, and it takes a lot of time to find the one you're after.

What do I want out of Meta today?

I am happy with any answer, if that's the consensus. Simply doing nothing is fine for me. I solved the problem, and I can write my own blog post on "favourite sed one-liners" if making them easier to find at Stack Overflow is frowned upon.


Note about duplicate question marker. At this time, 3 people have voted to close in favour of duplicate of this. The answers there, however, are not highly-upvoted, none are accepted, and the leading answer is negative, cynical and defeatist, and there's no evidence of any obvious consensus. Is close-voting here another way of saying, "Just do whatever you want, we don't care?" Or failing that, at least add some comment on how close-voting is supposed to be interpreted. As I noted, if someone could tell me: Is there a Bash chatroom? That would also be good answer.

15
  • 5
  • Those questions don't really tell me much about what the bash gold hammer people think about my question. Mar 30, 2019 at 8:05
  • 1
    The answer there is also frankly rather defeatist and cynical: Don't even bother. There is nothing you can do to convince users who happen across your question that you are posting this for the sake of the community instead of earning points. Is that really the best answer? Why do we even bother trying to make the site better then? Mar 30, 2019 at 8:13
  • I did find this excellent suggestion though. I guess my question could then become, does Bash have such a chatroom? If so, I could ask there. Mar 30, 2019 at 8:23
  • 7
    I noticed that your last question neither shows any attempt to solve your problem nor does it show the results of your research. It just looks like a task dump. Maybe that is a reason for some of the downvotes. I'm not surprised that it got duplicate close votes. Users might have thought: Well, try <this> first and update your question if it doesn't.
    – honk
    Mar 30, 2019 at 8:38
  • @honk, I posted the question with a solution (and also a note explaining why the question wasn't a dupe), so that probably covers off research? Also, posting research in a canonical question is distracting. If I want a sed one-liner and I'm the person finding it in the archives, I certainly won't be interested in the research someone once did years ago. Mar 30, 2019 at 8:40
  • 6
    Well, as far as I understand the community, we don't lower the quality requirements for a question, just because it's self-answered. The question still has to stand on its own. And your first revision just refers to "many similar questions" without mentioning any specific ones. For readers it's unclear which answers you inspected and which you might have missed.
    – honk
    Mar 30, 2019 at 10:59
  • @honk, my point is, adding text about research adds extraneous information that isn't relevant to documenting a problem/solution. It lowers the quality. It gets in the way and slows the reader down, who just wants the Q&A. Likewise, you won't find any such text in published "Cookbooks" e.g. here. Admittedly, the first version of my question wasn't perfect. Input from the community improved it greatly. Awesome. Mar 30, 2019 at 11:06
  • 2
    @honk, in any case I'm always happy to comply with a community consensus, even if I disagree. So, how do I improve the quality of that question? Should I invent a story about having tried things that didn't work, or what should I do? Because the true story is I spent about 15 minutes finding bits and pieces of code at various sites until I found something I considered concise and elegant enough to actually use in my code. Mar 30, 2019 at 12:07
  • 1
    @honk, ...we don't lower the quality requirements for a question, just because it's self-answered... Conversely, "we" [not me personally, but the "community"] downvote if we think you are rep-hunting as well as if we think your question is poor. OP, for (relatively..) better feedback, make sure you community-wiki your self-answer if it's a one-liner. Won't stop people downvoting your question, though. Good luck.
    – jpp
    Mar 30, 2019 at 12:22
  • @jpp, it's ok. I'm fairly satisfied by this stage that I won't be creating another canonical, at least not in the sed/awk space. I don't suppose you have a view on writing a good canonical? I just had this one down-voted (probably someone reading this discussion ;-) In my mind, it's a perfect Q&A. Evidently not everyone agrees. As in, do you also agree there should be some section on research? Mar 30, 2019 at 12:35
  • 5
    @AlexHarvey, I think your canonical is perfectly fine. I've experienced the same as you. I've often wanted to share small snippets which I find immensely helpful but not described elsewhere, but am discouraged to do so by the current attitude to self-answered questions. There's a belief that the standards should be higher for self-answered Q&A to prove the question is interesting. Officially, the standard should be the same as for any Q&A.
    – jpp
    Mar 30, 2019 at 13:45
  • 1
    The problem I have with the duplicate marker here is the answers on the target are all different and none are materially preferred by the community. Indeed, one of them suggests that for popular specific/related questions (as in here) clarity should be sought from Meta. Which is what AlexHarvey is doing...
    – jpp
    Mar 30, 2019 at 13:51
  • 1
    Thanks @jpp. Fortunately or unfortunately, however you look at it, I feel I do have my answer now. I'm not going to get involved in cleaning up sed/awk. And I did discover from one of those answers that Python seems to be a far more civilised corner of SO. ;-) Mar 30, 2019 at 13:54
  • There is a Bash chat room: chat.stackoverflow.com/rooms/98569/bin-bash -- there is a link to it in the question you linked to, though it's just in one of the comments. The room gets very sporadic activity, other than by a bot and my responses to it.
    – tripleee
    Apr 23, 2019 at 7:30

4 Answers 4

13

As a Bash gold badge holder and one-time closer of your earlier question, I would say don't do it.

We have many duplicates of questions like these.

Extract lines between 2 tokens in a text file using bash is the current canonical which is linked from the bash tag info page.

How to print lines between two patterns, inclusive or exclusive (in sed, AWK or Perl)? is linked as a canonical from the sed tag info page and is arguably a better fit for what I describe below. You will find that it has 16 duplicates already.

If you are set out to create a duplicate, how do you make sure yours is going to be easier to find than the previous attempts? Only time will tell if your question is going to be making it to the top Google ranks, and if the existing duplicates are any indication, if you try to cover the informal terminology many people use for this kind of thing is only going to add to the landfill of questions about "extracting something between two other things"; and you will attract answers about extracting other somethings between two other things than what you actually asked about.

If you want to try to create a canonical anyway, I would say go for covering the various corner cases in a single question. How do I include the starting pattern but exclude the ending pattern? (This is a pretty common one.) How do I add a default if this pattern is not found? But a better way to spend your time is probably to help clean up the trivial duplicates which are not yet linked as duplicates of a good canonical, and figure out if you can add anything to the existing questions (perhaps a stellar new answer?) to make people more likely to find them.

9
  • 2
    I can see the Bash gold hammer holders are alike in closing questions as duplicates somewhat arbitrarily. That's the first problem. I also can see there's a gentleman's agreement in that tag to suspend Stack Overflow's rules about "how to ask", because the "lines between patterns, inclusive, exclusive" is at least 4 questions in one. And because it's 4 questions in 1, the info underneath it is hard to find. SO's rules exist to stop precisely this from happening. Now, I don't mind this question existing- but it shouldn't be used to block creation of legitimate canonicals. Apr 23, 2019 at 6:23
  • 3
    But that is already a legitimate canonical. It is phrased precisely that way in order to cover the myriad minor variations we see every week. (Not my canonical, mind; but very clearly designed precisely to cover the ground I outlined in my answer, even before adding the link to that answer,) The use of duplicates as signposts to direct users to the canonical should still work, by the way; even if the canonical itself is hard to discover, you are (somewhat) likely to find one of the duplicates, and click through. Though probably more duplicates should be discovered and similarly linked.
    – tripleee
    Apr 23, 2019 at 6:28
  • There you go. The "minor variations" you see. A minor variation != duplicate. And as can be seen from the solutions, the so-called "minor variations" end up being huge differences in the solution. Especially in a language like sed, where going backwards in a file is next to impossible, and going forwards is trivial. Totally different problems, and anyone who knows anything about sed knows this. Apr 23, 2019 at 6:32
  • 1
    So are you saying that canonical covers completely different ground than yours?
    – tripleee
    Apr 23, 2019 at 6:53
  • 1
    No, I am not saying that. I am saying that the so-called "canonical" is at least 4 questions in 1, and it is therefore not even a legitimate question. Technically, it should be closed as "too broad". How to print between patterns inclusive is trivial in sed, AWK & Perl. Exclusive, is difficult and different in each language. That this is 4 question can easily be seen from the fact that every "answer" is 4 answers. My question above, was an attempt to start a legitimate canonical that asks only 1 question. Apr 23, 2019 at 7:16
  • 5
    We can close that as too broad if you like but that doesn't mean it's okay to post another question which covers the same ground. I'll remind you of the guidance that we judge duplicates on whether it's "same answer", not on whether it's "same question". And I'll concede that the canonical question is technically "too broad" but that there is a legitimate reason for that, and I doubt that closing it is going to make anyone happier or solve an actual problem for anyone.
    – tripleee
    Apr 23, 2019 at 7:20
  • I don't want to make people unhappy. Honestly, I can't see a problem having the 4 case question there alongside individual case questions. The existing question is a useful resource. The new questions would also be, I daresay, even more useful. Anyway, as I mentioned above, I created my own blog post for this stuff because it's pointless trying to create the missing canonicals. Apr 23, 2019 at 7:41
  • 3
    Dunno about "pointless" but without coordination, this sort of friction is precisely what I would predict, because it's hard to navigate partially conflicting goals. This meta question is a good start though I perceive it as somewhat frustrated in tone. Please do feel free to ping me in the Bash chat room to discuss what sort of arrangement we would actually like to see and attempt to form a consensus. (Other interested parties should obviously feel free to join.)
    – tripleee
    Apr 23, 2019 at 7:46
  • 1
    And of course, this isn't really a Bash question at all - part of the problem is the overlap between Bash, other shells, Awk, sed, Perl, Python, etc etc etc.
    – tripleee
    Apr 23, 2019 at 7:48
5

Frankly, if there are to be canonical questions asked about awk one-liners, I'd prefer that they be consolidated into a single canonical post, much like this one: Hidden Features of C#?. Something like "Common One-Liners in Awk."

We get regex questions all the time asking about a very specific point-solution, and that is fine, but none of them make good canonicals, because they are so specific.

If you want good guidance on posting canonical questions, have a look at what the PHP folks have done. Their tag wiki is one of the best wikis on Stack Overflow, and it lists a number of excellent canonical posts, like this one. The C++ community has also done an excellent job.

4
  • 1
    I'm not following because Hidden Features of C# is closed as "not constructive". I don't think regex questions either is a good analogy. I rarely think to google for "regex for..." because usually my regex problem can't be easily put into words (unless it's something common, like "regex for IPv4 address" etc). My thinking, however, is that if someone is going to google for a specific problem using a predictable set of search terms, they should easily find the answer at SO. Often- e.g. in Git- that is true. Sed/awk- rarely. Mar 31, 2019 at 2:01
  • Actually regex probably is a good analogy, and that's why it's great to have a question like this, which has been viewed 127,529 times, so surely that's a huge contribution? Mar 31, 2019 at 14:20
  • I see you edited out my answer that there is no consensus. From what I can see, different tags are doing different things, and this question is yet to draw any responses at all from people who edit in the Bash tag. I'm sure that you feel what you're doing in the C# area is good and I don't mean to suggest otherwise. But without buy-in from Bash people, there is no consensus is there? Apr 1, 2019 at 5:09
  • 2
    @AlexHarvey: Folks can decide on their own by looking at the answers here whether there is or is not consensus. No need for an executive summary. Apr 1, 2019 at 14:38
1

I think that the following principle ought to always hold:

  • If a developer types an easily predicted sequence of search terms into Google in search of a canonical solution to a common programming problem, they ought to immediately find the said canonical here at Stack Overflow.

That is, as far as I understand it, the problem Stack Overflow itself is supposed to solve for the world. Finding the canonicals should not require knowledge of a Stack Overflow Wiki, or an index that lives off-site.

In some topic areas, I find Stack Overflow's organisation is perfect. I have typed, "git find the commit that deleted a file" into Google an embarrassing number of times, and, thankfully, I always immediately found the answer. (Maybe that's why I still can't remember it?!)

On the other hand, when I search for sed/AWK one-liners, I often begin a 15 minute search before I find anything that satisfies my quality standards.

Returning to this specific problem:

I think that this question:

Is too broad and thus not suitable as a canonical. It could be extended further into hundreds of related answers.

If that question was closed and new canonicals for the various cases were created, as I tried to do, we'd all be better off. Fewer people would ask duplicate questions, and when they did ask duplicates, it would be clearer which question to close them in favour of.

As for me, I'm going to do nothing for now.

6
  • 1
    Wow, I see you were talking about my question here. It would've been good to at least get a ping so I could participate while this was fresh and I could've explained in detail what brought me to post that wannabe canonical question, after years of seeing too similar questions.
    – fedorqui
    Apr 17, 2019 at 22:06
  • @fedorqui, so I agree I should have pinged you but I'm not aware of conventions around when/how to actually do that here in Meta. Apr 18, 2019 at 0:24
  • @fedorqui, you begin at the outset by acknowledging it yourself, that the problem you wanted to solve was questions that were "too similar". But "too similar" != "duplicate". The question you've created is, in fact, too broad. It really asks at least 10 or more questions rolled into one. I can understand that you may feel this topic is best organised this way, but it is not consistent with SO's rules. For instance, if a new user asked your question, so worded, it would be immediately closed as "too broad" or "unclear what you're asking". (part 1) Apr 18, 2019 at 0:29
  • Even if all the answers on SO to so-called "duplicates", as you defined them, were today merged into that question now, it would immediately expand to hundreds of answers. If the question was then extended to include all possible answers, it would expand into thousands of answers. My single variant of the question alone could have 10 answers--just in sed. Now add in another 20 for AWK solutions and 20 for Perl. It's untenable. (part 2) Apr 18, 2019 at 0:33
  • As an end user (as opposed to a SO contributor), the current organisation simply is not useful. I had my question, and I couldn't find an answer to it anywhere on SO, despite an extensive search of at least 20 different questions. Remember, it may be easy for you to find what you want, because this is your personal organisation. Others out there won't understand why you organised it this way. (part 3) Apr 18, 2019 at 0:37
  • Finally, I accept that we will no doubt have to agree to disagree on this. I'm afraid that, at this point, I don't have much motivation to contribute to cleaning up AWK/Sed one-liners. I've decided to create yet-another-sed/AWK-oneliners blog post where I'll organise them, for myself, off-site. But I strongly believe that, until this organisation is fixed, SO will never be a go-to resource for sed/AWK one-liners. Apr 18, 2019 at 0:40
-9

I'm very against this. Unlike on UL where we have a tag dedicated for these tasks (called text-processing), Stack Overflow doesn't work for such topics. I mean, if there was a tag for the general task of processing text, as this question is actually about (no, the question isn't about sed, awk or perl, is about a specific task), someone wanting to extract or transform text have many more tools than what it is usually available to the Unix & Linux user.

Such questions could easily each turn into a 1000+ answers long, breaking the Q&A model. I recommend Stack Overflow to stick to what it knows best: how to solve problems exclusive to software development. Yes, text processing can happen on software development, but there is a context that naturally constrains those questions: the programming language target the asker is working with.

9
  • 3
    As someone who began their career as a Unix sysadmin, there is almost no such thing as a problem that is "unique to software development". The entire Git tag ought to be consumed as frequently by sysadmins as developers, because there is no excuse for a sysadmin to be not revision controlling their files. As for text-processing, however, that is clearly software development, and it is far more broad than sed/AWK one-liners. Have you considered Big Data? It doesn't matter if you're writing a 100,000 line application or a 10 line shell script- you are still developing software. Mar 31, 2019 at 6:05
  • 1
    I should also add that I did just find the answer to the question I wanted to ask at UL here. And I do agree the quality of sed/AWK at that site is superior to SO. Doesn't change that these are programming questions of course. Mar 31, 2019 at 8:55
  • @AlexHarvey well, actually, there are problems unique to software development, they aren't just the kind of problems that a system administrator would usually face.
    – Braiam
    Mar 31, 2019 at 12:50
  • I suppose that's right, @Braiam. All the same, SO isn't truly about the ones that are "unique" to software development. Otherwise we'd be in a silly situation where a language like Ruby would have be over in UL because it was actually designed for system administration, despite later popularity as a web framework. Mar 31, 2019 at 13:18
  • 1
    @AlexHarvey is the task, not the tool, that makes someone on topic for SO. If you are using ruby for software development, that could be asked here. There would be very little cases where you can use ruby in a non software development context, that most have inverted the above relationship.
    – Braiam
    Apr 1, 2019 at 11:07
  • What you appear to be not considering is that development is itself a text-processing task. There is writing code, and then there is also editing text files in a code base. Just today I needed sed inplace editing to edit a bunch of JSON files in a code base I maintain. Devs use sed/AWK all the time. And if they don't use these or something similar (like Ruby or Perl etc), then I hate to think how slow and inefficient they must be. Apr 2, 2019 at 2:14
  • 2
    @AlexHarvey But because developer does, doesn't mean that it becomes a programming task. To use some of these tools, you even have to be impractical like on this question. Do you really believe using column -t is software development?
    – Braiam
    Apr 2, 2019 at 11:02
  • I don't see why calling column -t is different to doing similar things using printf in C. Both are functions that have docs and inputs and outputs. It's precisely the same thought process using one as using the other. I will say that many sysadmins are less disciplined in their thinking, more susceptible to "cargo cult" programming etc. It doesn't change the nature of the process. Apr 2, 2019 at 11:09
  • As far as your question there, I don't see anything impractical going on. Looks to me just like fewer people willing to downvote at the drop of a hat in that forum when someone asks a question that isn't precise. Apr 2, 2019 at 11:16

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .