Background
When I initially encountered How can I efficiently remove elements by index from a very large list? it was titled "Remove int elements from a very large list in C#", had two close votes, and looked largely as it does now...
I have a very large list of integers (about 2 billion elements) and a list with indices (couple thousand elements) at which I need to remove elements from the first list. My current approach is to loop over all indices in the second list, passing each to the
RemoveAt()
method of the first list:indices.Sort(); indices.Reverse(); for (i = 0; i < indices.Count; i++) { largeList.RemoveAt(indices[i]); }
However, it takes about 2 minutes to finish. I really need to perform this operation much faster. Is there any way of optimizing this?
I have a Intel i9X CPU with 10 cores, so maybe some way of parallel processing?
I now see that the question originally did not contain any code, so I am assuming the thrice-upvoted comment "please show your slow code!" explains those close votes.
With the question more concrete, 6 answers soon followed, and those were later incorporated into my own where I presented my solution and provided benchmark code and results for it, the other answers, and the question's implementation. I continued to improve my answer over the following days, spending (too) many hours on this (writing and testing my own solution, writing and running the benchmark for all solutions, writing and refining my answer) and related matters (i.e. fighting/filing bugs encountered along the way in the benchmark library); such are the rabbit holes Stack Overflow provides.
So, yes, I have a vested interest in this question being in a state such that people can actually read and benefit from my efforts instead of them going to waste. Now, of course, if I answer an obviously-awful question and it ends up disappearing then that's my own fault, but I don't think this is that kind of question all. Though I can't really argue with anyone who feels that question was deserving of downvotes, as I'll try to explain, I don't think it warranted any votes stronger than that.
Closure
"Needs more focus"
Finishing my work by editing the question apparently attracted the attention needed for the final close vote and, evidently, punitive downvotes for the top-voted answer and my own. The stated close reason of "This question needs to be more focused." didn't make sense, and I commented as such...
What additional focus does this question require? A specific definition of "much faster"? If the votes are keying on the "if your question has many valid answers..." part of the close description, then couldn't any performance question fail that criteria? How should the author know how many ways there are to make their code faster? Don't ignore the "(but no way to determine which - if any - are correct)` that follows, though; there is a way to evaluate the correctness — or, at least, merit — of an answer here: benchmarks, which is exactly what I did in my answer!
To elaborate on that, this is not the usual "What's the fastest way to do Z?" question with no code or effort that's actually asking "How do I do Z? (And make it fast, please!)"; this is really "What's a way to do Z that's faster than this?" Yes, it would have been nice if the author had made some attempt at, say, the parallel processing solution they proposed, but then they did include some attempt at solving the problem: the working code they already have that's the basis of the question.
Further, I believe "needs more focus" is sometimes used to indicate "By not attempting a solution, you haven't narrowed the realm of possible answers", but then how much scope is really required for a question about improving the performance of specific code? If one has to include not only their working baseline code but some incomplete attempt at improving it, doesn't that risk it becoming an XY question? The question changes from "What's a way to do Z that's faster than this?" to "Why isn't Q working to do Z?" with the "(faster than) this" part potentially getting lost. It seems to me that either framing has its issues if held to that standard of "focus".
The XY problem
On the subject of XY questions, I can see there being the issue that the question doesn't give any background as to why it's storing so many integers this way. This was mentioned in one of the later comments...
The question is..... why do you keep 2 billions element in memory? That's about ~8G in memory for a list of int. How about using a database for dealing with data?
It's a good point and including that kind of context in the question might have been helpful, but I don't think that's a reason for closure; Should I flag questions w/ XY problem?...
No, don't use moderator flags for this.
In general, moderator flags are for situations that cannot be handled by the community. Answerable questions are answerable, even if the answer is "You're doing it wrong."
...and What should I do when the OP asks the wrong question?...
IMHO it's questionable to ban XY problem questions from the site as off topic in general.
These can give great enlightenment for future researchers, no matter how poor the original question was asked (regarding the [MCVE] or other formal close reasons).
...seem to agree. Even if using a huge List<int>
like that happens to be a poor choice for the author, it's conceivable that someone out there has a legitimate reason to do something like that and would find the question useful; just tweak the parameters — removing, say, 200,000 indices from a million-element List<Foo>
— and the question still applies. If nothing else, it's an interesting question for academic purposes.
Other close reasons
It has since occurred to me that the reason in the close banner is not necessarily what ultimately got the question closed, but then I'm not sure what reason it could have been...
Duplicate: None was proposed. Other than the question linked in the comments, I didn't find any others that deal with removing multiple elements from a list by index (not value), let alone with the performance aspect of doing so in a huge list.
Off-topic: No, this is a programming question. It's asking one question. It's not seeking recommendations (other than, of course, code).
Needs debugging details: I will grant that the provided code is not an MVCE because it doesn't show the declaration of the two lists, but then there's nothing to debug because it's already working code. 7 answers found that code to be sufficient to understand the task at hand.
Needs details or clarity: I will grant that the question never explicitly mentions the
List<>
class, but the long-obsoleteArrayList
is the only other BCL class I know of with those same methods and I think it would have said so if it used that.Otherwise, the framework (and, therefore, operating system), language, types, sizes, operation, and time to improve upon are all clear from the question, and the author responded to two comments seeking specific details, one of which yielded an answer (by a moderator, no less). What more information or specificity could be needed?
Primarily opinion-based: Did my attempt to improve the title perhaps make the question worse? Is "efficiently" considered subjective even when given an inefficient base for comparison? Otherwise, "Is there any way of optimizing this?" can be definitively answered via theory or measurements.
Whatever the reason for closure was, I couldn't do anything to fix the question because I'd already made all the improvements I could see in my edit a minute prior. Unfortunately, only one other person joined me in voting to reopen, and the question remained closed. Fine. I don't agree with it, but at least the existing answers will still be visible for future readers.
...until I noticed a couple weeks later the question was deleted, too!
Deletion
Admittedly, it's only rarely that I happen to encounter even an egregiously and unsalvageably bad question being deleted, so I am not too familiar with the process — maybe this happens all the time to other much-upvoted, much-answered questions and I've just been oblivious to it — but it's very surprising to see that one of the original close voters and two other users voted for this to happen. What's so wrong with that question that closure wasn't enough and it needed to be hidden from readers?
Bringing myself up to speed on the subject, in Privileges → Access to moderator tools I see...
When should I delete questions?
Closed questions that are of no lasting value whatsoever should be flagged and deleted.
Before voting to delete, please check whether there are any good answers; if so, then the question should be flagged for moderator attention as a potential merge candidate. We don't like to lose great answers!
How does a unique question with a +12/-3 score and 6 answers (excluding one self-deleted) with a combined score of +20/-2 even begin to qualify as "no lasting value whatsoever"?
Further, the relevant FAQ says...
What are the criteria for deletion?
For questions, a post that no longer adds anything to the site should be deleted. Basically, this includes most closed questions that cannot be improved and reopened.
I don't know how much work "no longer" is supposed to be doing in that sentence — the question no longer adds as much as it could to the site because it was closed — but I would strongly disagree that it doesn't add anything at all.
Also, I don't believe the question "cannot be improved", but then it's hard to say because, after the initial call to add code was addressed and despite the author demonstrating their responsiveness to feedback, no one took the time comment on what problem(s) remained with the question, so I can't speak to the feasibility of fixing them.
Following the quoted link is an answer to When to vote to delete question? that restates...
You delete a question when the content no longer adds anything to the site.
...and goes on to address handling each type of closure. Another answer on the same question says...
As per Jeff's post here, and answer here:
Questions which contain useful content contributed by your peers should generally be merged, not deleted.
I wish people wouldn't delete questions with good answers. You're destroying the useful contributions of your peers!
It's all about usefulness. Many questions get closed, but have useful answers that we want to preserve. Deletion votes should be based on usefulness, and that means usefulness to somebody, not just because it doesn't apply to your individual programming needs.
This Jeff person seems very wise and makes a great point about the potential for destroying quality content.
Another answer says...
If it's closed, it's fair game: vote to delete unless you can see that it has some value for the site (in which case you should probably vote to re-open).
...and yet another answer says...
It pretty much comes down to if you think the post is helpful on the site or not.
I'm seeing a trend for how to determine whether a question lives or dies. So, does a simple and clear question that hasn't been asked before with answers presenting a variety of creative approaches not have "usefulness" and "value"?
On the matter of merging questions, I did not know that's a thing. Unfortunately, it sounds like our answers could live on in that way if only the question author had made the mistake of posting an exact duplicate. Also, it's not my nature to summarily declare mine as being one of those "good", "useful" answers, but I will just say this: if mine isn't then I have no idea what one is nor what more I could possibly have done to write one. And that's not to suggest that mine was the only such answer, either.
I realize the various types of votes are subjective and, within the rules, it's up to each user to decide when and how to use theirs; what votes a post receives very much depends on who happens to encounter it. So, I really hate coming here to challenge how some users decided to use their votes, but it's just completely baffling to see the downfall of this non-terrible (at the least) question — and many person-hours of effort from myself and the other respondents null
ified as a result — with no obvious (to me) reason why.
In case all the mostly-rhetorical questions in the text above get me into "too broad" territory, TL;DR here is what I'd like to know:
- Was it appropriate to close the linked question?
- If it was appropriate to close the linked question, was it also appropriate to delete it and, therefore, its answers?