Posting a comment, or another answer, would have accomplished nothing. Two people already posted comments about the issue I tried to address, and accomplished nothing. They got almost completely ignored. One of them posted on day 1 of the answer's existence, but in the over 7 years between when the comment was posted and when I saw it, it only got 3 upvotes. The answer got 44 during that time period. The answerer abandoned the post and never responded to anything.
Adding a third response to the pile wouldn't help. But my impression is that people have mostly made up their minds one way or the other about this, and rehashing the debate won't accomplish anything either, so how about this:
Proposal
Instead of my original edit, I propose a different edit. The original answer text is this:
But if there are many elements, it's a better idea to create
independent iterators using
tee()
:
import itertools
it1, it2 = itertools.tee(db[3], 2) # create as many as needed
Now we can loop over each one in turn:
for e in it1:
print("doing this one time")
for e in it2:
print("doing this two times")
The author has heard that tee
helps save memory on long inputs, but has made a mistake about what circumstances that works in. It only works when you're iterating over all the tees together, so their positions in the data stream stay close to each other. Several people have pointed this out in the comments.
I propose to edit it to say this:
Sometimes you can save memory over the list approach by using
itertools.tee
. For use cases like
l = list(iterator)
for thing, next_thing in zip(l, l[1:]):
...
where you're iterating over several copies of the input "in sync", you
can save memory by creating iterators with tee
:
import itertools
it1, it2 = itertools.tee(iterator, 2) # create as many as needed
next(it2)
for thing, next_thing in zip(it1, it2):
...
tee
consumes memory proportional to how far apart the tee iterators
are in the data stream, so as long as the iterators don't get too far
away from each other, this will save memory.
Is this good? Let's see what the help center has to say.
What the help center has to say
All contributions are licensed under Creative Commons, and this site is collaboratively edited, like Wikipedia. If you see something that needs improvement, click edit!
The post needed improvement, so I clicked "edit"! Let's keep reading.
Editing is important for keeping questions and answers clear,
relevant, and up-to-date. If you are not comfortable with the idea of
your contributions being collaboratively edited by other trusted
users, this may not be the site for you.
When should I edit posts?
Any time you see a post that needs improvement and are inclined to
suggest an edit, you are welcome to do so. The original author of a
question or answer may always edit their own post, regardless of
reputation level.
Edits are expected to be substantial and to leave the post better than
you found it.
This edit is substantial and leaves the post better than I found it.
Common reasons for edits include:
- To fix grammar and spelling mistakes
- To clarify the meaning of the post (without changing that meaning)
- To include additional information only found in comments, so all of the information relevant to the post is contained in one place
- To correct minor mistakes or add updates as the post ages
- To add related resources or hyperlinks
How does this edit compare to the "common reasons"? The edit
- could be argued to change the meaning (but less so than the previous edit),
- incorporates information previously only in the mostly-overlooked comments, and
- corrects the author's minor mistake about when
tee
saves memory.
Mostly good! But does it change the meaning? And is that a deal-breaker? Let's think about it.
Any edit necessarily changes the meaning of a post to some degree. Edits by the OP often change the meaning a lot, but even non-OP edits will change the meaning somewhat. One of the help-center-approved edit reasons is adding information from comments, and that changes the meaning. The meaning of the post before such an edit didn't include that extra information, and after the edit, it does.
The meaning of the existing version of this answer section is "use tee
for nebulous benefits on long iterators". The meaning of the edited version is "use tee
to save memory on long iterators in the specific type of situation where it helps". That seems good to me.
Anyway, the help center says to edit "If you see something that needs improvement". Not editing would violate that.
tee
doesn't work like that. It's just a worse version oflist
if you use it that way.tee
avoids saving all the elements in memory, or does something else useful for large inputs. It only does that for use cases where you're iterating over both tees at the same time, likefor thing, next_thing in zip(tee1, islice(tee2, 1))
. If you're looping over the input twice in sequence,tee
offers no benefit.tee
being "a better idea", and it wrongly implies the existence of benefits that do not exist. Multiple commenters have pointed this out, but the answerer has done nothing.tee
isn't useful here and when it is useful, got no upvotes until I upvoted it. People just don't pay attention to that kind of thing. Adding a third "tee isn't useful here" response would be unlikely to actually accomplish anything.list
for long iterators". They did not achieve that intent, due to a misunderstanding oftee
. The edit achieves that intent. The rollback does not.