[RFC] locking decorators and iterators

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Aaron Bentley wrote:
> Robert Collins wrote:
>>> I ran into a nice bug today, and its structural in our code.
>>>
>>> This is because while extras() requires a read lock, and even takes one,
>>> the lock expires before any code from extras() executes: calling the
>>> method simply creates the iterator object.
> 
> Definitely an anti-pattern I've run into a few times.  I agree that
> iterators should not take locks.  In Python 2.5, iterators can have
> try/finally blocks around a yield, so it's a different story, but the
> lock still needs to be taken inside the iterator.
> 
> So yes, lock + iterator bad.  A related issue is that iterators should
> not update progress bars, because they can't know what is happening to
> the progress bar while they aren't running.
> 
> I'm not sure whether the bug with extras is actually related to locking,
> though.  Let's say extras actually did take out and hold a read lock for
> the duration of its iteration.

Looking at the output of which entries failed after the new decorators,
though, I can see that 'revision_trees' and 'get_deltas_for_revisions'
are both things that should require a lock, and that they are both
iterators which can actually help performance (being able to read ahead,
but return things as they are available/in the requested order)

At the very least it helps keep memory down. But I've also found it
helps tremendously with some of the readv() code. Specifically, when we
are doing file_ids_altered_by_revisions, we read in the inventory.knit.
And if we can start processing it before we have to read everything, we
get to mix some I/O time with processing time.

For something like revision_trees() you are talking disk i/o time, but
there is still the potential to do a bit of processing while you are
waiting for some disk i/o. (You may need to spawn a thread to get any
real benefit, though, versus network I/O where it is queuing up for you)

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFF0c3eJdeBCYSNAAMRAhB1AJ4gPM7MEF2Ur5nCrbM8gI6aNDmpWgCgsZYo
kyniOeOSEXpew0N2NB8g7YI=
=Djip
-----END PGP SIGNATURE-----