Loggerhead directions

[Michael Hudson]

On 01/04/10 21:10, Kapil Thangavelu wrote:
> On Tue, Mar 30, 2010 at 8:26 PM, Ian Clatworthy<
> ian.clatworthy at canonical.com>  wrote:
>
>> Hi all,
>>
>> After I wrap up improving the Windows installer build process, I'm hoping
>> to get more involved in Loggerhead development and maintenance. I'm sending
>> out this email now though because I want to raise some topics for
>> debate/discussion.
>>
>> At this point, I expect my immediate priorities to be "make it faster and
>> get it easier to deploy", i.e.:
>>
>> 1. Help review and land the dozen or so proposed branches.
>> 2. Improve packaging so that Loggerhead becomes a standard part of our
>>    Windows (and OS X hopefully) installer.
>>
>> I have some ideas on how to make Loggerhead more beautiful and more
>> functional but they can come later. :-)
>>
>> There's some *great* work going on in this space right now. In particular,
>> John Arbash Meinel and Kapil Thangavelu have put together some branches that
>> speed things up considerably. IIUIC, John's changes leverage recent
>> improvements to bzrlib so, apart from maintaining backwards compatibility to
>> earlier bzr versions, they are relatively straight forward. Kapil's
>> branch[1] uses chameleon[2] templates instead of simpleTAL. As such, we need
>> to discuss whether that's the best direction or not before we can land his
>> changes.
>>
>> More broadly, I'd like to get some feedback on the current technology stack
>> under Loggerhead (simpleTAL, simplejson and Paste). In terms of templating
>> languages, both Chameleon and Jinja2[3] are supposedly much faster than
>> simpleTAL. I also understand that Ian Bicking is refactoring the core of
>> Paste into WebOj. So looking forward, some options would be:
>>
>> * switch to Jinja2 + simplejson + WebOj
>> * switch to repoze.bfg (which wraps WebOj and supports both
>>   Chameleon and Jinja2).
>>
>> If we made such a switch, I'd suggest we keep the 1.17 branch on the
>> current technology stack and start a "2.0" branch on the new stack.
>>
>> Thoughts?
>>
>
>
> Its really not clear to me what value for loggerhead, which is pretty close
> to barebones wsgi, adopting a framework brings for the goals of making it
> fast and packaged?

I think this is what I was trying to say expressed much more clearly.

> As for speed the ideal scenario are figuring out how to
> make the bzr web browsing experience web speedy for large repositories and
> effectively O(1) wrt to history.
>
> Towards a goal of making the web experience faster, most of the gains will
> be from reexamining the cache usage and its structure. On the chameleon
> branch, i've experimented with some fairly trivial changes like the
> following two which remove the history cache usage from the file diff and
> revision ui respectively, and result in significant (~3x) performance
> improvements on request times.

I'd had the need for this in the back of my mind, I didn't realize it 
would make so much difference though.

> http://bazaar.launchpad.net/~hazmat/loggerhead/speed-rules/revision/408
> http://bazaar.launchpad.net/~hazmat/loggerhead/speed-rules/revision/409(this
> one might be a bit more controversial, but the view was previously
> walking back 10k revisions against the launchpad history and then discarding
> the result, for no apparent reason).
>
> Examining the views one by one and figuring out what they need access to,
> and to what extent they need to utilize the cache is probably a good first
> step in this direction. John Meinel's work on the branch is a great step
> forward in that it updates the cache generation to modern bzr apis and
> minimizes the time to generate the whole history cache. But even loading
> cache is relatively expensive, and it seems like its fairly common for views
> to calculate data from it without using the full result. John mentioned he
> had some ideas on a new cache structure that might be more efficient which
> would be great to see.
>
> There's also a bit of bit rot i think in the loggerhead code, for example it
> wasn't clear to me that the directory_ui view was used anywhere.

directory_ui is used when you say "./serve-branches ~/some/directory".

> I think if your going to go for 2.0, it might be worth dropping features or
> at least not making the common default views nesc. depend on them (ie.
> annotated default file view vs. just a file view).

Ah yes, that's certainly a good idea.  The annotation information could 
be loaded via Ajax.

> If i were going from the
> ground up, i'd perhaps even consider dropping revnos from the ui completely
> which will remove much of the need for a cache. I know it probably won't
> fly, but i think its worth consideration. i tried it out using trac-bzr as a
> guinea pig (and also sans the svn directory style change propogration) and
> it definitely validates at least for myself that fast bzr browsing with O(1)
> wrt to history and sans cache is possible.
>
> Regarding template engines, both jinja2 and chameleon, basically take their
> respective template languages (django and zope page templates) and compile
> that out to python code for execution. Given the existing template
> implementation i'd probably just continue with zpt/chameleon.
>
> wrt. to packaging, with a switchout of simpletal, loggerhead is effectively
> just an easy_install/pip installation away. Not that it obviates the need
> for binary installers, but its nice for those who don't use them ;-)

That's an advantage I hadn't thought of!

Cheers,
mwh