Loggerhead directions

[Kapil Thangavelu]

On Tue, Mar 30, 2010 at 8:26 PM, Ian Clatworthy <
ian.clatworthy at canonical.com> wrote:

> Hi all,
>
> After I wrap up improving the Windows installer build process, I'm hoping
> to get more involved in Loggerhead development and maintenance. I'm sending
> out this email now though because I want to raise some topics for
> debate/discussion.
>
> At this point, I expect my immediate priorities to be "make it faster and
> get it easier to deploy", i.e.:
>
> 1. Help review and land the dozen or so proposed branches.
> 2. Improve packaging so that Loggerhead becomes a standard part of our
>   Windows (and OS X hopefully) installer.
>
> I have some ideas on how to make Loggerhead more beautiful and more
> functional but they can come later. :-)
>
> There's some *great* work going on in this space right now. In particular,
> John Arbash Meinel and Kapil Thangavelu have put together some branches that
> speed things up considerably. IIUIC, John's changes leverage recent
> improvements to bzrlib so, apart from maintaining backwards compatibility to
> earlier bzr versions, they are relatively straight forward. Kapil's
> branch[1] uses chameleon[2] templates instead of simpleTAL. As such, we need
> to discuss whether that's the best direction or not before we can land his
> changes.
>
> More broadly, I'd like to get some feedback on the current technology stack
> under Loggerhead (simpleTAL, simplejson and Paste). In terms of templating
> languages, both Chameleon and Jinja2[3] are supposedly much faster than
> simpleTAL. I also understand that Ian Bicking is refactoring the core of
> Paste into WebOj. So looking forward, some options would be:
>
> * switch to Jinja2 + simplejson + WebOj
> * switch to repoze.bfg (which wraps WebOj and supports both
>  Chameleon and Jinja2).
>
> If we made such a switch, I'd suggest we keep the 1.17 branch on the
> current technology stack and start a "2.0" branch on the new stack.
>
> Thoughts?
>


Its really not clear to me what value for loggerhead, which is pretty close
to barebones wsgi, adopting a framework brings for the goals of making it
fast and packaged? As for speed the ideal scenario are figuring out how to
make the bzr web browsing experience web speedy for large repositories and
effectively O(1) wrt to history.

Towards a goal of making the web experience faster, most of the gains will
be from reexamining the cache usage and its structure. On the chameleon
branch, i've experimented with some fairly trivial changes like the
following two which remove the history cache usage from the file diff and
revision ui respectively, and result in significant (~3x) performance
improvements on request times.

http://bazaar.launchpad.net/~hazmat/loggerhead/speed-rules/revision/408
http://bazaar.launchpad.net/~hazmat/loggerhead/speed-rules/revision/409(this
one might be a bit more controversial, but the view was previously
walking back 10k revisions against the launchpad history and then discarding
the result, for no apparent reason).

Examining the views one by one and figuring out what they need access to,
and to what extent they need to utilize the cache is probably a good first
step in this direction. John Meinel's work on the branch is a great step
forward in that it updates the cache generation to modern bzr apis and
minimizes the time to generate the whole history cache. But even loading
cache is relatively expensive, and it seems like its fairly common for views
to calculate data from it without using the full result. John mentioned he
had some ideas on a new cache structure that might be more efficient which
would be great to see.

There's also a bit of bit rot i think in the loggerhead code, for example it
wasn't clear to me that the directory_ui view was used anywhere.

I think if your going to go for 2.0, it might be worth dropping features or
at least not making the common default views nesc. depend on them (ie.
annotated default file view vs. just a file view). If i were going from the
ground up, i'd perhaps even consider dropping revnos from the ui completely
which will remove much of the need for a cache. I know it probably won't
fly, but i think its worth consideration. i tried it out using trac-bzr as a
guinea pig (and also sans the svn directory style change propogration) and
it definitely validates at least for myself that fast bzr browsing with O(1)
wrt to history and sans cache is possible.

Regarding template engines, both jinja2 and chameleon, basically take their
respective template languages (django and zope page templates) and compile
that out to python code for execution. Given the existing template
implementation i'd probably just continue with zpt/chameleon.

wrt. to packaging, with a switchout of simpletal, loggerhead is effectively
just an easy_install/pip installation away. Not that it obviates the need
for binary installers, but its nice for those who don't use them ;-)

cheers,

Kapil
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.ubuntu.com/archives/bazaar/attachments/20100401/b51ce878/attachment.htm