Merge-o-Matic not updating

[Lucas Nussbaum]

On 23/12/09 at 11:20 -0800, Bryce Harrington wrote:
> On Wed, Dec 23, 2009 at 01:28:23PM +0100, Lucas Nussbaum wrote:
> > On 22/12/09 at 21:38 +0100, Lucas Nussbaum wrote:
> > > On 22/12/09 at 13:25 -0500, Jordan Mantha wrote:
> > > > To find what packages are likely to need a merge you can use the
> > > > "Outdated in Lucid, and Lucid has local changes" section of the lists
> > > > at:
> > > > 
> > > > http://qa.ubuntuwire.com/multidistrotools/
> > > 
> > > I've also put together a list at
> > > http://udd.debian.org/cgi-bin/merges.cgi
> > 
> > Hi,
> > 
> > I've changed the architecture a bit.
> > http://udd.debian.org/cgi-bin/merges.json.cgi now generates a JSON dump
> > with the info from UDD (version in debian & ubuntu, LP bugs).
> > That dump is processed by a cronjob on people.ubuntuwire.com to generate
> > a web page, also using launchpadlib to query for the last uploader.
> > 
> > So, from now on, please use http://people.ubuntuwire.com/~lucas/merges.html
> > 
> > Patches are very much welcomed to improve the script. I've written it in
> > python to maximize the chances that other will participate. Also, my
> > Python is not very good, so don't hesitate to rewrite it ;)
> 
> Nice!  I have a few ideas for improving it...
> 
> Sorting the rows based on component/source/last-uploader would be
> helpful.  I have some javascript that implements this:
> 
>   http://www2.bryceharrington.org:8080/js/sorttable.js
> 
> This adds an arrow in the column header you can use to sort with.  You
> can see how it works and is implemented in html tables from this page:
> 
>   http://www2.bryceharrington.org:8080/X/Reports/ubuntu-x-swat/patches.html

Cool! A patch would be appreciated for that.

> For the json data, it might be helpful to prett-print it so the
> structure is clearer to those using it.  You can do this by adding the
> 'indent=4' parameter:
> 
>   print json.dumps(report, indent=4)

I'm not sure how to do that with the ruby json lib.

> (Btw, is the source code for the merges.json.cgi program online?)

http://svn.debian.org/wsvn/collab-qa/udd/web/cgi-bin/merges.json.cgi

> The output will be marginally larger in size due to the tabs, but IMHO
> it's worth it since it saves some time for whomever will be using it.
> 
> 
> In merges.py it appears that the only thing you need launchpadlib for is
> to get the package creator name.  One suggestion I'd make is if you
> could move that into the json file, then it'd let you simplify merges.py
> a whole bunch because you could take out all the launchpadlib stuff, and
> just have it all encapsulated in merges.json.cgi.

I can't do that. All the data from the CGI comes from UDD, so I would
have to add the creator name to UDD somehow. However, I can't easily
know in advance which creators I will need, so the only solution would
be to fetch the creators for each source package in the archive.
And doing that with launchpadlib would take hours or days.

> I notice you're using EDGE_SERVICE_ROOT... it's been my experience that
> edge is really unreliable.  An approach I took to work around this that
> you might consider, is instead of calling launchpad from a .cgi file, to
> do it from a cron job and write the json data out to a static file.
> This way, if EDGE goes bonkers and gives you the 503 error, you still
> have the most recent data in a file on disk your program can use.

RTT between UDD and launchpad: 149ms.
RTT between people.ubuntuwire.com and launchpad: <2ms.

It is much more efficient to run the launchpadlib script on
people.ubuntuwire.com.
Regarding EDGE, I've heard that a bug in the method I'm using was fixed,
and is not fixed yet on the main instance. Also, the script on
ubuntuwire.com is run by cron, and if launchpadlib fails, the previous
result won't be overwritten.

I'm all open for collaborating to get more data about Ubuntu into UDD,
since it would simplify writing such tools a lot.  However, the only
answer I got so far was "use the API", even if it would take ages to
fetch all the needed data using it. And there's already a cron job that
takes several hours every day to import the status of Ubuntu bugs into
UDD.

Ideally, what I would like is a way to replicate the public data from
Launchpad's DB to UDD. As long as this is not done, all imports of
Ubuntu data into UDD will be hackish. I can live with that, and
understand that manpower is not infinite, but please do not throw more
work on me because of that.
-- 
| Lucas Nussbaum
| lucas at lucas-nussbaum.net   http://www.lucas-nussbaum.net/ |
| jabber: lucas at nussbaum.fr             GPG: 1024D/023B3F4F |