1.16 Release

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Vincent Ladeuil wrote:
>>>>>> "martin" == Martin Pool <mbp at sourcefrog.net> writes:
> 
>     martin> 2009/6/9 Vincent Ladeuil <v.ladeuil+lp at free.fr>:
>     >> But, we also talked about versioning these files directly in
>     >> bzr.dev which would address the above problem and make it easier
>     >> to build C extensions without requiring pyrex[1].
> 
>     martin> Let's do it, and then make the build just fail if they're not there.
> 
> Rhaaa, I forgot to mention the most important point :-(
> 
> So, we don't have a strict policy regarding the required pyrex
> version. The pyrex files themselves are not heavily modified so
> the C files shouldn't change a lot either *iff* we stick to one
> pyrex version when committing such changes.

Not really true. The autogenerated C files include comments on just
about every line indicating the absolute path of the file and line
number that it came from. Such as:

  /*
"c:\Users\jameinel\dev\bzr\jam-integration\bzrlib/_bencode_pyx.pyx":84 */
  __pyx_2 = (((struct __pyx_obj_6bzrlib_12_bencode_pyx_Decoder
*)__pyx_v_self)->size != 0);

So regardless the pyrex version, *every* comment like that is going to
change anytime your system thinks it needs to update the .c file. (Which
could easily happen whenever you do 'bzr up' and it gets new files.)

I understand the desire to version them. I'm really against it until we
get those things sorted out.

As for 'min pyrex version', I would *really* like to depend on pyrex
0.9.8+ as it provides things like:

  cdef list myobject

  myobject = []
  myobject.append(foo)

Gets automatically translated into:

  PyList_Append(myobject, foo)

Rather than needing to manually define "PyList_Append" and write that out.

> 
> The questions are:
> 
> - should we chose the pyrex version as proposed by the
>   Ubuntu-du-jour at the the time of the release ?
> 
> - will pqm try to rebuild its own versions or not ?

I would guess it depends how sensitive the timestamps are for file
creation. Off-hand if we create the files in 'filename' order, I think
they'll get regenerated. (foo.c comes before foo.pyx.) However, I'm
pretty sure we build the working tree in "whatever order is best for the
repository to extract" so it is likely to be quite random whether they
get rebuilt or not.

> 
> I know John invested quite some time on the subject so his
> feedback is highly desired :)
> 
> But from there I'd be happy to do the related changes if our RM
> can't.
> 
>   Vincent
> 

Some requirements

1) Add a stage to the build script that post processes the .c files to
change the absolute paths into relative paths.

2) Define a pyrex version that we will build with. I think building with
the latest is good. Though I'll note that I'm sure PQM has a rather old
version, because it rejects submissions with "foo += 1".

I sort of like that PQM is running python2.4 and an old version of
Pyrex, since it helps enforce that we stay backwards compatible. Though
that would have to go if we want to start versioning these files.

(1) is a bit tricky, because you want it to happen after the .c has been
generated, but before the .so is created from that.

I think it is doable, just tricky, as I think it means monkey-patching
the Pyrex.Distutils.build_ext. Though I'll note that we already have a
custom subclass, so that we can decide whether failing to build is a
strict failure or a soft failure (--allow-python-fallback)

3) I guess according to Martin we would change 'add_pyrex_extension' so
that it doesn't 'soft-fail' if Pyrex isn't available and the .c is
missing. (We would now require the .c files to be present.)

4) In light of (1) we may want to just move to an explicit "make pyrex"
step, which then runs "for f in bzrlib/*.pyx; pyrexc $f", rather than
having them built from setup.py.

Heck if we wanted to, "make pyrex" could even ensure you have the
correct pyrex version...

Anyway, pyrexc just generates the .c files, which would give us a very
clear time to go post-process them. And then "py setup.py build_ext -i"
doesn't try to generate .c files, it only compiles them.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkouZ0kACgkQJdeBCYSNAAPUhgCeP06pykL+nqq96DamZSysoCyQ
/bkAn1VuwOoTgFt7ijMzimrG69wTkh7K
=pLRi
-----END PGP SIGNATURE-----