bzr 1.14: Installing from source broken by default on Windows when compiling extensions

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Richard B. wrote:
> I  haven't  got  a  problem  building  bzr  from  sources.  I was just
> wondering  if  a  standard  procedure  had  been  agreed upon (per the
> original  message  list  thread  message  "RFC:  including zlib.dll on
> Win32"),  and  when  this  automation  might appear in an official bzr
> release.
> 
> I'm a bit surprised the official bzr 1.14 release was made without
> this being resolved first.
> 

Well, it is all a matter of time and energy... I've been sick the last
couple of days, and have been distracted with other things.

For now, I'm just going to get a win32 installer built "by any means
necessary" and hopefully document it a bit.

I think it will just end up that we need to add some steps to how to
build bzr for win32, which includes getting the zlib dll from
www.zlib.net, and then performing whatever operations to get it to
compile with your compiler. (The steps for MSVC being slightly different
than mingw.)

I'm *personally* only planning on working out the details for the mingw
stuff, as that is what works for me. And I prefer to use open-source
tools where possible. (I don't like that TortoiseBzr can only be built
with the for-pay version of MSVC, because of dependencies on ATL which
apparently isn't even included in the free version of MSVC, but I wasn't
the one spending the time writing that code.)

As far as depending on zlib.dll... The only reason I originally wrote an
extension that used it, was because I knew that python itself used zlib.
It was late when I found out that on Windows they statically compile it,
and *don't* export the symbols, so you have to bring in another copy...
I'm sincerely reconsidering the extension.

At the time I wrote it, it was a performance benefit for what I was
benchmarking. But I seem to remember restructuring the code to avoid
creating as many objects that needed to be crc'ed.

For my setup, I'm using cygwin + gcc-mingw32 install. Mostly because
using cygwin as my shell makes things easy, and it at least comes with a
simple package manager.
As I investigated, I found that for some reason my local machine has:
 /usr/lib/mingw32/libz.a
and
 /usr/lib/mingw32/libz.dll.a
as well as:
 /usr/include/mingw32/zlib.h

I don't have any idea how these got to be installed, it certainly could
have been something that I did manually a long-long time ago.


Anyway, these are the steps I took to get 1.14 built, using cygwin's gcc
- -mno-cygwin install:

1) Update my cygwin install because I was foolish

2) Find out that /usr/bin/gcc no longer exists as a real .exe and is now
a symlink pointing to /etc/alternatives/gcc, etc, etc. Which breaks all
of 'py setup.py build_ext' because there is now no .exe for it to find...

3) cry a bit

4) rm /usr/bin/gcc-3exe ; cp /usr/bin/gcc-3.exe /usr/bin/gcc.exe
   # You have to delete it first, otherwise it seems to follow the
   # symlink, and claims you can't copy a file onto itself

5) Download the zlib123-dll.zip from http://www.zlib.net
6) mkdir zlib; cd zlib; unzip ../zlib123-dll.zip
   # Why doesn't zlib.zip have a root dir like most other people...
7) Run
   dlltool -D zlib1.dll -d lib/zlib.def -l lib/libz.a

   # Note that this differs in using 'libz.a' rather than 'libzdll.a'


8) cp include/zconf.h include/zlib.h /usr/include/mingw/
   cp lib/libz.a /usr/lib/mingw/
   cp zlib1.dll /cygdrive/c/WINDOWS/system32

   # This puts the appropriate files in places that mingw will look for
   # *first*
   # Note that if you don't copy libz.a to /usr/lib/mingw/ it will end
   # up using /usr/lib/libz.a which is a statically compiled version,
   # which might be better...

9) Update build_release.py so that it uses:

VERSIONS = {
    'bzr': '1.14',
    'qbzr': '0.9.9',
    'bzrtools': '1.14.0',
    'bzr-svn': '0.5.4',
    'subvertpy': '0.6.5',
}

10) Run into the problem that bzrtools is now using rich-roots, so go
out and nuke the dir, init a rich-root repo, and repopulate the bzrtools
subdir.

11) run 'py build_release.py'

12) cd release-1.14
    # The simple python installers don't include plugins, mostly because
    # including them turned out to be a dependency disaster, because
    # then the user has to install the dependencies (PyQT, subvertpy,
    # etc), before they can just get 'bzrlib' installed.
    # So the choice was.. if you want to install everything piece by
    # piece, you are welcome to, but you have to install *everything*
    # separately, and handle dependencies yourself.

    rm -rf bzrlib/plugins/{qbzr,svn,bzrtools}
    make PYTHON=/cygdrive/c/Python25/python python-installer

    # Realize that the standalone installers probably now depend on the
    # user manually installing zlib1.dll....

13) explorer .
    right click on win32_bzr.exe
    Select 7zip/Add to Archive...

    Set the 'self extracting executable' flag
    set the name to 'bzr-nonadmin-1.14-1.exe'
    Go

14) Rename all of the files from 1.14 to 1.14-1
    mv bzr-1.14.win32-py2.4.exe bzr-1.14-1.win32-py2.4.exe
    mv bzr-1.14.win32-py2.5.exe bzr-1.14-1.win32-py2.5.exe
    mv bzr-setup-1.14.exe bzr-setup-1.14-1.exe

15) Upload all of the files to my local server to be signed
    rsync -avP bzr*.exe ....

16) Switch to my server, sign the files:
    for f in *.exe; do gpg --detach-sign --armor $f; done

17) Upload the files using a script I wrote:
    ./upload.py bzr*.exe

    While it takes care of getting the URL information and syntax to
    'curl' correct, it still requires me to enter my password 4 times,
    so I have to 'babysit' the process. (If I wait too long, connections
    timeout, and I have to start over.)


Anyway, this is the hints as to what it takes to actually build and
upload the windows releases. And as I'm the only one who is doing it,
ATM, you can also see why it takes me at least 1/2 a day to get it done,
and why we don't always get every minor release built. There are a lot
of steps (like 3 :) that are hard to automate, which in turn means that
*I personally* spend hours on every release just to get something that
is 'mostly' functional uploaded. (In the past several releases bzr-svn
has changed things enough that it took several iterations of this
process before we had something that would actually work for people...)

If you can't tell, I'm slightly burnt out on the process... (which is
also why there weren't entirely 1.13.1 and 1.14rcX windows builds.)

John
=:->



-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkn51K0ACgkQJdeBCYSNAAP6BACbBogNGxelPoCU/TgBbtkltKvm
VykAn0Tnq4rtukUrBurUk5bNdeVakonl
=JdEC
-----END PGP SIGNATURE-----