[RFC] New feature: --batch to accept command sequences from stdin (or file)

[John Arbash Meinel]

Jari Aalto+mail.perl wrote:
> * Wed 2006-06-21 Aaron Bentley <aaron.bentley AT utoronto.ca>
> * Message-Id: 44997B1C.3010207 AT utoronto.ca
>>> bzr startup time, as for all interpreted languages (Perl, Python ...)
>>> is quite constly. Especially under Win32/Cygwin it takes 3-5 seconds
>>> before simple "bzr ci" is run under old hardware.
>> We're working on reducing the startup time so that it's comparable with
>> Hg's.  That's what the recent discussion of "demandload" and "profiling
>> imports" was about.  One of the reasons for the new "dirstate" format is
>> to avoid having to import ElementTree.
> 
> Thanks,
> 
> Originally I was more thinking the language interpreter startup time,
> which I expected to be the problem. In PII 400Mhz/512M/W2k/Cygwin this
> command
> 
>    $ time python -c "exit;"
> 
> takes average of 1.5 sec (real). To run 3 commands in series, it would
> take total:
> 
>     1.5 x 3
> 
> hence my question if --batch could be considered.
> 
>>> It would be nice if bzr somehow accepted commands to run from 
>>> stdin, from file or command line. Something like:
>> There is already the "bzr serve" plugin.  See also the "bzr shell"
>> command in bzrtools, for interactive use.
> 
> OK, Where can I learn more about the "serve" plugin?
> 
> Could the "serve" plugin be included in the Core? This sounds like it
> would help people using Cygwin version of python.
> 
> Jari

Sorry, I wrote it long enough ago, I forgot it was called 'service'.
http://bzr.arbash-meinel.com/plugins/service/

It needs a little bit of work to make it secure, but I did just update
it to make sure that it runs.

Basically, when you run 'bzr service' it starts a program that binds to
localhost:11111, which can then be requested to do work for you.
The client just passes the command line arguments and the current
working directory, and the service forks and runs the command.
Then the client waits on the socket and returns all of stdout and stderr
in one stream.

It could be improved with:

1) Secure connections. As in using a local named socket (like ssh-agent
or gpg-agent do), which can be created inside a temp directory, and have
permissions locked down. I believe this will work on both cygwin and
Linux. It won't work in plain win32, but forking doesn't work either in
plain win32. If you are using this, you also need some sort of way to
indicate to clients what socket to connect to. (most *agents use an
environment variable for this)

2) Instead of sockets, you could use FIFO's, and then create a new FIFO
for each client who connects.

3) After forking, create 2 return sockets, one for stdin and one for
stdout. Use some sort of async in the client to poll those for output,
and return it as soon as you get results. If using FIFOs, create a 2 new
 and just replace file descriptors stdout and stdin with the given FIFO.
The client would still have to poll properly to make sure that the
screen gets updated, but the server side gets easier.

4) It would also be possible to change the messages being sent back and
forth, to indicate whether the text should be on stdout or stderr.

5) Improved messaging would be also be nice, since weird things happen
if you have signing turned on. (The signing command uses the terminal of
the service, not the terminal of the client).
So an improved message system could tell the client to spawn this
command with these arguments, and pass back the results.

The #1 thing that needs to be done is to get more secure connections. I
really don't know how to do it properly for win32. It is also really
difficult to share env variables under win32 (or cygwin). My ssh-agent
stuff creates a file on disk which records the current env vars, and
when bash starts up, it checks to see if ssh-agent is running.

Also, win32 native python is faster than cygwin python. By quite a bit.
The only thing you lose is the ability to handle executable bits (and
there is currently a bug that if you do 'bzr revert' it toggles the exec
bits, so it takes 2 'bzr revert' commands to get the right bits).

I've never seen it take 1.5s to just spawn python before.

Just to give the final results of performance, I have:
$ time bzr root
/home/jameinel/dev/bzr/plugins/service

real    0m0.449s
user    0m0.368s
sys     0m0.060s
jameinel at liliana ~/dev/bzr/plugins/service
$ time ./client root
/home/jameinel/dev/bzr/plugins/service

real    0m0.020s
user    0m0.000s
sys     0m0.004s
$ time ./client.py root
/home/jameinel/dev/bzr/plugins/service

real    0m0.070s
user    0m0.036s
sys     0m0.016s

This is on a reasonably fast machine. But you can see that we spend
about 400+ ms just loading libraries. Running the python interpreter on
this machine adds ~50ms over a C compiled client. But it is a good place
to start if you are wanting to improve the communication support.

Ultimately we are wanting to reduce the bzr startup overhead. We have
quite a few ways to do it, but this might get you going in the short term.

I'm sure it won't make it into the core before #1 happens. We may
introduce it afterwards, but we would have to make it pretty
bulletproof. (Probably only after all steps, and even then it might just
reach 'recommended plugin' status).

John
=:->

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 254 bytes
Desc: OpenPGP digital signature
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20060621/c70059e3/attachment.pgp