[PATCH] ignore log to stdout in benchmarks

[Martin Pool]

On 16 Aug 2006, Robert Collins <robertc at robertcollins.net> wrote:
> On Tue, 2006-08-15 at 18:47 -0500, John Arbash Meinel wrote:
> > 
> > I don't know whether it is better to read the log file, and cache the
> > contents and always delete the log file, or to do it as you have done.
> > My concern is that the log file might go into /tmp, in which case it
> > really needs to be deleted, so that we don't clutter up /tmp.
> > 
> > We already have too much leakage into /tmp. I currently have 95
> > testbzr*.log files in /tmp.
> > 
> > I'm guessing that if a test fails, or if a test explicitly requests
> > that
> > the log file is kept, then it is okay to buffer the contents in
> > memory.
> 
> I was suggesting the following:
> By default:
>  - successful or skipping tests delete the log from disk and throw away
> the contents
>  - failing or erroring tests put the log on disk until get_log_content()
> is called, when they read the file into memory and delete the log from
> disk.
> 
> Tests that need to can request that the log on disk be explicitly kept.

That sounds good to me: keep them if it fails, remove them otherwise,
and let this be overridden.

At the moment there is some undesirable coupling between

a- logs that are kept for debugging a failing test
b- logs that are examined by the test to make assertions about blackbox 
   behaviour, or about how the logging works

There are a few special cases, such as tests that (a) works.
I have a branch outstanding that tries to separate them.  

-- 
Martin