[LTP] LTP - Include upstart whitebox / blackbox testing API's?

[Subrata Modak]

On Fri, 2008-10-24 at 14:05 +0530, Subrata Modak wrote:
> On Thu, 2008-10-23 at 14:26 -0700, Garrett Cooper wrote:
> > On Wed, Oct 22, 2008 at 5:44 AM, Subrata Modak
> > <subrata at linux.vnet.ibm.com> wrote:
> > > Garrett,
> > >
> > > Is there any headway with upstart developers regarding this initiative.
> > > I dug out this mail from my mailbox to find this. Let me know if we can
> > > resume this discussion once again.
> > >
> > > Regards--
> > > Subrata
> > >
> > > On Fri, 2008-06-27 at 19:06 +0530, Subrata Modak wrote:
> > >> On Thu, 2008-06-26 at 05:26 -0700, Garrett Cooper wrote:
> > >> > Hello LTP gurus (and upstart gurus),
> > >> >       As I mentioned before on the upstart-devel list, one of the
> > >> > goals of the groups that I'm working with is to bring upstart -- the
> > >> > init replacement -- to Cisco's Linux based platform for process
> > >> > monitoring and management. As part of that we (my teammates and I)
> > >> > were thinking of including whitebox and blackbox tests with LTP (Linux
> > >> > test project) to try and unify testing of critical Linux components,
> > >> > and also provide deterministic output also with greater visibility in
> > >> > the testing community.
> > >> >       LTP has a number of whitebox and blackbox tests in place [3],
> > >> > most of the whitebox tests being C API's and the blackbox tests being
> > >> > shell invocations of Unix commands, as well as a well-defined set of
> > >> > test reporting API's and functions already in place.
> > >>
> > >> Ah!. That reminds me of the testcases for commands in LTP:
> > >>
> > >> http://ltp.cvs.sourceforge.net/ltp/ltp/testcases/commands/
> > >>
> > >> I have been merging lots of patches and we were totally engaged with our
> > >> white box test cases, that we completely forgot about those black box
> > >> test cases, which are of immense help for:
> > >>
> > >> 1) Increasing code coverage for the kernel,
> > >> 2) Testing the actual/mostly-used interfaces to the Linux OS.
> > >>
> > >> Thanks Garrett for reminding this valuable testcases piece. And the
> > >> important point here to make is:
> > >>
> > >> Writing white box test cases requires fair knowledge of Kernel
> > >> Internals, whereas the Blackbox test cases just requires user knowledge
> > >> of the OS. With guidance from the Man Pages information, a huge
> > >> community of administrators and normal users can write these black box
> > >> tests. And they are a huge group of people to count. I need to look into
> > >> this seriously from now.
> > >>
> > >> >       So, my question is two-fold:
> > >> >       1. Would the upstart project be willing to work with LTP (via my
> > >> > team as a proxy in the beginning) to enter some unit test code and
> > >> > other test cases into LTP's test framework / overall testsuite, and
> > >> > improve acceptance in the Linux testing community?
> > >>
> > >> I would be providing you the support with testing on the architectures i
> > >> have at my disposal and speedy patch merge to LTP. We definitely need to
> > >> do something to increase the code coverage.
> > >>
> > >> >       2. Would either group be willing to work with my team to help
> > >> > maintain these testcases and develop new ones?
> > >>
> > >> Of course, i will.
> > >>
> > >> > Thanks,
> > >> > -Garrett
> > >> >
> > >> > PS. Sorry for the cross-posting ; I try not to do this, but
> > >> > considering that both groups can benefit from the discussion I wanted
> > >> > to involve both.
> > >>
> > >> Nothing to worry about. When it comes to making Linux better, we need
> > >> collaboration on various fronts. The livest example being the work done
> > >> by Masatake Yamato from Red Hat in porting Crackerjack´s
> > >> (https://sourceforge.net/projects/crackerjack) regression tests to LTP
> > >> format. Thanks Garrett for taking this initiative. We need to
> > >> collaborate much more with others as well.
> > >>
> > >> Regards--
> > >> Subrata
> > >>
> > >> >
> > >> > 1. LTP -- Linux test project: http://ltp.sourceforge.net/
> > >> > 2. Upstart -- init(1) replacement: http://upstart.ubuntu.com/
> > >> > 3. LTP cvsweb -- http://ltp.cvs.sourceforge.net/ltp/ltp/ (see docs for
> > >> > relevant documentation items, lib/ltp for test lib API's, and
> > >> > testcases/commands for existing Linux command blackbox tests).
> > 
> > I haven't followed this up, but to be honest our group using upstart
> > has started using Python nose to write testcases for blackbox level
> > testing, and it's proven to be largely successful in finding basic
> > issues within the provided spec by the upstart folks.
> > 
> > I don't know if the test code can be easily committed back because it
> > has Cisco IP -- I'll talk to Sarvi (tech lead) and Corey (the manager)
> > about that.

Garret,

Can we revive this ?

Regards--
Subrata

> 
> It would be great in such a case.
> 
> > 
> > As for whitebox testing, we should definitely follow up the intiative
> > for using tst_res.
> > 
> 
> Yes. And as you said, keep the momentum going for having the tst_*
> functions under varied programming language. Let it take itś own course
> and time, but, we should keep up the gear.
> 
> Regards--
> Subrata
> 
> > -Garrett
> 
> 
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> Ltp-list mailing list
> Ltp-list at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/ltp-list