[LTP] LTP - Include upstart whitebox / blackbox testing API's?

[Garrett Cooper]

2008/12/13 Garrett Cooper <yanegomi at gmail.com>:
> On Fri, Dec 12, 2008 at 6:52 AM, Subrata Modak
> <subrata at linux.vnet.ibm.com> wrote:
>>
>> On Fri, 2008-10-24 at 14:05 +0530, Subrata Modak wrote:
>>> On Thu, 2008-10-23 at 14:26 -0700, Garrett Cooper wrote:
>>> > On Wed, Oct 22, 2008 at 5:44 AM, Subrata Modak
>>> > <subrata at linux.vnet.ibm.com> wrote:
>>> > > Garrett,
>>> > >
>>> > > Is there any headway with upstart developers regarding this initiative.
>>> > > I dug out this mail from my mailbox to find this. Let me know if we can
>>> > > resume this discussion once again.
>>> > >
>>> > > Regards--
>>> > > Subrata
>>> > >
>>> > > On Fri, 2008-06-27 at 19:06 +0530, Subrata Modak wrote:
>>> > >> On Thu, 2008-06-26 at 05:26 -0700, Garrett Cooper wrote:
>>> > >> > Hello LTP gurus (and upstart gurus),
>>> > >> >       As I mentioned before on the upstart-devel list, one of the
>>> > >> > goals of the groups that I'm working with is to bring upstart -- the
>>> > >> > init replacement -- to Cisco's Linux based platform for process
>>> > >> > monitoring and management. As part of that we (my teammates and I)
>>> > >> > were thinking of including whitebox and blackbox tests with LTP (Linux
>>> > >> > test project) to try and unify testing of critical Linux components,
>>> > >> > and also provide deterministic output also with greater visibility in
>>> > >> > the testing community.
>>> > >> >       LTP has a number of whitebox and blackbox tests in place [3],
>>> > >> > most of the whitebox tests being C API's and the blackbox tests being
>>> > >> > shell invocations of Unix commands, as well as a well-defined set of
>>> > >> > test reporting API's and functions already in place.
>>> > >>
>>> > >> Ah!. That reminds me of the testcases for commands in LTP:
>>> > >>
>>> > >> http://ltp.cvs.sourceforge.net/ltp/ltp/testcases/commands/
>>> > >>
>>> > >> I have been merging lots of patches and we were totally engaged with our
>>> > >> white box test cases, that we completely forgot about those black box
>>> > >> test cases, which are of immense help for:
>>> > >>
>>> > >> 1) Increasing code coverage for the kernel,
>>> > >> 2) Testing the actual/mostly-used interfaces to the Linux OS.
>>> > >>
>>> > >> Thanks Garrett for reminding this valuable testcases piece. And the
>>> > >> important point here to make is:
>>> > >>
>>> > >> Writing white box test cases requires fair knowledge of Kernel
>>> > >> Internals, whereas the Blackbox test cases just requires user knowledge
>>> > >> of the OS. With guidance from the Man Pages information, a huge
>>> > >> community of administrators and normal users can write these black box
>>> > >> tests. And they are a huge group of people to count. I need to look into
>>> > >> this seriously from now.
>>> > >>
>>> > >> >       So, my question is two-fold:
>>> > >> >       1. Would the upstart project be willing to work with LTP (via my
>>> > >> > team as a proxy in the beginning) to enter some unit test code and
>>> > >> > other test cases into LTP's test framework / overall testsuite, and
>>> > >> > improve acceptance in the Linux testing community?
>>> > >>
>>> > >> I would be providing you the support with testing on the architectures i
>>> > >> have at my disposal and speedy patch merge to LTP. We definitely need to
>>> > >> do something to increase the code coverage.
>>> > >>
>>> > >> >       2. Would either group be willing to work with my team to help
>>> > >> > maintain these testcases and develop new ones?
>>> > >>
>>> > >> Of course, i will.
>>> > >>
>>> > >> > Thanks,
>>> > >> > -Garrett
>>> > >> >
>>> > >> > PS. Sorry for the cross-posting ; I try not to do this, but
>>> > >> > considering that both groups can benefit from the discussion I wanted
>>> > >> > to involve both.
>>> > >>
>>> > >> Nothing to worry about. When it comes to making Linux better, we need
>>> > >> collaboration on various fronts. The livest example being the work done
>>> > >> by Masatake Yamato from Red Hat in porting Crackerjack´s
>>> > >> (https://sourceforge.net/projects/crackerjack) regression tests to LTP
>>> > >> format. Thanks Garrett for taking this initiative. We need to
>>> > >> collaborate much more with others as well.
>>> > >>
>>> > >> Regards--
>>> > >> Subrata
>>> > >>
>>> > >> >
>>> > >> > 1. LTP -- Linux test project: http://ltp.sourceforge.net/
>>> > >> > 2. Upstart -- init(1) replacement: http://upstart.ubuntu.com/
>>> > >> > 3. LTP cvsweb -- http://ltp.cvs.sourceforge.net/ltp/ltp/ (see docs for
>>> > >> > relevant documentation items, lib/ltp for test lib API's, and
>>> > >> > testcases/commands for existing Linux command blackbox tests).
>>> >
>>> > I haven't followed this up, but to be honest our group using upstart
>>> > has started using Python nose to write testcases for blackbox level
>>> > testing, and it's proven to be largely successful in finding basic
>>> > issues within the provided spec by the upstart folks.
>>> >
>>> > I don't know if the test code can be easily committed back because it
>>> > has Cisco IP -- I'll talk to Sarvi (tech lead) and Corey (the manager)
>>> > about that.
>>
>> Garret,
>>
>> Can we revive this ?
>>
>> Regards--
>> Subrata
>>
>>>
>>> It would be great in such a case.
>>>
>>> >
>>> > As for whitebox testing, we should definitely follow up the intiative
>>> > for using tst_res.
>>> >
>>>
>>> Yes. And as you said, keep the momentum going for having the tst_*
>>> functions under varied programming language. Let it take itś own course
>>> and time, but, we should keep up the gear.
>>>
>>> Regards--
>>> Subrata
>>>
>>> > -Garrett
>
>    No time.
> -Garrett

Ok, I have a bit more time now and it's become an issue Cisco side, so
it's important that this gets done ASAP.

We (my group in Cisco) needs a library that's LGPL2.1/BSD licensed so
we don't violate the GPL, and I didn't get a response from SGI about
relicensing the components, so I'll need to recreate the C library
provided the API's provided on the LTP page.

I'll see about getting this all ironed out in the next two weeks.

Thanks,
-Garrett