NAK/cmnt: [ACT][PATCH 1/4] UBUNTU: SAUCE: fix git clone failures and setups not being idempotent

[Sean Feole]

See inline below,

On Tue, Jun 8, 2021 at 2:12 AM Krzysztof Kozlowski <
krzysztof.kozlowski at canonical.com> wrote:

> On 07/06/2021 16:02, Po-Hsu Lin wrote:
> > I agree that the second run on the same SUT should not alter the test
> > result.
> >
> > We don't have this failed-with-second-run issue before because we will
> > clean up the autotest directory by rsync'ing a clean tree from our
> > jenkins server no matter it's a freshly deployed system or a manually
> > provisioned one. The old autotest/client/tmp/<test-suite> directory
> > will be gone, the test will start fresh.
> >
> > This is defintely an issue we need to deal with if we want to run tests
> > for multiple times manually on the same SUT (I just clean out the
> > aformentioned tmp directory manually).
>
> I would expect that's regular process for autotest since they
> implemented this setup() caching (being called only once).
>

+1 , Running the test setup() should always be step one. Even if the test
has already been run on the SUT. I feel that is the correct workflow, i
never like the autotest versioning scheme that required
the version to increment `foo+=1`

Autotest is old and EOL. it's no longer maintained, so for the time being
we gotta fix the pieces that do not work correctly.


>
>
> > As discussed on Mattermost, I think this fix is base on the assumption
> > that the change to autotest f6e444df45f45 ("UBUNTU: SAUCE: fix missing
> > setup on second run") is correct.
>
> Mentioned autotest change indeed looks wrong. However this does not make
> this change here wrong, either.
>
> Consider case when test version is bumped, while being executed for
> second time. The setup() will be called and all git clones will fail.
>
> Making setup() idempotent or resistent to existing stuff is good idea IMHO.
>

Moving forward imo, all tests should be written/updated with the intention
that we are not using autotest.

I know that sounds sort of confusing, I'm trying to convey that each of the
tests should be able to effectively and quickly accomplish their goal
without
any added helpers from the autotest client program. I know and well aware
that many of them are tightly intertwined with decorators and functions
that call on the autotest client.
But if the test is actually under review and going through some sort of
triage like Krzsztof has done, then we can take that time to fix this.

I believe each of the tests should indeed have a setup() that is
responsible for ensuring the SUT is correctly staged prior to running the
desired test. regardless of how many executions occur on the SUT

Sean



>
> >
> > With a quick test on the ubuntu_ltp_syscalls test, this will make the
> > test clone the source and build it twice before running the actual
> > syscall tests from LTP. From this fact it looks like the setup() was
> > called twice with one execution.
> >
> > On the other hand, this issue does not exist in libhugetlbfs test,
> > which will do a git clone for an external source as well, therefore
> > it doesn't look like a real fix to me.
>
>
>
> Best regards,
> Krzysztof
>
> --
> kernel-team mailing list
> kernel-team at lists.ubuntu.com
> https://lists.ubuntu.com/mailman/listinfo/kernel-team
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/kernel-team/attachments/20210608/12f17b91/attachment.html>