Gherkin DSL for testcase description and automation

[Gema Gomez]

+1 to brendand for putting it so clearly.

I am not very keen on rewriting everything twice, we should be creating
test cases that whether manual or automated, are reusable and not need
changing unless the behavior of Ubuntu changes. Otherwise, we are
shooting ourselves on the foot... we will always be catching up.

On the other hand, I am happy to do the intellectual exercise of trying
to imagine what a long test case would look like with Gherkin method,
and figure out if it would work. Can someone have a go at, let's say..
http://testcases.qa.ubuntu.com/Install/DesktopWhole

I think we need to ask ourselves at which level are we creating test
cases here. My take, is that we are trying to do system testing, because
we test on Alpha and Beta releases, and it is too late already for any
other kind of testing. So we are targeting Ubuntu as a whole, either
Desktop or Server. Our test cases should run on any architecture, though
(i386, AMD64, ARM Server whenever it comes along?).

I am happy to contribute to Alex's wiki about how to write test cases
whenever it is in place.

Gherkin.. worth considering, but it doesn't look very promising for the
problem at hand.

Thanks,
Gema


On 09/12/2011 13:51, Руаньяк wrote:
> Hi all,
> 
> 2011/12/9 Alex Lourie <djay.il at gmail.com>:
>>
>>
>> On Fri, Dec 9, 2011 at 12:41 PM, Brendan Donegan
>> <brendan.donegan at canonical.com> wrote:
>>>
>>> Not sure I buy some of the criticisms of the ISTQB definitions, but it's
>>> certainly true that Gherkin encodes the tests in such a way as to make
>>> it possible for tools to parse them, whereas perhaps ISTQB definitions
>>> don't have that. I do think the language ends up looking slightly weird
>>> for manual testers who may not have that much experience with the
>>> software though. Here's a weakness, at least in the definitions given
>>> below:
>>>
>>>  Scenario: TC_CAL-002: Calculate sums
>>>   Given Calculator is in Basic mode
>>>   When I input "1+1"
>>>    And I calculate result <- How?
>>>   Then Result is "2" <- Where?
>>>    And No error occurs <- Where does the error not occur?
> Yes, a poorly written scenario here. Anyway, these steps can be
> re-written in any possible way.
> The most important parts are keywords - Given, When, Then, And, But
> etc. - and step consistency (mostly for automation).
> We can even use bzr+launchpad to store testcases and Launchpad Code
> Merge for testcase review and merge. That would be great to
> find/develop a system, which will present the tests in Gherkin in cool
> sexy-looking way, which will be at the same time a) easy for manual
> testers, b) useful for automated testing team.
> 
>>> I will concede that those things are probably rectifiable though. We
>>> just need to acknowledge that given a.) time and resource constraints
>>> and b.) the very nature of certain tests, we won't be able to automate
>>> everything and whereas ISTQB provides a definition which is probably
>>> easier to understand for the manual tester but not so much for tools,
>>> Gherkin DSL provides one that is easier for tools to understand but
>>> maybe vulnerable to providing definitions that are more difficult for
>>> manual testers.
>>>
>>
>> So, in fact, we kind of need to decide who should we cater more with the
>> test cases - manual
>> testers or automatic ones? I think we need to handle manual ones now, and
>> try to move as much
>> as possible for automation later. So ISTQB definition would get my +1 for
>> the nearest future.
> I guess, Gherkin tests are not that hard to understand and they have
> another feature - they are much easier to create new ones (simply as
> they use less symbols ;)
> If we use Gherkin DSL for both manual and automated tests, we will
> avoid converstion from manual tests to automated unit-test - and this
> takes a lot of time, as I've been developing some tests for Mago
> recently.
> Simply, automation team just have to write actions for each step, for instance:
> @Given('Calculator is in (.+) mode')
> def switch_to_mode(mode):
> <code for switching Gcalc mode>
> @When('I click "(.+)" button')
> def click_button(button):
> <code for clicking "button">
> etc. etc.
> 
>>
>>>
>>> It also seems that it's background is unit test generation and I'd be
>>> curious to see what a larger test case written in this syntax would look
>>> like.
>>>
>>
>> And so would I :-)
> I havn't written many tests using Gherkin, here are some Mago tests
> rewritten - lp:~roignac/+junk/lettuce_experiment
> --
> Vadim Rutkovsky
>