Gherkin DSL for testcase description and automation

[Gema Gomez]

Hi,

I agree with Charlie, but I think it is worth discussing this idea and
the reasons why (imo) this way of developing test cases wouldn't work
in our case.

When I was reading about this method in this thread it altogether
reminded me of the days when we were trying to move the development of
Symbian OS to agile methodology, we were asked to plan in a way that
tasks had this format (also called stories in SCRUM):

> As a <USER> I want <FUNCTION> so that <BUSINESS VALUE> (see
http://www.thedailyscrum.co.uk/post/86171940/effective-story-writing-in-scrum)

We underwent 2 years of intensive work trying to figure out how to do
testing in such environment and trying to figure out how system
testing, integration testing and unit testing fitted in this world.
And believe you me, the Ubuntu dev world is much more complex than
Symbian ever was.

The first problem was that SCRUM doesn't really care about integration
and system testing, it cares only about unit testing. The only reason
I have been able to come up with for this is that SCRUM was initially
developed for small projects where developers could talk to each other
on daily basis plus the QA role wasn't a separate one. Only unit
testing was required.

I learned that for a big project this is not enough and you still need
QA people and testers so that you get the integration of the product
and the packaging of it right.

But Agile was cool and we were forced to move to it. We did, and we
implemented the lower layers of it, we wrote stories, tracked backlogs
and did daily stand ups like champions. When it came to write test
cases, ISTQB format still made sense for the whole company and was
widely used. Even for unit testing we were using that when tests were
written in something more formal than lines of code.

The Gherkin method seems to be an evolution of Xtreme Programming or
Agile (en.wikipedia.org/wiki/Behavior_Driven_Development). It is used
for doing Behaviour Driven Development, i.e. the developer writes such
test case, then implements the test code or generates it with a
cucumber (I haven't figured out this bit but as funny as it sounds, it
seems to be the way... a code generator from stories, call me old
school, but I still prefer to write code with an editor) and then,
when the test case fails, they go to the code and create the piece of
software that will make that test case pass. That is unit testing,
doesn't matter which language you use to write the test case.

As we've seen with the example that roignac put together (thanks!),
the language doesn't scale very well in terms of readability. Despite
making the test case logically very accurate, it doesn't really make
the manual tester's life easier, and at the end of the day, we are
trying to attract people to do more manual testing. Irrespective of
the subject having or not having a degree. It probably works for some
programmers, but definitely doesn't work for testers (imho).

In a diverse environment like ours where developers use different
development life cycles (even though they all try to fit in the 6
month bound releases), I am not sure how we would be able to push this
forward nor sure if we would want to do that if it is going to be more
complicated than just getting the wording of the test cases right and
accurate enough so that in front of the same behavior of the system,
two different people fail the test in the same way (consistency).

Those are my thoughts. More ideas?

Thanks,
Gema

On 10/12/2011 14:43, Charlie Kravetz wrote:
> On Sat, 10 Dec 2011 12:35:21 +0200 Руаньяк <roignac at gmail.com>
> wrote:
> 
>> Hi,
> 
>> 2011/12/10 Alex Lourie <djay.il at gmail.com>:
>>> I have the following questions:
>>> 
>>> 1. How long does it take to write something like that for
>>> someone who's not a programmer and have no idea about Cucumber
>>> or unit testing in general?
>> I guess, it should not take much time. The only rule to be
>> followed is 'Action after When, Expected result after Then,
>> Concatenation is And'. Maybe, someone inexperienced in Gherkin
>> could measure the time and post results here? For instance, this
>> conversion of DesktopWhole case took about 10 min. for me.
> 
> As someone completely ignorant of Gherkin, it took me over 10
> minutes to read through this test case. In reading through, I
> attempted to picture most of the steps, and found the flow doesn't
> work right here. I test a lot of images, and I test images daily.
> This test is much too specific. Also, it does not read well, from
> an english language user. It reads much like something written by
> lawyers reads to the lay person.
> 
> I don't think I could actually write such a test case, in less than
> at least an hour, and perhaps that is not enough enough time.
> 
> 
>>> 2. How would one execute this in LiveCD environment?
>> Manual testers may use this as a usual test case. Whenever bug 
>> appears, the tester may break the instruction, e.g. --- When I
>> double click 'Install Ubuntu' icon Then Ubuquity starts  #here we
>> can check main window etc.
> 
>> Result: Ubiquity crashes with error 'ImportError:..." ---
>>> 3. Is it possible to run something like that in automated VM
>>> environment?
>> Yes, as there is nose framework plugin Freshen (see 
>> https://github.com/rlisagor/freshen), which generates xUnit
>> reports from feature files. Another option is Lettuce (see 
>> https://github.com/gabrielfalcao/lettuce)
> 
>> All steps should be defined (using ldtp) in definition file(s),
>> which contain the actual automation code for each step, something
>> like this: -- from steps.py -- @step("I double click 'Install
>> Ubuntu' icon"): def double_click_on_ubuntu_icon: 
>> ltdp.double_click("btnInstallUbuntu");
> 
>> Then, using steps definitions and scenarios, automation suite
>> will do the following: a) create a virtual machine b) start up
>> from live cd c) fire up LDTP server on virtual machine d) connect
>> to LDTP server and execute steps from scenarios e) collect xUnit
>> results and post them to CI (Jenkins)
> 
>> I haven't tried this at home, so we should contact automation
>> guys, as they are using something similar for automation daily
>> precise images -- Vadim Rutkovsky
> 
> 
> While I realize that many of you are degree holders, community
> members as a rule are users who decided to get involved in helping
> their favorite software. For some, this is a way to give back for
> the ability to use the software.
> 
> For the common user, these types of tests are not actually
> something they can follow easily. The test cases as written, using
> steps, are usable by all of us.
> 
>