Ubuntu Certified Professionals

[Alan McKinnon]

On Saturday 08 April 2006 04:57, Bry Melvin wrote:
> >Alan McKinnon wrote:
> >>My personal anecdotal evidence differs :-) In school I used to
> >>consistently score $MARK>=B+ for technical stuff and
> >> $MARK='awful' for Latin no matter how much I studied or how
> >> confident I felt or how I was feeling on the day.
> >
> >Your experience doesn't suggest that multiple choice tests aren't
> > affected by your mood. It suggests that you have an aptitude for
> > technical stuff and not for Latin :)
>
> Just an "outsider's"  two cents worth: I have no first hand
> knowledge of Computer professional test developement, but spent a
> large part of my career developing tests in another technical area.
> So this thread hit an area of interest for me.
>
> A Good validation process is what makes or breaks the tests.
> OTOH I can't see a written without practical evaluation as
> adequate.
>
> I come from a field that B+ (85%) is minumum written passing grade
> however and a serious mistake in practice is often lethal. I have
> washed out many who could pass a written that couldn't actually do
> the job however, often giving thanks that my student didn't kill me
> that day! :-)

Are you a flight instructor perhaps? I know a teeny bit about that 
field, enough to know that I'd insist a student get 85%+ on a written 
test *and* 100% perfect on a practical one before I'd give him the 
keys to a plane. If he's carrying passengers, make that 5 practicals 
in a row all rated 100% including one done at 3am, plus 1000 hours 
flying time experience.

There are many similar fields like medicine, firemen, law enforcement, 
driver's license, legal, military where the cost of a mistake is 
high. Because of that cost, and because the technology in those 
fields is relatively static, they all have well-established protocols 
laid down on how to the job, and everyone working in the field knows 
where to get them. The practical test is testing adherence to the 
protocol. So far so good, it's a workable system and served us well 
for many years.

Here's the catch: please show me the established laid down protocols 
for *nix system administration. There isn't one - the subject is more 
of an art than a science and relies on deep understanding of the 
underlying technical fundamentals. The culture of the subject 
actively encourages experimentation and rigid protocols are 
discouraged.

Basic practical tests are easy to form, and I do them all the time. 
They are the exercises at the end of a chapter in the courseware. On 
an exam, I could ask the student to add a user with a specific UID 
and create a home dir. Watching him type 'useradd -m -u $UID' tells 
me nothing extra about skill level than getting him to pick the right 
answer from a list. Either way I know for sure the student knows 
useradd, -u and -m.

Now get more complex and make the question: Set up an HTTP server and 
install a valid SSL certificate; disallow connections from a 
specified list of IP blocks and from a specified list of domains; and 
just for fun throw in authentication via ldap. 

Sounds like a good test, but it has problems: which httpd server? 
Which provider provided the certificate? Would you use apache itself 
to allow and deny connections, or use a separate firewall and if so, 
which one? How about the web server in a DMZ protected by a FreeBSD 
firewall (not a bad idea in real life). The problem is that there is 
no established protocol to do any of these things, and if I were to 
suggest one I'd be (rightly) shot down in flames. So what is this 
question actually testing? You don't know, because the testing domain 
has not been defined. Not to mention the risk of false negatives.

Don't get me wrong, I'm not anti-practical tests. I've just never seen 
one for Linux that is a cost-effective implementation, that can 
clearly state exactly what is being tested without ambiguity and gray 
areas, and is not subject to the opinion and whim of the assessor. If 
you know how to overcome these problems in a cost effective efficient 
way, I'd love to hear them.

> If mood seriously would affect your performance level either in
> testing or at work, maybe you are in the wrong career?

Good point, but that's a job for career counseling, not 
examination :-)

> As far as test validation in computer systems I see the practical
> MORE important possibly because the technology changes too rapidly
> to be able to properly statistically validate the written (multiple
> question) test. That form is the best in technical areas as it
> nearly eliminates subjectivity however. Basically you would have to
> validate by throwing away questions an expert test group missed and
> push ahead....then elimanate questions from the test base that the
> top scorers got wrong and the bottom scorers all got right. This is
> a faulted process that in time would generally end up with too many
> of the more difficult questions eliminated...
> The test would gradually deteriorate in value.

Yes it would, but only if the pool of questions is not updated 
regularly. You also need a way to test new questions before release 
and to do enough of them in a short period of time to be 
statistically meaningful. LPI's solution is to seed the current exam 
with 15 extra new questions and extend the time from 1.5 hours to 2. 
Candidates don't know which questions are the 15 but the database 
does, and they do not contribute to the testee's score. 

Also keep in mind what is being tested here. The exam does not test 
for expert knowledge, it is an attempt to certify someone who has the 
minimum knowledge required to get something useful done, where you 
can reasonably trust the person not to break the box through 
ignorance. There are other better ways of testing expert knowledge 
like having your name in the ChangeLog many times, continuous 
on-the-job observation and assessment, and good old reputation.

> Much like some of the FAA's current tests. :-)
>
> BTW I didn't do that hot with Latin either :-)

It wasn't all bad - at least now English grammar mostly makes sense. 
And I still can't cope with Julius Caesar's ramblings and sentences a 
whole paragraph long with the verb right at the end...

-- 
Alan McKinnon
alan at linuxholdings dot co dot za
+27 82, double three seven, one nine three five