Ubuntu Certified Professionals

[Alan McKinnon]

On Friday 07 April 2006 21:21, Daniel Carrera wrote:
> Alan McKinnon wrote:
> > Red Hat has
> > been asked many times to show the psychometric validity of their
> > exams, and to date they have not been able to do so.
>
> I'm not familiar with RH's exam, but it may be that making the exam
> "psychometically measurable" would also make it less useful.

I'm not an expert on RH either so I don't speak gospel where they are 
concerned, but consider this - do RH really care about having their 
test verifiable across all domains? In essence they are answering one 
simple question: Has this dude demonstrated that he can use OS vx.y 
to do specific useful stuff? Answer: Well, we asked him to set up a 
webserver with valid SSL certificates, he did it and it worked in the 
browser so the answer is yes.

Very neat, very clean and the best approach if the aim is to certify 
people on specific software on a specific OS version. You run into 
trouble with it if the problem domain is wide, or even worse when 
it's unbounded like as in "Linux competency".

Don't get me wrong, I'm not anti-RH. I'm actually a fan (they GPL 
everything - that counts for a lot) but I do get frustrated when 
people at large claim things for their test that are just not true. 
It's like the frustration I feel when some folk claim that Linux 
*cannot* have viruses - technically that's not true and most of us 
here know why.

> > Research shows that practical tests are influenced by the
> > testee's mood - having a bad day can ruin your exam results. But
> > multiple choice by measurement doesn't suffer from this
>
> Yes, multiple choice test absolutely suffers from this. I can say
> that because I've been a testee many times. If I can't concentrate
> because my gf broke up with me or I have a cold that'll hurt me
> just as much on a multiple choice.

My personal anecdotal evidence differs :-) In school I used to 
consistently score $MARK>=B+ for technical stuff and $MARK='awful' 
for Latin no matter how much I studied or how confident I felt or how 
I was feeling on the day. 

Large scale studies seem to indicate that by and large most folk are 
unaffected, but people differ so YMMV. The topic is still somewhat 
controversial and it's real hard to set up rigorous studies.

> >>You can't compare two people writing different multiple choice
> >>exams and prove you are comparing apples and apples.
> >
> > But you can, using Angoff Standards Setting. First, the testee
> > profile is defined, then every question in the database is rated
> > by subject matter experts as to what % of testees are expected to
> > get it right. The exam objectives themselves are assigned weights
> > according to their relative importance. This gives a common base
> > from which to start.
>
> [snip]
>
> INGOTs has 5 staff for the UK. 1 clerk, 1 techie (that's me) and 3
> moderators (one of the moderators designs the criteria). This
> covers 2600 pupils in the UK distributed over 50 academies. That's
> thanks to the large network of assessors (whom we train) who do
> most of the work.

Now that's lean. There's nothing like getting those who benefit most 
from the program to do the majority of the contribution, heh? Kind of 
like OSS itself...

> > Historically, LPI exams delivered computer based style cost $100,
> > and more than half of that goes to Prometric or Thomson Vue.
> > Paper based exams cost $25 and that includes the cost of
> > couriering them all over the world. For security reasons, exams
> > are printed and marked only in Canada, hence the need to courier
> > stuff.
>
> Who are Prometric and Thomson Vue? Why do they take such a big
> chunk of the exam? I guess that the $100 is not bureocracy after
> all...

They are the large computer based tending vendors who deliver exams 
like MCSE etc etc. Colleges, training houses and other institutions 
license with them, they set up a testing room with 20 or so pcs and 
candidates do their exams online.

With a world-wide program you need a reliable testing vendor that's 
present in most places, which leaves only two realistic choices - 
Prometric and Thomson Vue. If you want to use them to deliver a 
program, you pretty much have to go along with their pricing. And 
they don't come cheap. I know that other alternatives were tried by 
various exam bodies in the last 5 years or so but none seem to have 
succeeded.

Hand written (dead tree) exams get around this nicely. As long as the 
logistics are done properly, you can test hundreds of people in a day 
just like Universities do and the price drops considerably. The 
downside is you get to wait three weeks for the results just like 
university exams. 

-- 
Alan McKinnon
alan at linuxholdings dot co dot za
+27 82, double three seven, one nine three five