THE LOEBNER PRIZE COMES TO THE UK FOR THE FIRST TIME

On April 28th 1997 in New York the Annual Loebner Competition was won by a program called CONVERSE, entered by Intelligent Research Ltd. of London, and designed and largely written by members of the Natural Language Processing research group in the Computer Science Department of the University of Sheffield [1].

The Loebner Prize Medal is awarded annually to the designer of the computer system that best succeeds in passing a variant of the Turing Test, in which human judges communicate with workstation and try to decide which of the systems is a program and which a person. the winning program is the one the judges are least able to distinguish from the human interlocutors also taking part. Complex competition rules control typing speeds etc.--so that the machine entries do not give themselves away by typing too fast!

The competition is overseen by the ACM, the main US organization for computer professionals, and for the last two years there has been no domain restriction on what can be talked about---and programs entering must in principle be prepared to talk about anything. CONVERSE has the personality of Catherine, a 26-year-old journalist from New York, born in the UK, who can discuss about 60 subjects including Christmas, shopping, travelling, the royal family, crime, abortion etc. CONVERSE had strong views on the lesbian couple Bill Clinton had welcomed to the White House the night before the competition, and of course on Clinton himself. It narrowly beat out last year's winner, an Australian program that claimed to be a 14-year old girl marooned on a desert island and appealing for help over the World Wide Web.

Competition included American, Canadian and Australian programs and it is the first time it has been won by a British team. It is hoped to adapt the underlying method to areas needing natural dialogue with computers like instructions on using software, or counselling certain classes of medical patients. David Levy, Director of Intelligent Research, claimed that in twenty years people will be falling in love with these programs. They are certainly and more stimulating than Tamagochi pets.

The efforts to make a program converse go back a long way in Artificial Intelligence, and Colby's PARRY program of the 1970s was the one we had in mind to beat--though that was simulating a paranoid patient, which made the situation a bit different, as its crazier utterances could always be taken as symptoms of paranoia. Usually, too, people want to connect the notion back to Turing's famous paper of 1950 where he considered what it would be for a machine to think. He considered this a pointless philosophical question, and proposed something like what is now called the Turing test: a set up with typing to a distant entity but, and this is important, Turing thought the questions ought to be as to whether the unseen entity was a man or a woman--a tricky, but qiite different, question. Then he thought, a machine could be smuggled in as a participant and the issue would be whether the human noticed the swap.

The key point here, in the original Turing set up, is that the questioner is never asking "Are you a machine", explicitly or implicitly. That is important, because it is not a question we ever ask in real conversations--we just assume our interlocutor is human IF they talk about, say, men and women in a normal way.

So, the Loebner competition is not the real Turing test, but the modern misunderstood version, where you ARE asking, directly or indirectly, if the other entity is a machine. Direct questions are useless, of course: CONVERSE, if asked if it's a machine, will say something like ""Well, that's for you to find out--don't ask silly questions!".

No one claims CONVERSE thinks: it is better to stick with Turing's own point of view that that is not a scientific question--let us get a really good imitation and then think what to say about it. CONVERSE was built to embody some new ideas on how to do the task: basically that, like people, it should have something it wants to say and not just field input passively as ELIZA used to do (and PARRY did not). It also has access to large linguistic data bases--one is a huge thesaurus called WordNet---and it knows about the meanings of about a thousand proper nouns, so you can ask who Mozart or Al Gore or Churchill is. It also "knows" a range of facts about itself and others and can adapt these as the conversation proceeds.

Sometimes it says surprisingly serendipitous things we had not predicted, as all good AI programs surprise their inventors. In the future, There will be many applications of technology like this: in hospitals, patients may want calm advice and counselling from a program that "understands" their condition and can give personalised advice, diets, explain risks and outcomes, and even try to get consent to planned treatments--which would be a nice reversal of the Turing test, for it would be the machine judging if the patient understood the planned operation. Old and lonely people may soon want to talk to (possibly, furry, 5-pound weight) companions who know who they are and remember their stories and laugh at their jokes.

More details on past and future competitions and full transcripts of the 1997 Loebner competition can be found on the following Web site: http://acm.org/~loebner/loebner-prize.htmlx

[1] Levy, D., Batacharia, B., Catizone, R., Krotov, A., and Wilks, Y. (1997). CONVERSE -- a companion with potential. In Proceedings of 1st Workshop on Human-Computer Communication, Bellagio, Italy. To appear in Machine Conversations (ed. Y. Wilks), Amsterdam: Kluwer.1


[CLUK home] Last modified: 10th September 1997
Mark Lee <M.Lee@dcs.shef.ac.uk>