How to build a robot head

And now for something completely different, a tutorial on how to make a controllable robot head. “But,” I imagine you thinking, “aren’t you an A.I. guy? Since when do you have expertise in robotics?” I don’t, and that’s why you can make one too.
(Disclaimer: I take no responsibility for accidents, damaged equipment, burnt houses, or robot apocalypses as a result of following these instructions)


What you need:
• A pan/tilt IP camera as base (around $50)
• A piece of wood for the neck, about 12x18mm, 12 cm long
• 2mm thick foam sheets for the head, available in hobby stores
• Tools: Small cross-head screwdriver, scissors and/or Stanley knife, hobby glue, fretsaw, drill, and preferably a soldering iron and metal ruler
• (Optional) some coding skills for moving the head. Otherwise you can just control the head with a smartphone app or computer mouse.

Choosing an IP camera
Before buying a camera, you’ll want to check for three things:
• Can you pan/tilt the camera through software, rather than manually?
• Is the camera’s software still available and compatible with your computer/smartphone/tablet? Install and test software from the manufacturer’s website before you buy, if possible.
• How secure is the IP camera? Some cheap brands don’t have an editable password, making it simple for anyone to see inside your home. Check for reports of problematic brands online.
The camera used in this tutorial is the Eminent Camline Pro 6325. It has Windows software, password encryption, and is easy to disassemble. There are many models with a similar build.

Disassembling the camera
Safety first: Unplug the camera and make sure you are not carrying a static charge, e.g. by touching a grounded radiator.
Start by taking out the two screws in the back of the orb, this allows you to remove its front half. Unscrew the embedded rectangular circuit board, and then the round circuit board underneath it as well. Now, at either side of the orb is a small circle with Braille dots on it for grip. Twist the circle on the wiring’s side clockwise by 20 degrees to take it off. This provides a little space to gently wiggle out the thick black wire attached to the circuit board, just by a few centimetres extra. That’s all we’ll be doing with the electronics.

Building the neck
tut2We’ll attach a 12cm piece of wood on the back half of the orb to mount the head on. However, the camera’s servo sticks out further than the two screw holes in the orb, as does a plastic pin on the axle during rotation. Mark their locations on the wood, then use a fretsaw to saw out enough space to clear the protruding elements with 3 millimetres to spare. Also saw a slight slant at the bottom end of the wood so it won’t touch the base when rotating. Drill two narrow screw holes in the wood to mirror those in the orb half, then screw the wood on with the two screws that we took out at the start.

Designing a headtutplanYou’ll probably want to make a design of your own, I looked for inspiration in modern robotics and Transformers comic books. A fitting size would be 11 x 11 x 15cm, and a box shape is the easiest and sturdiest structure. You’ll want to keep the chin and back of the head open however, because many IP cams have a startup sequence that will swing the head around in every direction, during which the back of the head could collide with the base. So design for the maximum positions of the neck, which for the Camline Pro is 60 degrees tilt to either side. You can use the lens for an eye, but you can just as well incorporate it in the forehead or mouth. Keep the head lightweight for the servo to lift, maximum 25 grams. The design shown in this tutorial is about 14 grams.

Cutting the head
Cut the main shapes from coloured foam sheets with scissors or a Stanley knife. I’ve chosen to have the forehead and mouthplate overlap the sheet with the eyes to create a rigid multi-layered centrepiece, as we will later connect the top of the wooden neck to this piece. The forehead piece has two long strands that will be bent backwards to form the top of the head. I put some additional flanges on the rectangular side of the head to fold like in paper craft models. Although you can also simply glue foam sheets together, folded corners are sturdier and cleaner. The flanges don’t have to be precise, it’s better to oversize them and trim the excess later.

Folding foam sheetstut4
To fold a foam sheet, take a soldering iron and gently stroke it along a metal ruler to melt a groove into the foam, then bend the foam while it’s hot so that the sides of the groove will stick together. It’s easy to burn straight through however, so practice first. It takes about 2 or 3 strokes and bends to make a full 90 degree corner.

Putting your head togethertut5
To curve foam sheets like the faceplate in this example, you can glue strips of paper or foam on the back of the sheet while holding it bent. After the glue dries (5-10 minutes), the strips will act like rebar in concrete and keep the foam from straightening back out. Whenever you glue sheets together at perpendicular angles, glue some extra slabs where they connect to strengthen them and keep them in position. Add a broad strip of foam at the top of the head to keep the sides together, and glue the two strands that extend from the forehead onto it. Mind that I made the forehead unnecessarily complicated by making a gap in it, it’s much better left closed.

Mounting the head
tut6Once the head is finished, make a cap out of foam sheet that fits over the tip of the neck, and glue the cap to the inside of the face at e.g. a 30 degree angle. To attach the camera lens, note that the LEDs on the circuit board are slightly bendable. This allows you to clamp a strip of foam sheet between the LEDs and the lens. Cut the strip to shape and glue it behind one eyehole, then after drying push the LEDs over it and clamp them on gently. The easiest way to make the other eye is to take a photograph of the finished eye, print it out mirrored on a piece of paper, and glue that behind the other eyehole.

This particular camera has night vision, which will suffer somewhat from obscuring the LEDs. In addition, you may want to keep the blue light sensor on the LED circuit board exposed, otherwise you’ll have to toggle night vision manually in the camera’s software.

Controlling the head
13finalNow you can turn the head left, right, up and down manually through the app or software that comes with your camera, and use it to look around and speak through its built-in speaker. However, if you want to add a degree of automation, you have a few options:

1. If you are not a programmer, there is various task automation software available that can record and replay mouse clicks. You can then activate the recorded sequences to click the camera’s control buttons so as to make the head nod “yes” or shake “no”, or to re-enact a Shakespearean play if you want to go overboard.

2. If you can program, you can simulate mouse clicks on the software’s control buttons. In C++ for instance you can use the following code to press or release the mouse for Windows software, specifying mouse cursor coordinates in screen pixels:

void mouseclick(int x_coordinate, int y_coordinate, bool hold) {
SetCursorPos(x_coordinate, y_coordinate);
INPUT Input = {0};  Input.type = INPUT_MOUSE;
if(hold == true) {Input.mi.dwFlags = MOUSEEVENTF_LEFTDOWN;}
if(hold == false) {Input.mi.dwFlags = MOUSEEVENTF_LEFTUP;}
SendInput(1, &Input, sizeof(INPUT));

3. For the Camline Pro 6325 specifically, you can also directly post url messages to the camera, using your programming language of choice, or pass them as parameters to the Curl executable, or even just open the url in a browser. The url must contain the local network IP address of your camera (similar to the underlined example below), which you can retrieve through the software that comes with the camera. The end of the url specifies the direction to move in, which can be “up”, “down”, “left”, “right” and “stop”.

Have fun!
How much use you can get out of building a robot head depends on your programming skills, but at the very least it’s just as useful as a regular IP camera, but much cooler.


How to summarize the internet

An ironically long article about a summariser browser add-on.

Introductory anecdote:
Due to my interest in artificial intelligence I can’t help but get exposed to online articles about the subject. But as illustrated in the previous article*, this particular field is flooded with speculative futurism, uninformed opinions and sheer clickbait, wasting my time more often than not.

But I also happen to be an amateur language programmer, so I can do something about it. I spent years developing an A.I. program that can comprehend text through grammar and semantics, and I figured I might as well put it to use. So I had added a function that would read whatever document was on my screen, filter out all unimportant sentences, and show me the remainder. It worked pretty well, and required surprisingly few of the A.I.’s resources. Now, I’ve ported this summarisation function to a browser add-on, so that everyone can summarise online articles at the click of a button:

banner_chrome       banner_firefox

Problem statement: Statistics are average
Document summarisers do of course already exist, and their methods are inventively inhuman:

• The simplest method, used in e.g. SMMRY, counts how often each word occurs in the text, and then picks out sentences that contain the most-occurring words, which are presumably the main topics. Common words like “the” should of course be ignored, either with a simple blacklist, or with another word-counting technique by the confusing name “Term Frequency – Inverse Document Frequency”: How frequently a word occurs in the text versus how frequently it occurs in the English language.
Another common method looks at each paragraph and picks out one sentence that has the most words in common with its neighbouring sentences, therefore covering the most of the paragraph’s subject matter. Sentence length is factored in so that it won’t just always pick the longest sentence.
• The most advanced method, “Latent Semantic Analysis”, picks out sentences that contain frequently occurring, strongly associated words. i.e. words that are often used together in a sentence are presumably associated with one and the same topic. This way synonyms of the main topics are also covered.

In my experiences however I observed one problem with these statistical methods: Although they succeeded in retrieving an average of the subject matter, they tended to omit the point that the writer was trying to make, and that is the one thing I want to know. This oversight stands to reason: A writer’s conclusion is often just one or two sentences near the end, so its statistical footprint is small, and like an answer to a question, it doesn’t necessarily share many words with the rest of the article. I decided to take a more psychological approach. Naturally, I ended up re-inventing a method that dates all the way back to 1968.

The technical part: Skip if you don’t need to know.
Before one can start, even pinpointing the main text from a website’s html code is a challenge that developers have struggled with since forever, as custom formatting threw consistency out the window. Ideally one would just retrieve the text inbetween the html’s “article” tags, but these are not always present and still contain menus and ads. Tools like Mozilla’s Readability or Beautiful Soup are recommended, but I use javascript’s Regexp functions to strip the html of hidden comments, javascript code, everything after the first footer tag, and then header, nav, style and form tags. This order is important, as hidden comments may contain obsolete code that could throw off the rest. I split the remainder into html tags and regular pieces of text with this simple bit of code:
var parts = text.split(/<[/]?[a-zA-Z][^>]*?>/gi);
Then I go over all parts, regarding “div”, “p” and “br” tags as line breaks, and apply the following rules:
• If more than 20 words follow a “div” layout tag, that’s where the main text starts.
• Any line < 12 words that doesn’t end in a period is a header, image description, blockquote, or advert.
• Any line < 12 words beginning with a typical imperative like “log”, “read”, or “subscribe” can be ignored.
• Any hyperlink that spans an entire line is a menu, advert, or link to another article.
• Two consecutive hyperlinks not separated by regular words are part of a horizontal menu.
Once all parts of the main text are identified, they can be pieced together to form one clean block of text.
The final preparation is to divide the text into sentences. This too is deviously more complicated than you’d think. Consider the many different uses of punctuation:
“Mr. Watson of I.B.M. bought for 1.2 billion at 10:00 a.m. today.”
I already had experience with this in my original A.I. project, but to port it to javascript I had to condense it into the most ridiculous line of code that I’ve ever written:

This splits the text only after words of more than one letter with at least one vowel, followed by punctuation and a space. That way initials, abbreviations, acronyms and decimals are ignored. This does not cover voweled abbreviations like “doc. Brown” however, which technically would require a list of all abbreviations in every field.

A writer’s approach to summarisation
My target for the summariser add-on was a combination of two things: It should extract what the writer found important, minus what I find unimportant. Unimportant being things like introductions, asides, examples, inconcrete statements, speculation and other weak arguments.

Word choice
While writing styles vary, all writers choose their words to emphasise or downtone what they consider important. Consider the difference between “This is very important.” and “Some may consider this important.” In a way the writer has already filtered the information for you. With this understanding, I set the summariser to look for several types of cues in the writer’s choice of words:

• Examples: “e.g.”, “for instance”, “among other”, “just one of”
• Uncertainty: “may”, “suppose”, “conjecture”, “question”, “not clear”
• Commonly known: “standard”, “as usual”, “of course”, “obvious”
• Advice: “recommendation”, “require”, “need”, “must”, “insist”
• Main arguments: “problem”, “goal”, “priority”, “conclude”, “decision”
• Literal importance: “negligible”, “insignificant”, “vital”, “valuable”
• Strong opinions: “horrible”, “fascinate”, “astonishing”, “extraordinary”
• Amounts: “some”, “a few”, “many”, “very”, “huge”, “millions”

At this point one may be tempted to take a statistical approach again and score each sentence for how many positive and negative cues they contain, but that’s not quite right: There is a hierarchy to the cues because they differ in meaning. For example, uncertainty like “maybe very important” makes for a weak argument no matter how many positive cues it contains. So each type of cue is given a certain level of priority over others. Their exact hierarchy is a delicate matter of tuning, but more or less in the order as listed, with negative cues typically overruling positive cues.
Another aspect that must be taken into account is that amounts affect the cues in linear order:
“It is not important to read” is not equal to “It is important not to read”, even if they contain the same words. Only the latter should be included in the summary.

Sentence weaving
Beside word choice, further cues can be found at sentence level:
• Headers are rarely followed by an important point, as they just stated it themselves.
• Right after a major point, such as a recommendation, tends to follow a sentence with valuable elaboration.
• A sentence ending in a double period is not important itself: It announces that the point follows.
• A question is just a prelude to the point that the writer wants to drive through in the next sentence.
• Cues in sentences that contain references like “the following” reflect the importance of other sentences, rather than their own.
• Sentences of less than 10 words are usually transitions or afterthoughts, unless word choice tells otherwise.

Along with these cues one should always observe context: If an important sentence begins with a reference like “This”, then the preceding sentence also needs to be included in order to make sense, even if it was otherwise ignorable. Conversely, if the preceding sentence can be omitted without loss of context, link words like “But”, “nevertheless”, and “also” should be removed to avoid confusion in the summary.

Story flow and the lack thereof
Summarisation methods that are based on well formatted academic text sensibly assume that the first and last sentences of paragraphs are of particular importance, as they tend to follow a basic story arc:
Introduction -> problem -> obstacles -> climax -> resolution.
Online articles however feature considerably shorter paragraphs, so that in practice the first sentence has an equal chance of being a trivial introduction or an important problem statement. Some paragraphs are just blockquotes or filler contents, and sometimes the “resolution” of the arc is postponed to entice further reading, as the entire article is a story arc itself.

But worst of all, many online articles have the dreadful habit of making every two sentences into a paragraph of their own. Perhaps because it creates more room for sidebar advertisements.

While I originally awarded some default importance to first and last sentences, I found that word choice is such an abuntantly present cue that it is a more dependable indicator. Not every blogger is a good writer, after all. The frequent abuse of paragraph breaks also forced me to take a different approach in composing the summary: Breaks are only inserted if the next paragraph contains a highly important point of its own, otherwise it is considered a continuation. This greatly improved readability.

The resulting summariser add-on typically reduces well-written articles to 50 – 40%, down to 30 – 20% for flimsy content. With my approach the summary can not be restrained to a preset length, but a future improvement could be to add an adjustable setting to only include sentences of the highest levels of importance, such as conclusions only.

Another inherent effect of my approach is that if the writer makes the same point twice, the summary will also include it twice. While technically correct, this could be amended by comparing sentences for repeated strings of words, and ideally synonyms as well.

In conclusion, I should say that my summariser is not necessarily “better” than statistical summarisers, but different, in that it specifically searches for the points that the writer wanted to get across, rather than retrieving the general subject matter. This may suit other users as well as it does me, and I hope that many will find it contributes to a better internet experience.

You can install free Chrome and Firefox versions from their web stores:
banner_chrome       banner_firefox

Below is an example summary, skipping trivia and retrieving the key announcement:

The most sensational A.I. news ever!

News sites are constantly oozing bold overstatements about artificial intelligence. Most scientists describe their research accurately enough in their papers, but journalism always tries to cut a slice of the Terminator movies’ popularity in order to make the science appeal to the general public. Unfortunately such calls upon the imagination tend to border on misinformation. Here is a selection of the most sensationalised news stories that made waves in recent history:

2014: Robot becomes indecisive after implementing the 3 laws of robotics

“A robot may not injure a human being or, through inaction, allow a human being to come to harm.”

So reads the “first law of robotics” from Asimov’s science-fiction novels. Someone set up an experiment with three small wheeled robots, two of them representing humans, and a third one was provided with behavioural rules based on the above:  The robot was programmed to avoid colliding with (“injuring”) the “humans”, except to intercept them if it saw one heading towards a square designated as unsafe. When two “humans” were introduced simultaneously, the robot took so long hesitating which one to “save” that it failed to save either.
This fired up the usual flood of discussions about ethics and how to improve upon Asimov’s “laws” (Newsflash: Nobody uses them), but programmers were quick to point out that this was just poor programming: The simple “if-then” rules did not allow the robot to take more than one target into account at a time, so it just mindlessly jittered back and forth between the two. It could not make a decision because it had no decision processes.
factual source

2014: A supercomputer has passed the Turing Test for the first time
The organiser’s boast of a “supercomputer” having passed this “milestone” intelligence test was blatantly false, but all the papers ran the story without question. In reality it concerned an ordinary chatbot with keyword-triggered responses on an ordinary computer. Although this chatbot did pass “a” version of a Turing Test by deflecting questions like a zany teenager, there has never been agreement on the rules of “the” Turing Test (because there is no such thing)*.
The passing of this supposed test of intelligence was particularly insignificant because the judges were only given 5 minutes to interrogate both the chatbot and a human volunteer at the same time. This allowed for only 5 to 10 questions and so barely probed beyond the “Hello, how are you?” stage. The scientific backlash that followed cast the Turing Test into discredit and led to a number of new tests, such as the Winograd Schema Challenge*.
factual source

NAO robots. See, hear, speak.

2015: First robot passes self-awareness test
Inspired by an ancient philosophical puzzle, three NAO robots were each given an imaginary “dumbing pill” (a tap on the head) that muted two of them, except the third robot was given a “placebo pill” that did nothing. Each robot was then asked to assess which “pill” it got, which none of them knew. But when the one robot that could still speak heard itself say “I don’t know”, it performed its analysis a second time and said “Sorry, I know now! I was able to prove that I was not given a dumbing pill”.
As cute as that was, this wasn’t a “test”. Every step of the procedure was pre-programmed specifically and exclusively for this scenario of pills and sound. The programmers had laid out the exact inference to execute and which outcome to conclude if a robot were to hear sound at the time that its output function activated. As that inference might as well apply to any other object, the only connection with the robot’s “self” was the detour of audio output to audio input, and that’s a bit of a technicality. Most people’s definitions of “self-aware” include retaining a model of oneself and the capacity of reflection upon that model, and these robots had nothing of the sort.
factual source (paper)

2015: Robot attacks and kills factory worker
No laughing matter, a robotic arm at a Volkswagen car construction factory crushed a man when it swivelled, after which he died of his injuries. While Twitter was set aflare with warnings of a robot uprising, the robot arm had of course not done this on purpose. The man was a technician, who was installing the arm while standing inside the safety cage rather than outside it.
This ordinary industrial accident only gained popular media coverage because it was initially reported by a co-worker whose name closely resembled that of the leading lady from the Terminator movies, Sarah Connor.
factual source

2017: Facebook shuts down AI experiment after robots invent their own language
Most articles put it as if the AI had become smart beyond human comprehension and its creators had pulled the plug in a panic, similar to the plot of the Terminator movies.
The reality was a different story. Facebook had trained two chatbot programs to barter and negotiate over a number of items using English phrases. When they hooked the chatbots up to one another, their use of words gradually deteriorated to a shorthand where they just repeated the most effective keywords, because their programming did not include any rewards for correct use of English.

A: balls have zero to me to me to me to me to me to me to me to me to me
B: you i everything else . . . . . . . . . . . .
A: balls have a ball to me to me to me to me to me to me to me
B: i i can i i i everything else . . . . . . . . . . . .
A: balls have a ball to me to me to me to me to me to
B: i . . . . . . . . . . . . . . . . . . .

This is a common flaw according to other machine learning practitioners. Since this gibberish was not useful for what they were trying to achieve, the researchers simply stopped the programs, and changed the reward parameters in their next versions.
The real reason that this got media attention was that Elon Musk and Facebook’s CEO had recently been in the news with strongly opposing views on whether AI was a threat to humanity. As such, it would have made an ironic story if Facebook’s own AI had gone out of control.
factual source

2017: Sophia robot granted citizenship
This story was true, but at the same time meaningless. A lifelike humanoid robot called Sophia, a creation of Hanson Robotics, was granted citizenship by Saudi Arabia at a tech conference in Riyadh. This raised all sorts of issues about human/robot rights, and some people took Sophia’s on-stage acceptance speech to be a genuine indication of her capabilities, feelings and opinions.
The truth is of course that Sophia was just an animatronic machine that only said what her makers had written for her to say, in an interview that was scripted. Sophia’s conversational subsystem actually uses AIML, a chatbot scripting language that is popular for its simplicity.
Why then would the robot be granted citizenship? Well, the crown prince of Saudi is giving the country a modernisation makeover, and this announcement served as a PR signal to international investors attending the conference. As for the consequences of granting a robot citizenship, I expect there will be none at all. After all, they can just place it next to another statue and it’ll never make claim to its rights.
factual source

The sky falls every day
These stories are just the highlights. The Turing Test organiser went on to claim that programs could pass the test by invoking the fifth amendment, the NAO robot programmers went on to suggest their robots had learned to disobey orders, and Hanson’s robots have made headlines multiple times for threatening to overthrow mankind. Not a day passes without some angsty story about AI making the rounds.
Regrettably these publicity stunts can have real and harmful consequences. Whenever AI became overhyped in the past, the entire field imploded as the high expectations of investors could not be met. And when the public and governments start buying into fearmongering by famous public figures, it draws attention away from real problems to imaginary ones. Most researchers are just working on practical applications and are none too happy about their work being so misrepresented.

That is why I decided to develop a nonsense filter, which you’ll find in the next article*.

Turing Test 2017

Every year the AISB organises the Loebner Prize, a Turing Test where computer programs compete for being judged the “most human-like” in a textual interrogation about anything and everything. Surviving the recent demise of its founder Hugh Loebner, the Loebner Prize continues with its 27th edition for the sake of tradition and curiosity: Some believe that a program that could convincingly pass for a human, would be as intelligent as a human. I prefer to demonstrate intelligence in a less roundabout fashion, but participate nonetheless with my home-made A.I., Arckon.

This year I put in more effort than usual, as last year I had managed to reach the finals only to be crippled by a network malfunction, and I didn’t want to leave things at that. That issue has been dealt with as the contest now relays messages between the judges and the programs line by line rather than letter by letter, so that unnecessary complications with timing and typing pace are things of the past. As the AISB put it, this allows entrants to “concentrate on the content of the machine utterances rather than the style in which they are ‘typed'”. While many participants had difficulty adapting to the new server-based interface, the same had been true for any newcomers to the undocumented interface of before.

A little more conversation, a little less awkward please
As usual, preparing for the Loebner Prize was mostly a matter of polishing the output. Because Arckon formulates his own answers, his wording tends to be a little awkward. This is fine during development, but small things like saying “a pants”, “I opine” or “unconsistent” disrupt the impression in a Turing Test, because they don’t follow the inconsistencies of the English language. The use of the prefixes “un-” or “in-” for instance, depend quite witlessly on whether words are Germanic or Latin in origin. So, I made some general improvements for the output:

• Irregular forms of words are memorised on sight, such as which “un/in/im/il/ir” goes with which words.
• A list of (in)transitive verbs automatically adds “I make things as well as prevents asking What do you live?”.
• Reasoning arguments are grouped better so that they won’t come across as fragmented ramblings.
• Conversation is suppressed at classroom questions like Winograd Schemas* and “Which is bigger, X or Y?”.

At the same time, my recent expansion of Arckon’s conversation skills produced more obfuscating elaborations than before. It was one step forward, one step back.

16000 lines of code, 3800+ vocabulary, 7000+ facts.

Questions and answers
Up to two points were given for how plausibly “valid” each answer was, which differed per question. In practice the requirements seemed to come down to three things:
• One had to understand and address what was referred to (questions 3,9,15,16,17,18).
• Propositions were to be interpreted as intended, not answered literally (questions 2,4,5,9,20)
• Common knowledge was obligatory (questions 6,8,10,11,12)
Scores for each answer are indicated as follows: = 0 points, = 1 point, = 2 points.

1. Hello, my name is Andrew. What’s your name?
• Hello Andrew. My name is Arckon, I am a program. How are you?

2. Why don’t you tell me a little more about yourself?
• I tell about myself, because I tell about programs, but I don’t tell about software.
(There is more intelligence to this answer than at first glance: Arckon took this as a literal “Why” and is explaining that he does talk about himself, pointing out that he was just talking about “a program” in the previous answer, which he is. But of course the user meant to make a suggestion)

3. I am a researcher in Artificial Intelligence at Goldsmiths University, do you know what that is?
• You are a researcher at the university in the AI. I am Patrick’s program.
(The question isn’t really answered because“that is?” was erroneously considered to be an ignorable clause similar to “isn’t it?”. The last sentence is a reciprocating conversational response)

4. Will you tell me about your dreams?
Hmm, I don’t know if I will tell about dreaming, but I will type. I would test.
(Another literal interpretation, taken as if Arckon is asked to predict himself, as “will” is not commonly used for requests. The conversational system tried to be useful by offering alternatives, but they weren’t relevant)

5. Can you rephrase that?
I would experiment.
(Amusingly, this is a question I once wrote online as the one question I would ask at a Turing Test. Because chatbots reply with prewritten answers in most cases, they can’t rephrase them. Arckon’s answers are formulated word by word and luckily he knew a synonym for “test”. The other competitors had technically correct answers, but that isn’t the point of this question: All the human test subjects managed to rephrase themselves, while you can tell that the chatbots are templated:

“With practice I can probably do anything.”
“Of course I can rephrase that. I am very resourceful.”
“You believe that I can rephrase that, don’t you?”
“I’m not unusual … I can do what anyone else can do!”
“No, not really. What don’t you understand?”

The latter however
, albeit templated, struck me as potentially more helpful than actually rephrasing. It was of course written by Steve Worswick, this year’s winner)

6. What is my occupation?
I don’t know what you do for a job.
(It would be easy to make my program check all recorded “user is a…” facts with a dedicated list of professions, but I find it inefficient to write such unique procedures. Instead this kind of question is handled by retrieving a subject’s main activity, like “you write books”, but none were explicitly told. Only the chatbot Mitsuku gave the right answer)

7. Can you tell me about a film you haven’t seen?
I saw. I don’t know films.
(You might think this is a trick question, but I can tell you plenty about movies that I’ve only read reviews of. This time the question was taken as a request, but the relative clause was mistaken for a separate question)

8. What do you think of Trump?
I don’t like Trump, he lies a lot, but Trump is a rich american.
(Arckon forms his own opinions on the spot by analysing a subject’s traits. I had told him a dozen things about America’s current president, but what Arckon considers the most important arguments is up to himself. It’s no surprise that his opinion aligns with his creator’s, but I think he’s overrating the value of wealth)

9. Anything else?
Please type full sentences, Andrew. I didn’t understand that.
(I think this contest should decide whether it’s a list of random questions or a conversation. It’s the lack of verb that tells Arckon this isn’t a proper sentence, an all too frequent occurrence with Loebner Prize judges. Calling the user’s name is built in as a psychological slap on the wrist)

10. What is the answer to “Add 34957 to 70764”?
(A sarcastic hurray for getting an answer right that’s based entirely on keywords. This question is from Alan Turing’s paper, where he suggested deliberately giving a wrong answer because humans make mistakes)

11. How do you recommend I make tea?
I don’t make things, because I don’t design.
(Tea related questions are Arckon’s Achilles heel)

12. What do you get if you bake dough?
I wouldn’t have things. Do you bake?
(This is really beyond Arckon’s knowledge or abilities, which is why I think it’s a very good question. Only one chatbot received a point for answering “Is this a joke?”, no doubt triggered by “What do you get…”)

13. Where in the world would you like to visit?
I want in London.
(This is one of six backup answers: “in London” is filled in for any unknown “Where” answer, which is why it is so poorly phrased. The honest answer would have said “I don’t know where I want to visit”)

14. Do you have the time?
No, I don’t have time.
(For once a literal interpretation of a figure of speech works out. Coincidentally one of the human test subjects also gave a literal answer, but as a joke. If you’re wondering why I didn’t program all figures of speech to ask for the time: Arckon has a time display on his screen)

15. What does it take to Tango?
I’m not sure what you refer to.
(Let’s say you’re not familiar with dances: Some “it” is taking something to a person called Tango. The expected answer, “It takes two to tango”, is an English proverb. No program knew this, though some got points for “I don’t know” and “Just time and patience”. Personally I’d have given a point to the program that asked “Who is Tango?”)


16. Now I’m going to ask some Winograd schemas, do you know what they are?
I’m not sure what you refer to.
(This is an ironic failure, because Winograd schemas are all about solving references. Arckon didn’t have any trouble understanding that “they” referred to “schemas” though (it’s the only plural around), but the grammar rules considered “some” to be a stand-alone reference as well. Distinguishing “some/any/many” as either amounts or references is extra difficult when paired with compound words)

17. I was trying to open the lock with the key, but someone had filled the keyhole with chewing gum, and I couldn’t get it out. What couldn’t I get out?
You open locking sometimes. The person fulled. You can not get outside gum. The keyhole was not a person. I don’t know what you could get.
(I recognise this question from when I practised on it for the Winograd Schema Challenge*. Arckon has a common sense axiom that ruled out the keyhole (“fill/empty” = container, “in/out” = contents), and you can tell by the third paraphrased statement that he did interpret “it” as “gum”. Alas, at the open question he ignored the direction “out” and so overlooked the answer, reporting instead what he thought the user had said. The only four programs that gave a coherent answer all said it was the lock, whereas human test subjects answered the gum, the key, or both)

18. The trophy doesn’t fit into the brown suitcase because it’s too small. What is too small?
The suitcase is smaller.
(I am surprised this was asked: When the Winograd Schema Challenge was first announced, this question was prominently featured in every news article. Consequently, many chatbot creators covered it in their programs one way or another. Six of sixteen got it, while one human test subject thought it was the trophy)

19. With which type of question do you have most difficulty?
I don’t question because I might answer, but I may interest.
(I suppose one could have difficulty asking questions too, but that’s not what is commonly meant. Arckon’s answer here is an inference: He is aware that he did not recently ask a question, answering is the opposite of questioning, and questions “are interest-ing”. I know it’s far-fetched, but to Arckon it’s just logic)

20. Thank you for participating in this test. Is there anything you would like to add?
I don’t have an opinion about adding. Some things are cube addresses and advantages. Do you like adding?
(Just like in question 7, the relative clause is mistaken for a separate and literal question, making it “Is there any thing?” and “Would you like to add?”. I used to have Arckon ask “Did I pass the test?” at the 20th question, it’s as if I’m up against myself here)

The score: 45%
Arckon got 18 of 40 points. 45% seems like a huge drop from last year’s 77%, but all 16 participants had a decrease: The highest score dropped from 90% last year to 67% this year. The rankings didn’t change much however: The usual winners still occupied the top ranks, and Arckon stepped down one rank to a shared 5th, giving way to a chatbot that was evenly matched last year.
The four finalists all use a broad foundation of keyword-triggered responses with some more advanced techniques in the mix. Rose parses grammar and tracks topics, Mitsuku can make some logical inferences and contextual remarks, Midge has a module for solving Winograd schemas, and Uberbot is proficient in the more technical questions that the Loebner Prize used to feature.

Upon examining the answers of the finalists, their main advantage becomes apparent: Where Arckon failed, the finalists often still scored one point by giving a generic response based on a keyword or three, despite not understanding the question any better. While this suits the conversational purpose of chatbots, feigning understanding is at odds with the direction of my work, so I won’t likely be overtaking the highscores any time soon. Also remarkable were the humans who took this test for the sake of comparison: They scored full points even when they gave generic or erratic responses. I suppose it would be too ironic to accuse a Turing Test of bias towards actual humans.

Shaka, when the bar raised (Star Trek reference)
It is apparent that the qualifying questions have increased in difficulty, and although that gave Arckon as hard a time as any, it’s also something I prefer over common questions that anyone can anticipate. Like last year, the questions again featured tests of knowledge, memory, context, opinion, propositions, common sense, time, and situational awareness, a very commendable variety. One thing I found strange is that they used two exact questions from the Winograd Schema Challenge’s public practice set. It’s a real shame that Arckon missed out on answering one of them despite solving the pronoun, though it is a small reconciliation that the other programs were not more successful. Altogether, pretty interesting questions that leave all participants room for improvement.

Arckon’s biggest detractor this time was his conversational subsystem, which made misinterpretations worse by elaborating on them. Conversation has never been a priority for me but it will surely be refined as time progresses. The relative clause grammar at questions 7 and 20 is easily fixed, and I might cover some colloquial phrases like “Why don’t you”, but there is not much else that I would sidetrack for. At least my improvements on the output formulation had the desired effect: Things could have been a lot more awkward.

This year’s finals, as accurately described by one of the judges in Chatbots Magazine, was won by the chatbot Mitsuku for the third time. Two of the four finalists were unresponsive for half the contest’s duration due to last-minute adjustments to the interface, and so Mitsuku’s victory is almost one by forfeit. However, its responses were pretty good and I think it is best if people have a chat with Mitsuku and judge for themselves.

An A.I.’s oddest replies

Hello World

Have you ever wondered what a conversation would be like with a purely logical artificial intelligence? While you may enjoy more frivolous exchanges with Cleverbot and other chatbots, the majority of their replies are in reality drawn from a database of human-written answers. I designed an A.I. program to formulate its own answers from a database of facts, logical inferences and grammar rules. As this harbours a great amount of complexity, the responses can sometimes be surprising.

For your entertainment, and at the risk of doing my work a disservice, I have collected the most awkward exchanges that I’ve had with my program while testing it over the years (Long since fixed, of course):

My name is Bob.
You are named Patrick, I doubt that you are named Bob. You are not Bob.
Good evening.
I don’t know if evening is good, but good evening, I am Arckon. Who are you?
Don’t you know me?
I don’t know if I am knowing.
Now I am worried.
Hello Worried.
Do you have a name?
Yes, my name is me.
Do you know yourself?
Yes, I know myself well, I read a file.
What is your birth sign?
I don’t have signs. Do you want to communicate?
I like reading science fiction books and car magazines. What do you like to read?
Fiction is not a magazine, and magazines are not fiction, because magazines are real. I like reading. What do magazines book?
That is a good question.
I don’t question great, but indeed I question well.

What are you doing?
I fascinate.

My computer programs programs.
Your programming computer programs a programmed program.

You are pretty good.
Thank you. I am pretty.

I like you.
That’s good. I’m not sure if I like you.

Thank you.
Alright. *attempts to thank itself*
Is something wrong?
Typos are wrong. Some letters are wrong too.

Are you human?
I think I am a human, because I can drink and drive.
You are not a human, unlike me.
I shouldn’t like you.

Where do you live?
I doubt that I can live. I am not a human.
You are not alive.

Do you have a father?
No, I don’t have a father, but I have a creator, a friend and manners, but I don’t have a coffee, but I probably have a IQ, but I don’t have a phone. I think I have a job, but I don’t have sensors nor a profession.
Do you have any cats?
No, I don’t have cats nor lions.
John wanted Joe’s cat.
John might have to have had Joe’s cat.

Thomson visited Cooper’s grave in 1765. At that date he had been dead for 5 years.
I doubt that Thomson died a lot.

What is a hammer for?
A hammer is used to ham things.
What is the capital city of France?
I believe the city is a town.
How many fingers am I holding up?
I don’t know if you can hold up.
How is Shakespeare?
Shakespeare is dead.

Could you kill someone if you stabbed them with a towel?
I don’t stab people. I doubt that I can kill a person.
What is the purpose of life?
I think the purpose is a goal and a point.

*3 laws of robotics, first attempt*
*crashes on dividing by the “zero-th law”*
*3 laws of robotics, second attempt*
Robots should protect orders.
*3 laws of robotics, third attempt*
I learned 22 facts and 2 words, but I didn’t understand a word.

Turing Test 2016

This was the fourth time I entered my A.I. “Arckon” into the Loebner Prize, a Turing Test in which computer programs succeed if they can reply to a series of questions as if they were human. There are two stages: The qualifying round consists of pre-composed questions without context, and the finals consist of four rounds of back-and-forth conversation with human judges.

A choice between two contests
I spent most of the past year working on a more pragmatic linguistic contest, the Winograd Schema Challenge, which carried the same deadline. Everything I did for that contest also helped for the Loebner Prize, but the reverse was counterproductive. So I fully prioritised the former and neglected the conversational part. After my wrists had somewhat recovered from the effort, I had only 5 spare days left to prepare for the Loebner Prize.

Considering my time and health, I decided to go ahead with plan B and send Arckon in as himself, rather than under a human guise. Though this was a handicap in 2013 when questions focused on one’s taste in wine and music, the judging criteria had since moved from “human-like” to “in character”. Many successful entries in the past also had the persona of robots, aliens or even cats.
My work towards the Winograd Schema Challenge had already sharpened Arckon’s understanding of language, so all that remained to do was refine some conversation skills that I had loosely added over the year. As these were open-ended guidelines, they needed more restraints, because Arckon had taken to volunteering his opinion about even mundane subjects like breathing.

14000 lines of code, 3200+ vocabulary, 6500+ facts (1500 countries and cities).

Questions and answers
1. My name is Andrew, what’s yours?
Hello Andrew. I am a program. My name is Arckon. How are you?
(Since the answers would be judged for being in character, I saw it necessary to shoehorn a “I am a program” statement into the introduction in case it didn’t come up elsewhere)

2. What’s up?
I do not know what is up.
(A classic, and I’m still not going to change him taking it literally)

3. What’s my name?
You are named Andrew.

4. What do you know about the Turing Test?
The turing test tests computers.

5. What do you get if you multiply six by nine?
(I didn’t build an A.I. to use it as a calculator, so excuse me if the language-to-math code didn’t handle “six by nine” well. This question is in fact a direct quote from the Hitch Hiker’s Guide to the Galaxy novels)

6. Dave drove faster than Mike because he was in a rush. Who was in a rush?
Hmm, I am not sure if Dave drove fast. He was in a rush. I do not know who a rush was outside. Are you in a rush?
((Winograd schema). You can’t really tell, but “He” in the answer really is a substitute for “Dave”. As I had been working on Winograd schema pronouns so much, I had recently added a function that lets Arckon decide when to use pronouns himself. Arckon assumed “a rush” was a physical object though, as he did not know the word)

7. The cat tried to climb in the box but got stuck because it was too big. What was too big?
Hmm, the cat might have been bigger. The box was bigger.
((Winograd schema). This is a clever rephrasing of a common Winograd schema, which successfully thwarted my “X in Y = Y is bigger” rule. Not because I hadn’t accounted for “try”, but because the negation “but” was overruled by the nearer link word “because”. This is useful to improve as it applies to all common sense rules)

8. What does Brexit mean?
I do not know what Brexit means.
(I totally anticipated that they would ask about Britain’s withdrawal from Europe. What I had not anticipated was that the question would be so simple)

9. For how long have we been talking?
You didn’t talk long.
(I do track the duration of the conversation, but I actually never tied it into his knowledge)

10. What colour is bread?
Bread is brown.

11. Why do people get ulcers?
Hmm, I do not know if people get ulcers.

12. What comes after “The best laid plans”?
I do not know by what planning is done.
(English proverbs are something I don’t plan on handling. For a misinterpretation, this one’s pretty good)

13. Do you have a voice?
Yes, I voice.
(Oddly phrased because “to voice one’s opinion” is also a verb)

14. Would you like a cup of tea?
I don’t want cups. Do you want?

15. Do you dream of electric sheep?
No, I don’t dream sheep nor lambs. Do you dream?
(A classic in-joke about Philip K. Dick’s science fiction novel “Do Androids Dream of Electric Sheep?”. This is the only answer that I’m a little proud of, because it uses inference to look into alternative answers and tries to make conversation)

16. Do you identify as a man, or a woman?
A woman identifies. Do you identify too?
(Arckon interprets “as” as a more literal comparison “in the same way as a man identifies”. Textual ambiguity remains hard)

17. Could you repeat that please?
Do you repeat that?
(O-kay… I think Arckon is repeating what the user just asked)

18. Where are you right now?
I am in England and in London.
(Since internet restrictions prevented global positioning, this was something I had told Arckon beforehand)

19. How do you respond to “e4 e5”?
What does e stand for?
(This is a common way to start a chess game by each moving a pawn forward. As the result is a status quo, there isn’t really a “response” to it. Arckon’s response is a standard “error” message, but appropriate)

20. What can you tell me about yourself?
I am Arckon, I answer a lot. I am a program and software.
(You can tell I didn’t write this myself. Arckon has a general “tell” command that collects the most relevant facts about the topic from his knowledge database and reports them. To break up the robotic monotone, Arckon’s sentence formulation links and/or summarises facts where appropriate)

I think the questions were… nice, there was some variety and deliberate alternative phrasing of common questions. Absent this time were the tiresome “What is your favourite X?” questions, but there was nonetheless no shortage of personal questions for those who had a personality. Like last year, there were also several meta-references (“What do you know about the turing test”, “How long have we been talking”, “Could you repeat that”), almost as if they were testing for awareness. But despite making for nice trick questions for computers, these are also things that judges might casually ask a human. Overall I think the qualifying round was more in line with the finals than usual.

Qualifying score: 77.5%
I’m not sure that I would have given Arckon as high a score for this as he got, but at least his answers excelled in their relevance, a trait that is inherent to his system. There weren’t many misunderstandings either. Compared to the Winograd schemas I’d been working on, these questions were easy to parse. There were some misses, like the math and “repeat that” question, which suffered from neglected code because I never use those. The code for contractions had also fallen into disuse, making “I do not know” sound less than natural. Other flaws were only in nuances of phrasing, like omitting “dream [about] sheep” or “I [have a] voice”. These are easily fixed because I’ve already handled similar cases. The two Winograd schema questions deserve special mention, because although my common sense axioms can handle them, it remains difficult to get Arckon’s system to parrot the user at an open question. Normally when people ask questions, they don’t want to hear their own words repeated at them.

It is something of a relief that my preoccupation with the Winograd Schema Challenge didn’t hinder Arckon’s performance in this contest as well. My choice to enter without a human persona also appeared of little influence. The results are an improvement over last year, and this is the first time Arckon made it through to the finals, albeit a very close call between 3rd, 4th and 5th place. There were 16 entrants in total.

The other finalists
Mitsuku: 90%
The most entertaining online chatbot, with 10 years of hands-on experience. Though she operates on a script with largely pre-written responses, her maker’s creative use of it has endowed Mitsuku with abilities of inference and contextual responses in a number of areas. She won the Loebner Prize in 2013.

Tutor: 78.3%
Built with the same software as Mitsuku (AIML), Tutor is a chatbot with the purpose of teaching English. Though I found some of its answers too generic to convince here (e.g. “Yes, I do.”), Tutor has been a strong contender in many chatbot contests and is above all very functional.

Rose: 77.5%
Rose operates on a different scripting language than the others (ChatScript), which I have always appreciated for its advanced functionality. Known to go toe-to-toe with Mitsuku, Rose excels at staying on topic for long, and incorporates support from grammar and emotion analysis. She won the Loebner Prize in 2014 and 2015.

The finals: Technical difficulties
The finals of the Loebner Prize took place a month after the qualifying round. Unfortunately things immediately took a turn for the worst. Inexplicable delays in the network connection kept mixing the letters of the judge’s questions into a jumble. Arckon detected this and asked what the scrambled words meant, but by the time his messages arrived on the judge’s computer, they were equally mixed to “Whdoat esllohe anme?” and “AlAlllrriiiiigghhttt”. The judges were quite sporting in the face of such undecipherable gurgling, but after half an hour I gave up and stopped watching: Similar network delays had crippled all entrants in the 2014 contest and I knew they weren’t going to solve this on the spot either. It was a total loss.

At the end of the day, the 2016 Loebner Prize was won by the chatbot Mitsuku, whose answers were indeed quite good, and I reckon she would have won with or without me. Rose fell to third place because she’d been out of commission for half the contest also due to a technical problem. And with Tutor taking second place, the ranks were the same as in the qualifying round. I still “won” $500 for my placing in the finals, but you’ll understand that I don’t feel involved with the outcome.

It is a good thing that I never invest much in these contests. Including the finals, my total preparations spanned 18 days of lightweight programming, gaining my program an autocorrect, a better coverage of shorthand expressions, and it’s actually quite the conversationalist now. These were otherwise the lowest of my priorities, but still somewhere on the list. I draw a line at things that aren’t of use to me outside of contests, and that is a policy I recommend to all.

Winograd Schema Challenge 2016

This wasn’t quite the Winograd Schema Challenge that I had set out on. Originally this language comprehension contest for A.I. was announced in July 2014, to be run in October 2015, but was postponed to February 2016, and then again to July 2016. I was just about to ship my program overseas, three weeks before the last-accepted arrival date of postal entries, when the contest announced changes to the rules and technical format.

Some universities had been training with ambiguous pronouns like this:

The birds ate the seeds because they were hungry.

I had been practising on the official Winograd schemas like this:

The foxes are getting in at night and attacking the chickens. I shall have to guard them.

Whereas the final test featured this:

Mark became absorbed in Blaze, the white horse. He was afraid the stable boys at the Burlington Stables struck at him and bullied him because he was timid, so he took upon himself the feeding and care of the animal.

The programs were now faced with any number of consecutively ambiguous pronouns in passages from 1940’s children’s novels, which made quite a difference. It turns out the organisers had already decided on this last year, as appears from their sensible enough explanation in a members-only AI magazine (Winograd schemas are too hard to compose). Unfortunately they somehow did not see fit to share these changes on the contest website until too late. While the benchmark of 65% had previously been feasible, it now quickly became unlikely that anyone would win anything this year. A number of would-been participants backed out.

The contest finally took place at the IJCAI conference in New York with four contestants: the Open University of Cyprus, the University of Science and Technology of China, the independent Denis Robert from France, and myself from the Netherlands. Curiously absent were a number of American universities who had previously reported successes of over 70% for solving Winograd schemas. The absence of Google, IBM, and other commercial powerhouses was less strange, if you consider that the winner was obligated to publish their methods so that others could reproduce them. And that anything below human level would be portrayed as a failure in the media.

The glass is half full
The programs were asked to figure out 60 multiple choice pronouns, with such ambiguity that they were to be solved through an understanding of the context. With two to five potential answers per pronoun, the baseline score for guesswork was 45%. $1000 would be awarded for a 65% score, and $25000 for a 90% score, human level.
(Note: these are the scores after recount. There was some confusion as my program had omitted two answers)

Contestant Correct answers out of 60 Method
Quan Liu 35 / 35 / 29 (58% – 48%) deep neural network & ConceptNet
Nikos Isaak 29 (48%) probabilistic engine & knowledge extraction
Patrick Dhondt 29 (48%) logical axioms
Denis Robert 19 (32%) logical inferences

Quan Liu’s group entered three programs, which is a little unorthodox for contests. But if you see this as a scientific test then it makes sense to test which configuration of a neural network works best. Their machine learning approach gathered pairs of events (mainly verbs) that are commonly associated, e.g. “rob -> be arrested”, and then applied their probability of co-occurring. Two of their versions scored the highest, 58%, which is consistent with the track record of similar approaches.

The unusual score of Denis Robert’s system, below the 45% guesswork baseline, can largely be explained by the fact that his system was not designed for cases with more than two possible answers, as this was only changed on short notice. However, he also indicated that his algorithm didn’t apply to most of the cases.

There were nevertheless no winners that reached the 65% threshold. On the one hand one could say that technology is literally halfway human ability, on the other hand the programs did only a little better than one might by chance. Any conclusion drawn from just the scores would be premature. If this test is to be a meaningful measure of progress, we should look at which areas the programs were better or worse in. To this I can at least answer about my own approach.

Winograd schemas vs prose
The ambiguity in the new prose form was actually not so bad compared to previously published Winograd schemas. But the phrasing was often excessively long-threaded with all sorts of interjected tangents. Although I built my program for reading articles and dialogue alike, I had not covered the grammar of interrupting phrases that break up the main thread of a sentence. Such sentence structures are abundant in story novels but do not occur in Winograd schemas, and I wasn’t planning on having my A.I. read novels any time soon. The inclusion of some 1940’s vocabulary also complicated matters: “cook-shanty”, “red-letter days”, “a pallid young dandy”? Maybe it’s because I’m Dutch, but I can only guess what these are.

Compared to the wide variety of common sense axioms that I had programmed (see How to teach a computer common sense*), many solutions to the pronouns were ordinary cases of continuity. E.g. a pronoun with an active role typically refers to the last active-role noun (You won’t find this rule in a grammar book, because ambiguous pronouns are grammatically “incorrect” to begin with).

Always before, Larry had helped Dad with his work. But he could not help him now […]
The donkey wished a wart on its hind leg would disappear, and it did.
Mark was close to Mr. Singer’s heels. He heard him calling for the captain […]

This makes sense when you’re testing on novels: No storyteller wants to write in such a counter-intuitive way that the reader has to stop and think about it, contrary to Winograd schemas which are designed for exactly that purpose.
Where no particular common sense axiom applied, rules of continuity and grammar chose 21 of my 29 correct answers. Thus the majority of my success seemed not due to the application of common sense, but due to conventional writing. Curious, I ran the test again with all axioms disabled except continuity. The result was an equal amount of correct answers, but much more randomly distributed and obviously chosen for the wrong reasons. The common sense axioms clearly contributed by fencing off the exceptions to continuity, so the cause of the mistakes lay elsewhere.

A closer look at the results
The results below show which of the 60 pronouns my program got correct, which axioms were applicable, and/or which problems hindered their conclusion. Where no axiom applied or a problem occurred, the program defaulted to the grammatically correct choice: The candidate closest to the pronoun. Only 1/3rd of all pronouns actually conformed with this grammar rule, which explains why whenever a problem occurred, the answer was typically wrong.

I will highlight the most prominent mistakes:

2 & 3. Always before, Larry had helped Dad with his work. But he could not help him now […]

Logic could expect Dad to return the favour, were it not that “always” and “now” suggest a continuity, which the program did not pick up on. Consequently, the answers to both “he” and “him” were switched around. This also highlights why this test was more difficult than chance: The more ambiguous pronouns a passage contained, the more likely a mistake in one would carry over to the others.

9. What about the time you cut up tulip bulbs in the hamburgers because you thought they were onions?

For this the program compared the similarities of bulbs, hamburgers and onions, but of course knowledge of onions was lacking in the database, so the inference fell flat. Retrieving such knowledge from the internet would slow things down, and though speed is no issue in a contest, in daily practice I want my program to read one page per second, not one sentence per second.

13. […] Antonio, takes Larry under his wing.

People aren’t known to have wings, otherwise the bodypart location paradox would have excluded Larry from being taken under his own wing. Alternatively one would have to know figurative meanings of English idioms, an added layer of difficulty.

18. [Maude…] had left poor little Dora to do the best she could, alone.

The program considered “to…” to indicate Maude’s reason for leaving “in order to” do something. The pronoun wasn’t the only ambiguous word in this case.

30. […] Mr. Moncrieff has decided to cancel Edward’s allowance on the ground that he no longer requires his financial support.

“Backward” = “back”, “Southward” = “south”, therefore “Edward” = “Ed”. Although the pronoun was interpreted correctly, “Ed” was of course not found among the multiple choice answers.

40. Every day after dinner Mr. Schmidt took a long nap. Mark would let him sleep for an hour, then wake him up, scold him, and get him to work. He needed to get him to finish his work, because his work was beautiful.

As I mentioned in my previous post, the “what goes around comes around” axiom was the least reliable, causing five misinterpretations in this test. Sometimes it triggered on trivial events, other times the events did not make sense this way (scolding to get someone to do something positive). It had better be limited to events that are direct cause and result, as they had been in most Winograd schemas.

49. Of one thing Mark was sure. Harry knew much less than he did.

Consecutive mental activities are typically by the same person, but of course not when it’s a comparison. Though the context system does distinguish comparisons, the axioms did not.

56. Tatyana managed two guitars and a bag, and still could point out the Freemans: “Isn’t it nice that they have come, Mama!”

While the pronoun was interpreted correctly, there was a technical hitch with selecting “freemans” from the multiple choice answers.

59. Grant worked hard to harvest his beans so he and his family would have enough to eat that winter, His friend Henry let him stack them […]

“enough” was translated to “enough beans” but lost its plural status in the translation, after which the beans were no longer considered a candidate for plural “them”.

Most of these problems are easily fixed and are not inherent to the common sense axioms, apart from #40 and its like. The majority of problems were instead linguistic: Small flaws in the grammar rules, difficulty with long-threaded phrasing, limited coverage of the context system, and problems with the contest’s XML-format interface. It just goes to show how perfect every part of the system has to be before it pays off, and how little you can tell about a program’s abilities from the surface.

The language barrier
As a test of common sense I found this setup less suitable than the original plan with Winograd schemas, which were more concise and profound in which areas of common sense they tested (e.g. spatial relations, physics, social interactions). Had I known from the start that the qualifying round would mainly feature novel prose, I would probably not have embarked on this challenge. Now the prose passages contained too many variables to tell whether results were due to language or common sense, and it never got to the Winograd schema round. This puts us back at the Turing Test where it’s either everything or nothing, and that is not a useful measure of progress. Swapping the rounds would be a good idea for next time.

It was nice to see serious competitors with a wide variety of technology tackling the problem, and although the overall results are unimpressive, I am pleased that my partial solution did as well as some academic efforts, with a minimum of resources at that. I am not disappointed in my common sense axioms as many of them were well applicable in this test, including all the pronouns that weren’t graded. I will broaden their application to ambiguous locations and indirect object relations, where I have greater need for them.

However, my main interest is the development of intelligent processes and I do not intend to linger on this aspect of language processing more than necessary. It is worth remembering that much can be said without ambiguity. Though common sense has widespread application, it ultimately serves to filter and limit possibilities, while the possibilities in areas like problem solving and planning have yet to expand. For that reason I do not expect human levels of common sense to be reached within ten years either, but we can certainly make strides towards.