Facebook Develops Artificial Intelligence Test

Facebook has developed a simple test that can help determine the intelligence level of an artificial intelligence software.

The test (found along with the paper 'Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks' here), developed by researchers at Facebook's Artificial Intelligence lab, involves 20 tasks, which get progressively harder.

Any potential artificial intelligence (AI) must pass all of them if it is ever to develop true intelligence, researchers said.

Computing pioneer Alan Turing introduced his own test for AI, called The Turing test, in 1950.

In the test a human judge engages in natural language conversations with a human and a machine designed to generate performance indistinguishable from that of a human being.

If the judge cannot tell the machine from the human, the machine is said to have passed the test.

However, this approach has a downside.

"The Turing test requires us to teach the machine skills that are not actually useful for us," said Matthew Richardson, an AI researcher at Microsoft.

For instance, to pass the test an AI must learn to lie about its true nature and pretend not to know facts a human would not.

AI researchers everywhere are developing more comprehensive exams to challenge their machines.

The AI intelligence test by Facebook has tasks involving short descriptions followed by some questions, like a reading comprehension quiz.

For example the AI may have to answer the following question, of the 'Factoid QA With Two Support Facts' type:

John is in the playground.
Bob is in the office.
John picked up the football.
Bob went to the kitchen.
Q1. Where is the football?
Q2. Where was Bob before the kitchen?

A1: Playground
A2: Office

Harder examples include figuring out whether one object could fit inside another, or why a person might act a certain way, 'New Scientist' reported.

"We wanted tasks that any human who can read can answer," said Facebook's Jason Weston, who led the research.

Having a range of questions challenges the AI in different ways, meaning systems that have a single strength fall short, researchers said.

The Facebook team used its exam to test a number of learning algorithms, and found that none managed full marks.

The best performance was by a variant of a neural network with access to an external memory. But even this fell down on tasks like counting objects in a question or spatial reasoning, researchers said.

Test yourself on some other example questions, ranging from two argument relations to yes/ no questions, compound coreference, and positional reasoning.

Questions:

The office is north of the bedroom.
The bedroom is north of the bathroom.
Q1: What is north of the bedroom?
Q2: What is the bedroom north of?

John is in the playground.
Daniel picks up the milk.
Q3: Is John in the classroom?
Q4: Does Daniel have the milk?

Daniel picked up the football.
Daniel dropped the football.
Daniel got the milk.
Daniel took the apple.
Q5: How many objects is Daniel holding?

Daniel and Sandra journeyed to the office.
Then they went to the garden.
Sandra and John travelled to the kitchen.
After that they moved to the hallway.
Q6: Where is Daniel?

The triangle is to the right of the blue square.
The red square is on top of the blue square.
The red sphere is to the right of the blue square.
Q7: Is the red sphere to the right of the blue square?
Q8: Is the red square to the left of the triangle?

Answers:

A1: Office
A2: Bathroom
A3: No
A4: Yes
A5: Two
A6: Garden
A7: Yes
A8: Yes

Written with inputs from PTI