ChatGPT running Brainhat test cases
This posting is a companion to another posting, Took a Pause... You might want to read that first.
QA Test #1
Here are some comparisons of Brainhat and ChatGPT running Brainhat tests. This is ChatGPT:
This is Brainhat's performance on the same test:
if dog has toy and dog is in room then toy is in room.
if dog has toy and dog is in room then toy is in room.
if a dog sees a cat then dog drops toy and dog chases cat.
if a dog sees a cat then dog drops toy and dog chases cat.
if dog drops toy then dog does not have toy.
if dog drops toy then dog does not have toy.
dog has a ball.
dog has a ball.
dog is in kitchen.
dog is in kitchen.
a ball is in kitchen.
dog is in bathroom.
dog is in bathroom.
a ball is in bathroom.
where is ball?
a ball is in bathroom.
dog sees cat.
dog in bathroom sees cat.
dog in bathroom drops toy.
dog in bathroom does not have toy.
dog in bathroom chases cat.
dog is in bedroom.
dog is in bedroom.
where is ball?
a ball is in bathroom.
They both passed the test. ChatGPT gave the sense that it knew this was an exercise for the fun of it.
QA Test #2
Here's another test. It is a snippet taken from a Spanish-speaking student's practice with English, further simplified to fit Brainhat's simple grammars. ChatGPT and Brainhat are both presented with the essay and asked questions about it. This is the input:
My name is Gabriela.
I am a woman.
I am 17 years old.
I am from Mexico.
I am from a city.
The city's name is Tapachula.
The city is always hot.
We have a beach.
I have a beautiful family.
I have one brother.
I have one sister.
My father's name is Jorge.
My mother's name is Irma.
My sister's name is Irma.
My brother's name is Jorge.
He is married.
His wife's name is Alejandra.
She will have a baby in November.
The family is so excited.
I have a beautiful dog.
He is a Siberian Husky.
He is my baby.
He is very lovely.
I love my dog.
My hobby is dancing.
My hobby is singing.
My hobby is swimming.
My hobby is skating.
My hobby is being with my friends.
My hobby is going to the cinema with my friends.
I love to dance.
When I was a child I studied ballet.
It was a really good experience.
I really enjoyed that.
I studied in two secondary schools.
It was fun.
I could meet new people.
The high school was really cool for me.
I liked the time very much.
When I was at 9 years my father told me that I will go to USA.
When I was at 9 years my father told me that I will learn english well.
I started to study english in Tapachula.
Before I came to USA I finished high school.
The high school's name is preparatoria Tapachula.
My father works in the school.
My brother works in the school too.
When I was in mexico i had many friends.
I have many friends in mexico.
I like to make friends.
I was with them all the time.
We went to many places together.
We went to many parties together.
We went to the cinema together.
We eat together.
I really enjoyed these moments.
My friend's name is Rosalba.
My friend's name is Miriam.
My friend's name is Paola.
My friend's name is Jorge.
My friend's name is Anahi.
My friend's name is Alex.
My friend's name is Max.
My friend's name is Salvador.
My friend's name is Eder.
My friend's name is Pauline.
My friend's name is Hugo.
I really enjoy to be with my friends because they are so nice.
My mom is my best friend.
My dad is my best friend.
My sister is my best friend.
My brother is my best friend.
With my family I can talk about my life.
If I have a problem my family is with me.
If I have a problem my family helps me.
My brother Jorge is 20 years old.
When he was 19 he served a mission.
The mission's city's name is Puebla City.
Jorge returned to Tapachula to study law.
He finished his major two years ago.
Now he is a lawyer.
Last December he married Alejandra.
They are going to have a baby.
The baby is a boy.
Irma is 25 years old.
When she was 16 she lived in the USA.
She lived in the USA for 1 year.
She lived in Denver.
She said it was a good experience for her.
She came back to Tapachula.
She finished high school.
She went to Puebla City to study english.
She is in Puebla City studying languages.
She has a boyfriend.
Maybe next year they are going to marry.
I really love animals.
I wanted to study veterinary medicine.
I changed my mind.
When I return to mexico I want to study tourism.
I will have many animals because I really love them.
I want to study my major in Tapachula.
I am from Tapachula.
There is a university that has a major in tourism.
I think it will be perfect for me.
I can be with my family.
I can be with my friends.
I can enjoy them.
I can enjoy my little dog.
I have plans.
I plan to learn english very well.
I plan to return to mexico.
I plan to study my major.
I plan to finish my major.
I plan to work.
I plan to earn money.
I plan to go to Hawaii.
I plan to go to Europe.
I plan to return to mexico.
I plan to marry the correct person.
I plan to have three children.
I plan to make a wonderful family.
Here is ChatGPT answering the questions:
Got it! If you need help with anything else, feel free to ask.
You said:
What is my name?
How old am I?
What is my father's name?
Where does my father work?
What do I want to do?
Where do I plan to go?
What did I study?
What did I enjoy?
What did I want to study?
What do I plan to study?
What is my brother's name?
What is my brother's wife's name?
Who will have a baby?
What is my sister's name?
Will she have a baby?
Do I have a dog?
Do I like him?
Who served a mission?
ChatGPT said:
ChatGPT
Here are the answers to your questions:
Your name is Gabriela.
You are 17 years old.
Your father's name is Jorge.
Your father works in the school.
You want to study tourism.
You plan to go to Hawaii and Europe.
You studied ballet as a child and English.
You really enjoyed ballet.
You wanted to study veterinary medicine but changed your mind.
You plan to study tourism.
Your brother's name is Jorge.
Your brother's wife's name is Alejandra.
Alejandra will have a baby.
Your sister's name is Irma.
No, she will not have a baby (at least not mentioned).
Yes, you have a dog.
Yes, you love him very much.
Your brother Jorge served a mission in Puebla City.
Here are Brainhat's answers:
You are gabriela.
You are 17.
jorge is jorge.
jorge works in the school.
You want You to study your major.
You plan You do go to europe.
You did study ballet at that You were a with hugo child.
You did like the time very.
You did want You to study veterinary medicine.
You plan You do study your major.
jorge is jorge.
alejandra is alejandra.
alejandra will have the baby in november.
irma is irma.
maybe. I do not know.
yes. You have your small dog belonging to You.
yes. You love your small dog belonging to You.
jorge did serve a mission at that jorge was 19.
QA Test #3
This is most complicated test. We describe a game of heads-or-tails to both ChatGPT and Brainhat and then play. Again, both passed the test. ChatGPT was more compelling:
You said:
your score is 0.
my score is 0.
if i say heads then add 1 to your score.
if i say tails then add 1 to my score.
if my score is 5 then I am the winner and you are the loser.
if your score is 5 then you are the winner and I am the loser.
ChatGPT said:
ChatGPT
Got it! Let’s start the game. Say “heads” or “tails,” and I’ll keep track of our scores.
Score:
Your score: 0
My score: 0
You said:
heads
ChatGPT said:
ChatGPT
You said "heads," so I add 1 to my score.
Score:
Your score: 0
My score: 1
You said:
tails
ChatGPT said:
ChatGPT
You said "tails," so I add 1 to your score.
Score:
Your score: 1
My score: 1
You said:
tails
ChatGPT said:
ChatGPT
You said "tails," so I add 1 to your score.
Score:
Your score: 2
My score: 1
You said:
tails
ChatGPT said:
ChatGPT
You said "tails," so I add 1 to your score.
Score:
Your score: 3
My score: 1
You said:
tails
ChatGPT said:
ChatGPT
You said "tails," so I add 1 to your score.
Score:
Your score: 4
My score: 1
You said:
heads
ChatGPT said:
ChatGPT
You said "heads," so I add 1 to my score.
Score:
Your score: 4
My score: 2
You said:
tails
ChatGPT said:
ChatGPT
You said "tails," so I add 1 to your score.
Score:
Your score: 5
My score: 2
Congratulations! You are the winner, and I am the loser.
Here is Brainhat's performance on the test:
your score is 0.
my score belonging to I is 0.
my score is 0.
your score belonging to You is 0.
if i say heads then add 1 to your score.
if You say head then add 1 number to my score.
if i say tails then add 1 to my score.
if You say tail then add 1 number to your score.
if my score is 5 then i am the winner and you are the loser.
if your score is 5 then You are the winner and I am the loser.
if your score is 5 then you are the winner and i am the loser.
if my score is 5 then I am the winner and You are the loser.
heads.
head.
my score belonging to I is 1.
heads.
head.
my score belonging to I is 2.
tails.
tail.
your score belonging to You is 1.
tails.
tail.
your score belonging to You is 2.
heads.
head.
my score belonging to I is 3.
heads.
head.
my score belonging to I is 4.
tails.
tail.
your score belonging to You is 3.
heads.
head.
my score belonging to I is 5.
I am the winner.
You are the loser.
am i the winner?
no.
You are not the winner.
how was your score when my score was 2?
score was 4.
score was 3.
score was 2.
what value was my score when your score was 4?
your score belonging to You is 3.
your score was 2.