This time the model played with us, live, by ear. It answered most questions right. It lost the most points on the questions where it heard the words and dropped the «правда ли».
В этот раз модель играла вместе с нами, вживую и на слух. На большинство вопросов она ответила правильно. Больше всего очков ушло на вопросах, где она расслышала слова, но потеряла «правда ли».
How it worked / Как это устроено
Last time Claude played a game of our Zoom league blind, after the fact, from a transcript. On October 22 it played the whole game live, in the background, for 2 hours 39 minutes.
The setup:
- Capture: one Mac captured the screen and the call’s sound.
- Reading: Haiku read questions off the screen.
- Listening: Gemini Flash listened to the last two minutes of sound every five seconds and wrote down each question once the host finished reading it.
- Answering: Opus answered.
- Rating: on multiple-choice questions, Jev from TypeSafe rated the options.
The team didn’t see any answer during play. Each one appeared only after the host announced the correct answer, because our job is to protect the game, not spoil it.
В прошлый раз Claude сыграл игру нашей Zoom-лиги вслепую, после игры, по расшифровке. 22 октября он отыграл всю игру вживую, в фоне, 2 часа 39 минут. Haiku читал вопросы с экрана, Gemini Flash каждые пять секунд слушал последние две минуты звука, Opus отвечал, а Jev от TypeSafe оценивал варианты в вопросах с выбором. Команда во время игры ответов не видела: каждый ответ открывался только после того, как ведущий объявлял правильный.
Score / Счёт
46 questions whose correct answers we caught:
| Type | Questions | Right | Counting implied true/false |
|---|---|---|---|
| Multiple choice | 6 | 6 | 6 |
| True or false | 6 | 1 | 4 |
| Open | 28 | 21 | 21 |
| Clue chains, counted once each | 4 | 4 | 4 |
| With a picture | 2 | 1 | 1 |
| Total | 46 | 33 | 36 |
| «Пары» anagrams, in pairs | 15 | 5 |
Answers followed questions by a median of 8.9 seconds, and by 15.3 seconds at the 90th percentile. The whole day on OpenRouter cost $9.78, including tests.
The model knew the answers. Most of the lost points went somewhere else.
Where the points went / Куда ушли очки
1. «Правда ли, что…». Six facts were read out as «правда ли, что…». Every time, the listener wrote down the fact and dropped the «правда ли», even though its instructions say to transcribe verbatim. Opus received a statement instead of a question, and in five of the six cases it explained the fact instead of saying «правда» or «неправда». That’s 1 of 6 strictly, or 4 of 6 if we count the stance it implied. This was the biggest single loss: the listener kept the words and lost the speech act.
2. The last answer wins / Побеждает последний ответ. Most questions were heard two or three times, and the later hearings were often fragments: the second half of the question, or the host’s aside. The page kept whichever answer came last, so fragments replaced correct answers three times:
- The Swedish idiom: «Smultronställe (земляничная поляна)» was on the page for about twenty seconds, then «Место силы» replaced it.
- The proverb «Пиар костей не ломит»: «Бранд на вороту не виснет», which was close, was replaced by answers to two fragments.
- The Hemingway quote: the screen reader had the full quote, and Opus answered «фрегат». Nineteen seconds later the version heard without the quote got «Барка».
The model wasn’t wrong in any of these. The system threw away the right answer.
3. Clue chains without memory / Цепочки без памяти. Each clue went to Opus as a separate question. On the chastity-belt chain it found «пояс верности» on clue 3, then answered “shoes with nailed soles” on clue 4. On the others it got the vampires on clue 3 of 5 and Chekhov on clue 1. Anyone at the table would have kept the earlier clues in mind.
4. Answers in English / Ответы по-английски. 19 of the 110 answers came back mostly in English. The answer prompt is in English and doesn’t say what language to answer in, so the model chose English.
Jev: the test couldn’t have gone the other way / Jev: тест, который не мог показать обратное
Jev rated the options of seven multiple-choice questions. It was right on five. On the other two it was confidently wrong: «в Гознаке» at 65% for Fyodor Fedorovsky, the designer of the ruby stars, who worked at the Bolshoi Theatre, and «Илья» at 81% for the painter of «Богатыри», whose name was Viktor. On those two it gave the correct answer 4% and 2%.
But Opus answered all seven correctly. So Jev had nothing to rescue: on this sample it could only agree or make things worse. “Jev added nothing” is true and uninformative, because the test had no question where Jev could have helped. A fair test needs the questions Opus gets wrong, which means the open ones with versions the team types in, and nobody typed any.
Limits / Ограничения
One game, one league, one evening. The table counts only questions whose correct answer our reveal watcher caught; it missed about nine announcements. Seven multiple-choice questions say nothing about Jev in general. The answer model was almost never the bottleneck, and the rest of the system will get fixes before the next game.
Next: the listener writes down the type of question as well as its text, repeated hearings of one question update one answer instead of overwriting it, clues accumulate, and answers come back in the question’s language. Then another game.
