They then hypothesized a general factor, “g,” to explain this pattern. Early tests (e.g., Binet–Simon; later Stanford–Binet and Wechsler) sampled a wide range of tasks, and researchers used correlations and factor analysis to extract the common component, then norm it around 100 with a SD of 15 and call it IQ.

IQ tend to meaningfully predicts performance across some domains especially education and work, and shows high test–retest stability from late adolescence through adulthood. It is also tend to be consistent between high quality tests, despite a wide variety of testing methods.

It looks like this site just uses human rated public IQ tests. But it would have been more interesting if an IQ test was developed specifically for AI. I.e. a test that would aim to Factor out the strength of a model general cognitive ability across a wide variety of tasks. It is probably doable by doing principal component analysis on a large set of benchmarks available today.

This website's method doesn't work at all for humans the way it works for LLMs. For humans, there is a strict time limit on these IQ tests (at least in officially recognised settings like Mensa). This kind of sequence completion is mostly a question of how fast your brain can iterate on problems. Being able to solve more questions within the time limit means you get a higher score because your brain essentially switches faster. But for LLMs, they just give them all the time in the world in parallel and see how many questions they can solve at all. If you look at the examples, you'll see some high end models struggling with some the first questions, that most humans would normally get easily. Only the later ones get hard where you really have to think through multiple options. So a 100 IQ LLM in here is not technically more intelligent in IQ test questions than 50% of humans.

If anything, this shows that some LLMs might win against humans because they can spend more time thinking per wall clock time interval thanks to the underlying hardware. Not because they are fundamentally smarter.

None of the models did actually "reason" about what the problem could possibly be - like none of them considered that more intricate patterns are possible in a 3x3 grid (having taken this kinds of test earlier in life, I still had a few seconds of indecision, thinking whether this is the same kind of test that I've seen and not some more elaborate one), and none of them tried solving the problem column-wise (it is still possible by the way) - personally, I think that indicates a strong bias present in the pretraining. For what it's worth, I would consider a model that would come up with at least a few different interpretations of the pattern while "reasoning" to be the most intelligent one - irrespective of the correctness of the answer.

Just glancing at the bar graphs, the vision models mostly suck across the board for each question. Whereas verbal ones do OK.

And today's example of clock faces (#17) does a good job of demonstrating why: because when a lot of the diagrams are explained verbally, it makes it significantly easier to solve.

Maybe it's just me, but #17 for example - it's not immediately obvious those are even supposed to represent clocks, and yet the verbal prompt turns each one into clock times for the model (e.g. 1:30) which feels like 50% of the problem being solved before the model does anything at all.

Human IQ is norm-referenced psychometrics under embodied noise. Calling both “IQ” isn’t harmless, it invites bad policy and building decisions on a false equivalence. Don’t promote it.

They run each model through the political leaning quiz.

Spoiler alert: They all fall into the Left/Liberal box. Even Grok. Which I guess I already knew but still find interesting.

As I understand it, chess AI isn't actually particularly good at playing chess intelligently, it's just that high level chess devolves into memorization and computers have an infinite ability to memorize scenarios.

In a similar way, from what I remember from taking an IQ test as a child, the tests are built on the assumption that the test taker has a finite memory and a finite amount of time to learn. In that case, having learned and remembered an unusually large amount for one's age could reasonably correlate with intelligence. However, without that limitation, the ability to answer the questions may not actually correlate with intelligence.

I'd say, as a group, those with a higher IQ than another group from a random selection of a normally distributed population, they can be expected to perform better on mental tasks that we care about. But at the individual level? Meaningless. Feynman was ~120. I, who have not contributed to anything like quantum physics scored higher, much higher.

For AI, an IQ test is interesting, but I would randomize the temperature (and other knobs) and take lots of samples. Keep in mind a relatively low IQ can blow away an AI on all kinds of things like compassion, understanding the pains of the human condition, self-sacrifice, etc. (etc. means a whole book could be worth exploring).

It seems fairly obvious to me that an LLM is the projection intelligence in the language domain. In other words, if you killed Intelligence and gave it a push in direction of language, the chalk outline you could draw around its dead body on the ground would be an LLM.

Full disclosure: I have taken 2 IQ tests, both online and timed. First was in late 90's after graduated electonics eng. was free, scored 149. After 4 years and obtaining theoretical physics degree, I did another scoring 169. The second test was not free, but I did not pay. I got the second test results because the test site owner personally emailed me my results for free with congrads, because they were the highest ever recorded on the site to date. I did both for fun just see the questions, I think both results are meaningless, the same variability occurs on farmers studied as mentioned above.

For whatever this benchmark is worth, it's yet another metric showing Gemini 2.5 Pro is really one of the best all-around models (despite being a bit older now), and available without a subscription.

These "IQ" results are so different than metrics like GPQA, AIME, SWE Bench, etc.

https://artificialanalysis.ai/leaderboards/models

Or the visual one, where you have two images that are very similar, and have to find the difference.

In fact, the test depends on vision - yet all models perform poorly with that capability?

Even assuming that companies prune out authoritarianism from their models for whatever reason, surely we'd expect at least one of them to drift over into mild economic right-wing territory. It'd be interesting to know what is causing that bias.

https://emiruz.com/post/2020-12-01-iq-rabbit-hole/

Overall this is fun but not sure anyone in their right mind will be selecting an LLM based on this IQ benchmark

But that's the first IQ test.

If the idea is to measure the ability of an LLM to correctly lookup the correct answer in its encyclopedic database, then surely there are better ways to measure that performance than using a test designed for humans without giving humans the answers in advance.

Within 2–3 years, we’ll see practical “personal LLMs” with effectively infinite memory via retrieval + lightweight updates, feeling continuous but not actually rewriting the core brain.

Within 5–10 years, we’ll likely get true continual-learning systems that can safely update weights live, with mechanisms to prune bad habits and compress knowledge—closer to how a human learns daily.

The rub is less can we and more should we: infinite memory + unfiltered feedback loops risks building a paranoid mirror that learns every user’s quirks, errors, and biases as gospel. In other words, your personal live-updating LLM might become your eccentric twin.

It is interesting to look at the political spectrum as well (https://www.trackingai.org/political-test) - ar are liberals, even Grok 4. The political leaning isn’t surprising either. Mainstream models need to be broadly acceptable, which in practice means being respectful of all groups. An authoritarian right-wing model might work for one country, group, or religion, but would almost certainly be offensive elsewhere.

Isn't this like saying that a spellchecker is "very smart" because it did well at a spelling bee? It isn't, it just has a list of answers.

Even looking at the reasoning, in a majority of the cases you cannot prove that the LLM got it right because it actually found the right pattern instead of on a fluke.

Here's an example reasoning that got the right answer but that is not specific enough and therefore could apply to literally any answer (model is Bing Copilot, picked randomly):

> Option D : A shape resembling a clock. The clock shows the time 9:00.* The pattern involves shifting times across rows and columns in a logical progression. Observing the sequence in the third row, where the first two clocks show times moving forward in increments, the next logical step is a clock displaying 9:00 to fit the established rhythm. This ensures symmetry and continuity within the overall grid.

Here's a comparison to "OpenAI o4 mini high" which is a very specific answer and shows it got the logic of the puzzle correctly:

> D Each row adds +1:30, then +3:00. - Row 1: 12:00 → 1:30 (+1:30), 1:30 → 4:30 (+3:00) - Row 2: 3:00 → 4:30 (+1:30), 4:30 → 7:30 (+3:00) - Row 3: 4:30 → 6:00 (+1:30), so 6:00 → *9:00* (+3:00) (Down each column it’s +3:00 then +1:30, which also fits.)

After spitting out some disclaimers, it gave a 5 point range that nailed what my IQ has been professionally tested at. I found this very interesting and a bit alarming.

https://www.johnstossel.com/ai-is-coming-for-your-job/

All standard talking points on one page (he forgot lump of labor though). So I'd take this test with a big grain of salt.