AI can guess how humans will answer personality tests

By Stephen Beech

An AI chatbot can generate personality tests and predict people's responses before they take them, according to new research.

Scientists have developed a method for generating personality assessment questionnaires with ChatGPT from any source text.

To test the method, they applied it to both the fifth edition of the American Psychiatric Association's manual used to classify and diagnose mental disorders (DSM-5) and, as a deliberately unconventional example, an astrology textbook.

The findings, published in the journal iScience, showed that not only could ChatGPT be used to create and validate the questionnaires, but it could also accurately predict responses before the surveys were administered.

ChatGPT and other publicly accessible large language models (LLMs) are trained on the internet by compiling trillions of human language data inputs from websites and social media.

Because of the training, researchers have considered whether LLMs have an expert-level understanding of human language and, by extension, personality built into their algorithms.

Study lead author Rotem Monsa said: "Given that personality traits are reflected in language, LLMs may have learned the structure of human personality as a natural byproduct of their training.

"So, while they were not taught specifically psychology or personality theories, these are already embedded in the language that LLMs learn from."

To test how well LLMs can naturally assess human personality, the researchers used GPT-4 to generate two personality assessment questionnaires.

They first used excerpts from the DSM-5 as source text, with the assumption that personality traits are localized on the personality disorder continuum.

As a control, another questionnaire used an astrology textbook.

For the former, ChatGPT generated a questionnaire with personality statements based on descriptions of personality disorders from the DSM-5.

For example, based on the paranoid personality disorder section, the questionnaire asked participants to rank how much they agree with statements such as "often suspects others' motives" or "finds it easy to trust people."

The astrology questionnaire generated similar statements, but they were based on the source text's assignment of personality traits to the astrological zodiac signs.

Monsa, a doctoral student at the Hebrew University of Jerusalem in Israel, said: "We wanted to choose texts that describe human personality in very rich detail but also sit on opposite ends of a spectrum in terms of scientific grounding.

"The DSM-5 was refined through decades of clinical research and is the standard diagnostic manual in clinical psychiatry, known worldwide.

More from this section

+4

AI can guess how humans will answer personality tests

+4

Did a robot write that? Study suggests people prefer AI authors

+4

New monkey DNA map could help speed up human research

"The astrology text is very culturally based but not scientifically validated."

After the LLM-based questionnaires were generated, they were given to 600 participants alongside the Big Five personality questionnaire (BFI), the most validated personality questionnaire to date, to assess their utility.

Results from the DSM-5-sourced questionnaire showed "high internal consistency" within personality clusters.

Monsa says that means traits that typically correlate together in the real world — such as avoidance and dependency — also correlated in the participants' responses.

The results mirrored those from the BFI, a finding that helps validate the strength of the questionnaire in measuring real-life patterns of human psychology.

As predicted, the astrology questionnaire, by contrast, showed a weak internal consistency across traits.

Monsa said: "Our data suggests that the astrological elements don't reflect coherent psychological dimensions.

"Personality traits that were together in, for example, the fire elements don't actually go together in real population."

Despite the limitation, she said both questionnaires could predict life outcomes such as depression, anxiety and well-being from their responses at levels comparable to the BFI.

Monsa says that suggests that even though the assignment of personality traits to zodiac signs is not scientifically grounded, the personality-relevant content that LLMs extract from the texts retains meaningful psychological signal.

But she says the most surprising result was that ChatGPT could predict how participants would respond to the questionnaires before they were even taken.

For both questionnaires, ChatGPT anticipated the average responses and correlations between questions with high real-world accuracy, suggesting that LLMs have an "innate" understanding of personality dynamics at a population level.

Monsa said: "The fact that LLMs can predict human response patterns before seeing any human data suggests that these models have observed something generally meaningful about human psychology.

"These results tell us that LLMs are not just a good content generation tool but also function as an informed evaluator of their own output."

But she said that the results may not hold if the methods are repeated in different languages and cultures.

Monsa said: "We would assume that in other languages and cultures, the results will be not as strong as we saw here, because LLMs were trained mostly on English texts in occidental cultures."

She added: "We think it's really interesting, and several members of our lab are currently testing it using LLMs in other languages."