NAISS
SUPR
NAISS Projects
SUPR
Evaluating the Effects of Reasoning and Input Language on the Theory of Response Sampling
Dnr:

NAISS 2026/4-1217

Type:

NAISS Small

Principal Investigator:

Thomas Vakili

Affiliation:

Stockholms universitet

Start Date:

2026-06-30

End Date:

2027-01-01

Primary Classification:

10208: Natural Language Processing

Webpage:

Allocation

Abstract

The project aims to evaluate Large Language Model (LLM) decision-making through the Theory of Response Sampling, which posits that LLMs' responses are driven by a concept's statistical reality (descriptive) with its idealized version (prescriptive). The project aims to investigate how input language affects the concept distribution, specifically investigating the effects of larger reasoning models. This will be done by generating sampling distributions of average, sample, and ideal prompts (N=10) and then performing statistical tests, mechanistic analysis of logits, and faithfulness analysis of thinking tokens. The goal of using Arrhenius is to run larger LLMs, such as Gemma-4-31B and Qwen-3.6-35B, to generate sampling distributions, which requires access to more GPU resources. This will be repeated across multiple languages to measure differences in responses among the LLMs. The goal is to publish the results in an academic conference exploring cross-cultural and linguistic bias in a similar setup as Sivaprasad et al. (2025) paper from ACL 2025 on response sampling bias.