NAISS
SUPR
NAISS Projects
SUPR
Computational Ablation Study of an LLM-Assisted Framework for Simulation-Based Multi-Objective Manufacturing Optimization
Dnr:

NAISS 2026/4-892

Type:

NAISS Small

Principal Investigator:

Kaveh Amouzgar

Affiliation:

Uppsala universitet

Start Date:

2026-06-23

End Date:

2026-10-01

Primary Classification:

10299: Other Computer and Information Science

Webpage:

Allocation

Abstract

This project will conduct a computational ablation study of a large language model (LLM)-assisted framework for simulation-based multi-objective optimization of manufacturing systems. The framework automatically generates Python-based discrete-event simulation models and multi-objective evolutionary optimization workflows from structured production data and user input. It also includes human-in-the-loop mechanisms and an inspector agent for validating, repairing, and interpreting generated code and results. The purpose of the project is to quantify the contribution of the main components of the framework and provide statistically grounded evidence for a journal manuscript targeting Computers & Industrial Engineering. The ablation study will compare a full baseline framework against several modified configurations, including removal of the simulation blueprint, removal of the multi-objective optimization blueprint, removal of the inspector agent, removal of human-in-the-loop decisions, alternative LLM models, and different temperature settings. All configurations will be evaluated on the same manufacturing case and with the same set of random seeds to enable paired statistical comparisons. The experiments will record code quality, simulation fidelity, optimization quality, cost, and runtime metrics. Code quality will be assessed through generation success rate and counts of syntax, structural, and reasoning errors. Simulation fidelity will be evaluated by comparing key performance indicators such as throughput, work-in-progress, and energy consumption against a reference model. Optimization quality will be assessed using indicators such as hypervolume, inverted generational distance, Pareto front cardinality, and solution diversity. Statistical analysis will include paired comparisons between the baseline and ablated configurations. The computational workload consists of repeated CPU-based Python simulations and multi-objective evolutionary optimization runs using SimPy and pymoo. The LLM inference itself is performed externally through API calls, while the requested compute resources are needed to execute the simulation and optimization experiments in a controlled, reproducible, and parallelized manner. The project will generate a complete experimental dataset, summary tables, statistical results, and figures for manuscript preparation. The expected contribution is a clearer understanding of which architectural components are most important for reliable LLM-assisted simulation-based optimization. This will support the development of more transparent, robust, and accessible decision-support tools for manufacturing optimization, particularly for users who do not have expert knowledge in simulation modelling, coding, or multi-objective optimization.