College - Author 1
College of Engineering
Department - Author 1
Industrial and Manufacturing Engineering Department
Advisor
Puneet Agarwal, College of Engineering, Industrial and Manufacturing Engineering Department
Funding Source
Cal Poly's College of Engineering Dean's Innovation Fund, Paul & Sandi Bonderson, Kim Vorrath, and The Sprague Foundation.
Date
10-2025
Abstract/Summary
This research project will investigate the ability of advanced Large Language Models (LLMs) to identify and assess misinformation across diverse forms of media, including text, images, and video. In an age where misleading content spreads rapidly across digital platforms, evaluating the reliability and integrity of AI systems tasked with fact-checking is critical. We will develop a comprehensive dataset composed of factual and misleading examples drawn from various well-known and reliable fact-checking organizations. Each item will be independently reviewed and transparently labeled to ensure reproducibility. We will then prompt a curated group of state-of-the-art LLMs—including GPT-4, Claude, Gemini, Perplexity, Grok, and Meta's LLaMA—to return a "misleadingness score" and a self-reported "confidence score" for each item. These responses will be evaluated against ground-truth labels to assess each model's accuracy and consistency. Ultimately, we will develop a tool that allows users to input various types of media. The tool will then collect responses from multiple LLMs and use a statistical model to determine its own score for how misleading the information is and the confidence in that statement. This tool will use LLMs to fact-check other LLMs, avoiding bias or hallucination from a single inaccurate answer/model. The project emphasizes reproducibility, open-source tooling, and ethical safeguards to ensure that all findings are verifiable and responsibly communicated.
October 1, 2025.
Included in
URL: https://digitalcommons.calpoly.edu/ceng_surp/114