Mendel and UMass Amherst Unveil Groundbreaking Research on AI-driven Hallucination Detection in Healthcare

News provided by

Aug 06, 2024, 06:30 ET

Innovative framework developed to ensure the accuracy and reliability of AI-generated medical summaries.

SAN JOSE, Calif. and AMHERST, Mass., Aug. 6, 2024 /PRNewswire/ -- Mendel, a leader in healthcare AI, and the University of Massachusetts Amherst (UMass Amherst) have jointly published pioneering research addressing the critical issue of faithfulness hallucinations in AI-generated medical summaries. This collaborative effort marks a significant advancement in ensuring the safety and reliability of AI applications in healthcare settings.

Research Overview

In recent years, large language models (LLMs) such as GPT-4o and Llama-3 have shown remarkable capabilities in generating medical summaries. However, the risk of hallucinations—where AI outputs include false or misleading information—remains a significant concern. This study aimed to systematically detect and categorize these hallucinations to improve the trustworthiness of AI in clinical contexts.

The research team developed a robust detection framework, categorizing hallucinations into five types. A pilot study of 100 summaries from GPT-4o and Llama-3 models revealed that GPT-4o produced longer summaries (>500 words) and often made bold, two-step reasoning statements, leading to hallucinations. Llama-3 hallucinated less by avoiding extensive inferences, but it's summaries were of lower quality. Below is a table summarizing the inconsistencies identified in both models:

Model	Medical Event Inconsistency	Incorrect Reasoning	Chronological Inconsistency
GPT-4o	21	44	2
Llama-3	19	26	1

"Our findings highlight the critical risks posed by hallucinations in AI-generated medical summaries," said Andrew McCallum, Distinguished Professor of Computer Science, University of Massachusetts Amherst. "Ensuring the accuracy of these models is paramount to preventing potential misdiagnoses and inappropriate treatments in healthcare."

The study also explored automated detection methods to mitigate the high costs and time associated with human annotations. The Hypercube system, leveraging medical knowledge bases, symbolic reasoning and NLP, played a crucial role in detecting hallucinations. It provided a comprehensive representation of patient documents, aiding in the initial detection step before human expert review.

"We are committed to continually enhancing Hypercube's capabilities. The future of healthcare AI depends on reliable, accurate tools, and Hypercube's evolving features, including real-time data processing and adaptive learning algorithms, will keep it at the forefront of clinical innovation," said Dr. Wael Salloum, Chief Scientific Officer of Mendel AI.

Future Prospects

As AI continues to integrate into healthcare, addressing hallucinations in LLM outputs will be vital. Future research will focus on refining detection frameworks and exploring more advanced automated systems like Hypercube to ensure the highest levels of accuracy and reliability in AI-generated medical content. Hypercube's real-time data processing and adaptive learning algorithms will be essential in maintaining its position at the forefront of clinical innovation.

Accepted Paper

Mendel's work on Hypercube in detecting hallucinations is recognized by the academic community. The research paper, "Faithfulness Hallucination Detection in Healthcare AI," is accepted for the KDD AI conference, August 2024. It details the methodologies and technologies underpinning Hypercube's success.

For more information about the Hypercube platform try the Hypercube demo.

About Mendel

Mendel AI supercharges clinical data workflows by coupling large language models with a proprietary clinical hypergraph, delivering scalable clinical reasoning without hallucinations and ensuring 100% explainability. Headquartered in San Jose, California, Mendel is backed by blue-chip investors, including Oak HC/FT and DCM. For more information, visit Mendel or contact [email protected].

About UMass Amherst

UMass Amherst, the flagship campus of the University of Massachusetts system, is a nationally ranked public research university known for its excellence in teaching, research, and community engagement. The university fosters innovation and collaboration across a wide range of disciplines. For more information, visit UMass Amherst.

Press Contact: Jessica McNellis, Gale Strategies ([email protected])

SOURCE Mendel