When building a knowledge graph, there are a wide variety of options available to you.

You can use a Named Entity Recognition (NER) model, such as Azure AI Text Analytics, or you can use one of the leading Large Language Models (LLMs) such as OpenAI GPT-4o, Anthropic Claude 3.5 Sonnet or Google Gemini 1.5.

In this article, we will compare the resulting output from entity extraction across several LLMs, and compare against the NER approach.

We used a 6 page PDF as a source, which contained a patient's sample medical history.

For each example, we show the rendered graph that resulted from entity extraction, and a histogram of the number of extracted entities of each type.

At the end, we compare the relative cost of each approach.

Azure AI Text Analytics

Counts:
LABEL: 6
PRODUCT: 13
ORGANIZATION: 2

Claude 3.5 Sonnet (2024-06-20)

Counts:
MEDICAL_CONDITION: 80
MEDICAL_PROCEDURE: 13
LABEL: 4
MEDICAL_TEST: 10
MEDICAL_DRUG: 10
PERSON: 1
ORGANIZATION: 1
MEDICAL_THERAPY: 1

Claude Sonnet 3.5 (2024-10-22)

Counts:
MEDICAL_CONDITION: 70
MEDICAL_PROCEDURE: 9
MEDICAL_DRUG_CLASS: 4
LABEL: 2
MEDICAL_DRUG: 8
ORGANIZATION: 1
MEDICAL_THERAPY: 2
PERSON: 1

GPT-4o

Counts:
MEDICAL_CONDITION: 42
LABEL: 3
MEDICAL_TEST: 1
MEDICAL_PROCEDURE: 8
MEDICAL_DRUG: 6
MEDICAL_DRUG_CLASS: 1
MEDICAL_CONTRAINDICATION: 1
MEDICAL_THERAPY: 1
PERSON: 1

GPT-4o Mini

Counts:
MEDICAL_THERAPY: 6
MEDICAL_CONDITION: 34
PERSON: 2
MEDICAL_PROCEDURE: 3
MEDICAL_TEST: 2

Gemini 1.5 Flash 002

Counts:
MEDICAL_CONDITION: 62
MEDICAL_DRUG: 10
LABEL: 6
PRODUCT: 1
MEDICAL_PROCEDURE: 3
MEDICAL_TEST: 4
MEDICAL_CONTRAINDICATION: 1
PERSON: 2
MEDICAL_THERAPY: 1
MEDICAL_DRUG_CLASS: 2

Gemini 1.5 Pro 002

Counts:
MEDICAL_CONDITION: 78
MEDICAL_PROCEDURE: 10
MEDICAL_DRUG_CLASS: 5
LABEL: 4
MEDICAL_TEST: 4
MEDICAL_DRUG: 6
PERSON: 1

Entity Extraction Results

Across all the LLMs tested, Claude 3.5 Sonnet (2024-06-20) provided the highest quantity of extracted entities. Gemini 1.5 Pro 002 was a runner-up, and had a better price/quantity ratio.

Interestingly, Claude 3.5 Sonnet (2024-06-20) performed even better than the recently released 2024-10-22 version of the model.

Claude 3.5 Sonnet (2024-06-20)

MEDICAL_CONDITION: 80
MEDICAL_PROCEDURE: 13
LABEL: 4
MEDICAL_TEST: 10
MEDICAL_DRUG: 10
PERSON: 1
ORGANIZATION: 1
MEDICAL_THERAPY: 1

Claude 3.5 Sonnet (2024-10-22)

MEDICAL_CONDITION: 70
MEDICAL_PROCEDURE: 9
MEDICAL_DRUG_CLASS: 4
LABEL: 2
MEDICAL_DRUG: 8
ORGANIZATION: 1
MEDICAL_THERAPY: 2
PERSON: 1

Cost Comparison

(Converted at $0.10/credit, on Hobby Plan)

NER: 2.122070 credits ($0.21)
Claude 3.5 Sonnet (2024-06-20): 11.909668 credits ($1.19)
Claude 3.5 Sonnet (2024-10-22): 8.926537 credits ($0.89)
GPT-4o: 9.893492 credits ($0.99)
GPT-4o Mini: 2.037695 credits ($0.20)
Gemini 1.5 Flash 002: 2.128356 credits ($0.21)
Gemini 1.5 Pro 002: 7.532407 credits ($0.75)

Notes:

• Claude 3.5 Sonnet (2024-06-20) returned more output tokens than the 2024-10-22 model, for the same entity extraction process, which incurred the extra cost.

Overall, the 'lite' models like GPT-4o Mini and Gemini 1.5 Flash are the cheapest, but can come with the disadvantage of extracting less entities.

However, Gemini Flash does an excellent job at entity extraction, even with it being a cheaper class of model.

GPT-4o:

MEDICAL_THERAPY: 6
MEDICAL_CONDITION: 34
PERSON: 2
MEDICAL_PROCEDURE: 3
MEDICAL_TEST: 2

Gemini 1.5 Flash 002:

MEDICAL_CONDITION: 62
MEDICAL_DRUG: 10
LABEL: 6
PRODUCT: 1
MEDICAL_PROCEDURE: 3
MEDICAL_TEST: 4
MEDICAL_CONTRAINDICATION: 1
PERSON: 2
MEDICAL_THERAPY: 1
MEDICAL_DRUG_CLASS: 2

As with many applications of LLMs, you will need to compare the output vs cost based on your own content. But the speed and low cost of Google Gemini 1.5 Flash 002 is very promising as a cost-effective approach for knowledge graph generation.

Summary

Please email any questions on this article or the Graphlit Platform to questions@graphlit.com.

For more information, you can read our Graphlit Documentation, visit our marketing site, or join our Discord community.