A simple way to increase the accuracy of GPT4 API

You can increase the accuracy of the response for the Structured Output from GPT4 API by asking for an Explanation

How to ask for an explanation?

Just add an explanation class to each output parameter you expect, like below!

(Yes, GPT and Gemini LLMs are both smart enough to infer this)

Here is an example where I ask for the outcome from a VAERS report.

import json
import os
from openai import OpenAI
from pydantic import BaseModel
from dotenv import load_dotenv
load_dotenv()
api_key = os.getenv('GPT_API_KEY')
client = OpenAI(api_key=api_key)

class Explanation(BaseModel):
    matching_sentence_if_any: str
    matching_phrase_if_any: str
    explanation: str

class PatientInfo(BaseModel):
    hospitalized_post_vaccination: bool
    hospitalized_post_vaccination_explanation: Explanation
    required_er_or_ed_visit_post_vaccination: bool
    required_er_or_ed_visit_post_vaccination_explanation: Explanation
    had_life_threatening_event_post_vaccination: bool
    had_life_threatening_event_post_vaccination_explanation: Explanation
    disabled_post_vaccination: bool
    disabled_post_vaccination_explanation: Explanation

symptom_text = '''
    Cardio-respiratory arrest (CPA); Ventricular fibrillation (Vf); Acute myocardial infarction; Booster; This is a spontaneous report received from a contactable reporter(s) (Physician).  A 56 year-old male patient received bnt162b2 (COMIRNATY), intramuscular, administered in arm left, administration date 07Jan2022 15:30 (Lot number: FJ5929, Expiration Date: 30Apr2022) at the age of 56 years as dose 3 (booster), single for covid-19 immunisation. Relevant medical history included: "Hypertension" (unspecified if ongoing). Concomitant medication(s) included: TERAMURO. Vaccination history included: Covid-19 vaccine (Dose 1, Single, MANUFACTURER UNKNOWN), for COVID-19 immunisation; Covid-19 vaccine (Dose 2, Single, MANUFACTURER UNKNOWN), for COVID-19 immunisation. The following information was reported: IMMUNISATION (hospitalization, medically significant, life threatening) with onset 07Jan2022 15:30, outcome "unknown", described as "Booster"; On 07Jan2022 at 17:30 (2 hours after the vaccination), approximately 2 hours after the vaccination, the patient was found collapsed due to Cardio-respiratory arrest (CPA). Automated External Defibrillator (AED) showed findings of Ventricular fibrillation (Vf). Emergency cardiac catheterization revealed complete occlusion of segment #6, indicating acute myocardial infarction. The patient was hospitalized for immunisation, cardio-respiratory arrest, ventricular fibrillation, acute myocardial infarction (start date: Jan2022). On an unspecified date in Jan2022, the outcome of the events was recovered with treatment including Percutaneous transluminal coronary angioplasty/Mechanical ventilation. The reporter stated the events results in Emergency room/department or urgent care and classified the events as Hospitalized/Life-threatening (death risk caused by an adverse event). Since the vaccination, the patient has not been tested for COVID-19.   No follow-up attempts are possible. No further information is expected.; Sender's Comments: Based on the information available, a possible contributory role of the suspect BNT162B2 cannot be excluded for the reported events. The impact of this report on the benefit/risk profile of the Pfizer product is evaluated as part of Pfizer procedures for safety evaluation, including the review and analysis of aggregate data for adverse events. Any safety concern identified as part of this review, as well as any appropriate action in response, will be promptly notified to regulatory authorities, Ethics Committees, and Investigators, as appropriate
'''
completion = client.beta.chat.completions.parse(
        model="gpt-4o-2024-08-06",
        messages=[
            {"role": "system",
             "content": "You are a biomedical expert. If the value is not available, use 'Unknown' for string and -1 for int. If there is no matching sentence, leave the field empty. If there is no matching phrase, leave the field empty. "},
            {"role": "user",
             "content": f'''
             Writeup:
             {symptom_text}
             '''},
        ],
        response_format=PatientInfo,
)

result = completion.choices[0].message.content
response = json.loads(result)
print(json.dumps(response, indent=2))

Here is the response

{
  "hospitalized_post_vaccination": true,
  "hospitalized_post_vaccination_explanation": {
    "matching_sentence_if_any": "The patient was hospitalized for immunisation, cardio-respiratory arrest, ventricular fibrillation, acute myocardial infarction (start date: Jan2022).",
    "matching_phrase_if_any": "hospitalized for immunisation",
    "explanation": "The report indicates the patient was hospitalized for immunisation, cardio-respiratory arrest, ventricular fibrillation, and acute myocardial infarction."
  },
  "required_er_or_ed_visit_post_vaccination": true,
  "required_er_or_ed_visit_post_vaccination_explanation": {
    "matching_sentence_if_any": "The reporter stated the events results in Emergency room/department or urgent care and classified the events as Hospitalized/Life-threatening.",
    "matching_phrase_if_any": "results in Emergency room/department or urgent care",
    "explanation": "The report explicitly states that the events resulted in a visit to the Emergency room/department or urgent care."
  },
  "had_life_threatening_event_post_vaccination": true,
  "had_life_threatening_event_post_vaccination_explanation": {
    "matching_sentence_if_any": "The reporter stated the events results in Emergency room/department or urgent care and classified the events as Hospitalized/Life-threatening (death risk caused by an adverse event).",
    "matching_phrase_if_any": "classified the events as Hospitalized/Life-threatening",
    "explanation": "The report classifies the events as life-threatening, indicating a death risk caused by an adverse event."
  },
  "disabled_post_vaccination": false,
  "disabled_post_vaccination_explanation": {
    "matching_sentence_if_any": "",
    "matching_phrase_if_any": "",
    "explanation": "There is no information in the report indicating the patient was left disabled post-vaccination."
  }
}

In my GPT vs Gemini accuracy course, I explain how adding this field increases the accuracy and why it is also beneficial in other ways.

Check out the transcripts for this chapter

Similar Posts

One Comment

Leave a Reply