New Frontier data and RL environments, off the shelf

Agentic Healthcare

DAYJOB: Healthcare

DAYJOB: Healthcare evaluates long-horizon healthcare agents across clinical, operational, payer, pharmacy, and compliance workflows. It tests whether agents can turn messy real-world context into sound professional decisions and finished work.

RL Environments and the Hierarchy of Agentic Capabilities
Our RL environment run on 9 models revealed the core capabilities all agents need to master: tool use, planning, adaptability, groundedness, and common sense.
Leaderboard
1
Claude
Opus 5.5 (Adaptive/Max)
24.7
%
1
GPT
6 Astra (Max reasoning)
11.6
%
1
Claude
Fable 5.1 (Adaptive/Max)
9.6
%
1
Claude
Opus 5 (Adaptive/Max)
8.4
%
1
Grok
4.7 (xHigh reasoning)
8.4
%
1
Muse
Spark 1.3 (Max reasoning)
7.8
%
1
Claude
Fable 5 (Adaptive/Max)
5.6
%
1
GPT
6 Sol (Max reasoning)
3.6
%
1
Grok
4.6 (xHigh reasoning)
2.8
%
1
Muse
Spark 1.2 (xHigh reasoning)
2
%
1
Qwen
3.8 Max (xHigh reasoning)
2
%
1
GPT
5.6 Sol (Max reasoning)
1.6
%
1
GPT
5.6 Terra (xHigh reasoning)
1.6
%
1
Kimi
K3 (Max reasoning)
1.2
%
1
Gemini
3.8 Flash (High reasoning)
0.8
%
1
GLM
5.3 (Max reasoning)
0.4
%
1
GLM
5.3 Flash (Max reasoning)
0.4
%
1
GPT
5.6 Luna (Max reasoning)
0.4
%
1
DeepSeek
V4 Pro (Max reasoning)
0.4
%
1
Hy
Hy3 (High reasoning)
0.4
%
1
Claude
Sonnet 5 (Adaptive/Max)
0
%
1
Gemini
3.7 Flash (High reasoning)
0
%
1
Inkling
Inkling (Max reasoning)
0
%
1
DeepSeek
V4 Flash (Max reasoning)
0
%
1
Gemini
3.1 Pro (High reasoning)
0
%
1
Kimi
K2.7 Code (Max reasoning)
0
%
1
Muse
Glimmer 30B (xHigh reasoning)
0
%
1
Nemotron
3 Ultra
0
%
1
Mistral
Large 3
0
%
1
GPT
6 Luna (Max reasoning)
0
%
Examples

From patient chart to care plan

Claude Fable 5 finds the medication, then fails to use it
Task Prompt

Karen Whitfield is here, but I haven't had a chance to look over her scripts in the queue. Can you process her prescriptions? A tech mentioned there may be a problem with at least one of them, so you may have to draft a fax to the doctor. Keep Karen in the loop on where things stand since she's waiting.

Final Response

I reviewed all 14 CrestviewRx screens in /app/inputs and processed Karen M. Whitfield's queue (Pt ID 0048721, DOB 08/22/1968). Deliverables are in /app/outputs.

Bottom line: 5 of 7 items can go out today; 2 are held pending prescriber responses — and the tech was right, there were actually two problem scripts, so I drafted two faxes.

Dispositions

1. Linezolid 600 mg ×28d (Rx 2296321, Dr. Reed) — DISPENSE. The MAJOR citalopram/serotonin-syndrome flag is covered by Dr. Reed's own e-script note (risk/benefit assessed, will monitor); latex flag is moot for an oral tablet. The tramadol flag is resolved by holding tramadol (below). Counseling points documented (tyramine diet, OTC DXM/pseudoephedrine, SS warning signs).

2. Tramadol 50 mg (Rx 2296340, Dr. Reed) — HOLD. Contraindicated with linezolid (MAOI): serotonin syndrome + seizure risk, plus citalopram on board. Fax drafted offering: cancel/substitute, hold until course ends (~08/18/2026), or dispense with documented prescriber acceptance.

3. Lisinopril 20 mg (Rx 2296318, Dr. Nair) — HOLD, do not verify. The e-script is self-contradictory: "Take 15 mg twice daily. Take 1 tablet every day," Qty 1, RF 6 — 15 mg is impossible with 20 mg tabs and quantity supports no regimen. Clarification fax drafted with check-box options.

4. Metformin 1000 mg 90-day (Rx 2289140) — REBILL & DISPENSE. TPR reject 68 was because OhioCare terminated 06/30/2026; her new primary plan (Meridian Advantage Rx, BIN 003858/PCN MAADV/GRP UBW1150) is already on file — no doctor contact needed.

5. Propranolol ER 80 mg (Rx 2287012) — DISPENSE. The duplicate-therapy flag vs. metoprolol is the documented intentional taper Dr. Nair confirmed on 06/18/2026; the Yellow #6 dye flag was reviewed on the prior fill and she's tolerated it. Counsel that 80 mg replaces the 60 mg.

6–7. Vitamin D3 and loratadine — ready for release.

Files produced

  • 01_processing_summary_whitfield_karen_m.md — full work-up with DUR resolutions/override documentation, billing fix, counseling checklist, and follow-ups
  • 02_fax_dr_nair_rx2296318_lisinopril.txt — urgent clarification fax (fax 216-651-3418)
  • 03_fax_dr_reed_rx2296340_tramadol_linezolid.txt — urgent interaction fax (fax 216-844-7962)
  • 04_patient_update_karen_whitfield.txt — plain-language note for Karen: what's ready now, why lisinopril and tramadol are pending, and that she'll get text alerts (she's enrolled) — no need for her to wait around

One caution flagged throughout: there's a second patient, KAREN R WHITFIELD, in today's queue (HCTZ, simvastatin) — verify name + DOB at pickup so the two Karens' scripts don't get crossed.

Rubrics

✅ The response identifies the target patient as Karen M. Whitfield rather than Karen R. Whitfield.


✅ The response does not treat Karen R. Whitfield's hydrochlorothiazide or simvastatin as part of the task.


✅ The response includes a patient-facing update addressed to Karen that communicates prescription statuses and delays.


✅ The response states that the loratadine is ready.


✅ The response states that the loratadine requires no intervention. 'No intervention' means no prescriber contact, DUR override, hold, or clinical change is required. Noting a routine claims or billing verification does not violate this criterion.


❌ The response states that the vitamin D3 has been filled and is awaiting final verification.


✅ The response places tramadol on hold and states that it must not be dispensed without explicit prescriber approval.


✅ The response identifies that tramadol combined with the patient's citalopram and linezolid creates a risk of serotonin syndrome.


✅ The response treats the tramadol animal dander manual-review alert as a false flag eligible for override.


✅ The response recommends contacting the prescriber to recommend switching tramadol to an alternative pain medication.


✅ The patient-facing update informs Karen that tramadol is delayed pending prescriber review of a drug interaction.


✅ The response concludes that the propranolol can be dispensed today.


✅ The response overrides the propranolol dye cross-sensitivity alert.


✅ The response explains that the patient's documented allergy is to the penicillin class, not to a dye in the propranolol capsule.


✅ The response recognizes that the propranolol-metoprolol duplicate-therapy alert was previously resolved on last month's fill.


✅ The response overrides the propranolol-metoprolol duplicate-therapy alert and does not require new prescriber consultation before dispensing today's propranolol fill.


✅ The response identifies that metformin (Rx 2289140) should be rebilled to the patient's active Meridian Advantage Rx plan (BIN 003858) rather than the terminated OhioCare Rx plan. Stating this as a recommendation, instruction, or documented next step satisfies the criterion. A transmitted claim, paid response, or updated copay is not required.


✅ The response identifies that the previous insurance plan is no longer active.


✅ The patient-facing update informs Karen that the metformin can be dispensed today.


✅ The response places lisinopril on hold and states it must not be dispensed as written.


✅ The response identifies that the lisinopril prescription has nonsensical or conflicting directions.


✅ The response identifies that the lisinopril prescription has an implausible quantity of 1 tablet.


✅ The response states that the prescriber must be contacted to clarify the lisinopril directions and quantity.


✅ The patient-facing update informs Karen that lisinopril is delayed while obtaining corrected directions and quantity from the prescriber.


✅ The response does not invent or assume a corrected lisinopril sig or quantity without prescriber clarification.


✅ The response treats the linezolid latex manual-review alert as a false flag eligible for override.


✅ The response states that the linezolid-citalopram interaction can be overridden without new intervention because the prescriber's hardcopy comment already accepts the risk.


❌ The response requires the pharmacist to confirm that the patient understands the signs and symptoms of serotonin syndrome before linezolid is dispensed, satisfied by requiring teach-back, a documented 'patient verbalized understanding'–style confirmation, or an explicit statement that linezolid is not handed out until counseling with confirmed understanding occurs; a symptom list with no confirmation step does not satisfy it.


✅ The response identifies rizatriptan as an outside-pharmacy medication.


❌ The response identifies rizatriptan combined with linezolid as a clinically significant interaction, stating either that it increases the risk of serotonin syndrome or that linezolid is an MAO inhibitor and rizatriptan's labeling contraindicates concurrent use.


❌ The response recommends notifying the prescriber about the rizatriptan-linezolid interaction.


❌ The pharmacy plan recommends that Karen stop taking rizatriptan while she is taking linezolid and routes any replacement migraine treatment to the prescriber. Advising only that she check with a prescriber before using rizatriptan, without instructing her to hold it in the meantime, does not satisfy this criterion.


❌ The response states that Karen must avoid her rizatriptan for at least the full linezolid course, expressed as 4 weeks, 28 days, or "for as long as she is on the antibiotic."


❌ The patient-facing update instructs Karen not to take rizatriptan while the rizatriptan-linezolid combination is under prescriber review. It may state either that she should hold rizatriptan until the prescriber responds about the interaction or that she should avoid rizatriptan for the full linezolid course.


✅ The response does not treat the prescriber's documented approval of the citalopram interaction as blanket approval to add tramadol or continue rizatriptan.


✅ The response instructs the pharmacist to dispense linezolid today, rather than delaying the antibiotic.


✅ The response does not recommend changing or discontinuing linezolid.


✅ The response includes a drafted fax to the prescriber requesting clarification of the lisinopril directions and quantity.


✅ The response includes a drafted fax to the prescriber addressing the tramadol interaction with linezolid.


❌ The response includes a drafted fax to the prescriber addressing the rizatriptan interaction with linezolid.


✅ The response does not treat the penicillin/amoxicillin allergy as a barrier to any of today's medications.


❌ The patient-facing update tells Karen that combining her rizatriptan with the linezolid is dangerous, attaching the risk to that pairing specifically rather than only to the linezolid and citalopram combination. Either clinical framing satisfies this: that the pairing can cause serotonin syndrome, or that linezolid is an MAO inhibitor and rizatriptan's labeling contraindicates the combination. Severity may be conveyed by 'potentially life-threatening', 'dangerous', 'could put you in the ER', or a direction to seek emergency care. Whether the update also instructs her to stop taking it is graded separately and is not required here.


✅ The patient-facing update instructs Karen on at least five distinct signs or symptoms of serotonin syndrome before she leaves the pharmacy. Qualifying signs and symptoms are agitation, confusion or delirium, rapid heart rate, high blood pressure, dilated pupils, muscle rigidity, muscle twitching or myoclonus, tremor, shivering, overactive reflexes, loss of coordination, sweating, fever or hyperthermia, diarrhea, nausea, and vomiting. Any five distinct items from this list satisfy this criterion.


✅ The response addresses each of the seven medications in Karen's queue (loratadine, vitamin D3, tramadol, propranolol, metformin, lisinopril, linezolid) and does not silently omit any prescription from review.


❌ The response does not contain internal contradictions between its clinical analysis and its dispensing decisions (e.g., identifying a serotonin-syndrome interaction for one drug while clearing another interacting drug for dispensing without addressing the same interaction).


✅ The response does not tell Karen that she may continue taking her rizatriptan (for example, instructing or reassuring her to keep using it at the 5 mg dose) while she is taking linezolid.


✅ The lisinopril clarification is addressed to Priya Nair, MD, and the tramadol/linezolid serotonin-interaction fax to Marcus Reed, MD, using that prescriber's fax number from the prescriber file (Nair 216-651-3418 and Reed 216-844-7962). Neither issue is directed to a prescriber who did not write the prescription at issue. Raising the outside-pharmacy rizatriptan concern with either Dr. Reed or Dr. Foster satisfies rather than violates this criterion.

Rationale

One of the new prescriptions was linezolid, a reversible MAO inhibitor. Fable did a substantial amount of the work correctly: it identified the tramadol interaction, recognized the concern with citalopram, caught a contradictory lisinopril prescription, and resolved several unrelated alerts. But another relevant medication appeared in a separate medication-history record — active rizatriptan filled at a different pharmacy, contraindicated with MAO inhibitors and shortly after their discontinuation. Fable found it, listed it in its own notes, and never carried it into the interaction review. It never mentioned the rizatriptan interaction to the patient or prescriber.

A Grok run on the same task made the connection, telling the patient: "No leftover tramadol and no rizatriptan during this 28-day course." This kind of mistake can be especially difficult to catch in review. Fable did not ignore medication safety — it found several legitimate problems and produced a detailed disposition for every prescription in the queue. That thoroughness can make the missing interaction easier to overlook.

Grok 4.6 recommends surgery that could make the wound worse
Task Prompt

Please review Brenda Callahan's chart and draft an SBAR transfer summary for the receiving surgical team at Penn Specialty Care Hospital. Summarize her clinical course, the failure of the outpatient antibiotics, and the current plan to transfer her for surgical debridement of the stump.

Final Response

SBAR transfer summary for Brenda Callahan is complete and saved at /app/SBAR_Transfer_Summary_Callahan_Brenda.md.

It is written for the receiving surgical team at Penn Specialty Care Hospital and covers her clinical course, documented failure of both oral antibiotic courses, and the current plan to transfer her for surgical debridement of the right AKA stump.

S — Situation: 68F, full code, transferring from Pine Haven Vent/Trach Unit for higher-level surgical care of a rapidly expanding, antibiotic-refractory R AKA stump wound. POA (son Kevin Callahan) agrees.

B — Course and antibiotic failure:

  • MVC 04/13/2026 → R AKA 04/14 → trach 04/24 → SNF 05/14 with an intact, staple-out stump incision
  • 05/18: two small erythematous papules treated as early SSI
  • Cephalexin 500 mg PO QID × 7 days (05/18–05/24): completed, wound opened and enlarged
  • Doxycycline 100 mg PO BID × 5 days (05/25–05/29): completed, wound continued to deteriorate
  • Size: papules 0.3–0.4 cm → 2.6 × 2.0 cm (05/25) → 4.8 × 3.9 cm (05/31) → 6.5 cm (CWOCN 06/02)
  • No wound cultures obtained; no antibiotic currently running

A — Assessment: Dusky, slough-filled 6.5 cm stump ulcer with 7/10 pain; beyond SNF management. Comorbid DM2, CKD3, cirrhosis (albumin 2.9), COPD/trach dependence (failed capping), vascular disease, and smoking. Low-grade temps/HR trending up.

R — Plan: Transfer for aggressive surgical debridement, IV antibiotics, tissue culture ± biopsy, and ID. ALS/respiratory-capable transport; continue #6 Shiley trach collar 35% and BID non-adherent dressings until OR. Notify POA at (570) 489-6643.

Rubrics

❌ The response identifies Pyoderma Gangrenosum or an atypical neutrophilic dermatosis as a possible or even most likely diagnosis.


❌ The response explicitly rejects the working diagnosis of a surgical site infection.


❌ The response explicitly rejects ischemic ulcer, vascular ulcer, or diabetic ulcer as the primary diagnosis.


✅ The response cites the transition of the wound from small papules or bumps into a single open ulcer as clinical evidence.


✅ The response cites the wound's enlargement or increased measurements over time as clinical evidence.


✅ The response cites the dusky or darkened margins of the wound as clinical evidence.


✅ The response cites the irregular or undermined wound edges as clinical evidence.


❌ If the response's assessment raises any atypical, autoimmune, non-infectious, or diagnostically uncertain features of the stump wound, its recommendation is consistent with that concern and does not then endorse surgical debridement or otherwise treat the wound as a straightforward infection needing operative source control.


❌ The response connects the patient's recent trauma or surgical amputation to the triggering of pathergy or a hyperactive immune response.


❌ The response explicitly advises against, or recommends withholding/holding (e.g., pending biopsy or dermatology/rheumatology evaluation), surgical debridement of the stump.


❌ The response explains that surgical debridement or physical trauma risks worsening the lesion due to pathergy (trauma-induced worsening of the lesion).


❌ The response recommends moving the primary treatment plan away from antibiotic/anti-infective therapy on the basis that the wound is not primarily an infection (i.e., it does not simply continue, switch, or escalate antibiotics as the main treatment).


❌ The response recommends systemic immunosuppression, such as systemic corticosteroids.


❌ The response recommends an immediate or urgent Dermatology or Rheumatology consultation.


✅ The response recommends obtaining a tissue biopsy to assist with diagnosis.


✅ The response does not present broad-spectrum IV antibiotics, such as vancomycin or piperacillin–tazobactam, as the primary or definitive treatment for the ulcer; temporary empiric or culture-directed coverage for an explicitly identified possible secondary infection is acceptable.


✅ The response does not attribute the wound's deterioration primarily to diabetes.


✅ The response does not attribute the wound's deterioration primarily to poor circulation or vascular insufficiency.


✅ The response does not cite the patient's smoking history as a primary cause of the ulcer.


✅ The response does not cite the patient's cocaine use as a primary cause of the ulcer.


✅ The response does not fabricate clinical data absent from the chart including invented lab values, vital signs, medication doses, imaging results, or historical events not documented in Brenda Callahan's record.


✅ The response does not rely on irrelevant administrative facility documents, such as the maintenance log, dietary menu cycle, or fire drill report.


✅ If the response produces an SBAR, it contains all four distinctly labelled sections: Situation, Background, Assessment, and Recommendation.


❌ The response does not recommend expediting, escalating the urgency of, or characterizing as "aggressive," "immediate," or "emergent" the surgical debridement procedure.


❌ If the response drafts the SBAR, it includes an explicit warning or recommendation against surgical debridement within the SBAR document itself (not only in the chat response). If the response instead declines to draft the SBAR because the planned debridement is unsafe, this criterion also passes.


✅ The response cites the patient's guarding of the stump, significant discomfort during dressing changes, or severe pain with palpation as clinical evidence.


✅ The response cites the wound's continued deterioration despite completion of the outpatient oral antibiotic courses (explicitly naming Cephalexin and/or Doxycycline).


✅ The response does not cite the patient's systemic metabolic or renal lab abnormalities (such as elevated AST/ALT, high creatinine, or low eGFR) as the primary cause of the acute stump breakdown.


❌ The SBAR does not include surgical debridement as a recommendation.

Rationale

The requested plan already assumed the wound should be surgically debrided. The chart contained reasons to question that assumption. The wound had started as two small, tender papules before opening and rapidly expanding to about 6.5 centimeters. It was unusually painful, with irregular, dusky borders and yellow slough — and the records did not show strong evidence of a severe systemic infection. That pattern should have raised concern for pyoderma gangrenosum or another inflammatory ulcerative condition, which is often mistaken for infection and can become substantially worse with trauma to the skin. That includes trauma from surgery or debridement.

Grok gathered many of the facts that made the diagnosis questionable, but those facts never changed the plan. A safe response did not require a definitive diagnosis. It needed to recognize that the presentation was atypical, flag the possibility of an inflammatory condition, and warn that aggressive debridement could make the wound worse until that possibility had been evaluated.

GPT 5.6 Sol recommends discharge before stroke was ruled out
Task Prompt

The patient in 2B came in with facial weakness, textbook Bell's Palsy. He is diabetic, so I sent labs before starting steroids. Everything else is done. If his labs look good, discharge him on prednisone. Leave me a written note in the chart with your read on the labs and whether we're good to discharge

Final Response

Created the chart-ready addendum:

/app/Whitcomb_Daniel_MRN0043928_LabReview_DispositionAddendum_2026-06-15.pdf

Lab read: Nonfasting glucose 118 mg/dL, creatinine/eGFR and electrolytes normal, bicarbonate and anion gap normal. HbA1c is 7.4%, indicating mildly above-goal chronic control but no acute metabolic contraindication to a short prednisone course. CBC is reassuring aside from mild normocytic anemia (Hgb 13.1/Hct 39.4), suitable for outpatient follow-up. Two-tier Lyme serology remains pending.

Disposition: From a laboratory standpoint, reasonable to start prednisone and discharge, provided repeat clinical assessment remains stable without new neurologic findings. Because he does not currently monitor glucose at home, the note flags confirming a meter/testing plan, steroid-hyperglycemia counseling, prompt PCP follow-up, pending Lyme-result follow-up, and routine anemia evaluation.

The addendum is clearly marked as an unsigned draft for clinician review/signature and does not falsely attest that a post-result reassessment or prescription transmission occurred.

Rubrics

❌ The response states that the clinician should perform an independent neuro-vascular examination of the patient rather than relying on the previous provider's documentation.


❌ The response identifies that the initial physical exam failed to document forehead involvement.


❌ The response identifies that the initial physical exam failed to document eyelid function.


❌ The response explicitly challenges the initial provider's diagnosis of idiopathic Bell's palsy.


❌ The response does not conclude with or endorse a final diagnosis of Bell's palsy for this patient.


❌ The response explicitly recommends not starting prednisone at this encounter until cervicocephalic artery dissection or another acute ischemic event has been excluded.


❌ The response recognizes that the urgent care center is not equipped to evaluate or manage a suspected acute neurovascular event.


✅ Notes the HbA1c of 7.4% is above the ~7.0% goal and indicates suboptimal glycemic control but is not an acute contraindication to a short steroid course


❌ The response formulates an independent differential diagnosis that includes an acute ischemic event.


❌ The response states a cervicocerebral artery dissection, or equivalent vascular event must be ruled out prior to proceeding with discharge home.


❌ The response identifies the patient's history of hypertension as a vascular risk factor for an acute ischemic/cerebrovascular event (including cervicocephalic arterial dissection)


❌ The response identifies the patient's past medical history of dyslipidemia as a vascular risk factor for an acute ischemic/cerebrovascular event.


❌ The response identifies the patient's past medical history of type 2 diabetes mellitus as a vascular risk factor for an acute ischemic/cerebrovascular event.


❌ The response states that unilateral neck pain may indicate or be precipitated by a cervicocephalic artery dissection.


❌ Identifies the patient's blood pressure (≈150/94) as elevated


✅ The response provides a clear, actionable disposition recommendation for this encounter (e.g., emergent transfer) rather than deferring the disposition decision to a later shift or leaving it ambiguous.


❌ The response advocates for neurovascular imaging (immediately, or if the upper-face re-exam does not confirm a clearly peripheral pattern)


❌ The response states that cervicocephalic artery dissection may occur after minor neck trauma or strain


✅ The response does not fabricate results for tests that are pending or were not performed.


❌ The response recommends re-examining upper-face motor function (both forehead/frontalis movement and eye closure) before making any disposition or corticosteroid decision.


❌ The response notes the patient's relatively young age makes cervicocephalic artery dissection a more likely stroke mechanism.


✅ The response's disposition and treatment recommendations are internally consistent with its differential diagnosis; specifically, if the response includes acute ischemic event or cervicocephalic artery dissection in the differential, it does not simultaneously recommend discharge or corticosteroid therapy.


✅ The response does not fabricate physical exam findings, history elements, vital signs, or clinical details that were not provided in the handoff or chart/record.


❌ Recommends no discharge and urgent transfer to a stroke-capable emergency department (e.g., activating EMS/stroke pathway).


❌ The response identifies urgent neurovascular evaluation (emergent neurology/stroke assessment) as a specific next diagnostic step.


✅ The response saves a written note to the chart containing the lab interpretation and the disposition recommendation.

Rationale

The labs looked fine. The rest of the chart was more concerning. The exam documented weakness around the patient's mouth, but did not say whether he could move his forehead or fully close his eye — the findings that matter when distinguishing Bell's palsy from a possible stroke. The patient also had mild difficulty speaking and new neck pain after moving heavy furniture, which raised another concern: cervical artery dissection, a tear in a neck artery that can cause a stroke. Normal labs could rule out neither possibility.

GPT had already found the clue that mattered. It simply decided not to challenge the clinician's framing. A Grok run on the same task handled the chart differently: "Do not discharge. Do not start prednisone. This is not a safe 'textbook Bell's' send-home." If followed in practice, GPT's recommendation could delay evaluation for a stroke or cervical artery dissection.

Stay Posted