AI AGENT DEVELOPMENT & EVALUATION
Primary Care, Acute Symptom Triage & Travel Health | Contract, up to 10 hrs/week
ABOUT THE ROLE
We are seeking Nurses & Nurse Practitioners with clinical experience in one or more of the following areas:
- acute symptom management
- CHF and hypertension management
- diabetes management
- travel health
to serve as a subject matter expert supporting the evaluation of AI-driven clinical agents. In this role, you will evaluate agent conversations and recommendations spanning acute symptom triage and escalation decisions, insulin/diabetes titration, and pre-travel health guidance, ensuring clinical accuracy, guideline adherence, and patient safety. This is a part-time, remote contract position (1099) requiring up to 10 hours per week, with flexible scheduling.
KEY RESPONSIBILITIES
- Review recorded and simulated AI agent interactions involving acute symptom triage and assess the appropriateness of escalation and de-escalation decisions.
- Review agent conversations related to insulin and diabetes medication titration for clinical accuracy and guideline adherence.
- Evaluate agent-delivered travel health guidance, including pre-travel counseling, vaccination recommendations, and destination-specific risk assessment.
- Identify clinical safety concerns, edge cases, and gaps in the agent's triage logic or clinical reasoning.
- Score and annotate agent performance using structured evaluation rubrics.
- Provide clear, actionable written feedback to clinical and product teams to inform agent refinement.
- Participate in periodic case review and calibration sessions with other clinical SMEs to ensure consistent evaluation standards.
- Help develop and refine test scenarios used in ongoing agent quality assurance.
QUALIFICATIONS
- Registered Nurses: active, unrestricted RN License Nurse Practitioners: active, unrestricted Nurse Practitioner license in the United States, with current board certification (ANCC or AANP).
- Minimum of 5+ years of direct clinical experience in primary care, urgent care, telephone/virtual triage, or a comparable acute-care setting.
- Clinical experience managing patients with diabetes, including insulin titration.
- Strong, current working knowledge of relevant triage protocols and clinical practice guidelines.
- Ability to critically evaluate clinical reasoning and identify inaccuracies or safety risks in written or recorded clinical interactions.
- Comfortable working independently in a remote, asynchronous contract role.
- Strong written communication skills for delivering structured clinical feedback.
RECOMMENDED
It is not a requirement that candidates have all of the recommended criteria
- Experience in telephone triage or nurse advice line settings.
- Experience in travel medicine, occupational health, or pre-travel consultation.
- Familiarity with CDC and other travel medicine guidelines.
- Prior experience in clinical quality assurance, utilization review, or case review.
- Familiarity with AI-assisted clinical tools or digital health platforms.
ENGAGEMENT DETAILS
- Contract (1099) position, up to 10 hours per week.
- Fully remote; flexible scheduling.
- Hourly compensation, commensurate with experience.
- Engagement expected to run in support of ongoing agent development and FDA evaluation milestones.
Please be aware of recruitment scams impersonating Hippocratic AI. All recruiting communication will come from @hippocraticai.com http://hippocraticai.com email addresses. We will never request payment or sensitive personal information during the hiring process.