Senior Data Scientist AI Evaluation
Alpaca · Remote - Americas
At a glance
- Where
- Remote — open to applicants in the United States
- Pay
- Not disclosed by Alpaca
- Level
- Senior
- Experience
- 6+ years
- Posted
- October 7, 2026
- Last confirmed open
- 53m ago
- At Alpaca
- 17 open remote roles — mostly Finance & Legal (3), Cybersecurity (3), Software Development (3).
- How to apply
- Directly on Alpaca's own site — no NodJobs account needed.
About the role
This role designs how Alpaca measures whether its models and agents are correct, defining ground truth, scoring methods and eval loops for the company, and uses results to improve the systems. It suits a senior individual contributor with 6 to 10 years in quantitative data science or ML and focused experience in measurement or evaluation.
What you'll do
- Design AI evaluations by defining ground truth, metrics, and scoring methods for models and agents.
- Build repeatable eval loops that track quality over time and catch regressions before release.
- Work with engineering and analytics engineering to operationalize eval harnesses.
- Translate eval results into actionable recommendations for system improvements.
- Set evaluation guidelines, documentation, and review practices.
- Foster evaluation best practices and build a culture of measurable AI quality across the team.
- Partner with Product, Engineering, Analytics Engineering, and business stakeholders to define what good looks like and close the loop so evals drive iteration.
- Own the quality bar for models and agents, independent of the teams that build and optimize those systems.
- Turn ambiguous quality questions into ground truth, scoring methods, and eval loops the company can trust.
- Establish and own a greenfield eval practice that raises quality and speeds up safe rollout.
What you'll need
- Track record of quantitative measurement rigor, such as LLM/model evaluation, metric validation, or experimentation.
- Strong statistical and ML foundation, treating evaluations as experiments (sample sizing, confidence intervals, significance, handling non-determinism) and validating automated graders against human ground truth.
- Proficiency in Python and SQL, with experience evaluating models in production environments.
- Strong judgment in defining quality metrics and ground truth for ambiguous outputs.
- Excellent communication and cross-functional collaboration skills to align technical teams and leadership.
- Strong problem-solving ability in fast-paced, greenfield environments.
- 6 to 10 years in quantitative data science or ML, with focused experience in measurement or evaluation.
- Quantitative degree is a plus; equivalent industry experience is equally welcome.
Nice to have
- Hands-on LLM/agent evaluation in production, including eval harnesses, LLM-as-judge calibration, and CI regression gates.
- Experience evaluating text-to-SQL, analytics agents, or other systems where correctness is verifiable against data.
- Background in fintech, brokerage, or other domains where a wrong answer has real business or risk consequences.
- Fluency with AI tools in research and engineering workflows.
Benefits listed by Alpaca
- Competitive salary and stock options
- Health benefits
- New hire home-office setup allowance
- Monthly stipend via a company card
Tools & skills mentioned
Summarized by NodJobs from Alpaca's posting.
Full posting from Alpaca
Your Role: We're looking for a Senior Data Scientist, AI Evaluation to design how Alpaca measures whether our models and agents are actually right. You'll be a senior individual contributor who turns ambiguous quality questions into ground truth, scoring methods, and eval loops that the company can trust—and uses those results to make the systems better. You'll build on an established data foundation, so the focus is raising quality and speeding up safe rollout. You'll own the quality bar, independent of the teams that build and optimize those systems. This role is for someone who cares as much about whether an answer is correct as about whether a model can generate one. You'll partner with Product, Engineering, Analytics Engineering, and business stakeholders to define what "good" looks like, build the evaluations that test it, and close the loop so evals drive iteration. If you have a strong quantitative background, have shipped rigorous, measurable work (evaluation, experimentation, or model validation), and want ownership over a greenfield eval practice at a fast-growing brokerage-infrastructure company, this is the role. What You'll Do • Design AI evaluations: Define ground truth, metrics, and scoring methods for models and agents. • Build repeatable eval loops: Track quality over time and catch regressions before release. • Partner on infrastructure: Work with engineering and analytics engineering to operationalize eval harnesses. • Drive iteration: Translate eval results into actionable recommendations for system improvements. • Establish quality standards: Set evaluation guidelines, documentation, and review practices. • Mentor and align: Foster evaluation best practices and build a culture of measurable AI quality across the team. What We're Looking For • Track record of quantitative measurement rigor (e.g., LLM/model evaluation, metric validation, or experimentation). • Strong statistical and ML foundation—you treat evaluations as experiments (sample sizing, confidence intervals, significance, handling non-determinism) and validate automated graders against human ground truth. • Proficiency in Python and SQL, with experience evaluating models in production environments. • Strong judgment in defining quality metrics and ground truth for ambiguous outputs. • Excellent communication and cross-functional collaboration skills to align technical teams and leadership. • Strong problem-solving ability in fast-paced, greenfield environments. • 6–10 years in quantitative data science or ML, with focused experience in measurement or evaluation. A quantitative degree is a plus; equivalent industry experience is equally welcome. Nice to Have • Hands-on LLM/agent evaluation in production, including eval harnesses, LLM-as-judge calibration, and CI regression gates. • Experience evaluating text-to-SQL, analytics agents, or other systems where correctness is verifiable against data. • Background in fintech, brokerage, or other domains where a wrong answer has real business or risk consequences. • Fluency with AI tools in research and engineering workflows. How We Take Care of You: • Competitive Salary & Stock Options • Health Benefits • New Hire Home-Office Setup: One-time USD $500 • Monthly Stipend: USD $150 per month via a Brex Card Alpaca is proud to be an equal opportunity workplace dedicated to pursuing and hiring a diverse workforce. Recruitment Privacy Policy
NodJobs is free for job seekers — we never charge to apply, and this role takes you straight to the company's own application.
Similar live jobs
Staff Machine Learning Engineer, Ads Forecasting & Scenarios
Staff Machine Learning Engineer
Staff Applied Scientist
Upstart
Principal Agentic Engineer, AI Ecosystem
Life360
Director, Internal AI Operations
LaunchDarkly