I am sourcing for a Machine Learning Engineer who will take full ownership of evaluating Honcho end-to-end—building the core benchmarking harness that informs our research-driven team whether our entity representations are truly improving. Plastic Labs develops the core memory and identity layer powering agentic systems, and our product is already live in production. Because this involves complex multi-agent architectures with emergent black-box dynamics, your primary challenge is establishing rigorous standards for what constitutes quality in evolving representations without fixed ground truth, then architecting the underlying measurement infrastructure.
If you thrive on constructing evaluation metrics from scratch, analyzing deep execution traces to uncover root-cause issues, and directly engineering production-grade solutions, this team is an ideal fit.
- Architecting custom eval suites: establish objective scoring mechanisms and benchmark definitions for evolving entity representations, continually adapting them as system capabilities mature.
- Building robust evaluation infrastructure: construct scalable pipelines, automated labeling pipelines, LLM-as-a-judge frameworks, and versioned regression runs to ensure rapid response cycles.
- Extracting actionable insights: inspect execution traces and benchmark outputs to isolate systemic failures, delivering high-conviction system design updates that directly enhance output fidelity.
- Scaling agent simulation environments: expand continuous multi-agent testing capabilities to unlock higher throughput for concurrent entity evaluation.
- Driving complete execution loops: lead problem definition, data pipeline construction, automated benchmarking, and technical writeups with total autonomy and minimal administrative overhead.