Observability & Evaluation Engineer – Agentic AI
Save this job and keep your search organized
Create a free account to save jobs, create alerts and return to this listing from your dashboard.
By continuing, you agree to our Terms & Privacy Policy.
Location:
Charlotte, NC (Onsite) # Hybrid # W2 only We are looking for an experienced Observability & Evaluation Engineer to support production-grade Agentic AI systems. This role focuses on building telemetry, tracing, dashboards, alerts, evaluation frameworks, and operational readiness processes for Tachyon agent releases. The ideal candidate will have strong Python engineering skills, hands-on observability experience, and a solid understanding of LLMs, AI agents, RAG, and AI evaluation methodologies. 🔹
Key Responsibilities
• Implement observability and telemetry for LLM-powered applications, AI agents, tools, and platform services.
• Build evaluation suites for agent behavior, prompt quality, response quality, retrieval performance, latency, reliability, and safety.
• Develop dashboards, metrics, traces, alerts, SLOs, and production monitoring reports.
• Collaborate with platform engineers and Product Owners to define monitoring requirements and evaluation metrics.
• Automate evidence collection for release readiness, operational reviews, and governance checkpoints.
• Create runbooks and support documentation for priority agent releases.
• Analyze production behavior and recommend improvements in reliability, performance, and AI quality. 🔹 Required Qualifications
• 7+ years of engineering experience in observability, monitoring, test automation, platform operations, or AI/ML systems.
• 5+ years of hands-on Python development experience.
• 5+ years of experience with dashboards, metrics, alerts, tracing, logs, SLOs, and production monitoring.
• 5+ years of experience or demonstrated expertise in LLM evaluation, prompt evaluation, RAG evaluation, or AI quality assessment.
• Experience working in Agile engineering teams and production support environments. 🔹 Required Technical Skills
• Python, telemetry, tracing, monitoring, dashboards, alerting, and SLOs.
• Evaluation frameworks, test automation, and production operations.
• Understanding of LLMs, AI agents, RAG, prompt performance, retrieval quality, latency, and reliability metrics.
• Observability tools and OpenTelemetry concepts. 🔹 Preferred Qualifications
• Experience with GenAI observability, AI evaluation tools, ML monitoring, or platform reliability engineering.
• Experience in regulated environments requiring evidence collection and readiness documentation.
• Familiarity with Kubernetes, cloud platforms, and CI/CD pipelines. 🔹 Expected Outcomes
• Production-ready observability dashboards and evaluation suites for priority agent releases.
• Clear release-readiness evidence, actionable alerts, SLOs, and runbooks.
• Improved quality, reliability, and trust in production Agentic AI systems.
-
AI Observability Engineer
2 days ago
Charlotte, North Carolina, United States GL Global Full-timeAI Observability Engineer - GenAI & Agentic AINot sure what skills you will need for this opportunity Simply read the full description below to get a complete picture of candidate requirements.We’re looking for an experienced AI Observability Engineer to support the development and production of next-generation GenAI and Agentic AI systems.This is a great...
-
Agentic AI Engineer
2 days ago
Charlotte, North Carolina, United States EY Full-time $144,000 - $374,000 ContractLocation: Anywhere in Country At EY, we’re all in to shape your future with confidence. We’ll help you succeed in a globally connected powerhouse of diverse teams and take your career wherever you want it to go. Join EY and help to build a better working world. The opportunity Build AI agents that carry meaningful work from intent to a verified...
-
Senior Engineer- Agentic AI
5 days ago
Charlotte, North Carolina, United States Bank of America Full-time $122,000 - $200,000 ContractJob Description:At Bank of America, we are guided by a common purpose to help make financial lives better through the power of every connection. We do this by driving Responsible Growth and delivering for our clients, teammates, communities and shareholders every day.Being a Great Place to Work is core to how we drive Responsible Growth. This includes our...
-
AI Architect- Agents Development
6 hours ago
Charlotte, North Carolina, United States Synechron Full-time $120,000 - $130,000 ContractWe are: At Synechron, we believe in the power of digital to transform businesses for the better. Our global consulting firm combines creativity and innovative technology to deliver industry-leading digital solutions. Synechron’s progressive technologies and optimization strategies span end-to-end Artificial Intelligence, Consulting, Digital, Cloud &...
-
AI Observability Engineer
5 days ago
charlotte, north carolina, United States GL Global Full-timeAI Observability Engineer - GenAI & Agentic AI We’re looking for an experienced AI Observability Engineer to support the development and production of next-generation GenAI and Agentic AI systems. This is a great opportunity for someone with a background in Observability, SRE, MLOps, Platform Engineering, AI/ML or Test Automation who wants to work on...
-
Observability and Evaluation Engineer
1 week ago
Charlotte, North Carolina, United States Business Transformation Services Powered Full-timeCompany Overview: None Job Description: Position Summary The Overwatch Observability & Evaluation Engineer will build telemetry, tracing, dashboards, evaluation suites, alerts, service objectives, runbooks, and readiness evidence for Tachyon agent releases. This role ensures production AI systems can be monitored, evaluated, improved, and...
-
Principal Agentic AI Architect
3 days ago
Charlotte, North Carolina, United States Cyber Space Technologies LLC Full-timeHi, Job Title: Principal Agentic AI Architect Location: Charlott NC Work Model: Day 1 Onsite Please prioritize local candidates only. Candidates must be willing to attend an in-person interview and should be able to work onsite from Day 1 without exception KEY RESPONSIBILITIES Technical Leadership & Architecture · Define and drive the long-term technology...
-
Machine Learning Engineer
2 days ago
Charlotte, North Carolina, United States NTT DATA Full-time $75 - $84 PermanentAt NTT DATA, we know that with the right people on board, anything is possible. The quality, integrity, and commitment of our employees have been key factors in our company's growth and market presence. By hiring the best people and helping them grow both professionally and personally, we ensure a bright future for NTT DATA and for the people who work...
-
Machine Learning Engineer
3 days ago
Charlotte, North Carolina, United States Business Transformation Services Powered Full-time $75 - $84 PermanentAt NTT DATA, we know that with the right people on board, anything is possible. The quality, integrity, and commitment of our employees have been key factors in our company's growth and market presence. By hiring the best people and helping them grow both professionally and personally, we ensure a bright future for NTT DATA and for the people who work...
-
Machine Learning Engineer
3 days ago
Charlotte, North Carolina, United States NTT DATA Services Full-time $65 - $80 PermanentAt NTT DATA, we know that with the right people on board, anything is possible. The quality, integrity, and commitment of our employees have been key factors in our company's growth and market presence. By hiring the best people and helping them grow both professionally and personally, we ensure a bright future for NTT DATA and for the people who work...
