Responsibilities
Production Monitoring and Operations
Monitor availability, latency, error rates, model and token usage, service costs, tool-call failures, and workflow completion metrics for production AI services.
Integrate AI service health and agent-workflow telemetry into observability dashboards, monitoring systems, and alerting processes.
Investigate failed, stalled, repetitive, or abnormal AI workflows and coordinate remediation with software, platform, and AI engineering teams.
Maintain operational runbooks, escalation procedures, troubleshooting guidance, and recovery processes for AI-enabled applications.
Distinguish infrastructure and application failures from issues associated with model behavior, prompts, retrieval sources, or agent orchestration.
AI Evaluation and Operational Assurance
Develop and maintain repeatable evaluation scenarios for critical AI workflows, developer assistants, and model-service updates.
Assess task completion, output quality, error patterns, inappropriate tool use, and other indicators of operational suitability.
Coordinate regression testing before changes to models, prompts, tools, workflow logic, or retrieval sources are deployed to production.
Analyze user feedback and operational results to identify where workflows require refinement, additional controls, or restrictions on automated execution.
Document evaluation results, limitations, and recommendations to support evidence-based deployment decisions.
Change and Configuration Management
Maintain version control and change records for prompts, agent instructions, tool definitions, model selections, evaluation datasets, and workflow configurations.
Coordinate model and service updates with AI engineering, cybersecurity, platform operations, and architecture stakeholders.
Support rollback to validated configurations when changes negatively affect reliability, performance, or workflow quality.
Ensure AI-related changes follow established review, testing, approval, and deployment procedures.
Monitor configuration drift and verify that production services remain aligned with approved configurations.
Security, Audit, and Compliance Support
Coordinate with cybersecurity and identity teams to verify access controls, tool permissions, logging, and auditability for production AI services.
Support investigations involving anomalous AI behavior, unauthorized actions, or potential data-handling concerns.
Collect and maintain operational evidence supporting security assessments, continuous monitoring, and service reviews.
Verify that production workflows use approved models, endpoints, tools, and data sources in accordance with established policies.
Support the implementation and validation of operational safeguards for AI-enabled functions.
Operational Improvement and Reporting
Identify repetitive operational tasks and support activities suitable for controlled AI-assisted automation.
Measure the impact of deployed workflows on processing time, support workload, service quality, and operational efficiency.
Collaborate with service delivery, Tier 2 support, SRE, and platform engineering teams to improve workflows based on operational demand.
Report trends in service health, model usage, operating costs, evaluation results, and unresolved issues.
Recommend improvements to monitoring, testing, configuration control, and operational procedures.
Location: This position is fully onsite in Chantilly, VA.
Qualifications
Required Qualifications
Active TS/SCI clearance with CI Polygraph.
BS/BA and 5 years of experience, an additional 4 years of experience may be considered in lieu of a degree.
Approximately 5–8 years of experience in platform operations, site reliability engineering, cloud engineering, software operations, applied AI/ML operations, or production application support.
Experience operating and troubleshooting production software applications or cloud services.
Experience with monitoring, logging, metrics, incident response, and configuration or change management.
Experience evaluating software behavior through repeatable test cases, regression testing, quality measures, or operational performance analysis.
Working knowledge of foundation-model or LLM-based applications, including prompts, model endpoints, tool calls, and retrieval-augmented applications.
Experience with AWS or comparable cloud platforms.
Strong analytical, troubleshooting, documentation, and communication skills.
Desired Qualifications
Experience with AWS Bedrock or comparable managed foundation-model services.
Familiarity with LLM or agent evaluation frameworks, automated evaluation harnesses, or AI application testing.
Experience with OpenTelemetry, Amazon CloudWatch, OpenSearch, Prometheus, Grafana, or comparable observability technologies.
Familiarity with ServiceNow, IT service management, and incident-management processes.
Experience with GitLab, CI/CD pipelines, and controlled configuration promotion.
Familiarity with Kubernetes and containerized application environments.
Experience monitoring cloud utilization, model consumption, and service costs.
Knowledge of risk management frameworks, continuous monitoring, or security evidence collection.
Experience supporting classified, restricted, or highly regulated environments.
Target Salary Range $112,000 - $179,000. This represents the typical salary range for this position. Salary is determined by various factors, including but not limited to, the scope and responsibilities of the position, the individual’s experience, education, knowledge, skills, and competencies, as well as geographic location and business and contract considerations. Depending on the position, employees may be eligible for overtime, shift differential, and a discretionary bonus in addition to base pay.
Benefits Statement
Peraton offers eligible employees a variety of benefits including medical, dental, vision, life, health savings account, short/long term disability, EAP, parental leave, 401(k), paid time off (PTO) for vacation, and company paid holidays. A full listing of available benefits can be viewed at https://www.careers.peraton.com/benefits.
Application Statements The application period for the job is estimated to be 30 days from the job posting date. However, this timeline may be shortened or extended depending on business needs and the availability of qualified candidates. By applying to this job, you are expressing interest in the role and the Company. During the review of your application, you may be required to participate in an on-camera interview, as well as participate in a process to verify your identity. Use of artificial intelligence (AI) tools of any kind during Peraton interviews is strictly prohibited unless the candidate has obtained prior written authorization. All interview responses must be the candidate’s own.
- EEO
- EEO: Equal opportunity employer, including disability and protected veterans, or other characteristics protected by law.