In this role, you’ll own the performance, quality, and production-readiness of select innovation projects running on Databricks – the ones where we dig deep to learn what excellence really looks like, then turn those lessons into patterns we can reuse on future projects. Other teams take care of the platform and infrastructure; you’ll own the layer above it: the data foundations, governance, FinOps, lineage, runbooks, and the handoff to Enterprise Technology. When a project leaves our hands, it’s fully governed, tagged at the project level, and optimized for cost, with a low L3 incident rate, complete runbooks, and self-healing wherever possible.
This is a lead individual contributor role: you’ll set technical direction and raise the bar across teams without managing people. You’ll partner with teams across the enterprise, take on hard problems on Databricks, and help shape how we put the platform’s newest capabilities to work on the goals that matter most to State Farm. It’s a great fit if you want to grow your skills at the leading edge of Databricks and agentic AI, work on high-visibility initiatives, and do your best work in a fast-moving, experimental setting.
What You’ll Own
Data Foundations – Design and run medallion architecture (Bronze → Silver → Gold) with solid data quality, schema management, and Delta Lake tuning. Everything downstream depends on getting this right.
Application Performance – Hunt down bottlenecks in Spark queries, partitioning, caching, and Databricks cluster sizing, then fix them and keep them from coming back.
Unity Catalog & Lineage – Own schemas, access controls, and project-level tagging, with lineage-driven ticketing so upstream breaks surface right away.
MLOps / MLflow 3.0 – Take models through their full lifecycle, from experiment to production to retirement, including retraining and drift detection.
FinOps – Make Databricks cost visible by default: track, alert on, and optimize spend for every project.
ET Handoff – Define and hold the line on “done”: low L3 incident rate, complete runbooks, self-healing where possible, and governance and FinOps buttoned up. When a project crosses the handoff plane, it’s ready.
Patterns & Practices – Turn what you learn into templates and standards that make every future project better.
Agentic Integration – Bring agentic AI into everything you do: data validation, pipeline generation, testing, monitoring, documentation. If it can be augmented, it should be.
Technical Leadership & Collaboration – Set technical direction, mentor engineers, and pair with pod teams on architecture, code reviews, and debugging. You teach by doing.
Qualifications
Databricks (Non-Negotiable) – At least a year running Databricks in production at the application layer. You’ve designed and operated medallion architecture, and you know Delta Lake, Unity Catalog, and PySpark cold.
Production ML (Non-Negotiable) – You’ve kept models healthy in production, not just trained them and handed them off. You’ve been paged, written the RCA, and caught drift and silent degradation before they became someone else’s problem.
Agentic-Native (Non-Negotiable) – GenAI and agentic tools are already part of how you work every day, not an experiment. You’re at home on the command line and always looking for the next thing to automate.
MLflow 3.0 – Hands-on with the full lifecycle: registry, experiment tracking, deployment automation, retraining pipelines, and drift detection.
FinOps – You’ve tracked and optimized Databricks spend at the project level, and you treat compute budgets with the same rigor as SLAs.
Data Lineage & Observability – You’ve built or run systems that track lineage end to end and open tickets automatically on upstream breaks, schema changes, or SLA misses.
Production Handoff – You’ve prepared systems for operations teams with runbooks, self-healing patterns, incident targets, and governance checklists.
Systems Thinking – You see the whole system, not just the model or the pipeline. You’ve read Hidden Technical Debt in Machine Learning Systems, and you’ve lived it.
Engineering – Production-grade Python, PySpark, SQL; CI/CD for data and ML pipelines; and testing strategies for non-deterministic systems.
Omnigent (Nice to Have) – Bonus points if you’ve worked with Omnigent, Databricks’ open-source agent meta-harness
Our Benefits
Because work-life balance is a priority at State Farm, compensation is based on our standard 38:45-hour work week!
Potential starting salary range: $100,000 - $200,000. Starting salary will be based on skills, background, and experience. High end of the range limited to applicants with significant relevant experience and living in high cost of labor location like CA/NY/NJ.
Potential yearly incentive pay up to 15% of base salary
At State Farm, we offer more than just a paycheck. Check out our suite of benefits designed to give you the flexibility you need to take care of you and your family!
Get Paid! On top of our competitive pay, you are eligible for an annual raise and bonus.
Stay Well! Focus on you and your family’s health with our robust health and wellbeing programs. State Farm pays most of your healthcare premium, and we offer multiple healthcare plan options, including a high deductible plan. All medical plans provide 100% coverage for in-network preventative care, AND you and your family have access to vision, dental, telemedicine, 24/7 mental health professionals, and much more!
Develop and Grow! Take advantage of educational benefits like industry leading training programs, top-notch tuition assistance programs, employee resource groups, and mentoring.
Plan Ahead! Plan for those big moments in life with benefits like fertility/IVF/adoption assistance, college coaching, national discount programs, interactive monthly financial workshops, free financial coaching, and more. You can also start a savings account or consider financing through our State Farm Federal Credit Union!
Take a Little “You” Time! You will have access to our generous time off policies designed so you can plan around holidays, family events, volunteering, or just to take a relaxing day off. With the opportunity to initially earn up to 20 days annually plus parental leave, paid holidays, celebration day, life leave (40 hours/year), bereavement leave, and community service/education support days, there will be plenty of time for you!
Give Back! We offer several ways to give back through our Matching Gift Program, Good Neighbor Grant Program, and the Employee Assistance Fund.
Finish Strong! Plan for retirement using free financial advisors and a 401(k) plan with company contributions of up to 7% of your salary.
Visit our State Farm Careers page for more information on our benefits, locations, and the hiring process of joining the State Farm team!
At the State Farm Applied Innovation Lab, we’re building an Agentic SDLC control plane, where GenAI helps with every part of how we build, test, deploy, and run software. We’re looking for someone who has run real workloads on Databricks in production and knows Delta Lake, Unity Catalog, and MLflow inside and out. Just as important, they already work the way we do, reaching for agentic tools naturally whether they’re building something new or tracking down a bug. If you’re happiest at the command line, you’ll feel right at home here.
In this role, you’ll own the performance, quality, and production-readiness of select innovation projects running on Databricks – the ones where we dig deep to learn what excellence really looks like, then turn those lessons into patterns we can reuse on future projects. Other teams take care of the platform and infrastructure; you’ll own the layer above it: the data foundations, governance, FinOps, lineage, runbooks, and the handoff to Enterprise Technology. When a project leaves our hands, it’s fully governed, tagged at the project level, and optimized for cost, with a low L3 incident rate, complete runbooks, and self-healing wherever possible.
This is a lead individual contributor role: you’ll set technical direction and raise the bar across teams without managing people. You’ll partner with teams across the enterprise, take on hard problems on Databricks, and help shape how we put the platform’s newest capabilities to work on the goals that matter most to State Farm. It’s a great fit if you want to grow your skills at the leading edge of Databricks and agentic AI, work on high-visibility initiatives, and do your best work in a fast-moving, experimental setting.
What You’ll Own
Data Foundations – Design and run medallion architecture (Bronze → Silver → Gold) with solid data quality, schema management, and Delta Lake tuning. Everything downstream depends on getting this right.
Application Performance – Hunt down bottlenecks in Spark queries, partitioning, caching, and Databricks cluster sizing, then fix them and keep them from coming back.
Unity Catalog & Lineage – Own schemas, access controls, and project-level tagging, with lineage-driven ticketing so upstream breaks surface right away.
MLOps / MLflow 3.0 – Take models through their full lifecycle, from experiment to production to retirement, including retraining and drift detection.
FinOps – Make Databricks cost visible by default: track, alert on, and optimize spend for every project.
ET Handoff – Define and hold the line on “done”: low L3 incident rate, complete runbooks, self-healing where possible, and governance and FinOps buttoned up. When a project crosses the handoff plane, it’s ready.
Patterns & Practices – Turn what you learn into templates and standards that make every future project better.
Agentic Integration – Bring agentic AI into everything you do: data validation, pipeline generation, testing, monitoring, documentation. If it can be augmented, it should be.
Technical Leadership & Collaboration – Set technical direction, mentor engineers, and pair with pod teams on architecture, code reviews, and debugging. You teach by doing.
Databricks (Non-Negotiable) – At least a year running Databricks in production at the application layer. You’ve designed and operated medallion architecture, and you know Delta Lake, Unity Catalog, and PySpark cold.
Production ML (Non-Negotiable) – You’ve kept models healthy in production, not just trained them and handed them off. You’ve been paged, written the RCA, and caught drift and silent degradation before they became someone else’s problem.
Agentic-Native (Non-Negotiable) – GenAI and agentic tools are already part of how you work every day, not an experiment. You’re at home on the command line and always looking for the next thing to automate.
MLflow 3.0 – Hands-on with the full lifecycle: registry, experiment tracking, deployment automation, retraining pipelines, and drift detection.
FinOps – You’ve tracked and optimized Databricks spend at the project level, and you treat compute budgets with the same rigor as SLAs.
Data Lineage & Observability – You’ve built or run systems that track lineage end to end and open tickets automatically on upstream breaks, schema changes, or SLA misses.
Production Handoff – You’ve prepared systems for operations teams with runbooks, self-healing patterns, incident targets, and governance checklists.
Systems Thinking – You see the whole system, not just the model or the pipeline. You’ve read Hidden Technical Debt in Machine Learning Systems, and you’ve lived it.
Engineering – Production-grade Python, PySpark, SQL; CI/CD for data and ML pipelines; and testing strategies for non-deterministic systems.
Omnigent (Nice to Have) – Bonus points if you’ve worked with Omnigent, Databricks’ open-source agent meta-harness
