Audit the platform's highest-priority assets — clusters, jobs, pipelines, notebooks, and access controls — and deliver an evidence-backed picture of the biggest risks within 30 days.
Own the design of critical Databricks standards — compute policies, Spark configuration, notebook and pipeline patterns, and Unity Catalog governance — and get teams to actually adopt them.
Serve as the escalation point for complex Databricks failures, performance regressions, and cost anomalies, digging to root cause every time.
Turn every fix into something reusable — better defaults, templates, reference architectures, and automation instead of one-off patches.
Become the person teams check in with before they build, guiding responsible adoption of AI-assisted engineering practices along the way.
Qualifications
8+ years in data engineering, platform engineering, or related software engineering roles, including 3+ years of deep hands-on Databricks experience.
Production experience with Spark internals and performance tuning — shuffle behavior, query planning, adaptive query execution, memory pressure, skew, and partitioning.
Strong working knowledge of Databricks compute options, cluster policies, instance pools, autoscaling, Photon, and workload right-sizing.
A track record of reviewing and improving notebooks, jobs, and pipelines built by other people, quickly spotting what's driving runtime, cost, or reliability risk.
Production experience with Delta Lake, Structured Streaming, Auto Loader, and data quality practices.
Strong command of Unity Catalog governance, lineage, permission models, and secure handling of sensitive data.
Experience designing resilient job orchestration, including dependency management, retries, timeouts, and alerting.
Strong communication skills, including the judgment to translate technical risk into terms stakeholders can act on.
Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent professional experience.
Preferred
Databricks certification (Certified Data Engineer or Certified Machine Learning Associate).
Experience with MLOps or feature-store platforms.
Contributions to open-source data engineering tooling.
PowerPlan is an EOE
Applicant and Candidate Privacy Notice
Please note that this is a hybrid role that involves a combination of onsite work from our corporate office as well as work from home. While we strive to accommodate flexible working arrangements when sensible, there will be times when onsite work is required. This could include scheduled office days, team meetings, client meetings, or special events.
