The Data Platform Engineer will execute the following responsibilities:
Design, configure, and maintain the enterprise Databricks Lakehouse Platform, including workspaces, Unity Catalog, and scalable data architectures.
Administer and optimize Databricks compute, clusters, SQL warehouses, serverless capabilities, and workload management for performance, reliability, and cost efficiency.
Develop and optimize data pipelines, ETL/ELT processes, and ingestion frameworks using Apache Spark, Delta Lake, Delta Live Tables, and Databricks Workflows.
Create, maintain, and govern Unity Catalog catalogs, schemas, tables, roles/groups, and RBAC/access-control lists, aligned with AO security, IAM, and compliance policies.
Implement and manage Unity Catalog, RBAC, and data governance controls.
Automate platform provisioning, configuration, and deployments using Terraform, Python, SQL, Databricks Asset Bundles, and Databricks APIs.
Implement platform monitoring, logging, alerting, observability, capacity planning, and performance optimization.
Perform Spark and SQL performance tuning, scalability testing, and troubleshooting of platform, pipeline, and data-processing issues.
Implement cost optimization strategies, including autoscaling, auto-termination, right-sizing, workload isolation, and contribute to DBU consumption forecasting, financial reporting, and TCO analysis.
Support platform upgrades, patching, versioning, release management, and adoption of new Databricks capabilities.
Integrate Databricks with enterprise data sources, governance tools, BI platforms, and AI/ML environments.
Implement security, encryption, networking, auditing, compliance, and high-availability/disaster-recovery controls.
Support security audits, ATO activities, incident response, root-cause analysis, and operational reporting.
Establish platform standards, reusable engineering patterns, reference architectures, runbooks, and technical documentation.
Collaborate with architecture, security, governance, and data engineering teams and provide technical guidance and mentorship to platform users.
Maintain cloud monitoring dashboards, capacity planning, and KPI metrics; evaluate new Databricks features and recommend adoption to improve performance, cost, or operability.