Security and compliance: Implement PII protection, encryption and tokenization, row/column-level security, lineage, retention and audit readiness with Security, Risk, Compliance and Data Governance.
Design reviews: Lead technical design reviews, write architecture decision records, and resolve cross-team technical trade-offs.
Mentoring: Mentor senior and mid-level engineers and raise engineering quality across the data organization.
Data products: Work with Analytics, Data Science/ML, Risk, Fraud and Finance to turn business needs into reliable, documented data products.
Evaluation: Evaluate new Databricks capabilities and tools and recommend adoption where there is clear value.
REQUIRED SKILLS
Experience: 12+ years in data or software engineering, with at least 5 years in a technical lead, staff or principal role.
SQL: Expert-level SQL: complex analytical queries, window functions, CTEs, query plan analysis and tuning on very large datasets (billions of rows).
Spark and big data: Deep experience with Apache Spark (PySpark and/or Scala) and the ecosystem: Hadoop, Hive, Kafka, Parquet/ORC/Delta.
Databricks in production: Hands-on production use of Delta Lake, Unity Catalog, Workflows/Jobs, Databricks SQL, and cluster and cost management.
Pipelines: Proven design and operation of large-scale batch and streaming pipelines, including Structured Streaming, Delta Live Tables and Kafka, on a major cloud.
Migration: Experience migrating on-premises or EMR-based Hadoop/Hive platforms to Databricks.
Performance and cost: Proven Spark and Databricks performance tuning and cost optimization, including cluster policies.
Data modeling: Dimensional, entity-centric, Data Vault or medallion modeling, including slowly changing dimensions, incremental/CDC loads and soft deletes.
Orchestration and CI/CD: Airflow, Databricks Workflows or Oozie; Git, Terraform or Databricks Asset Bundles, and automated testing and data quality checks.
Data security: Implementation of PII protection: encryption, tokenization, and row/column-level security.
Leadership and communication: Mentoring engineers, and explaining technical trade-offs clearly to engineers and executives.
EDUCATION: Graduate in Computer Science, Data Science, or a related field. 12+ years of experience in data modelling or data analysis.
KEY SKILLS:Databricks, Big Data Technologies
Lakehouse design: Own the detailed design of the Databricks lakehouse: ingestion, bronze/silver/gold layers, Delta Lake design, Unity Catalog setup, and workspace and cluster configuration.
Migration: Lead the hands-on migration of legacy Hadoop/Hive/Spark and warehouse workloads to Databricks, including re-platforming HQL and SQL, validating data parity, and safe cutover.
Engineering standards: Set standards for SQL, PySpark and Delta Live Tables/Lakeflow pipelines, covering code review, testing, CI/CD and observability.
Performance and cost: Tune performance and cost: partitioning, liquid clustering/Z-ordering, file compaction, join and shuffle tuning, Photon, cluster right-sizing and DBU spend governance.
Data modeling: Design canonical, auditable entity models (applications, accounts, customers, transactions) across source systems.
Security and compliance: Implement PII protection, encryption and tokenization, row/column-level security, lineage, retention and audit readiness with Security, Risk, Compliance and Data Governance.
Design reviews: Lead technical design reviews, write architecture decision records, and resolve cross-team technical trade-offs.
Mentoring: Mentor senior and mid-level engineers and raise engineering quality across the data organization.
Data products: Work with Analytics, Data Science/ML, Risk, Fraud and Finance to turn business needs into reliable, documented data products.
Evaluation: Evaluate new Databricks capabilities and tools and recommend adoption where there is clear value.
Graduate in Computer Science, Data Science, or related field. 12+ years of experience in data engineering or related field.