Technical Leadership
Serve as the technical lead for Databricks and Data Engineering programs.
Establish engineering best practices, coding standards, and architecture guidelines.
Conduct architecture reviews, code reviews, and solution assessments.
Mentor and guide Data Engineers, Technical Leads, and junior team members.
Collaborate with Enterprise Architects, Product Owners, Data Scientists, and Business Stakeholders.
Cloud & Platform Engineering
Design cloud-native data solutions on Azure, AWS, or GCP.
Implement secure, scalable, and resilient data architectures.
Optimize costs, performance, reliability, and availability of data platforms.
Lead CI/CD and Infrastructure-as-Code implementation for data workloads.
Define and implement monitoring, observability, and operational excellence practices.
Governance & Data Quality
Establish data governance, lineage, cataloging, and security controls.
Ensure compliance with organizational and regulatory requirements.
Implement data quality frameworks and monitoring solutions.
Collaborate with security and compliance teams to enforce data protection standards.
Stakeholder & Delivery Management
Partner with business leaders to understand strategic data requirements.
Translate business requirements into scalable technical solutions.
Drive delivery planning, risk management, and resource coordination.
Present architecture decisions and technical recommendations to leadership teams.
Required Technical Skills Databricks
Databricks Lakehouse Platform
Databricks Workflows
Delta Lake
Unity Catalog
Structured Streaming
Databricks SQL
Delta Live Tables (DLT)
Databricks Asset Bundles
Big Data Technologies
Apache Spark
PySpark
Spark SQL
Kafka
Real-Time Streaming Architectures
Data Lake & Lakehouse Architectures
Cloud Platforms (Strong Experience in One or More) Azure
Azure Databricks
Azure Data Factory (ADF)
ADLS Gen2
Azure Synapse Analytics
Azure Key Vault
Event Hub
AWS
S3
EMR
Glue
Redshift
Lambda
IAM
GCP
BigQuery
Cloud Storage
Dataflow
Pub/Sub
Composer
Programming
Python
SQL
PySpark
Scala (Preferred)
Shell Scripting
DevOps & Automation
Git
Azure DevOps
GitHub Actions
Jenkins
Terraform
Docker
Kubernetes
Data Warehousing
Dimensional Modeling
Star & Snowflake Schema
Data Vault
Master Data Management
Data Governance Frameworks
Required Experience
12-15 years of overall IT experience.
Minimum 6+ years of hands-on Databricks experience.
7+ years of experience in Spark/PySpark-based data engineering.
Experience designing enterprise-scale Data Lake and Lakehouse solutions.
Strong experience leading large data transformation and modernization programs.
Experience handling structured, semi-structured, and unstructured data processing.
Experience supporting analytics, AI/ML, and reporting ecosystems.
Proven experience managing technical teams and driving architectural decisions.
Preferred Qualifications
Bachelor's or Master's Degree in Computer Science, Engineering, Information Technology, or related field.
Databricks Certified Data Engineer Professional.
Azure Data Engineer Associate.
AWS Data Analytics Specialty.
Google Professional Data Engineer Certification.
TOGAF or Cloud Architecture certifications are a plus.
Nice to Have
MLOps and AI Engineering exposure.
Snowflake and Databricks integration experience.
Microsoft Fabric exposure.
Data Mesh implementation experience.
Data Governance tools such as Collibra, Alation, or Informatica.
Expertise in Data Quality frameworks such as Great Expectations or Deequ.
Key Competencies
Solution Architecture
Technical Leadership
Stakeholder Management
Program Delivery
Data Strategy & Governance
Team Mentoring & Coaching
Problem Solving & Decision Making
Communication & Executive Presentation Skills
Performance Optimization
Innovation & Continuous Improvement
Technology Classification
- Technology Tower: Data & AI
- Practice Area: Data Engineering
- Primary Skill: Databricks Lakehouse Engineering
- Secondary Skills: Apache Spark, PySpark, Cloud Data Platforms (Azure/AWS/GCP), Data Architecture, Data Governance, MLOps.
Design and implement data pipelines, Optimize data processing and storage, Ensure data solutions meet performance standards, Provide technical support, Collaborate with stakeholders.
Bachelor's/Master's in Engineering 5-8 years
