Perform advanced debugging and troubleshooting across distributed Spark workloads (data skew, shuffle tuning, memory/compute optimization).
Implement engineering best practices: modular design, efficient coding, code reviews, and CI-friendly development approaches.
Use GitHub/Bitbucket and standard version control workflows to manage codebase, peer reviews, and releases.
Partner with cross-functional stakeholders to convert requirements into robust big data solutions.
Uses enterprise-authorized AI capabilities within the work environment to accelerate data pipeline/design analysis and documentation, validating outputs and handling data according to sensitivity and security requirements.
Applies reuse-first, AI-assisted practices to strengthen SDLC-quality routines for data pipelines (e.g., test generation and control validation), ensuring traceability/auditability and alignment to resiliency and security expectations.
Required qualifications, capabilities, and skills
Formal training or certification on data engineering concepts and 3+ years applied experience
Experience in data engineering / big data engineering, with strong hands-on delivery and Expert-level SQL skills (must be extremely strong): complex joins, window functions, CTEs, optimization, and analytical problem solving at scale.
Strong hands-on coding experience with Python in production environments; demonstrated ability to write efficient, maintainable code.
Deep expertise in Apache Spark (in depth) and PySpark, including performance tuning and distributed processing fundamentals.
Strong experience with Databricks for large-scale data processing and pipeline development.
Strong understanding of data warehousing concepts and best practices; proven capability in data modelling and solution design for scalable, maintainable data platforms/products.
Experience implementing both batch and streaming data processing solutions.
Familiarity with AWS S3 and common AWS services used in data platforms and processing
Excellent debugging, troubleshooting, problem-solving skills and experience with GitHub, Bitbucket, and version control best practices.
Demonstrated experience using enterprise-authorized AI capabilities within the work environment to support data engineering workflows with strong validation habits and awareness of data sensitivity.
Ability to review and validate AI-assisted outputs (e.g., query suggestions, test ideas, or model change summaries) before use, escalating when uncertain and following data handling requirements.
Preferred qualifications, capabilities, and skills
Good to have: infrastructure provisioning in AWS using Infrastructure as Code (IaC) (e.g., Terraform, AWS CloudFormation).
