Work with AWS data engineering services for data ingestion, transformation, orchestration, storage, cataloging, and analytics.
Develop ETL/ELT workflows integrating structured, semi-structured, and unstructured data from multiple sources.
Implement data transformation, data quality, validation, and reconciliation frameworks.
Optimize Spark jobs, SQL queries, cluster configurations, and distributed processing workloads.
Implement workflow orchestration using Airflow, AWS Step Functions, Databricks Workflows, or equivalent services.
Work with streaming technologies and services for real-time data ingestion and processing.
Develop reusable frameworks, automation scripts, and standardized data engineering components.
Implement security, access control, encryption, monitoring, logging, and governance across data platforms.
Collaborate with architects, analysts, data scientists, and business stakeholders to translate requirements into scalable technical solutions.
Participate in architecture discussions, technical design, code reviews, and performance optimization initiatives.
Required Experience & Qualifications
5+ years of experience in Data Engineering.
Strong hands-on experience with AWS data engineering services.
Very good practical experience with Databricks and Apache Spark.
Strong programming skills in Python, PySpark, and SQL.
Practical exposure to at least one of GCP or Azure, with familiarity with the data engineering ecosystem of both.
Experience designing and implementing end-to-end data pipelines.
Understanding of modern data lakehouse architecture and distributed data processing.
Experience working with large-scale datasets and performance optimization.
Good to Have
Experience working on multi-cloud data engineering architectures.
Exposure to AWS, Azure, and GCP migrations.
Experience with Apache Kafka, Amazon MSK, or other streaming platforms.
Knowledge of Apache Iceberg, Delta Lake, and open table formats.
Exposure to data governance, lineage, and metadata management.
Knowledge of DevOps practices and CI/CD automation.
Experience with Infrastructure as Code (Terraform / CloudFormation).
Exposure to Generative AI data pipelines and vector databases.
Ideal Candidate Profile
The ideal candidate should be a hands-on Data Engineer who can independently design, develop, troubleshoot, and optimize complex data engineering pipelines.
We are specifically looking for someone who
Has strong practical expertise in AWS data engineering services.
Has very good hands-on experience with Databricks and PySpark.
Understands data engineering ecosystems across AWS, GCP, and Azure.
Has strong fundamentals in distributed data processing, SQL, Python, and data architecture.
Can work independently on end-to-end data engineering implementations.
Can contribute to technical architecture and engineering best practices.
