Data Modeling & Architecture
Collaborate with Data Architects to design dimensional (Star/Snowflake), normalized, and Data Vault schemas optimized for analytical workloads.
Ensure efficient data storage, partitioning strategies, and file format selection (Parquet, ORC, Avro).
Orchestration, CI/CD & DevOps
Schedule and orchestrate complex workflow DAGs using workflow orchestrators such as Apache Airflow, Prefect, or Dagster.
Implement CI/CD pipelines (Git, Jenkins, GitLab CI, or GitHub Actions) for automated testing, packaging, and deployment of data pipelines.
Write unit tests, integration tests, and data validation routines (e.g., Great Expectations, PyTest).
Governance, Security & Quality
Implement enterprise data governance, lineage tracking, metadata management, and data quality frameworks.
Ensure strict compliance with data security, masking, tokenization, RBAC, and regulatory privacy standards (e.g., GDPR, CCPA).
Required Technical Qualifications & Skills
Core Technologies - Apache Spark, PySpark (Spark SQL, DataFrames, Datasets, Structured Streaming), Python 3.x
Database & Querying - Advanced SQL, Query optimization, Indexing, Window functions
Data Lake & Lakehouse - Delta Lake, Apache Iceberg, Apache Hudi, Parquet, Avro, ORC
Workflow & Orchestration - Apache Airflow, Luigi, Prefect, or Databricks Workflows
Cloud Platforms - AWS (EMR, Glue, S3, Redshift), Azure (Synapse, ADLS, HDInsight), or GCP (Dataproc, BigQuery, GCS)
Streaming Technologies - Apache Kafka, Spark Structured Streaming, Event Hubs, or Kinesis
Key Competencies & Experience
Distributed Computing Fundamentals: In-depth knowledge of Spark internals (RDD, Catalyst Optimizer, DAG execution, Shuffling, Narrow vs. Wide dependencies).
Production Experience: Minimum 4+ years of hands-on experience building and deploying production-grade PySpark applications at enterprise scale.
Problem Solving: Proven track record in debugging memory bottlenecks, Out-Of-Memory (OOM) errors, and resolving data skew issues.
Data Engineering Best Practices: Experience implementing modular code architectures, logging, monitoring (Prometheus, Grafana, CloudWatch), and alerting.
5. Preferred Qualifications (Nice-to-Have)
Hands-on experience with Databricks platform (Delta Lake, Unity Catalog, Photon engine, MLflow).
Familiarity with Scala/Java in addition to Python.
Knowledge of DataOps and Infrastructure as Code (Terraform, CloudFormation).
Experience with modern analytical data warehouses (Snowflake, Google BigQuery, Amazon Redshift).
Relevant certifications: Databricks Certified Data Engineer Associate / Professional AWS Certified Data Engineer / Azure Data Engineer Associate (DP-203)CCA Spark and Hadoop Developer (or equivalent)
6. Educational Requirements
Bachelor’s or Master’s degree in Computer Science, Software Engineering, Information Technology, Data Science, or a related quantitative field (or equivalent practical experience).
Primary : Py Spark, Python
Secondary : Bigdata, Abinitio, Unix, Oracle
Job Family Group
Technology
Job Family
Applications Development
Time Type
Full time
- Most Relevant Skills
- Please see the requirements listed above.
- Other Relevant Skills
- For complementary skills, please see above and/or contact the recruiter.
Citi is an equal opportunity employer, and qualified candidates will receive consideration without regard to their race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran, or any other characteristic protected by law.
If you are a person with a disability and need a reasonable accommodation to use our search tools and/or apply for a career opportunity review Accessibility at Citi.
View Citi’s EEO Policy Statement and the Know Your Rights poster.