We are seeking a Data Engineer for a contract position (6+ months) based in Malvern, Pennsylvania. In this role, you will design, build, and optimize enterprise-scale data pipelines using Databricks and related cloud technologies. You will orchestrate data extraction from diverse sources, migrate workloads to modern platforms, and provide production support to ensure system reliability and data integrity across financial data ecosystems.
Responsibilities
Design and implement high-performance, scalable ETL and ELT pipelines using Databricks, Apache Spark, and AWS data services
Orchestrate data extraction and transformation from legacy and modern data sources
Migrate data processing jobs from EMR to Databricks
Develop and maintain CI/CD pipelines for data infrastructure
Collaborate with business analysts and stakeholders to ensure data availability, quality, and integrity
Provide expert-level production support and troubleshooting for data platforms
Create documentation and job aids to support team operations and training
Implement dimensional data modeling and data lake architectures (Medallion Architecture)
Qualifications
Required
Bachelor's degree in Computer Science, Engineering, Information Systems, or related discipline
8+ years of enterprise Data Engineering experience
Strong hands-on Python development experience
Advanced SQL programming skills
Extensive experience with Apache Spark and PySpark
Show the full description
Strong hands-on experience with Databricks
Hands-on experience with AWS data services: S3, AWS Glue, Athena, Lambda, Step Functions, and EventBridge
Experience building enterprise ETL and ELT pipelines
Strong understanding of distributed data processing
Experience working with large-scale cloud data platforms
Strong knowledge of dimensional data modeling
Experience implementing CI/CD pipelines
Experience with Git and GitHub Actions
Familiarity with Agile/Scrum methodologies
Excellent communication and stakeholder management skills
Preferred
Financial Services or Asset Management industry experience
Knowledge of Apache Iceberg, Delta Lake, or Hudi
Experience with Apache Airflow
Knowledge of Kafka or Kinesis streaming
Experience supporting AI/ML and advanced analytics platforms
AWS Professional or Specialty Certifications