We are seeking Data Engineers for a contract position based in Malvern, Pennsylvania. This is a 6+ month engagement focused on designing, developing, and maintaining high-volume data transformation logic and batch processes.
Responsibilities
Design, develop, and maintain high-volume data transformation logic primarily using AWS Glue jobs written in Python
Develop custom code and potentially AWS Lambda functions for handling complex logic within batch processes
Utilize PySpark and SQL for data querying, filtering, and manipulation against various data stores, including modernized data sources and legacy DB2 tables
Collaborate with internal mainframe experts to understand legacy system logic and implement requirements for modernized batch processes
Handle data ingestion from multiple sources, including various vendors, flat files, CSVs, and APIs
Engage with the build and deployment pipeline, demonstrating a strong understanding of DevOps concepts and proficiency with Git and GitHub
Work closely with a dedicated Tech Lead while operating with a high degree of independence
Qualifications
Proficiency in Python for data transformation and ETL development
Experience with AWS Glue and related AWS data services
Strong knowledge of PySpark and SQL for data manipulation and querying
Experience with Git and GitHub for version control
Understanding of DevOps concepts and CI/CD practices
Familiarity with data ingestion from multiple source types and formats
Show the full description
Ability to work independently and collaboratively within a team structure