Description
Remote
Our client seeks a Data Engineer to design and implement high-performance data pipelines at scale. The role will focus on building reliable ingestion and transformation processes, enforcing data quality and lineage, and delivering resilient solutions across AWS services.
Due to client requirements, applicants must be willing and able to work on a w2 basis. For our w2 consultants, we offer a great benefits package that includes Medical, Dental, and Vision benefits, 401k with company matching, and life insurance.
Rate: $50.00 to $57.00/hr. w2
Responsibilities
Design and implement high-performance data pipelines that handle large volumes of data.
Ensure data integrity through advanced quality checks and lineage tracking.
Experience Requirements
Expert-level proficiency in Python and PySpark.
Deep hands-on experience with AWS Glue and Amazon EMR.
Proven track record building scalable pipelines for large-scale data, preferably at terabyte scale.
Experience implementing frameworks for data lineage and data quality.
Strong experience with S3-based data lakes and relational databases such as RDS, PostgreSQL, or MySQL.
Experience in Financial Services is a plus.
Experience with real-time processing using Kinesis or Kafka.
Familiarity with CI/CD for data pipelines and Infrastructure as Code.