Must-Have Skills
Programming and Development
Java – Expert -level proficiency in building production grade ETL framework, customers actions, and reusable libraries. Must-Have have experience with Maven multi-model projects, dependency management with shaded JARs, and handling Hadoop class path conflicts.
Groovy – Strong scripting skills for configuration DSLs, row processing, and partition logic in declarative ETL framework.
Git and version controls - Proficient with branching strategies, pull request workflows, code reviews and Jira -integrated commit discipline.
Scala – Functional programming experience for spark development with ability to write concise idiomatic Scala for data processing at scale.
SQL – Ability to write complex SQL queries across multiple databases (MySQL, PG, Oracle, SQL server etc.)
Data Engineering
ETL Pipeline Development - Proven track record building production ETL framework with reusable components. Deep understanding of ETL concepts, load patterns, CDC processing, and data lineage.
Apache Spark – Deep expertise with spark on production scale datasets. Must-Have have hands on experience optimizing spark jobs, tuning memory/shuffle and working with DataFrame/RDDs in Scala or Java.
Kafka/ Golden Gate streaming – Real/Near real-time data processing with Kafka, Golden gate or spark streaming.
Airflow – Experience migrating from Oozie to modern workflow orchestration
Databricks – Exposure to unified analytics platforms, Delta Lake and notebook driven development.
Big Data and Distributed systems
Hadoop Ecosystem - Strong production scale experience with HDFS operations, Hive query optimization and managing portion tables at scale. Understanding of Data locality, replication factors and cluster resource management.
Hive/HQL - Experience in complex analytics queries on multi – million row tables.
Oozie – Proficient in designing coordinator/workflow orchestration with dataset dependency, SLA monitoring and handling late – data scenarios.
Sqoop – Hands-on on experience Configuring bulk imports with parallel rappers, split by strategies, incremental modes and handling large scale database exports
Cloud and Infrastructure
AWS – Production experience with S3 data lakes (lifecycle policies, versioning, encryption), EMR cluster operations, IAM role-based access, and Secrets Manager for credentials management. Understanding of AWS networking, VPC and security groups.
Data Security and compliance - Hands-on on experience implementing PII encryption/ decryption, SafeNet, Data masking, and working on regulated environment.
Good-to-Have Skills
Banking, lending or other regulated financial services experience
Data modeling
Observability and monitoring frameworks for data pipelines
Databricks Certified Data Engineer Professional
Presto/Trino
NoSQL databases
Soft Skills
Strong problem solving and communication skills with ability to explain data engineering concepts to non-technical stakeholders.
Experience working on client facing roles.
Location and Work Arrangements
San Francisco Bay Area
This position requires onshore presence in our San Francisco office.
Manage data engineering projects. Provide strategic guidance. Oversee data engineering team. Ensure continuous improvement of data processes.
Graduate in Computer Science, Data Science, or related field. 3-4 years of experience in data engineering or a related field.