What you’ll be doing
Design, build, and maintain scalable and reliable data pipelines for batch and near-real-time processing in Python and PySpark on Databricks and AWS.
Develop and optimize data models, ETL/ELT workflows, and data integration across multiple systems and platforms, using Delta Lake and lakehouse (medallion) patterns.
Support data integrations with Reltio and Transcend, including data loads, extracts, and reconciliation.
Build data quality, lineage, and reconciliation checks so that profile and consent data is accurate, complete, and auditable.
Implement data governance, security, and quality standards across data assets, including access controls, masking, and retention for sensitive data (PII/PHI) and consent data.
Collaborate with the Technical Lead, product owners, architects, analysts, and business stakeholders to understand data requirements and deliver solutions.
Own pipelines and work streams from design through production, and contribute to architectural and design decisions.
Promote best practices in coding, code review, automated testing, and deployment, and release code through CI/CD pipelines.
Monitor, troubleshoot, and improve performance, reliability, and cost of data pipelines, and support production incidents and root cause analysis.
Automate manual processes and identify opportunities to optimize data workflows and reduce costs.
How You Will Succeed
Deliver scalable solutions by designing robust data pipelines and architectures that meet performance and reliability standards.
Build in privacy and consent by treating consent and data protection as a design principle in pipelines from the start, and by raising risks early.
Collaborate effectively with cross-functional teams to turn business needs into technical outcomes.
Share expertise, mentoring junior engineers and adopting best practices in data engineering and cloud technologies.
Continuously improve systems through automation, performance tuning, and proactive issue resolution.
Communicate with clarity to ensure alignment across technical and non-technical stakeholders.
Basic Qualifications and Experience Requirement
Bachelor’s degree in Computer Science, Information Technology, Management Information Systems
Overall 5+ years of experience in data engineering using core technologies such as SQL, Python, PySpark, and cloud platforms (AWS, Azure, or GCP) services including Lambda, Glue, EMR S3, Redshift, Athena, and IAM roles/policies.
2+ years of experience working in Agile environments, with hands-on experience using GitHub and CI/CD pipelines for code deployment.
Hands-on experience with Databricks, including workspace configuration, notebook development, Delta Lake, and performance optimization.
1+ years of experience with orchestration tools like Airflow or Databricks Workflows for workflow automation.
Qualified applicants must be authorized to work in the United States on a full-time basis. Lilly will not provide support for or sponsor work authorization or visas for this role, including but not limited to F-1 CPT, F-1 OPT, F-1 STEM OPT, J-1, H-1B, TN, O-1, E-3, H-1B1, or L-1.
Additional Skills/Preferences
Experience building high-performance, scalable data pipelines following Data Lakehouse, Data Warehouse, and Data Mart standards and deep understanding of ETL/ELT workflows.
Strong expertise in data modelling, managing large datasets, and implementing secure, compliant data governance practices.
Working knowledge of data governance, security, and data quality practices, including handling sensitive data (PII/PHI) in a regulated environment.
Ability to collaborate with business stakeholders to translate key business requirements into scalable technical solutions.
Strong communication and collaboration skills to work with both technical and non-technical stakeholders; a problem-solving mindset and attention to detail.
Pharmaceutical or healthcare industry experience, including privacy requirements such as HIPAA.
Hands-on experience with Reltio (or a comparable MDM platform), including match and merge, survivorship, and integration patterns.
Familiarity with consent and preference management (for example opt-in/opt-out, purpose, and channel); experience with Transcend or a similar consent platform.
Familiarity with streaming frameworks such as Kafka, Kinesis, or MSK.
Experience with Unity Catalog, infrastructure-as-code (for example Terraform), and data quality frameworks (for example Great Expectations or dbt tests).
Partner with and influence vendor resources on solution development to ensure understanding of data and technical direction for solutions as well as delivery.
AWS Certified Data Engineer – Associate
Databricks Certified Data Engineer (Associate or Professional)
Lilly is dedicated to helping individuals with disabilities to actively engage in the workforce, ensuring equal opportunities when vying for positions. If you require accommodation to submit a resume for a position at Lilly, please complete the accommodation request form (https://careers.lilly.com/us/en/workplace-accommodation) for further assistance. Please note this is for individuals to request an accommodation as part of the application process and any other correspondence will not receive a response.
Lilly is proud to be an EEO Employer and does not discriminate on the basis of age, race, color, religion, gender identity, sex, gender expression, sexual orientation, genetic information, ancestry, national origin, protected veteran status, disability, or any other legally protected status.
Our employee resource groups (ERGs) offer strong support networks for their members and are open to all employees. Our current groups include: Africa, Middle East, Central Asia (AMECA), Black Employees at Lilly (BE@Lilly), Chinese Culture Network (CCN), EnAble, Evolve, Lilly Indian Network (LIN), Organization of Latinx at Lilly (OLA), Pride (LGBTQ+ Allies), Veterans Leadership Network (VLN) and Women’s Initiative for Leading at Lilly (WILL).
- Actual compensation will depend on a candidate’s education, experience, skills, and geographic location. The anticipated wage for this position is
- $66,000 - $158,400
Full-time equivalent employees also will be eligible for a company bonus (depending, in part, on company and individual performance). In addition, Lilly offers a comprehensive benefit program to eligible employees, including eligibility to participate in a company-sponsored 401(k); pension; vacation benefits; eligibility for medical, dental, vision and prescription drug benefits; flexible benefits (e.g., healthcare and/or dependent day care flexible spending accounts); life insurance and death benefits; certain time off and leave of absence benefits; and well-being benefits (e.g., employee assistance program, fitness benefits, and employee clubs and activities).Lilly reserves the right to amend, modify, or terminate its compensation and benefit programs in its sole discretion and Lilly’s compensation practices and guidelines will apply regarding the details of any promotion or transfer of Lilly employees.
#WeAreLilly
