This role combines deep technical ownership of enterprise Lakehouse and data platform architecture with hands-on experience building Agentic AI and Retrieval-Augmented Generation (RAG) solutions on Databricks. The ideal candidate is equally comfortable designing large-scale, production-grade data pipelines, architecting modern AI-driven solutions, and engaging directly with clients on strategic proposals and solution design.
Your key responsibilities
Architect, design, and deliver scalable Data & AI solutions on the Databricks Lakehouse platform, covering ingestion, transformation, storage, governance, and consumption layers.
Design and optimize large-scale distributed data processing pipelines using Hadoop, Apache Spark, and PySpark.
Build reusable, production-grade Python frameworks and pipelines for batch and streaming data engineering workloads.
Define and implement cloud data architecture on AWS and/or Azure, covering storage, IAM, networking, security, and monitoring, with native Databricks integration.
Own end-to-end ETL/ELT design, data ingestion, transformation, orchestration, and data quality frameworks across the platform.
Design dimensional data models (star/snowflake schemas, fact/dimension tables, SCD) to support analytical and reporting use cases.
Design and build Agentic AI solutions on Databricks, including LLM integration, tool/function calling, agent workflows, guardrails, and enterprise data integration.
Architect and implement RAG (Retrieval-Augmented Generation) solutions using Databricks — document ingestion, chunking, embeddings, vector search, retrieval, grounding, and evaluation.
Establish and enforce engineering best practices for Git-based version control, code reviews, and CI/CD across development, test, and production environments.
Lead performance optimization and production deployment of Databricks jobs, workflows, and pipelines at scale.
Participate in strategic RFP/RFI responses, technical proposals, solution shaping, effort estimation, and architecture presentations.
Act as a trusted technical advisor in customer workshops, translating business requirements into robust technical architectures.
Mentor engineering teams on Databricks, Spark, cloud architecture, and modern AI/RAG engineering practices.
Skills and attributes for success
13+ years of experience in Data Engineering / Data Architecture, with strong hands-on experience across Databricks, Hadoop, Spark/PySpark, and cloud data platforms.
Databricks
Strong expertise in Databricks Lakehouse, Delta Lake, Databricks SQL, and Unity Catalog.
Hands-on experience with Jobs/Workflows, performance optimization, and production deployments.
Hadoop & Spark
Strong knowledge of the Hadoop ecosystem, HDFS, and Hive.
Strong hands-on experience with Apache Spark and PySpark for large-scale distributed data processi
Python
Strong proficiency in Python for building scalable, reusable, production-grade data pipelines and frameworks.
Cloud
Strong experience with AWS and/or Azure, including cloud storage, IAM, networking, security, and monitoring.
Experience integrating Databricks with cloud-native services.
Data Engineering
Expertise in ETL/ELT, batch and streaming processing, data ingestion, transformation, orchestration, and data quality.
Proven ability to design scalable, production-ready pipeline architectures.
SQL & Data Modelling
Strong SQL skills, with experience in dimensional modelling, star/snowflake schemas, fact/dimension models, SCD, and analytical data models.
Agentic AI
Hands-on experience designing and building Agentic AI solutions with Databricks, including LLMs, tool/function calling, agent workflows, guardrails, and enterprise data integration.
RAG (Retrieval-Augmented Generation)
Strong experience building RAG solutions using Databricks, including document ingestion, chunking, embeddings, vector search, retrieval, grounding, and evaluation.
Git & CI/CD
Strong knowledge of Git, branching strategies, pull requests, and code reviews.
Experience with CI/CD pipelines and deployment across multiple environments.
Experience with Databricks Asset Bundles and/or Terraform is a plus.
Strategic RFPs & Solutioning
Ability to participate in strategic RFP/RFI responses, solution shaping, technical proposals, and effort estimations.
Experience delivering architecture presentations and leading customer workshops.
Client Communication
Strong client-facing and communication skills, with the ability to understand business requirements, translate them into scalable technical solutions, and clearly communicate architectural decisions to both technical and non-technical stakeholders.
Ideally, you’ll also have
Prior experience in a consulting, systems integrator, or client-facing delivery role.
Relevant cloud or Databricks certifications (e.g., Databricks Certified Data Engineer / Solutions Architect, AWS/Azure certifications).
Exposure to enterprise data governance, MLOps, or LLMOps practices.
What working at EY offers
At EY, we’re dedicated to helping our clients, from start–ups to Fortune 500 companies — and the work we do with them is as varied as they are.
You get to work with inspiring and meaningful projects. Our focus is education and coaching alongside practical experience to ensure your personal development. We value our employees and you will be able to control your own development with an individual progression plan. You will quickly grow into a responsible role with challenging and stimulating assignments. Moreover, you will be part of an interdisciplinary environment that emphasizes high quality and knowledge exchange. Plus, we offer:
Support, coaching and feedback from some of the most engaging colleagues around
Opportunities to develop new skills and progress your career
The freedom and flexibility to handle your role in a way that’s right for you
EY | Building a better working world
EY exists to build a better working world, helping to create long-term value for clients, people and society and build trust in the capital markets.
Enabled by data and technology, diverse EY teams in over 150 countries provide trust through assurance and help clients grow, transform and operate.
Working across assurance, consulting, law, strategy, tax and transactions, EY teams ask better questions to find new answers for the complex issues facing our world today.
Apply now »
Apply now
Start applying with LinkedIn
Apply Now
Start
Please wait...
Find similar jobs
- Consulting DE,
- Consulting CN,
- Consulting CA,
- Consulting BR,
- Consulting
Connect with us
Our locations
My EY
Site map
Legal and privacy
Labor condition applications
Cookie Consent Manager
Opens in a new tab.
Opens in a new tab.
Opens in a new tab.
Opens in a new tab.
Opens in a new tab.
EY refers to the global organization, and may refer to one or more, of the member firms of Ernst & Young Global Limited, each of which is a separate legal entity. Ernst & Young Global Limited, a UK company limited by guarantee, does not provide services to clients.
×
Cookie Consent Manager
When you visit any website, it may store or retrieve information on your browser, mostly in the form of cookies. Because we respect your right to privacy, you can choose not to allow some types of cookies. However, blocking some types of cookies may impact your experience of the site and the services we are able to offer.
Required Cookies
These cookies are required to use this website and can't be turned off.
Required Cookies
Show More Details
Required Cookies
- Provider
- Description
- Enabled
SAP as service provider
We use the following session cookies, which are all required to enable the website to function:
"route" is used for session stickiness
"careerSiteCompanyId" is used to send the request to the correct data center
"JSESSIONID" is placed on the visitor's device during the session so the server can identify the visitor
"Load balancer cookie" (actual cookie name may vary) prevents a visitor from bouncing from one instance to another
Cookies from provider SAPasserviceprovider are required and cannot be turned off
Advertising Cookies
These cookies serve ads that are relevant to your interests. You may freely choose to accept or decline these cookies at any time. Note that certain functionality that these third parties make available may be impacted if you do not accept these cookies.
Consent to all Advertising Cookies
Show More Details
Advertising Cookies
- Provider
- Description
- Enabled
AddThis
Google Analytics is a web analytics service offered by Google that tracks and reports website traffic.
Cookie Information
Privacy Policy
Terms and Conditions
Consent to cookies from provider AddThis
LinkedIn
LinkedIn is an employment-oriented social networking service. We use the Apply with LinkedIn feature to allow you to apply for jobs using your LinkedIn profile. Opting out of LinkedIn cookies will disable your ability to use Apply with LinkedIn.
Cookie Policy
Cookie Table
Privacy Policy
Terms and Conditions
Consent to cookies from provider LinkedIn
Google Analytics
Google Analytics is a web analytics service offered by Google that tracks and reports website traffic.
Cookie Information
Privacy Policy
Terms and Conditions
Consent to cookies from provider GoogleAnalytics
Google Tag Manager
Google Tag Manager is a tag management system for conversion tracking, site analytics, remarketing, and more.
Privacy Policy
Terms and Conditions
Consent to cookies from provider GoogleTagManager
Confirm My Choices
- Reject All Cookies
- Accept All Cookies