Senior Data Engineering owns the modelling standards and the parity proof that a dbt mart matches the legacy procedure it replaces. You own the path those models sit on; the pipeline that feeds them and the serving layer that consumes them and you build marts to that standard rather than to your own. The seam is real and it is worked together: you bring the source contract, the schedule and the serving behaviour; they bring the layer rules and the value comparison.
You will work in an AI-assisted engineering workflow, where agentic tooling carries the mechanical parts and your judgement stays on the design: what a source's contract is, where a pipeline should fail loudly, and what a dashboard can afford to query.
What you will be doing
Onboard sources end to end: authentication, pagination, rate limits, incremental watermarks, landing, bronze, silver with quality gates, warehouse sync as one DAG family per data source, not a bespoke script per table.
Own orchestration: DAGs triggered on data arrival, idempotent re-runs, backfills that do not double-count, pod sizing that fits the work, and alerts that page a human only when a human is needed.
Build the last mile: the marts and the semantic-layer definitions on top of them, so a metric in Zenlytic or a chart in Superset resolves to a column that means what its name says.
Keep the lakehouse healthy: compaction, Z-ordering chosen from real query patterns, vacuum only inside retention, and schema evolution that does not break a consumer.
Make performance a measured number: before and after on the pipeline you tuned, capacity spend on the warehouse, caching or pre-aggregation where a dashboard is paying for the same scan every morning.
Set and defend freshness and completeness expectations per source; when one breaks, diagnose to root cause and write it down.
Run the platform underneath: AKS workloads, Helm or Kustomize, Terraform, image builds and CI/CD, Key Vault and managed identity, least-privilege access to every store you touch.
Review pull requests on pipelines and models, and keep runbooks and standards current enough that the next engineer does not need to ask you.
Work with the Analytics Engineers and Product Analysts who own the business definitions your marts serve, so a definition change and a pipeline change land in the right order.
What we need from you
5+ years in data engineering, with production ownership of pipelines other people depended on.
Expert Python and SQL: you can read a query plan and say why it is slow.
A dataframe engine at volume: Polars, duckDB, Spark and the judgement to know when not to pull the data into memory at all.
Airflow in production (or Dagster/Prefect and the willingness to move); you have debugged a scheduler, not only written DAGs.
Containers and Kubernetes: an image, a pod spec, a resource limit, and what happens when the limit is wrong.
Lakehouse table formats: Delta or Iceberg with partitioning, compaction and schema evolution treated as operational concerns, not theory.
Dimensional modelling judgement and hands-on dbt: you can state a grain, choose a key, and place logic in the right layer.
One cloud in depth: Azure, AWS or GCP including storage, identity, secrets, and the networking that makes a private endpoint actually work.
Infrastructure as code: Terraform, and Helm or Kustomize. You have shipped a change through a pipeline rather than by hand.
Observability habits: metrics, logs, alerts, and an on-call triage you can describe step by step.
API integration craft, pagination, throttling, retry with backoff, incremental state and the difference between a source that is down and a source that changed shape.
Working fluency with a BI or semantic layer, enough to know why a model that is correct in SQL can still be unusable in a dashboard.
Clear writing. The incident report and the runbook are part of the job, not paperwork around it.
It would be great if you also have
Azure Database for PostgreSQL Flexible Server including Citus / Elastic Cluster in your background.
dbt at scale: Elementary, exposures, column-level lineage, CI-enforced builds.
Cost engineering: capacity units, warehouse sizing, and a saving you can attribute to a change you made.
A platform migration you helped carry warehouse engine, orchestrator, or both.
Superset, Zenlytic, Power BI, or another tool where you owned the semantic model rather than only the chart.
CDC or streaming: Fabric mirroring, Debezium, event-driven ingestion.
Standards or tooling you authored that other engineers actually follow.
OPSWAT is an equal opportunity employer. We celebrate diversity and are committed to providing an environment where equal employment opportunities are extended to all employees and applicants, free of discrimination and harassment of any type. All employment decisions are based on individual qualifications, job requirements, and business needs without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other category protected by federal, state, or local laws.
Recruiting Agencies: we do not accept unsolicited resumes from third party agencies for any of our open positions. To submit resumes for our jobs, there must be a recruiting contract approved by our legal team and endorsed by both parties. We are currently not accepting additional 3rd party agencies at this time.
