Job Purpose
We are looking for a talented and experienced Senior Data Engineer / Backend SQL Developer to join our Data Aggregation team. In this role, you will be a core contributor to our SQL Server backend, writing complex stored procedures, views, and aggregation logic that feeds downstream reporting and analytical workloads. Alongside traditional SQL Server development, you will extend these capabilities into the modern lakehouse using Python and Databricks, enabling scalable batch and near-real-time data processing across large datasets. The ideal candidate brings deep expertise in SQL Server (T-SQL, stored procedures, query tuning, indexing strategies) combined with hands-on experience in PySpark, Apache Iceberg, and Databricks Workflows — and is comfortable bridging the gap between on-premise SQL environments and cloud-based data platforms.
Responsibilities
Design and implement robust ETL/ELT pipelines using PySpark and Python on Databricks
Build and manage Apache Iceberg table formats for efficient data storage, versioning, and time-travel queries on the data lakehouse
Develop and optimize complex SQL queries, stored procedures, and data models for analytical workloads
Automate infrastructure provisioning, deployment, and operational tasks using PowerShell scripts
Monitor and tune Databricks cluster performance, job scheduling, and cost optimization
Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver reliable data products
Show the full description
Enforce data quality, governance, and lineage best practices across the platform
Participate in code reviews, technical design sessions, and Agile ceremonies
Knowledge and Experience
6+ years of experience in data engineering or similar role
Experience working with cloud platforms (AWS / GCP)
Familiarity with version control (Git) and CI/CD practices
Strong understanding of data warehouse and data lakehouse concepts
Apache Iceberg- Experience managing Iceberg table format — schema evolution, partitioning, compaction, time-travel
Databricks- Hands-on with Databricks Workflows, Unity Catalog, Delta/Iceberg integration, cluster management
Python - Proficient in writing clean, production-grade Python for data engineering tasks
PySpark - Strong experience with distributed data processing, DataFrame API, optimizations
SQL - Advanced SQL — window functions, CTEs, query optimization, schema design
PowerShell - Scripting for automation, CI/CD pipelines, Windows/Azure environment management
Preferred Knowledge and Experience
Experience with Azure Data Factory, dbt, or Airflow for orchestration
Knowledge of SSIS and PowerBI/Tableau