Data Governance: Implement data governance practices, including data security, privacy, and compliance, to ensure data integrity and regulatory compliance.
Performance Optimization: Optimize data pipelines and queries for improved performance and scalability in Azure and Databricks environments.
Monitoring and Troubleshooting: Monitor data pipelines, identify and resolve performance issues, and troubleshoot data-related problems in collaboration with other teams.
Data Visualization: Build BI reports to enable faster decision making.
Collaboration: Work with product managers to ensure superior product delivery to drive business value & transformation
Documentation: Document data engineering processes, data flows, and system configurations for future reference and knowledge sharing.
Job Qualifications
Experience: Bachelor's or master's degree in computer science, data engineering, or a related field, along with 4+ year work experience in data engineering and cloud platforms.
Azure and Databricks: Strong proficiency in Azure services such as Azure Data Factory, Azure Databricks, Azure SQL Database, Azure Data Lake Storage, and Azure Synapse Analytics.
ETL Tools: Experience with ETL (Extract, Transform, Load) tools and frameworks, such as Apache Spark, Databricks Delta, or Azure Data Factory, for data integration and transformation.
Programming: Proficiency in programming languages such as PySpark, Python, SQL, or Scala for data manipulation, scripting, and automation.
Data Modeling: Knowledge of data modeling techniques and experience with data modeling tools.
Database Technologies: Familiarity with relational databases (e.g., SQL Server) for data storage and retrieval.
Data Warehousing: Understanding of data warehousing concepts, dimensional modeling, and experience with data warehousing technologies such as Azure Synapse Analytics or Azure SQL Data Warehouse or Azure Databricks DeltaLake
Data Governance: Knowledge of data governance principles, data security, privacy regulations, and experience implementing data governance practices.
Data Visualization: Experience of working with Microsoft Power BI to build semantic data model & BI reports/dashboards.
Cloud Computing: Familiarity with cloud computing concepts and experience working with cloud platforms, particularly Microsoft Azure.
Problem-Solving: Strong analytical and problem-solving skills to identify and resolve data-related issues.
Proficiency in DevOps Tools and CICD tools (e.g. Azure DevOps, Chef, Puppet, Github)
- Job Segmentation
- Experienced Professionals