Support downstream analytics workflows, including advanced analytics, data science, visualization, and industrial process analytics.
Data Ingestion & Integration
Design, build, and maintain robust pipelines to ingest data from relational databases, operational systems, industrial historians, APIs, cloud platforms, and flat files.
Establish reliable and secure connections between source systems, cloud data platforms, and industrial data platforms.
Write optimized SQL queries and extraction logic for high-performance data retrieval.
Implement flexible ingestion patterns including batch, incremental, streaming, and replication-based approaches based on business latency requirements.
Integrate high-volume time-series data and industrial metadata while maintaining relationships between assets, measurements, events, and other operational entities.
Support ingestion and integration patterns used by platforms such as Cognite Data Fusion (CDF), Seeq, and Microsoft Fabric.
Data Storage & Lakehouse Engineering
Design and manage scalable storage structures across raw (bronze), intermediate (silver), and curated (gold) layers.
Leverage Spark-native formats and engines such as Delta Lake to support scalable analytics workloads.
Design curated data structures that support both enterprise analytics and industrial data consumption.
Enforce data retention, archiving, and purging policies to manage data lifecycles.
Monitor and optimize storage, compute costs, and cloud resource utilization.
Data Transformation
Develop Python and SQL-based transformations to clean, standardize, enrich, and unify data.
Develop, tune, and proactively optimize scalable PySpark applications to efficiently process high-volume datasets.
Build modular, reusable transformation components rather than siloed, application-specific logic.
Resolve complex data-quality anomalies including schema drift, missing values, duplicates, timestamp alignment, and inconsistent identifiers.
Engineer efficient incremental processing patterns to handle high-volume, frequently changing datasets.
Transform raw operational and historian data into analytics-ready time series, calculations, conditions, events, and contextualized datasets for downstream consumption.
Enable reusable data transformations that can support platforms such as Seeq, Cognite, Fabric, and custom analytics applications.
Semantic Modeling & Industrial Data Contextualization
Translate complex source schemas into intuitive, business-aligned semantic entities, asset hierarchies, and industrial data models.
Build and maintain relationships between assets, equipment, time series, events, files, and other operational data.
Standardize naming conventions, identifiers, metadata, and relationship mappings across source systems.
Perform data contextualization to connect operational data with relevant equipment, process, and business context.
Support asset-centric and knowledge-graph-style data models commonly used in industrial data platforms such as Cognite.
Pipeline Orchestration & Operations
Architect end-to-end workflows that seamlessly coordinate ingestion, transformation, contextualization, validation, and publication.
Orchestrate data movement between source systems, lakehouse environments, industrial data platforms, and analytics applications.
Implement resilient error handling including retries, automated recovery, checkpointing, and alerting.
Monitor pipeline health, SLA compliance, data freshness, and data drift using centralized observability tools.
Design pipelines capable of supporting both traditional enterprise data and high-frequency industrial time-series workloads.
Data Governance, Security & DevOps
Enforce data security and privacy standards, including row- and column-level security, RBAC (Role-Based Access Control), and masking of sensitive data.
Apply consistent access-control and governance patterns across enterprise data platforms and industrial analytics environments.
Embed automated validation checks as needed to enforce data quality.
Maintain Git-based development workflows using branching, code reviews, and pull requests.
Implement automated CI/CD pipelines for code testing, data validation, configuration management, and repeatable environment deployments.
Support deployment and lifecycle management of data pipelines, transformation logic, semantic models, and analytics configurations across development, test, and production environments.
Maintain comprehensive documentation of data architectures, pipeline schemas, source-to-target mappings, contextual relationships, and lineage maps.
Preferred Experience
Bachelor’s Degree in Computer Science, Software Engineering, Data Science, Engineering, or related field + 2 years of relevant experience; equivalent Master’s degree accepted; non-relevant bachelor’s degree requires 5+ years of equivalent hands-on experience.
Experience working with
SQL and Python for data engineering and transformation.
PySpark and Microsoft Fabric for lakehouse transformations.