As a Data Engineer, you will play a pivotal role in constructing the data foundation for our AI and quant model initiatives. Your responsibilities will encompass managing diverse data pipelines, from market and economic data to internal and news feeds. Additionally, you will be tasked with creating and maintaining a robust data architecture, ensuring data quality, and developing metadata systems. Your work will directly impact the performance and reliability of our client tools and quant models.
Build and maintain market, economic, research, news, and internal data pipelines, as well as Fenics/BGC data integration.
Design and implement a scalable bronze/silver/gold data architecture to support our AI and quant models.
Establish and monitor data quality gates to ensure data integrity and accuracy.
Develop and manage instrument, ticker, country, client, source, and theme metadata for effective data categorization and retrieval.
Create APIs and tables for seamless integration with agent tools, dashboards, and quant models.
Implement data lineage tracking, user entitlements, and permissions management systems.
Monitor and optimize data pipelines for performance and efficiency, ensuring smooth data flow.
Collaborate with data scientists and analysts to understand their data needs and provide model-ready datasets.
Stay updated with the latest trends in time-series and complex-data analysis, incorporating best practices into our data platform.
Show the full description
Ensure data security and privacy compliance, adhering to industry standards and regulations.
Strong proficiency in Python and SQL, with experience in production-level ETL/ELT processes.
In-depth knowledge of cloud data stack technologies, such as Azure, Databricks, Delta Lake, or Snowflake.
Experience with time-series data and complex data structures, demonstrating an understanding of data quality and monitoring.
Ability to clean and transform messy data into structured, model-ready datasets, ensuring data integrity.
Familiarity with ticker-level data processing, having worked with kdb+ or similar systems is an advantage.
Excellent problem-solving and analytical skills, with a keen eye for detail and data accuracy.
Strong communication and collaboration skills, able to work effectively with cross-functional teams.
A proactive and self-motivated approach to work, with a passion for data engineering and continuous improvement.
A bachelor's degree in Computer Science, Engineering, or a related field is preferred.
2+ years of relevant work experience in data engineering or a similar role.
