This includes creating robust data infrastructure for our MCP, ensuring data quality and availability for personalized learning paths and skill validation.
Data Quality Assurance: Continuously work towards detecting, alerting, and fixing quality issues to build trust in our data
System Enhancement: Improve existing systems and services using latest tooling and engineering standards
Cost Optimization: Improve unit economics across ingestion, storage, and processing components
Technical Leadership & Ownership
Complete Development Lifecycle: Own the entire development process from ideation to deployment
Cross-functional Collaboration: Work closely with different teams to achieve common goals through efficient coordination
Technical Documentation: Maintain up-to-date documentation with architecture diagrams for entire projects
Service Ownership: Take complete ownership of services within your workstream and drive continuous improvement
Operational Excellence: Establish and maintain robust monitoring, alerting, and operational procedures for mission-critical data systems.
Team Leadership & Growth
Knowledge Sharing: Write blogs and articles to help team members and the community
Architecture Breakdown: Break down high-level architecture into manageable services and tasks
Technology Trends: Learn and share new trends in technology, engineering, and productivity
Team Productivity: Actively work towards improving team productivity and knowledge
Mentorship: Guide and mentor junior team members
Business Impact
Data Governance: Govern data assets across the entire company and audit data posture
Time to Insight: Reduce time to insight for data team stakeholders
Business Decision Support: Assist and challenge business decisions using data insights
Stakeholder Service: Make data available to all stakeholders in an efficient, quick, quarriable, and intuitive manner
Skills and attributes for success
Experience: 8+ years of progressive professional experience in data engineering, with a proven track record of designing, building, and operating large-scale, high-throughput data platforms, ideally in a consumer-facing or streaming industry.
Extreme Scale Experience: Demonstrated experience working with data systems that handle hundreds of millions of events/requests per minute, billions of daily events, and petabytes of storage (e.g., S3, HDFS).
Programming: Strong proficiency in at least one major programming language such as Python, Java, Scala, or Go, with a focus on building robust, production-grade data applications.
Distributed Data Processing: Deep expertise and practical experience with distributed data processing frameworks like Apache Spark, Apache Flink, or equivalent technologies for batch and streaming workloads.
AI/ML Data Experience: Specific experience in building data pipelines for Machine Learning model training, inference, and feature stores. Familiarity with data needs for AI Agents or similar intelligent systems.
Messaging & Streaming: Extensive experience with high-throughput messaging queues and stream processing platforms like Apache Kafka, Kinesis, or equivalent.
Cloud Platforms: Hands-on expertise with at least one major cloud platform (AWS, GCP, or Azure) and their data-related services, with a strong preference for AWS (S3, EMR, Kinesis, Glue, Redshift, etc.).
Data Warehousing/Lakes: Strong understanding and hands-on experience with modern data warehousing concepts and data lake architectures (e.g., Snowflake, BigQuery, Redshift, Delta Lake, Iceberg, Hudi).
ETL/Orchestration: Expertise in building, optimizing, and orchestrating complex ETL/ELT pipelines using tools like Apache Airflow, Prefect, or other workflow management systems.
Databases: Strong SQL skills and experience with both relational (e.g., PostgreSQL, MySQL) and NoSQL databases (e.g., Cassandra, DynamoDB).
Data Modeling: Advanced data modeling skills (dimensional, Kimball, data vault) for large, complex datasets.
Problem-Solving: Exceptional analytical and problem-solving abilities, with a proven track record of debugging and optimizing complex distributed systems.
Communication: Excellent communication, collaboration, and interpersonal skills, with the ability to articulate complex technical concepts to diverse audiences.
Mentorship: Proven ability to mentor, guide, and technically lead senior and junior engineers.
