Senior Data Engineer (AI & Machine Learning)

Role overview

A large-scale AI product team is hiring a Senior Data Engineer to build the data infrastructure behind machine-learning and recommendation systems making very high volumes of automated decisions. The role owns Spark-based production pipelines and works with ML engineers, data scientists and platform engineers to improve reliability, scalability and performance of the data platform.

Responsibilities

• Design, build and optimize data pipelines for AI/ML workloads.
• Develop and tune Apache Spark/PySpark batch pipelines on Google Cloud.
• Maintain centralized data infrastructure for training, experimentation and inference.
• Provide reliable datasets to machine-learning and data-science teams.
• Troubleshoot performance and reliability issues in distributed systems.
• Evolve data architecture with platform and distributed-systems engineers.

• 5+ years in Data Engineering.
• Strong Apache Spark and PySpark DataFrame expertise.
• Distributed data-processing pipeline design and optimization experience.
• Strong Python engineering practices including tests, source control, reviews and CI/CD.
• Ability to collaborate across ML, data science and platform engineering.

Permanent role that can be worked remotely from Canada, associated with Toronto. The current GuruLink page offers a $750 referral reward and direct application.