You will join a data team of about ten people who are developing a data & AI product to help malt plant operators monitor and optimize their production.
Already deployed on a dozen of the group’s plants, the application is intended to be generalized to all sites by 2026. It is developed in Python/PySpark on Databricks, with data ingested every 30 minutes via API, files or SQL databases, then exposed via DeltaLake or PostgreSQL.
Future priorities are reliability, performance optimization and cost reduction. You will work alongside 3 data engineers and a product owner, in close collaboration with the BI and AI teams to provide them with quality data, quickly and efficiently.
As a Lead Data Engineer, you will join the Data teams to:
- Evolve data architectures to be robust and scalable.
- Model data from industrial processes and IoT sensors
- Develop and optimize data pipelines using the mentioned technologies
. - Ensure data quality and integrity through validation and
cleaning processes. - Collaborate with data science and analytics teams to understand their
needs and deliver tailored solutions. - Carry out a technological watch to propose improvements and innovations.
You contribute to agile dynamics by participating in the improvement of practices and by
sharing via BBLs (moments of informal sharing), meetups or workshops.
You participate in the construction of a software product with multi-disciplinary teams (UX, UI,
OPS, PO, Developer) aligned on a common goal.