Grigori Fursin
Founder and CEO of cKnowledge SAS, France
Invited Talk: Enabling Reproducible ML&Systems Research: The Good, the Bad and the Ugly
Abstract:
10 years ago we released our novel ML-based MILEPOST compiler with all the related experimental data at cTuning.org. Unfortunately, this research quickly stalled after we struggled to reproduce performance results and predictive models shared by volunteers across rapidly changing systems.
In this talk, I will describe my 10-year effort to solve numerous reproducibility issues in ML&systems research. I will share my experience reproducing 150+ systems and ML papers during artifact evaluation at ASPLOS, MLSys, CGO, PPoPP and Supercomputing. This tedious experience motivated me to develop the cKnowledge.org framework and the open CodeReef.ai portal to bring DevOps principles to our research. I will also present CodeReef solutions - a new way to package and share research artifacts and results with common Python APIs, CLI actions, portable workflows and JSON meta descriptions. Such solutions can be used to automatically build, benchmark and validate ML&system experiments across continuously evolving platforms.
I will conclude with several practical use-cases of our technology in collaboration with Arm, IBM, General Motors, the Raspberry Pi foundation and MLPerf. Our long-term goal is to help researchers share their new ML techniques as production-ready packages along with published papers and participate in collaborative and reproducible benchmarking, co-design and comparison of efficient ML/software/hardware stacks.
Biography:
Grigori Fursin is a computer scientist with 20+ years of experience pioneering novel Machine Learning and Knowledge Sharing techniques to modernize the development of efficient computer software and hardware. After completing a PhD in Computer Science from the University of Edinburgh in 2004, he was a tech lead of the joint MILEPOST project with IBM, a senior research scientist at INRIA and a co-director of the Intel Exascale Lab.
Grigori is the founder of cKnowledge.org to enable reproducible ML&systems research and portable MLOps. He is an active open-source contributor and reproducibility champion notably through his non-profit cTuning foundation and the involvement in artifact evaluation. Grigori is the author of the cKnowledge.org framework to bring DevOps principles to research with reusable automation tasks, portable workflows, reproducible experiments and auto-generated “live” papers.