Designing and optimizing applications becomes increasingly tedious, time consuming, ad-hoc and error prone due to ever changing and complex hardware and software stack. At the same time, it becomes difficult or even impossible to validate, reproduce and extend many proposed optimization and auto-tuning techniques from numerous publications. One of the main reasons is a lack of common and practical way to preserve, systematize and reuse available knowledge and artifacts including developments, optimizations and experimental data.

In this talk, I will present modular, extensible, python-based Collective Mind framework and web-based schema-free repository (c-mind.org) that I developed at first to systematize my own research and experimentation on machine learning based program optimization and compiler design. This infrastructure can be customized to preserve, describe, share and reproduce whole experimental setups including benchmarks, data sets, libraries, tools, predictive models and optimization information with related JSON-based meta-data. I will also present and discuss positive and negative feedback during several recent academic and industrial usages of this framework to systematize benchmarking and program optimization, and to initiate new publication model where experimental results and related research artifacts are shared, reproduced and validated by the community. In a long term, I hope that such approach and collective knowledge will eventually help us to squeeze maximum performance from computer systems while minimizing energy, development time and other costs.