Michele Simoncelli

PSA for machine learning force field authors who already have or are planning to submit models to Matbench Discovery. There's a new dynamic scatter plot on the landing page (matbench-discovery.materials…) that allows comparing all 30 existing models against each other across 2 dozen metrics, hyper-parameters and model design choices. We hope that as new models and metrics get added to the leaderboard, this will surface empirical insights about which aspects of a model's design space positively or negatively impact certain metrics. For users of these ML force fields, we hope the dynamic nature of this plot will allow constructing exactly the Pareto front of metrics they care about to make a more informed decision on which potential best suits their research needs. To make this plot more informative, we encourage future model submission to be as detailed as possible in reporting the hyper-parameters and training procedure chosen for a given model. Existing submissions are welcome to backfill any data that wasn't originally recorded but could be helpful in this plot! There's an existing model schema on how to report e.g. graph construction cutoffs, max neighbor limits, learnings rates, layer counts, number of trainable parameters, etc. which will require extension to capture less common hyper-parameters as well as pre-training and fine-tuning stages in a standardized way. Happy to collaborate with everyone on that! The next step for this dynamic scatter plot is to report each model's inference time and memory usage at different system sizes. This will become the default values shown on the X axis as they establish a continuously updated cost-accuracy Pareto front for any metric on Matbench Discovery.

@lonepair better MLIP metrics, as promised :) much work left to do but putting this out early to solicit feedback from all stakeholders linkedin.com/feed/update/urn…