DVC on HPC with CML and large(r) number of experiments

Hi, we are currently testing and configuring a new HPC setup and I will be working on setting up a best practice guide and template projects for running ML/Data science project using DVC, CML and possibly other tools.

Based on the examples I read, I am wondering if the CML workflow also supports large numbers of parameter grid searches?

Thank you!

The DVC/CML documentation does mention something like this, but not how. Link: CI/CD for Machine Learning | Data Version Control · DVC Bottom of the page.

Not sure I follow, but this doesn’t seem particularly CML-related. The bottom of that article (regarding grid search) links to DVC Get Started: Experiments (which you could of course run in a CML workflow)?

How I understand it, which might be incomplete: One makes some changes to the code or params etc. then pushes these changes to a git remote, this triggers the training (at least I would like to have that triggered) of that specific commit in a single experiment.

I don’t understand how to perform a gridsearch (or just several experiments) using CML, without following the workflow described above for each entry.