Tuning Hyperparameters with Reproducible Experiments

Blog Comments
1

Please leave comments and ask questions regarding @milecia’s post on hyperparameter tuning with experiments!

2

Hey @milecia,
Thanks for the tutorial. However, I have some doubts regarding remote training using CML. Can’t we run all these queued experiments using CICD (CML)? do we have to create individual branch for each run?

3

Hi @RaghavaAlajangi, yes you could run them all using CML. Could you expand more on what you are trying to do and we can try to provide some more specific guidance?

1 Like
4

@dberenbaum Thanks for the reply.
I have a DVC repo configured with remote storage and a self-hosted GPU runner (CML). So, my idea is to have a params.yaml file with all the parameter combinations (like batch_size: [8,16,32]) and then train those models using Gitlab CICD and CML. Finally, persist the run which gives me better performance. Basically, I do not want to change anything in the pipeline except params.yaml file to run new experiments. However, in this tutorial, you trained models for local hyperparameter tuning but did not mention how to do remote training.

Moreover, I have some other questions like:

  • In case I create an experiment queue and want to push them as branches, should I manually create those branches?
  • Can I create the experiment queue in GitLab CI and then run all the experiments?
# gitlab-ci.yaml
queue experiments:  
stage: queue
   dvc exp run --queue -S params.yaml
run experiments:
stage: train
   dvc exp run --run-all
1 Like
5

Thanks @RaghavaAlajangi! Yes, your proposed workflow makes sense. You can use dvc exp push --rev HEAD origin after running all the experiments to push them all without having to create a branch in each one. Unfortunately, there are a few tricky permissions issues to handle in the CI script, and our docs for it are still a work in progress. Would you mind posting your full YAML file and we can try to help you make it work? Also, do you plan to run them all sequentially, and how long does each training job take?

1 Like