Dynamic parameter import

Questions
1

Hi, I have the following scenario: I’m working with a Computer Vision Deep Learning framework for which I basically can write a config file only and train. This config file is in python. I would like to input this config file as parameter dependency (under params in dvc.yaml) such that I have one place where I can change the parameters. If I don’t I need to maintain two places because some things I have to set in the config.py file…

I know I can specify the config.py as a params file. However, I have different architectures with potentially different parameters I would like to change. I made a stage using templating that switches the config.py based on the architecture name. Is it possible to create a deps.yaml (or any file) that contains a list of the keys present in config.py that should be considered parameters and ignore all the other valid potential parameters in config.py?

Thanks!

2

Hi @RiCk, sorry to suggest something that’s not actually released yet, but we have an upcoming integration with https://hydra.cc/ that aims to address this kind of scenario. In fact, there is an example in exp run: Support composing and dumping Hydra config. by daavoo · Pull Request #8093 · iterative/dvc · GitHub that seems similar to what you need. Does that look like it would work for you? If so, we will hopefully have that feature released shortly if you can wait a few days.

3

Hi @dberenbaum, thanks! I skimmed over the hydra documentation as I wasn’t familiar with it. And I saw the feature is now supported in Release 2.24.0 :owl: · iterative/dvc (github.com). I will need to check it out, but looks useful :slight_smile: for my case.