# Statistical significant stage best practice
**URL:** https://discuss.dvc.org/t/statistical-significant-stage-best-practice/784
**Category:** Questions
**Created:** [June 8, 2021, 10:09am UTC](https://discuss.dvc.org/t/statistical-significant-stage-best-practice/784 "2021-06-08T10:09:15Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@kwon-young](https://discuss.dvc.org/u/kwon-young)
#### Post date: [June 8, 2021, 10:09am UTC](https://discuss.dvc.org/t/statistical-significant-stage-best-practice/784/1 "2021-06-08T10:09:15Z")
Hello,
In machine learning, it is important to repeat multiple time the same training (often with a different seed) and then compute the average and standard deviation for a metric to evaluate the stability of the model and check the statistical significance of the results.
Let’s say we have a stage `trainmodel` which has a parameter `seed` and we want to run the training 10 times with 10 different seeds. Once all trainings have been completed, we have a script that takes as input the results (metric files) from the 10 runs and produces the average and standard deviation for each metric.
The thing is that I don’t want to store the 10 trained models (since it is very heavy) but only keep the best trained model.
However, I want to keep the metrics of all 10 runs in order to be able to add a stage that compute the average and std of all metrics.
I also want to launch all 10 trainings in parallel.
What would be the best way to do this with dvc ?
---
### Author:  [@Paffciu](https://discuss.dvc.org/u/Paffciu)
#### Post date: [June 8, 2021, 11:21am UTC](https://discuss.dvc.org/t/statistical-significant-stage-best-practice/784/2 "2021-06-08T11:21:49Z")
Hello @kwon-young!
As to parallelization: we have an open issue for that, and running multiple stages in parallel is currently not possible:
[https://github.com/iterative/dvc/issues/755](https://github.com/iterative/dvc/issues/755)
It is possible to parallelize experiment runs when using `dvc exp run` with `--jobs` ([https://dvc.org/doc/command-reference/exp/run](https://dvc.org/doc/command-reference/exp/run)) option, however it seems that it is not applicable in your case if you later want to join the results of particular runs.
* * *
Approach #1:
It seems to me that you could be interested in pipelines files parametrization ([https://dvc.org/doc/user-guide/project-structure/pipelines-files#templating](https://dvc.org/doc/user-guide/project-structure/pipelines-files#templating)) which essentialy allows to parametrize stages. In your case you could use seed to parametrize your pipeline and later down the road join the results of the stages.
As to storing metrics and not storing intermediate results:
when creating your pipeline, use `-O/--outs-no-cache` to prevent dvc from caching the results (in that case remember to not add them to git, sometimes it happens when one uses `git add -A`), and add additional step that will choose the best and output it as dvc tracked file.
In case of metrics use `-m` (dvc will track it) or `-M` (in this case you need to track it with git)
* * *
Alternative approach:
1. Make list of seeds a parameter in `params.yaml`
2. Make train scrip accept list of seeds and create the directory of models (not tracked in dvc, look previous approach) and file with all metrics.
3. Add step choosing the best model
This approach has an advantage, that it can run multiple trainings in parallel if you implement it in your script.
---
### Author:  [@kwon-young](https://discuss.dvc.org/u/kwon-young)
#### Post date: [June 8, 2021, 12:06pm UTC](https://discuss.dvc.org/t/statistical-significant-stage-best-practice/784/3 "2021-06-08T12:06:40Z")
Thank you very much for your thoughts !
> [@Paffciu](#):
>
> Approach #1:
>
> It seems to me that you could be interested in pipelines files parametrization ([dvc.yaml Files](https://dvc.org/doc/user-guide/project-structure/pipelines-files#templating)) which essentialy allows to parametrize stages. In your case you could use seed to parametrize your pipeline and later down the road join the results of the stages.
I suppose I could use a variable to parameterize my metric output name so that the 10 runs won’t overwrite each others.
> [@Paffciu](#):
>
> As to storing metrics and not storing intermediate results:
> when creating your pipeline, use `-O/--outs-no-cache` to prevent dvc from caching the results
Thanks, I did not thought of this. However, this means that I need another almost identical stage which will save the trained model with maybe the best seed or parameter combination…
I still need to decide what is the best tradeoff between diskspace/computation time…
> [@Paffciu](#):
>
> Alternative approach:
>
> 1. Make list of seeds a parameter in `params.yaml`
> 2. Make train scrip accept list of seeds and create the directory of models (not tracked in dvc, look previous approach) and file with all metrics.
> 3. Add step choosing the best model
>
> This approach has an advantage, that it can run multiple trainings in parallel if you implement it in your script.
Unfortunately, i parallelize my trainings using a slurm like cluster, so it is really hard to do what you described.
---
### Author:  [@Paffciu](https://discuss.dvc.org/u/Paffciu)
#### Post date: [June 8, 2021, 12:32pm UTC](https://discuss.dvc.org/t/statistical-significant-stage-best-practice/784/4 "2021-06-08T12:32:03Z")
> [@kwon-young](#):
>
> I suppose I could use a variable to parameterize my metric output name so that the 10 runs won’t overwrite each others.
No need to worry about overriding if you will make code and dvc aware of param. Ex:
`train.py --seed 10` could produce `model_seed_10`. In dvc pipeline file you would need to define the output as `model_seed_${seed}`. I recommend reading through pipelines file docs page. This is really powerful feature.
> [@kwon-young](#):
>
> Unfortunately, i parallelize my trainings using a slurm like cluster, so it is really hard to do what you described.
Ah, then its not as trivial as simple python run. How does invoking the job for particular seed look like? maybe we could abstract that somehow? Is it a bash script or some cli command?
---
### Author:  [@kwon-young](https://discuss.dvc.org/u/kwon-young)
#### Post date: [June 8, 2021, 12:49pm UTC](https://discuss.dvc.org/t/statistical-significant-stage-best-practice/784/5 "2021-06-08T12:49:23Z")
> [@Paffciu](#):
>
> How does invoking the job for particular seed look like? maybe we could abstract that somehow? Is it a bash script or some cli command?
The basic idea is that you have to use a job scheduler with a bash script in which i run my script. So currently, my bash script consists of:
```auto
dvc exp run $@
dvc exp push
```
You can submit a job from a frontend server using `myscheduler-submit -S mybashscript expename -S someparams=3` and the job will be run when there are available nodes.
Once a job is finished, i get back the experiment results using `dvc pull origin myexpe`
> [@Paffciu](#):
>
> I recommend reading through pipelines file docs page. This is really powerful feature.
I have read that page and saw the `foreach` feature. This is cool because it allows to write down clearly the parameter sweeps I have done. But can each different stage produced be run in parallel ?
---
### Author:  [@dberenbaum](https://discuss.dvc.org/u/dberenbaum)
#### Post date: [June 8, 2021, 1:00pm UTC](https://discuss.dvc.org/t/statistical-significant-stage-best-practice/784/6 "2021-06-08T13:00:59Z")
Could you design your process like:
1. Make list of seeds a parameter in params.yaml.
2. Make a training script that takes a single seed and trains a single model.
3. Make a metascript that (a) takes all seeds; (b) iterates over each seed and submits a job for each that calls the training script; (c) collects the results, determines the best model, and saves only the outputs you need.
4. Make that metascript your dvc stage.
You wouldn’t get the granularity of having each model training be in its own stage, but it would parallelize your jobs and allow you to only track what you need in the end.
---
### Author:  [@kwon-young](https://discuss.dvc.org/u/kwon-young)
#### Post date: [June 8, 2021, 1:14pm UTC](https://discuss.dvc.org/t/statistical-significant-stage-best-practice/784/7 "2021-06-08T13:14:44Z")
> [@dberenbaum](#):
>
> Could you design your process like:
>
> 1. Make list of seeds a parameter in params.yaml.
> 2. Make a training script that takes a single seed and trains a single model.
> 3. Make a metascript that (a) takes all seeds; (b) iterates over each seed and submits a job for each that calls the training script; (c) collects the results, determines the best model, and saves only the outputs you need.
> 4. Make that metascript your dvc stage.
I often wanted to do just this but there are multiple problems with this approach.
So (a) and (b) is easy. But (c) is really hard to do since the execution of jobs is asynchronous:
- knowing when all jobs are finished: I would need to parse the output of cli command and check them periodically, while keeping track of which jobs I have launched
- collects the results: currently, I use `dvc exp push/pull` but that would not be possible with a metascript
- logging in the training script: currently I use dvclive, but that also would not be available with a metascript
Also, if I make my pipeline depends on the cluster, now I need to run my pipeline on the cluster frontend server, which is just a weak server for submitting job and where installing dvc is out of question.
---
### Author:  [@kwon-young](https://discuss.dvc.org/u/kwon-young)
#### Post date: [June 8, 2021, 1:48pm UTC](https://discuss.dvc.org/t/statistical-significant-stage-best-practice/784/8 "2021-06-08T13:48:40Z")
How about this strategy:
Let’s say we have a pipeline with 3 stages:
preprocessing → training → evaluation
- preprocessing: transforms data deterministcally, so can be cached and retrieve for each experiments
- training: produces metrics and a trained model, which is very large to store on disk
- evaluation: uses a trained model to produce more metrics, maybe visualization…
Let’s say we want to do a large grid-search on parameters and seeds for the training stage.
We want to keep all the metrics produced by the training stage but not the models.
However, for a small number of manually picked and interesting parameter combinations, we want to use the trained model to do the evaluation stage.
So, for the training stage, I actually use two version of the training stage:
- training: the original stage which saves the results and the model
- training-gs: the stage to use for a grid-search which won’t save the model and save metrics in files with names different for each parameter combination. all grid-search search parameter combination is stored using the `foreach` feature of a stage
So the pipeline looks now like this:
preprocessing → training → evaluation
------------------- → training-gs
First, I start by running all my training grid-search stages. I get the results, analyse them, reduce them and pick a few good parameter combinations.
Relaunch theses combinations using the training stage in order to do the evaluation stage of each of them.
---
### Author:  [@dberenbaum](https://discuss.dvc.org/u/dberenbaum)
#### Post date: [June 8, 2021, 4:07pm UTC](https://discuss.dvc.org/t/statistical-significant-stage-best-practice/784/9 "2021-06-08T16:07:49Z")
That sounds like a reasonable strategy if it works for you. A couple questions:
1. Are you doing 10 experiments with different seeds for each parameter combo, aggregating metrics for all 10 experiments in each parameter combo, and then choosing the best parameter combo?
2. Are you using the full training dataset for each experiment in the grid-search or are you sampling the data?
There are some related Github issues to your use case:
>
>
> I had an idea while trying the new experiment feature. For the use case of grid-…search or hyper-parameter optimization, could we imagine an option for \`dvc experiments checkout\` to checkout the best experiment regarding one of the metrics? Something like \`dvc experiments checkout --best metrics.json:accuracy\`. With an option like this, you can run (manually or automatically) a lot of experiments and ask dvc to return the best experiment, which can be then commited to git. Or instead of \`--best\`, it could be \`--lowest\` or \`--highest\` to take into account that some metrics should be minimized and some others maximized.
>
>
> \## Summary
> As is my current understanding of the experiment feature, after perf…orming many experiments the user \*\*has to\*\* choose a single one with the \`dvc experiments checkout \\` command, discarding all the others.
> The proposal is to allow the user to choose some of the experiments (or all of them) in order for dvc to retain all associated plots, metrics and artifacts (for example ml models).
>
> \## Use Case
> My use case is academical: I would like to correlate different hyperparameter settings with the effect they have on some metric of the resulting models, so that experiments seem to fit the usage, except that I value all of the different runs, and I would like to save them all, not only the "best performant" one.
>
> More generally, there can be cases where a single best model can't be determined at the current time: for example, it may be the case that two models have a very similar accuracy and we want to reserve to test them on other data, or we want to save them both to build a model ensemble.
>
> \## Proposed change
> Add a command to save a subset of the experiments or the special flag \`--all\` to save all of the performed ones.
> Concerning the repository structure, each of the saved experiments could be stored in a separate commit, which would then be referenced by the experiment commit as multiple ancestors.
>
> \[!\[\](https://mermaid.ink/img/eyJjb2RlIjoiZ3JhcGggVERcbkFbQW5jZXN0b3JdIC0tPiBCW0V4cGVyaW1lbnQgMV1cbkEgLS0-IENbRXhwZXJpbWVudCAyXVxuQSAtLT4gRFtFeHBlcmltZW50IDNdXG5CIC0tPiBFW0pvaW50IGNvbW1pdF1cbkMgLS0-IEVcbkQgLS0-IEUiLCJtZXJtYWlkIjp7InRoZW1lIjoiZGVmYXVsdCIsInRoZW1lVmFyaWFibGVzIjp7ImJhY2tncm91bmQiOiJ3aGl0ZSIsInByaW1hcnlDb2xvciI6IiNFQ0VDRkYiLCJzZWNvbmRhcnlDb2xvciI6IiNmZmZmZGUiLCJ0ZXJ0aWFyeUNvbG9yIjoiaHNsKDgwLCAxMDAlLCA5Ni4yNzQ1MDk4MDM5JSkiLCJwcmltYXJ5Qm9yZGVyQ29sb3IiOiJoc2woMjQwLCA2MCUsIDg2LjI3NDUwOTgwMzklKSIsInNlY29uZGFyeUJvcmRlckNvbG9yIjoiaHNsKDYwLCA2MCUsIDgzLjUyOTQxMTc2NDclKSIsInRlcnRpYXJ5Qm9yZGVyQ29sb3IiOiJoc2woODAsIDYwJSwgODYuMjc0NTA5ODAzOSUpIiwicHJpbWFyeVRleHRDb2xvciI6IiMxMzEzMDAiLCJzZWNvbmRhcnlUZXh0Q29sb3IiOiIjMDAwMDIxIiwidGVydGlhcnlUZXh0Q29sb3IiOiJyZ2IoOS41MDAwMDAwMDAxLCA5LjUwMDAwMDAwMDEsIDkuNTAwMDAwMDAwMSkiLCJsaW5lQ29sb3IiOiIjMzMzMzMzIiwidGV4dENvbG9yIjoiIzMzMyIsIm1haW5Ca2ciOiIjRUNFQ0ZGIiwic2Vjb25kQmtnIjoiI2ZmZmZkZSIsImJvcmRlcjEiOiIjOTM3MERCIiwiYm9yZGVyMiI6IiNhYWFhMzMiLCJhcnJvd2hlYWRDb2xvciI6IiMzMzMzMzMiLCJmb250RmFtaWx5IjoiXCJ0cmVidWNoZXQgbXNcIiwgdmVyZGFuYSwgYXJpYWwiLCJmb250U2l6ZSI6IjE2cHgiLCJsYWJlbEJhY2tncm91bmQiOiIjZThlOGU4Iiwibm9kZUJrZyI6IiNFQ0VDRkYiLCJub2RlQm9yZGVyIjoiIzkzNzBEQiIsImNsdXN0ZXJCa2ciOiIjZmZmZmRlIiwiY2x1c3RlckJvcmRlciI6IiNhYWFhMzMiLCJkZWZhdWx0TGlua0NvbG9yIjoiIzMzMzMzMyIsInRpdGxlQ29sb3IiOiIjMzMzIiwiZWRnZUxhYmVsQmFja2dyb3VuZCI6IiNlOGU4ZTgiLCJhY3RvckJvcmRlciI6ImhzbCgyNTkuNjI2MTY4MjI0MywgNTkuNzc2NTM2MzEyOCUsIDg3LjkwMTk2MDc4NDMlKSIsImFjdG9yQmtnIjoiI0VDRUNGRiIsImFjdG9yVGV4dENvbG9yIjoiYmxhY2siLCJhY3RvckxpbmVDb2xvciI6ImdyZXkiLCJzaWduYWxDb2xvciI6IiMzMzMiLCJzaWduYWxUZXh0Q29sb3IiOiIjMzMzIiwibGFiZWxCb3hCa2dDb2xvciI6IiNFQ0VDRkYiLCJsYWJlbEJveEJvcmRlckNvbG9yIjoiaHNsKDI1OS42MjYxNjgyMjQzLCA1OS43NzY1MzYzMTI4JSwgODcuOTAxOTYwNzg0MyUpIiwibGFiZWxUZXh0Q29sb3IiOiJibGFjayIsImxvb3BUZXh0Q29sb3IiOiJibGFjayIsIm5vdGVCb3JkZXJDb2xvciI6IiNhYWFhMzMiLCJub3RlQmtnQ29sb3IiOiIjZmZmNWFkIiwibm90ZVRleHRDb2xvciI6ImJsYWNrIiwiYWN0aXZhdGlvbkJvcmRlckNvbG9yIjoiIzY2NiIsImFjdGl2YXRpb25Ca2dDb2xvciI6IiNmNGY0ZjQiLCJzZXF1ZW5jZU51bWJlckNvbG9yIjoid2hpdGUiLCJzZWN0aW9uQmtnQ29sb3IiOiJyZ2JhKDEwMiwgMTAyLCAyNTUsIDAuNDkpIiwiYWx0U2VjdGlvbkJrZ0NvbG9yIjoid2hpdGUiLCJzZWN0aW9uQmtnQ29sb3IyIjoiI2ZmZjQwMCIsInRhc2tCb3JkZXJDb2xvciI6IiM1MzRmYmMiLCJ0YXNrQmtnQ29sb3IiOiIjOGE5MGRkIiwidGFza1RleHRMaWdodENvbG9yIjoid2hpdGUiLCJ0YXNrVGV4dENvbG9yIjoid2hpdGUiLCJ0YXNrVGV4dERhcmtDb2xvciI6ImJsYWNrIiwidGFza1RleHRPdXRzaWRlQ29sb3IiOiJibGFjayIsInRhc2tUZXh0Q2xpY2thYmxlQ29sb3IiOiIjMDAzMTYzIiwiYWN0aXZlVGFza0JvcmRlckNvbG9yIjoiIzUzNGZiYyIsImFjdGl2ZVRhc2tCa2dDb2xvciI6IiNiZmM3ZmYiLCJncmlkQ29sb3IiOiJsaWdodGdyZXkiLCJkb25lVGFza0JrZ0NvbG9yIjoibGlnaHRncmV5IiwiZG9uZVRhc2tCb3JkZXJDb2xvciI6ImdyZXkiLCJjcml0Qm9yZGVyQ29sb3IiOiIjZmY4ODg4IiwiY3JpdEJrZ0NvbG9yIjoicmVkIiwidG9kYXlMaW5lQ29sb3IiOiJyZWQiLCJsYWJlbENvbG9yIjoiYmxhY2siLCJlcnJvckJrZ0NvbG9yIjoiIzU1MjIyMiIsImVycm9yVGV4dENvbG9yIjoiIzU1MjIyMiIsImNsYXNzVGV4dCI6IiMxMzEzMDAiLCJmaWxsVHlwZTAiOiIjRUNFQ0ZGIiwiZmlsbFR5cGUxIjoiI2ZmZmZkZSIsImZpbGxUeXBlMiI6ImhzbCgzMDQsIDEwMCUsIDk2LjI3NDUwOTgwMzklKSIsImZpbGxUeXBlMyI6ImhzbCgxMjQsIDEwMCUsIDkzLjUyOTQxMTc2NDclKSIsImZpbGxUeXBlNCI6ImhzbCgxNzYsIDEwMCUsIDk2LjI3NDUwOTgwMzklKSIsImZpbGxUeXBlNSI6ImhzbCgtNCwgMTAwJSwgOTMuNTI5NDExNzY0NyUpIiwiZmlsbFR5cGU2IjoiaHNsKDgsIDEwMCUsIDk2LjI3NDUwOTgwMzklKSIsImZpbGxUeXBlNyI6ImhzbCgxODgsIDEwMCUsIDkzLjUyOTQxMTc2NDclKSJ9fSwidXBkYXRlRWRpdG9yIjpmYWxzZX0)\](https://mermaid-js.github.io/mermaid-live-editor/#/edit/eyJjb2RlIjoiZ3JhcGggVERcbkFbQW5jZXN0b3JdIC0tPiBCW0V4cGVyaW1lbnQgMV1cbkEgLS0-IENbRXhwZXJpbWVudCAyXVxuQSAtLT4gRFtFeHBlcmltZW50IDNdXG5CIC0tPiBFW0pvaW50IGNvbW1pdF1cbkMgLS0-IEVcbkQgLS0-IEUiLCJtZXJtYWlkIjp7InRoZW1lIjoiZGVmYXVsdCIsInRoZW1lVmFyaWFibGVzIjp7ImJhY2tncm91bmQiOiJ3aGl0ZSIsInByaW1hcnlDb2xvciI6IiNFQ0VDRkYiLCJzZWNvbmRhcnlDb2xvciI6IiNmZmZmZGUiLCJ0ZXJ0aWFyeUNvbG9yIjoiaHNsKDgwLCAxMDAlLCA5Ni4yNzQ1MDk4MDM5JSkiLCJwcmltYXJ5Qm9yZGVyQ29sb3IiOiJoc2woMjQwLCA2MCUsIDg2LjI3NDUwOTgwMzklKSIsInNlY29uZGFyeUJvcmRlckNvbG9yIjoiaHNsKDYwLCA2MCUsIDgzLjUyOTQxMTc2NDclKSIsInRlcnRpYXJ5Qm9yZGVyQ29sb3IiOiJoc2woODAsIDYwJSwgODYuMjc0NTA5ODAzOSUpIiwicHJpbWFyeVRleHRDb2xvciI6IiMxMzEzMDAiLCJzZWNvbmRhcnlUZXh0Q29sb3IiOiIjMDAwMDIxIiwidGVydGlhcnlUZXh0Q29sb3IiOiJyZ2IoOS41MDAwMDAwMDAxLCA5LjUwMDAwMDAwMDEsIDkuNTAwMDAwMDAwMSkiLCJsaW5lQ29sb3IiOiIjMzMzMzMzIiwidGV4dENvbG9yIjoiIzMzMyIsIm1haW5Ca2ciOiIjRUNFQ0ZGIiwic2Vjb25kQmtnIjoiI2ZmZmZkZSIsImJvcmRlcjEiOiIjOTM3MERCIiwiYm9yZGVyMiI6IiNhYWFhMzMiLCJhcnJvd2hlYWRDb2xvciI6IiMzMzMzMzMiLCJmb250RmFtaWx5IjoiXCJ0cmVidWNoZXQgbXNcIiwgdmVyZGFuYSwgYXJpYWwiLCJmb250U2l6ZSI6IjE2cHgiLCJsYWJlbEJhY2tncm91bmQiOiIjZThlOGU4Iiwibm9kZUJrZyI6IiNFQ0VDRkYiLCJub2RlQm9yZGVyIjoiIzkzNzBEQiIsImNsdXN0ZXJCa2ciOiIjZmZmZmRlIiwiY2x1c3RlckJvcmRlciI6IiNhYWFhMzMiLCJkZWZhdWx0TGlua0NvbG9yIjoiIzMzMzMzMyIsInRpdGxlQ29sb3IiOiIjMzMzIiwiZWRnZUxhYmVsQmFja2dyb3VuZCI6IiNlOGU4ZTgiLCJhY3RvckJvcmRlciI6ImhzbCgyNTkuNjI2MTY4MjI0MywgNTkuNzc2NTM2MzEyOCUsIDg3LjkwMTk2MDc4NDMlKSIsImFjdG9yQmtnIjoiI0VDRUNGRiIsImFjdG9yVGV4dENvbG9yIjoiYmxhY2siLCJhY3RvckxpbmVDb2xvciI6ImdyZXkiLCJzaWduYWxDb2xvciI6IiMzMzMiLCJzaWduYWxUZXh0Q29sb3IiOiIjMzMzIiwibGFiZWxCb3hCa2dDb2xvciI6IiNFQ0VDRkYiLCJsYWJlbEJveEJvcmRlckNvbG9yIjoiaHNsKDI1OS42MjYxNjgyMjQzLCA1OS43NzY1MzYzMTI4JSwgODcuOTAxOTYwNzg0MyUpIiwibGFiZWxUZXh0Q29sb3IiOiJibGFjayIsImxvb3BUZXh0Q29sb3IiOiJibGFjayIsIm5vdGVCb3JkZXJDb2xvciI6IiNhYWFhMzMiLCJub3RlQmtnQ29sb3IiOiIjZmZmNWFkIiwibm90ZVRleHRDb2xvciI6ImJsYWNrIiwiYWN0aXZhdGlvbkJvcmRlckNvbG9yIjoiIzY2NiIsImFjdGl2YXRpb25Ca2dDb2xvciI6IiNmNGY0ZjQiLCJzZXF1ZW5jZU51bWJlckNvbG9yIjoid2hpdGUiLCJzZWN0aW9uQmtnQ29sb3IiOiJyZ2JhKDEwMiwgMTAyLCAyNTUsIDAuNDkpIiwiYWx0U2VjdGlvbkJrZ0NvbG9yIjoid2hpdGUiLCJzZWN0aW9uQmtnQ29sb3IyIjoiI2ZmZjQwMCIsInRhc2tCb3JkZXJDb2xvciI6IiM1MzRmYmMiLCJ0YXNrQmtnQ29sb3IiOiIjOGE5MGRkIiwidGFza1RleHRMaWdodENvbG9yIjoid2hpdGUiLCJ0YXNrVGV4dENvbG9yIjoid2hpdGUiLCJ0YXNrVGV4dERhcmtDb2xvciI6ImJsYWNrIiwidGFza1RleHRPdXRzaWRlQ29sb3IiOiJibGFjayIsInRhc2tUZXh0Q2xpY2thYmxlQ29sb3IiOiIjMDAzMTYzIiwiYWN0aXZlVGFza0JvcmRlckNvbG9yIjoiIzUzNGZiYyIsImFjdGl2ZVRhc2tCa2dDb2xvciI6IiNiZmM3ZmYiLCJncmlkQ29sb3IiOiJsaWdodGdyZXkiLCJkb25lVGFza0JrZ0NvbG9yIjoibGlnaHRncmV5IiwiZG9uZVRhc2tCb3JkZXJDb2xvciI6ImdyZXkiLCJjcml0Qm9yZGVyQ29sb3IiOiIjZmY4ODg4IiwiY3JpdEJrZ0NvbG9yIjoicmVkIiwidG9kYXlMaW5lQ29sb3IiOiJyZWQiLCJsYWJlbENvbG9yIjoiYmxhY2siLCJlcnJvckJrZ0NvbG9yIjoiIzU1MjIyMiIsImVycm9yVGV4dENvbG9yIjoiIzU1MjIyMiIsImNsYXNzVGV4dCI6IiMxMzEzMDAiLCJmaWxsVHlwZTAiOiIjRUNFQ0ZGIiwiZmlsbFR5cGUxIjoiI2ZmZmZkZSIsImZpbGxUeXBlMiI6ImhzbCgzMDQsIDEwMCUsIDk2LjI3NDUwOTgwMzklKSIsImZpbGxUeXBlMyI6ImhzbCgxMjQsIDEwMCUsIDkzLjUyOTQxMTc2NDclKSIsImZpbGxUeXBlNCI6ImhzbCgxNzYsIDEwMCUsIDk2LjI3NDUwOTgwMzklKSIsImZpbGxUeXBlNSI6ImhzbCgtNCwgMTAwJSwgOTMuNTI5NDExNzY0NyUpIiwiZmlsbFR5cGU2IjoiaHNsKDgsIDEwMCUsIDk2LjI3NDUwOTgwMzklKSIsImZpbGxUeXBlNyI6ImhzbCgxODgsIDEwMCUsIDkzLjUyOTQxMTc2NDclKSJ9fSwidXBkYXRlRWRpdG9yIjpmYWxzZX0)
>
> However I still have doubts that this is the best approach.
>
> \## References
> \- \[Reference to the original discord messages.\](https://discordapp.com/channels/485586884165107732/563406153334128681/746683542703964161)
> \- \[DVC Wiki page on experiments\](https://github.com/iterative/dvc/wiki/Experiments-development-status)
Please add any feedback you have. Experiments are still a pretty new feature, so how to best support grid searches is still an open question.
By the way, I don’t now the details of your environment or scheduler, but at least in slurm I think you can use `srun` usually if you need jobs to be synchronous, and maybe even nest those inside of `sbatch` jobs to get the best of both worlds. You might be able to submit the full pipeline through `sbatch`, which does a bunch of `dvc exp run --queue` commands followed by `dvc exp run --run-all -j [number of jobs]`, which executes training stages that each include an `srun` command.
---
### Author:  [@kwon-young](https://discuss.dvc.org/u/kwon-young)
#### Post date: [June 9, 2021, 7:40am UTC](https://discuss.dvc.org/t/statistical-significant-stage-best-practice/784/10 "2021-06-09T07:40:54Z")
> [@dberenbaum](#):
>
> Are you doing 10 experiments with different seeds for each parameter combo, aggregating metrics for all 10 experiments in each parameter combo, and then choosing the best parameter combo?
Yes, that’s the plan
> [@dberenbaum](#):
>
> Are you using the full training dataset for each experiment in the grid-search or are you sampling the data?
For now, I’m using the same training/validation split for every experiment.
Adding kfold cross-validation is way too complex for now since it adds another dimension to the grid-search.
> [@dberenbaum](#):
>
> By the way, I don’t now the details of your environment or scheduler, but at least in slurm I think you can use `srun` usually if you need jobs to be synchronous, and maybe even nest those inside of `sbatch` jobs to get the best of both worlds. You might be able to submit the full pipeline through `sbatch` , which does a bunch of `dvc exp run --queue` commands followed by `dvc exp run --run-all -j [number of jobs]` , which executes training stages that each include an `srun` command.
Unfortunately, my scheduler does not have the equivalent `srun` command but I understand your strategy. The idea is to translate the pipeline execution using dvc to the relevant scheduler commands. Thank you for the insight.