If we assume our currently deployed model is on the main branch. When we train a better model on a new branch and we decide to merge into main, we want to run some CI/CD tests that will confirm our branch is ok to merge. In particular we want to test that 1) all of the dvc tracked files have been pushed to the remote and 2) if you run dvc status it return something like ‘All data and pipeline up to date.’. We have set up some python tests to confirm this but it doesn’t seem the most robust, I wondered if there was a more formal approach using dvc/cml functionality for doing this?
@davesteps right now CML does not have such functionality.
It sounds like a nice feature to have. Not sure if it should be a CML’s or DVC’s one.
Could you please open a ticket in CMLs issues in the meantime?
You might also want to check out New command: dvc verify - check that the pipeline is up to date without having to pull or run it · Issue #5369 · iterative/dvc · GitHub. If that fits your needs, you could comment there to help drive the direction of that feature request.
Thanks @davidgortega and @dberenbaum. The dvc verify feature sounds great and I think will largely solve the problem, I have indicated my support in the ticket. I have opened a CML issue also as it feels as though there could be some way to integrate this feature into CML also. Feature Request: Automated testing when merging to main · Issue #681 · iterative/cml · GitHub
Many thanks.