# DVC and MLFlow - reproduce experiments using git commit ids **URL:** \ **Category:** Questions\ **Created:** [November 25, 2020, 8:03pm UTC](https://discuss.dvc.org/t/dvc-and-mlflow-reproduce-experiments-using-git-commit-ids/561 "2020-11-25T20:03:57Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![al.ponomar](https://avatars.discourse-cdn.com/v4/letter/a/6de8d8/32.png) [@al.ponomar](https://discuss.dvc.org/u/al.ponomar)\ **Post date:** [November 25, 2020, 8:03pm UTC](https://discuss.dvc.org/t/dvc-and-mlflow-reproduce-experiments-using-git-commit-ids/561/1 "2020-11-25T20:03:57Z")
Hello! I’ve started using DVC and I love it, thank you for your work! I have a question regarding DVC and MLFlow combination. I hope someone can help me with that. I am using DVC to build/run pipelines and version data and models. And I am using MLFlow to have a nice overview of all the experiments and to visualize/store metrics and plots of the experiments. During the training and evaluation stages, I am logging stuff to MLFlow (including current git commit id for reproducibility). Let’s say I want to experiment with a different learning rate. My actions are: - I am updating the learning rate in params.yaml - `git commit -m 'starting a new run with updated learning rate'` - `dvc repro` (new MLFlow run with metrics and current git commit id is created) - I check the results and I like them. I want to save the model. - `git add 'dvc.lock'` - `git commit -m 'awesome learning rate'` - `dvc push` - `git push` So I have 2 commits here. Commit A before the run and commit B after. Let’s say I checked my MLFlow dashboard and I want to get the trained model of the last run. It is linked to the commit A. If I `git checkout` commit A, I won’t get the trained model until I `dvc repro` again. And I can’t log commit B to the MLFlow run during training/evaluation because I haven’t created commit B yet. Can’t really see the whole picture… How to do it properly? Any ideas would be very helpful! Thanks in advance ---
**Author:** ![dmitry](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.dvc.org/dmitry/32/7_2.png) [@dmitry](https://discuss.dvc.org/u/dmitry)\ **Post date:** [November 26, 2020, 12:46am UTC](https://discuss.dvc.org/t/dvc-and-mlflow-reproduce-experiments-using-git-commit-ids/561/2 "2020-11-26T00:46:13Z")
Hi @al.ponomar, that’s a great question. DVC + MLflow (or other experiment loggers) is a quite common use case. > [@al.ponomar](#): > > So I have 2 commits here. Commit A before the run and commit B after. The first commit is not required by DVC. You can change params and/or code, do `repro` and then commit all the code and models together if you like the result. However, the 1st commit is required by MlFlow if you’d like to attach a proper commit to the metrics. You can mitigate the issue by reporting params in mlflow but it might explode the number of entities you report - not everyone likes this idea. > [@al.ponomar](#): > > If I `git checkout` commit A, I won’t get the trained model until I `dvc repro` again. `dvc checkout` will get you the model. You can set up Git hooks to do that automatically [install](https://dvc.org/doc/command-reference/install#installed-git-hooks) > [@al.ponomar](#): > > Can’t really see the whole picture… How to do it properly? Your workflow makes sense except the 1st commit that is required by mlflow - people tend to avoid doing that. PS: We are working on the experimentation experience improvements. Two big changes are expected in DVC 2.0 (in a month or two): 1. DVC experiments without explicit commit - [Home · iterative/dvc Wiki · GitHub](https://github.com/iterative/dvc/wiki/Experiments). Plus model checkpoints - mostly for deep learning. 2. Integrations with metrics loggers (it is in a closed repository for now). ---
**Author:** ![al.ponomar](https://avatars.discourse-cdn.com/v4/letter/a/6de8d8/32.png) [@al.ponomar](https://discuss.dvc.org/u/al.ponomar)\ **Post date:** [November 26, 2020, 11:59am UTC](https://discuss.dvc.org/t/dvc-and-mlflow-reproduce-experiments-using-git-commit-ids/561/3 "2020-11-26T11:59:52Z")
Wow, what a fast and elaborate response! Thank you a lot, very helpful. Looking forward to DVC 2.0! Meanwhile, I was also thinking to add commit id B to the MLFow run after the experiment. I can log to the right experient using **mlflow\_run\_id**. The file with **mlflow\_run\_id** is an output of the training pipeline stage and tracked by dvc. So after `dvc repro` is done and I commit my results, I can run a short script that will add the current commit id (commit B) to the right MLFlow run ( **mlflow\_run\_id** ). ---
**Author:** ![dmitry](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.dvc.org/dmitry/32/7_2.png) [@dmitry](https://discuss.dvc.org/u/dmitry)\ **Post date:** [November 26, 2020, 7:57pm UTC](https://discuss.dvc.org/t/dvc-and-mlflow-reproduce-experiments-using-git-commit-ids/561/4 "2020-11-26T19:57:17Z")
> [@al.ponomar](#): > > Looking forward to DVC 2.0! 🙂 We will be doing pre-release pretty soon. I can include you in the beta users program if you are up to it. I’ll just need a contact - email or your name in our Discord chat. > [@al.ponomar](#): > > Meanwhile, I was also thinking to add commit id B to the MLFow run after the experiment. Good idea. It might work well. ---
**Author:** ![dmitry](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.dvc.org/dmitry/32/7_2.png) [@dmitry](https://discuss.dvc.org/u/dmitry)\ **Post date:** [November 26, 2020, 8:57pm UTC](https://discuss.dvc.org/t/dvc-and-mlflow-reproduce-experiments-using-git-commit-ids/561/6 "2020-11-26T20:57:37Z")
Amazing! I’ll let you know when it’s ready for beta testing. ---
**Author:** ![suparna.bhattacharya](https://avatars.discourse-cdn.com/v4/letter/s/df788c/32.png) [@suparna.bhattacharya](https://discuss.dvc.org/u/suparna.bhattacharya)\ **Post date:** [December 16, 2020, 4:35pm UTC](https://discuss.dvc.org/t/dvc-and-mlflow-reproduce-experiments-using-git-commit-ids/561/7 "2020-12-16T16:35:40Z")
Integrations with metrics loggers sounds very useful, esp. if this would enable a way to plugin any ML logging/tracking system such as mlflow, guildai, aim, wandb, mlmd etc. Is this code going to be moved to a git branch in the dvc open source tree, or will this remain closed source? It is nice to have the bits in an experimental branch when the feature is being developed. ---
**Author:** ![kupruser](https://avatars.discourse-cdn.com/v4/letter/k/e95f7d/32.png) [@kupruser](https://discuss.dvc.org/u/kupruser)\ **Post date:** [December 16, 2020, 10:37pm UTC](https://discuss.dvc.org/t/dvc-and-mlflow-reproduce-experiments-using-git-commit-ids/561/8 "2020-12-16T22:37:01Z")
Hi 🙂 Logger project will likely live in a separate git repo(we are not 100% sure yet), but it will be open-source, same as dvc. ---
**Author:** ![suparna.bhattacharya](https://avatars.discourse-cdn.com/v4/letter/s/df788c/32.png) [@suparna.bhattacharya](https://discuss.dvc.org/u/suparna.bhattacharya)\ **Post date:** [December 17, 2020, 1:07pm UTC](https://discuss.dvc.org/t/dvc-and-mlflow-reproduce-experiments-using-git-commit-ids/561/9 "2020-12-17T13:07:21Z")
Glad to hear that. The open source “by the community, for the community” aspect is one of the things that I find so exciting about the way DVC is evolving 🙂 Integration with metrics loggers would make it so much easier to enable DVC naturally in those settings. Look forward to it and curious to learn more about the design, which loggers are covered etc. I can see why a separate repo might make sense if not all users of DVC need it. I guess there may be some dependencies to sort out. ---
**Author:** ![suparna](https://avatars.discourse-cdn.com/v4/letter/s/7cd45c/32.png) [@suparna](https://discuss.dvc.org/u/suparna)\ **Post date:** [February 15, 2021, 3:43pm UTC](https://discuss.dvc.org/t/dvc-and-mlflow-reproduce-experiments-using-git-commit-ids/561/10 "2021-02-15T15:43:55Z")
Is this available now and open for contributions? From a quick look at the DVC 2.0 release plan I don’t see this project mentioned, so I am guessing this is still in another repository. ---
**Author:** ![Paffciu](https://avatars.discourse-cdn.com/v4/letter/p/d07c76/32.png) [@Paffciu](https://discuss.dvc.org/u/Paffciu)\ **Post date:** [February 15, 2021, 3:48pm UTC](https://discuss.dvc.org/t/dvc-and-mlflow-reproduce-experiments-using-git-commit-ids/561/11 "2021-02-15T15:48:43Z")
Hi @suparna, `live` is still not released, though it shouldn’t be much longer before we release it. DVC 2.0 will support it. It is not mentioned in release plan, because it will be standalone project, though DVC will have proper integration with it. ---
**Author:** ![Paffciu](https://avatars.discourse-cdn.com/v4/letter/p/d07c76/32.png) [@Paffciu](https://discuss.dvc.org/u/Paffciu)\ **Post date:** [February 17, 2021, 2:27pm UTC](https://discuss.dvc.org/t/dvc-and-mlflow-reproduce-experiments-using-git-commit-ids/561/12 "2021-02-17T14:27:12Z")
@al.ponomar @suparna @suparna.bhattacharya we have released the logger. Prerelease version is avialable under: > **[dvclive](https://pypi.org/project/dvclive/)** > > Metric logger for ML projects. Stay tuned, for release, which is planned to go together with `DVC 2.0` release. ---
**Author:** ![suparna](https://avatars.discourse-cdn.com/v4/letter/s/7cd45c/32.png) [@suparna](https://discuss.dvc.org/u/suparna)\ **Post date:** [February 17, 2021, 2:58pm UTC](https://discuss.dvc.org/t/dvc-and-mlflow-reproduce-experiments-using-git-commit-ids/561/13 "2021-02-17T14:58:38Z")
Thanks! Is the code available on github? How does one integrate this with MLflow and other tools which have their own logging APIs and UIs for analysis and visualization? Or lets say we are using Kubeflow with MLMD? ---
**Author:** ![Paffciu](https://avatars.discourse-cdn.com/v4/letter/p/d07c76/32.png) [@Paffciu](https://discuss.dvc.org/u/Paffciu)\ **Post date:** [February 17, 2021, 3:06pm UTC](https://discuss.dvc.org/t/dvc-and-mlflow-reproduce-experiments-using-git-commit-ids/561/14 "2021-02-17T15:06:01Z")
@suparna > Is the code available on github Not yet, though we will probably release it soon. So, `dvclive` is intending to replace MLFlow, rather than integrate with it. Its a metric logger producing outputs understandable by `dvc`. So, the metrics logs produced by `dvclive` can be used for example in `dvc plots` command. Currently neither `dvc` nor `dvclive` integrate with MLFlow or MLMD. ---
**Author:** ![suparna](https://avatars.discourse-cdn.com/v4/letter/s/7cd45c/32.png) [@suparna](https://discuss.dvc.org/u/suparna)\ **Post date:** [February 17, 2021, 6:24pm UTC](https://discuss.dvc.org/t/dvc-and-mlflow-reproduce-experiments-using-git-commit-ids/561/15 "2021-02-17T18:24:42Z")
Hmm … I am afraid that wasn’t what I was expecting. There are way too many open source metrics logging tools/APIs in use as it is mlflow, guildai, mlmd, aim, not mention other proprietary tools like wandb. I was looking forward to a design that would integrate with these ecosystems not more parallel universes to choose from. ---
**Author:** ![Paffciu](https://avatars.discourse-cdn.com/v4/letter/p/d07c76/32.png) [@Paffciu](https://discuss.dvc.org/u/Paffciu)\ **Post date:** [February 18, 2021, 8:54am UTC](https://discuss.dvc.org/t/dvc-and-mlflow-reproduce-experiments-using-git-commit-ids/561/16 "2021-02-18T08:54:08Z")
@suparna I understand your concerns. The thing is that the tools that you mentioned (`MLFlow`, `wandb`) are tools that, in a way, aim to solve similar problem DVC tries to solve: versioning your ML project and training pipelines. `dvclive` is library created to provide monitoring for the training loop (similar to mlflow and wandb), while not requiring user to run some kind of a server on training machine (contrary to both of them). Integrating DVC with wandb or mlflow would be hard if not impossible due to different approach those tools use for versioning - DVC is a CLI tool, while `wandb` and `mlflow` are server apps. What is your use case? How would you see the integration between DVC and, for example, `wandb`?