# Workflow on slurm-like clusters

**URL:** <https://discuss.dvc.org/t/workflow-on-slurm-like-clusters/484>\
**Category:** Questions\
**Created:** [September 6, 2020, 2:09pm UTC](https://discuss.dvc.org/t/workflow-on-slurm-like-clusters/484 "2020-09-06T14:09:01Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![xyz](https://avatars.discourse-cdn.com/v4/letter/x/b5a626/32.png) [@xyz](https://discuss.dvc.org/u/xyz)\
**Post date:** [September 6, 2020, 2:09pm UTC](https://discuss.dvc.org/t/workflow-on-slurm-like-clusters/484/1 "2020-09-06T14:09:01Z")

</div>

Hi,  
I have access to a [slurm](https://slurm.schedmd.com/overview.html) (gpu) cluster and my current workflow is quite a mess. Is there any advice or experience with that? So some problems I see:

- you have to submit jobs:

- docker unfriendly

- testing the whole pipeline from data processing to “result management” is hard

- data security: everybody has access to everything:  
There are tools like [rclone](https://rclone.org/) and [crypten](https://github.com/facebookresearch/CrypTen). Can dvc help with this problem or interact with those tools?  
Can dvc / cml help there to automate things?  
Thanks in advance. 😀

---

<div class="post-metadata">

**Author:** ![xyz](https://avatars.discourse-cdn.com/v4/letter/x/b5a626/32.png) [@xyz](https://discuss.dvc.org/u/xyz)\
**Post date:** [September 19, 2020, 5:32am UTC](https://discuss.dvc.org/t/workflow-on-slurm-like-clusters/484/2 "2020-09-19T05:32:55Z")

</div>

Really no one has any thought on one of the points? Or is something unclear?

---

<div class="post-metadata">

**Author:** ![jorgeorpinel](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.dvc.org/jorgeorpinel/32/46_2.png) [@jorgeorpinel](https://discuss.dvc.org/u/jorgeorpinel)\
**Post date:** [September 19, 2020, 7:30pm UTC](https://discuss.dvc.org/t/workflow-on-slurm-like-clusters/484/3 "2020-09-19T19:30:55Z")

</div>

Oops sorry about that @xyz, this must have slipped through the cracks.

It would be helpful if you can elaborate more on your workflow and what you’re trying to achieve that maybe DVC could help with. Some doubts:

- What does your pipeline look like? What are you processing, what stages are there?
- Have you started using DVC at all? If so how?
- What do you mean by “result management”? Tracking experiment results perhaps? If so, do you do this with certain metrics, which ones?

> [@xyz](#):
>
> data security: everybody has access to everything… Can dvc help with this problem?

DVC leaves data access management up to you. It supports multiple [remote storage](https://dvc.org/doc/command-reference/remote/add#supported-storage-types) platforms, and most of them let you setup user access controls. For example if you use SSH data storage, you control that via system users. If you use Azure for remote storage, then similarly each user can be required to provide their own connection string, letting you control granular access.

Thanks

---

<div class="post-metadata">

**Author:** ![xyz](https://avatars.discourse-cdn.com/v4/letter/x/b5a626/32.png) [@xyz](https://discuss.dvc.org/u/xyz)\
**Post date:** [September 22, 2020, 9:03am UTC](https://discuss.dvc.org/t/workflow-on-slurm-like-clusters/484/5 "2020-09-22T09:03:33Z")

</div>

Hi,  
the main goal is automating as much as possible.

> Have you started using DVC at all? If so how?

I have used dvc before but not in this project. So it’s a more general question about a good slurm cluster workflow.

> What does your pipeline look like? What are you processing, what stages are there?

This is what I am using:

- mlflow for tracking results
- optuna for hyperparameter optimization
- gitlab for sharing code

My experiment workflow looks like this:

1. change a bigger part of my code locally (for example different network architecture)
2. test locally
3. push to gitlab
4. login to cluster and pull changes
5. submit several jobs: since I use optuna I don’t have to change parameters or config files. It sets parameters more or less random on the fly and I can submit several jobs at once.
6. wait for long time until first job runs to make sure there is no error (I cant test everything locally)
7. wait long time until last job finished and commit results, (mlruns folder, optuna database)
8. push results
9. pull results locally

> What do you mean by “result management”? Tracking experiment results perhaps? If so, do you do this with certain metrics, which ones?

With “result management” I mean “everything you do with your results after you collected it”. For example creating a static html with condensed information that everyone (also “non-techies”) can use and interpret. But that is not special to those clusters I think. Probably I just wanted to say that I find it hard to test everything. I track metrics, parameters, artifacts (some images) but no bigger files at the moment

> For example if you use SSH data storage, you control that via system users. If you use Azure for remote storage, then similarly each user can be required to provide their own connection string, letting you control granular access.

The bigger the data the less often you want to move it. So on this cluster you have a home folder where you can put the data and the running nodes have access to that. It’s not an important thing for me right now, just wanted to know if there is a “nice” solution at the moment.

Thanks

---

<div class="post-metadata">

**Author:** ![jorgeorpinel](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.dvc.org/jorgeorpinel/32/46_2.png) [@jorgeorpinel](https://discuss.dvc.org/u/jorgeorpinel)\
**Post date:** [September 22, 2020, 4:25pm UTC](https://discuss.dvc.org/t/workflow-on-slurm-like-clusters/484/6 "2020-09-22T16:25:49Z")

</div>

> [@xyz](#):
>
> general question about a good slurm cluster workflow  
> the main goal is automating as much as possible…
> 
> - mlflow for tracking results
> - optuna for hyperparameter optimization (I don’t have to change parameters or config files.)

Gotcha. I’ll check if other team members have specific tips for those tools.

I know that DVC and MLFlow can happily coexist. But it you mainly use the latter for tracking/visualizing results, and since you’re using GitLab, you may be interested in our sister project [CML](https://cml.dev/) (for this or other projects).

> [@xyz](#):
>
> testing the whole pipeline from data processing to “result management” is hard.  
> I track metrics, parameters, artifacts (some images) but no bigger files at the moment.

This seems like a good area where DVC can help. I’m just not sure about how to integrate it with the existing tools you’re using for hyperparams and metrics (DVC has it’s own solutions for that), you’d have to play around with it a bit — feel free to send us follow-up questions here or at [dvc.org/chat](https://dvc.org/chat).

But the tracking of large files/dirs is [one of the core features](https://dvc.org/doc/use-cases/versioning-data-and-model-files) of DVC.

> [@xyz](#):
>
> - login to cluster and pull changes
> - submit several jobs  
> So on this cluster you have a home folder where you can put the data and the running nodes have access to that

Convenient! With DVC, you could use a “[shared server](https://dvc.org/doc/use-cases/shared-development-server)” approach (variation): construct the pipeline stages using [external dependencies](https://dvc.org/doc/user-guide/external-dependencies) to the data in `~/`, since you know it will be there at run time. Locally you would need dummy or sample data files with the same file names to test your code.

> [@xyz](#):
>
> The bigger the data the less often you want to move it.

Another option is to use [external outputs](https://dvc.org/doc/user-guide/managing-external-data) — if you want to track changes in those data files/dirs.