# Lock error with parallelized dvc repro

**URL:** <https://discuss.dvc.org/t/lock-error-with-parallelized-dvc-repro/929>\
**Category:** Questions\
**Created:** [October 20, 2021, 7:00am UTC](https://discuss.dvc.org/t/lock-error-with-parallelized-dvc-repro/929 "2021-10-20T07:00:11Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![maartenb](https://avatars.discourse-cdn.com/v4/letter/m/d6d6ee/32.png) [@maartenb](https://discuss.dvc.org/u/maartenb)\
**Post date:** [October 20, 2021, 7:00am UTC](https://discuss.dvc.org/t/lock-error-with-parallelized-dvc-repro/929/1 "2021-10-20T07:00:11Z")

</div>

I have a pipeline with many parallel stages. I use a combination of a (generated) `params.yaml` file and `foreach` approach. See below for simplified examples of `dvc.yaml`, `params.yaml`, and `dvc.lock`.

I wrote a simple scheduler in Python to run the stages with `dvc repro` in parallel using multiprocessing. Essentially, it’s running running `dvc repro setup@sim1`, `dvc repro setup@sim2`, etc. in parallel.

From what I [understand](https://dvc.org/doc/command-reference/repro#parallel-stage-execution) this should work. However, I’m getting lock errors: “ERROR: Unable to acquire lock. Most likely another DVC process is running or was terminated abruptly.”

dvc.yaml:

```auto
stages:
  setup:
    foreach: ${sims}
    do:
      cmd: python my_script.py --simlabel ${item.label}
      deps:
        - my_script.py
      outs:
        - ${item.work_dir}/outputs.txt

```

params.yaml

```auto
sims:
	sim1:
		label: label1
		parameters:
		  param1: 1
		  param2: 2
		work_dir: label1_workdir
	sim2:
		label: label2
		parameters:
		  param1: 1
		  param2: 2
		work_dir: label2_workdir

```

dvc.lock:

```auto
schema: '2.0'
stages:
  setup@sim1
    cmd: python python my_script.py --simlabel label1
    deps:
    - path: my_script.py
      md5: 614d9f23b42a36de65981aa76026600e
      size: 17401
    outs:
    - path: label1_workdir/outputs.txt
      md5: eaf7cdf7540bb96a6307394ad421ad7b
      size: 4412
  setup@sim2
    cmd: python python my_script.py --simlabel label2
    deps:
    - path: my_script.py
      md5: 614d9f23b42a36de65981aa76026600e
      size: 17401
    outs:
    - path: label2_workdir/outputs.txt
      md5: eaf7cdf7540bb96a6307394ad421ad7b
      size: 4412

```

---

<div class="post-metadata">

**Author:** ![Paffciu](https://avatars.discourse-cdn.com/v4/letter/p/d07c76/32.png) [@Paffciu](https://discuss.dvc.org/u/Paffciu)\
**Post date:** [October 26, 2021, 6:57pm UTC](https://discuss.dvc.org/t/lock-error-with-parallelized-dvc-repro/929/2 "2021-10-26T18:57:42Z")

</div>

Hi @maartenb sorry for late response,  
So the problem here is that both stages (while different due to parameters use) use the same dependency:

- my\_script.py

dvc is locking it and hence you get an error. So this is a little bit different use case than the one linked by you in your post, where the pipelines are separate completely.  
I am afraid there is no straightforward workaround for that, besides creating several copies of the script and naming them differently, though that would probably not be very elegant.

Having said so, you might be interested in `exp` command, which should allow you to achieve your goal:

> **[exp run](https://dvc.org/doc/command-reference/exp/run#exp-run)**
>
> Open-source version control system for Data Science and Machine Learning projects. Git-like experience to organize your data, models, and experiments.

You could get parallel execution with the following commands:

`dvc exp run setup@sim1 --queue`  
`dvc exp run setup@sim2 --queue`  
`dvc exp run -j {2}`

Please read on `exp` and share your thoughts on whether this is something you might find useful.

---

<div class="post-metadata">

**Author:** ![gregstarr](https://avatars.discourse-cdn.com/v4/letter/g/c77e96/32.png) [@gregstarr](https://discuss.dvc.org/u/gregstarr)\
**Post date:** [November 3, 2023, 9:58pm UTC](https://discuss.dvc.org/t/lock-error-with-parallelized-dvc-repro/929/3 "2023-11-03T21:58:34Z")

</div>

I want to run several training pipelines in parallel, but they all depend on the same data. This is giving me the same error, that the data file is locked by the other process. If I run each pipeline as an experiment, will it be easy to merge them back together?

---

<div class="post-metadata">

**Author:** ![gregstarr](https://avatars.discourse-cdn.com/v4/letter/g/c77e96/32.png) [@gregstarr](https://discuss.dvc.org/u/gregstarr)\
**Post date:** [November 3, 2023, 10:09pm UTC](https://discuss.dvc.org/t/lock-error-with-parallelized-dvc-repro/929/4 "2023-11-03T22:09:40Z")

</div>

posted in discord:

Hello, I want to run multiple DVC pipelines in parallel which share a dependency. Specifically these are training pipelines which share the data dependency and some code dependency, but have separate output directories. When I try to run `dvc repro <pipeline>` twice from separate processes, the second one says that the dependency stage is locked:

```auto
Stage 'pipelines/data/dvc.yaml:prepare_dataset@H' didn't change, skipping
ERROR: failed to reproduce 'pipelines/data/dvc.yaml:make_folds@H': ' _____ /data/folds/H' is busy, it is being blocked by:
  (PID 317937): _____ dvc repro pipelines/twod/dvc.yaml

If there are no processes with such PIDs, you can manually remove ' ________ /.dvc/tmp/rwlock' and try again.

```

How can I get around this? It seems like one possibility would be running the pipelines as experiments, but would it be easy to merge the experiment results back together? I think it should be because the pipelines have separate dvc.yaml files and dvc.lock files so I think I could just `dvc exp apply` each of the runs individually. Any advice?

---

<div class="post-metadata">

**Author:** ![kupruser](https://avatars.discourse-cdn.com/v4/letter/k/e95f7d/32.png) [@kupruser](https://discuss.dvc.org/u/kupruser)\
**Post date:** [November 6, 2023, 3:36am UTC](https://discuss.dvc.org/t/lock-error-with-parallelized-dvc-repro/929/5 "2023-11-06T03:36:15Z")

</div>

For the record: responded in [Discord](https://discord.com/channels/485586884165107732/485596304961962003/1170122256966483968)

---

<div class="post-metadata">

**Author:** ![gregstarr](https://avatars.discourse-cdn.com/v4/letter/g/c77e96/32.png) [@gregstarr](https://discuss.dvc.org/u/gregstarr)\
**Post date:** [November 9, 2023, 4:53pm UTC](https://discuss.dvc.org/t/lock-error-with-parallelized-dvc-repro/929/6 "2023-11-09T16:53:21Z")

</div>

Hello,

I am running the stages as experiments and they are running in parallel, but they are each reproducing the common dependency.

Here is a sketch of my DAG:

 ![image](https://canada1.discourse-cdn.com/flex035/uploads/dataversioncontrol/original/1X/d46df5db44a3bc56e0a2a3371b8c0d556a258870.png)

I manually ran `train model 1` / `evaluate model 1`, so it made sure `prepare dataset` was up to date. I verified this with `dvc status`. Then I queue up the other two pipelines with

```auto
dvc exp run --queue pipelines/model_2/dvc.yaml
dvc exp run --queue pipelines/model_3/dvc.yaml

```

But then when I run the two of them with `dvc exp run --run-all -j 2`, they both reproduce `prepare dataset`. `prepare dataset` has a directory output like this: `remote://records/${item.p1}/${item.p2}` where `item` comes from a matrix stage. The model training stages depend on `remote://records/${var}/${item}` where var is defined in the `vars` section and `item` is from the foreach stage.

Any ideas how to make the training stages not reproduce the dependency which should already be up to date?

---

<div class="post-metadata">

**Author:** ![dberenbaum](https://avatars.discourse-cdn.com/v4/letter/d/5e9695/32.png) [@dberenbaum](https://discuss.dvc.org/u/dberenbaum)\
**Post date:** [November 13, 2023, 1:44pm UTC](https://discuss.dvc.org/t/lock-error-with-parallelized-dvc-repro/929/7 "2023-11-13T13:44:36Z")

</div>

Hm, it doesn’t sound right that `prepare dataset` gets triggered if it’s already up to date. Is it possible to produce a simple reproducible example of this happening?

---

<div class="post-metadata">

**Author:** ![gregstarr](https://avatars.discourse-cdn.com/v4/letter/g/c77e96/32.png) [@gregstarr](https://discuss.dvc.org/u/gregstarr)\
**Post date:** [November 13, 2023, 4:36pm UTC](https://discuss.dvc.org/t/lock-error-with-parallelized-dvc-repro/929/8 "2023-11-13T16:36:00Z")

</div>

I think this was actually an issue on my end, I had an extra dependency in one of my stages. Thanks

---

<div class="post-metadata">

**Author:** ![daniel.castillo99](https://avatars.discourse-cdn.com/v4/letter/d/e19b73/32.png) [@daniel.castillo99](https://discuss.dvc.org/u/daniel.castillo99)\
**Post date:** [July 21, 2026, 2:37pm UTC](https://discuss.dvc.org/t/lock-error-with-parallelized-dvc-repro/929/9 "2026-07-21T14:37:18Z")

</div>

Hi, how did you solve this problem? I’m getting a similar error when running two phases synchronously. First, I build the data in a stage that the rest of the pipeline depends on. Then, in one branch, I run the training for phase 1 followed by its hyperparameter tuning, and in the other branch, I do the same for phase 2. The problem arises when I run phases 2 and 1 sequentially, as it returns an error.