True parallel dvc repro of independent branches on shared workspace (Databricks) — rwlock prevents concurrent execution

Bug Reports
1

Environment

  • DVC version: 3.67.1 (pip)
  • Platform: Databricks Asset Bundle (Azure).

Problem

I have a pipeline with two independent branches (ModelA and ModelB) that share a common root but diverge completely after data_explorer:

dvc dag
+------------+     +------------+
| transform  |     | transform  |   (same stage)
+------------+     +------------+
      |                   |
+------------+     +------------+
| data_expl  |     | data_expl  |   (same stage)
+------------+     +------------+
      |                   |
+------------+     +------------+
| train_A  |     | train_B  |   ← INDEPENDENT
+------------+     +------------+
      ...                 ...
+------------+     +------------+
| hyper_A  |     | hyper_B  |
+------------+     +------------+
          \         /
        +----------+
        | assembly |
        +----------+

Each branch takes ~3 hours. I’m using Databricks multi-task jobs to orchestrate the pipeline. When I configure tunning_A(hyper_A ) and tunning_B(hyper_B) as parallel tasks (both depends_on: setup), I get:

ERROR: Unable to acquire lock. Most likely another DVC process is running
or was terminated abruptly. (.dvc/tmp/rwlock)

Root cause identified

All Databricks tasks in the same job share the same workspace path (/Workspace/Users/.../.bundle/project/files/), so they share the same rwlock. DVC acquires an exclusive lock for the entire dvc repro operation, making concurrent execution on the same workspace impossible.

Question

The official docs say:

“you can launch dvc repro multiple times concurrently (e.g. in separate terminals)”

But this seems to only work in truly separate working directories (different clones). Is there:

  1. A way to configure DVC’s lock timeout/retry behavior so one process waits instead of immediately failing?
  2. A --no-lock flag or equivalent for dvc repro when the user guarantees the stages are independent?
  3. A recommended pattern for running independent DAG branches in parallel within a CI/CD system that shares a single workspace (not multiple git clones)?
  4. Would running each branch task in a separate temp directory (copy of the workspace files) on local disk, then merging dvc.lock results, be a supported/safe approach?

What I’ve tried

  • Cleaning rwlock before each task: helps with stale locks from aborted runs but not with active concurrent locks
  • Sequential execution: works but totals ~6h instead of ~3h

Any guidance appreciated.