ERROR: failed to reproduce 'prepare': [ERRNO 16] Device or resource busy: '/train/data/prepare'

Questions
1

I try to dockerize a full training DVC pipeline offline. And I have the same error (as mention in the title). I’ve mounted (and unmount) the prepare folder in my docker-compose.yaml. I’ve copied each folder in my container. Maybe there is some problem of compatibility between DVC and docker…
I don’t know what could have caused the error. I’m open to hear about some resolution track or even some example of Dockerfile using working dvc environment.

Thank you in advance.

Docker version : 20.10.21

1 Like
2

hey, @d9r2e3 . Here is the repo - GitHub - shcheklein/dvc-docker-example: An example of DVC pipeline with a Docker-wrapped command - I’ve recently prepared while experimenting with different ways to run Docker + DVC. I’ve tested all the scenarios on Win and Mac. I don’t remember seeing the error. Could you share the compose file, or a small repo to try to reproduce it?

1 Like
3

hello @shcheklein, thank you for you answer.
Here is some extra information.
The tree of my project to understand how I mount my volumes:

.
├── core
│   ├── modules
│   └── utils
├── data
│   ├── clean
│   └── prepare
├── .dvc
├── .git
├── models
├── pipeline
│   ├── instruct
│   │   ├── dvc.yaml
│   │   ├── instruct.docker-compose.yml
│   │   ├── instruct.Dockerfile
│   │   └── params.yaml
│   └── pretrain
└── requirements.txt

My dockerfile:

FROM <cuda:12.0.1-cudnn8-runtime-ubuntu22.04>

WORKDIR /train

RUN apt update -y && apt-get install -y git python3.10 python3-pip
RUN python3.10 -m pip install dvc[s3]

COPY ./requirements.txt /train/requirements.txt
COPY ./pipelines/instruct/dvc.yaml /train/pipelines/instruct/dvc.yaml
COPY ./pipelines/instruct/params.yaml /train/pipelines/instruct/params.yaml
COPY ./.git /train/.git
COPY ./.gitignore /train/.gitignore
COPY ./.dvc /train/.dvc
COPY ./core /train/core
COPY ./data /train/data
COPY ./models/train/models

RUN python3.10 -m pip install -r /train/requirements.txt

ENTRYPOINT ["dvc", "repro"]

And my docker-compose file:

version: "3.2"

services:
  train:
    build:
      context: ../..
      dockerfile: ./pipeline/instruct/instruct.Dockerfile
    volumes:
      - ../../models:/train/models:rw
      - ../../data:/train/data:rw
      - ../../.git:/train/.git
      - ../../.dvc:/train/.dvc
      - ../../core:/train/core

Maybe I will try to explore your solutions, thanks for your repo!