We are using Poetry to manage Python environments. It is generally possible to invoke a python script B to run in a given Poetry environment from within python program A. I have so far done this for instance via issuing shell commands from within a Python program. Like This:
Given this structure with two differnt envs:
# within prog_a.py import subprocess cmd = "cd folder_b; poetry run python prog_b.py" subprocess.run(cmd, shell=True, check=True)
This worked without problem.
Now, when I try the same from DVC it unfortunately does not work. Say I now have the following setup:
- pyproject.toml # has DVC installed
If I specify a
cmd in a
dvc.yaml like this:
cmd : "cd abs/path/to/folder_b; poetry run python prog_b.py", then upon
dvc repro ... I expected prog_b to be run in the Poetry env from folder_b. Instead it is run in the env from folder_a (where DVC is installed).
There seems to be no way to escape this. I have even tried instead calling a wrapper through
cmd that does the exact same subprocess call as in the first example (meaning it gets the full rest of the command as string, including the cd…). Still, no luck, the env stays the same.
Does anyone have any clue why this is the case? How does DVC execute
cmd? In some special way that prevents running a process in a new env? I have tried a lot of variants of DVC / Poetry configs, to no avail.
Alternatively I appreciate suggestions how else to deal with the need to execute different stages in different environments.
Thanks lots and lots.