H!
I have a couple of DVC projects for ML model development.
I have been using the queue functionality for experiments quite a lot and it seems like all of a sudden queued experiments don’t work.
I can run an experiment to completion with dvc exp run
But the following fails:
dvc exp run --queue
dvc queue start
dvc queue logs {exp_id}
> ERROR: No output logs found for experiment {exp_id}
It fails a few seconds after starting. This happens across all my dvc projects running in different python environments.
dvc doctor
output
DVC version: 3.58.0 (pip)
-------------------------
Platform: Python 3.10.16 on macOS-15.0.1-arm64-arm-64bit
Subprojects:
dvc_data = 3.16.7
dvc_objects = 5.1.0
dvc_render = 1.0.2
dvc_task = 0.40.2
scmrepo = 3.3.9
Supports:
http (aiohttp = 3.11.11, aiohttp-retry = 2.9.1),
https (aiohttp = 3.11.11, aiohttp-retry = 2.9.1),
s3 (s3fs = 2024.12.0, boto3 = 1.35.81)
Config:
Global: /Users/user/Library/Application Support/dvc
System: /Library/Application Support/dvc
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk3s1s1
Caches: local
Remotes: s3
Workspace directory: apfs on /dev/disk3s1s1
Repo: dvc (subdir), git
Repo.site_cache_dir: /Library/Caches/dvc/repo/a4ff6d6a37d7fcdb70adbcdc6d8a45d6
Any suggestions of further debugging or solutions is much appreciated!