I think I broke my dvc installation…
My config file is:
[core]
autostage = true
remote = local_remote
['remote "local_remote"']
url = /home/fuchsfa/foundation-models/remote
dvc remote list returns:
dvc remote list
local_remote /home/fuchsfa/foundation-models/remote
but dvc exp push origin -vv still errors with:
dvc exp push origin -vv
2024-08-07 13:59:22,989 DEBUG: v3.52.0 (pip), CPython 3.10.14 on Linux-5.15.0-113-generic-x86_64-with-glibc2.31
2024-08-07 13:59:22,989 DEBUG: command: /home/fuchsfa/foundation-models/.venv/bin/dvc exp push origin -vv
2024-08-07 13:59:22,990 TRACE: Namespace(quiet=0, verbose=2, cprofile=False, cprofile_dump=None, yappi=False, yappi_separate_threads=False, viztracer=False, viztracer_depth=None, viztracer_async=False, pdb=False, instrument=False, instrument_open=False, show_stack=False, cd='.', cmd='push', all_commits=False, rev=None, num=1, force=False, push_cache=True, dvc_remote=None, jobs=None, run_cache=True, git_remote='origin', experiment=[], func=<class 'dvc.commands.experiments.push.CmdExperimentsPush'>, parser=DvcParser(prog='dvc', usage=None, description='Data Version Control', formatter_class=<class 'dvc.cli.formatter.RawTextHelpFormatter'>, conflict_handler='error', add_help=False))
2024-08-07 13:59:24,789 DEBUG: git push experiment ['refs/exps/6b/c0fc63809e34fdd99e6ea0c93cd7f6afc25cd1/aspen-loge:refs/exps/6b/c0fc63809e34fdd99e6ea0c93cd7f6afc25cd1/aspen-loge'] -> 'origin'
Pushing git refs| |0.00/? [00:00, ?obj/s]/home/fuchsfa/foundation-models/.venv/lib/python3.10/site-packages/asyncssh/crypto/cipher.py:29: CryptographyDeprecationWarning: ARC4 has been moved to cryptography.hazmat.decrepit.ciphers.algorithms.ARC4 and will be removed from this module in 48.0.0.
from cryptography.hazmat.primitives.ciphers.algorithms import AES, ARC4
/home/fuchsfa/foundation-models/.venv/lib/python3.10/site-packages/asyncssh/crypto/cipher.py:30: CryptographyDeprecationWarning: TripleDES has been moved to cryptography.hazmat.decrepit.ciphers.algorithms.TripleDES and will be removed from this module in 48.0.0.
from cryptography.hazmat.primitives.ciphers.algorithms import TripleDES
Enter passphrase for key '/home/fuchsfa/.ssh/id_ed25519':
2024-08-07 13:59:51,053 DEBUG: dvc push experiment '[ExpRefInfo(baseline_sha='6bc0fc63809e34fdd99e6ea0c93cd7f6afc25cd1', name='aspen-loge')]'
2024-08-07 13:59:51,226 TRACE: switching fs to workspace
2024-08-07 13:59:51,230 TRACE: switching fs to revision 0f93e29
2024-08-07 13:59:51,432 TRACE: Context during resolution of stage train:
{'encoder_decoder': {'mode': 'LINEAR_PROBING', 'checkpoint': None}, 'backbone': {'config_name': 'CONVNEXTV2_ATTO'}, 'decoder': {'out_channels': 1}, 'optimization': {'cudnn_benchmark': False}, 'training': {'lr': 0.001, 'epochs': 5, 'batch_size': 128, 'device': 'cuda', 'train_loader': {'num_workers': 2, 'shuffle': True, 'persistent_workers': True, 'pin_memory': True}, 'val_loader': {'num_workers': 2, 'shuffle': False, 'persistent_workers': True, 'pin_memory': True}, 'optimizer': {'name': 'AdamW', 'weight_decay': 0.05}, 'loss': {'name': 'MSELoss'}, 'lr_scheduler': {'name': 'OneCycleLR'}, 'metrics': {'PeakSignalNoiseRatio': {'data_range': 1}}}}
2024-08-07 13:59:51,438 TRACE: 138.33 ms in collecting stages from /
2024-08-07 13:59:51,439 TRACE: 17.44 mks in collecting stages from /.vscode
2024-08-07 13:59:51,441 TRACE: 32.03 mks in collecting stages from /dvclive
2024-08-07 13:59:51,442 TRACE: 7.99 mks in collecting stages from /dvclive/plots
2024-08-07 13:59:51,443 TRACE: 13.49 mks in collecting stages from /dvclive/plots/metrics
2024-08-07 13:59:51,444 TRACE: 10.66 mks in collecting stages from /dvclive/plots/metrics/train
2024-08-07 13:59:51,445 TRACE: 12.41 mks in collecting stages from /dvclive/plots/metrics/val
2024-08-07 13:59:51,445 TRACE: 8.08 mks in collecting stages from /src
2024-08-07 13:59:51,446 TRACE: 19.81 mks in collecting stages from /src/foundation_models
2024-08-07 13:59:51,447 TRACE: 16.18 mks in collecting stages from /src/foundation_models/backbones
2024-08-07 13:59:51,448 TRACE: 12.39 mks in collecting stages from /src/foundation_models/data
2024-08-07 13:59:51,448 TRACE: 12.99 mks in collecting stages from /src/foundation_models/decoders
2024-08-07 13:59:51,449 TRACE: 17.54 mks in collecting stages from /tests
Collecting |0.00 [00:00, ?entry/s]
2024-08-07 13:59:51,487 DEBUG: Preparing to transfer data from '/home/fuchsfa/foundation-models/.dvc/cache/files/md5' to '/dvc_cache/files/md5'
2024-08-07 13:59:51,487 DEBUG: Preparing to collect status from '/dvc_cache/files/md5'
2024-08-07 13:59:51,488 DEBUG: Collecting status from '/dvc_cache/files/md5'
2024-08-07 13:59:51,491 DEBUG: Preparing to collect status from '/home/fuchsfa/foundation-models/.dvc/cache/files/md5'
2024-08-07 13:59:51,492 DEBUG: Collecting status from '/home/fuchsfa/foundation-models/.dvc/cache/files/md5'
Pushing
2024-08-07 13:59:51,746 DEBUG: notifying Studio (https://studio.dvc.ai) about updated experiments
2024-08-07 13:59:51,746 TRACE: Sending {'repo_url': 'git@github.com:f-fuchs/foundation-models.git', 'client': 'dvc', 'refs': {'pushed': ['refs/exps/6b/c0fc63809e34fdd99e6ea0c93cd7f6afc25cd1/aspen-loge']}} to https://studio.dvc.ai/webhook/dvc
2024-08-07 13:59:52,369 TRACE: 404 Client Error: Not Found for url: https://studio.dvc.ai/webhook/dvc
Traceback (most recent call last):
File "/home/fuchsfa/foundation-models/.venv/lib/python3.10/site-packages/dvc/utils/studio.py", line 72, in notify_refs
r = post("webhook/dvc", token, data, base_url=base_url)
File "/home/fuchsfa/foundation-models/.venv/lib/python3.10/site-packages/dvc/utils/studio.py", line 47, in post
r.raise_for_status()
File "/home/fuchsfa/foundation-models/.venv/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://studio.dvc.ai/webhook/dvc
2024-08-07 13:59:52,375 WARNING: failed to notify Studio: permission denied or repo not found in studio (status=404)
2024-08-07 13:59:52,376 TRACE: received response: {'detail': 'Permission denied or repo not found in Studio'} (status=404)
Experiment aspen-loge is up to date on Git remote 'origin'.
2024-08-07 13:59:52,572 ERROR: failed to push cache: [Errno 30] Read-only file system: '/dvc_cache'
Traceback (most recent call last):
File "/home/fuchsfa/foundation-models/.venv/lib/python3.10/site-packages/dvc/repo/experiments/push.py", line 126, in push
result["uploaded"] = _push_cache(repo, pushed_refs_info, **kwargs)
File "/home/fuchsfa/foundation-models/.venv/lib/python3.10/site-packages/dvc/repo/experiments/push.py", line 182, in _push_cache
return repo.push(
File "/home/fuchsfa/foundation-models/.venv/lib/python3.10/site-packages/dvc/repo/__init__.py", line 58, in wrapper
return f(repo, *args, **kwargs)
File "/home/fuchsfa/foundation-models/.venv/lib/python3.10/site-packages/dvc/repo/push.py", line 147, in push
push_transferred, push_failed = ipush(
File "/home/fuchsfa/foundation-models/.venv/lib/python3.10/site-packages/dvc_data/index/push.py", line 76, in push
result = transfer(
File "/home/fuchsfa/foundation-models/.venv/lib/python3.10/site-packages/dvc_data/hashfile/transfer.py", line 224, in transfer
failed = _do_transfer(
File "/home/fuchsfa/foundation-models/.venv/lib/python3.10/site-packages/dvc_data/hashfile/transfer.py", line 121, in _do_transfer
failed_ids.update(_add(src, dest, file_ids, **kwargs))
File "/home/fuchsfa/foundation-models/.venv/lib/python3.10/site-packages/dvc_data/hashfile/transfer.py", line 165, in _add
dest.add(
File "/home/fuchsfa/foundation-models/.venv/lib/python3.10/site-packages/dvc_data/hashfile/db/__init__.py", line 99, in add
transferred = super().add(
File "/home/fuchsfa/foundation-models/.venv/lib/python3.10/site-packages/dvc_objects/db.py", line 171, in add
self._init(parts)
File "/home/fuchsfa/foundation-models/.venv/lib/python3.10/site-packages/dvc_objects/db.py", line 82, in _init
self.makedirs(self.fs.join(self.path, dname))
File "/home/fuchsfa/foundation-models/.venv/lib/python3.10/site-packages/dvc_data/hashfile/db/local.py", line 43, in makedirs
makedirs(path, exist_ok=True, mode=self._dir_mode)
File "/home/fuchsfa/foundation-models/.venv/lib/python3.10/site-packages/dvc_objects/fs/utils.py", line 126, in makedirs
makedirs(head, exist_ok=exist_ok, mode=mode)
File "/home/fuchsfa/foundation-models/.venv/lib/python3.10/site-packages/dvc_objects/fs/utils.py", line 126, in makedirs
makedirs(head, exist_ok=exist_ok, mode=mode)
File "/home/fuchsfa/foundation-models/.venv/lib/python3.10/site-packages/dvc_objects/fs/utils.py", line 126, in makedirs
makedirs(head, exist_ok=exist_ok, mode=mode)
File "/home/fuchsfa/foundation-models/.venv/lib/python3.10/site-packages/dvc_objects/fs/utils.py", line 136, in makedirs
os.mkdir(path, mode)
OSError: [Errno 30] Read-only file system: '/dvc_cache'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/fuchsfa/foundation-models/.venv/lib/python3.10/site-packages/dvc/cli/__init__.py", line 211, in main
ret = cmd.do_run()
File "/home/fuchsfa/foundation-models/.venv/lib/python3.10/site-packages/dvc/cli/command.py", line 27, in do_run
return self.run()
File "/home/fuchsfa/foundation-models/.venv/lib/python3.10/site-packages/dvc/commands/experiments/push.py", line 55, in run
result = self.repo.experiments.push(
File "/home/fuchsfa/foundation-models/.venv/lib/python3.10/site-packages/dvc/repo/experiments/__init__.py", line 364, in push
return push(self.repo, *args, **kwargs)
File "/home/fuchsfa/foundation-models/.venv/lib/python3.10/site-packages/dvc/repo/__init__.py", line 58, in wrapper
return f(repo, *args, **kwargs)
File "/home/fuchsfa/foundation-models/.venv/lib/python3.10/site-packages/dvc/repo/scm_context.py", line 143, in run
return method(repo, *args, **kw)
File "/home/fuchsfa/foundation-models/.venv/lib/python3.10/site-packages/dvc/repo/experiments/push.py", line 134, in push
raise UploadError("failed to push cache", result) from e
dvc.repo.experiments.push.UploadError: failed to push cache
2024-08-07 13:59:52,613 DEBUG: Analytics is enabled.
2024-08-07 13:59:52,617 TRACE: Saving analytics report to /tmp/tmpfi3rk_jm
2024-08-07 13:59:53,210 DEBUG: Trying to spawn ['daemon', 'analytics', '/tmp/tmpfi3rk_jm', '-vv']
2024-08-07 13:59:53,230 DEBUG: Spawned ['daemon', 'analytics', '/tmp/tmpfi3rk_jm', '-vv'] with pid 99236
2024-08-07 13:59:53,241 TRACE: Process 99070 exiting with 255
