Hi there! I’ve just started using DVC and am looking to set up DVC experiments to track my experiments. I have got the following experiments that I have run and logged using dvclive, which I am now trying to push:
dvc_experiments_test_branch:
c50b40a [boozy-aqua]
72f3a5a [fined-zest]
349b33e [herby-lama]
But I am getting the following error when running dvc exp push origin -v:
2024-04-12 09:45:16,851 DEBUG: v3.49.0 (pip), CPython 3.10.12 on Linux-6.5.0-1016-aws-x86_64-with-glibc2.35
2024-04-12 09:45:16,851 DEBUG: command: /home/ubuntu/Documents/dvc_test_env/bin/dvc exp push origin -v
2024-04-12 09:45:17,115 DEBUG: git push experiment ['refs/exps/98/558612ded7fb57ebf0bc1837b3adbe76c32fc9/boozy-aqua:refs/exps/98/558612ded7fb57ebf0bc1837b3adbe76c32fc9/boozy-aqua', 'refs/exps/98/558612ded7fb57ebf0bc1837b3adbe76c32fc9/fined-zest:refs/exps/98/558612ded7fb57ebf0bc1837b3adbe76c32fc9/fined-zest', 'refs/exps/98/558612ded7fb57ebf0bc1837b3adbe76c32fc9/herby-lama:refs/exps/98/558612ded7fb57ebf0bc1837b3adbe76c32fc9/herby-lama'] -> 'origin'
2024-04-12 09:45:20,057 ERROR: unexpected error - The remote server unexpectedly closed the connection.
Traceback (most recent call last):
File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/dvc/cli/__init__.py", line 211, in main
ret = cmd.do_run()
File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/dvc/cli/command.py", line 27, in do_run
return self.run()
File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/dvc/commands/experiments/push.py", line 54, in run
result = self.repo.experiments.push(
File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/dvc/repo/experiments/__init__.py", line 364, in push
return push(self.repo, *args, **kwargs)
File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/dvc/repo/__init__.py", line 58, in wrapper
return f(repo, *args, **kwargs)
File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/dvc/repo/scm_context.py", line 143, in run
return method(repo, *args, **kw)
File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/dvc/repo/experiments/push.py", line 111, in push
push_result = _push(repo, git_remote, exp_ref_set, force)
File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/dvc/repo/experiments/push.py", line 153, in _push
results: Mapping[str, SyncStatus] = repo.scm.push_refspecs(
File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/scmrepo/git/__init__.py", line 307, in _backend_func
result = func(*args, **kwargs)
File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/scmrepo/git/backend/dulwich/__init__.py", line 663, in push_refspecs
result = client.send_pack(
File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/dulwich/client.py", line 2125, in send_pack
ref_status = self._handle_receive_pack_tail(
File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/dulwich/client.py", line 934, in _handle_receive_pack_tail
for chan, data in _read_side_band64k_data(proto.read_pkt_seq()):
File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/dulwich/client.py", line 501, in _read_side_band64k_data
for pkt in pkt_seq:
File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/dulwich/protocol.py", line 272, in read_pkt_seq
pkt = self.read_pkt_line()
File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/dulwich/protocol.py", line 215, in read_pkt_line
raise HangupException
dulwich.errors.HangupException: The remote server unexpectedly closed the connection.
2024-04-12 09:45:20,089 DEBUG: link type reflink is not available ([Errno 95] no more link types left to try out)
2024-04-12 09:45:20,089 DEBUG: Removing '/home/ubuntu/Documents/.0xFMgvC9qXuZBpDPIH5A0g.tmp'
2024-04-12 09:45:20,089 DEBUG: Removing '/home/ubuntu/Documents/.0xFMgvC9qXuZBpDPIH5A0g.tmp'
2024-04-12 09:45:20,089 DEBUG: Removing '/home/ubuntu/Documents/.0xFMgvC9qXuZBpDPIH5A0g.tmp'
2024-04-12 09:45:20,089 DEBUG: Removing '/home/ubuntu/Documents/raids-u109-castle-repo/.dvc/cache/files/md5/.KMGEqzV2_6k4yD-zTyW6Zw.tmp'
2024-04-12 09:45:20,100 DEBUG: Version info for developers:
DVC version: 3.49.0 (pip)
-------------------------
Platform: Python 3.10.12 on Linux-6.5.0-1016-aws-x86_64-with-glibc2.35
Subprojects:
dvc_data = 3.15.1
dvc_objects = 5.1.0
dvc_render = 1.0.1
dvc_task = 0.4.0
scmrepo = 3.3.1
Supports:
http (aiohttp = 3.9.3, aiohttp-retry = 2.8.3),
https (aiohttp = 3.9.3, aiohttp-retry = 2.8.3),
s3 (s3fs = 2024.3.1, boto3 = 1.34.51)
Config:
Global: /home/ubuntu/.config/dvc
System: /etc/xdg/dvc
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/nvme0n1p1
Caches: local
Remotes: s3
Workspace directory: ext4 on /dev/nvme0n1p1
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/c615bcd70e1f35acafcebbca7063e140
Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2024-04-12 09:45:20,102 DEBUG: Analytics is enabled.
2024-04-12 09:45:20,150 DEBUG: Trying to spawn ['daemon', 'analytics', '/tmp/tmp03zu_gaf', '-v']
2024-04-12 09:45:20,156 DEBUG: Spawned ['daemon', 'analytics', '/tmp/tmp03zu_gaf', '-v'] with pid 4332
My git repo is in AWS CodeCommit and the remote storage for DVC is set as an S3 bucket. For information, when just tracking a model file with DVC (not experiments), dvc push works fine and I can see my models in S3.
I would really appreciate some help on this! If you need any further information or printouts in order to help, please let me know 