Hi there! I am using DVC experiments to track my experiments, but am coming across an error when trying to push. I have the following experiments which I have run and logged using dvclive, which I would like to push:
dvc_experiments_test_branch:
c50b40a [boozy-aqua]
72f3a5a [fined-zest]
349b33e [herby-lama]
But when running dvc exp push origin -v, I get the following error:
2024-04-12 09:45:16,851 DEBUG: v3.49.0 (pip), CPython 3.10.12 on Linux-6.5.0-1016-aws-x86_64-with-glibc2.35
2024-04-12 09:45:16,851 DEBUG: command: /home/ubuntu/Documents/dvc_test_env/bin/dvc exp push origin -v
2024-04-12 09:45:17,115 DEBUG: git push experiment ['refs/exps/98/558612ded7fb57ebf0bc1837b3adbe76c32fc9/boozy-aqua:refs/exps/98/558612ded7fb57ebf0bc1837b3adbe76c32fc9/boozy-aqua', 'refs/exps/98/558612ded7fb57ebf0bc1837b3adbe76c32fc9/fined-zest:refs/exps/98/558612ded7fb57ebf0bc1837b3adbe76c32fc9/fined-zest', 'refs/exps/98/558612ded7fb57ebf0bc1837b3adbe76c32fc9/herby-lama:refs/exps/98/558612ded7fb57ebf0bc1837b3adbe76c32fc9/herby-lama'] -> 'origin'
2024-04-12 09:45:20,057 ERROR: unexpected error - The remote server unexpectedly closed the connection.
Traceback (most recent call last):
File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/dvc/cli/__init__.py", line 211, in main
ret = cmd.do_run()
File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/dvc/cli/command.py", line 27, in do_run
return self.run()
File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/dvc/commands/experiments/push.py", line 54, in run
result = self.repo.experiments.push(
File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/dvc/repo/experiments/__init__.py", line 364, in push
return push(self.repo, *args, **kwargs)
File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/dvc/repo/__init__.py", line 58, in wrapper
return f(repo, *args, **kwargs)
File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/dvc/repo/scm_context.py", line 143, in run
return method(repo, *args, **kw)
File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/dvc/repo/experiments/push.py", line 111, in push
push_result = _push(repo, git_remote, exp_ref_set, force)
File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/dvc/repo/experiments/push.py", line 153, in _push
results: Mapping[str, SyncStatus] = repo.scm.push_refspecs(
File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/scmrepo/git/__init__.py", line 307, in _backend_func
result = func(*args, **kwargs)
File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/scmrepo/git/backend/dulwich/__init__.py", line 663, in push_refspecs
result = client.send_pack(
File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/dulwich/client.py", line 2125, in send_pack
ref_status = self._handle_receive_pack_tail(
File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/dulwich/client.py", line 934, in _handle_receive_pack_tail
for chan, data in _read_side_band64k_data(proto.read_pkt_seq()):
File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/dulwich/client.py", line 501, in _read_side_band64k_data
for pkt in pkt_seq:
File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/dulwich/protocol.py", line 272, in read_pkt_seq
pkt = self.read_pkt_line()
File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/dulwich/protocol.py", line 215, in read_pkt_line
raise HangupException
dulwich.errors.HangupException: The remote server unexpectedly closed the connection.
2024-04-12 09:45:20,089 DEBUG: link type reflink is not available ([Errno 95] no more link types left to try out)
2024-04-12 09:45:20,089 DEBUG: Removing '/home/ubuntu/Documents/.0xFMgvC9qXuZBpDPIH5A0g.tmp'
2024-04-12 09:45:20,089 DEBUG: Removing '/home/ubuntu/Documents/.0xFMgvC9qXuZBpDPIH5A0g.tmp'
2024-04-12 09:45:20,089 DEBUG: Removing '/home/ubuntu/Documents/.0xFMgvC9qXuZBpDPIH5A0g.tmp'
2024-04-12 09:45:20,089 DEBUG: Removing '/home/ubuntu/Documents/raids-u109-castle-repo/.dvc/cache/files/md5/.KMGEqzV2_6k4yD-zTyW6Zw.tmp'
2024-04-12 09:45:20,100 DEBUG: Version info for developers:
DVC version: 3.49.0 (pip)
-------------------------
Platform: Python 3.10.12 on Linux-6.5.0-1016-aws-x86_64-with-glibc2.35
Subprojects:
dvc_data = 3.15.1
dvc_objects = 5.1.0
dvc_render = 1.0.1
dvc_task = 0.4.0
scmrepo = 3.3.1
Supports:
http (aiohttp = 3.9.3, aiohttp-retry = 2.8.3),
https (aiohttp = 3.9.3, aiohttp-retry = 2.8.3),
s3 (s3fs = 2024.3.1, boto3 = 1.34.51)
Config:
Global: /home/ubuntu/.config/dvc
System: /etc/xdg/dvc
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/nvme0n1p1
Caches: local
Remotes: s3
Workspace directory: ext4 on /dev/nvme0n1p1
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/c615bcd70e1f35acafcebbca7063e140
Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2024-04-12 09:45:20,102 DEBUG: Analytics is enabled.
2024-04-12 09:45:20,150 DEBUG: Trying to spawn ['daemon', 'analytics', '/tmp/tmp03zu_gaf', '-v']
2024-04-12 09:45:20,156 DEBUG: Spawned ['daemon', 'analytics', '/tmp/tmp03zu_gaf', '-v'] with pid 4332
For information, my git repo is in AWS CodeCommit and the dvc remote storage is in an S3 bucket. When using dvc (not experiments), I can push a tracked model to the remote using dvc push and it works fine, with the model appearing in S3. This makes me think that it is the link into CodeCommit that is the issue, but I am not sure exactly what the issue is or how to resolve.
I would really appreciate some help on this! If you require any further information or printouts in order to help, please let me know and I will provide 