Dvc exp push to AWS CodeCommit fails

Hi there! I am using DVC experiments to track my experiments, but am coming across an error when trying to push. I have the following experiments which I have run and logged using dvclive, which I would like to push:

	c50b40a [boozy-aqua]
	72f3a5a [fined-zest]
	349b33e [herby-lama]

But when running dvc exp push origin -v, I get the following error:

2024-04-12 09:45:16,851 DEBUG: v3.49.0 (pip), CPython 3.10.12 on Linux-6.5.0-1016-aws-x86_64-with-glibc2.35
2024-04-12 09:45:16,851 DEBUG: command: /home/ubuntu/Documents/dvc_test_env/bin/dvc exp push origin -v
2024-04-12 09:45:17,115 DEBUG: git push experiment ['refs/exps/98/558612ded7fb57ebf0bc1837b3adbe76c32fc9/boozy-aqua:refs/exps/98/558612ded7fb57ebf0bc1837b3adbe76c32fc9/boozy-aqua', 'refs/exps/98/558612ded7fb57ebf0bc1837b3adbe76c32fc9/fined-zest:refs/exps/98/558612ded7fb57ebf0bc1837b3adbe76c32fc9/fined-zest', 'refs/exps/98/558612ded7fb57ebf0bc1837b3adbe76c32fc9/herby-lama:refs/exps/98/558612ded7fb57ebf0bc1837b3adbe76c32fc9/herby-lama'] -> 'origin'
2024-04-12 09:45:20,057 ERROR: unexpected error - The remote server unexpectedly closed the connection.
Traceback (most recent call last):
  File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/dvc/cli/__init__.py", line 211, in main
    ret = cmd.do_run()
  File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/dvc/cli/command.py", line 27, in do_run
    return self.run()
  File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/dvc/commands/experiments/push.py", line 54, in run
    result = self.repo.experiments.push(
  File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/dvc/repo/experiments/__init__.py", line 364, in push
    return push(self.repo, *args, **kwargs)
  File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/dvc/repo/__init__.py", line 58, in wrapper
    return f(repo, *args, **kwargs)
  File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/dvc/repo/scm_context.py", line 143, in run
    return method(repo, *args, **kw)
  File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/dvc/repo/experiments/push.py", line 111, in push
    push_result = _push(repo, git_remote, exp_ref_set, force)
  File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/dvc/repo/experiments/push.py", line 153, in _push
    results: Mapping[str, SyncStatus] = repo.scm.push_refspecs(
  File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/scmrepo/git/__init__.py", line 307, in _backend_func
    result = func(*args, **kwargs)
  File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/scmrepo/git/backend/dulwich/__init__.py", line 663, in push_refspecs
    result = client.send_pack(
  File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/dulwich/client.py", line 2125, in send_pack
    ref_status = self._handle_receive_pack_tail(
  File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/dulwich/client.py", line 934, in _handle_receive_pack_tail
    for chan, data in _read_side_band64k_data(proto.read_pkt_seq()):
  File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/dulwich/client.py", line 501, in _read_side_band64k_data
    for pkt in pkt_seq:
  File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/dulwich/protocol.py", line 272, in read_pkt_seq
    pkt = self.read_pkt_line()
  File "/home/ubuntu/Documents/dvc_test_env/lib/python3.10/site-packages/dulwich/protocol.py", line 215, in read_pkt_line
    raise HangupException
dulwich.errors.HangupException: The remote server unexpectedly closed the connection.

2024-04-12 09:45:20,089 DEBUG: link type reflink is not available ([Errno 95] no more link types left to try out)
2024-04-12 09:45:20,089 DEBUG: Removing '/home/ubuntu/Documents/.0xFMgvC9qXuZBpDPIH5A0g.tmp'
2024-04-12 09:45:20,089 DEBUG: Removing '/home/ubuntu/Documents/.0xFMgvC9qXuZBpDPIH5A0g.tmp'
2024-04-12 09:45:20,089 DEBUG: Removing '/home/ubuntu/Documents/.0xFMgvC9qXuZBpDPIH5A0g.tmp'
2024-04-12 09:45:20,089 DEBUG: Removing '/home/ubuntu/Documents/raids-u109-castle-repo/.dvc/cache/files/md5/.KMGEqzV2_6k4yD-zTyW6Zw.tmp'
2024-04-12 09:45:20,100 DEBUG: Version info for developers:
DVC version: 3.49.0 (pip)
Platform: Python 3.10.12 on Linux-6.5.0-1016-aws-x86_64-with-glibc2.35
	dvc_data = 3.15.1
	dvc_objects = 5.1.0
	dvc_render = 1.0.1
	dvc_task = 0.4.0
	scmrepo = 3.3.1
	http (aiohttp = 3.9.3, aiohttp-retry = 2.8.3),
	https (aiohttp = 3.9.3, aiohttp-retry = 2.8.3),
	s3 (s3fs = 2024.3.1, boto3 = 1.34.51)
	Global: /home/ubuntu/.config/dvc
	System: /etc/xdg/dvc
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/nvme0n1p1
Caches: local
Remotes: s3
Workspace directory: ext4 on /dev/nvme0n1p1
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/c615bcd70e1f35acafcebbca7063e140

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2024-04-12 09:45:20,102 DEBUG: Analytics is enabled.
2024-04-12 09:45:20,150 DEBUG: Trying to spawn ['daemon', 'analytics', '/tmp/tmp03zu_gaf', '-v']
2024-04-12 09:45:20,156 DEBUG: Spawned ['daemon', 'analytics', '/tmp/tmp03zu_gaf', '-v'] with pid 4332

For information, my git repo is in AWS CodeCommit and the dvc remote storage is in an S3 bucket. When using dvc (not experiments), I can push a tracked model to the remote using dvc push and it works fine, with the model appearing in S3. This makes me think that it is the link into CodeCommit that is the issue, but I am not sure exactly what the issue is or how to resolve.

I would really appreciate some help on this! If you require any further information or printouts in order to help, please let me know and I will provide :slight_smile: