S3 pushing error

Questions
1

I’m trying to push file with 860Gb size. And I have the following error:

ERROR: failed to transfer 'f5a305d4584f07ea756e2d33b039b63d' - [Errno 22] Argument partNumber must be an integer between 1 and 10000.: An error occurred (InvalidArgument) when calling the UploadPart operation: Argument partNumber must be an integer between 1 and 10000.

I’ve tried to search what I can do, but found nothing. I don’t use AWS console and use only dvc_s3 package. I’ve tried to set “multipart_chunksize” manually to 20Gb, but it did’t help. What can I do?

$ dvc doctor

DVC version: 3.49.0 (pip)
-------------------------
Platform: Python 3.10.12 on Linux-5.15.0-71-generic-x86_64-with-glibc2.35
Subprojects:
        dvc_data = 3.15.1
        dvc_objects = 5.1.0
        dvc_render = 1.0.1
        dvc_task = 0.4.0
        scmrepo = 3.3.1
Supports:
        gdrive (pydrive2 = 1.19.0),
        http (aiohttp = 3.9.3, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.9.3, aiohttp-retry = 2.8.3),
        s3 (s3fs = 2024.3.1, boto3 = 1.34.69)
Config:
        Global: /home/ermolaev/.config/dvc
        System: /etc/xdg/dvc
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/sdc1
Caches: local
Remotes: gdrive, gdrive, gdrive, s3
Workspace directory: ext4 on /dev/sdc1
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/ba8968b2882f4c6075c2ef5ec4d678b1
2

Can you try dvc push --jobs 1?

3

Unfortunatelly nothing has changed :frowning:

$ dvc push -r yadrive MosMed_0823/h5/train.dvc --jobs 1
Collecting                                                                                                                                                                        |0.00 [00:00,    ?entry/s]
ERROR: failed to transfer 'f5a305d4584f07ea756e2d33b039b63d' - [Errno 22] Argument partNumber must be an integer between 1 and 10000.: An error occurred (InvalidArgument) when calling the UploadPart operation: Argument partNumber must be an integer between 1 and 10000.                                                                                                                                           
Pushing                                                                                                                                                                                                     
ERROR: failed to push data to the cloud - 2 files failed to upload
4

Hi. What kind of storage are you using? Is it s3 or s3-compatible?

And what are the size of files that you are trying to upload?

We try to upload multiple chunks of a file at once (each of 50 MB), so to go out of range of 10k, you’d have to be uploading a 500 GB file. Is that what’s happening?

5

Hi!

  • It’s s3-compatible, yandex bucket actually
  • The size of the file is 860Gb. All files less than 500Gb have been uploaded correcty.

I’ve tried to change size of a chunk. I didn’t find a parameter for it, but I changed dvc-s3 code manually to 20Gb, it also didn’t help:

image