Error in pushing experiments

ermolaev · July 3, 2024, 2:39pm

Hello!

I faced with a problem when trying to push an experiment:

$ dvc exp push origin rf_sgm_v6.01
ERROR: unexpected error - can't concat tuple to bytes                                                                                                                                                                                                   

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!

The same error running other command:

$ dvc exp list origin
ERROR: unexpected error - can't concat tuple to bytes

while local run works fine:

$ dvc exp list
feature/228-ribs-spec-metrics:                                        
        f3a4ea4 [rf_sgm_v6.01]

What is wrong and how this error can be reffered?

$ dvc doctor
DVC version: 3.49.0 (pip)
-------------------------
Platform: Python 3.10.12 on Linux-6.8.0-35-generic-x86_64-with-glibc2.35
Subprojects:
        dvc_data = 3.15.1
        dvc_objects = 5.1.0
        dvc_render = 1.0.1
        dvc_task = 0.4.0
        scmrepo = 3.3.1
Supports:
        gdrive (pydrive2 = 1.19.0),
        http (aiohttp = 3.9.3, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.9.3, aiohttp-retry = 2.8.3)
Config:
        Global: /home/ermolaev/.config/dvc
        System: /etc/xdg/dvc
Cache types: symlink
Cache directory: ext4 on /dev/sdc1
Caches: local
Remotes: gdrive, gdrive, gdrive
Workspace directory: ext4 on /dev/sdb1
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/7205a6ce3131e59a2db7211a94dd5faa

dberenbaum · July 3, 2024, 8:16pm

Can you show the output of dvc exp push -v origin rf_sgm_v6.01?

ermolaev · July 4, 2024, 12:12pm

$ dvc exp push origin rf_sgm_v6.01 -v                                  
2024-07-04 15:09:45,204 DEBUG: v3.51.2 (pip), CPython 3.10.12 on Linux-6.8.0-35-generic-x86_64-with-glibc2.35                                               
2024-07-04 15:09:45,204 DEBUG: command: /home/ermolaev/projects/radml/venv/bin/dvc exp push origin rf_sgm_v6.01 -v                                          
2024-07-04 15:09:45,390 DEBUG: git push experiment ['refs/exps/84/1f5b33a660735f00ae10400250c8b68e4244ca/rf_sgm_v6.01:refs/exps/84/1f5b33a660735f00ae1040025
0c8b68e4244ca/rf_sgm_v6.01'] -> 'origin'                                                                                                                    
2024-07-04 15:09:45,568 ERROR: unexpected error - can't concat tuple to bytes                                                                               
Traceback (most recent call last):                                                                                                                          
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc/cli/__init__.py", line 211, in main                                             
    ret = cmd.do_run()                                                                                                                                      
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc/cli/command.py", line 27, in do_run                                             
    return self.run()                                                                                                                                       
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc/commands/experiments/push.py", line 55, in run                                  
    result = self.repo.experiments.push(                                                                                                                    
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc/repo/experiments/__init__.py", line 364, in push                                
    return push(self.repo, *args, **kwargs)                                                                                                                 
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc/repo/__init__.py", line 58, in wrapper                                          
    return f(repo, *args, **kwargs)                                                                                                                         
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc/repo/scm_context.py", line 143, in run                                          
    return method(repo, *args, **kw)                                                                                                                        
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc/repo/experiments/push.py", line 111, in push                                    
    push_result = _push(repo, git_remote, exp_ref_set, force)                                                                                               
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc/repo/experiments/push.py", line 153, in _push                                   
    results: Mapping[str, SyncStatus] = repo.scm.push_refspecs(                                                                                             
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/scmrepo/git/__init__.py", line 307, in _backend_func                                
    result = func(*args, **kwargs)                                                                                                                          
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/scmrepo/git/backend/dulwich/__init__.py", line 663, in push_refspecs                
    result = client.send_pack(                                                                                                                              
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dulwich/client.py", line 1051, in send_pack                                         
    proto, unused_can_read, stderr = self._connect(b"receive-pack", path)                                                                                   
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dulwich/client.py", line 1799, in _connect                                          
    con = self.ssh_vendor.run_command(                                                                                                                      
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/fsspec/asyn.py", line 118, in wrapper                                               
    return sync(self.loop, func, *args, **kwargs)                                                                                                           
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/fsspec/asyn.py", line 103, in sync                                                  
    raise return_result                                                                                                                                     
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/fsspec/asyn.py", line 56, in _runner                                                
    result[0] = await coro                                                                                                                                  
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/scmrepo/git/backend/dulwich/asyncssh_vendor.py", line 297, in _run_command
    conn = await asyncssh.connect(
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/asyncssh/connection.py", line 8269, in connect
    return await asyncio.wait_for(
  File "/usr/lib/python3.10/asyncio/tasks.py", line 408, in wait_for
    return await fut
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/asyncssh/connection.py", line 436, in _connect
    await options.waiter
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/asyncssh/connection.py", line 1087, in _reap_task
    task.result()
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/asyncssh/auth.py", line 462, in _start
    await self.send_request(Boolean(False), String(password),
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/asyncssh/packet.py", line 73, in String
    return len(value).to_bytes(4, 'big') + value
TypeError: can't concat tuple to bytes

2024-07-04 15:09:45,618 DEBUG: link type reflink is not available ([Errno 95] no more link types left to try out)
2024-07-04 15:09:45,618 DEBUG: Removing '/home/ermolaev/projects/radml/.wIsplIKYc0sISmbKPLwLGw.tmp'
2024-07-04 15:09:45,618 DEBUG: link type hardlink is not available ([Errno 95] no more link types left to try out)
2024-07-04 15:09:45,618 DEBUG: Removing '/home/ermolaev/projects/radml/.wIsplIKYc0sISmbKPLwLGw.tmp'
2024-07-04 15:09:45,618 DEBUG: Removing '/home/ermolaev/projects/radml/.wIsplIKYc0sISmbKPLwLGw.tmp'
2024-07-04 15:09:45,618 DEBUG: Removing '/data/projects/radml/DVC_CACHE/files/md5/.INaDvNAnn5LmTGfxqdyZnQ.tmp'
2024-07-04 15:09:45,626 DEBUG: Version info for developers:
DVC version: 3.51.2 (pip)
-------------------------
Platform: Python 3.10.12 on Linux-6.8.0-35-generic-x86_64-with-glibc2.35
Subprojects:
        dvc_data = 3.15.1
        dvc_objects = 5.1.0
        dvc_render = 1.0.1
        dvc_task = 0.4.0
        scmrepo = 3.3.5
Supports:
        gdrive (pydrive2 = 1.19.0),
        http (aiohttp = 3.9.3, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.9.3, aiohttp-retry = 2.8.3)
Config:
        Global: /home/ermolaev/.config/dvc
        System: /etc/xdg/dvc
Cache types: symlink
Cache directory: ext4 on /dev/sdc1
Caches: local
Remotes: gdrive, gdrive, gdrive
Workspace directory: ext4 on /dev/sdb1
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/7205a6ce3131e59a2db7211a94dd5faa

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2024-07-04 15:09:45,628 DEBUG: Analytics is enabled.
2024-07-04 15:09:45,667 DEBUG: Trying to spawn ['daemon', 'analytics', '/tmp/tmp0pjfzw19', '-v']
2024-07-04 15:09:45,671 DEBUG: Spawned ['daemon', 'analytics', '/tmp/tmp0pjfzw19', '-v'] with pid 262799

dberenbaum · July 5, 2024, 12:43pm

It looks like asynssh is trying to read your password but finding a tuple instead of string. Do you have anything unusual in your SSH config or Git config (especially in credential helpers)?

ermolaev · July 5, 2024, 3:11pm

I have several ssh keys and my config of .ssg/config looks as following:

Host cvisionlab_inhouse.gitlab.com
    HostName gitlab.cvisionlab.com
    User ermolaev
    PreferredAuthentications publickey
    IdentityFile ~/.ssh/work
    IdentitiesOnly yes

ermolaev · July 5, 2024, 3:45pm

Actually I run this command from the server to which I used to connect the following command:

ssh -A -o "UserKnownHostsFile /dev/null" -o "StrictHostKeyChecking no" <server_name> -p <port on which docker is running>

dberenbaum · July 8, 2024, 2:25pm

It looks like the scmrepo package was not properly handling empty password fields. Added a fix for that in Handle missing password by dberenbaum · Pull Request #369 · iterative/scmrepo · GitHub, although I’m wondering whether it is expected for the password field to be empty here. Can you try to upgrade scmrepo with pip install --upgrade scmrepo?

ermolaev · July 8, 2024, 9:36pm

Thanks you a lot! Now it works!

But, unfortunatelly, I’ve faced with the other problem. During push I’ve got the following error:

2024-07-08 23:37:47,730 DEBUG: failed to load ('1c', 'bddf98ac99bbf716816b36811e3fb6.dir') from storage local (/data/projects/radml/DVC_CACHE/) - [Errno 2] 
No such file or directory: '/data/projects/radml/DVC_CACHE//1c/bddf98ac99bbf716816b36811e3fb6.dir'                                                          
Traceback (most recent call last):                                                                                                                          
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc_data/index/index.py", line 611, in _load_from_storage                           
    _load_from_object_storage(trie, entry, storage)                                                                                                         
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc_data/index/index.py", line 547, in _load_from_object_storage                    
    obj = Tree.load(storage.odb, root_entry.hash_info, hash_name=storage.odb.hash_name)                                                                     
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc_data/hashfile/tree.py", line 193, in load                                       
    with obj.fs.open(obj.path, "r") as fobj:                                                                                                                
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc_objects/fs/base.py", line 324, in open                                          
    return self.fs.open(path, mode=mode, **kwargs)                                                                                                          
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc_objects/fs/local.py", line 131, in open                                         
    return open(path, mode=mode, encoding=encoding)  # noqa: SIM115                                                                                         
FileNotFoundError: [Errno 2] No such file or directory: '/data/projects/radml/DVC_CACHE//1c/bddf98ac99bbf716816b36811e3fb6.dir'

And this file is really not in cache. This file is from other pipeline, from other part of the repo and it’s ok for me. I don’t know why dvc exp push rf_sgm_v6.01 command tries to push files far away from the pipeline to which this exp is related.

Also there is other error below. I don’t know is it connected to the first one and will push finish ok if the last error won’t happen:

...
FileNotFoundError: [Errno 2] No such file or directory: '/data/projects/radml/DVC_CACHE//1c/bddf98ac99bbf716816b36811e3fb6.dir'                             
                                                                                                                                                            
2024-07-08 23:37:47,747 DEBUG: Preparing to transfer data from '/data/projects/radml/DVC_CACHE/' to 'gdrive://1GkI-Tgwdi1r2oat1E0wxFG3sXc6oL2UL'            
2024-07-08 23:37:47,747 DEBUG: Preparing to collect status from '1GkI-Tgwdi1r2oat1E0wxFG3sXc6oL2UL'                                                         
2024-07-08 23:37:47,748 DEBUG: Collecting status from '1GkI-Tgwdi1r2oat1E0wxFG3sXc6oL2UL'                                                                   
2024-07-08 23:37:47,751 DEBUG: Querying 1 oids via object_exists                                                                                            
2024-07-08 23:37:48,793 DEBUG: Querying 12 oids via object_exists                                                                                           
2024-07-08 23:37:52,411 DEBUG: Estimated remote size: 256 files                                                                                             
2024-07-08 23:37:52,413 DEBUG: Querying 45 oids via traverse                                                                                                
Pushing                                                                                                                                                     
2024-07-08 23:37:52,859 DEBUG: Studio token not found.                                                                                                      
Pushed experiment rf_sgm_v6.01 to Git remote 'origin'.                                                                                                      
2024-07-08 23:37:52,914 ERROR: failed to push cache: <HttpError 400 when requesting https://www.googleapis.com/drive/v2/files returned "The query is too com
plex.". Details: "[{'message': 'The query is too complex.', 'domain': 'global', 'reason': 'queryTooComplex', 'location': 'q', 'locationType': 'parameter'}]"
>                                                                                                                                                           
Traceback (most recent call last):                                                                                                                          
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/pydrive2/files.py", line 84, in _GetList                                            
    self.auth.service.files()                                                                                                                               
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/googleapiclient/http.py", line 938, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 400 when requesting https://www.googleapis.com/drive/v2/files returned "The query is too complex.". Details: "[
{'message': 'The query is too complex.', 'domain': 'global', 'reason': 'queryTooComplex', 'location': 'q', 'locationType': 'parameter'}]">

During handling of the above exception, another exception occurred:
Traceback (most recent call last):                                                                                                                 [44/1848]
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc/repo/experiments/push.py", line 126, in push                                    
    result["uploaded"] = _push_cache(repo, pushed_refs_info, **kwargs)                                                                                      
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc/repo/experiments/push.py", line 182, in _push_cache                             
    return repo.push(                                                                                                                                       
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc/repo/__init__.py", line 58, in wrapper                                          
    return f(repo, *args, **kwargs)                                                                                                                         
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc/repo/push.py", line 147, in push                                                
    push_transferred, push_failed = ipush(                                                                                                                  
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc_data/index/push.py", line 76, in push
    result = transfer(
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc_data/hashfile/transfer.py", line 203, in transfer
    status = compare_status(
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc_data/hashfile/status.py", line 179, in compare_status
    dest_exists, dest_missing = status( 
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc_data/hashfile/status.py", line 151, in status
    exists.update(odb.oids_exist(hashes, jobs=jobs, progress=pbar.callback))
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc_objects/db.py", line 454, in oids_exist
    return list(oids & set(wrap_iter(remote_oids, callback)))
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc_objects/db.py", line 35, in wrap_iter
    for index, item in enumerate(iterable, start=1):
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc_objects/db.py", line 346, in _list_oids_traverse
    yield from self._list_oids(prefixes=traverse_prefixes, jobs=jobs)
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc_objects/db.py", line 250, in _list_oids
    for path in self._list_prefixes(prefixes=prefixes, jobs=jobs):
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc_objects/db.py", line 225, in _list_prefixes
    yield from self.fs.find(paths, batch_size=jobs, prefix=prefix)
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc_objects/fs/base.py", line 529, in find
    yield from self.fs.find(path)
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/pydrive2/fs/spec.py", line 490, in find
    for item in self._gdrive_list_ids(query_ids):
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/funcy/decorators.py", line 47, in wrapper
    return deco(call, *dargs, **dkwargs)
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/funcy/flow.py", line 99, in retry
    return call()
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/funcy/decorators.py", line 68, in __call__
    return self._func(*self._args, **self._kwargs)
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/pydrive2/fs/spec.py", line 308, in <lambda>
    get_list = _gdrive_retry(lambda: next(file_list, None))
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/pydrive2/apiattr.py", line 150, in __next__
    result = self._GetList()
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/pydrive2/auth.py", line 85, in _decorated
    return decoratee(self, *args, **kwargs)
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/pydrive2/files.py", line 89, in _GetList
    raise ApiRequestError(error)
pydrive2.files.ApiRequestError: <HttpError 400 when requesting https://www.googleapis.com/drive/v2/files returned "The query is too complex.". Details: "[{'
message': 'The query is too complex.', 'domain': 'global', 'reason': 'queryTooComplex', 'location': 'q', 'locationType': 'parameter'}]">

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc/cli/__init__.py", line 211, in main
    ret = cmd.do_run()
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc/cli/command.py", line 27, in do_run
    return self.run()
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc/commands/experiments/push.py", line 55, in run
    result = self.repo.experiments.push(
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc/repo/experiments/__init__.py", line 364, in push
    return push(self.repo, *args, **kwargs)
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc/repo/__init__.py", line 58, in wrapper
    return f(repo, *args, **kwargs)
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc/repo/scm_context.py", line 143, in run
    return method(repo, *args, **kw)
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc/repo/experiments/push.py", line 134, in push
    raise UploadError("failed to push cache", result) from e
dvc.repo.experiments.push.UploadError: failed to push cache

2024-07-08 23:37:52,922 DEBUG: Analytics is enabled.
2024-07-08 23:37:52,974 DEBUG: Trying to spawn ['daemon', 'analytics', '/tmp/tmpny78mjkd', '-v']
2024-07-08 23:37:52,982 DEBUG: Spawned ['daemon', 'analytics', '/tmp/tmpny78mjkd', '-v'] with pid 706

dberenbaum · July 9, 2024, 11:25am

dvc exp push is basically git push && dvc push, so it will push anything in your project, just like git won’t only push your current directory. If you need more granularity, you could do dvc exp push --no-cache and then dvc push only the targets you want pushed.

ermolaev · July 11, 2024, 2:56pm

I have run this command, but it looks like it didn’t push any targets in the remote cache:

$ dvc exp push origin rf_sgm_v6.01 -v --no-cache
2024-07-11 17:09:57,717 DEBUG: v3.51.2 (pip), CPython 3.10.12 on Linux-6.8.0-35-generic-x86_64-with-glibc2.35
2024-07-11 17:09:57,717 DEBUG: command: /home/ermolaev/projects/radml/venv/bin/dvc exp push origin rf_sgm_v6.01 -v --no-cache
2024-07-11 17:09:57,904 DEBUG: git push experiment ['refs/exps/f6/0bc58f3f1ee9bbe7842f164d360270fc677032/rf_sgm_v6.01:refs/exps/f6/0bc58f3f1ee9bbe7842f164d360270fc677032/rf_sgm_v6.01'] -> 'origin'
2024-07-11 17:09:59,395 DEBUG: Studio token not found.                                                                                                      
Experiment rf_sgm_v6.01 is up to date on Git remote 'origin'.
To push cached outputs for this experiment to DVC remote storage,re-run this command without '--no-cache'.
2024-07-11 17:09:59,447 DEBUG: Analytics is enabled.
2024-07-11 17:09:59,488 DEBUG: Trying to spawn ['daemon', 'analytics', '/tmp/tmpy3w8oiof', '-v']
2024-07-11 17:09:59,493 DEBUG: Spawned ['daemon', 'analytics', '/tmp/tmpy3w8oiof', '-v'] with pid 6879

I’ve also tried --no-run-cache because I supposed this flag could help, but got the following error:

$ dvc exp push origin rf_sgm_v6.01 -v --no-run-cache                   
2024-07-11 17:23:36,273 DEBUG: v3.51.2 (pip), CPython 3.10.12 on Linux-6.8.0-35-generic-x86_64-with-glibc2.35                                               
2024-07-11 17:23:36,273 DEBUG: command: /home/ermolaev/projects/radml/venv/bin/dvc exp push origin rf_sgm_v6.01 -v --no-run-cache                           
2024-07-11 17:23:36,477 DEBUG: git push experiment ['refs/exps/f6/0bc58f3f1ee9bbe7842f164d360270fc677032/rf_sgm_v6.01:refs/exps/f6/0bc58f3f1ee9bbe7842f164d3
60270fc677032/rf_sgm_v6.01'] -> 'origin'                                                                                                                    
2024-07-11 17:23:38,032 DEBUG: dvc push experiment '[ExpRefInfo(baseline_sha='f60bc58f3f1ee9bbe7842f164d360270fc677032', name='rf_sgm_v6.01')]'             
Collecting                                                                                                                        |0.00 [00:00,    ?entry/s]
<...>
FileNotFoundError: [Errno 2] No such file or directory: '/data/projects/radml/DVC_CACHE//1c/bddf98ac99bbf716816b36811e3fb6.dir'                             
                                                                                                                                                            
2024-07-11 17:29:57,551 DEBUG: Preparing to transfer data from '/data/projects/radml/DVC_CACHE/' to 'gdrive://1GkI-Tgwdi1r2oat1E0wxFG3sXc6oL2UL'            
2024-07-11 17:29:57,551 DEBUG: Preparing to collect status from '1GkI-Tgwdi1r2oat1E0wxFG3sXc6oL2UL'                                                         
2024-07-11 17:29:57,552 DEBUG: Collecting status from '1GkI-Tgwdi1r2oat1E0wxFG3sXc6oL2UL'                                                                   
2024-07-11 17:29:57,555 DEBUG: Querying 1 oids via object_exists                                                                                            
2024-07-11 17:29:58,629 DEBUG: Querying 12 oids via object_exists                                                                                           
2024-07-11 17:30:01,965 DEBUG: Estimated remote size: 256 files                                                                                             
2024-07-11 17:30:01,967 DEBUG: Querying 45 oids via traverse                                                                                                
Pushing                                                                                                                                                     
2024-07-11 17:30:39,931 DEBUG: Studio token not found.                                                                                                      
Experiment rf_sgm_v6.01 is up to date on Git remote 'origin'.                                                                                               
2024-07-11 17:30:39,987 ERROR: failed to push cache: <HttpError 400 when requesting https://www.googleapis.com/drive/v2/files returned "The query is too com
plex.". Details: "[{'message': 'The query is too complex.', 'domain': 'global', 'reason': 'queryTooComplex', 'location': 'q', 'locationType': 'parameter'}]"
>                                                                                                                                                           
Traceback (most recent call last):                                                                                                                          
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/pydrive2/files.py", line 84, in _GetList                                            
    self.auth.service.files()                                                                                                                               
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper                       
    return wrapped(*args, **kwargs)                                                                                                                         
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/googleapiclient/http.py", line 938, in execute                                      
    raise HttpError(resp, content, uri=self.uri)                                                                                                            
googleapiclient.errors.HttpError: <HttpError 400 when requesting https://www.googleapis.com/drive/v2/files returned "The query is too complex.". Details: "[
{'message': 'The query is too complex.', 'domain': 'global', 'reason': 'queryTooComplex', 'location': 'q', 'locationType': 'parameter'}]">                  
                                                                                                                                                            
During handling of the above exception, another exception occurred:
Traceback (most recent call last):                                                                                                                          
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc/repo/experiments/push.py", line 126, in push                                    
    result["uploaded"] = _push_cache(repo, pushed_refs_info, **kwargs)                                                                                      
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc/repo/experiments/push.py", line 182, in _push_cache                             
    return repo.push(                                                                                                                                       
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc/repo/__init__.py", line 58, in wrapper                                          
    return f(repo, *args, **kwargs)                                                                                                                         
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc/repo/push.py", line 147, in push                                                
    push_transferred, push_failed = ipush(                                                                                                                  
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc_data/index/push.py", line 76, in push                                           
    result = transfer(                                                                                                                                      
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc_data/hashfile/transfer.py", line 203, in transfer                               
    status = compare_status(                                                                                                                                
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc_data/hashfile/status.py", line 179, in compare_status                           
    dest_exists, dest_missing = status(                                                                                                                     
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc_data/hashfile/status.py", line 151, in status                                   
    exists.update(odb.oids_exist(hashes, jobs=jobs, progress=pbar.callback))                                                                                
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc_objects/db.py", line 454, in oids_exist                                         
    return list(oids & set(wrap_iter(remote_oids, callback)))                                                                                               
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc_objects/db.py", line 35, in wrap_iter                                           
    for index, item in enumerate(iterable, start=1):                                                                                                        
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc_objects/db.py", line 346, in _list_oids_traverse                                
    yield from self._list_oids(prefixes=traverse_prefixes, jobs=jobs)                                                                                       
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc_objects/db.py", line 250, in _list_oids                                         
    for path in self._list_prefixes(prefixes=prefixes, jobs=jobs):                                                                                          
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc_objects/db.py", line 225, in _list_prefixes                                     
    yield from self.fs.find(paths, batch_size=jobs, prefix=prefix)
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc_objects/fs/base.py", line 529, in find
    yield from self.fs.find(path)
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/pydrive2/fs/spec.py", line 490, in find
    for item in self._gdrive_list_ids(query_ids):
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/funcy/decorators.py", line 47, in wrapper
    return deco(call, *dargs, **dkwargs)
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/funcy/flow.py", line 99, in retry
    return call()
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/funcy/decorators.py", line 68, in __call__
    return self._func(*self._args, **self._kwargs)
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/pydrive2/fs/spec.py", line 308, in <lambda>
    get_list = _gdrive_retry(lambda: next(file_list, None))
File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/pydrive2/apiattr.py", line 150, in __next__
    result = self._GetList()
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/pydrive2/auth.py", line 85, in _decorated
    return decoratee(self, *args, **kwargs)
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/pydrive2/files.py", line 89, in _GetList
    raise ApiRequestError(error)
pydrive2.files.ApiRequestError: <HttpError 400 when requesting https://www.googleapis.com/drive/v2/files returned "The query is too complex.". Details: "[{'
message': 'The query is too complex.', 'domain': 'global', 'reason': 'queryTooComplex', 'location': 'q', 'locationType': 'parameter'}]">

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc/cli/__init__.py", line 211, in main
    ret = cmd.do_run()
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc/cli/command.py", line 27, in do_run
    return self.run()
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc/commands/experiments/push.py", line 55, in run
    result = self.repo.experiments.push(
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc/repo/experiments/__init__.py", line 364, in push
    return push(self.repo, *args, **kwargs)
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc/repo/__init__.py", line 58, in wrapper
    return f(repo, *args, **kwargs)
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc/repo/scm_context.py", line 143, in run
    return method(repo, *args, **kw)
  File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc/repo/experiments/push.py", line 134, in push
    raise UploadError("failed to push cache", result) from e
dvc.repo.experiments.push.UploadError: failed to push cache

2024-07-11 17:30:39,990 DEBUG: Analytics is enabled.
2024-07-11 17:30:40,043 DEBUG: Trying to spawn ['daemon', 'analytics', '/tmp/tmpp2q20wf1', '-v']
2024-07-11 17:30:40,049 DEBUG: Spawned ['daemon', 'analytics', '/tmp/tmpp2q20wf1', '-v'] with pid 8052

It looks like that dvc push is ok with missed data anyway, but the final request to the gdrive is too complex. I’ve tried also the following command “dvc push” and got the same issue while pushing specific target finishes with ok status.

I don’t know how to interpret this message. What should I do? Could it be that repo contains too many targets to be pushed into gdrive (I’ve checked limits by size and count of files and it’s still ok)?

I also decided to try to push the same experiment on S3 and it’s finished with ok status:

$ dvc exp push origin rf_sgm_v6.01 -v --no-run-cache -r yadrive
<...>
2024-07-11 17:47:23,298 DEBUG: Preparing to transfer data from '/data/projects/radml/DVC_CACHE/files/md5' to 's3://cvisionrad-ml-data/files/md5'            
2024-07-11 17:47:23,298 DEBUG: Preparing to collect status from 'cvisionrad-ml-data/files/md5'                                                              
2024-07-11 17:47:23,299 DEBUG: Collecting status from 'cvisionrad-ml-data/files/md5'                                                                        
2024-07-11 17:47:23,300 DEBUG: Querying 214 oids via object_exists                                                                                          
2024-07-11 17:47:26,525 DEBUG: Indexing new .dir '6cba07d1ca2e93b30166b8bdc2e21a73.dir' with '2' nested files                                               
2024-07-11 17:47:26,689 DEBUG: 'cvisionrad-ml-data/files/md5/00' doesn't look like a cache file, skipping                                                   
2024-07-11 17:47:26,690 DEBUG: Estimated remote size: 4096 files                                                                                            
2024-07-11 17:47:26,691 DEBUG: Querying 1403 oids via traverse                                                                                              
2024-07-11 17:47:26,790 DEBUG: Preparing to collect status from '/data/projects/radml/DVC_CACHE/files/md5'                                                  
2024-07-11 17:47:26,790 DEBUG: Collecting status from '/data/projects/radml/DVC_CACHE/files/md5'                                                            
2024-07-11 17:47:26,830 WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:                                   
md5: ad7272d675a124f37f10bc398465ef1d                                                                                                                       
md5: d434b7e969ccea967016fcdbfd77eee6                                                                                                                       
md5: 7c3fe9919a5a653cc452dba9b49a8211                                                                                                                       
md5: c0ed4d8be64bd04b652f6d1605ad35b9.dir
<...>
2024-07-11 17:47:26,908 DEBUG: transfer dir: md5: 2f98661e28ec4d1a3cd6fb3ea63f9156.dir with 4 files
2024-07-11 17:48:33,659 DEBUG: transfer dir: md5: 61f81d4920dea83b745edd5aa482c00a.dir with 1 files                                                         
2024-07-11 17:48:33,734 DEBUG: directory 'md5: 61f81d4920dea83b745edd5aa482c00a.dir' contains missing files, skipping .dir file upload                      
2024-07-11 17:48:33,880 DEBUG: transfer dir: md5: 46ac83add73a3028cbadecee1d2a480d.dir with 4 files                                                         
2024-07-11 17:52:21,563 DEBUG: transfer dir: md5: eddc406f7c5257821fe9429e6cf1c55c.dir with 3 files                                                         
2024-07-11 17:52:22,036 DEBUG: transfer dir: md5: 06511bea29d83440a5e7ac3e34c573eb.dir with 3 files                                                         
2024-07-11 17:52:22,336 DEBUG: transfer dir: md5: 24900e46d96e4cc7c10097e02c3fa35e.dir with 6 files                                                         
2024-07-11 17:52:24,198 DEBUG: transfer dir: md5: 48772dedc99c75d126200daea53a9757.dir with 4 files                                                         
2024-07-11 17:53:12,348 DEBUG: transfer dir: md5: da5bcbd96cbc59c869b4e04a14445c5e.dir with 0 files                                                         
2024-07-11 17:53:12,350 DEBUG: directory 'md5: da5bcbd96cbc59c869b4e04a14445c5e.dir' contains missing files, skipping .dir file upload                      
2024-07-11 17:53:12,432 DEBUG: transfer dir: md5: a6e9497ab771d6f7b18b9a6f852038d9.dir with 0 files                                                         
2024-07-11 17:53:12,433 DEBUG: directory 'md5: a6e9497ab771d6f7b18b9a6f852038d9.dir' contains missing files, skipping .dir file upload                      
2024-07-11 17:53:12,523 DEBUG: transfer dir: md5: 1a6bae2f85bc9b30d446c91ac9602c32.dir with 4 files                                                         
2024-07-11 17:53:52,643 DEBUG: transfer dir: md5: f15aa5cafe956ac709f52ef977db2d16.dir with 4 files                                                         
2024-07-11 17:54:29,195 DEBUG: transfer dir: md5: 295a62e47c066006b3ff583c8c09148d.dir with 4 files                                                         
Pushing                                                                                                                                                     
  0%|          |Pushing to s3

P.S. I don’t provide full log because this command tries to push everything from the repo into S3 and it’s not what I’d like to do.

So, it looks like something is in pydrive2 package or somewhere near google drive, but I don’t know how to create the same conditions synthetically.

dberenbaum · July 11, 2024, 4:30pm

I think practically if you want to push only a few files to gdrive, you should do dvc exp push --no-cache followed by dvc push last_stage_of_your_pipeline, which will push the outputs of all stages in that pipeline.

As far as the error you are getting from google drive, you could try opening an issue on GitHub - iterative/PyDrive2: Google Drive API Python wrapper library. Maintained fork of PyDrive..

ermolaev · July 12, 2024, 7:31am

So, if I do the following

dvc exp run <...> --queue
dvc queue start
<exp takes some time>

dvc exp push <exp> --no-cache
dvc push dvc.yaml

Then DVC will take dvc.yaml and check its .lock version is ALL visible experiments? How this push will find my experiment if it’s in temp space, because it was queued?

ermolaev · July 12, 2024, 10:29am

Actually, I did this commands and didn’t get result that I expected. While google-drive doesn’t work correcly, I used S3 as remote.

So, starting from the begining. My case is a git-repo with several different ML tasks. Some ML tasks are really huge, more than 10Tb of data and I don’t want to work with it while I’m working on something much smaller.

I have several different yaml files describing different pipelines, we call them “scenarios” in our project. Each scenario is also pretty huge, more than 500+ lines of commands, that’s why we split them. Also we don’t need to have all stages together, it’s hard to cope with them in a such way.

So I run an experiment for one specific task - processing human ribs data. I run this experiment in a queue to left cloned repo clean for other tasks. Then I’d like to push experiment data when it’s finished. I run command dvc exp push <name> --no-cache and dvc push scenario/dvc.yaml. I think that the last command only pushed the data from the current repo state e.g. not from experiment dvc.lock, but from the current dvc.lock. And I proffed my theory when tried to apply experiment at my local machine (I used dvc exp branch because dvc exp apply is also very heavy, tries to get all .dvc files and can not be configured). I got dvc.lock for this experiment and all the data of an experiment is missing on the remote:

$ dvc status dvc.yaml -r gdrive
missing:            ../../data/frac_segmentation/qa/metrics/fractures_0124_seg_val_epoch_hit_accuracy_0.50_test.h5
missing:            ../../data/frac_segmentation/qa/metrics/fractures_1123_seg_val_epoch_hit_f1_0.50_val.h5
missing:            ../../data/frac_segmentation/qa/metrics/fractures_0124_seg_val_epoch_hit_accuracy_0.50_train.h5
missing:            ../../data/frac_segmentation/qa/metrics/fractures_0124_seg_val_epoch_hit_f1_0.50_train.h5

Then I decided to check - what if I apply experiment on the server on which experiment was run, push the data from its dvc.lock file and then try to pull again. And it works. So, for now I see my working pipeline as following:

Run experiment on machine#1
Push without cache on machine#1
Apply exp on machine#1
Push outs on on machine#1
Pull experiment without cache on on machine#2
Pull experiment data on machine#2 using changed .lock file

So, my questions are:

Is there any easier way to get the same result? Maybe I’ve missed something?
What is for --no-run-cache flag?
dvc exp apply tries to pull all the data from all .dvc files presenting in the repo, is it possible to localize it’s work? I used branch method, it’s very good, but I don’t like having a lot of branches, actually.
CLI of DVC is very friendly for huge repos when coping with .dvc files directly, but running experiments becomes pretty difficult. Can DVC CLI be extending with some flags like -R for dvc exp push? I think that pipeline described above can be simplified for my case, but I don’t see such arguments in the currect CLI.

dberenbaum · July 12, 2024, 11:49am

Is there any easier way to get the same result? Maybe I’ve missed something?

No, what you wrote makes sense for queued experiments if you don’t want to push everything for each experiment.

What is for --no-run-cache flag?

That’s related to pushing the run-cache, which doesn’t push outputs but a record of stages run so they aren’t re-run unnecessarily.

dvc exp apply tries to pull all the data from all .dvc files presenting in the repo, is it possible to localize it’s work? I used branch method, it’s very good, but I don’t like having a lot of branches, actually.

No, that level of granularity is not supported and would be quite complicated since it is performing git operations, which happen at the repo level.

CLI of DVC is very friendly for huge repos when coping with .dvc files directly, but running experiments becomes pretty difficult. Can DVC CLI be extending with some flags like -R for dvc exp push? I think that pipeline described above can be simplified for my case, but I don’t see such arguments in the currect CLI.

Unfortunately, it doesn’t fit cleanly into dvc exp push because again that is also pushing git references, which are at the repo level.

ermolaev · July 12, 2024, 1:06pm

Thank you!

I understand. One more idea of how it can be implemented - additional argument -R can define what paths are crucial to the expriment i.e. dvc exp run dvc.yaml --name <path> -R path1 -R path2 --queue. Then, when experiment will be pushed or applied or something else, dvc will use only those paths that were added on its creation if such arguments were passed. Or use the whole repo, exactly how it works now.

JFYI: I’ve created an issue for pydrive2. But since today I can’t push or pull anything from gdrive. I’ve tried service-acc, clean docker container, nothing helped. Actually I’m blocked. It seems like something has broken / changed at google API, I’m not sure. I also can’t find this google drive API error description.

Topic		Replies	Views
Dvc exp push origin fails to push to AWS CodeCommit Questions	0	50	April 12, 2024
Dvc exp push to AWS CodeCommit fails Questions	0	64	April 12, 2024
Facing issue on dvc push Questions	6	80	June 19, 2024
DVC push raises errno 95 Bug Reports	2	1192	April 28, 2022
Dvc push error: unexpected keyword argument Questions	2	774	September 21, 2021

Error in pushing experiments

Related Topics