Problem with dvc list

Questions
1

Hi. New to dvc and trying to get things to work with a little sample project.

I was able to initialize git and dvc and use an S3 bucket as my data store. I committed and pushed my code to Bitbucket (my company’s internal Bitbucket instance) and I see everything there but the data (as expected). I am able to clone the git repo and pull the data from S3 as well.

I am trying to run dvc list to see the data tracked by dvc.

I’ve tried running the following commands unsuccessfully:
dvc list https://sourcecode.mycompany.com/scm/asx-jbbd/dvc
dvc list https://sourcecode.mycompany.com/scm/asx-jbbd dvc

ERROR: failed to list ‘ssh://cesc@sourcecode.mycompany.com:3268/asx-jbbd’ - Failed to clone repo ‘ssh://cesc@sourcecode.mycompany.com:3268/asx-jbbd’ to ‘C:\Users\cesc\AppData\Local\Temp\1\tmp43z389r6dvc-clone’

What am I doing wrong?

2

@cesc What is the url for your git repository?
https://sourcecode.mycompany.com/scm/asx-jbbd/dvc ?

3

To see the repository in the browser I go to https://sourcecode.mycompany.com/projects/ASX-JBBD/repos/dvc/browse

4

The forum is blocking me from posting links and my previous post was hidden automatically.

Let me try again… To view the repo in the browser I go to sourcecode.__.com/projects/ASX-JBBD/repos/dvc/browse

5

git remote -v
origin https://sourcecode..com/scm/asx-jbbd/dvc.git (fetch)
origin https://sourcecode..com/scm/asx-jbbd/dvc.git (push)

In case this helps.

6

Huh, I don’t seem to be able to unblock your message, Ill see, maybe I am not an admin.
Anyway, please try running dvc list https://sourcecode.com/scm/asx-jbbd/dvc.git

7

dvc list https://sourcecode..com/scm/asx-jbbd/dvc.git
ERROR: failed to list ‘url above’ - Failed to clone repo 'https://sourcecode..com/scm/asx-jbbd/dvc.git’ to ‘C:\Users\cesc\AppData\Local\Temp\1\tmpj22ewz1kdvc-clone’

8

Are you able to git clone it?

9

git clone https://sourcecode.com/scm/asx-jbbd/dvc.git
Cloning into ‘dvc’…
Username for first part of the URL above up to .com’: cesc
Password for ‘https://cesc@sourcecode.com’:
remote: Enumerating objects: 25, done.
remote: Counting objects: 100% (25/25), done.
remote: Compressing objects: 100% (20/20), done.
remote: Total 25 (delta 6), reused 0 (delta 0), pack-reused 0
Receiving objects: 100% (25/25), done.
Resolving deltas: 100% (6/6), done.

10

list should have no problem clonging that.
Could you post the result for dvc version and dvc list https://sourcecode.com/scm/asx-jbbd/dvc.git -v? Not that the latter might contain information you don’t want to share. If so, strip it or don’t post it at all, we will try to workaround that.

11

dvc version
DVC version: 2.9.3 (exe)

Platform: Python 3.8.10 on Windows-10-10.0.18363-SP0
Supports:
azure (adlfs = 2022.2.0, knack = 0.9.0, azure-identity = 1.7.1),
gdrive (pydrive2 = 1.10.0),
gs (gcsfs = 2022.1.0),
hdfs (fsspec = 2022.1.0, pyarrow = 7.0.0),
webhdfs (fsspec = 2022.1.0),
http (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
https (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
s3 (s3fs = 2022.1.0, boto3 = 1.20.24),
ssh (sshfs = 2021.11.2),
oss (ossfs = 2021.8.0),
webdav (webdav4 = 0.9.4),
webdavs (webdav4 = 0.9.4)

12

And when you are running dvc list -v - can you find a stact trace of an error? It should not contain anything regarding your setup.

13

LINK below refers to https://sourcecode.com/scm/asx-jbbd/dvc.git (I can only post 2 links, which makes it super hard to explain things…)

dvc list LINK -v
2022-03-03 11:30:37,869 DEBUG: Creating external repo LINK@None
2022-03-03 11:30:37,870 DEBUG: erepo: git clone ‘LINK’ to a temporary dir
2022-03-03 11:30:40,845 ERROR: failed to list ‘LINK’ - Failed to clone repo ‘LINK’ to ‘C:\Users\cesc\AppData\Local\Temp\tmpw7o1__fmdvc-clone’

Traceback (most recent call last):
File “scmrepo\git\backend\gitpython.py”, line 185, in clone
File “git\repo\base.py”, line 1148, in clone_from
File “git\repo\base.py”, line 1078, in _clone
File “git\cmd.py”, line 175, in handle_process_output
File “git\util.py”, line 386, in finalize_process
File “git\cmd.py”, line 501, in wait
git.exc.GitCommandError: Cmd(‘git’) failed due to: exit code(128)
cmdline: git clone -v --no-single-branch --progress LINK C:\Users\cesc\AppData\Local\Temp\tmpw7o1__fmdvc-clone

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “dvc\scm.py”, line 104, in clone
File “scmrepo\git_init_.py”, line 121, in clone
File “scmrepo\git\backend\gitpython.py”, line 190, in clone
scmrepo.exceptions.CloneError: Failed to clone repo ‘LINK’ to ‘C:\Users\cesc\AppData\Local\Temp\tmpw7o1__fmdvc-clone’

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “dvc\command\ls_init_.py”, line 30, in run
File “dvc\repo\ls.py”, line 31, in ls
File “contextlib.py”, line 113, in enter
File “dvc\external_repo.py”, line 35, in external_repo
File “dvc\external_repo.py”, line 155, in _cached_clone
File “funcy\decorators.py”, line 45, in wrapper
File “funcy\flow.py”, line 274, in wrap_with
File “funcy\decorators.py”, line 66, in call
File “dvc\external_repo.py”, line 220, in _clone_default_branch
File “dvc\scm.py”, line 106, in clone
dvc.scm.CloneError: Failed to clone repo ‘LINK’ to ‘C:\Users\cesc\AppData\Local\Temp\tmpw7o1__fmdvc-clone’

2022-03-03 11:30:40,896 DEBUG: Analytics is enabled.
2022-03-03 11:30:42,564 DEBUG: Trying to spawn ‘[‘daemon’, ‘-q’, ‘analytics’, ‘C:\Users\cesc\AppData\Local\Temp\tmp2ix9tked’]’
2022-03-03 11:30:42,760 DEBUG: Spawned ‘[‘daemon’, ‘-q’, ‘analytics’, ‘C:\Users\cesc\AppData\Local\Temp\tmp2ix9tked’]’

14

Ok it seems that its git failing itself.
Can you try running:
git clone -v --no-single-branch --progress LINK C:\Users\cesc\AppData\Local\Temp\tmpw7o1__fmdvc-clone

BTW if you want to post stack trace/code surrender the text with triple ` sign

Just like: ```code goes here```

15 
git clone -v --no-single-branch --progress https://sourcecode.com/scm/asx-jbbd/dvc.git C:\Users\cesc\AppData\Local\Temp\tmp1in8ic2kdvc-clone
Cloning into 'C:\Users\cesc\AppData\Local\Temp\tmp1in8ic2kdvc-clone'...
Username for 'https://sourcecode.com': cesc
Password for 'https://cesc@sourcecode.com':
POST git-upload-pack (174 bytes)
POST git-upload-pack (262 bytes)
remote: Enumerating objects: 25, done.
remote: Counting objects: 100% (25/25), done.
remote: Compressing objects: 100% (20/20), done.
remote: Total 25 (delta 6), reused 0 (delta 0), pack-reused 0
Receiving objects: 100% (25/25), done.
Resolving deltas: 100% (6/6), done.
16

This must be a problem with dvc and Windows. For the heck of it, I tried on Linux and it’s working as expected…

17

Well then, I will try to reproduce it on windows, I need to dig up my windows machine :slight_smile:

18

@cecs
I’ve been trying to reproduce the problem on my windows. Tried both cmd and powershell, installed DVC 2.9.3 as an exe package (from dvc version I assessed that is what you did, right?). I am still unable to reproduce.

Could you update to latest version of DVC and check if its still failing?

19

I will give it a shot and let you know.

20

Isn’t version 2.9.3 the latest? I downloaded it from the Download button in the home page.