Unexpected "changed outs" on status, and "link type reflink is not available" on pull

Questions
1

I’m quite new to DVC and I’m still confued by some concepts. What I’m trying to do now is to understand the output of dvc status. This is what I see:

$ dvc status
somefolder/somefile.dvc:
        changed outs:
                deleted:            somefolder/somefile

What does “changed outs” mean here? I’ve just cloned this git repository, and I have not made any changes at all. Why would dvc claim that something has been deleted?

The .dvc file looks correct:

$ cat somefolder/somefile.dvc
outs:
- md5: c06520abe0140c72004dbe4494a78b23.dir
  size: 692847854
  nfiles: 8
  hash: md5
  path: somefile

This is how I’ve configured DVC:

$ cat .dvc/config
[cache]
    dir = /srv/shared-dvc-cache/
    shared = group
    type = hardlink
[core]
    remote = myremote
['remote "myremote"']
    url = gs://some-bucket/dvc

And I do see the md5 for somefolder/somefile.dvc in the cache directory (/srv/shared-dvc-cache/):

$ ls -ahl /srv/shared-dvc-cache/files/md5/c0/6520abe0140c72004dbe4494a78b23.dir
-r--r--r-- 1 stian stian 686 Aug  2 14:56 /srv/shared-dvc-cache/files/md5/c0/6520abe0140c72004dbe4494a78b23.dir

I can also find it in the bucket:

$ gsutil ls -ahl gs://some-bucket/dvc/files/md5/c0/6520abe0140c72004dbe4494a78b23.dir
     686 B  2023-08-03T07:01:49Z  gs://s/dvc/files/md5/c0/6520abe0140c72004dbe4494a78b23.dir#1691046109228732  metageneration=1
TOTAL: 1 objects, 686 bytes (686 B)

If I try to pull, then I get a weird error:

$ dvc pull -vv
2023-08-22 11:31:15,846 DEBUG: v3.10.1 (pip), CPython 3.10.12 on Linux-5.4.0-152-generic-x86_64-with-glibc2.31
2023-08-22 11:31:15,846 DEBUG: command: /home/stian/some_repo/venv/bin/dvc pull -vv
2023-08-22 11:31:15,846 TRACE: Namespace(quiet=0, verbose=2, cprofile=False, cprofile_dump=None, yappi=False, yappi_separate_threads=False, viztracer=False, viztracer_depth=None, viztracer_async=False, pdb=False, instrument=False, instrument_open=False, show_stack=False, cd='.', cmd='pull', jobs=None, targets=[], remote=None, all_branches=False, all_tags=False, all_commits=False, force=False, with_deps=False, recursive=False, run_cache=False, glob=False, allow_missing=False, func=<class 'dvc.commands.data_sync.CmdDataPull'>, parser=DvcParser(prog='dvc', usage=None, description='Data Version Control', formatter_class=<class 'argparse.RawTextHelpFormatter'>, conflict_handler='error', add_help=False))
2023-08-22 11:31:16,149 TRACE:     3.64 ms in collecting stages from /home/stian/some_repo
<redacted>
2023-08-22 11:31:16,376 DEBUG: Preparing to transfer data from 'some-bucket/dvc/files/md5' to '/srv/shared-dvc-cache/files/md5'
2023-08-22 11:31:16,376 DEBUG: Preparing to collect status from '/srv/shared-dvc-cache/files/md5'
2023-08-22 11:31:16,376 DEBUG: Collecting status from '/srv/shared-dvc-cache/files/md5'
2023-08-22 11:31:16,394 DEBUG: failed to create '/home/stian/some-repo/testdata/some-folder/some-file.txt' from '/srv/shared-dvc-cache/files/md5/3e/ee94fe51c2bc3978b48d0205b6c77d' - [Errno 95] no more link types left to try out: [Errno 1] Operation not permitted: '/srv/shared-dvc-cache/files/md5/3e/ee94fe51c2bc3978b48d0205b6c77d' -> '/home/stian/some-repo/testdata/some-folder/some-file.txt'
Traceback (most recent call last):
  File "/home/stian/some-repo/venv/lib/python3.10/site-packages/dvc_objects/fs/generic.py", line 250, in _try_links
    _link(link, from_fs, from_path, to_fs, to_path)
  File "/home/stian/some-repo/venv/lib/python3.10/site-packages/dvc_objects/fs/generic.py", line 62, in _link
    func(from_path, to_path)
  File "/home/stian/some-repo/venv/lib/python3.10/site-packages/dvc_objects/fs/base.py", line 381, in link
    return self.fs.link(from_info, to_info)
  File "/home/stian/some-repo/venv/lib/python3.10/site-packages/dvc_objects/fs/local.py", line 166, in link
    return system.hardlink(path1, path2)
  File "/home/stian/some-repo/venv/lib/python3.10/site-packages/dvc_objects/fs/system.py", line 32, in hardlink
    os.link(src, link_name)
PermissionError: [Errno 1] Operation not permitted: '/srv/shared-dvc-cache/files/md5/3e/ee94fe51c2bc3978b48d0205b6c77d' -> '/home/stian/some-repo/testdata/some-folder/some-file.txt'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/stian/some-repo/venv/lib/python3.10/site-packages/dvc_objects/fs/generic.py", line 308, in transfer
    _try_links(
  File "/home/stian/some-repo/venv/lib/python3.10/site-packages/dvc_objects/fs/generic.py", line 267, in _try_links
    raise OSError(errno.ENOTSUP, "no more link types left to try out") from error
OSError: [Errno 95] no more link types left to try out

2023-08-22 11:31:16,395 ERROR: unexpected error - list index out of range
Traceback (most recent call last):
  File "/home/stian/some-repo/venv/lib/python3.10/site-packages/dvc/cli/__init__.py", line 209, in main
    ret = cmd.do_run()
  File "/home/stian/some-repo/venv/lib/python3.10/site-packages/dvc/cli/command.py", line 26, in do_run
    return self.run()
  File "/home/stian/some-repo/venv/lib/python3.10/site-packages/dvc/commands/data_sync.py", line 31, in run
    stats = self.repo.pull(
  File "/home/stian/some-repo/venv/lib/python3.10/site-packages/dvc/repo/__init__.py", line 64, in wrapper
    return f(repo, *args, **kwargs)
  File "/home/stian/some-repo/venv/lib/python3.10/site-packages/dvc/repo/pull.py", line 43, in pull
    stats = self.checkout(
  File "/home/stian/some-repo/venv/lib/python3.10/site-packages/dvc/repo/__init__.py", line 64, in wrapper
    return f(repo, *args, **kwargs)
  File "/home/stian/some-repo/venv/lib/python3.10/site-packages/dvc/repo/checkout.py", line 184, in checkout
    apply(
  File "/home/stian/some-repo/venv/lib/python3.10/site-packages/dvc_data/index/checkout.py", line 351, in apply
    _create_files(
  File "/home/stian/some-repo/venv/lib/python3.10/site-packages/dvc_data/index/checkout.py", line 122, in _create_files
    transfer(
  File "/home/stian/some-repo/venv/lib/python3.10/site-packages/dvc_objects/fs/generic.py", line 296, in transfer
    if links[0] == "copy":
IndexError: list index out of range

2023-08-22 11:31:16,414 DEBUG: link type reflink is not available ([Errno 95] no more link types left to try out)
2023-08-22 11:31:16,414 DEBUG: Removing '/home/stian/.PKve26BjXmzCXh4ZsBJg8H.tmp'
2023-08-22 11:31:16,415 DEBUG: Removing '/home/stian/.PKve26BjXmzCXh4ZsBJg8H.tmp'
2023-08-22 11:31:16,415 DEBUG: Removing '/home/stian/.PKve26BjXmzCXh4ZsBJg8H.tmp'
2023-08-22 11:31:16,415 DEBUG: Removing '/srv/shared-dvc-cache/files/md5/.R67LoavoyLcfYk2qnVcXn3.tmp'
2023-08-22 11:31:16,421 DEBUG: Version info for developers:
DVC version: 3.10.1 (pip)
-------------------------
Platform: Python 3.10.12 on Linux-5.4.0-152-generic-x86_64-with-glibc2.31
Subprojects:
        dvc_data = 2.8.1
        dvc_objects = 0.24.1
        dvc_render = 0.5.3
        dvc_task = 0.3.0
        scmrepo = 1.1.0
Supports:
        gs (gcsfs = 2023.6.0),
        http (aiohttp = 3.8.5, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.8.5, aiohttp-retry = 2.8.3)
Config:
        Global: /home/stian/.config/dvc
        System: /etc/xdg/dvc
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/md2
Caches: local
Remotes: gs
Workspace directory: ext4 on /dev/md2
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/e42a95986bfa50153b451739105fc96b

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2023-08-22 11:31:16,423 DEBUG: Analytics is enabled.
2023-08-22 11:31:16,478 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmppwgh5izm']'
2023-08-22 11:31:16,479 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmppwgh5izm']'

Does this make any sense at all?

Here are the dvc package versions:

$ pip list | grep -i dvc
dvc                      3.10.1
dvc-data                 2.8.1
dvc-gs                   2.22.1
dvc-http                 2.30.2
dvc-objects              0.24.1
dvc-render               0.5.3
dvc-studio-client        0.11.0
dvc-task                 0.3.0
1 Like
2

This happens primarily because we can’t distinguish really if something was deleted or it’s a fresh repo clone. Can be “missing”, but I think it’s already overloaded also. So, I believe there was not better ideas. Please let us know if you have some other options in mind.

It seems there is a minor bug / ugly implementation. IndexError: list index out of range should not be happening. It doesn’t look like it’s critical or changing the outcome. It’s an internal implementation detail. I see that /srv/shared-dvc-cache/ is located on a different volume (mounted?). hardlink do not work in between different volumes, so you would need to use symlink type of cache for this to work.

From the user perspective OSError: [Errno 95] no more link types left to try out means that we can’t find a right link type that would work in this situation.

Please let me know if that works for you.

1 Like
3

I think it’s on the same volume:

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev             63G     0   63G   0% /dev
tmpfs            13G  1.8M   13G   1% /run
/dev/md2        875G  611G  220G  74% /
tmpfs            63G  8.0K   63G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            63G     0   63G   0% /sys/fs/cgroup
/dev/sda1       1.8T  1.5T  208G  88% /data
/dev/md1        485M  161M  299M  35% /boot
tmpfs            13G     0   13G   0% /run/user/1021
tmpfs            13G     0   13G   0% /run/user/1007

$ ll /srv/shared-dvc-cache/
total 12
drwxrwxrwx 3 root  root  4096 Aug  2 14:56 ./
drwxr-xr-x 3 root  root  4096 Aug  2 13:27 ../
drwxrwxrwx 3 stian stian 4096 Aug  2 14:56 files/

Or did I misunderstand?

We had to use hardlink because the processes that are using our data weren’t able to work with the symlinks.

4

Back to report that switching from using /srv/shared-dvc-cache/ as the cache directory to instead using /home/shared/dvc-cache solved the problem. Would still like to figure out what exactly caused it, though.

5

Back yet again to report that my fix of changing to that other directory didn’t work after all. I’m still getting these:

2023-08-23 13:11:53,763 DEBUG: failed to create '/opt/actions-runner/_work/my-repo/my-repo/some-data/some_file' from '/home/shared/dvc-cache/files/md5/88/712af9626d5cbba82007cf8e3f90b2' - [Errno 95] no more link types left to try out: [Errno 1] Operation not permitted: '/home/shared/dvc-cache/files/md5/88/712af9626d5cbba82007cf8e3f90b2' -> '/opt/actions-runner/_work/some-repo/some-repo/some-data/some_file'
Traceback (most recent call last):
  File "/opt/actions-runner/_work/some-repo/some-repo/venv/lib/python3.10/site-packages/dvc_objects/fs/generic.py", line 250, in _try_links
    _link(link, from_fs, from_path, to_fs, to_path)
  File "/opt/actions-runner/_work/some-repo/some-repo/venv/lib/python3.10/site-packages/dvc_objects/fs/generic.py", line 62, in _link
    func(from_path, to_path)
  File "/opt/actions-runner/_work/some-repo/some-repo/venv/lib/python3.10/site-packages/dvc_objects/fs/base.py", line 381, in link
    return self.fs.link(from_info, to_info)
  File "/opt/actions-runner/_work/some-repo/some-repo/venv/lib/python3.10/site-packages/dvc_objects/fs/local.py", line 166, in link
    return system.hardlink(path1, path2)
  File "/opt/actions-runner/_work/some-repo/some-repo/venv/lib/python3.10/site-packages/dvc_objects/fs/system.py", line 32, in hardlink
    os.link(src, link_name)
PermissionError: [Errno 1] Operation not permitted: '/home/shared/dvc-cache/files/md5/88/712af9626d5cbba82007cf8e3f90b2' -> '/opt/actions-runner/_work/some-repo/some-repo/some-data/some_file'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/actions-runner/_work/some-repo/some-repo/venv/lib/python3.10/site-packages/dvc_objects/fs/generic.py", line 308, in transfer
    _try_links(
  File "/opt/actions-runner/_work/some-repo/some-repo/venv/lib/python3.10/site-packages/dvc_objects/fs/generic.py", line 267, in _try_links
    raise OSError(errno.ENOTSUP, "no more link types left to try out") from error
OSError: [Errno 95] no more link types left to try out

2023-08-23 13:11:53,768 ERROR: unexpected error - list index out of range
Traceback (most recent call last):
  File "/opt/actions-runner/_work/some-repo/some-repo/venv/lib/python3.10/site-packages/dvc/cli/__init__.py", line 209, in main
    ret = cmd.do_run()
  File "/opt/actions-runner/_work/some-repo/some-repo/venv/lib/python3.10/site-packages/dvc/cli/command.py", line 26, in do_run
    return self.run()
  File "/opt/actions-runner/_work/some-repo/some-repo/venv/lib/python3.10/site-packages/dvc/commands/data_sync.py", line 31, in run
    stats = self.repo.pull(
  File "/opt/actions-runner/_work/some-repo/some-repo/venv/lib/python3.10/site-packages/dvc/repo/__init__.py", line 64, in wrapper
    return f(repo, *args, **kwargs)
  File "/opt/actions-runner/_work/some-repo/some-repo/venv/lib/python3.10/site-packages/dvc/repo/pull.py", line 43, in pull
    stats = self.checkout(
  File "/opt/actions-runner/_work/some-repo/some-repo/venv/lib/python3.10/site-packages/dvc/repo/__init__.py", line 64, in wrapper
    return f(repo, *args, **kwargs)
  File "/opt/actions-runner/_work/some-repo/some-repo/venv/lib/python3.10/site-packages/dvc/repo/checkout.py", line 184, in checkout
    apply(
  File "/opt/actions-runner/_work/some-repo/some-repo/venv/lib/python3.10/site-packages/dvc_data/index/checkout.py", line 351, in apply
    _create_files(
  File "/opt/actions-runner/_work/some-repo/some-repo/venv/lib/python3.10/site-packages/dvc_data/index/checkout.py", line 122, in _create_files
    transfer(
  File "/opt/actions-runner/_work/some-repo/some-repo/venv/lib/python3.10/site-packages/dvc_objects/fs/generic.py", line 296, in transfer
    if links[0] == "copy":
IndexError: list index out of range

2023-08-23 13:11:53,790 DEBUG: link type reflink is not available ([Errno 95] no more link types left to try out)
2023-08-23 13:11:53,790 DEBUG: Removing '/opt/actions-runner/_work/some-repo/.2BE82g6YfXN8CcBZLEd4B8.tmp'
2023-08-23 13:11:53,790 DEBUG: Removing '/opt/actions-runner/_work/some-repo/.2BE82g6YfXN8CcBZLEd4B8.tmp'
2023-08-23 13:11:53,790 DEBUG: Removing '/opt/actions-runner/_work/some-repo/.2BE82g6YfXN8CcBZLEd4B8.tmp'
2023-08-23 13:11:53,790 DEBUG: Removing '/home/shared/dvc-cache/files/md5/.GEVj4mn4hVF8ryVsUu3hAA.tmp'

I suspect that the files that I’ve added to the cache are not being set with the correct bits:

$ ll /home/shared/dvc-cache/files/md5/88/712af9626d5cbba82007cf8e3f90b2
-r--r--r-- 2 stian stian 3099922624 Aug 16 14:38 /home/shared/dvc-cache/files/md5/88/712af9626d5cbba82007cf8e3f90b2

This file was added by me doing a dvc push -r prod-data type of command. When my github actions runner then later, on the same machine, tries to access the file, maybe it’s not able to? The github actions runner is using the unix user “ghworker”, different from my own user “stian”. How should that be configured? I’m not very knowledgeable on unix users and permissions, but here’s some info:

The “ghworker” user is in these groups:

$ sudo su ghworker
$ groups
ghworker wheel docker

My own user “stian” is in these:

$ groups
stian wheel admin docker

Maybe I missed that step in the setup tutorial at How to Share a Cache Among Projects ? To add a new group and use that for the dvc config cache.shared group command?

6

It seems the documentation is out of date. The current value for files is indeed 0o444. I’ve created the PR to update it fix(config): update file mode in group shared cache by shcheklein · Pull Request #4796 · iterative/dvc.org · GitHub

Now, back to the issue.

I’ve tried to reproduce it locally (I’m using Mac), with this config:

[cache]
    dir = /tmp/test-cache
    shared = group
    type = hardlink

With a simple repo like this:

(.venv) √ Projects/test-cache % tree .
.
├── test.txt
└── test.txt.dvc

0 directories, 2 files

I can’t reproduce it.

Can you try it on a simpler repo? Can you also try to run ln /home/shared/dvc-cache/files/md5/88/712af9626d5cbba82007cf8e3f90b2 /opt/actions-runner/_work/some-repo/some-repo/some-data/some_file

Could you try to run it with a cache located inside /opt/actions-runner/ ? Could you also check the permissions of the /opt/actions-runner/_work/some-repo/some-repo/some-data directory?

7

It may well be my usecase which doesn’t work. We have multiple shared development machines and I would prefer to use a single dvc cache for all users, including our github actions runner user.

I found that using a cache dir in this location:

[cache]
    # Set the cache directory to be two levels up from this config file, which
    # means it'll be one level above the git repository.
    dir = ../../dvc-cache

works, but then the cache is not shared among users.

I could live with a solution like that for now, even though it takes a lot more space, but then I would prefer this modification:

[cache]
    # Set the cache directory to be in the users home directory
    dir = ~/dvc-cache

But that doesn’t seem to work. Then I get this error:

FileNotFoundError: [Errno 2] No such file or directory: '/opt/actions-runner/_work/some-repo/some-repo/.dvc/~/.gitignore'

Is there a way to make “the-users-home-directory/dvc-cache” work?

8

That should be fixed in the latest DVC version. Could you try to upgrade it?

works, but then the cache is not shared among users.

Your scenario (single shared cache) makes total sense. Something weird is happening with the links here.

including our github actions runner user

Could you clarify please? Is it something like self-hosted runner? How is it working (ww might have people in the team who have this setup to try and / or have experience).

Have you had a chance to try to create a link manually between those locations?

We have multiple shared development machines and I would prefer to use a single dvc cache for all users

If there are multiple dev machines and they all share the cache it means that that cache should be located on a separate volume on each of those machines, right? It means that to make hardlinks work you would have to put the home dir to the same mounted volume.

9

Thanks again for responding!

I was thinking that I had the latest version, but I see now that I have 3.10.1 and that the latest released is dvc · PyPI. I’ll try to upgrade!

My usecase is that I have many shared development machines, and some of them also act as self-hosted github actions runners. I want each of these machines to have their own shared DVC cache (shared amongst all users on the single machine). So no sharing between machines - only between users on a single machine.

Sorry, not yet, but I will try to reproduce it.