ERROR: unexpected error - [Errno 2] No such file or directory:

Hi all,

I ran into an ERROR: unexpected error - [Errno 2] No such file or directory from running dvc run on my stage in my dvc.yaml. I am running into a similar error when I try to do dvc repro. I am not sure how to debug the error.

Full stacktrace from `dvc run --force -vvv ...`
$ dvc run --force -vvv -n split_state_and_fed -d data/source -d split_state_and_fed.py -o data/fed/LabeledRegulations_Fed.csv -o data/state/LabeledRegulations_State.csv python split_state_and_fed.py split_state_and_fed --csv_path=data/source --output_state=data/state/LabeledRegulations_State.csv --output_fed=data/fed/LabeledRegulations_Fed.csv
2022-11-05 21:00:28,578 TRACE: Namespace(cprofile=False, yappi=False, yappi_separate_threads=False, viztracer=False, viztracer_depth=None, cprofile_dump=None, pdb=False, instrument=False, instrument_open=False, show_stack=False, quiet=0, verbose=3, version=None, cd='.', cmd='run', name='split_state_and_fed', file=None, single_stage=False, force=True, deps=['data/source', 'split_state_and_fed.py'], params=[], outs=['data/fed/LabeledRegulations_Fed.csv', 'data/state/LabeledRegulations_State.csv'], outs_no_cache=[], checkpoints=[], external=False, outs_persist=[], outs_persist_no_cache=[], metrics=[], metrics_no_cache=[], plots=[], plots_no_cache=[], live=None, live_no_cache=None, live_no_summary=False, live_no_html=False, wdir=None, always_changed=False, desc=None, command=['python', 'split_state_and_fed.py', 'split_state_and_fed', '--csv_path=data/source', '--output_state=data/state/LabeledRegulations_State.csv', '--output_fed=data/fed/LabeledRegulations_Fed.csv'], no_exec=False, no_commit=False, no_run_cache=False, func=<class 'dvc.commands.run.CmdRun'>, parser=DvcParser(prog='dvc', usage=None, description='Data Version Control', formatter_class=<class 'argparse.RawTextHelpFormatter'>, conflict_handler='error', add_help=False))
2022-11-05 21:00:29,131 TRACE: params.yaml does not exist, it won't be used in parametrization
2022-11-05 21:00:29,132 TRACE: Context during resolution of stage split_state_and_fed:
{}
2022-11-05 21:00:29,133 DEBUG: Lockfile for 'dvc.yaml' not found
2022-11-05 21:00:29,139 TRACE:    41.75 ms in collecting stages from C:\Users\bd122850\OneDrive - The Bureau of National Affairs, Inc\Documents\projects\reports\tracker_healthcare_qa
2022-11-05 21:00:29,147 TRACE:     7.54 ms in collecting stages from C:\Users\bd122850\OneDrive - The Bureau of National Affairs, Inc\Documents\projects\reports\tracker_healthcare_qa\data
2022-11-05 21:00:29,150 TRACE:   958.90 mks in collecting stages from C:\Users\bd122850\OneDrive - The Bureau of National Affairs, Inc\Documents\projects\reports\tracker_healthcare_qa\src
2022-11-05 21:00:29,152 TRACE:    12.30 mks in collecting stages from C:\Users\bd122850\OneDrive - The Bureau of National Affairs, Inc\Documents\projects\reports\tracker_healthcare_qa\src\tracker_healthcare_qa
2022-11-05 21:00:29,154 TRACE:    17.80 mks in collecting stages from C:\Users\bd122850\OneDrive - The Bureau of National Affairs, Inc\Documents\projects\reports\tracker_healthcare_qa\tests
2022-11-05 21:00:29,336 TRACE: params.yaml does not exist, it won't be used in parametrization
2022-11-05 21:00:29,336 TRACE: Context during resolution of stage split_state_and_fed:
{}
2022-11-05 21:00:29,338 DEBUG: Lockfile for 'dvc.yaml' not found
2022-11-05 21:00:29,348 DEBUG: Removing output 'data\fed\LabeledRegulations_Fed.csv' of stage: 'split_state_and_fed'.
2022-11-05 21:00:29,349 DEBUG: Removing 'C:\Users\bd122850\OneDrive - The Bureau of National Affairs, Inc\Documents\projects\reports\tracker_healthcare_qa\data\fed\LabeledRegulations_Fed.csv'
2022-11-05 21:00:29,350 DEBUG: Removing output 'data\state\LabeledRegulations_State.csv' of stage: 'split_state_and_fed'.
2022-11-05 21:00:29,352 DEBUG: Removing 'C:\Users\bd122850\OneDrive - The Bureau of National Affairs, Inc\Documents\projects\reports\tracker_healthcare_qa\data\state\LabeledRegulations_State.csv'
2022-11-05 21:00:29,371 DEBUG: built tree 'object 24fdb62da3d871204e3c29da681a9939.dir'
2022-11-05 21:00:29,378 DEBUG: built tree 'object 24fdb62da3d871204e3c29da681a9939.dir'
2022-11-05 21:00:29,379 DEBUG: {'data\\source': 'modified'}
2022-11-05 21:00:29,387 DEBUG: built tree 'object 24fdb62da3d871204e3c29da681a9939.dir'
2022-11-05 21:00:29,398 DEBUG: built tree 'object 24fdb62da3d871204e3c29da681a9939.dir'
2022-11-05 21:00:29,424 DEBUG: built tree 'object 24fdb62da3d871204e3c29da681a9939.dir'
Running stage 'split_state_and_fed':
> python split_state_and_fed.py split_state_and_fed --csv_path=data/source --output_state=data/state/LabeledRegulations_State.csv --output_fed=data/fed/LabeledRegulations_Fed.csv
2022-11-05 21:00:33,363 DEBUG: built tree 'object 24fdb62da3d871204e3c29da681a9939.dir'
2022-11-05 21:00:33,400 DEBUG: Computed stage: 'split_state_and_fed' md5: 'bfb841b6a7ea23cf95f75c8b2a8e7ea3'
2022-11-05 21:00:33,406 DEBUG: built tree 'object 24fdb62da3d871204e3c29da681a9939.dir'
2022-11-05 21:00:33,412 DEBUG: built tree 'object 24fdb62da3d871204e3c29da681a9939.dir'
2022-11-05 21:00:33,440 ERROR: unexpected error - [Errno 2] No such file or directory: 'C:\\Users\\bd122850\\OneDrive - The Bureau of National Affairs, Inc\\Documents\\projects\\reports\\tracker_healthcare_qa\\.dvc\\cache\\runs\\5c\\5cfa45a43abf76c684266397388c048a50ede05c4e0ed9d7ccb8107c1ee70601\\e0d88e8d4dd7e19c58922c66af34a089d91cd70f419cbb28773bd22b72e5f967.hAv7gnwFPvcZY9cUZGJiHN'
------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\bd122850\Anaconda3\envs\tracker_healthcare_qa\lib\shutil.py", line 825, in move
    os.rename(src, real_dst)
FileNotFoundError: [WinError 3] The system cannot find the path specified: 'C:\\Users\\bd122850\\OneDrive - The Bureau of National Affairs, Inc\\Documents\\projects\\reports\\tracker_healthcare_qa\\.dvc\\cache\\runs\\5c\\5cfa45a43abf76c684266397388c048a50ede05c4e0ed9d7ccb8107c1ee70601\\tmpiy81j0vt' -> 'C:\\Users\\bd122850\\OneDrive - The Bureau of National Affairs, Inc\\Documents\\projects\\reports\\tracker_healthcare_qa\\.dvc\\cache\\runs\\5c\\5cfa45a43abf76c684266397388c048a50ede05c4e0ed9d7ccb8107c1ee70601\\e0d88e8d4dd7e19c58922c66af34a089d91cd70f419cbb28773bd22b72e5f967.hAv7gnwFPvcZY9cUZGJiHN'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\bd122850\Anaconda3\envs\tracker_healthcare_qa\lib\site-packages\dvc\cli\__init__.py", line 185, in main
    ret = cmd.do_run()
  File "C:\Users\bd122850\Anaconda3\envs\tracker_healthcare_qa\lib\site-packages\dvc\cli\command.py", line 22, in do_run
    return self.run()
  File "C:\Users\bd122850\Anaconda3\envs\tracker_healthcare_qa\lib\site-packages\dvc\commands\run.py", line 47, in run
    self.repo.run(**kwargs)
  File "C:\Users\bd122850\Anaconda3\envs\tracker_healthcare_qa\lib\site-packages\dvc\repo\__init__.py", line 48, in wrapper
    return f(repo, *args, **kwargs)
  File "C:\Users\bd122850\Anaconda3\envs\tracker_healthcare_qa\lib\site-packages\dvc\repo\scm_context.py", line 156, in run
    return method(repo, *args, **kw)
  File "C:\Users\bd122850\Anaconda3\envs\tracker_healthcare_qa\lib\site-packages\dvc\repo\run.py", line 33, in run
    stage.run(no_commit=no_commit, run_cache=run_cache)
  File "C:\Users\bd122850\Anaconda3\envs\tracker_healthcare_qa\lib\site-packages\funcy\decorators.py", line 45, in wrapper
    return deco(call, *dargs, **dkwargs)
  File "C:\Users\bd122850\Anaconda3\envs\tracker_healthcare_qa\lib\site-packages\dvc\stage\decorators.py", line 43, in rwlocked
    return call()
  File "C:\Users\bd122850\Anaconda3\envs\tracker_healthcare_qa\lib\site-packages\funcy\decorators.py", line 66, in __call__
    return self._func(*self._args, **self._kwargs)
  File "C:\Users\bd122850\Anaconda3\envs\tracker_healthcare_qa\lib\site-packages\dvc\stage\__init__.py", line 574, in run
    self.save(allow_missing=allow_missing)
  File "C:\Users\bd122850\Anaconda3\envs\tracker_healthcare_qa\lib\site-packages\dvc\stage\__init__.py", line 486, in save
    self.repo.stage_cache.save(self)
  File "C:\Users\bd122850\Anaconda3\envs\tracker_healthcare_qa\lib\site-packages\dvc\stage\cache.py", line 181, in save
    self.repo.odb.local.move(tmp, path)
  File "C:\Users\bd122850\Anaconda3\envs\tracker_healthcare_qa\lib\site-packages\dvc_data\hashfile\db\local.py", line 34, in move
    super().move(from_info, to_info)
  File "C:\Users\bd122850\Anaconda3\envs\tracker_healthcare_qa\lib\site-packages\dvc_objects\db.py", line 91, in move
    self.fs.move(from_info, to_info)
  File "C:\Users\bd122850\Anaconda3\envs\tracker_healthcare_qa\lib\site-packages\dvc_objects\fs\base.py", line 371, in mv
    self.fs.mv(from_info, to_info)
  File "C:\Users\bd122850\Anaconda3\envs\tracker_healthcare_qa\lib\site-packages\dvc_objects\fs\local.py", line 107, in mv
    move(path1, path2)
  File "C:\Users\bd122850\Anaconda3\envs\tracker_healthcare_qa\lib\site-packages\dvc_objects\fs\utils.py", line 64, in move
    shutil.move(src, tmp)
  File "C:\Users\bd122850\Anaconda3\envs\tracker_healthcare_qa\lib\shutil.py", line 845, in move
    copy_function(src, real_dst)
  File "C:\Users\bd122850\Anaconda3\envs\tracker_healthcare_qa\lib\shutil.py", line 444, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "C:\Users\bd122850\Anaconda3\envs\tracker_healthcare_qa\lib\shutil.py", line 266, in copyfile
    with open(dst, 'wb') as fdst:
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\bd122850\\OneDrive - The Bureau of National Affairs, Inc\\Documents\\projects\\reports\\tracker_healthcare_qa\\.dvc\\cache\\runs\\5c\\5cfa45a43abf76c684266397388c048a50ede05c4e0ed9d7ccb8107c1ee70601\\e0d88e8d4dd7e19c58922c66af34a089d91cd70f419cbb28773bd22b72e5f967.hAv7gnwFPvcZY9cUZGJiHN'
------------------------------------------------------------
2022-11-05 21:00:33,798 DEBUG: link type reflink is not available ([Errno 129] no more link types left to try out)
2022-11-05 21:00:33,798 DEBUG: Removing 'C:\Users\bd122850\OneDrive - The Bureau of National Affairs, Inc\Documents\projects\reports\.69MpunVkddMa4TZdFpmKaC.tmp'
2022-11-05 21:00:33,800 DEBUG: Removing 'C:\Users\bd122850\OneDrive - The Bureau of National Affairs, Inc\Documents\projects\reports\.69MpunVkddMa4TZdFpmKaC.tmp'
2022-11-05 21:00:33,803 DEBUG: link type symlink is not available ([WinError 1314] A required privilege is not held by the client: 'C:/Users/bd122850/OneDrive - The Bureau of National Affairs, Inc/Documents/projects/reports/tracker_healthcare_qa/.dvc/cache/.ZBKsiaxpxhhyMox828VWLX.tmp' -> 'C:/Users/bd122850/OneDrive - The Bureau of National Affairs, Inc/Documents/projects/reports/.69MpunVkddMa4TZdFpmKaC.tmp')
2022-11-05 21:00:33,803 DEBUG: Removing 'C:\Users\bd122850\OneDrive - The Bureau of National Affairs, Inc\Documents\projects\reports\.69MpunVkddMa4TZdFpmKaC.tmp'
2022-11-05 21:00:33,804 DEBUG: Removing 'C:\Users\bd122850\OneDrive - The Bureau of National Affairs, Inc\Documents\projects\reports\tracker_healthcare_qa\.dvc\cache\.ZBKsiaxpxhhyMox828VWLX.tmp'
2022-11-05 21:00:33,854 DEBUG: Version info for developers:
DVC version: 2.33.2 (pip)
---------------------------------
Platform: Python 3.9.13 on Windows-10-10.0.19043-SP0
Subprojects:
        dvc_data = 0.25.2
        dvc_objects = 0.12.2
        dvc_render = 0.0.12
        dvc_task = 0.1.4
        dvclive = 1.0
        scmrepo = 0.1.3
Supports:
        http (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.8.3, aiohttp-retry = 2.8.3)
Cache types: hardlink
Cache directory: NTFS on C:\
Caches: local
Remotes: local
Workspace directory: NTFS on C:\
Repo: dvc, git

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2022-11-05 21:00:33,868 DEBUG: Analytics is enabled.
2022-11-05 21:00:34,127 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', 'C:\\Users\\bd122850\\AppData\\Local\\Temp\\tmpsxs91nti']'
2022-11-05 21:00:34,157 DEBUG: Spawned '['daemon', '-q', 'analytics', 'C:\\Users\\bd122850\\AppData\\Local\\Temp\\tmpsxs91nti']'

DVC doctor output:

DVC version: 2.33.2 (pip)
---------------------------------
Platform: Python 3.9.13 on Windows-10-10.0.19043-SP0
Subprojects:
        dvc_data = 0.25.2
        dvc_objects = 0.12.2
        dvc_render = 0.0.12
        dvc_task = 0.1.4
        dvclive = 1.0
        scmrepo = 0.1.3
Supports:
        http (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.8.3, aiohttp-retry = 2.8.3)
Cache types: hardlink
Cache directory: NTFS on C:\
Caches: local
Remotes: local
Workspace directory: NTFS on C:\
Repo: dvc, git

Working through stack traces on the discord channel. Answering the question about if the parent directory is being created:

  1. The parent directory in the path for the python command used in the stage of the DVC yaml is present. The python script does not produce any error and I see the expected output
  2. The parent directory 5373a268696130b39cc91eb0a84fc6aa3d319c5f5457738204f311227a48d67b.fBqeujqRpR6mBP8JnoCst7 mentioned in the stack trace does not exist. Here is the full path: .dvc\\cache\\runs\\61\\6110fe992548e0f3a21a5db003fb5dad206d04196519039f39ff70f58540c9d7\\5373a268696130b39cc91eb0a84fc6aa3d319c5f5457738204f311227a48d67b.fBqeujqRpR6mBP8JnoCst7'

Below is a screenshot of the project directory highlighting the missing parent.

Thanks. The last part in the path is a file which will be created when open() succeeds.

What happens if you try in python REPL yourself with the following command?

with open("<path>", mode="wb") as fdst:
    pass

Does that above work?

Okay, I wonder if this is due to long path issue.
Could you please enable long path? See this guide:
https://dvc.org/doc/user-guide/how-to/run-dvc-on-windows#enable-long-folderfile-paths.

Okay, I wonder if this is due to long path issue.
Could you please enable long path? See this guide:

It could be this, but I don’t have admin rights to edit the group policy on my machine.

The specified path in stack traces ends with a directory that is not there, but I try:

with open("<path>", mode="wb") as fdst:
    pass

Where <path> is the parent of the file missing within the project directory, I get a PermissionError: [Errno 13] Permission denied:

I wasn’t able to verify if the long path issue was the exact reason, but I moved the project out my OneDrive folder into another directory. This fixed the ERROR: unexpected error - [Errno 2] No such file or directory when running dvc repro