ERROR: unexpected error - [Errno 2] No such file or directory:

bdatko · November 6, 2022, 1:01am

Hi all,

I ran into an ERROR: unexpected error - [Errno 2] No such file or directory from running dvc run on my stage in my dvc.yaml. I am running into a similar error when I try to do dvc repro. I am not sure how to debug the error.

Full stacktrace from `dvc run --force -vvv ...`

$ dvc run --force -vvv -n split_state_and_fed -d data/source -d split_state_and_fed.py -o data/fed/LabeledRegulations_Fed.csv -o data/state/LabeledRegulations_State.csv python split_state_and_fed.py split_state_and_fed --csv_path=data/source --output_state=data/state/LabeledRegulations_State.csv --output_fed=data/fed/LabeledRegulations_Fed.csv
2022-11-05 21:00:28,578 TRACE: Namespace(cprofile=False, yappi=False, yappi_separate_threads=False, viztracer=False, viztracer_depth=None, cprofile_dump=None, pdb=False, instrument=False, instrument_open=False, show_stack=False, quiet=0, verbose=3, version=None, cd='.', cmd='run', name='split_state_and_fed', file=None, single_stage=False, force=True, deps=['data/source', 'split_state_and_fed.py'], params=[], outs=['data/fed/LabeledRegulations_Fed.csv', 'data/state/LabeledRegulations_State.csv'], outs_no_cache=[], checkpoints=[], external=False, outs_persist=[], outs_persist_no_cache=[], metrics=[], metrics_no_cache=[], plots=[], plots_no_cache=[], live=None, live_no_cache=None, live_no_summary=False, live_no_html=False, wdir=None, always_changed=False, desc=None, command=['python', 'split_state_and_fed.py', 'split_state_and_fed', '--csv_path=data/source', '--output_state=data/state/LabeledRegulations_State.csv', '--output_fed=data/fed/LabeledRegulations_Fed.csv'], no_exec=False, no_commit=False, no_run_cache=False, func=<class 'dvc.commands.run.CmdRun'>, parser=DvcParser(prog='dvc', usage=None, description='Data Version Control', formatter_class=<class 'argparse.RawTextHelpFormatter'>, conflict_handler='error', add_help=False))
2022-11-05 21:00:29,131 TRACE: params.yaml does not exist, it won't be used in parametrization
2022-11-05 21:00:29,132 TRACE: Context during resolution of stage split_state_and_fed:
{}
2022-11-05 21:00:29,133 DEBUG: Lockfile for 'dvc.yaml' not found
2022-11-05 21:00:29,139 TRACE:    41.75 ms in collecting stages from C:\Users\bd122850\OneDrive - The Bureau of National Affairs, Inc\Documents\projects\reports\tracker_healthcare_qa
2022-11-05 21:00:29,147 TRACE:     7.54 ms in collecting stages from C:\Users\bd122850\OneDrive - The Bureau of National Affairs, Inc\Documents\projects\reports\tracker_healthcare_qa\data
2022-11-05 21:00:29,150 TRACE:   958.90 mks in collecting stages from C:\Users\bd122850\OneDrive - The Bureau of National Affairs, Inc\Documents\projects\reports\tracker_healthcare_qa\src
2022-11-05 21:00:29,152 TRACE:    12.30 mks in collecting stages from C:\Users\bd122850\OneDrive - The Bureau of National Affairs, Inc\Documents\projects\reports\tracker_healthcare_qa\src\tracker_healthcare_qa
2022-11-05 21:00:29,154 TRACE:    17.80 mks in collecting stages from C:\Users\bd122850\OneDrive - The Bureau of National Affairs, Inc\Documents\projects\reports\tracker_healthcare_qa\tests
2022-11-05 21:00:29,336 TRACE: params.yaml does not exist, it won't be used in parametrization
2022-11-05 21:00:29,336 TRACE: Context during resolution of stage split_state_and_fed:
{}
2022-11-05 21:00:29,338 DEBUG: Lockfile for 'dvc.yaml' not found
2022-11-05 21:00:29,348 DEBUG: Removing output 'data\fed\LabeledRegulations_Fed.csv' of stage: 'split_state_and_fed'.
2022-11-05 21:00:29,349 DEBUG: Removing 'C:\Users\bd122850\OneDrive - The Bureau of National Affairs, Inc\Documents\projects\reports\tracker_healthcare_qa\data\fed\LabeledRegulations_Fed.csv'
2022-11-05 21:00:29,350 DEBUG: Removing output 'data\state\LabeledRegulations_State.csv' of stage: 'split_state_and_fed'.
2022-11-05 21:00:29,352 DEBUG: Removing 'C:\Users\bd122850\OneDrive - The Bureau of National Affairs, Inc\Documents\projects\reports\tracker_healthcare_qa\data\state\LabeledRegulations_State.csv'
2022-11-05 21:00:29,371 DEBUG: built tree 'object 24fdb62da3d871204e3c29da681a9939.dir'
2022-11-05 21:00:29,378 DEBUG: built tree 'object 24fdb62da3d871204e3c29da681a9939.dir'
2022-11-05 21:00:29,379 DEBUG: {'data\\source': 'modified'}
2022-11-05 21:00:29,387 DEBUG: built tree 'object 24fdb62da3d871204e3c29da681a9939.dir'
2022-11-05 21:00:29,398 DEBUG: built tree 'object 24fdb62da3d871204e3c29da681a9939.dir'
2022-11-05 21:00:29,424 DEBUG: built tree 'object 24fdb62da3d871204e3c29da681a9939.dir'
Running stage 'split_state_and_fed':
> python split_state_and_fed.py split_state_and_fed --csv_path=data/source --output_state=data/state/LabeledRegulations_State.csv --output_fed=data/fed/LabeledRegulations_Fed.csv
2022-11-05 21:00:33,363 DEBUG: built tree 'object 24fdb62da3d871204e3c29da681a9939.dir'
2022-11-05 21:00:33,400 DEBUG: Computed stage: 'split_state_and_fed' md5: 'bfb841b6a7ea23cf95f75c8b2a8e7ea3'
2022-11-05 21:00:33,406 DEBUG: built tree 'object 24fdb62da3d871204e3c29da681a9939.dir'
2022-11-05 21:00:33,412 DEBUG: built tree 'object 24fdb62da3d871204e3c29da681a9939.dir'
2022-11-05 21:00:33,440 ERROR: unexpected error - [Errno 2] No such file or directory: 'C:\\Users\\bd122850\\OneDrive - The Bureau of National Affairs, Inc\\Documents\\projects\\reports\\tracker_healthcare_qa\\.dvc\\cache\\runs\\5c\\5cfa45a43abf76c684266397388c048a50ede05c4e0ed9d7ccb8107c1ee70601\\e0d88e8d4dd7e19c58922c66af34a089d91cd70f419cbb28773bd22b72e5f967.hAv7gnwFPvcZY9cUZGJiHN'
------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\bd122850\Anaconda3\envs\tracker_healthcare_qa\lib\shutil.py", line 825, in move
    os.rename(src, real_dst)
FileNotFoundError: [WinError 3] The system cannot find the path specified: 'C:\\Users\\bd122850\\OneDrive - The Bureau of National Affairs, Inc\\Documents\\projects\\reports\\tracker_healthcare_qa\\.dvc\\cache\\runs\\5c\\5cfa45a43abf76c684266397388c048a50ede05c4e0ed9d7ccb8107c1ee70601\\tmpiy81j0vt' -> 'C:\\Users\\bd122850\\OneDrive - The Bureau of National Affairs, Inc\\Documents\\projects\\reports\\tracker_healthcare_qa\\.dvc\\cache\\runs\\5c\\5cfa45a43abf76c684266397388c048a50ede05c4e0ed9d7ccb8107c1ee70601\\e0d88e8d4dd7e19c58922c66af34a089d91cd70f419cbb28773bd22b72e5f967.hAv7gnwFPvcZY9cUZGJiHN'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\bd122850\Anaconda3\envs\tracker_healthcare_qa\lib\site-packages\dvc\cli\__init__.py", line 185, in main
    ret = cmd.do_run()
  File "C:\Users\bd122850\Anaconda3\envs\tracker_healthcare_qa\lib\site-packages\dvc\cli\command.py", line 22, in do_run
    return self.run()
  File "C:\Users\bd122850\Anaconda3\envs\tracker_healthcare_qa\lib\site-packages\dvc\commands\run.py", line 47, in run
    self.repo.run(**kwargs)
  File "C:\Users\bd122850\Anaconda3\envs\tracker_healthcare_qa\lib\site-packages\dvc\repo\__init__.py", line 48, in wrapper
    return f(repo, *args, **kwargs)
  File "C:\Users\bd122850\Anaconda3\envs\tracker_healthcare_qa\lib\site-packages\dvc\repo\scm_context.py", line 156, in run
    return method(repo, *args, **kw)
  File "C:\Users\bd122850\Anaconda3\envs\tracker_healthcare_qa\lib\site-packages\dvc\repo\run.py", line 33, in run
    stage.run(no_commit=no_commit, run_cache=run_cache)
  File "C:\Users\bd122850\Anaconda3\envs\tracker_healthcare_qa\lib\site-packages\funcy\decorators.py", line 45, in wrapper
    return deco(call, *dargs, **dkwargs)
  File "C:\Users\bd122850\Anaconda3\envs\tracker_healthcare_qa\lib\site-packages\dvc\stage\decorators.py", line 43, in rwlocked
    return call()
  File "C:\Users\bd122850\Anaconda3\envs\tracker_healthcare_qa\lib\site-packages\funcy\decorators.py", line 66, in __call__
    return self._func(*self._args, **self._kwargs)
  File "C:\Users\bd122850\Anaconda3\envs\tracker_healthcare_qa\lib\site-packages\dvc\stage\__init__.py", line 574, in run
    self.save(allow_missing=allow_missing)
  File "C:\Users\bd122850\Anaconda3\envs\tracker_healthcare_qa\lib\site-packages\dvc\stage\__init__.py", line 486, in save
    self.repo.stage_cache.save(self)
  File "C:\Users\bd122850\Anaconda3\envs\tracker_healthcare_qa\lib\site-packages\dvc\stage\cache.py", line 181, in save
    self.repo.odb.local.move(tmp, path)
  File "C:\Users\bd122850\Anaconda3\envs\tracker_healthcare_qa\lib\site-packages\dvc_data\hashfile\db\local.py", line 34, in move
    super().move(from_info, to_info)
  File "C:\Users\bd122850\Anaconda3\envs\tracker_healthcare_qa\lib\site-packages\dvc_objects\db.py", line 91, in move
    self.fs.move(from_info, to_info)
  File "C:\Users\bd122850\Anaconda3\envs\tracker_healthcare_qa\lib\site-packages\dvc_objects\fs\base.py", line 371, in mv
    self.fs.mv(from_info, to_info)
  File "C:\Users\bd122850\Anaconda3\envs\tracker_healthcare_qa\lib\site-packages\dvc_objects\fs\local.py", line 107, in mv
    move(path1, path2)
  File "C:\Users\bd122850\Anaconda3\envs\tracker_healthcare_qa\lib\site-packages\dvc_objects\fs\utils.py", line 64, in move
    shutil.move(src, tmp)
  File "C:\Users\bd122850\Anaconda3\envs\tracker_healthcare_qa\lib\shutil.py", line 845, in move
    copy_function(src, real_dst)
  File "C:\Users\bd122850\Anaconda3\envs\tracker_healthcare_qa\lib\shutil.py", line 444, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "C:\Users\bd122850\Anaconda3\envs\tracker_healthcare_qa\lib\shutil.py", line 266, in copyfile
    with open(dst, 'wb') as fdst:
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\bd122850\\OneDrive - The Bureau of National Affairs, Inc\\Documents\\projects\\reports\\tracker_healthcare_qa\\.dvc\\cache\\runs\\5c\\5cfa45a43abf76c684266397388c048a50ede05c4e0ed9d7ccb8107c1ee70601\\e0d88e8d4dd7e19c58922c66af34a089d91cd70f419cbb28773bd22b72e5f967.hAv7gnwFPvcZY9cUZGJiHN'
------------------------------------------------------------
2022-11-05 21:00:33,798 DEBUG: link type reflink is not available ([Errno 129] no more link types left to try out)
2022-11-05 21:00:33,798 DEBUG: Removing 'C:\Users\bd122850\OneDrive - The Bureau of National Affairs, Inc\Documents\projects\reports\.69MpunVkddMa4TZdFpmKaC.tmp'
2022-11-05 21:00:33,800 DEBUG: Removing 'C:\Users\bd122850\OneDrive - The Bureau of National Affairs, Inc\Documents\projects\reports\.69MpunVkddMa4TZdFpmKaC.tmp'
2022-11-05 21:00:33,803 DEBUG: link type symlink is not available ([WinError 1314] A required privilege is not held by the client: 'C:/Users/bd122850/OneDrive - The Bureau of National Affairs, Inc/Documents/projects/reports/tracker_healthcare_qa/.dvc/cache/.ZBKsiaxpxhhyMox828VWLX.tmp' -> 'C:/Users/bd122850/OneDrive - The Bureau of National Affairs, Inc/Documents/projects/reports/.69MpunVkddMa4TZdFpmKaC.tmp')
2022-11-05 21:00:33,803 DEBUG: Removing 'C:\Users\bd122850\OneDrive - The Bureau of National Affairs, Inc\Documents\projects\reports\.69MpunVkddMa4TZdFpmKaC.tmp'
2022-11-05 21:00:33,804 DEBUG: Removing 'C:\Users\bd122850\OneDrive - The Bureau of National Affairs, Inc\Documents\projects\reports\tracker_healthcare_qa\.dvc\cache\.ZBKsiaxpxhhyMox828VWLX.tmp'
2022-11-05 21:00:33,854 DEBUG: Version info for developers:
DVC version: 2.33.2 (pip)
---------------------------------
Platform: Python 3.9.13 on Windows-10-10.0.19043-SP0
Subprojects:
        dvc_data = 0.25.2
        dvc_objects = 0.12.2
        dvc_render = 0.0.12
        dvc_task = 0.1.4
        dvclive = 1.0
        scmrepo = 0.1.3
Supports:
        http (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.8.3, aiohttp-retry = 2.8.3)
Cache types: hardlink
Cache directory: NTFS on C:\
Caches: local
Remotes: local
Workspace directory: NTFS on C:\
Repo: dvc, git

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2022-11-05 21:00:33,868 DEBUG: Analytics is enabled.
2022-11-05 21:00:34,127 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', 'C:\\Users\\bd122850\\AppData\\Local\\Temp\\tmpsxs91nti']'
2022-11-05 21:00:34,157 DEBUG: Spawned '['daemon', '-q', 'analytics', 'C:\\Users\\bd122850\\AppData\\Local\\Temp\\tmpsxs91nti']'

DVC doctor output:

DVC version: 2.33.2 (pip)
---------------------------------
Platform: Python 3.9.13 on Windows-10-10.0.19043-SP0
Subprojects:
        dvc_data = 0.25.2
        dvc_objects = 0.12.2
        dvc_render = 0.0.12
        dvc_task = 0.1.4
        dvclive = 1.0
        scmrepo = 0.1.3
Supports:
        http (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.8.3, aiohttp-retry = 2.8.3)
Cache types: hardlink
Cache directory: NTFS on C:\
Caches: local
Remotes: local
Workspace directory: NTFS on C:\
Repo: dvc, git

bdatko · November 7, 2022, 2:03pm

Working through stack traces on the discord channel. Answering the question about if the parent directory is being created:

The parent directory in the path for the python command used in the stage of the DVC yaml is present. The python script does not produce any error and I see the expected output
The parent directory 5373a268696130b39cc91eb0a84fc6aa3d319c5f5457738204f311227a48d67b.fBqeujqRpR6mBP8JnoCst7 mentioned in the stack trace does not exist. Here is the full path: .dvc\\cache\\runs\\61\\6110fe992548e0f3a21a5db003fb5dad206d04196519039f39ff70f58540c9d7\\5373a268696130b39cc91eb0a84fc6aa3d319c5f5457738204f311227a48d67b.fBqeujqRpR6mBP8JnoCst7'

Below is a screenshot of the project directory highlighting the missing parent.

skshetry · November 7, 2022, 2:16pm

Thanks. The last part in the path is a file which will be created when open() succeeds.

What happens if you try in python REPL yourself with the following command?

with open("<path>", mode="wb") as fdst:
    pass

Does that above work?

skshetry · November 7, 2022, 2:25pm

Okay, I wonder if this is due to long path issue.
Could you please enable long path? See this guide:
https://dvc.org/doc/user-guide/how-to/run-dvc-on-windows#enable-long-folderfile-paths.

bdatko · November 7, 2022, 2:29pm

Okay, I wonder if this is due to long path issue.
Could you please enable long path? See this guide:

It could be this, but I don’t have admin rights to edit the group policy on my machine.

bdatko · November 7, 2022, 2:31pm

The specified path in stack traces ends with a directory that is not there, but I try:

with open("<path>", mode="wb") as fdst:
    pass

Where <path> is the parent of the file missing within the project directory, I get a PermissionError: [Errno 13] Permission denied:

bdatko · November 8, 2022, 5:55pm

I wasn’t able to verify if the long path issue was the exact reason, but I moved the project out my OneDrive folder into another directory. This fixed the ERROR: unexpected error - [Errno 2] No such file or directory when running dvc repro

stevejonn · April 26, 2023, 10:18pm

By specifying the name of a file as “filename.ext” when opening it with the open() function, you are indicating that the file is located in the current working directory. This type of file path is referred to as a relative path.

file = open('filename.ext') //relative path

In the above code, you are not giving the full path to a file to the open() function, just its name - a relative path. The error “FileNotFoundError: [Errno 2] No such file or directory” is telling you that there is no file of that name in the working directory. So, try using the exact, or absolute path.

file = open(r'C:\path\to\your\filename.ext') //absolute path

In the above code, all of the information needed to locate the file is contained in the path string - absolute path.

If the user fails to provide a full path to the file (which, on Unix-like systems, means a path starting with a slash), Python interprets the file path relatively to the current working directory. Typically, the current working directory is the directory where the program was launched from. For this to work correctly, the directory containing the Python executable must be listed in the PATH environment variable, which contains a set of directories that the system searches when searching for executable files. Regardless, if your Python script file and the input data file are not located in the same directory, you will need to specify either a relative path between them or use an absolute path for one of the files.

Topic		Replies	Views
What to do if the file is not found in the repo? Questions	1	429	July 23, 2022
No such lock file error Questions	1	15	December 11, 2024
First run of DVC - getting a "failed to reproduce" error Questions	7	6001	April 11, 2019
Unexpected error [Errno2] No storage files available Questions	2	40	October 9, 2024
DVC push gives "ERROR: unexpected error - [Errno 21] Is a directory" Bug Reports	3	548	November 21, 2023

ERROR: unexpected error - [Errno 2] No such file or directory:

Related topics