DVC Heartbeat - Discord gems

Announcements
2

Discord gems from the April Heartbeat:

  • It supports Windows, Mac, Linux. Python 2 and 3.

  • No specific CPU or RAM requirements — it’s a lightweight command line tool and should be able run pretty much everywhere you can run Python.

  • It depends on a few Python libraries that it installs as dependencies (they are specified in the requirements.txt).

  • It does not depend on Git and theoretically could be run without any SCM. Running it on top of a Git repository however is recommended and gives you an ability to actually save history of datasets, models, etc (even though it does not put them into Git directly).

No server licenses for DVC. It is 100% free and open source.

There is no limit. None enforced by DVC itself. It depends on the size of your local or remote storages. You need to have some space available on S3, your SSH server or other storage you are using to keep these data files, models and their version, which you would like to store.

DVC figures out the pipeline by looking at the dependencies and outputs of the stages. For example, having the following:

$ dvc run -f download.dvc \
          -o joke.txt \
          "curl https://geek-jokes.sameerkumar.website/api > joke.txt"
$ dvc run -f duplicate.dvc \
          -d joke.txt \
          -o dulpicate.txt \
          "cat joke.txt joke.txt > duplicate.txt"

you will end up with two stages: download.dvc and duplicate.dvc. The download one will have joke.txt as an output . The duplicate one defined joke.txt as a dependency, as it is the same file. DVC detects that and creates a pipeline by joining those stages.

You can inspect the content of each stage file here (they are human readable).

Yes! It’s a frequent scenario for multiple repos to share remotes and even local cache. DVC file serves as a link to the actual data. If you add the same DVC file (e.g. data.dvc) to the new repo and do dvc pull -r remotename data.dvc- it will fetch data. You have to use dvc remote add first to specify the coordinates of the remote storage you would like to share in every project. Alternatively (check out the question below), you could use — global to specify a single default remote (and/or cache dir) per machine.

Use — global when you specify the remote settings. Then remote will be visible for all projects on the same machine. — global — saves remote configuration to the global config (e.g. ~/.config/dvc/config) instead of a per project one — .dvc/config. See more details here.

We would recommend to skim through our get started tutorial, to summarize the data versioning process of DVC:

  • You create stage (aka DVC) files by adding, importing files (dvc add / dvc import) , or run a command to generate files (dvc run — out file.csv “wget https://example.com/file.csv").

  • This stage files are tracked by git

  • You use git to retrieve previous stage files (e.g. git checkout v1.0)

  • Then use dvc checkout to retrieve all the files related by those stage files

All your files (with each different version) are stored in a .dvc/cache directory, that you sync with a remote file storage (for example, S3) using the dvc push or dvc pull commands (analogous to a git push / git pull, but instead of syncing your .git, you are syncing your .dvc directory)

on a remote repository (let’s say an S3 bucket).

If you need to move your dvc file somewhere, it is pretty easy, even if done manually:

$ mv my.dvc data/my.dvc
# and now open my.dvc with your favorite editor and change wdir in it to 'wdir: ../'.

This is an expected behaviour. DVC saves files under the name created from their checksum in order to prevent duplication. If you delete “pushed” file in your project directory and perform dvc pull, dvc will take care of pulling the file and renaming it to “original” name.

Below are some details about how DVC’s cache works, just to illustrate the logic. When you add a data source:

$ echo "foo" > data.txt
$ dvc add data.txt

It computes the (md5) checksum of the file and generates a DVC file with related information:


md5: 3bccbf004063977442029334c3448687
outs:
- cache: true
  md5: d3b07384d113edec49eaa6238ad5ff00
  metric: false
  path: data.txt
wdir: ..

The original file is moved to the cache and a link or copy (depending on your filesystem) is created to replace it on your working space:

.dvc/cache
└── d3
    └── b07384d113edec49eaa6238ad5ff00

Absolutely! There are three ways you could interact with DVC:

  1. Use subprocess to launch DVC;

  2. Use from dvc.main import main and use it with regular CLI logic like ret = main(‘add’, ‘foo’)

  3. Use our internal API (see dvc/repo and dvc/command in our source to get a grasp of it). It is not officially public yet, and we don’t have any special docs for it, but it is fairly stable and could definitely be used for a POC. We’ll add docs and all the official stuff for it in the not-so-distant future.

There are two options:

  1. Use dvc add to track models and/or input datasets. It should be enough if you use git commit on DVC files produced by dvc add. This is the very minimum you can get with DVC and it does not require using DVC run. Check the first part (up to the Pipelines/Add transformations section) of the DVC get started.

  2. You could use — no-exec in dvc run and then just dvc commit and git commit the results. That way you’ll get your DVC files with all the linkages, without having to actually run your commands through DVC.

1 Like