# Dealing with large numbers of small files **URL:** \ **Category:** Questions\ **Created:** [April 23, 2020, 4:12pm UTC](https://discuss.dvc.org/t/dealing-with-large-numbers-of-small-files/362 "2020-04-23T16:12:30Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Seanny123](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.dvc.org/seanny123/32/60_2.png) [@Seanny123](https://discuss.dvc.org/u/Seanny123)\ **Post date:** [April 23, 2020, 4:12pm UTC](https://discuss.dvc.org/t/dealing-with-large-numbers-of-small-files/362/1 "2020-04-23T16:12:30Z")
A project I’m working on has many small files, which don’t work well with DVC. Instead, I zip up all the files and push them remotely. However, it would be nice to know if I’ve modified any of the files in the zipped folder and diff them with the remote. Am I supposed to be using a pipeline to accomplish this? Is this just a silly goal to have? ---
**Author:** ![pmrowla](https://avatars.discourse-cdn.com/v4/letter/p/7ab992/32.png) [@pmrowla](https://discuss.dvc.org/u/pmrowla)\ **Post date:** [April 24, 2020, 1:52am UTC](https://discuss.dvc.org/t/dealing-with-large-numbers-of-small-files/362/2 "2020-04-24T01:52:31Z")
Hi @Seanny123! DVC does not currently provide any way to compare the contents of a zip file between your local copy and the remote. However, the [next release](https://github.com/iterative/dvc/releases/tag/untagged-817d11944b3efb9afcf9) of DVC will include several optimizations that substantially improve performance for handling many small files. Ideally, you should be able to push your data directly (without using an intermediate zip file), and run `dvc status -c` to see which files differ between your local copy and the remote. It sounds like your data set has a large number of files, but only expect that a small percentage of them will have been modified each time you want to push to the remote? Performance in this case will be significantly improved in the upcoming release, and I would recommend upgrading DVC whenever the new version is released to see if the performance without using an intermediate zip meets your needs. ---
**Author:** ![Seanny123](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.dvc.org/seanny123/32/60_2.png) [@Seanny123](https://discuss.dvc.org/u/Seanny123)\ **Post date:** [April 24, 2020, 2:07am UTC](https://discuss.dvc.org/t/dealing-with-large-numbers-of-small-files/362/3 "2020-04-24T02:07:13Z")
Are these changes already in the master branch? Alternatively, could you please link me to the Pull Request where these changes are being prototyped? ---
**Author:** ![pmrowla](https://avatars.discourse-cdn.com/v4/letter/p/7ab992/32.png) [@pmrowla](https://discuss.dvc.org/u/pmrowla)\ **Post date:** [April 24, 2020, 2:22am UTC](https://discuss.dvc.org/t/dealing-with-large-numbers-of-small-files/362/4 "2020-04-24T02:22:04Z")
These changes are available in the master branch. If you are interested, [https://github.com/iterative/dvc/pull/3634](https://github.com/iterative/dvc/pull/3634) also contains some benchmarks (using an S3 remote) for the new/optimized behavior. ---
**Author:** ![Seanny123](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.dvc.org/seanny123/32/60_2.png) [@Seanny123](https://discuss.dvc.org/u/Seanny123)\ **Post date:** [May 15, 2020, 3:42pm UTC](https://discuss.dvc.org/t/dealing-with-large-numbers-of-small-files/362/5 "2020-05-15T15:42:26Z")
Those changes work great for adding many small files. However, when pushing them, they are still pushed individually when it would be helpful to zip them together before pushing. Is this “zip before pushing” functionality something I should be able to do with a pipeline? ---
**Author:** ![jorgeorpinel](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.dvc.org/jorgeorpinel/32/46_2.png) [@jorgeorpinel](https://discuss.dvc.org/u/jorgeorpinel)\ **Post date:** [May 18, 2020, 6:51pm UTC](https://discuss.dvc.org/t/dealing-with-large-numbers-of-small-files/362/6 "2020-05-18T18:51:22Z")
@Seanny123 that’s an existing feature request, please upvote it and feel to comment your case/support for it on Github: [https://github.com/iterative/dvc/issues/1239](https://github.com/iterative/dvc/issues/1239) Thanks! ---
**Author:** ![shcheklein](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.dvc.org/shcheklein/32/173_2.png) [@shcheklein](https://discuss.dvc.org/u/shcheklein)\ **Post date:** [May 19, 2020, 5:05pm UTC](https://discuss.dvc.org/t/dealing-with-large-numbers-of-small-files/362/7 "2020-05-19T17:05:38Z")
@Seanny123 could elaborate why do you think zipping them together work better in your case? To my mind - treating them individually should be better for two reasons: 1. We can upload in parallel in a very aggressive manner (`--jobs` options) 2. DVC can upload only delta (only new files) instead of the whole bundle every time. ---
**Author:** ![Seanny123](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.dvc.org/seanny123/32/60_2.png) [@Seanny123](https://discuss.dvc.org/u/Seanny123)\ **Post date:** [May 19, 2020, 6:00pm UTC](https://discuss.dvc.org/t/dealing-with-large-numbers-of-small-files/362/8 "2020-05-19T18:00:24Z")
I mostly just wanted to reduce the amount of time it took to upload a large file set of files. Specifically, the whole folder is 20GBs. However, this might be a case wherein I’m abusing DVC. Additionally, I haven’t checked if I’m using the `--jobs` option, so this might just be premature optimization on my end. ---
**Author:** ![shcheklein](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.dvc.org/shcheklein/32/173_2.png) [@shcheklein](https://discuss.dvc.org/u/shcheklein)\ **Post date:** [May 19, 2020, 11:48pm UTC](https://discuss.dvc.org/t/dealing-with-large-numbers-of-small-files/362/9 "2020-05-19T23:48:21Z")
20GB does not sound like an abuse. But I still hope that pushing/pulling them as a directory should be faster. DVC still takes times to check change in the directory before pushing/pulling and it might take a bit of time, but overall it should still be better experience, especially if expect removing/adding small amount of files to the dir.