# Best practice for handling large data **URL:** \ **Category:** Questions\ **Created:** [April 13, 2021, 9:07am UTC](https://discuss.dvc.org/t/best-practice-for-handling-large-data/721 "2021-04-13T09:07:39Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![nsorros](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.dvc.org/nsorros/32/104_2.png) [@nsorros](https://discuss.dvc.org/u/nsorros)\ **Post date:** [April 13, 2021, 9:07am UTC](https://discuss.dvc.org/t/best-practice-for-handling-large-data/721/1 "2021-04-13T09:07:39Z")
I am working with a 25GB raw dataset which when processed to be prepared for training produces another 10+GB file. Using `dvc push` and `dvc pull` with s3 takes a significant amount of time to move data through the network. Is there a way to keep the data only in s3? or use a flag that chooses when to pull or push but that knows that a stage has run and does not need reruning? What would be the recommended way of working with the data both locally and using a remote? ---
**Author:** ![jorgeorpinel](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.dvc.org/jorgeorpinel/32/46_2.png) [@jorgeorpinel](https://discuss.dvc.org/u/jorgeorpinel)\ **Post date:** [April 14, 2021, 5:48pm UTC](https://discuss.dvc.org/t/best-practice-for-handling-large-data/721/2 "2021-04-14T17:48:40Z")
Hey @nsorros, Can you please clarify why you need to push/pull data constantly? Remote storage is an “optional” (thought very common) feature, meaning that you can do your entire pipeline locally and only sync to/from remote once (in a while). > Is there a way to keep the data only in s3? DVC can track data externally without ever downloading it, yes. But it’s a very advanced/tricky feature that we need to redesign so not something I would definitely recommend… See [External Data](https://dvc.org/doc/user-guide/managing-external-data). > or use a flag that chooses when to pull or push but that knows that a stage has run This is the part I’m not getting. When do you want push/pull to happen? If you can share more details on your workflow we can probably give you a better answer. > recommended way of working with the data both locally and using a remote? To recap, remote storage is meant for sharing and backing up raw data, results, models, etc. But if you’re using a network connection to sync it then it’s probably not a good idea to rely on it for day-to-day pipeline executions involving very large files. You may be interested in things like [https://dvc.org/doc/use-cases/shared-development-server](https://dvc.org/doc/use-cases/shared-development-server) or [add](https://dvc.org/doc/command-reference/add#example-transfer-to-remote-storage) but again, depends on your goals. Thanks ---
**Author:** ![nsorros](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.dvc.org/nsorros/32/104_2.png) [@nsorros](https://discuss.dvc.org/u/nsorros)\ **Post date:** [April 15, 2021, 3:50pm UTC](https://discuss.dvc.org/t/best-practice-for-handling-large-data/721/3 "2021-04-15T15:50:05Z")
Sorry for the confusion, what I am thinking is a flow where the code and data lives in 2 places, an EC2 instance and locally and synced through GitHub and DVC. This is our current setup for most projects. We use an EC2 instance to train when needed, otherwise we train locally. So in this setup sometimes I develop locally but want to train in the EC2 instance. Ideally I do not want to run dvc pull and get all data locally, I want to be able to exclude some due to the size of the files. And same for other members of the team, if they go to my project and want to test a pipeline works, I do not want them to have to download 25-100GB of data with dvc pull. Ideally I want to include a sample dataset that they can use in a sample pipeline or something. I might use that as well for local development. If that flow does not make sense and a better solution is to never get the data through dvc pull both locally and in the ec2 instance and always treat them as external data then I guess that is my answer. This problem should be more prominent as the data and the models get larger I would assume. Hope this makes sense, I understand it might be confusing, very curious to hear more about how you would think about it and I am happy to provide more context. ---
**Author:** ![jorgeorpinel](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.dvc.org/jorgeorpinel/32/46_2.png) [@jorgeorpinel](https://discuss.dvc.org/u/jorgeorpinel)\ **Post date:** [April 15, 2021, 6:46pm UTC](https://discuss.dvc.org/t/best-practice-for-handling-large-data/721/4 "2021-04-15T18:46:42Z")
> [@nsorros](#): > > We use an EC2 instance to train when needed, otherwise we train locally. Got it. > [@nsorros](#): > > I do not want to run dvc pull and get all data locally, I want to be able to exclude some You can give specific targets to `dvc pull`. With the `--glob` option you can even use wildcards and other patterns. See [pull](https://dvc.org/doc/command-reference/pull) (Same for `dvc push`) > [@nsorros](#): > > Ideally I want to include a sample dataset that they can use in a sample pipeline or something. You can make a separate Git branch in your repo, put the sample there, `dvc commit/repro` to update DVC metafiles, and `git+dvc push` it. They can then get that branch and `dvc pull` will download the smaller sample. Does that make sense or am I misunderstanding the situation? Thanks ---
**Author:** ![nsorros](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.dvc.org/nsorros/32/104_2.png) [@nsorros](https://discuss.dvc.org/u/nsorros)\ **Post date:** [April 16, 2021, 7:29am UTC](https://discuss.dvc.org/t/best-practice-for-handling-large-data/721/5 "2021-04-16T07:29:39Z")
Hi Jorge, That makes sense. I guess I missed the glob option which seems to do what I am looking for. Interesting idea about the separate branch, thanks for the suggestion. Super helpful and again thanks for the quick response. ---
**Author:** ![jorgeorpinel](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.dvc.org/jorgeorpinel/32/46_2.png) [@jorgeorpinel](https://discuss.dvc.org/u/jorgeorpinel)\ **Post date:** [April 16, 2021, 6:37pm UTC](https://discuss.dvc.org/t/best-practice-for-handling-large-data/721/6 "2021-04-16T18:37:16Z")
Glad I could help! > Post must be at least 20 characters