When handling large data imports, there’s a few things to consider.

Validate Source Data

• Always validate data as the first step. The goal here is to catch any problems as quickly as possible in order to not hinder anything else down the line.
• Validate formats given. You should know the expected formats given. This applies to text, images and any other files you receive. Spot check them to make sure they aren’t corrupted.
• Spot check the given data. Determine a number of entries you should test check. This will depend on the amount of time you have for the update and the amount of data given. Make sure to spread out the samples so they aren’t near to each other.
• Always keep an original of the data provided in the client folder with a date. Work with a copy of this data.

Import

• If the import will take a long time (anything over 15minutes), import in batches so you can validate a subset of the data before spending wait time on the entire thing.