Data Import

Data is only as useful as the information you can make from it. Whether it comes from a third party, another internal system, or a source you are gathering for a new application, the first step is getting it into a usable structure. Sources include data feeds, file drops, vendor APIs, direct database access, and website scraping where the site's terms allow it, and each has its own needs for scheduling, credentials, and change detection.

Getting the data is only part of the job. Imports can also need cleaning, de-duplication, and business rules applied before anyone can rely on the result. That work can be automated, manual, or both: automated rules handle routine cases, and records that match specific conditions go into a queue for a person to review.

Load settings deserve a close look, because bulk-load defaults can skip some malformed rows or bypass checks. SQL Server's BULK INSERT, for example, allows up to 10 syntax errors by default before canceling the load (MAXERRORS, which does not cover constraint checks), and without the CHECK_CONSTRAINTS option it ignores CHECK and foreign key constraints and marks them as not trusted. Setting error handling and validation explicitly, and reconciling the count of rows received against rows accepted, makes a short load visible as one.