This article gives one idea to avoid creating duplicates when importing from Excel, database or any other data source to a list or library.
In our example we have a list of job titles. Some are duplicated and we do not wish to import these.
We are going to use Import for SharePoint.
We will go to list settings and select the Job Title field (The field which needs to be unique).
We will make that field require a unique value.
We will use our spreadsheet that contains duplicates
The import configuration file for this scenario is included in the documentation and the installation.
When the import runs the tool will report the appropriate errors.
In your destination list the duplicate values will not be present.
This is a simple but effective way to avoid duplicates in a list that works both when importing or entering items via the UI.
Of course your needs may be more complex. Perhaps the duplicate is detected across multiple fields.
You can address more complex scenarios by creating a single unique column in Excel (using CONCAT) or in the database (compound key) or SQL select statement (e.g. ‘ColumnThis’+’ColumnThat’).
Additionally we can enhance Import for SharePoint to address your needs directly under our enhancement programme.