Duplicate Detection

This article gives one idea to avoid creating duplicates when importing from Excel, database or any other data source to a list or library.

Scenario

In our example we have a list of job titles.  Some are duplicated and we do not wish to import these.

Import Tool

We are going to use Import for SharePoint.

List Configuration

We will go to list settings and select the Job Title field (The field which needs to be unique).

We will make that field require a unique value.

Import Source

We will use our spreadsheet that contains duplicates

Import Configuration

The import configuration file for this scenario is included in the documentation and the installation.

Office365_JobDescriptions_Items_Excel_Select.xml

Import Execution

When the import runs the tool will report the appropriate errors.

Result

In your destination list the duplicate values will not be present.

Further Thoughts

This is a simple but effective way to avoid duplicates in a list that works both when importing or entering items via the UI.

Of course your needs may be more complex.  Perhaps the duplicate is detected across multiple fields.

You can address more complex scenarios by creating a single unique column in Excel (using CONCAT) or in the database (compound key) or SQL select statement (e.g. ‘ColumnThis’+’ColumnThat’).

Additionally we can enhance Import for SharePoint to address your needs directly under our enhancement programme.