6 Steps to Mastering Dynamics CRM Duplicate Detection

CRM Duplicate data prevents you from building a holistic view of customers. It can also make it nearly impossible to communicate with your customer base in a meaningful way. However, companies often don’t realize how big the problem is until it’s time to use, report on, or mine their data.  Imagine trying to pull contact information for a marketing campaign and having thousands of duplicate records, each with disparate pieces of information.

Using Microsoft Dynamics CRM duplicate detection rules and jobs, this post outlines the steps you can take to minimize the introduction of duplicate records in your system.

When Microsoft Dynamics CRM duplicate detection rules are enabled the application provides users creating or editing records with a warning when a duplicate record is detected. However, the duplicate is not automatically resolved and the warning is not enforced. This mean that users need to decided how to proceed. Our dream CRM user would resolve the potential duplicate. However, most CRM users choose to ignore the warning and save the record anyways.

In short, users have the freedom and autonomy to decided whether the record is a true duplicate. However, they also have have the responsibility not to ignore and dismiss warnings for true duplicate records.

THE DEFINITION OF DUPLICATE SHOULD BE DRIVEN BY HOW THE DATA IS BEING USED LATER.

For instance, if Marketing will be targeting campaigns at Leads and Contacts then Lead records should never be created for existing Contacts.

It is surprising how often the definition of duplicate varies between individual users or different departments. Consequently, it is necessary to build consensus across the organization on what constitutes a duplicate.

• Eg. If a user creates a brand new Lead in the system for a Contact that already exists, is this a duplicate?
• Eg. If a user creates a subsidiary Account of another corporate Account and they both share the same website or head office address, is this a duplicate?

Microsoft Dynamics CRM includes a number of pre-configured Duplicate Detection rules. However, it’s important to review these rules because many of the pre-configured options don’t make sense for most organizations.

For instance, “Contacts with the same first name and last name” is a rule that should not be enabled in most systems. There are a lot of unique individuals with the same First Name and Last Name. It’s definitely possible to do business with 10 different people named John Smith! We do not want to desensitize users by configuring superfluous rules that will pop up every time they try and create or edit a record.

Well-configured Dynamics CRM duplicate detection rules can be a powerful way to minimize bad data. However, you must fully understand how to build the rules. There are a couple less known facts to be aware of:

• The criteria in a duplicate detection rule is applied as an ‘AND’ statement, not as an ‘OR’ statement.
  • example: Some system administrators will create one massive duplicate detection rule for Contacts with the same Name, Email, Business Phone, and Street Address. Rarely will this duplicate detection rule be applied in the system, as it relies on users entering the data across all these different fields in exactly the same way. It may also rely on optional fields that users may not be required to fill out.
• When configuring duplicate detection rules, you must be strategic.  You can use the ‘AND’ statement to create powerful rules.
  • example: Remember earlier, when we mentioned that the ‘Contacts with the same first name and last name’ rule can lead to negative behaviour in end users? To avoid inundating users with useless duplicate warnings, we can easily tweak this rule to make it more useful. More specifically we can specify that, in addition to the first and last name, the Company Name must also match. With this rule in place a warning about a duplicate John Smith at ABC Company finally becomes useful for users.

Duplicate Detection Rules are only applied when they are published. Unpublished rules remain in a draft state and are not considered in the system. Using the Duplicate Detection Settings in CRM you can specify when published Duplicate Detection Rules are applied. It is recommended to have Duplicate Detection Rules published prior to data imports in order to avoid introducing duplicates into your system en masse.

TIP: Remember, duplicate detection rules in the system are not enforced.  Users may dismiss and ignore duplicate warnings. As part of a complete data quality management program it is important to proactively check for duplicate data in the system.  This means running a Duplicate Detection Job.

Going through the exercise of defining, communicating, and configuring Dynamics CRM duplicate detection rules is not enough. Users must be trained on what to do when they receive a duplicate alert. Otherwise it’s an unwritten adventure with regards to how users will react.

The busy user may dismiss an alert and save the record anyways. The conscientious user may even try and help by deleting one of the duplicates. It is important to provide instructions on how to handle duplicate warnings in the system. More specifically, either merging existing duplicates or not creating a new duplicate record altogether.

During training users should understand the importance of merging records as opposed to deleting them. After all, deleting records in CRM leads to lost data and orphaned records.  Any sales activities tracked against a deleted Contact will potentially disappear. Any Opportunities once related to a deleted Contact are potentially orphaned. Bad processes will lead to bad data.

As previously mentioned, minimizing the likelihood of duplicates in your system will require a bit of proactive effort. Luckily, duplicate detection jobs can be a big help.

The process to set up a Duplicate Detection Job involves defining which entity and records to run the job on. The job will apply published Duplicate Detection Rules to the data set specified.

A DUPLICATE DETECTION JOB WILL NOT RESOLVE DUPLICATES.

Instead, it will highlight potential duplicate records that you may action by merging. Properly trained users will know what action to take when duplicates are detected.

Setting up Dynamics CRM Duplicate Detection Rules and running a Duplicate Detection Job is not a one-time event.  Organizations must be diligent in data quality management.  Proactively managing and addressing duplicate records helps to ensure that users are not left scrambling to make sense of data during critical times such as Year End Reporting or Marketing Campaigns.

Duplicate management should include scheduling regular Duplicate Detection Jobs.  It’s possible to schedule CRM Duplicate Detection Jobs to run automatically based on a specified interval and to receive an email alert after each job. The image below is an example of a scheduled duplicate detection job with email notification.

Duplicate Detection Rules should be reviewed regularly after deployments to CRM. Many organizations do not realize that if they are publishing solutions to CRM which include metadata changes to an entity the Duplicate Detection Rules for that entity will be unpublished. This may not seem intuitive. For a deeper explanation check out Surviving CRM’s post.

A simple best practice is to add a Duplicate Detection Rule check to your organization’s post-deployment checklist.  This will ensure that Duplicate Detection Rules are reviewed after each deployment.