Strategy & Growth

Duplicate Contacts and Dirty CRM Data: What the Famous Numbers Actually Say

Bad-data statistics travel without their footnotes. Here is what the $3.1 trillion estimate, the "3% acceptable" study and the CRM vendor surveys actually measured, why one Israeli phone number can become nine records, and a one-afternoon audit.

oToK Team13 min read
Duplicate Contacts and Dirty CRM Data: What the Famous Numbers Actually Say

Not legal advice: The cost example below uses stated, hypothetical inputs (list size, duplicate share), not your data. Replace them with your own before you rely on any figure.

Search for what dirty CRM data costs and you will meet the same few numbers: $3.1 trillion a year for the US economy, 47% of new records containing a critical error, a headline that only 3% of companies' data meets basic quality standards, a contact list that is a quarter duplicates. They are repeated like measurements. Most are estimates, small samples or vendor surveys, and the original sources are more careful than the people quoting them.

We went back to the primary texts: Thomas Redman's 2016 and 2017 articles, the Harvard Business Review study behind the "3%" headline, and two Validity reports on CRM data, one of which assessed 264 billion records. The short version: dirty data is a real, measurable problem, but the famous cost figures cannot be scaled down to a clinic or a course business. What you can measure yourself in an afternoon is your own error rate, and the duplicates one person creates simply by being written down in several ways.

Key takeaways

  • The "$3.1 trillion a year" figure is IBM's estimate of what poor-quality data cost the US in 2016, quoted by Thomas Redman in Harvard Business Review. The article does not show the calculation, and it is a national estimate, not a cost per business.
  • The "only 3%" headline rests on 75 data-quality scores that executives in Irish executive-education programs gave their own departments' last 100 records (Nagle, Redman and Sammon, HBR, September 2017). On average 47% of new records had at least one work-impacting error. It is a small, self-scored sample, not a census of companies.
  • Redman's "rule of ten" says a unit of work costs ten times as much when its data are flawed. If 11 of 100 records are flawed, the same work costs $199 instead of $100; at the 47% average it would cost $523 (our arithmetic). It is a rule of thumb, not a measured constant.
  • Validity assessed 264 billion CRM records, but its "81% need improvement" figure comes from a 35-million-record subset of organizations that had not yet bought a data-management tool, graded on the vendor's own rubric. Its "47% of emails invalid or risky" is 13% invalid plus 34% "risky," meaning unverifiable.
  • In Validity's 2024 survey of 631 CRM users, 53% of those who struggle with data quality named duplicate records as an issue, and 31% of respondents said poor data costs them at least 20% of annual revenue. Both are self-reported opinions from a vendor survey.
  • One Israeli mobile number can be typed as at least nine different strings, so exact matching misses duplicates. Normalize phones to one international format and match on phone, email and a validated ID. The direct cost of messaging a duplicate is small (about $0.035 per WhatsApp marketing message to Israel); the bigger risks are a missed opt-out, split history and broken attribution.

Where does the "$3.1 trillion" bad-data figure come from?

From a Harvard Business Review article by Thomas Redman, published on 22 September 2016. Redman introduces it: "$3.1 trillion, IBM's estimate of the yearly cost of poor quality data, in the US alone, in 2016." It is IBM's number, relayed by Redman, and the article does not reproduce IBM's calculation. What it offers instead is an explanation of where the cost hides.

Redman calls it the "hidden data factory." When one department hands flawed data to the next, the receiving team quietly adds steps to check and correct it, and some errors still leak through to customers. "Salespeople waste time dealing with erred prospect data," he writes, and "data scientists spend an inordinate amount of time cleaning data." Those hidden steps, he says, "form the basis for IBM's $3.1 trillion per year figure."

The same article lists three company-level figures: 50%, 60% and 75%. Only the 60% names an outside source (CrowdFlower); the 50% has no source given, and the 75% is an estimate resting on the Friday Afternoon Measurement and the rule of ten. A year later Redman wrote in MIT Sloan Management Review (27 November 2017) that he and his co-authors estimate bad data costs "15% to 25% of revenue for most companies," based on research by Experian plc and two consultants. We could not see the underlying research.

Widely repeated figureWhat the source saysStatus
"Bad data costs the US $3.1 trillion a year"IBM's estimate for 2016, quoted by Redman; hidden data factories "form the basis" for itReal attribution; national estimate, calculation not shown
"Knowledge workers waste 50% of their time on bad data"Listed by Redman as "the amount of time that knowledge workers waste in hidden data factories"; no study is cited beside itStated, but unsourced in the article
"Data scientists spend 60% of their time cleaning data"Redman credits CrowdFlower for the 60% figureAttributed to CrowdFlower; about data scientists, not CRMs
"Bad data costs 15 to 25% of revenue"Redman, MIT Sloan Management Review, Nov 2017, attributed to Experian plc and two consultantsAttributed; underlying research not visible to us

None of these figures is about a five-person business. They describe large organizations or whole economies, so they say little about what a small list costs you. We left out other widely quoted dollar averages whose methodology we could not open.

What does the "Only 3% of Companies' Data Meets Basic Quality Standards" headline actually measure?

The headline comes from "Only 3% of Companies' Data Meets Basic Quality Standards" (Tadhg Nagle, Thomas Redman and David Sammon, HBR, 11 September 2017). The method is simple, and you can copy it. In executive programs in Ireland, the authors ask each participant to take 10 to 15 critical data attributes for "the last 100 units of work completed by their departments," mark obvious errors, and count the records with none. That count is the department's data-quality (DQ) score. They call it the Friday Afternoon Measurement.

Over two years they collected 75 scores. On average, "47% of newly-created data records have at least one critical (e.g., work-impacting) error." A quarter of the scores were below 30%, half were below 57%, and individual scores ranged from 0% to 99%. The authors add that managers they have asked, in classes and in consulting work, never thought a score below the "high nineties" acceptable; fewer than 3% of the 75 scores met that bar. Breakdowns by industry "have yielded no meaningful insights."

Read the denominator. The "3%" is 3% of 75 scores that managers produced for their own departments, in a teaching setting, using their own judgment of what counts as an "obvious error." The headline says "companies' data," but the study did not audit companies. It is good evidence that error rates in routine records are far higher than managers assume, and weak evidence about any particular business, which is why the useful step is to run the exercise on your own data.

47%

of newly created records had at least one work-impacting error, on average, across 75 self-scored samples of 100 records each.

Source: Nagle, Redman & Sammon, Harvard Business Review, 11 September 2017

The error counts are the managers' own markings of "obvious" errors in their department's records. Treat 47% as a warning about sloppy data creation in general, not as the error rate of CRM contacts.

What is the "rule of ten," and what does it mean for a small list?

In the same article, the authors give the usual way to turn an error rate into money: "it costs ten times as much to complete a unit of work when the data are flawed in any way as it does when they are perfect." Their example: 100 jobs at $1 each when the data are perfect cost $100. If 89 are perfect and 11 are flawed, the cost is 89 x $1 + 11 x $10 = $199.

Because the rule is linear, you can run it on any error rate. At the 47% average from the study, 53 x $1 + 47 x $10 = $523, more than five times the perfect-data cost. The authors say most people find "a good first approximation" in the rule of ten, and note that it does not count non-monetary costs such as lost customers or bad decisions. It is a heuristic for sizing a problem, not a measurement of yours.

Cost of the same 100 tasks ($1 each when data are perfect), by share of flawed records
47% flawed (study average)
$523.00
25% flawed
$325.00
11% flawed (the authors' example)
$199.00
0% flawed
$100.00
Source: Our arithmetic from the "rule of ten" as stated in Nagle, Redman & Sammon, HBR, 2017: cost = (unflawed records x $1) + (flawed records x $10). The 11% case is the authors' own worked example ($199).

How many CRM records are actually bad?

One widely circulated measurement of real CRM data comes from Validity, which sells data-cleaning software for Salesforce. In a short report, "5 Things We Learned from Evaluating 264 Billion CRM Records," it says it assessed 264 billion records from over ten thousand CRM organizations using its own DemandTools product. A record failed a "basic quality check" if it was a duplicate, was missing information in critical fields, or held invalid data.

The headline is "81 percent of records assessed require improvement." But read the sentence under it: that figure is for "over 35 million CRM records of organizations yet to invest in a data management solution." It is a subset, drawn from organizations that had not yet cleaned up, graded on the vendor's own scale. Within it, Validity reports 17% of records as "Acceptable," 32% "Poor," 25% "Very Poor" and 24% "Unactionable." On email, "nearly half (47 percent) of the lead and contact emails" were "invalid or risky": 13% invalid and 34% risky, where risky means the address cannot be fully verified and may include accept-all servers.

So the honest reading is narrower than the headline. Only 13 of those 47 points are addresses known to bounce; the rest are addresses nobody could confirm. And the 81% figure comes from organizations that had not yet bought a data-management solution, the audience the vendor sells to; the report does not describe the email sample further. It tells you data decays and that unmanaged lists are in poor shape, which is useful, but it is not a share of all businesses.

Record quality at organizations without a data-management tool (% of 35M+ records assessed)
Poor
32%
Very poor
25%
Unactionable
24%
Acceptable
17%
Source: Validity, "5 Things We Learned from Evaluating 264 Billion CRM Records" (2024), chart read from the report. A fifth category, "Validified," appears with no printed value and is left out. Vendor-defined grades; subset of organizations that had not yet bought a data-management tool.

Validity sells the cleanup tool its figures describe. That does not make the numbers wrong, but it is a reason to treat them as a signal about unmanaged data rather than an industry average.

What do CRM admins say about duplicates and the cost of bad data?

Validity's 2024 survey, "The State of CRM Data Management in 2024," asked 631 CRM users and stakeholders in the United States, the United Kingdom and Australia. Respondents who said they struggle with data-quality issues were asked which ones they face. Duplicate records came fourth at 53%, behind incomplete data (68%), missing data (65%) and incorrect data (61%), and ahead of expired data (49%). Note the base: these are percentages of people who already said they struggle, not of all respondents.

The survey's headline finding is that 31% of admins said poor-quality data costs them at least 20% of annual revenue, which Validity describes as a 933% increase from its 2021 study. That jump implies about 3% saying so three years earlier. A tenfold change in a stated opinion is a sign the number measures sentiment, not revenue. The report also says that "a recent Salesforce study found that the average customer's contact database contains more than 25 percent duplicates," and that most CRM vendors charge by the number of contacts stored. We could not find the Salesforce study itself, so the 25% figure is traceable only through Validity.

Data-quality issues named by CRM users who report struggling with data quality (%)
Incomplete data
68%
Missing data
65%
Incorrect data
61%
Duplicate records
53%
Expired data
49%
Source: Validity, "The State of CRM Data Management in 2024," Figure 3 (631 CRM users and stakeholders in the US, UK and Australia; multiple answers allowed; asked of respondents who said they struggle with data quality).

This is a vendor survey of CRM administrators. It shows what practitioners say is wrong, and duplicates are on the list. It does not say how many of your contacts are duplicates.

How does one customer end up as several records?

A duplicate is rarely the same person typed identically twice. It is one person written down in several ways, so a system that compares text exactly will see strangers. Phone numbers are the clearest example: the same Israeli mobile line can be stored as any of the nine strings below, all plausible variants of one number. The fix is to normalize before you compare: strip separators, drop the domestic leading zero, and store one international form such as +972501234567. Email needs the same care (case, trailing spaces, a work address and a personal address for one person). Names are the weakest key of all: "יוסי," "Yossi" and "Yosi" can be the same person.

String storedHow it can get there
0501234567Typed the way Israelis write it
050-1234567Hyphen after the prefix
050-123-4567Three-part grouping
050 123 4567Spaces from a form or copy-paste
+972501234567International format from a WhatsApp contact or an API
+972-50-123-4567International format with separators
972501234567The plus sign stripped by a tool
00972501234567International dialing prefix written out
501234567A spreadsheet dropped the leading zero

Merging goes wrong in the other direction too. Two different people can share a household landline or a family email address, and a false merge is harder to spot than a duplicate. That is why a merge you can review and undo matters more than a merge that is fast.

How does the check digit on an Israeli ID number work?

A national ID is the strongest match you can have, and in Israel it carries its own error check. The identity number has nine digits, the last of which is a check digit calculated with the Luhn algorithm (Wikipedia's description of the Israeli identity card; the open-source python-stdnum library pads shorter numbers with leading zeros before checking). Take 039337423, a number printed in that library's documentation as valid. Multiply the digits, left to right, by 1, 2, 1, 2, 1, 2, 1, 2, 1. Where a product is 10 or more, add its two digits. You get 0 + 6 + 9 + 6 + 3 + 5 + 4 + 4 + 3 = 40, a multiple of 10, so it passes. Change the last digit to 2 and the sum is 39, so it fails.

That check catches typos, nothing more. A well-formed number does not prove it was ever issued to anyone, so use it to reject a mistyped digit at entry, not to verify a person.

What does a duplicate cost a small business in practice?

Less in direct fees than the headlines suggest, and more in everything around them. Take a hypothetical list of 10,000 records in which 8% extra records (800) are duplicates of someone already on the list. Both inputs are our assumptions. If a campaign sends a WhatsApp marketing template to every record, Meta's Israel rate card effective 1 October 2026 is about $0.0353 per delivered marketing message, so 800 duplicates cost 800 x $0.0353 = $28.24 per campaign. Real money over a year of campaigns, but not the headline-sized figure.

The costs that are hard to put a price on are the ones that do the damage. They are mechanisms we can describe, not numbers we can quote:

Where it hurtsWhat happensCan you put a number on it?
Paid sendsEvery duplicate that gets a message is a message you pay forYes: records x rate (about $28 per campaign in the example above)
An opt-out that did not stickSomeone unsubscribes on one record while a second record of the same person stays subscribed, so the next campaign reaches a person who said stopNo, and the exposure depends on the channel and the facts
Split historyThe chat sits on one record and the purchase on another, so lifetime value and attribution are each wrongOnly by auditing your own data
Two people, one customerTwo team members contact the same lead without knowing itNo, but customers notice

On the opt-out row: Israel's Spam Law (Section 30A of the Communications Law) concerns commercial messages by email, SMS, fax and automated dialing, and its statutory damages are a ceiling a court may award, not an automatic amount. Our article on the law covers it in detail. We are not offering a legal opinion here.

Does a messy list raise your bill?

It depends on how your vendor charges. Validity's 2024 report says most CRM vendors charge by the number of contacts stored, so a bloated list can raise the bill. That is a pricing model, not a law of nature. In oToK, contacts are unlimited on every plan, so duplicates do not raise a contact-based fee, but sends still cost something: paid plans are priced by monthly marketing-email volume, and WhatsApp messages are billed by Meta per message.

How do you measure your own data quality in an afternoon?

You can adapt the Friday Afternoon Measurement to a CRM (8 to 10 fields instead of the study's 10 to 15). In our estimate it takes about an hour and gives you a number about your data instead of someone else's.

  1. 1Pull the last 100 contacts created, by any route: forms, imports, the inbox, your team, integrations.
  2. 2Pick the 8 to 10 fields your work depends on, for example a valid mobile number, an email, a name, an owner, the source, consent status for each channel.
  3. 3Go record by record and mark it clean only if every field you picked is right. Count the clean ones. That count is your DQ score out of 100.
  4. 4Size the problem with the rule of ten: cost multiplier = 1 + 9 x (share of flawed records). A score of 80 means a multiplier of 2.8; a score of 53 means 5.2.
  5. 5Check duplicates separately: export your list, normalize phones to international format, lowercase emails, then count how many phone numbers or emails appear on more than one record.
  6. 6Trace the three worst errors back to where they entered (a form, an import file, a manual entry) and fix the entry point before you clean the list.
Your DQ score (clean records out of 100)Rule-of-ten cost multiplierReading
971.3Close to the "high nineties" bar the authors say managers they asked require
802.8Work costs nearly three times what it should
535.2The study's average: more than five times

The multiplier is the rule of ten applied to your own count. It is a way to see how much a few points of error matter, not a forecast of your costs.

How does oToK keep duplicates out?

oToK works on the principle that it is cheaper to stop a duplicate than to merge one. Duplicate prevention, contact merge and duplicate management are available on every plan, with identity matched on three values: phone, email and national ID. National IDs are validated as you type, using a check-digit rule for Israel and a general rule elsewhere.

  • A duplicate warning as you type, before the contact is saved; you choose per workspace whether to warn or block
  • Import with a choice of which identifiers to match on (phone, email, national ID) and in what priority order, plus a downloadable row-by-row error report
  • Integrations, the API and webhooks never create a duplicate quietly: every identity conflict is resolved or raised for review
  • Compatible matches, such as a phone that matches one contact and an email that matches another, merge automatically on arrival; anything uncertain or conflicting waits for review
  • Side-by-side, field-by-field merge, with the reason contacts were grouped shown, and each value's change history so you keep the current one
  • Undo a merge for up to 24 hours, and a pair you decide to keep separate is remembered and does not resurface

None of this replaces the afternoon audit. Prevention at entry lowers the rate; measuring tells you whether it worked.

Share this article

Frequently asked questions

How much do duplicate contacts cost a small business?
No study we could find prices it for a small business. The direct cost is arithmetic: every duplicate that receives a paid message costs the per-message rate (about $0.0353 per WhatsApp marketing message to Israel on the rate card effective 1 October 2026), so 800 duplicates cost about $28 per campaign. The larger risks are an opt-out that does not carry across two records, split purchase and conversation history, and two team members contacting one lead.
What percentage of CRM contacts are duplicates?
There is no neutral figure. Validity's 2024 survey of 631 CRM users found 53% of those struggling with data quality named duplicates as an issue, and the same report relays a Salesforce-attributed claim of more than 25% duplicates that we could not trace to the original study. Measure your own: export, normalize phones and emails, and count records that share one.
Where does the $3.1 trillion bad-data figure come from?
It is IBM's estimate of the yearly cost of poor-quality data to the US economy in 2016, quoted by Thomas Redman in Harvard Business Review on 22 September 2016. Redman says hidden data factories, the extra work teams do to fix flawed data, form the basis for it. The article does not show the calculation, so treat it as an estimate for a whole economy, not a figure to scale down to your business.
What is the "rule of ten" in data quality?
It is the rule that completing a unit of work costs ten times as much when the data are flawed as when they are perfect, as stated by Nagle, Redman and Sammon in Harvard Business Review in 2017. With 100 jobs at $1 each, 89 perfect and 11 flawed, the cost is 89 x $1 + 11 x $10 = $199. The authors say most people find it a good first approximation, and it omits lost customers and bad decisions.
What is the 1-10-100 rule?
A related heuristic: roughly $1 to prevent a defect, $10 to correct it internally and $100 to fix it after it reaches the customer. It is often credited to Labovitz, Chang and Rosansky's 1992 book Making Quality Work, though attributions vary and we could not confirm the source. The ratios are illustrative, not measured constants. It is sometimes confused with Redman's rule of ten.
How do I check whether an Israeli ID number is valid?
The identity number has nine digits, the last a check digit calculated with the Luhn algorithm. Pad shorter numbers with leading zeros, multiply the digits by 1 and 2 alternately starting with 1, add the digits of each product, and the total must be a multiple of 10. This catches typos only; it does not show that the number was issued to a real person.
How should I store Israeli phone numbers so duplicates are caught?
In one international format, such as +972501234567 for a mobile number written 050-123-4567 domestically: drop the leading zero, add +972 and strip separators. The same line can otherwise appear as nine or more different strings, including one where a spreadsheet dropped the zero. Match on the normalized form, not the typed text.
Does oToK charge more if my list has duplicate contacts?
Not by contact count: contacts, deals and pipelines are unlimited on every plan. Sends still cost something. Paid plans are priced by monthly marketing-email volume, and WhatsApp messages are billed by Meta per message with nothing added by oToK, so a duplicate that receives a message is still a message you pay for. Duplicate prevention and merge are available on every plan.

Ready to upgrade your customer communication?

Get started with oToK — connecting a WhatsApp number takes minutes.

No credit card required — start right away