# The Dedupe API

> `POST https://dedupe.dev/api/dedupe` with a JSON body and get the deduplicated result back. Three modes are available: `lines`, `rows` (CSV), and `json`. First occurrence kept, order preserved. Metered in units, not calls: one unit is one row of up to 1 KB, and a free key gets 1,000 a month. CORS-open, and nothing you send is stored.

Canonical HTML version: https://dedupe.dev/api
Machine-readable: https://dedupe.dev/api.md (this file), https://dedupe.dev/openapi.json (OpenAPI 3.1)
Last updated: 2026-07-25

## Endpoint

`POST https://dedupe.dev/api/dedupe`. JSON in, JSON out, one endpoint, no versioning in the path. There is nothing else to call: no job queue, no polling, no webhooks. The response is the answer.

It is CORS-open, so browser code can call it directly. These four headers are on **every** response, success or error:

| Header | Value |
| --- | --- |
| `Content-Type` | `application/json` |
| `Access-Control-Allow-Origin` | `*` |
| `Access-Control-Allow-Methods` | `POST, OPTIONS` |
| `Access-Control-Allow-Headers` | `Content-Type` |

`OPTIONS` answers preflight with `204` and no body. `GET`, `PUT`, `PATCH` and `DELETE` answer `405`. `Retry-After` is the only other header the endpoint ever sets, and only on a burst-limit `429`. There are no `X-RateLimit-*` headers: if you need your remaining allowance, read `limit` and `used` off the `429` body or check [your dashboard](/app).

Requests are processed in memory and the result is returned. Nothing you send is written to storage. Keyed calls record usage metadata only: mode, input row count, rows removed, billable units, status and duration. Never the data.

## Authentication

A key is optional. Without one you get the keyless evaluation tier; with one you get your plan's limits. Two header forms are accepted, and they are equivalent:

```http
Authorization: Bearer dd_live_...
x-api-key: dd_live_...
```

`Authorization` is read first and must be `Bearer <token>`. If that header is absent or is not a bearer token, `x-api-key` is used instead. If neither is present the request is served as keyless.

### Creating and revoking keys

Create keys at [/app/keys](/app/keys). A token is `dd_live_` followed by 64 hex characters (32 random bytes). **The full token is shown once, at creation.** Only its SHA-256 hash and a 12-character display prefix are stored, so a lost key cannot be recovered, only replaced.

Revoking is immediate and permanent: a revoked key stops matching on the next request. Free plans may hold 1 active key, Starter 3, Pro unlimited.

An unknown, revoked, or malformed key returns `401` with `{"error":"invalid api key"}`. So does a key that cannot be checked at all: verification fails closed rather than letting an unverified request through.

The token is parsed out of the header and never logged or persisted. Treat it as a secret: it is a bearer credential, and anyone holding it spends your monthly units.

## Request body

The body is a single JSON object with three fields:

```json
{
  "mode": "lines" | "rows" | "json",
  "data": <string for lines and rows, array for json>,
  "options": { }
}
```

| Field | Type | Required | Notes |
| --- | --- | --- | --- |
| `mode` | string | yes | One of `lines`, `rows`, `json`. Anything else is a `400`. |
| `data` | string or array | yes | Type depends on the mode, see below. |
| `options` | object | no | Defaults to `{}`. Unrecognised keys are ignored. |

There is no `Content-Type` requirement: the body is read as text and parsed as JSON regardless of what you declare. Sending `application/json` is still the right thing to do.

### `lines` mode

`data` is one string of text. It is split on `\r\n`, `\n` or `\r`, and duplicate lines are dropped. The first occurrence is kept and the surviving order is the original order.

| Option | Type | Default | Effect |
| --- | --- | --- | --- |
| `trim` | boolean | `false` | Trim leading and trailing whitespace off each line before comparing. The trimmed line is what comes back. |
| `ignoreCase` | boolean | `false` | Compare case-insensitively. The output keeps the first spelling seen, it is not lower-cased. |
| `dropEmpty` | boolean | `false` | Discard empty lines entirely. Applied after `trim`, so whitespace-only lines go too when both are set. |

```bash
curl -s -X POST https://dedupe.dev/api/dedupe \
  -H "Content-Type: application/json" \
  -d '{"mode":"lines","data":"a\nb\na","options":{"trim":true,"ignoreCase":true}}'

→ 200 {"result":["a","b"],"kept":2,"removed":1}
```

### `rows` mode

`data` is a CSV **string**, not an array. It is parsed with a comma delimiter, double-quote quoting (`""` is a literal quote inside a quoted field), and `\r\n`, `\n` or `\r` row breaks. A leading UTF-8 BOM, which Excel writes when you save as CSV UTF-8, is stripped. The delimiter is not configurable over the API: tab-separated and semicolon-separated files are not parsed as such, so convert them to commas first or use the [in-browser CSV tool](/csv), which does detect them.

| Option | Type | Default | Effect |
| --- | --- | --- | --- |
| `keyColumns` | int[] | `[]` | Zero-based column indexes that define a duplicate. Empty means the whole row. A missing cell counts as an empty string. |
| `hasHeader` | boolean | `true` | Treat the first row as a header: never deduplicated, always returned first, and not billed as a row. |
| `caseSensitive` | boolean | `true` | `false` compares keys case-insensitively. Output rows are returned unchanged either way. Ignored when `fuzzy` is on, which normalizes case itself. |
| `fuzzy` | boolean | `false` | Fuzzy matching: keys are lowercased, punctuation-stripped and word-sorted, then compared by edit distance, so `Smith, Jane` matches `jane smith`. Requires an API key (any tier) and at most 50,000 data rows per request. |
| `threshold` | number | `0.9` | Similarity two keys must reach to count as duplicates, from 0.5 to 1. 0.85 to 0.92 works for names; 1 means exact-after-normalization. |

Fuzzy matching is the one comparison that cannot run in the browser tools: naively it is pairwise, and it used to freeze the tab on real-world lists. Server-side it uses candidate blocking (a match must share a word or a character n-gram with an already-kept key, which anything above a realistic threshold does), so a request can carry up to 50,000 data rows, the same ceiling as the paid per-request unit cap, and still answer in seconds. At thresholds below about 0.8 on very short values, blocking can miss rare pairs; the 0.9 default and the 0.85 to 0.92 band recommended for names are unaffected. The first occurrence is always kept, nothing is merged, and billing is the same as any other request.

```bash
curl -s -X POST https://dedupe.dev/api/dedupe \
  -H "Authorization: Bearer $DEDUPE_KEY" \
  -H "Content-Type: application/json" \
  -d '{"mode":"rows","data":"name\n\"Smith, Jane\"\njane smith\nJane Smyth\nBob Ray","options":{"keyColumns":[0],"fuzzy":true,"threshold":0.85}}'

→ 200 {"result":"name\n\"Smith, Jane\"\nBob Ray","kept":2,"removed":2}
```

The result is re-serialised CSV: fields containing a comma, a double quote, or a line break are quoted and inner quotes are doubled; rows are joined with `\n`. Cell values are otherwise untouched, so a file that round-trips through the API comes back byte-comparable minus its duplicates and its BOM.

```bash
curl -s -X POST https://dedupe.dev/api/dedupe \
  -H "Content-Type: application/json" \
  -d '{"mode":"rows","data":"email,name\na@x.com,Ann\na@x.com,Ann B","options":{"keyColumns":[0],"hasHeader":true}}'

→ 200 {"result":"email,name\na@x.com,Ann","kept":1,"removed":1}
```

### `json` mode

`data` is an array of anything: objects, strings, numbers, nested structures. Items are compared by their JSON serialisation, or by one value inside them if you pass `key`.

| Option | Type | Default | Effect |
| --- | --- | --- | --- |
| `key` | string | `""` | Dot path to the value that identifies an item, for example `user.id` or `profile.email`. Empty compares the whole item. |

A path that does not resolve yields `undefined`, and every item with an unresolvable path collapses to the same key, so only the first of them survives. That is usually a sign the path is wrong. Without `key`, comparison is on `JSON.stringify` of the item, which means key order matters: `{"a":1,"b":2}` and `{"b":2,"a":1}` are two different items.

```bash
curl -s -X POST https://dedupe.dev/api/dedupe \
  -H "Content-Type: application/json" \
  -d '{"mode":"json","data":[{"user":{"id":1}},{"user":{"id":1}},{"user":{"id":2}}],"options":{"key":"user.id"}}'

→ 200 {"result":[{"user":{"id":1}},{"user":{"id":2}}],"kept":2,"removed":1}
```

## Responses

A successful call returns `200` and the same three fields in every mode. Only the type of `result` changes:

| Mode | `result` | `kept` | `removed` |
| --- | --- | --- | --- |
| `lines` | string[], one per surviving line | surviving lines | duplicate lines dropped |
| `rows` | string, CSV including the header when `hasHeader` is true | surviving data rows, header excluded | duplicate rows dropped |
| `json` | array of the surviving items | surviving items | duplicate items dropped |

`kept + removed` equals the input count in `lines` and `json` mode. In `rows` mode it equals the input row count excluding the header. `dropEmpty` in `lines` mode is the one exception: dropped empty lines are counted in neither, because they were discarded rather than deduplicated.

## Errors

Every error is JSON with an `error` string, and carries the same CORS headers as a success. The strings below are the literal ones the handler emits.

| Status | `error` | Cause |
| --- | --- | --- |
| 400 | `invalid JSON body` | The request body is not parseable JSON. |
| 400 | `mode must be one of: lines, rows, json` | `mode` is missing, or is not one of the three modes. |
| 400 | `lines mode expects data: string` | `mode` is `lines` but `data` is not a string. |
| 400 | `rows mode expects data: CSV string` | `mode` is `rows` but `data` is not a string. |
| 400 | `json mode expects data: array` | `mode` is `json` but `data` is not an array. |
| 400 | `fuzzy matching is available in rows mode only` | `options.fuzzy` is set on a `lines` or `json` request. Fuzzy matching compares whole normalized records, which only means something for tabular data, so it is a `rows`-mode option. |
| 400 | `threshold must be a number between 0.5 and 1` | `options.threshold` is present but is not a number between 0.5 and 1. |
| 403 | `fuzzy matching requires an API key — create a free key at https://dedupe.dev/app/keys` | A keyless request asked for fuzzy matching. Fuzzy is the expensive pairwise path and needs an accountable caller: any key works, including the free tier's. |
| 401 | `invalid api key` | The presented key is unknown, revoked, malformed, or could not be verified. Verification fails closed, so an unverifiable key is treated as an invalid one. |
| 405 | `POST a JSON body — see https://dedupe.dev/api` | The method is not `POST` or `OPTIONS`. `GET`, `PUT`, `PATCH` and `DELETE` all answer with this. |
| 413 | `payload too large (256 KB plan limit)` | A keyless request is over the 256 KB keyless body cap. |
| 413 | `payload too large (512 KB Free plan limit)` | A keyed request declares a `Content-Length` over the plan's body cap. Refused before the body is read, so it never buffers. The same cap is re-checked after reading, and that second check reports `payload too large (512 KB plan limit)` without the plan name. |
| 413 | `fuzzy matching is capped at 50,000 rows per request (got 50,500); split the file or use exact matching` | A fuzzy request carries more than 50,000 data rows. That is the same ceiling as the paid plans' per-request unit cap, so in practice the plan cap is what you hit first. The number in the message is your row count. |
| 413 | `request too large: 812 units, over the 200-unit Free per-request limit` | The request parsed fine but costs more units than the plan allows in one call. A distinct message from the size cap: this one is about row count, not bytes. |
| 429 | `monthly unit limit reached` | This request would take the account past its monthly unit allowance. No `Retry-After`: `resets` is the ISO timestamp the allowance refills at. |
| 429 | `rate limit reached: 60 requests per hour on Free` | The key exceeded its burst rate limit. Sent with a `Retry-After` header in seconds. |
| 429 | `keyless daily unit limit reached — create a free API key at https://dedupe.dev/app/keys` | A keyless caller spent its 500-unit daily budget for that IP. No `Retry-After`. |

Checks run in this order for a keyed request, and the first one to fail is what you get back: key verification, declared body size, JSON parse and mode validation (including the fuzzy row cap), per-request unit cap, monthly unit quota, burst rate limit. So a request that is both unauthenticated and oversized returns `401`, not `413`.

### 400, invalid body

Seven distinct messages, in the order the parser reaches them. The first is a parse failure, the rest are shape failures found after the body parsed.

```json
{
  "error": "invalid JSON body"
}
```

```json
{
  "error": "mode must be one of: lines, rows, json"
}
```

```json
{
  "error": "lines mode expects data: string"
}
```

```json
{
  "error": "rows mode expects data: CSV string"
}
```

```json
{
  "error": "json mode expects data: array"
}
```

```json
{
  "error": "fuzzy matching is available in rows mode only"
}
```

```json
{
  "error": "threshold must be a number between 0.5 and 1"
}
```

### 401, bad key

```json
{
  "error": "invalid api key"
}
```

### 403, fuzzy without a key

```json
{
  "error": "fuzzy matching requires an API key — create a free key at https://dedupe.dev/app/keys"
}
```

### 405, wrong method

Returned with the CORS headers, so a misrouted browser call still sees the message.

```json
{
  "error": "POST a JSON body — see https://dedupe.dev/api"
}
```

### 413, too large

Three different limits. The first two messages are about bytes, the third about the fuzzy row cap, the fourth about units. A 100 KB body of very short rows can pass the size cap and still fail the unit cap.

```json
{
  "error": "payload too large (256 KB plan limit)"
}
```

```json
{
  "error": "payload too large (512 KB Free plan limit)"
}
```

```json
{
  "error": "fuzzy matching is capped at 50,000 rows per request (got 50,500); split the file or use exact matching"
}
```

```json
{
  "error": "request too large: 812 units, over the 200-unit Free per-request limit",
  "units": 812,
  "limit": 200
}
```

### 429, out of allowance

Three cases, and they are not interchangeable. The monthly one will not clear until `resets`. The burst one clears in `retryAfter` seconds and is the only response that carries a `Retry-After` header. The keyless one clears the next UTC day, or immediately if you create a key.

```json
{
  "error": "monthly unit limit reached",
  "limit": 1000,
  "used": 987,
  "resets": "2026-08-01T00:00:00.000Z"
}
```

```json
{
  "error": "rate limit reached: 60 requests per hour on Free",
  "limit": 60,
  "windowSeconds": 3600,
  "retryAfter": 42
}
```

```json
{
  "error": "keyless daily unit limit reached — create a free API key at https://dedupe.dev/app/keys",
  "limit": 500,
  "units": 12
}
```

## How billing works

**One unit is one row of up to 1 KB.** Every request costs:

```text
units = max(rows, ceil(bytes / 1024))
```

`rows` is what you sent us: lines in `lines` mode, data rows excluding the header in `rows` mode, array items in `json` mode. `bytes` is the size of the whole request body. Not what comes back: you are billed for the input you asked us to read, so removing more duplicates never costs more. A request that reaches the handler always costs at least one unit.

Worked examples:

- A 50,000-row CSV averaging 80 bytes a row = **50,000 units**. Rows win: 4 MB of body is only about 3.9k units, so the row count is the bigger of the two.
- The same 50,000 rows split across 50 calls of 1,000 rows each = **50,000 units**. Batching changes nothing, which is the point. The old per-call meter could be gamed by packing everything into one request.
- One 5 MB row of dense JSON = **about 4.9k units**. Bytes win. Bandwidth is our biggest cost per request, so a fat row is priced like the many small rows it costs us as much as.
- A 12-line email list in a 400-byte body = **12 units**. Neither term is large, so the row count decides again.

Units reset on the 1st of each month, UTC, and the allowance is for the whole account rather than per key. Three keys on Starter share the same 100,000. Free stops at 1,000 units with a `429`; Starter and Pro currently do the same at their caps, and metered overage pricing is on the way so paid plans can run past them instead of stopping.

**The in-browser tools are free and completely unmetered.** They run entirely in your browser and never touch this endpoint, so none of this applies to them. Only the API is billed. If a browser tab can do the job, use [the tools](/) and pay nothing.

A request we refuse is never billed. `401`, `413` and `429` responses record a status row for your dashboard and consume no units.

| Plan | Units/month | Max units/request | Max request | Keys |
| --- | --- | --- | --- | --- |
| Keyless | 500 per IP, per day | 500 | 256 KB | none |
| Free key (free) | 1,000 | 200 | 512 KB | 1 |
| Starter key ($19/mo) | 100,000 | 50,000 | 5 MB | 3 |
| Pro key ($49/mo) | 300,000 | 50,000 | 5 MB | unlimited |

The per-request byte cap is enforced twice: once against the declared `Content-Length`, before the body is buffered, and once against the real length after reading. No plan can be raised above 5 MB without changing hosting: the platform answers a larger body itself, before our handler runs.

## Rate limits

Separate from the monthly meter. The monthly quota is what you bought; the burst limiter just stops one runaway script from monopolising the function.

| Plan | Requests per window | Window | Retry-After |
| --- | --- | --- | --- |
| Keyless | not rate limited | daily 500-unit budget per IP | not sent |
| Free | 60 | 1 hour | sent |
| Starter | 60 | 60 seconds | sent |
| Pro | 300 | 60 seconds | sent |

It is a fixed window, counted per key. Exceeding it returns `429` with `Retry-After` set to the whole number of seconds until the current window ends, never less than 1. The same number is in the body as `retryAfter`.

It is enforced best effort, in the memory of whichever serverless instance answered, so it is not shared between instances and resets on a cold start. In practice you may occasionally get slightly more than the table says. Do not design around that: the monthly unit quota is the real meter and it is counted in the database.

### What a client should do

1. **Respect `Retry-After` before anything else.** Sleep at least that long. It is exact, not advisory.
2. **Add exponential backoff on top**, so a fleet of workers that all hit the wall together do not all wake up together. Sleep `Retry-After` plus a growing jitter.
3. **Treat a `429` without `Retry-After` as terminal.** That is the monthly quota or the keyless daily budget, and retrying will not clear it. Read `resets`, and alert rather than loop.
4. **Cap your attempts.** Five is plenty. A job that has retried five times has a different problem.
5. **Batch rather than parallelise.** Units are charged the same either way, so one 50,000-row call costs exactly what fifty 1,000-row calls cost and uses one request instead of fifty.

## Recipes

### Clean a CRM export before importing it

Sales exports arrive with the same contact three times. Dedupe on the email column and import the result. `keyColumns` is zero-based, so change the index if email is not the first column.

```bash
jq -Rs '{mode:"rows", data:., options:{keyColumns:[0], hasHeader:true, caseSensitive:false}}' \
    < contacts.csv \
  | curl -s -X POST https://dedupe.dev/api/dedupe \
      -H "Authorization: Bearer $DEDUPE_KEY" \
      -H "Content-Type: application/json" \
      --data-binary @- \
  | jq -r '.result' > contacts-clean.csv
```

### Dedupe an email list

One address per line. `trim` strips stray whitespace, `dropEmpty` removes blank lines, and `ignoreCase` compares case-insensitively while returning the first spelling it saw.

```bash
jq -Rs '{mode:"lines", data:., options:{trim:true, ignoreCase:true, dropEmpty:true}}' \
    < emails.txt \
  | curl -s -X POST https://dedupe.dev/api/dedupe \
      -H "Authorization: Bearer $DEDUPE_KEY" \
      -H "Content-Type: application/json" \
      --data-binary @- \
  | jq -r '.result[]' > emails-clean.txt
```

### Use it in a shell pipeline with jq

`json` mode takes an array straight out of another API and dedupes it on a dot path, so nothing has to touch disk.

```bash
curl -s https://api.example.com/v1/contacts \
  | jq '{mode:"json", data:., options:{key:"profile.email"}}' \
  | curl -s -X POST https://dedupe.dev/api/dedupe \
      -H "Authorization: Bearer $DEDUPE_KEY" \
      -H "Content-Type: application/json" \
      --data-binary @- \
  | jq '{removed, kept}'
```

### Run it from cron

A nightly job that fails loudly. Note the exit status check: a silent cron job that swallows a 429 will publish yesterday's list forever.

```bash
#!/usr/bin/env bash
# /etc/cron.daily/dedupe-leads
set -euo pipefail

: "${DEDUPE_KEY:?DEDUPE_KEY is not set}"

code=$(jq -Rs '{mode:"lines", data:., options:{trim:true, dropEmpty:true}}' < /srv/leads.txt \
  | curl -s -o /tmp/dedupe.json -w '%{http_code}' \
      -X POST https://dedupe.dev/api/dedupe \
      -H "Authorization: Bearer $DEDUPE_KEY" \
      -H "Content-Type: application/json" \
      --data-binary @-)

if [ "$code" != "200" ]; then
  echo "dedupe.dev returned $code: $(cat /tmp/dedupe.json)" >&2
  exit 1
fi

jq -r '.result[]' /tmp/dedupe.json > /srv/leads-clean.txt
```

## The validation endpoint

`POST https://dedupe.dev/api/validate` checks email addresses before you import or send. **Hygiene-grade, and we say exactly what that means**: syntax, MX-record presence, disposable-domain detection, and role-account flagging (info@, support@). It is NOT SMTP-grade verification: no mailbox pinging and no catch-all detection, so an address can pass every check and still bounce. Dedicated verifiers charge around $7 to $10 per 1,000 emails for the full job; this does the cheap, honest subset at API prices.

An API key is required (any tier; there is no keyless mode, because each unique domain costs a DNS lookup). Billing is **2 units per address** against the same monthly allowance as /api/dedupe, and the per-request unit cap applies to the billed units, so one request may carry at most maxUnitsPerRequest / 2 addresses (100 on Free, 25,000 on paid plans). Refused requests are never billed.

```bash
curl -s -X POST https://dedupe.dev/api/validate \
  -H "Authorization: Bearer $DEDUPE_KEY" \
  -H "Content-Type: application/json" \
  -d '{"emails":["ann@example.com","info@example.com","burner@mailinator.com","not-an-email"]}'

→ 200 {"results":[
  {"email":"ann@example.com","valid":true,"reason":"ok","mx":true,"disposable":false,"role":false},
  {"email":"info@example.com","valid":true,"reason":"ok","mx":true,"disposable":false,"role":true},
  {"email":"burner@mailinator.com","valid":false,"reason":"disposable","mx":true,"disposable":true,"role":false},
  {"email":"not-an-email","valid":false,"reason":"syntax","mx":null,"disposable":false,"role":false}
],"summary":{"total":4,"valid":2,"invalid":2,"disposable":1,"role":1}}
```

Per address: `valid` (syntax ok, not disposable, and MX did not come back definitely absent), `reason` (`ok`, `syntax`, `disposable`, or `no_mx`), `mx` (`true`, `false`, or `null` when the lookup itself failed, which never fails an address), `disposable`, and `role`. Role accounts stay `valid`; whether to mail info@ is your call, so it is a flag, not a verdict. Addresses are processed in memory and never stored, the same policy as every other endpoint.

## Code samples

The same call in five languages. Each one deduplicates a two-row CSV on its first column and
returns `kept: 1, removed: 1`.

### curl

```bash
curl -X POST https://dedupe.dev/api/dedupe \
  -H "Authorization: Bearer $DEDUPE_KEY" \
  -H "Content-Type: application/json" \
  -d '{"mode":"rows","data":"email,name\na@x.com,Ann\na@x.com,Ann B","options":{"keyColumns":[0],"hasHeader":true}}'

→ 200 {"result":"email,name\na@x.com,Ann","kept":1,"removed":1}
```

### node

```js
const res = await fetch("https://dedupe.dev/api/dedupe", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.DEDUPE_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    mode: "rows",
    data: "email,name\na@x.com,Ann\na@x.com,Ann B",
    options: { keyColumns: [0], hasHeader: true },
  }),
});
const { result, kept, removed } = await res.json();

// kept 1, removed 1 (first occurrence kept, order preserved)
```

### python

```python
import os, requests

res = requests.post(
    "https://dedupe.dev/api/dedupe",
    headers={"Authorization": f"Bearer {os.environ['DEDUPE_KEY']}"},
    json={
        "mode": "rows",
        "data": "email,name\na@x.com,Ann\na@x.com,Ann B",
        "options": {"keyColumns": [0], "hasHeader": True},
    },
    timeout=30,
)
print(res.json())

# {'result': 'email,name\na@x.com,Ann', 'kept': 1, 'removed': 1}
```

### go

```go
package main

import (
	"bytes"
	"fmt"
	"io"
	"net/http"
	"os"
)

func main() {
	body := []byte(`{"mode":"rows","data":"email,name\na@x.com,Ann\na@x.com,Ann B","options":{"keyColumns":[0],"hasHeader":true}}`)

	req, err := http.NewRequest("POST", "https://dedupe.dev/api/dedupe", bytes.NewReader(body))
	if err != nil {
		panic(err)
	}
	req.Header.Set("Authorization", "Bearer "+os.Getenv("DEDUPE_KEY"))
	req.Header.Set("Content-Type", "application/json")

	res, err := http.DefaultClient.Do(req)
	if err != nil {
		panic(err)
	}
	defer res.Body.Close()

	out, _ := io.ReadAll(res.Body)
	fmt.Println(res.StatusCode, string(out))
}

// 200 {"result":"email,name\na@x.com,Ann","kept":1,"removed":1}
```

### php

```php
<?php
$payload = json_encode([
    "mode" => "rows",
    "data" => "email,name\na@x.com,Ann\na@x.com,Ann B",
    "options" => ["keyColumns" => [0], "hasHeader" => true],
]);

$ch = curl_init("https://dedupe.dev/api/dedupe");
curl_setopt_array($ch, [
    CURLOPT_POST => true,
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_HTTPHEADER => [
        "Authorization: Bearer " . getenv("DEDUPE_KEY"),
        "Content-Type: application/json",
    ],
    CURLOPT_POSTFIELDS => $payload,
]);
$res = curl_exec($ch);
curl_close($ch);
echo $res;

# {"result":"email,name\na@x.com,Ann","kept":1,"removed":1}
```

## Retrying safely

The same backoff in three languages. Each one distinguishes the burst `429`, which is worth retrying, from the quota `429`, which is not.

```js
async function dedupe(body, attempt = 0) {
  const res = await fetch("https://dedupe.dev/api/dedupe", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.DEDUPE_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify(body),
  });
  if (res.status !== 429 || attempt >= 5) return res;

  // Retry-After is only sent for the burst limiter. The monthly-quota 429
  // carries "resets" in the body instead, and retrying will not help.
  const after = Number(res.headers.get("retry-after"));
  if (!after) throw new Error((await res.json()).error);

  await new Promise((r) => setTimeout(r, after * 1000 + 2 ** attempt * 250));
  return dedupe(body, attempt + 1);
}
```

```python
import os, time, requests

def dedupe(body, attempts=5):
    for attempt in range(attempts):
        res = requests.post(
            "https://dedupe.dev/api/dedupe",
            headers={"Authorization": f"Bearer {os.environ['DEDUPE_KEY']}"},
            json=body,
            timeout=30,
        )
        if res.status_code != 429:
            return res
        # Only the burst limiter sends Retry-After; a monthly-quota 429 does not.
        after = res.headers.get("Retry-After")
        if after is None:
            raise RuntimeError(res.json()["error"])
        time.sleep(int(after) + 2 ** attempt * 0.25)
    return res
```

```bash
#!/usr/bin/env bash
set -euo pipefail

for attempt in 0 1 2 3 4; do
  code=$(curl -s -o /tmp/dedupe.json -D /tmp/dedupe.hdr -w '%{http_code}' \
    -X POST https://dedupe.dev/api/dedupe \
    -H "Authorization: Bearer $DEDUPE_KEY" \
    -H "Content-Type: application/json" \
    --data-binary @payload.json)

  [ "$code" = "200" ] && break
  [ "$code" = "429" ] || { echo "failed: $code $(cat /tmp/dedupe.json)" >&2; exit 1; }

  # No Retry-After means the monthly quota, not the burst limiter: stop.
  after=$(awk 'tolower($1)=="retry-after:"{print $2+0}' /tmp/dedupe.hdr)
  [ -n "$after" ] || { cat /tmp/dedupe.json >&2; exit 1; }
  sleep $((after + 2 ** attempt))
done

jq -r '.result' /tmp/dedupe.json
```

## What this API is not

Scoping, honestly, so you do not integrate it and then find out. This endpoint removes exact duplicates from text lines, CSV rows, and JSON arrays. That is the whole product.

- **Fuzzy matching is bounded, not unlimited.** `rows` mode takes `fuzzy: true` with a similarity threshold, capped at 50,000 data rows per request (the paid per-request unit cap), and it needs an API key (the free one counts). It is record dedupe with normalization, blocking and edit distance, not probabilistic entity resolution: no training, no scoring model, no cross-request memory.
- **It does not deduplicate photos, files, or music.** There is no image hashing here. For duplicate photos see the [photo guides](/guides), which point at tools that read your device.
- **It does not connect to anything.** No CRM, no Google Sheets, no Salesforce, no OAuth, no webhooks. You send it data, it sends data back. Anything that fetches from your systems is code you write.
- **It does not store your data, which also means it cannot remember it.** There is no persistent index of what you have already sent, so it cannot tell you that a row you are uploading today duplicates one from last month. Deduplication is within a single request.
- **It does not merge records.** Duplicates are dropped, not combined. The first occurrence survives untouched and the rest are discarded, so if the second copy had the phone number the first was missing, that phone number is gone.
- **It is not an async job API.** One request, one response, bounded by your plan's size and unit caps. There is nothing to poll.

And the case for not using it at all: **if your data already lives in SQL or a dataframe, you probably do not need this.** `SELECT DISTINCT`, a `GROUP BY`, a window function over `ROW_NUMBER()`, or `df.drop_duplicates()` will be faster, free, and one less dependency. We wrote [the SQL guide](/guides/remove-duplicates-sql) and [the pandas guide](/guides/remove-duplicates-python-pandas) partly so you can check that first. This API earns its place when the data is in flight rather than at rest: a webhook payload, a file upload, a shell pipeline, a serverless function with no database to lean on.

And if you just have a file on your desk, do not write code at all. [The browser tools](/) are free, unmetered, and never upload anything.

## Frequently asked questions

**Do I need an API key?**

Not to try it. The endpoint answers unauthenticated requests on a small per-IP daily budget (500 units). Create a free key at /app/keys for 1,000 units a month, and upgrade if you need more: Starter is 100,000 units a month at $19, Pro 300,000 at $49.

**What is a unit?**

One row of up to 1 KB. A request costs max(rows, bytes ÷ 1024) units, so a 50,000-row CSV costs 50,000 units whether you send it in one call or fifty, and a single very large row is billed by its size. Only the API is metered. The in-browser tools are free and unlimited.

**What formats does it accept?**

Three modes: lines (a string of text lines), rows (a CSV string with optional key columns), and json (an array, optionally deduped by key path).

**Does the API do fuzzy matching?**

Yes, in rows mode: set options.fuzzy to true and names like "Smith, Jane" and "jane smith" match after normalization for case, punctuation and word order, with a similarity threshold you control (default 0.9). It requires an API key (any tier, including free) and handles up to 50,000 data rows per request, the same ceiling as the paid per-request unit cap. The browser tools are exact-only.

**Can the API validate email addresses?**

Yes: POST {"emails": [...]} to /api/validate for hygiene-grade checks (syntax, MX presence, disposable domains, role accounts) at 2 units per address against the same monthly allowance. It is not SMTP-grade verification and the docs say so plainly: no mailbox pinging, no catch-all detection. An API key is required; any tier works.

**Is my data stored?**

No. The function processes the request in memory and returns the result; nothing is written to storage or logs beyond standard error metadata. Keyed calls record usage counts, mode, status and timing, never request bodies.

**What happens when I hit a limit?**

Over your plan's per-request size or unit cap you get a 413. Past your monthly units you get a 429 carrying limit, used and resets. Too many requests too fast gets a 429 carrying a Retry-After header, which is the only case worth retrying.