How to Remove Duplicates from a JSON Array with jq

All jq tools and guides

Use unique to remove repeated values from an array, or unique_by(.id) to deduplicate objects by a field. Both sort their results. If input order matters, use the order-preserving example below.

Paste the filter into the playground, or run it in a Unix shell as jq 'FILTER' input.json. Each example below includes its own input and expected output.

Remove duplicate strings or numbers

Input:

[
  "web",
  "api",
  "web",
  "worker",
  "api"
]

Filter:

unique

Output:

[
  "api",
  "web",
  "worker"
]

Run this example in the jq playground

This is useful when an API returns repeated tags. The same filter works for numbers: [3, 1, 3, 2] becomes [1, 2, 3]. The result is sorted, rather than kept in arrival order.

Remove duplicate objects by a field

Input:

[
  {
    "id": 2,
    "name": "Lin"
  },
  {
    "id": 1,
    "name": "Ada"
  },
  {
    "id": 2,
    "name": "Lin"
  }
]

Filter:

unique_by(.id)

Output:

[
  {
    "id": 1,
    "name": "Ada"
  },
  {
    "id": 2,
    "name": "Lin"
  }
]

Run this example in the jq playground

Use this when the ID defines a duplicate, even if objects contain other fields. Do not use this filter to choose the newest record: make your conflict-resolution rule explicit when records with the same ID disagree. Objects without an id and objects with id: null share the same grouping value.

Keep the first occurrence and preserve order

Input:

[
  {
    "id": 2,
    "name": "Lin"
  },
  {
    "id": 1,
    "name": "Ada"
  },
  {
    "id": 2,
    "name": "Lin updated"
  }
]

Filter:

reduce .[] as $item ([]; if any(.[]; .id == $item.id) then . else . + [$item] end)

Output:

[
  {
    "id": 2,
    "name": "Lin"
  },
  {
    "id": 1,
    "name": "Ada"
  }
]

Run this example in the jq playground

This keeps the first object for each ID and retains arrival order. It scans the accumulated result for each item, so prefer the simpler sorted version for large arrays when order does not matter. For scalar values, replace .id == $item.id with . == $item.

Deduplicate an array inside an object

Input:

{
  "tags": [
    "web",
    "api",
    "web"
  ],
  "total": 3
}

Filter:

.tags |= unique

Output:

{
  "tags": [
    "api",
    "web"
  ],
  "total": 3
}

Run this example in the jq playground

Update assignment changes only tags. Other fields remain unchanged, including total: recalculate counts separately if they describe the deduplicated array.

Related guides

Reference: official jq manual.