Filtering Duplicates by ID (and Keeping the Rest!) in JavaScript

Smarter Data Handling:

Working with data arrays in JavaScript, especially when fetched from APIs or user input, often means dealing with imperfections like duplicate entries or missing information. A common challenge is ensuring uniqueness b…


This content originally appeared on DEV Community and was authored by groundhog

Smarter Data Handling:

Working with data arrays in JavaScript, especially when fetched from APIs or user input, often means dealing with imperfections like duplicate entries or missing information. A common challenge is ensuring uniqueness based on a specific property, like an id. But what if some items legitimately don't have an id, and you still want to include them in your final list?

This post dives into a robust JavaScript technique to filter out objects with duplicate IDs, while explicitly preserving any objects that don't have an id property (or whose id is null or undefined).

The Duplicate ID Dilemma

Imagine you have an array of data like this:

const rawData = [
  { id: 1, name: 'Product A' },
  { name: 'Uncategorized Item 1' }, // No ID!
  { id: 2, name: 'Product B' },
  { id: 1, name: 'Product A - Updated' }, // Duplicate ID!
  { id: null, name: 'Null ID Item' },     // ID is null!
  { id: 3, name: 'Product C' },
  { name: 'Uncategorized Item 2' }, // Another item without an ID
];

Our goal is to get a clean array where:

  1. Each id (if present) appears only once.
  2. Any item without an id (or with null/undefined as an id) is always included, regardless of other items.

This requires a slightly more nuanced approach than a simple duplicate filter.

The filter() + Set Strategy (with a Twist)

The most effective way to solve this is by combining Array.prototype.filter() with a Set object. The Set is perfect for efficiently tracking unique values (our IDs) because checking if a value has() been added to a Set is very fast.

The "twist" comes in how we handle objects that lack an id.

Let's break down the JavaScript implementation:

function getUniqueItemsPreservingNoId(arr) {
  // We use a Set to keep track of IDs we've already encountered.
  const seenIds = new Set();

  return arr.filter(item => {
    const id = item.id; // Get the ID for the current item

    // --- IMPORTANT: Handle items without a valid ID first ---
    // If 'id' is falsy (undefined, null, 0, etc.), we immediately return true.
    // This ensures such items are ALWAYS included in the filtered array.
    if (!id) {
      return true; 
    }

    // --- For items WITH a valid ID, check for duplicates ---
    // Check if we've already seen this specific ID.
    const isDuplicate = seenIds.has(id);

    // If it's the first time we're seeing this ID, add it to our Set.
    if (!isDuplicate) {
      seenIds.add(id);
    }

    // Return true only if it's NOT a duplicate (i.e., it's the first instance of this ID).
    return !isDuplicate;
  });
}

const uniqueAndPreserved = getUniqueItemsPreservingNoId(rawData);

console.log(uniqueAndPreserved);

What the Result Looks Like

Running the getUniqueItemsPreservingNoId function on our rawData will yield:

[
  { id: 1, name: 'Product A' },
  { name: 'Uncategorized Item 1' }, 
  { id: 2, name: 'Product B' },
  { id: null, name: 'Null ID Item' },     
  { id: 3, name: 'Product C' },
  { name: 'Uncategorized Item 2' }
]

Notice how Product A - Updated (which had a duplicate id: 1) was filtered out, but Uncategorized Item 1, Null ID Item, and Uncategorized Item 2 (all lacking a standard id) were kept.

How It Works: The Breakdown

  1. seenIds = new Set(): We initialize an empty Set. This will store all the unique id values we encounter.
  2. arr.filter(item => { ... }): The filter method iterates over each item in our rawData array. If the callback function returns true, the item is kept; if false, it's discarded.
  3. if (!id) { return true; }: This is the critical line.
    • It checks if item.id is a "falsy" value (i.e., undefined, null, 0, '', false).
    • If id is undefined or null (meaning no valid ID exists), we immediately return true. This tells filter to keep this item without any further duplicate checks.
  4. seenIds.has(id): For items that do have a valid id, we check if our seenIds Set already contains this id.
  5. seenIds.add(id): If the id hasn't been seen before (isDuplicate is false), we add it to our Set.
  6. return !isDuplicate;: Finally, for items with an id, we return true only if it's not a duplicate (i.e., it's the first time we've encountered this specific ID).

Connecting the dots...

This filter() and Set combination provides a clean, efficient, and readable way to handle a common data manipulation task. By adding the initial check for !id, you gain the flexibility to enforce uniqueness where it matters (for items with IDs) while gracefully preserving all other data points. Master this pattern, and your data processing logic will be much more robust!


This content originally appeared on DEV Community and was authored by groundhog


Print Share Comment Cite Upload Translate Updates
APA

groundhog | Sciencx (2025-10-21T02:03:50+00:00) Filtering Duplicates by ID (and Keeping the Rest!) in JavaScript. Retrieved from https://www.scien.cx/2025/10/21/filtering-duplicates-by-id-and-keeping-the-rest-in-javascript/

MLA
" » Filtering Duplicates by ID (and Keeping the Rest!) in JavaScript." groundhog | Sciencx - Tuesday October 21, 2025, https://www.scien.cx/2025/10/21/filtering-duplicates-by-id-and-keeping-the-rest-in-javascript/
HARVARD
groundhog | Sciencx Tuesday October 21, 2025 » Filtering Duplicates by ID (and Keeping the Rest!) in JavaScript., viewed ,<https://www.scien.cx/2025/10/21/filtering-duplicates-by-id-and-keeping-the-rest-in-javascript/>
VANCOUVER
groundhog | Sciencx - » Filtering Duplicates by ID (and Keeping the Rest!) in JavaScript. [Internet]. [Accessed ]. Available from: https://www.scien.cx/2025/10/21/filtering-duplicates-by-id-and-keeping-the-rest-in-javascript/
CHICAGO
" » Filtering Duplicates by ID (and Keeping the Rest!) in JavaScript." groundhog | Sciencx - Accessed . https://www.scien.cx/2025/10/21/filtering-duplicates-by-id-and-keeping-the-rest-in-javascript/
IEEE
" » Filtering Duplicates by ID (and Keeping the Rest!) in JavaScript." groundhog | Sciencx [Online]. Available: https://www.scien.cx/2025/10/21/filtering-duplicates-by-id-and-keeping-the-rest-in-javascript/. [Accessed: ]
rf:citation
» Filtering Duplicates by ID (and Keeping the Rest!) in JavaScript | groundhog | Sciencx | https://www.scien.cx/2025/10/21/filtering-duplicates-by-id-and-keeping-the-rest-in-javascript/ |

Please log in to upload a file.



There are no updates yet.
Click the Upload button above to add an update.

You must be logged in to translate posts. Please log in or register.

Related Posts