Combine two Lists and remove Duplicates in Python

Last updated: Apr 10, 2024
4 min

#Combine two lists and remove duplicates in Python

To combine two lists and remove duplicates:

1. Use the `set()` class to convert the lists to `set` objects.
2. Get the difference between the sets.
3. Use the `list()` class to convert the result back to a list.
4. Use the addition (+) operator to combine the two lists.
main.py
```Copied!```list1 = [1, 4, 6, 9]
list2 = [4, 6, 14, 7]

result = list1 + list(set(list2) - set(list1))
print(result)  # ๐๏ธ [1, 4, 6, 9, 14, 7]
``````

If you need a solution that doesn't involve `set` objects, scroll down to the next subheading.

The first step is to use the set() class to convert the lists to `set` objects.

main.py
```Copied!```list1 = [1, 4, 6, 9]
list2 = [4, 6, 14, 7]

print(set(list1))  # ๐๏ธ {1, 4, 6, 9}
print(set(list2))  # ๐๏ธ {4, 7, 6, 14}
``````

Set objects are an unordered collection of unique elements and implement a `difference()` method.

#Using subtraction vs using `set.difference()`

The following 2 examples combine the two lists and remove the duplicates.

The minus sign is a shorthand for calling the `difference()` method on the `set`.

main.py
```Copied!```list1 = [1, 4, 6, 9]
list2 = [4, 6, 14, 7]

result = list1 + list(set(list2) - set(list1))
print(result)  # ๐๏ธ [1, 4, 6, 9, 14, 7]

result = list1 + list(set(list2).difference(list1))
print(result)  # ๐๏ธ [1, 4, 6, 9, 7, 14]
``````

The difference() method returns a new `set` with elements in the `set` that are not in the provided iterable.

In other words, `set(list2).difference(list1)` returns a new `set` that contains the items in `list2` that are not in `list1`.

main.py
```Copied!```list1 = [1, 4, 6, 9]
list2 = [4, 6, 14, 7]

print(set(list2).difference(list1))  # ๐๏ธ {7, 14}
``````

The last step is to convert the `set` to a list and use the addition (+) operator to combine the two lists.

main.py
```Copied!```list1 = [1, 4, 6, 9]
list2 = [4, 6, 14, 7]

result = list1 + list(set(list2) - set(list1))
print(result)  # ๐๏ธ [1, 4, 6, 9, 14, 7]

result = list1 + list(set(list2).difference(list1))
print(result)  # ๐๏ธ [1, 4, 6, 9, 7, 14]
``````

When the addition (+) operator is used with two lists, it combines the lists into a single list.

main.py
```Copied!```print(['a'] + ['b']) # ๐๏ธ ['a', 'b']
``````

#Combine two lists and remove duplicates using `numpy`

If you use `numpy`, you can also use the `numpy.unique()` method.

main.py
```Copied!```import numpy as np

list1 = [1, 4, 6, 9]
list2 = [4, 6, 14, 7]

result = np.unique(list1 + list2).tolist()
print(result)  # ๐๏ธ [1, 4, 6, 7, 9, 14]
``````

Make sure you have the numpy module installed to be able to run the code sample.

main.py
```Copied!```pip install numpy

# ๐๏ธ or with pip3
pip3 install numpy
``````

The numpy.unique() method returns the sorted unique elements of the provided array-like object.

The tolist method converts an array to a list.

Alternatively, you can use the `list.extend()` method

#Combine two lists and remove duplicates using a list.extend()

This is a four-step process:

1. Use the `list.copy()` method to create a copy of the first list.
2. Use a generator expression to iterate over the second list.
3. Check if each item is not present in the copied list.
4. Use the `list.extend()` method to combine the two lists.
main.py
```Copied!```list1 = [1, 4, 6, 9]
list2 = [4, 6, 14, 7]

result = list1.copy()

result.extend(item for item in list2
if item not in result)

print(result)  # ๐๏ธ [1, 4, 6, 9, 14, 7]
``````

The list.copy() method returns a shallow copy of the object on which the method was called.

main.py
```Copied!```list1 = [1, 4, 6, 9]

result = list1.copy()
print(result)  # ๐๏ธ [1, 4, 6, 9]
``````

We used a generator expression to iterate over the second list.

Generator expressions are used to perform some operation for every element or select a subset of elements that meet a condition.

On each iteration, we check if the current item is not present in the copied list and return the result.

main.py
```Copied!```list1 = [1, 4, 6, 9]
list2 = [4, 6, 14, 7]

result = list1.copy()

result.extend(item for item in list2
if item not in result)

print(result)  # ๐๏ธ [1, 4, 6, 9, 14, 7]
``````

The in operator tests for membership. For example, `x in l` evaluates to `True` if `x` is a member of `l`, otherwise, it evaluates to `False`.

The last step is to extend the copy with the items of the second list that are not in the first.

The list.extend() method takes an iterable and extends the list by appending all of the items from the iterable.

main.py
```Copied!```my_list = ['bobby']

print(my_list)  # ๐๏ธ ['bobby', 'hadz', '.', 'com']
``````
The `list.extend()` method returns None as it mutates the original list.