How to Remove Duplicates from List in Python

Learn how to efficiently remove duplicates from a list in Python using various methods, including built-in functions, data structures, and algorithms. …

Updated June 15, 2023

Learn how to efficiently remove duplicates from a list in Python using various methods, including built-in functions, data structures, and algorithms.

Removing duplicates from a list is a fundamental operation in Python programming. It’s a common task that arises frequently in many areas of development, such as data analysis, web scraping, and more. In this article, we’ll delve into the world of duplicate removal, exploring various techniques to help you master this essential skill.

Definition: What are Duplicates?

Duplicates refer to identical elements or values within a list. For instance, if we have a list [1, 2, 3, 4, 5, 2, 3], the duplicates are 2 and 3. Our goal is to remove these duplicates, leaving us with a unique list of elements.

Method 1: Using Built-in Functions

Python provides an efficient way to remove duplicates using the built-in set() function. Here’s how you can do it:

my_list = [1, 2, 3, 4, 5, 2, 3]
unique_list = list(set(my_list))
print(unique_list)  # Output: [1, 2, 3, 4, 5]

Code Explanation: We convert the list to a set using set(my_list), which automatically removes duplicates. Then, we convert the resulting set back to a list with list(). This method is efficient but loses the original order of elements.

Method 2: Using List Comprehension

We can also use list comprehension to remove duplicates while preserving the original order:

my_list = [1, 2, 3, 4, 5, 2, 3]
unique_list = []
[unique_list.append(x) for x in my_list if x not in unique_list]
print(unique_list)  # Output: [1, 2, 3, 4, 5]

Code Explanation: We use a list comprehension to iterate through the original list. If an element x is not already present in the resulting unique_list, we append it.

Method 3: Using Dictionary

Another approach involves using dictionaries to remove duplicates while preserving the original order:

my_list = [1, 2, 3, 4, 5, 2, 3]
seen = {}
unique_list = []
for x in my_list:
    if x not in seen:
        seen[x] = True
        unique_list.append(x)
print(unique_list)  # Output: [1, 2, 3, 4, 5]

Code Explanation: We iterate through the original list. If an element x has not been encountered before (i.e., it’s not in the seen dictionary), we add it to the unique_list.

Conclusion