Python : How to Remove Duplicates from a List

A collection is an ordered list of values. There could be various types of values. A list is a mutable container. This means that existing ones can be added to, deleted from, or changed.

The Python list represents the mathematical concept of a finite sequence. List values are referred to as list items or list elements. The same value may appear multiple times in a list. Each event is regarded as a distinct element.

Example:

Input:

givenlist = ['hello', 'this', 'is', 'hello', 'BTechGeeks', 'is', 'this']

Output:

Given list with duplicates :  ['hello', 'this', 'is', 'hello', 'BTechGeeks', 'is', 'this']
Given list without duplicates :  ['hello', 'this', 'is', 'BTechGeeks']

Delete Duplicates elements from the list

We can delete duplicate elements by many methods some of them are:

Using set
Using in operator and lists (same order)
Using Counter() function (same order)

Method #1:Using set

A set is an unordered data structure with only unique elements.
So, using the set() function, convert the given list to a set and then back to a list.
Print the list

Below is the implementation:

# given list (which contains duplicates)
givenlist = ['hello', 'this', 'is', 'hello', 'BTechGeeks', 'is', 'this']
print('Given list with duplicates : ', givenlist)
# converting givenlist to set and then back to list to remove duplicates
givenlist = list(set(givenlist))
# print given list
print('Given list without duplicates : ', givenlist)

Output:

Given list with duplicates :  ['hello', 'this', 'is', 'hello', 'BTechGeeks', 'is', 'this']
Given list without duplicates :  ['this', 'is', 'hello', 'BTechGeeks']

Note:

While this method avoids duplicates, it does not hold the elements in the same order as the original.

Method #2:Using in operator and lists (same order)

For unique elements, we’ll need to make a new list. Then iterate through the original list, adding each element to the newList only if it isn’t already there.

Below is the implementation:

# given list (which contains duplicates)
givenlist = ['hello', 'this', 'is', 'hello', 'BTechGeeks', 'is', 'this']
print('Given list with duplicates : ', givenlist)
# taking empty list
newlist = []
# Travers the given list
for element in givenlist:
    # if the element is not present in new list then append it
    if element not in newlist:
        newlist.append(element)
# print new list
print('Given list without duplicates : ', newlist)

Output:

Given list with duplicates :  ['hello', 'this', 'is', 'hello', 'BTechGeeks', 'is', 'this']
Given list without duplicates :  ['hello', 'this', 'is', 'BTechGeeks']

Method #3:Using Counter() function (same order)

Counter() function is mainly used to calculate the frequencies of all elements .
In this case we calculate the frequencies using Counter function which returns dictionary having unique keys.
Convert this keys to list and print it.

Below is the implementation:

from collections import Counter
# given list (which contains duplicates)
givenlist = ['hello', 'this', 'is', 'hello', 'BTechGeeks', 'is', 'this']
print('Given list with duplicates : ', givenlist)
# counting the frequencies using Counter function
freq = Counter(givenlist)
# converting keys to list
uniquelist = list(freq.keys())
# print unqiue list
print('Given list without duplicates : ', uniquelist)

Output:

Given list with duplicates :  ['hello', 'this', 'is', 'hello', 'BTechGeeks', 'is', 'this']
Given list without duplicates :  ['hello', 'this', 'is', 'BTechGeeks']

Related Programs: