Python

How to find duplicate rows in a Dataframe based on all or selected columns using DataFrame.duplicate() in Python ?

In this article we will discuss about how we can find duplicate rows in a Dataframe based on all or selected columns using DataFrame.duplicated(). So first let’s know about this duplicated() function then we will see how it actually works.

DataFrame.duplicated()

Python’s Pandas library contains DataFrame class which provides a function i.e. duplicated() that helps in finding duplicate rows based on specific or all columns.

Synatx : DataFrame.duplicated (subset='None', keep='first')

where,

• subset : It represents single or multiple column labels which will be used for duplication check. If it is not provided then all columns will be checked for finding duplicate rows.
• keep : It represents the occurrence which needs to be marked as duplicate. Its value can be (first : here all duplicate rows except their first occurrence are returned and it is the default value also, last : here all duplicate rows except their last occurrence are returned and false : here all duplicate rows except occurrence are returned)

Find Duplicate rows based on all columns :

To find all the duplicate rows based on all columns, we should not pass any argument in subset while calling DataFrame.duplicate(). If any duplicate rows found, True will be returned at place of the duplicated rows expect the first occurrence as default value of keep argument is first.

import pandas as sc

# List of Tuples

players = [('MI', 'Surya', 487),

('RR', 'Buttler', 438),

('CSK', 'Jadeja', 456),

('CSK', 'Jadeja', 456),

('KKR', 'Gill', 337),

('SRH', 'Roy', 241),

('DC', 'Rahane', 221),

('CSK', 'Dhoni', 446),

('PK', 'Malan', 298)

]

# To create a DataFrame object

dfObjs = sc.DataFrame(players, columns=['Team', 'Player', 'Runs'])

# To select duplicate rows based on all columns except the first occurrence

dupliRows = dfObjs[dfObjs.duplicated()]

print("Duplicate rows based on all column excluding first occurrence is:")

print(dupliRows)

Output :
Duplicate rows based on all column excluding first occurrence is:
Team  Player  Runs
3  CSK  Jadeja   456

In the above example all duplicate values returned except the first occurrence, because the by default value of keep is first.

Note : If we make keep argument as last then while finding the duplicate rows last occurrence will be ignored.

Find Duplicate Rows based on selected columns :

If we want to find duplicate compare rows on selected column, then we should pass the columns names as argument in duplicate(), which will return the duplicate rows based on passed or selected columns. Similarly in this case also first occurrence is ignored.

#Program :

import pandas as sc

# List of Tuples

players = [('MI', 'Surya', 487),

('RR', 'Buttler', 438),

('DC', 'Pant', 337),

('CSK', 'Dhoni', 456),

('KKR', 'Gill', 337),

('SRH', 'Roy', 241),

('DC', 'Rahane', 337),

('DC', 'Iyer', 337),

('PK', 'Malan', 298)

]

# To create a DataFrame object

dfObjs = sc.DataFrame(players, columns=['Team', 'Player', 'Runs'])

# Select all duplicate rows based on one column

# To select the duplicated rows based on column that is passed as argument

dupliRows = dfObjs[dfObjs.duplicated(['Team','Runs'])]

print("Duplicate Rows based on a selected column are:", dupliRows, sep='\n')

Output :
Duplicate Rows based on a selected column are:
     Team   Player      Runs
6    DC      Rahane    337
7    DC      Iyer          337

By default value of keep is first, so only matched first row is ignored. Here, we have found rows based on selected columns. In this example we have selected  columns (Team and Runs) based on which 3 rows matches i.e

'DC', 'Pant', 337
'DC', 'Rahane', 337
'DC', 'Iyer', 337

Out of which last two rows are displayed and first row is ignored as the keep value is first(default value).

So it returned last two rows as output i.e

'DC', 'Rahane', 337
'DC', 'Iyer', 337

Want to expert in the python programming language? Exploring Python Data Analysis using Pandas tutorial changes your knowledge from basic to advance level in python concepts.

Read more Articles on Python Data Analysis Using Padas – Find Elements in a Dataframe

• Check if a value exists in a DataFrame using in & not in operator | isin()
• Find & Drop duplicate columns in a DataFrame
• Check if a DataFrame is empty in Python
• Find maximum values & position in columns or rows of a Dataframe
• Find indexes of an element in pandas dataframe

How to set work directory in R studio ?

In this article we are going to discuss about how we can set work directory R and also we will verify if that directory has been set perfectly or not. So, let’s start exploring the topic.

R is an interpreted programming language which is created by Ross Ihaka and Robert Gentleman. This programming language and open source environment is used for statistical computing, graphical presentation, data analytics and scientific research.

R studio is an  Integrated Development Environment (IDE) which provides free and open source tools for Rn language.

When working with R language sometimes we need to work with external files and for that we have to set that file directory as working directory otherwise the file won’t be accessible.

So, let’s see how we can set a directory as an working directory in R studio.

Setting up a Working Directory in R studio :

Steps to set working directory in R studio.

1. Go to session menu.
2. Select set working directory.
3. Then select Choose Directory.
4. From there browse the directory.

Set up Working Directory in R using setwd() function :

By using setwd() function we can set the working directory.

Syntax : setwd("D:/R_workingfile")

For example, we have a directory /Users/BtechGeeks and we want to set it as  working directory then we can do it like this

# command to set working directory.
setwd("Users/BtechGeeks")

If the path does not exist :

If you want to set a directory as working directory but that directory does not exist then it will give Error.

For example, a directory /Users/Btech which does not exists but we want to make it working directory.

Then

# Directory does not exist
setwd("/Users/Btech")

# Raising error as directory does not exit
Error in setwd("/Users/Btech") : cannot change working directory

Verifying the current working directory is set in R :

We can use getwd() function to check the current working directory.

If it returns the correct directory then the working directory has been set perfectly.

# It will give the current working directory
getwd()
>> "/Users/BtechGeeks"

Read CSV into a list of lists or tuples or dictionaries | Import csv to list in Python.

In this article, we will demonstrate how we can import a CSV into a list, list of lists or a list of tuples in python. We will be using pandas module for importing CSV contents to the list without headers.

Example Dataset :

CSV File name – data.csv

Id,Name,Course,City,Session
21,Jill,DSA,Texas,Night
22,Rachel,DSA,Tokyo,Day
23,Kirti,ML,Paris,Day
32,Veena,DSA,New York,Night

Read a CSV into list of lists in python :

1. Importing csv to a list of lists using csv.reader :

CSV.reader is a python built-in function from the CSV module which will help us read the CSV file into the python. Then passing the reader object into the list() will return a list of lists.

Let’s see the implementation of it.

#Program :

from csv import reader

#Opening the csv file as a list of lists in read mode
with open('data.csv', 'r') as csvObj:
 #The object having the file is passed into the reader
 csv_reader = reader(csvObj)
 #The reader object is passed into the list( ) to generate a list of lists
 rowList = list(csv_reader)
 print(rowList)

Output :
[['Id', 'Name', 'Course', 'City', 'Session'], 
['21', 'Jill', 'DSA', 'Texas', 'Night'], 
['22', 'Rachel', 'DSA', 'Tokyo', 'Day'], 
['23', 'Kirti', 'ML', 'Paris', 'Day'], 
['32', 'Veena', 'DSA', 'New York', 'Night']]

2. Selecting specific value in csv by specific row and column number :

 We can also select particular rows and columns from the CSV file by using Pandas. We have to read the CSV into a dataframe excluding the header and create a list of lists.

Let’s see the implementation of it.

#Program :

import pandas as pd

# Create a dataframe from the csv file
dfObj = pd.read_csv('data.csv', delimiter=',')
# User list comprehension 
# for creating a list of lists from Dataframe rows
rowList = [list(row) for row in dfObj.values]
# Print the list of lists i.e. only rows without the header
print(rowList)

Output :
[[21, 'Jill', 'DSA', 'Texas', 'Night'], 
[22, 'Rachel', 'DSA', 'Tokyo', 'Day'], 
[23, 'Kirti', 'ML', 'Paris', 'Day'], 
[32, 'Veena', 'DSA', 'New York', 'Night']]

3. Using Pandas to read csv into a list of lists with header :

To include the header row, we can first read the other rows like the previous example and then add the header to the list.

Let’s see the implementation of it.

#Program :

import pandas as pd

# Create a dataframe from the csv file
dfObj = pd.read_csv('data.csv', delimiter=',')
# User list comprehension 
# for creating a list of lists from Dataframe rows
rowList = [list(row) for row in dfObj.values]
#Adding the header
rowList.insert(0, dfObj.columns.to_list())
# Print the list of lists with the header
print(rowList)

Output :
[['Id', 'Name', 'Course', 'City', 'Session'], 
[21, 'Jill', 'DSA', 'Texas', 'Night'], 
[22, 'Rachel', 'DSA', 'Tokyo', 'Day'], 
[23, 'Kirti', 'ML', 'Paris', 'Day'], 
[32, 'Veena', 'DSA', 'New York', 'Night']]

Reading csv into list of tuples using Python :

Let’s add the contents of CSV file as a list of tuples. Each tuple will be representing a row and each value in the tuple represents a column value. Just like the way we added the contents into a list of lists from CSV, we will read the CSV file and then pass it into list function to create a list of tuples. The only difference here is the map( ) function that accepts function and input list arguments.

Let’s see the implementation of it.

#Program :

from csv import reader
# open file in read mode
with open('data.csv', 'r') as readerObj:
    # here passing the file object to reader() to get the reader object
    csv_reader = reader(readerObj)
    #Read all CSV files into the tuples
    tuplesList = list(map(tuple, csv_reader))
    # display the list of tuples
    print(tuplesList)

Output :

[('Id', 'Name', 'Course', 'City', 'Session'), ('21', 'Jill', 'DSA', 'Texas', 'Night'), ('22', 'Rachel', 'DSA', 'Tokyo', 'Day'), ('23', 'Kirti', 'ML', 'Paris', 'Day'), ('32', 'Veena', 'DSA', 'New York', 'Night')]

Reading csv into list of tuples using pandas & list comprehension :

We can load the contents of a CSV file into a dataframe by using read_csv( ) . Then using list comprehension we can convert the 2D numpy array into a list of tuples.

Let’s see the implementation of it.

#Program :

import pandas as pd
# Create a dataframe object from the csv file
dfObj = pd.read_csv('data.csv', delimiter=',')
# Create a list of tuples for Dataframe rows using list comprehension
tuplesList = [tuple(row) for row in dfObj.values]
# Print the list of tuple
print(tuplesList)

Output :
[(21, 'Jill', 'DSA', 'Texas', 'Night'), (22, 'Rachel', 'DSA', 'Tokyo', 'Day'), (23, 'Kirti', 'ML', 'Paris', 'Day'), (32, 'Veena', 'DSA', 'New York', 'Night')]

Reading csv into list of dictionaries using python :

We can also read the contents of a CSV file into dictionaries in python where each dictionary in the list will be a row from the CSV file. The CSV file contents are opened in read mode then they are passed into the Dict_reader( ) as a reader object, then it is passed into the list.

Let’s see the implementation of it.

#Program :

from csv import DictReader
# open file in read mode
with open('data.csv', 'r') as readerObj:
    # pass the reader file object to DictReader() to get the DictReader object
    dict_reader = DictReader(readerObj)
    # get a list of dictionaries from dct_reader
    dictList = list(dict_reader)
    # print the list of dict
    print(dictList)

Output :

[OrderedDict([('Id', '21'), ('Name', 'Jill'), ('Course', 'DSA'), ('City', 'Texas'), ('Session', 'Night')]), OrderedDict([('Id', '22'), ('Name', 'Rachel'), ('Course', 'DSA'), ('City', 'Tokyo'), ('Session', 'Day')]), OrderedDict([('Id', '23'), ('Name', 'Kirti'), ('Course', 'ML'), ('City', 'Paris'), ('Session', 'Day')]), OrderedDict([('Id', '32'), ('Name', 'Veena'), ('Course', 'DSA'), ('City', 'New York'), ('Session', 'Night')])]

Opening a file using ‘open with’ statement and benefits in Python.

In this article we will discuss about how to open a file using ‘open with’ statement, how to open multiple files in a single ‘open with’ statement and finally its benefits. So, let’s start the topic.

The need for “open with” statement :

To understand the “open with” statement we have to go through opening a file in python. For that we can make use of the open( ) function that is in-built in python

File.txt-

New File Being Read.
DONE!!

#program :

# opened a file 
fileObj = open('file.txt')
# Reading the file content into a placeholder
data = fileObj.read()
# print file content
print(data)
#close the file
fileObj.close()

Output :
New File Being Read.
DONE!!

In case the file does not exist it will throw a FileNotFoundError .

How to open a file using “open with” statement in python :

#Program :

# opened a file using open-with
with open('file.txt', "r") as fileObj:
    # Reading the file content into a placeholder
    data = fileObj.read()
    # print file content
    print(data)
# Check if file is closed
if fileObj.closed == False:
    print('File is not closed')
else:
    print('File is already closed')

New File Being Read.
DONE!!
File is closed

The with statements created an execution block that will automatically delete any object that was created in the program, in this case even if it was not closed the reader object was deleted that closed the file automatically. This saves us some memory in case we forgot to close the file.

Benefits of calling open() using “with statement” :

• Fewer chances of bug due to coding error

With “with” statement we don’t have to close the opened file manually. It takes care of that when the compiler goes out of the block and automatically closes file. So it reduces the chances of bugs, lines of code and releases the memory for other operations.

• Excellent handling in case of exception

If we have used “open-with” statement to open a file, and an exception occurs inside the with block, the file will be closed and the control moves to the except block.

# Python :

# Before handling the exception file will be closed 
try:
    # using "with statement" with open() function
    with open('file.txt', "r") as fileObj:
        # reading the file content
        data = fileObj.read()
        # Division by zero error
        x = 1 / 0
        print(data)
except:
    # handling the exception caused above
    print('Error occurred')
    if fileObj.closed == False:
        print('File is not closed')
    else:
        print('File is closed')

Output :
Error occurred
File is closed

• Open multiple files in a single “with statement” :

We can use open with statement to open multiple files at the same time. Let’s try reading from one file and writing into another-

# Program :

# Read from file.txt and write in output.txt
with open('output.txt', 'w') as fileObj2, open('file.txt', 'r') as fileObj1:
    data = fileObj1.read()
    fileObj2.write(data)
    # Both the files are automatically close when the control moves out of the with block.

This will generate a “outuput.txt” file that will have the same contents as our old “file.txt”.

Output : 
Output.txt- 
New File Being Read. 
DONE !!

The files will automatically close when the control moves outside the with block.