Python

In this Page, We are Providing Python Programming – Data Structures – List Using Square Brackets. Students can visit for more Detail and Explanation of Python Handwritten Notes Pdf.

Python Programming – Data Structures – List Using Square Brackets

List creation

The list can be created in many ways.

Using square brackets

As discussed before, the most common way of creating a list is by enclosing comma-separated values (items) between square brackets. Simply assigning a square bracket to a variable creates an empty list.

>>> a= [ ]
>>> a
[ ]
>>> type ( a ) 
< type ' list ’ >

Using other lists

A list can also be created by copying a list or slicing a list.

>>> a= [ ' spam ' , ' eggs ' , 100 , 1234 ]
>>> b=a [ : ]
>>> b
[ ' spam ' , ' eggs ' , 100 , 1234 ]
>>> c=a [ 1 : 3 ]
>>> c
[ ' eggs ' , 100 ]

List comprehension

List comprehension provides a concise way to create a list. A list comprehension consists of brackets containing an expression followed by a for clause, then zero or more for or if clauses. The result will be a new list resulting from evaluating the expression in the context of the for and if clauses that follow it.

Common applications are to make new lists where each element is the result of some operations applied to each member of another sequence or iterable or to create a sub-sequence of those elements that satisfy a certain condition. For example, creating a list of squares using an elaborate approach is as follows:

>>> squares= [ ]
>>> for x in range ( 10 ) :
. . . squares. append ( x**2 )
. . .
>>> squares
[ 0 , 1 , 4 , 9 , 16 , 25 , 36 , 49 , 64 , 81 ]

The same can be obtained using list comprehension as:

squares= [ x**2 for x in range ( 10 ) ] 
>>> squares
[ 0 , 1 , 4 , 9 , 16 , 25 , 36 , 49 , 64 , 81 ]

Alternatively, it can also be obtained using map ( ) built-in function:

squares=map ( 1ambda x : x**2 , range ( 10 ) )
>>> squares
[ 0 , 1 , 4 , 9 , 16 , 25 , 36 , 49 , 64 , 81 ]

The following list comprehension combines the elements of two lists if they are not equal:

>>> [ ( x , y ) for x in [ 1 , 2 , 3 ] for y in [ 3 , 1 , 4 ] if x !=y ]
[ ( 1 , 3 ) , ( 1 , 4 ) , ( 2 , 3 ) , ( 2 , 1 ) , ( 2 , 4 ) , ( 3 , 1 ) , ( 3 , 4 ) ]

and it’s equivalent to:

>>> combs= [ ]
>>> for x in [ 1 , 2 , 3 ] :
. . . for y in [ 3 , 1 , 4 ] :
. . . if x !=y :
. . . combs. append ( ( x , y ) )
. . . 
>>> combs
[ ( 1 , 3 ) , ( 1 , 4 ) , ( 2 , 3 ) , ( 2 , 1 ) , ( 2 , 4 ) , ( 3 , 1 ) , ( 3 , 4 ) ]

Using built-in function

The list can also be created using a built-in function list ( ). The list ( [iterable] ) function returns a list whose items are the same and in the same order as iterable items. The iterable can be either a sequence, a container that supports iteration, or an iterator object. If iterable is already a list, a copy is made and returned. If no argument is given, a new empty list is returned.

>>> list ( ( ' hi ' ' hello ' , ' bye ' ) )
[ ' hi ' , ' hello ' , ' bye ' ]
>>> list ( ' hello ' )
[ ' h ' , ' e ' , ' 1 ' , ' 1 ' , ' o ' ]
>>> list ( ( 10 , 50 , ) )
[ 10 , 50 ]
>>> list ( )
[ ]

Accessing list elements

Like string indices, list indices start at 0. To access values in a list, use the square brackets with the index or indices to obtain a slice of the list.

>>> a= [ ' spam ' , ' eggs ' , 100 , 1234 ]
>>> a [ 0 ] 
' spam ' 
>>> a [ 0 ] [ 1 ]
' p '
>>> a [ 1 : 3 ] 
[ ' eggs ' , 100 ]

In this Page, We are Providing Python Programming – List. Students can visit for more Detail and Explanation of Python Handwritten Notes Pdf.

Python Programming – List

List

Python has a number of built-in types to group together data items. The most versatile is the list, which is a group of comma-separated values (items) between square brackets. List items need not be of the same data type.

>>> a= [ ' spam ' , ' eggs ' , 100 , 1234 ]
>>> a
[ ' spam ' , ' eggs ' , 100 , 1234 ]

Python allows adding a trailing comma after the last item of list, tuple, and dictionary, There are several reasons to allow this:

• When the list, tuple, or dictionary elements spread across multiple lines, one needs to remember to add a comma to the previous line. Accidentally omitting the comma can lead to errors that might be hard to diagnose. Always adding the comma avoids this source of error (the example is given below).
• If the comma is placed in each line, it can be reordered without creating a syntax error.

The following example shows the consequence of missing a comma while creating a list.

>>> a=[
         ' hi ' , 
         ’ hello ’ 
         ' bye ’ , 
         ' tata ' , 
         ] 
>>> a
[ ' hi ' ,  ' hellobye ' ,  ' tata ' ]

This list looks like it has four elements, but it actually contains three: ‘ hi ‘,  ‘ hellobye ‘, and  ‘ tata ‘. Always adding the comma avoids this source of error.

In this Page, We are Providing Python Programming – List Methods. Students can visit for more Detail and Explanation of Python Handwritten Notes Pdf.

Python Programming – List Methods

List methods

Below are the methods of list-objects.

list . append ( x )
Add an item to the end of the list. It is same as list [ len ( list ) : len ( list ) ] = [ x ] .

>>> a= [ 66 . 25 , 333 , 333 , 1 , 1234 . 5 ]
>>> a . append ( 45 )
>>> a
[ 66 . 25 , 333 , 333 , 1 , 1234 . 5 , 45 ] 
>>> a= [ 66 . 25 , 333 , 333 , 1 , 1234 . 5 ]
>>> a [ len ( a ) : len ( a ) ]=[ 45 ]
>>> a
[ 66 . 25 , 333 , 333 , 1 , 1234 . 5 , 45 ]

list . extend ( L )
Extend a list by appending all the items of a given list. It is same as list [ len ( list ) : len ( list ) ] =L.

>>> a= [ 66 . 25 , 333 , 333 , 1 , 1234 . 5 ]
>>> b= [ 7 . 3 , 6 . 8 ]
>>> a . extend ( b )
>>> a
[ 66 . 25 , 333 , 333 , 1 , 1234 . 5 , 7 . 3 , 6 . 8 ]
>>> a= [ 66 . 25 , 333 , 333 , 1 , 1234 . 5 ]
>>> b= [ 7 . 3 , 6 . 8 ]
>>> a [ len ( a ) : len ( a ) ]=b 
>>> a
[ 66 . 25 , 333 , 333 , 1 , 1234 . 5 , 7 . 3 , 6 . 8 ]

list.insert ( i , x )
Insert an item at a given position in the list. The first argument i is the index before which an item x need to be inserted. It is same as list [i : i] = [x].

>>> a= [ 66 . 25 , 333 , 333 , 1 , 1234 . 5 ]
>>> a . insert ( 2 , 5 . 7 )
>>> a
[ 66 . 25, 333 , 5 . 7 , 333 , 1 , 1234 . 5 ]
>>> a= [ 66 . 25 , 333 , 333 , 1 , 1234 . 5 ]
>>> a [ 2 : 2 ] = [ 5 . 7 ]
>>> a
[ 66 . 25 , 333 , 5 . 7 , 333 , 1 , 1234 . 5 ]

list.index ( x [ , i [ , j ] ] )
Return the index in the list of the first occurrence of item x. In other words, it returns the smallest index k such that list [k]==x and i<=k<j. A ValueError exception is raised in absence of item x.

>>> a= [ 66 . 25 , 333 , 333 , 1 , 1234 . 5 ]
>>> a . index ( 333 )
1

list.remove ( x )
Remove the first item from the list whose value is x. An error (ValueError exception) occur in absence of item x. It is same as del list [list. index (x) ].

>>> a= [ 66 . 25 , 333 , 333 , 1 , 1234 . 5 ]
>>> a . remove ( 333 )
>>> a
[ 66 . 25 , 333 , 1 , 1234 . 5 ]
>>> a= [ 66 . 25 , 333 , 333 , 1 , 1234 . 5 ]
>>> del a [ a . index ( 333 ) ] 
>>> a
[ 66 . 25 , 333 , 1 , 1234 . 5 ]

list . pop ( [ i ] )
Remove the item at the given position i in the list, and return it. If no index is specified (defaults to -1), pop () removes and returns the last item in the list.

>>> a= [ 66 . 25 , 333 , 333 , 1 , 1234 . 5 ]
>>> a . pop ( 3 )
1
>>> a
[ 66 . 25 , 333 , 333 , 1234 . 5 ]
>>> a . pop ( )
1234 . 5 
>>> a
[ 66 . 25 , 333 , 333 ]

list.count ( x )
Return the number of times item x appears in the list.

>>> a= [ 66 . 25 , 333 , 333 , 1 , 1234 . 5 ]
>>> a . count ( 333 )
2

list.reverse ( )
Reverse the element’s position in the list; no new list is returned. It is same as list=list [: : -1 ].

>>> a= [ 66 . 25 , 333 , 333 , 1 , 1234 . 5 ]
>>> a . reverse ( )
>>> a 
[ 1234 . 5 , 1 , 333 , 333 , 66 . 25 ] 
>>> a= [ 66 . 25 , 333 , 333 , 1 , 1234 . 5 ]
>>> a=a [ : : -1 ] 
[ 1234 . 5 , 1 , 333 , 333 , 66 . 25 ]

list.sort ( [ cmp [ , key [ , reverse ] ] ] )
Sort the items of the list; no new list is returned. The optional arguments have same meaning as given in sorted () built-in function.

>>> a= [ 66 . 25 , 333 , 333 , 1 , 1234 . 5 ]
>>> a . sort ( ) 
>>> a
[ 1 , 66 . 25 , 333 , 333 , 1234 . 5 ]
>>> a= [ 66 . 25 , 333 , ' abc ' , 333 , 1 , ' ab ' , 1234 . 5 ] 
>>> a . sort ( )
>>> a
[ 1 , 66 . 25 , 333 , 333 , 1234 . 5 , ' ab ' , ' abc ' ]
>>> a= [ 66 . 25 , 333 , ' abc ' , 333 , 1 , ' ab ' , 1234 . 5 ]
>>> a . sort ( reverse=True )
>>> a
[ ' abc ' , ' ab ' , 1234 . 5 , 333 , 333 , 66 . 25 , 1 ]

Using list as Stack

The list methods make it very easy to use a list as a stack, where the last element added is the first element retrieved (“last-in, first-out” approach). To add an item to the top of the stack, use append (). To retrieve an item from the top of the stack, use pop ( ) without an explicit index. For example:

>>> stack= [ 3 , 4 , 5 ]
>>> stack . append ( 6 )
>>> stack . append ( 7 )
>>> stack
[ 3 , 4 , 5 , 6 , 7 ]
>>> stack . pop ( )
7
>>> stack [ 3 , 4 , 5 , 6 ]
>>> stack . pop ( )
6
>>> stack . pop ( )
5
>>> stack 
[ 3 , 4 ]

Using list as queue

It is also possible to use a list as a queue, where the first element added is the first element retrieved (“first-in, first-out”); however, list is not efficient for this purpose. While appending and popping of elements from the end of list are fast, doing inserting and popping from the beginning of a list is slow (because all of the other elements have to be shifted by one).

To implement a queue, use collections. deque which was designed to have fast appends and pops from both ends. For example:

>>> from collections import deque
>>> queue=deque ( [ " Eric " , " John " , " Michael " ] )
>>> queue . append ( " Terry " )
>>> queue . append ( " Graham " )
>>> queue 
deque ( [ ' Eric ' , ' John ' , ' Michael ' , ' Terry ' , ' Graham ' ] ) 
>>> queue . popleft ( )
' Eric '
>>> queue . popleft ( ) 
' John '
>>> queue
deque ( [ ' Michael ' , ' Terry ' , ' Graham ' ] )

Python’s implementation of an associative array, which is a data structure, is dictionaries. A dictionary is a collection of key-value pairs. Each key-value pair represents a key and its associated value.

Enclosing a comma-separated list of key-value pairs in curly braces defines a dictionary. A colon ‘ : ‘ separates each key from its associated value.

In this post, we’ll look at how to use the update() method of the dict class in Python, as well as some examples of how to use it.

Tutorial and Examples for the update() function

• Updating the value of a key in a Python dictionary
• Using update function if key is not in dictionary
• Updating Multiple keys in dictionary
• Update/Modify the key name in python dictionary

The update() method in Python Dictionary updates the dictionary with elements from another dictionary object or an iterable of key/value pairs.

Syntax:

dictioanry_name.update(iterable)

Parameters:

 As parameters, this method accepts a dictionary or an iterable object of key/value pairs (typically tuples).

Return:

It does not return a value, but instead updates the Dictionary with elements from a dictionary object or an iterable object of key/value pairs.

If a key appears in the sequence argument but does not exist in the dictionary, it is added to the dictionary along with the given value. If the update() function is called without an argument, the dictionary is not modified.

Updating the value of a key in a Python dictionary

In the python dictionary, just build a temporary dictionary containing the new value key and move it to update() function for the update value.

Below is the implementation:

# Given Dictionary
dictionary = {'hello': 50, 'this': 100, 'is': 200, 'BTechGeeks': 300}
# updating the value of hello key to 400
dictionary.update({'hello': 400})
# printing the updated dictionary
print("Updated Dictionary : ", dictionary)

Output:

Updated Dictionary :  {'hello': 400, 'this': 100, 'is': 200, 'BTechGeeks': 300}

Using update function if key is not in dictionary

If the update() function is passed a key-value pair and the given key does not exist in the dictionary, it creates it with the given value.

Below is the implementation:

# Given Dictionary
dictionary = {'hello': 50, 'this': 100, 'is': 200, 'BTechGeeks': 300}
# updating the new key using update() function
dictionary.update({'python': 500})
# printing the updated dictionary
print("Updated Dictionary : ", dictionary)

Output:

Updated Dictionary :  {'hello': 50, 'this': 100, 'is': 200, 'BTechGeeks': 300, 'python': 500}

Updating Multiple keys in dictionary

If we want to change the values of multiple keys in the dictionary, we can use the update() function and pass them as key-value pairs. We can use a list of tuples or a temporary dictionary to bind multiple key-value pairs together.

Below is the implementation:

# Given Dictionary
dictionary = {'hello': 50, 'this': 100, 'is': 200, 'BTechGeeks': 300}
# updating the multiple keys using update() function
dictionary.update({'python': 500, 'this': 150, 'is': 250})
# printing the updated dictionary
print("Updated Dictionary : ", dictionary)

Output:

Updated Dictionary :  {'hello': 50, 'this': 150, 'is': 250, 'BTechGeeks': 300, 'python': 500}

Update/Modify the key name in python dictionary

A dictionary’s key cannot be modified. So, if we want to change the key name in the dictionary, we must remove the current key-value pair from the dictionary and replace it with a new key that has the same value.

Approach:

• Using the pop function, remove the key and save the value of the current key in a variable.
• Make a new key with the above value and a new name(key).

Below is the implementation of above approach:

# Given Dictionary
dictionary = {'hello': 50, 'this': 100, 'is': 200, 'BTechGeeks': 300}
# storing value of any key say hello in keyvalue variable after removing it
keyvalue = dictionary.pop('hello')
# updating the new key with the above value
dictionary.update({'Helloworld': keyvalue})
# printing the updated dictionary
print("Updated Dictionary : ", dictionary)

Output:

Updated Dictionary :  {'this': 100, 'is': 200, 'BTechGeeks': 300, 'Helloworld': 50}

 
Related Programs:

• python filter function tutorial and examples
• python dictionary pop function and examples
• python dictionary get function tutorial examples
• python numpy ravel function tutorial with examples
• python numpy flatten function tutorial with examples
• python min function tutorial with examples
• python numpy reshape function tutorial with examples

A sequence of characters is referred to as a string.

Characters are not used by computers instead, numbers are used (binary). Characters appear on your computer, but they are internally stored and manipulated as a sequence of 0s and 1s.

In Python, a string is a set of Unicode characters. Unicode was designed to include every character in every language and to introduce encoding uniformity to the world. Python Unicode will tell you all about Unicode you need to know.

Example:

Input:

string = "this is btech geeks"

Output:

This Is Btech Geeks

Given a string, the task is to convert each word first letter to uppercase

Capitalize the First Letter of Each Word in a String

There are several ways to capitalize the first letter of each word in a string some of them are:

• Using capitalize() function
• Using title() function
• Using string.capwords() function
• Using Regex

Method #1:Using capitalize() function

Syntax: given_string.capitalize()
Parameters: No parameters will be passed
Return : Each word’s first letter is capitalised in the string.

Approach:

• Because spaces separate all of the words in a sentence.
• We must use split to divide the sentence into spaces ().
• We separated all of the words with spaces and saved them in a list.
• Using the for loop, traverse the wordslist and use the capitalise feature to convert each word to its first letter capital.
• Using the join function, convert the wordslist to a string.
• Print the string.

Below is the implementation of above approach:

# given string
string = "this is btech geeks"
# convert the string to list and split it
wordslist = list(string.split())
# Traverse the words list and capitalize each word in list
for i in range(len(wordslist)):
  # capitizing the word
    wordslist[i] = wordslist[i].capitalize()
# converting string to list using join() function
finalstring = ' '.join(wordslist)
# printing the final string
print(finalstring)

Output:

This Is Btech Geeks

Method #2:Using title() function

Before returning a new string, the title() function in Python converts the first character in each word to Uppercase and the remaining characters in the string to Lowercase.

• Syntax: string_name.title()
• Parameters: string in which the first character of each word must be converted to uppercase
• Return Value: Each word’s first letter is capitalised in the string.

Below is the implementation:

# given string
string = "this is btech geeks"
# using title() to convert all words first letter to capital
finalstring = string.title()
# print the final string
print(finalstring)

Output:

This Is Btech Geeks

Method #3:Using string.capwords() function

Using the spilt() method in Python, the string capwords() method capitalises all of the words in the string.

• Syntax: string.capwords(given_string)
• Parameters: The givenstring that needs formatting.
• Return Value: Each word’s first letter is capitalised in the string.

Break the statement into words, capitalise each word with capitalise, and then join the capitalised words together with join. If the optional second argument sep is None or missing, whitespace characters are replaced with a single space and leading and trailing whitespace is removed.

Below is the implementation:

# importing string
import string
# given string
givenstring = "this is btech geeks"
# using string.capwords() to convert all words first letter to capital
finalstring = string.capwords(givenstring)
# print the final string
print(finalstring)

Output:

This Is Btech Geeks

Method #4:Using Regex

We’ll use regex to find the first character of each word and convert it to uppercase.

Below is the implementation:

# importing regex
import re

# function which converts every first letter of each word to capital

def convertFirstwordUpper(string):
    # Convert the group 2 to uppercase and join groups 1 and 2 together. 
    return string.group(1) + string.group(2).upper()


# given string
string = "this is btech geeks"

# using regex
resultstring = re.sub("(^|\s)(\S)", convertFirstwordUpper, string)
# print the final string
print(resultstring)

Output:

This Is Btech Geeks

Related Programs:

• capitalize first letter of each word in a string in java
• python program to read a file and capitalize the first letter of every word in the file
• convert first letter of each word of a string to upper case in cpp
• how to uppercase the first letter of a string in javascript
• check the first or last character of a string in python
• python program to remove the ith occurrence of the given word in a list where words can repeat
• python program to sort a list of tuples in increasing order by the last element in each tuple

In this article, we will be applying given function to selected rows and column.

For example, we have a dataframe object,

matrix = [(22, 34, 23),
(33, 31, 11),
(44, 16, 21),
(55, 32, 22),
(66, 33, 27),
(77, 35, 11)
]
# Create a DataFrame object
dfObj = pd.DataFrame(matrix, columns=list('xyz'), index=list('abcdef'))

Original Dataframe
    x    y   z
a 22 34 23
b 33 31 11
c 44 16 21
d 55 32 22
e 66 33 27
f 77 35 11

modDfObj = dfObj.apply(lambda x: np.square(x) if x.name == 'z' else x)
print("Modified Dataframe : Squared the values in column 'z'", modDfObj, sep='\n')

Modified Dataframe : Squared the values in column 'z'
 x y z
a 22 34 529
b 33 31 121
c 44 16 441
d 55 32 484
e 66 33 729
f 77 35 121

Method 2: Using [] operator

Using [] operator we will select the column from dataframe and apply numpy.square() method. Later, we will assign it back to the column.

dfObj['z'] = dfObj['z'].apply(np.square)

It will square all the values in column ‘z’.

Method 3: Using numpy.square()

dfObj['z'] = np.square(dfObj['z'])

This function will also square all the values in ‘z’.

Apply a function to a single row in Dataframe

Now, we saw what we have done with the columns. Same thing goes with rows. We will square all the values in row ‘b’. We can use different methods for that.

Method 1:Using Dataframe.apply()

We will apply lambda function to all the rows and will use the above function. We will check the label inside the lambda function and will square the row.

Code:

modDfObj = dfObj.apply(lambda x: np.square(x) if x.name == 'b' else x, axis=1)
print("Modified Dataframe : Squared the values in row 'b'", modDfObj, sep='\n')

Output:

Modified Dataframe : Squared the values in row 'b'
 x y z
a 22 34 23
b 1089 961 121
c 44 16 21
d 55 32 22
e 66 33 27
f 77 35 11

Method 2 : Using [] Operator

We will do what we have done above. We will select the row from dataframe.loc[] operator and apply numpy.square() method on it. Later, we will assign it back to the row.

dfObj.loc['b'] = dfObj.loc['b'].apply(np.square)

It will square all the values in the row ‘b’.

Method 3 : Using numpy.square()

dfObj.loc['b'] = np.square(dfObj.loc['b'])

This will also square the values in row ‘b’.

Apply a function to a certain columns in Dataframe

We can apply the function in whichever column we want. For instance, squaring the values in ‘x’ and ‘y’.

Code:

modDfObj = dfObj.apply(lambda x: np.square(x) if x.name in ['x', 'y'] else x)
print("Modified Dataframe : Squared the values in column x & y :", modDfObj, sep='\n')

modDfObj = dfObj.apply(lambda x: np.square(x) if x.name in ['b', 'c'] else x, axis=1)
print("Modified Dataframe : Squared the values in row b & c :", modDfObj, sep='\n')

import pandas as pd
import numpy as np
def main():
 # List of Tuples
 matrix = [(22, 34, 23),
(33, 31, 11),
(44, 16, 21),
(55, 32, 22),
(66, 33, 27),
(77, 35, 11)]
# Create a DataFrame object
 dfObj = pd.DataFrame(matrix, columns=list('xyz'), index=list('abcdef'))
print("Original Dataframe", dfObj, sep='\n')
print('********* Apply a function to a single row or column in DataFrame ********')
print('*** Apply a function to a single column *** ')
 # Method 1:
 # Apply function numpy.square() to square the value one column only i.e. with column name 'z'
 modDfObj = dfObj.apply(lambda x: np.square(x) if x.name == 'z' else x)
print("Modified Dataframe : Squared the values in column 'z'", modDfObj, sep='\n')
 # Method 2
 # Apply a function to one column and assign it back to the column in dataframe
dfObj['z'] = dfObj['z'].apply(np.square)
 # Method 3:
 # Apply a function to one column and assign it back to the column in dataframe
 dfObj['z'] = np.square(dfObj['z']
print('*** Apply a function to a single row *** ')
 dfObj = pd.DataFrame(matrix, columns=list('xyz'), index=list('abcdef'))
 # Method 1:
 # Apply function numpy.square() to square the values of one row only i.e. row with index name 'b'
 modDfObj = dfObj.apply(lambda x: np.square(x) if x.name == 'b' else x, axis=1)
print("Modified Dataframe : Squared the values in row 'b'", modDfObj, sep='\n')
 # Method 2:
 # Apply a function to one row and assign it back to the row in dataframe
 dfObj.loc['b'] = dfObj.loc['b'].apply(np.square)
 # Method 3:
 # Apply a function to one row and assign it back to the column in dataframe
dfObj.loc['b'] = np.square(dfObj.loc['b'])
print('********* Apply a function to certains row or column in DataFrame ********')
dfObj = pd.DataFrame(matrix, columns=list('xyz'), index=list('abcdef'))
print('Apply a function to certain columns only')
# Apply function numpy.square() to square the value 2 column only i.e. with column names 'x' and 'y' only
modDfObj = dfObj.apply(lambda x: np.square(x) if x.name in ['x', 'y'] else x)
print("Modified Dataframe : Squared the values in column x & y :", modDfObj, sep='\n')
print('Apply a function to certain rows only') # 
Apply function numpy.square() to square the values of 2 rows only i.e. with row index name 'b' and 'c' only
 modDfObj = dfObj.apply(lambda x: np.square(x) if x.name in ['b', 'c'] else x, axis=1)
print("Modified Dataframe : Squared the values in row b & c :", modDfObj, sep='\n')
if __name__ == '__main__':
main()

Original Dataframe
x y z
a 22 34 23
b 33 31 11
c 44 16 21
d 55 32 22
e 66 33 27
f 77 35 11
********* Apply a function to a single row or column in DataFrame ********
*** Apply a function to a single column *** 
Modified Dataframe : Squared the values in column 'z'
 x y z
a 22 34 529
b 33 31 121
c 44 16 441
d 55 32 484
e 66 33 729
f 77 35 121
*** Apply a function to a single row *** 
Modified Dataframe : Squared the values in row 'b'
 x y z
a 22 34 23
b 1089 961 121
c 44 16 21
d 55 32 22
e 66 33 27
f 77 35 11
********* Apply a function to certains row or column in DataFrame ********
Apply a function to certain columns only
Modified Dataframe : Squared the values in column x & y :
 x y z
a 484 1156 23
b 1089 961 11
c 1936 256 21
d 3025 1024 22
e 4356 1089 27
f 5929 1225 11
Apply a function to certain rows only
Modified Dataframe : Squared the values in row b & c :
 x y z
a 22 34 23
b 1089 961 121
c 1936 256 441
d 55 32 22
e 66 33 27
f 77 35 11

In this article, we will be discussing how to count several elements in 1D, 2D, and 3D Numpy array. Moreover, we will be discussing the counting of rows and columns in a 2D array and the number of elements per axis in a 3D Numpy array.

Let’s get started!

Get the Dimensions of a Numpy array using ndarray.shape()

NumPy.ndarray.shape

This module is used to get a current shape of an array, but it is also used to reshaping the array in place by assigning a tuple of arrays dimensions to it. The function is:

ndarray.shape

We will use this function for determining the dimensions of the 1D and 2D array.

Get Dimensions of a 2D NumPy array using ndarray.shape:

Let us start with a 2D Numpy array.

2D Numpy Array 
[[11 12 13 11] 
[21 22 23 24] 
[31 32 33 34]]

Get the number of rows in this 2D NumPy array:

Code:

numOfRows = arr2D.shape[0]
print('Number of Rows : ', numOfRows)

Number of Rows : 3

Get a number of columns in this 2D NumPy array:

Code:

numOfColumns = arr2D.shape[1]
print('Number of Columns : ', numOfColumns)

Number of Columns: 4

Get the total number of elements in this 2D NumPy array:

Code:

print('Total Number of elements in 2D Numpy array : ', arr2D.shape[0] * arr2D.shape[1])

Total Number of elements in 2D Numpy array: 12

Get Dimensions of a 1D NumPy array using ndarray.shape

Now, we will work on a 1D NumPy array.

Code:

arr = np.array([4, 5, 6, 7, 8, 9, 10, 11])

Shape of 1D numpy array : (8,)
length of 1D numpy array : 8

arr2D = np.array([[11 ,12,13,11], [21, 22, 23, 24], [31,32,33,34]])
print('2D Numpy Array')
print(arr2D)

Get a number of rows and columns of this 2D NumPy array:

Code:

numOfRows = np.size(arr2D, 0)
# get number of columns in 2D numpy array
numOfColumns = np.size(arr2D, 1)
print('Number of Rows : ', numOfRows)
print('Number of Columns : ', numOfColumns)

Number of Rows : 3
Number of Columns: 4

print('Total Number of elements in 2D Numpy array : ', np.size(arr2D))

Total Number of elements in 2D Numpy array: 12

Now, we will be working on the 3D Numpy array.

Code:

arr3D = np.array([ [[11, 12, 13, 11], [21, 22, 23, 24], [31, 32, 33, 34]],
[[1, 1, 1, 1], [2, 2, 2, 2], [3, 3, 3, 3]] ])
print(arr3D)

[[[11 12 13 11]
[21 22 23 24]
[31 32 33 34]]
[[ 1 1 1 1]
[ 2 2 2 2]
[ 3 3 3 3]]]

Get a number of elements per axis in 3D NumPy array:

Code:

print('Axis 0 size : ', np.size(arr3D, 0))
print('Axis 1 size : ', np.size(arr3D, 1))
print('Axis 2 size : ', np.size(arr3D, 2))

Output:

Axis 0 size : 2
Axis 1 size : 3
Axis 2 size : 4

arr = np.array([4, 5, 6, 7, 8, 9, 10, 11])
print('Length of 1D numpy array : ', np.size(arr))

Length of 1D numpy array : 8

import numpy as np
def main():
print('**** Get Dimensions of a 2D numpy array using ndarray.shape ****')
# Create a 2D Numpy array list of list
 arr2D = np.array([[11 ,12,13,11], [21, 22, 23, 24], [31,32,33,34]])
print('2D Numpy Array')
print(arr2D)
 # get number of rows in 2D numpy array
 numOfRows = arr2D.shape[0]
 # get number of columns in 2D numpy array
 numOfColumns = arr2D.shape[1]
print('Number of Rows : ', numOfRows)
print('Number of Columns : ', numOfColumns)
print('Total Number of elements in 2D Numpy array : ', arr2D.shape[0] * arr2D.shape[1])
print('**** Get Dimensions of a 1D numpy array using ndarray.shape ****')
 # Create a Numpy array from list of numbers
arr = np.array([4, 5, 6, 7, 8, 9, 10, 11])
print('Original Array : ', arr)
print('Shape of 1D numpy array : ', arr.shape)
print('length of 1D numpy array : ', arr.shape[0])
print('**** Get Dimensions of a 2D numpy array using np.size() ****')
 # Create a 2D Numpy array list of list
 arr2D = np.array([[11, 12, 13, 11], [21, 22, 23, 24], [31, 32, 33, 34]])
print('2D Numpy Array')
print(arr2D)
 # get number of rows in 2D numpy array
 numOfRows = np.size(arr2D, 0)
 # get number of columns in 2D numpy array
 numOfColumns = np.size(arr2D, 1)
print('Number of Rows : ', numOfRows)
print('Number of Columns : ', numOfColumns)
print('Total Number of elements in 2D Numpy array : ', np.size(arr2D))
print('**** Get Dimensions of a 3D numpy array using np.size() ****')
 # Create a 3D Numpy array list of list of list
 arr3D = np.array([ [[11, 12, 13, 11], [21, 22, 23, 24], [31, 32, 33, 34]],
[[1, 1, 1, 1], [2, 2, 2, 2], [3, 3, 3, 3]] ])
print('3D Numpy Array')
print(arr3D)
print('Axis 0 size : ', np.size(arr3D, 0))
print('Axis 1 size : ', np.size(arr3D, 1))
print('Axis 2 size : ', np.size(arr3D, 2))
print('Total Number of elements in 3D Numpy array : ', np.size(arr3D))
print('Dimension by axis : ', arr3D.shape)
print('**** Get Dimensions of a 1D numpy array using numpy.size() ****')
 # Create a Numpy array from list of numbers
arr = np.array([4, 5, 6, 7, 8, 9, 10, 11])
print('Original Array : ', arr)
print('Length of 1D numpy array : ', np.size(arr))
if __name__ == '__main__':
main()

**** Get Dimensions of a 2D numpy array using ndarray.shape ****
2D Numpy Array
[[11 12 13 11]
[21 22 23 24]
[31 32 33 34]]
Number of Rows : 3
Number of Columns : 4
Total Number of elements in 2D Numpy array : 12
**** Get Dimensions of a 1D numpy array using ndarray.shape ****
Original Array : [ 4 5 6 7 8 9 10 11]
Shape of 1D numpy array : (8,)
length of 1D numpy array : 8
**** Get Dimensions of a 2D numpy array using np.size() ****
2D Numpy Array
[[11 12 13 11]
[21 22 23 24]
[31 32 33 34]]
Number of Rows : 3
Number of Columns : 4
Total Number of elements in 2D Numpy array : 12
**** Get Dimensions of a 3D numpy array using np.size() ****
3D Numpy Array
[[[11 12 13 11]
[21 22 23 24]
[31 32 33 34]]
[[ 1 1 1 1]
[ 2 2 2 2]
[ 3 3 3 3]]]
Axis 0 size : 2
Axis 1 size : 3
Axis 2 size : 4
Total Number of elements in 3D Numpy array : 24
Dimension by axis : (2, 3, 4)
**** Get Dimensions of a 1D numpy array using numpy.size() ****
Original Array : [ 4 5 6 7 8 9 10 11]
Length of 1D numpy array : 8

In this article, we will be checking up few things:

• Basic crawling setup In Python
• Basic crawling with AsyncIO
• Scraper Util service
• Python scraping via Scrapy framework

Web Crawler

A web crawler is an automatic bot that extracts useful information by systematically browsing the world wide web.

The web crawler is also known as a spider or spider bot. Some websites use web crawling for updating their web content. Some websites do not allow crawling because of their security, so on that websites crawler works by either asking for permission or exiting out of the website.

Web Scraping

Extracting the data from the websites is known as web scraping. Web scraping requires two parts crawler and scraper.

Crawler is known to be an artificial intelligence algorithm and it browses the web which leads to searching of the links we want to crawl across the internet.

Scraper is the tool that was specifically used for extracting information from the internet.

By web scraping, we can obtain a large amount of data which is in unstructured data in an HTML format and then it is converted into structured data.

Crawler Demo

Mainly, we have been using two tools:

• Python request module to make a crawler bot. We can get it from (https://pypi.org/project/requests/)
• Parsel Library which is used as a scraping tool. Access it from (https://parsel.readthedocs.io/en/latest/usage.html)

Task I

Scrap recurship website is used for extracting all the links and images present on the page.

Demo Code:

Import requests

from parsel import Selector

import time

start = time.time()

response = requests.get('http://recurship.com/')

selector  = Selector(response.text)

href_links = selector.xpath('//a/@href').getall()

image_links = selector.xpath('//img/@src').getall()

print("********************href_links****************")

print(href_links)

print("******************image_links****************")

print(image_links)

end = time.time()

print("Time taken in seconds:", (end_start)

Task II

Scrap recurship site and extract links, one of one navigate to each link and extract information of the images.

Demo code:

import requests
from parsel import Selector

import time
start = time.time()


all_images = {} 
response = requests.get('http://recurship.com/')
selector = Selector(response.text)
href_links = selector.xpath('//a/@href').getall()
image_links = selector.xpath('//img/@src').getall()

for link in href_links:
try:
response = requests.get(link)
if response.status_code == 200:
image_links = selector.xpath('//img/@src').getall()
all_images[link] = image_links
except Exception as exp:
print('Error navigating to link : ', link)

print(all_images)
end = time.time()
print("Time taken in seconds : ", (end-start))

Task II takes 22 seconds to complete. We are constantly using the python “parsel” and “request” package.

Let’s see some features these packages use.

Request package:

Python request module basically offers the following features:

• HTTP method calls
• Working with response codes and headers
• Maintains redirection and history for requests
• Maintains sessions
• Work with cookies
• Errors and exceptions

Parsel package

Python parsel package offers the following features:

• Extract text using CSS or XPath selectors
• Regular expression helper methods

Crawler service using Request and Parsel

The code:

import requests
import time
import random
from urllib.parse import urlparse
import logging

logger = logging.getLogger(__name__)

LOG_PREFIX = 'RequestManager:'


class RequestManager:
def __init__(self):
self.set_user_agents(); # This is to keep user-agent same throught out one request

crawler_name = None
session = requests.session()
# This is for agent spoofing...
user_agents = [
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36 Edge/12.246',
'Mozilla/4.0 (X11; Linux x86_64) AppleWebKit/567.36 (KHTML, like Gecko) Chrome/62.0.3239.108 Safari/537.36',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/601.3.9 (KHTML, like Gecko) Version/9.0.2 Safari/601.3.9',
'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko'
]

headers = {}

cookie = None
debug = True

def file_name(self, context: RequestContext, response, request_type: str = 'GET'):
url = urlparse(response.url).path.replace("/", "|")
return f'{time.time()}_{context.get("key")}_{context.get("category")}_{request_type}_{response.status_code}_{url}'

# write a file, safely
def write(self, name, text):
if self.debug:
file = open(f'logs/{name}.html', 'w')
file.write(text)
file.close()

def set_user_agents(self):
self.headers.update({
'user-agent': random.choice(self.user_agents)
})

def set_headers(self, headers):
logger.info(f'{LOG_PREFIX}:SETHEADER set headers {self.headers}')
self.session.headers.update(headers)

def get(self, url: str, withCookie: bool = False, context):
logger.info(f'{LOG_PREFIX}-{self.crawler_name}:GET making get request {url} {context} {withCookie}')
cookies = self.cookie if withCookie else None
response = self.session.get(url=url, cookies=cookies, headers=self.headers)
self.write(self.file_name(context, response), response.text)
return response

def post(self, url: str, data, withCookie: bool = False, allow_redirects=True, context: RequestContext = {}):
logger.info(f'{LOG_PREFIX}:POST making post request {url} {data} {context} {withCookie}')
cookies = self.cookie if withCookie else None
response = self.session.post(url=url, data=data, cookies=cookies, allow_redirects=allow_redirects)
self.write(self.file_name(context, response, request_type='POST'), response.text)
return response

def set_cookie(self, cookie):
self.cookie = cookie
logger.info(f'{LOG_PREFIX}-{self.crawler_name}:SET_COOKIE set cookie {self.cookie}')

Request = RequestManager()

context = {
"key": "demo",
"category": "history"
}
START_URI = "DUMMY_URL" # URL OF SIGNUP PORTAL
LOGIN_API = "DUMMY_LOGIN_API"
response = Request.get(url=START_URI, context=context)

Request.set_cookie('SOME_DUMMY_COOKIE')
Request.set_headers('DUMMY_HEADERS')

response = Request.post(url=LOGIN_API, data = {'username': '', 'passphrase': ''}, context=context)

Class “RequestManager” offers few functionalities listed below:

• POST and GET calls with logging
• Saving responses as log files of each HTTP request
• Setting headers and cookies
• Session management
• Agent spoofing

Scraping with AsyncIO

All we have to do is scrap the Recurship site and extract all the links, later we navigate each link asynchronously and extract information from the images.

Demo code

import requests
import aiohttp
import asyncio
from parsel import Selector
import time

start = time.time()
all_images = {} # website links as "keys" and images link as "values"

async def fetch(session, url):
try:
async with session.get(url) as response:
return await response.text()
except Exception as exp:
return '<html> <html>' #empty html for invalid uri case

async def main(urls):
tasks = []
async with aiohttp.ClientSession() as session:
for url in urls:
tasks.append(fetch(session, url))
htmls = await asyncio.gather(*tasks)
for index, html in enumerate(htmls):
selector = Selector(html)
image_links = selector.xpath('//img/@src').getall()
all_images[urls[index]] = image_links
print('*** all images : ', all_images)


response = requests.get('http://recurship.com/')
selector = Selector(response.text)
href_links = selector.xpath('//a/@href').getall()
loop = asyncio.get_event_loop()
loop.run_until_complete(main(href_links))


print ("All done !")
end = time.time()
print("Time taken in seconds : ", (end-start))

By AsyncIO, scraping took almost 21 seconds. We can achieve more good performance with this task.

Open-Source Python Frameworks for spiders

Python has multiple frameworks which take care of the optimization

It gives us different patterns. There are three popular frameworks, namely:

1. Scrapy
2. PySpider
3. Mechanical soup

Let’s use Scrapy for further demo.

Scrapy

Scrapy is a framework used for scraping and is supported by an active community. We can build our own scraping tools.

There are few features which scrapy provides:

• Scraping and parsing tools
• Easily export the data it collects in a number of formats like JSON or CSV and store the data on a backend of your choosing
• Has a number of built-in extensions for tasks like cookie handling, user-agent spoofing, restricting crawl depth, and others
• Has an API for easily building your own additions.
• Scrapy also offers a cloud to host spiders where spiders scale on-demand and run from thousand to billions. It’s like Heroku of spiders.

Now we have to do is scrap the Recurship site and extract all the links, later we navigate each link asynchronously and extract information from the images.

Demo Code

import scrapy


class AuthorSpider(scrapy.Spider):
name = 'Links'

start_urls = ['http://recurship.com/']
images_data = {}
def parse(self, response):
# follow links to author pages
for img in response.css('a::attr(href)'):
yield response.follow(img, self.parse_images)

# Below commented portion is for following all pages
# follow pagination links
# for href in response.css('a::attr(href)'):
# yield response.follow(href, self.parse)

def parse_images(self, response):
#print "URL: " + response.request.url
def extract_with_css(query):
return response.css(query).extract()
yield {
'URL': response.request.url,
'image_link': extract_with_css('img::attr(src)')
}

Commands

scrapy run spider -o output.json spider.py

The JSON file got export in 1 second.

Conclusion

We can see that the scrapy performed an excellent job. If we have to perform simple crawling, scrapy will give the best results.

Enjoy scraping!!

In this Page, We are Providing Python Programming – Compound statement. Students can visit for more Detail and Explanation of Python Handwritten Notes Pdf.

Python Programming – Compound statement

Compound statement

Compound statements contain groups of other statements, they affect or control the execution of those other statements in some way. In general, compound statements span multiple lines, although a whole compound statement can be contained in one line.

The if, while, and for statements are the traditional control flow compound statements, try to specify exception handlers and/or cleanup code for a group of statements. Function and class definitions are also syntactically compound statements.

Compound statements consist of one or more clauses. A clause consists of a header and a suite. The clause headers of a particular compound statement are all at the same indentation level. Each clause header begins with a uniquely identifying keyword and ends with a colon. A suite is a group of statements controlled by a clause. A suite can be one or more semicolon-separated simple statements on the same line as the header, following the header’s colon, or it can be one or more indented statements on subsequent lines. Only the latter form of the suite can contain nested compound statements.

The colon is required primarily to enhance readability.

if a == b
print aversus

if a == b:
print a

Notice how the second one is slightly easier to read. Another minor reason is that the colon makes it easier for editors with syntax highlighting; they can look for colons to decide when indentation needs to be increased.

A code block (or simply “block”) is a piece of Python program text that is executed as a unit. Few examples of blocks are a module, a function body, a class definition, etc. Each command typed interactively is a block.

If statement

The if statement is used for conditional execution:

if_stmt : : = " if " expression " : " suite
( " elif " expression " : " suite ) *
["else" " : " suite]

It selects exactly one of the suites by evaluating the expressions one by one until one is found to be true, then that suite is executed (and no other part of the if statement is executed or evaluated). If all expressions are false, the suite of the else clause (if present) is executed.

>>> var=100 
>>> if var==100 :
. . .             print ' var has value 100 '
. . .
var has a value of 100 
>>>
>>> var=100 
>>> if var<> 100 :
. . .           print ' var does not have value 100 '
. . . else:
. . .           print 'var has value 100'
. . .
var has a value of 100 
>>>
>>> var=100 
>>> if var<100 : 
. . .            print ' var has value less than 100 '
. . . elif var>100 : 
. . .            print ' var has value greater than 100'
. . . else:
. . .           print ’ var has value 100 '
. . .
var has a value of 100

The following example shows that if the evaluation of expression gives a result either None, an empty string (‘ ‘), zero (0), or Boolean False, then the condition is considered as false.

>>> if None or ' ' or 0 or False:
. . .          print ' Atleast one condition is true ' 
. . . else:
. . .           print 'None of the condition is true'
. . . 
None of the condition is true

There may be a situation when there is a need to check for another condition after a condition resolves to true. In such a situation, the nested if statement can be used.

>>> var=100
>>> if isinstance ( var , str) :
. . .                 print ' var is a string '
. . . elif (type(var) is int) :
. . .          if var<> 100 :
. . .             print ' var does not have value 100 '
. . . else:
. . .        print 'var has value 100'
. . .
var has a value of 100

While statement

The while statement is used for repeated execution of a group of statements as long as an expression is true:

while_stmt " while " expression " : "  suite
[ " else " suite ]

This compound statement repeatedly tests the expression, and if it is true, it executes the first suite; if the expression is false (which may be the first time it is tested) the suite of the else clause (if present) is executed and the loop terminates.

>>> var=1
>>> while var<=5: 
. . .  print ' Count ',var
. . .  var=var+l
. . .
Count 1
Count 2 
Count 3
Count 4 
Count 5
>>> print ' The loop ends '
The loop ends

The above code can also be witten as follows:

>>> var=l
>>> while var<=5 :
. . .         print ' Count ' , var
. . .   var=var+1
. . . else:
. . .     print 'The loop ends'
. . .
Count 1 
Count 2 
Count 3 
Count 4 
Count 5 
The loop ends

A break statement executed in the first suite terminates the loop without executing the else clause’s suite.

>>> var=l
>>> while var<=5:
. . .   print  ' Count ', var
. . .     var=var+1
. . .      if var==4:
. . .        break
. . . else:
. . .        print ' The loop ends '
. . .
Count 1 
Count 2 
Count 3

A continuous statement executed in the first suite skips the rest of the suite and goes back to testing the expression.

>>> var=0
>>> while var<=5:
. . .         var=var+1
. . .         if var>=3 and var<=4:
. . .              continue
. . .     print 'Count ',var
. . .
Count 1 
Count 2 
Count 5
Count 6

Consider a scenario where the condition never becomes false. This results in a loop that never ends; such a loop is called an infinite loop. One needs to use the keyboard Ctrl+C to come out of the program.

For statement

The for statement is used to iterate over the elements of a sequence (such as a string, tuple, or list) or another iterable object:

for_stmt :  : = " for " target_list " in " expression_list " : " suite 
                            [ " else " suite ]

The expression list expression_list is evaluated once; it should yield an iterable object. An iterator is created for the result of the expression_list. The suite is then executed once for each item provided by the iterator, in the order of ascending indices. Each item, in turn, is assigned to the target list target_list using the standard rules for assignments, and then the suite is executed. When the items are exhausted (which is immediately when the sequence is empty), the suite in the else clause (if present) is executed, and the loop terminates.

>>> for i in [ 1 , 2 , 3 , 4 , 5 ] :
. . . print ' Count ' , i
. . .
Count 1 
Count 2 
Count 3 
Count 4 
Count 5
>>> print ' The loop ends '
The loop ends

The above code can also be written as follows:

>>> for i in range ( 1 , 6 ) :
. . .         print ' Count ' , i
. . . else : 
. . .           print 'The loop ends'
. . .
Count 1 
Count 2 
Count 3
Count 4 
Count 5
The loop ends

A break statement executed in the first suite terminates the loop without executing the else clause’s suite.

>>> for i in [ 1 , 2 , 3 , 4 , 5 ] :
. . . if i==4 :
. . .  break
. . .    print ' Count ' , i
. . . else :
. . .      print  ' The loop ends '
. . .
Count 1 
Count 2 
Count 3

A continue statement executed in the first suite skips the rest of the suite and continues with the next item, or with the else clause if there was no next item.

>>> for i in [ 1 , 2 , 3 , 4 , 5 ] :
. . . if i==4 :
. . . continue
. . .         print  ' Count  ' , i
. . .  else :
. . .       print  ' The loop ends '
. . . 
Count 1 
Count 2 
Count 3 
Count 5 
The loop ends

There might be a scenario where one needs to print the item along with its index. The following example demonstrates this scenario using enumerate () built-in function (discussed later in this chapter).

>>> shuttles= [ ' Columbia ' , ' endeavour ' , ' challenger ' ]
>>> for index , value in enumerate ( shuttles ) :
. . . print index, value
. . .
0 Columbia
1 endeavor
2 challenger 
>>>
>>> value
'challenger'
>>> index 
2

It can also be observed that the target list is not deleted when the loop is finished.