NLTK in Python:
NLTK is a Python toolkit for working with natural language processing (NLP). It provides us with a large number of test datasets for various text processing libraries. NLTK can be used to perform a variety of tasks such as tokenizing, parse tree visualization, and so on.
Tokenization
Tokenization is the process of dividing a large amount of text into smaller pieces known as tokens. These tokens are extremely valuable for detecting patterns and are regarded as the first stage in stemming and lemmatization. Tokenization also aids in the replacement of sensitive data elements with non-sensitive data elements.
Natural language processing is utilized in the development of applications such as text classification, intelligent chatbots, sentiment analysis, language translation, and so on. To attain the above target, it is essential to consider the pattern in the text.
Natural Language Toolkit features an important module called NLTK tokenize sentences, which is further divided into sub-modules.
- word tokenize
- sentence tokenize
nltk.tokenize.SExprTokenizer() Function:
Using nltk.tokenize.SExprTokenizer() method, we can extract tokens from a string of characters or numbers. It is actually looking for correct brackets to produce tokens.
Syntax:
tokenize.SExprTokenizer()
Parameters: This method doesn’t accept any parameters
Return Value:
The tokens from a string of characters or numbers are returned.
NLTK nltk.tokenize.SExprTokenizer() Function in Python
Method #1: Using tokenize.SExprTokenizer() Function (Static Input)
Here, we are using the tokenize.SExprTokenizer() method to extract tokens from a stream of characters or numbers while taking brackets into account.
Approach:
- Import SExprTokenizer() function from tokenize of nltk using the import keyword
- Create a reference/Instance variable(Object) for the SExprTokenizer Class
- Give the string as static input and store it in a variable.
- Pass the above-given string as an argument to the tokenize() function to extract tokens from the given string (taking brackets into account).
- Store it in another variable.
- Print the above result.
- The Exit of the Program.
Below is the implementation:
# Import SExprTokenizer() function from tokenize of nltk using the import keyword
from nltk.tokenize import SExprTokenizer
# Creating a reference/Instance variable(Object) for the SExprTokenizer Class
tkn = SExprTokenizer()
# Give the string as static input and store it in a variable.
gvn_str = "( p * ( q + r ))st( u-v )"
# Pass the above given string as an argument to the tokenize() function to extract
# tokens from the given string (taking brackets into account).
# Store it in another variable.
rslt = tkn.tokenize(gvn_str)
# Print the above result
print(rslt)Output:
['( p * ( q + r ))', 'st', '( u-v )']
Method #2: Using tokenize.SExprTokenizer() Function (User Input)
Approach:
- Import SExprTokenizer() function from tokenize of nltk using the import keyword
- Create a reference/Instance variable(Object) for the SExprTokenizer Class
- Give the string as static input and store it in a variable.
- Pass the above-given string as an argument to the tokenize() function to extract tokens from the given string (taking brackets into account).
- Store it in another variable.
- Print the above result.
- The Exit of the Program.
Below is the implementation:
# Import SExprTokenizer() function from tokenize of nltk using the import keyword
from nltk.tokenize import SExprTokenizer
# Creating a reference/Instance variable(Object) for the SExprTokenizer Class
tkn = SExprTokenizer()
# Give the string as user input using the input() function and store it in a variable.
gvn_str = input("Enter some random string = ")
# Pass the above given string as an argument to the tokenize() function to extract
# tokens from the given string (taking brackets into account).
# Store it in another variable.
rslt = tkn.tokenize(gvn_str)
# Print the above result
print(rslt)Output:
Enter some random string = (p q r) st (u v w) xy ['(p q r)', 'st', '(u v w)', ' xy']