Python nltk.tokenize.TabTokenizer() Function

/ Python / By

NLTK in Python:

NLTK is a Python toolkit for working with natural language processing (NLP). It provides us with a large number of test datasets for various text processing libraries. NLTK can be used to perform a variety of tasks such as tokenizing, parse tree visualization, and so on.

Tokenization

Tokenization is the process of dividing a large amount of text into smaller pieces/parts known as tokens. These tokens are extremely valuable for detecting patterns and are regarded as the first stage in stemming and lemmatization. Tokenization also aids in the replacement of sensitive data elements with non-sensitive data elements.

Natural language processing is utilized in the development of applications such as text classification, intelligent chatbots, sentiment analysis, language translation, and so on. To attain the above target, it is essential to consider the pattern in the text.

Natural Language Toolkit features an important module called NLTK tokenize sentences, which is further divided into sub-modules.

  • word tokenize
  • sentence tokenize

nltk.tokenize.TabTokenizer() Function:

Using TabTokenizer() Function of the tokenize of nltk module we can extract tokens from a string of words based on the tabs between them.

Syntax:

tokenize.TabTokenizer()

Parameters: This method doesn’t accept any parameters

Return Value:

The tokens of words based on the tabs are returned by the TabTokenizer() Function

Examples:

Example1:

Input:

Given string = "hello python-programs\t@@&* \nwelcome\tgood morning"

Output:

['hello python-programs', '@@&* \nwelcome', 'good morning']

Example2:

Input:

Given string = "welcome\t to python-programs\t hi\tall"

Output:

['welcome', ' to python-programs', ' hi', 'all']

nltk.tokenize.TabTokenizer() Function in Python

Example1

Approach:

  • Import TabTokenizer() function from tokenize of nltk module using the import keyword
  • Creating a reference/Instance variable(Object) for the TabTokenizer Class
  • Give the string as static input and store it in a variable.
  • Pass the above-given string as an argument to extract tokens from a string of words based on the tabs between them and store it in another variable.
  • Print the above result.
  • The Exit of the Program.

Below is the implementation:

# Import TabTokenizer() function from tokenize of nltk module using the import keyword
from nltk.tokenize import TabTokenizer
    
# Creating a reference/Instance variable(Object) for the TabTokenizer Class
tkn = TabTokenizer()
    
# Give the string as static input and store it in a variable.
gvn_str = "hello python-programs\t@@&* \nwelcome\tgood morning"

# Pass the above given string as an argument to extract tokens from a string of words 
# based on the tabs between them.
rslt = tkn.tokenize(gvn_str)

# Print the above result 
print(rslt)

Output:

['hello python-programs', '@@&* \nwelcome', 'good morning']

Example2

# Import TabTokenizer() function from tokenize of nltk module using the import keyword
from nltk.tokenize import TabTokenizer
    
# Creating a reference/Instance variable(Object) for the TabTokenizer Class
tkn = TabTokenizer()
    
# Give the string as static input and store it in a variable.
gvn_str = "welcome\t to python-programs\t hi\tall"

# Pass the above given string as an argument to extract tokens from a string of words 
# based on the tabs between them and store it in another variable.
rslt = tkn.tokenize(gvn_str)

# Print the above result 
print(rslt)

Output:

['welcome', ' to python-programs', ' hi', 'all']