使用 TextBlob 进行词形还原和分词

2024 年 8 月 29 日 | 阅读 3 分钟

词形还原是将不同的屈折词形式连接起来，使之被视为同一个词。词形还原类似于词干提取。然而，它为词语提供了上下文含义。它还将共享相同含义并被视为同一个词的词语联系起来。文本预处理包括词干提取和词形还原。有时人们会混淆这两个术语，并且它们经常被视为相同。然而，词形还原比词干提取更有优势，因为它允许研究词语的形态学。

词形还原的应用

在像搜索引擎这样的综合检索系统中。
用于小型索引。

词形还原的例子

painters: painter
birds: bird
worst: bad

词干提取和词形还原之间的主要区别在于，词形还原需要词性参数“pos”。如果未提供，则默认为“名词”。下面是如何使用 TextBlob 实现词形还原的方法。

代码

# from textblob library we will import Word module
from textblob import Word as WD

# here, we will create a Word object.
P = WD("painters")

# here, we will apply lemmatization.
print("painters:", P.lemmatize())

# here, we will create a Word object again.
Q = WD("birds")

# here, we will apply lemmatization.
print("birds:", Q.lemmatize())

# here, we willcreate a Word object.
R = WD("worst")

# Now, we will apply lemmatization with
# parameter "a", "a" denotes adjective.
print("worst:", R.lemmatize("a"))

输出

painters: painter
birds: bird
worst: bad

使用 TextBlob 分词

TextBlob 模块是一个 Python 库，它提供了一个易于使用的 API 来执行简单的 NLP 任务。该模块基于 NLTK 模块开发。

在终端使用以下命令安装 TextBlob

!pip3 install textblob
from textblob import download_corpora

这将启用 TextBlob，并下载所需的 NLTK 语料库。由于需要下载大量的分词器、分块器、各种算法和整个语料库，因此上述过程可能需要很长时间。

常用的术语包括

语料库 (Corpus): 文本的集合，单数形式。Corpora 是其复数形式。
词汇表 (Lexicon): 单词的含义及其定义。
词元 (Token): 根据规则被划分为其他一部分的每个“实体”。在这种情况下，当一个句子被“分词”成单词时，每个单词都是一个词元。当您将段落中的句子分词时，一个句子也可以是一个词元。

代码

# First, we will import TextBlob method from textblob library
from textblob import TextBlob as Txb

text1 = ("There were three friends name, Jemmy, Jacky, Kenny.  " +
	"They have been friends forever since pre school." +
	"but somehow Jemmy's bestfriend is Jacky and whenever jemmy and kenny lefts alone, they endup being quite." +
	"One day they all decided to plan a trip together after graduation. " +
	"They all went to Kashmire" +
	"Kashmire trip was really good, they all created lifetime memories together. " +
	"After that trip they have to focus on there future. " +
	"Which stream they have to choose and career path they should choose for future.")

    
# Now, we will create a TextBlob object
blob_object1 = Txb(text1)
  
# Here, we will tokenize the paragraph into words.
print(" Word Tokenize from paragraph:\n", blob_object1.words)
  
# tokenize paragraph into sentences.
print("\n Sentence Tokenize from paragraph:\n", blob_object1.sentences)

输出

Word Tokenize from paragraph:
 ['There', 'were', 'three', 'friends', 'name', 'Jemmy', 'Jacky', 'Kenny', 'They', 'have', 'been', 'friends', 'forever', 'since', 'pre', 'school.but', 'somehow', 'Jemmy', "'s", 'bestfriend', 'is', 'Jacky', 'and', 'whenever', 'jemmy', 'and', 'kenny', 'lefts', 'alone', 'they', 'endup', 'being', 'quite.One', 'day', 'they', 'all', 'decided', 'to', 'plan', 'a', 'trip', 'together', 'after', 'graduation', 'They', 'all', 'went', 'to', 'KashmireKashmire', 'trip', 'was', 'really', 'good', 'they', 'all', 'created', 'lifetime', 'memories', 'together', 'After', 'that', 'trip', 'they', 'have', 'to', 'focus', 'on', 'there', 'future', 'Which', 'stream', 'they', 'have', 'to', 'choose', 'and', 'career', 'path', 'they', 'should', 'choose', 'for', 'future']

 Sentence Tokenize from paragraph:
 [Sentence("There were three friends name, Jemmy, Jacky, Kenny."), Sentence("They have been friends forever since pre school.but somehow Jemmy's bestfriend is Jacky and whenever jemmy and kenny lefts alone, they endup being quite.One day they all decided to plan a trip together after graduation."), Sentence("They all went to KashmireKashmire trip was really good, they all created lifetime memories together."), Sentence("After that trip they have to focus on there future."), Sentence("Which stream they have to choose and career path they should choose for future.")]

下一个话题如何在 Python 中四舍五入数字

使用 TextBlob 进行词形还原和分词

词形还原的应用

使用 TextBlob 分词

联系信息

关注我们

教程

面试题

在线编译器

Python

Java

.Net Framework

AI, ML and Data Science

Cloud Technology

B.Tech and MCA

Web Technology

PHP

Software Testing

Technical Interview

Java Interview

Python

Web Interview

Database Interview

B.Tech / MCA

Important Interview

Software Testing Interview

Company Interviews

Online Compilers

Multiple Choice Questions

Python 问题

使用 TextBlob 进行词形还原和分词

词形还原的应用

使用 TextBlob 分词

相关帖子

Python 统计学

如何检查 Python 版本

Python 常量模块

Python 中的分派装饰器

边缘计算项目创意列表 第二部分

如何在 Python 中安装 statsmodels

使用 Kivy 构建跨平台 GUI 应用

Python Getpass 模块

如何在 Pandas DataFrame 中基于条件选择行

使用 Railway API 查询任意列车的实时运行状态和 PNR

订阅 Tpoint Tech

联系信息

关注我们

教程

面试题

在线编译器

边缘计算项目创意列表第二部分