源碼請到：自然語言處理練習: 學習自然語言處理時候寫的一些代碼 (gitee.com)

數據來源：

搜狗新聞語料庫 由於鏈接失效，現在使用百度網盤分享

鏈接：https://pan.baidu.com/s/1RTx2k7V3Ujgg9-Rv8I8IRA?pwd=ujn3
提取碼：ujn3

停用詞 來源於網路

鏈接：https://pan.baidu.com/s/1ePrf4_gWx8_pTn6PEjTtCw?pwd=5jov
提取碼：5jov

字樣式文件 來源於網路

鏈接：https://pan.baidu.com/s/1uVreJY-MKhz1HXzAw5e4VQ?pwd=8ill
提取碼：8ill

一、tf-idf簡介

TF = 某詞在文章中出現的次數/該文章中出現最多詞出現的次數

IDF = log(文章總數/包含該詞的文章數+1)

TF-IDF = TF * IDF

二、載入數據集

# 載入數據集
df_news = pd.read_table('./data/val.txt', names=['category', 'theme', 'URL', 'content'], encoding='utf-8')
df_news = df_news.dropna()
print(df_news.head())
print(df_news.shape)

 可以看到有5000行4列的數據，其中第一列可以作為新聞分類的標簽，最後一列為新聞內容

三、分詞

首先將數據轉換為list格式

# 轉換為list格式
content = df_news.content.values.tolist()
print(content[1000])

 將最後一列數據摘出來轉換成了一個字元串列表，就可以進行分詞操作

# 分詞
content_S = []
for line in content:
    current_segment = jieba.lcut(line)
    if len(current_segment) > 1 and current_segment != '\r\n':
        content_S.append(current_segment)
print(content_S[1000])
df_content = pd.DataFrame({'content_S': content_S})
print(df_content.head())

 四、去掉停用詞

可以看出上面還有許多沒有價值的詞作干擾，所以我們載入停用詞庫並且去掉停用詞

# 載入停用詞
stopwords = pd.read_csv('./data/stopwords.txt', index_col=False, sep='\t', quoting=3, names=['stopword'],
                        encoding='utf-8')
print(stopwords.head(20))


# 去掉停用詞
def drop_stopwords(contents, stopwords):
    contents_clean = []
    all_words = []
    for line in contents:
        line_clean = []
        for word in line:
            if word in stopwords:
                continue
            line_clean.append(word)
            all_words.append(str(word))
        contents_clean.append(line_clean)
    return contents_clean, all_words


contents = df_content.content_S.values.tolist()
stopwords = stopwords.stopword.values.tolist()
contents_clean, all_words = drop_stopwords(contents, stopwords)
df_content = pd.DataFrame({'contents_clean': contents_clean})
print(df_content.head())
df_all_words = pd.DataFrame({'all_words': all_words})
print(df_all_words.head())

 五、計算詞頻

# 計算詞頻
words_count = df_all_words.groupby(by=['all_words'])['all_words'].agg(count='count')
words_count = words_count.reset_index().sort_values(by=['count'], ascending=False)
print(words_count.head())

 六、繪製詞雲

# 繪製詞雲
wordcloud = WordCloud(font_path='./data/SimHei.ttf', background_color='white', max_font_size=80)
word_frequence = {x[0]: x[1] for x in words_count.head(100).values}
wordcloud = wordcloud.fit_words(word_frequence)
plt.imshow(wordcloud)
plt.show()

 七、使用tf-idf提取關鍵詞

# tf-idf
index = 1000
print(df_news['content'][index])
content_S_str = ''.join(content_S[index])
print(' '.join(jieba.analyse.extract_tags(content_S_str, topK=5, withWeight=False)))

 八、使用主題模型提取關鍵詞

# LDA
dictionary = corpora.Dictionary(contents_clean)
corpus = [dictionary.doc2bow(sentence) for sentence in contents_clean]
lda = gensim.models.ldamodel.LdaModel(corpus=corpus, id2word=dictionary, num_topics=20)
print(lda.print_topic(1, topn=5))
for topic in lda.print_topics(num_topics=20, num_words=5):
    print(topic[1])

 可以看出第一類詞的成分權重

 這是所有類型的詞成分權重

九、使用貝葉斯演算法進行分類

# 貝葉斯演算法進行分類
df_train = pd.DataFrame({'contents_clean': contents_clean, 'label': df_news['category']})
print(df_train.tail())
print(df_train.label.unique())
label_mapping = {'汽車': 1, '財經': 2, '科技': 3, '健康': 4, '體育': 5, '教育': 6, '文化': 7, '軍事': 8, '娛樂': 9,
                 '時尚': 0}
df_train['label'] = df_train['label'].map(label_mapping)
print(df_train.head())
x_train, x_test, y_train, y_test = train_test_split(df_train['contents_clean'].values, df_train['label'].values)
print(x_train[0][1])

words = []
for line_index in range(len(x_train)):
    words.append(' '.join(x_train[line_index]))
print(words[0])
print(len(words))
# 計算詞頻構造向量
vec = CountVectorizer(analyzer='word', max_features=4000, lowercase=False)
vec.fit(words)
classifier = MultinomialNB()
classifier.fit(vec.transform(words), y_train)
test_words = []
for line_index in range(len(x_test)):
    test_words.append(' '.join(x_test[line_index]))
print(test_words[0])
print(len(test_words))
print(classifier.score(vec.transform(test_words), y_test))
# tf-idf構造詞向量
vec2 = TfidfVectorizer(analyzer='word', max_features=4000, lowercase=False)
vec2.fit(words)
classifier = MultinomialNB()
classifier.fit(vec2.transform(words), y_train)
print(classifier.score(vec2.transform(test_words), y_test))
# 詞頻構造多維向量形式構造詞向量
vec3 = CountVectorizer(analyzer='word', max_features=4000, lowercase=False, ngram_range=(1, 2))
vec3.fit(words)
classifier = MultinomialNB()
classifier.fit(vec3.transform(words), y_train)
print(classifier.score(vec3.transform(test_words), y_test))
# tfidf構造多維向量形式構造詞向量
vec4 = TfidfVectorizer(analyzer='word', max_features=4000, lowercase=False, ngram_range=(1, 2))
vec4.fit(words)
classifier = MultinomialNB()
classifier.fit(vec4.transform(words), y_train)
print(classifier.score(vec4.transform(test_words), y_test))

 可以看出不同方法構成詞向量對結果產生了影響，使用tf-idf方法構建詞向量比單純使用詞頻構建詞向量準確率高一些，將詞向量擴充多維比不擴充準確率稍微高一些