保险欺诈检测 - 机器学习

2025年6月25日 | 阅读 10 分钟

Insurance Fraud Detection -Machine Learning

保险公司面临着严重的保险欺诈问题，每年给它们造成数十亿美元的损失。保险欺诈可能以多种方式出现，包括捏造或夸大索赔。机器学习可以在这里用于检测保险欺诈。

机器学习算法可用于分析大量数据，以发现可能表明欺诈的趋势。这些实时数据处理方法使保险公司能够快速发现并阻止虚假索赔。

许多机器学习方法，包括决策树、随机森林、逻辑回归和神经网络，都可以用于检测保险欺诈。算法的选择将取决于应用程序的特定需求。这些算法各有优缺点。

机器学习在欺诈检测方面的优势

以下是使用机器学习进行保险欺诈检测的一些好处：

由于机器学习算法对海量数据进行实时处理，因此与传统技术相比，欺诈性索赔的识别和标记速度大大加快。
机器学习算法可以检查来自不同来源的数据，并发现可能指向欺诈的趋势。这可以减少误报，并提高欺诈检测的准确性。
如果欺诈性索赔得到及早发现，保险公司可以节省大量资金。通过利用机器学习算法，保险公司可以识别并阻止欺诈性索赔在支付之前，从而可以节省大量成本。
保险公司可以通过识别和避免虚假索赔来改善整体客户体验。欺诈不太可能导致有效索赔延迟或被拒绝，从而提高客户满意度。
机器学习算法可以根据保险公司的需求进行扩展或缩减。随着数据量的增加，机器学习算法可以处理增加的负载，而无需额外的资源。

数据不平衡是保险欺诈检测中的一个主要问题。由于欺诈性索赔相对于有效索赔的发生率较低，因此开发能够可靠识别欺诈的模型可能具有挑战性。可以通过过采样、欠采样和成本敏感学习等技术来平衡数据，以提高模型的性能，从而解决此问题。

Python 实现

在这里，我们将看到可用于保险欺诈检测的各种模型及其准确性。

导入库

# necessary imports

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import plotly.express as px

import warnings
warnings.filterwarnings('ignore')

plt.style.use('ggplot')

读取数据集

输出

该数据集包含 40 列。

# some missing values are denoted by '0', so let's replace missing values with np.nan

dataframe.replace('0', np.nan, inplace = True)
dataframe.describe()

输出

数据预处理

数据预处理是机器学习的关键步骤，涉及数据清理、转换、编码、选择、集成和缩减，以准备用于训练机器学习模型。数据的质量以及数据的准备方式会对模型的准确性和性能产生重大影响。

在这里，我们将执行以下操作：

可视化缺失值
处理缺失值
编码分类列
异常值检测

# looking for missing values
dataframe.isna().sum()

输出

我们的数据存在缺失值。

可视化缺失值

缺失值可能对机器学习模型造成问题，因为它们可能导致有偏见或不准确的结果。因此，可视化它们有助于理解缺失数据的范围和模式。

import missingno as msno

msno.bar(dataframe)
plt.show()

输出

处理缺失值

我们将把缺失值分配为 0 作为替代来处理缺失值。

dataframe['collision_type'] = dataframe['collision_type'].fillna(dataframe['collision_type'].mode()[0])
dataframe['property_damage'] = dataframe['property_damage'].fillna(dataframe['property_damage'].mode()[0])
dataframe['police_report_available'] = dataframe['police_report_available'].fillna(dataframe['police_report_available'].mode()[0])

dataframe.isna().sum()

输出

现在，我们的数据中没有缺失值。

# heatmap

plt.figure(figsize = (18, 12))

corr = dataframe.corr()

sns.heatmap(data = corr, annot = True, fmt = '.2g', linewidth = 1)
plt.show()

输出

# dropping columns that are not necessary for prediction

to_drop = ['policy_number','policy_bind_date','policy_state','insured_zip','incident_location','incident_date',
           'incident_state','incident_city','insured_hobbies','auto_make','auto_model','auto_year', '_c39']

dataframe.drop(to_drop, inplace = True, axis = 1)
dataframe.head()

输出

# checking for multicollinearity

plt.figure(figsize = (18, 12))

corr = dataframe.corr()
mask = np.triu(np.ones_like(corr, dtype = bool))

sns.heatmap(data = corr, mask = mask, annot = True, fmt = '.2g', linewidth = 1)
plt.show()

输出

从上图可以看出，年龄和客户月数之间存在高度相关性。我们将删除“年龄”列。此外，总索赔金额、伤害索赔、财产索赔和车辆索赔之间也存在高度相关性，因为总索赔是其他索赔的总和。因此，我们将删除总索赔列。

dataframe.drop(columns = ['age', 'total_claim_amount'], inplace = True, axis = 1)
dataframe.head()

输出

# separating the feature and target columns

X = dataframe.drop('fraud_reported', axis = 1)
y = dataframe['fraud_reported']

编码分类变量

它涉及将分类数据转换为机器学习模型可以处理的数值数据。

我们将把分类变量编码为数值数据，以便我们的模型更容易预测保险欺诈。

# extracting categorical columns
dataframe_cat = X.select_dtypes(include = ['object'])

输出

# printing unique values of each column
for col in dataframe_cat.columns:
    print(f"{col}: \n{dataframe_cat[col].unique()}\n")

输出

dataframe_cat = pd.get_dummies(dataframe_cat, drop_first = True)
dataframe_cat.head()

输出

# extracting the numerical columns

dataframe_num = X.select_dtypes(include = ['int64'])
dataframe_num.head()

输出

# combining the Numerical and Categorical dataframes to get the final dataset

X = pd.concat([dataframe_num, dataframe_cat], axis = 1)
X.head()

输出

plt.figure(figsize = (25, 20))
plotnumber = 1

for col in X.columns:
    if plotnumber <= 24:
        ax = plt.subplot(5, 5, plotnumber)
        sns.distplot(X[col])
        plt.xlabel(col, fontsize = 15)
       
    plotnumber += 1
   
plt.tight_layout()
plt.show()

输出

数据看起来不错。让我们检查一下异常值。

异常值检测

称为异常值的数据点与其他数据集中的数据点差异很大。异常值可能出现的原因有多种，包括测量错误、数据输入问题或固有数据变异性。统计分析和机器学习模型可能会受到异常值的影响，因为它们可能会提供有偏见的估计或不准确的预测。

我们将尝试查找数据中的异常值。

plt.figure(figsize = (20, 15))
plotnumber = 1

for col in X.columns:
    if plotnumber <= 24:
        ax = plt.subplot(5, 5, plotnumber)
        sns.boxplot(X[col])
        plt.xlabel(col, fontsize = 15)
   
    plotnumber += 1
plt.tight_layout()
plt.show()

输出

某些数值列存在异常值。稍后我们将对数值列进行缩放。

# splitting data into a training set and test set

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.25)
X_train.head()

输出

dataframe_num= X_train[['months_as_customer', 'policy_deductable', 'umbrella_limit',
       'capital-gains', 'capital-loss', 'incident_hour_of_the_day',
       'number_of_vehicles_involved', 'bodily_injuries', 'witnesses', 'injury_claim', 'property_claim',
       'vehicle_claim']]

# Scaling the numeric values in the dataset

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
scaled_data = scaler.fit_transform(dataframe_num)
scaled_dataframe_num = pd.DataFrame(data = scaled_data, columns = dataframe_num.columns, index = X_train.index)
scaled_dataframe_num.head()

输出

X_train.drop(columns = scaled_dataframe_num.columns, inplace = True)
X_train = pd.concat([scaled_dataframe_num, X_train], axis = 1)
X_train.head()

输出

型号

现在，我们将训练和测试以下模型：

支持向量分类器
Knn
决策树分类器
随机森林分类器
Ada Boost 分类器
梯度提升分类器
随机梯度提升 (SGB)
XgBoost
Cat Boost 分类器
Extra Trees 分类器
LGBM 分类器
投票分类器

我们还将检查模型的准确性。

1. SVM

from sklearn.svm import SVC

svc = SVC()
svc.fit(X_train, y_train)

y_pred = svc.predict(X_test)

# accuracy_score, confusion_matrix and classification_report

from sklearn.metrics import accuracy_score, confusion_matrix, classification_report

acc_svc_train = accuracy_score(y_train, svc.predict(X_train))
acc_svc_test = accuracy_score(y_test, y_pred)

print(f"Training accuracy of Support Vector Classifier is : {acc_svc_train}")
print(f"Test accuracy of Support Vector Classifier is : {acc_svc_test}")

print(confusion_matrix(y_test, y_pred))
print(classification_report(y_test, y_pred))

输出

2. KNN

from sklearn.neighbors import KNeighborsClassifier

knn = KNeighborsClassifier(n_neighbors = 30)
knn.fit(X_train, y_train)

y_pred = knn.predict(X_test)
# accuracy_score, confusion_matrix and classification_report

from sklearn.metrics import accuracy_score, confusion_matrix, classification_report

acc_knn_train = accuracy_score(y_train, knn.predict(X_train))
acc_knn_test = accuracy_score(y_test, y_pred)

print(f"Training accuracy of KNN is : {acc_knn_train}")
print(f"Test accuracy of KNN is : {acc_knn_test}")

print(confusion_matrix(y_test, y_pred))
print(classification_report(y_test, y_pred))

输出

3. 决策树分类器

from sklearn.tree import DecisionTreeClassifier

dt = DecisionTreeClassifier()
dt.fit(X_train, y_train)

y_pred = dt.predict(X_test)
# accuracy_score, confusion_matrix and classification_report

from sklearn.metrics import accuracy_score, confusion_matrix, classification_report

acc_dt_train = accuracy_score(y_train, dt.predict(X_train))
acc_dt_test = accuracy_score(y_test, y_pred)

print(f"Training accuracy of Decision Tree is : {acc_dt_train}")
print(f"Test accuracy of Decision Tree is : {acc_dt_test}")

print(confusion_matrix(y_test, y_pred))
print(classification_report(y_test, y_pred))

输出

# hyper parameter tuning

from sklearn.model_selection import GridSearchCV

params_grid = {
    'criterion' : ['gini', 'entropy'],
    'max_depth' : [3, 5, 7, 10],
    'min_samples_split' : range(2, 10, 1),
    'min_samples_leaf' : range(2, 10, 1)
}

search_grid = GridSearchCV(dt, params_grid, cv = 5, n_jobs = -1, verbose = 1)
search_grid.fit(X_train, y_train)

输出

# best parameters and the best score

print(search_grid.best_params_)
print(search_grid.best_score_)

输出

# best estimator

dt = search_grid.best_estimator_

y_pred = dt.predict(X_test)
# accuracy_score, confusion_matrix and classification_report

from sklearn.metrics import accuracy_score, confusion_matrix, classification_report

acc_dt_train = accuracy_score(y_train, dt.predict(X_train))
acc_dt_test = accuracy_score(y_test, y_pred)

print(f"Training accuracy of Decision Tree is : {acc_dt_train}")
print(f"Test accuracy of Decision Tree is : {acc_dt_test}")

print(confusion_matrix(y_test, y_pred))
print(classification_report(y_test, y_pred))

输出

4. 随机森林分类器

from sklearn.ensemble import RandomForestClassifier

rfc = RandomForestClassifier(criterion= 'entropy', max_depth= 10, max_features= 'sqrt', min_samples_leaf= 1, min_samples_split= 3, n_estimators= 140)
rfc.fit(X_train, y_train)

y_pred = rfc.predict(X_test)
# accuracy_score, confusion_matrix and classification_report

from sklearn.metrics import accuracy_score, confusion_matrix, classification_report

acc_rfc_train = accuracy_score(y_train, rfc.predict(X_train))
acc_rfc_test = accuracy_score(y_test, y_pred)

print(f"Training accuracy of Random Forest is : {acc_rfc_train}")
print(f"Test accuracy of Random Forest is : {acc_rfc_test}")

print(confusion_matrix(y_test, y_pred))
print(classification_report(y_test, y_pred))

输出

5. Ada Boost 分类器

from sklearn.ensemble import AdaBoostClassifier

ada = AdaBoostClassifier(base_estimator = dt)

parameters = {
    'n_estimators' : [50, 70, 90, 120, 180, 200],
    'learning_rate' : [0.001, 0.01, 0.1, 1, 10],
    'algorithm' : ['SAMME', 'SAMME.R']
}

search_grid = GridSearchCV(ada, parameters, n_jobs = -1, cv = 5, verbose = 1)
search_grid.fit(X_train, y_train)

输出

# best parameter and the best score

print(search_grid.best_params_)
print(search_grid.best_score_)

输出

# best estimator

ada = search_grid.best_estimator_

y_pred = ada.predict(X_test)
# accuracy_score, confusion_matrix and classification_report

acc_ada_train = accuracy_score(y_train, ada.predict(X_train))
acc_ada_test = accuracy_score(y_test, y_pred)

print(f"Training accuracy of Ada Boost is : {acc_ada_train}")
print(f"Test accuracy of Ada Boost is : {acc_ada_test}")

print(confusion_matrix(y_test, y_pred))
print(classification_report(y_test, y_pred))

输出

6. 梯度提升分类器

from sklearn.ensemble import GradientBoostingClassifier

gb = GradientBoostingClassifier()
gb.fit(X_train, y_train)

# accuracy score, confusion matrix, and classification report of gradient boosting classifier

acc_gb = accuracy_score(y_test, gb.predict(X_test))

print(f"Training Accuracy of Gradient Boosting Classifier is {accuracy_score(y_train, gb.predict(X_train))}")
print(f"Test Accuracy of Gradient Boosting Classifier is {acc_gb} \n")

print(f"Confusion Matrix :- \n{confusion_matrix(y_test, gb.predict(X_test))}\n")
print(f"Classification Report :- \n {classification_report(y_test, gb.predict(X_test))}")

输出

7. 随机梯度提升 (SGB)

sgb = GradientBoostingClassifier(subsample = 0.90, max_features = 0.70)
sgb.fit(X_train, y_train)

# accuracy score, confusion matrix, and classification report of stochastic gradient boosting classifier

acc_sgb = accuracy_score(y_test, sgb.predict(X_test))

print(f"Training Accuracy of Stochastic Gradient Boosting is {accuracy_score(y_train, sgb.predict(X_train))}")
print(f"Test Accuracy of Stochastic Gradient Boosting is {acc_sgb} \n")

print(f"Confusion Matrix :- \n{confusion_matrix(y_test, sgb.predict(X_test))}\n")
print(f"Classification Report :- \n {classification_report(y_test, sgb.predict(X_test))}")

输出

8. XGBoost 分类器

from xgboost import XGBClassifier

xgb = XGBClassifier()
xgb.fit(X_train, y_train)

y_pred = xgb.predict(X_test)
# accuracy_score, confusion_matrix and classification_report

acc_xgb_train = accuracy_score(y_train, xgb.predict(X_train))
acc_xgb_test = accuracy_score(y_test, y_pred)

print(f"Training accuracy of XgBoost is : {acc_xgb_train}")
print(f"Test accuracy of XgBoost is : {acc_xgb_test}")

print(confusion_matrix(y_test, y_pred))
print(classification_report(y_test, y_pred))

输出

grid_params = {"n_estimators": [10, 50, 100, 130], "criterion": ['gini', 'entropy'],
                               "max_depth": range(2, 10, 1)}

grid = GridSearchCV(estimator=xgb, grid_params=grid_params, cv=5,  verbose=3,n_jobs=-1)
search_grid.fit(X_train, y_train)

输出

# best estimator

xgb = search_grid.best_estimator_

y_pred = xgb.predict(X_test)
# accuracy_score, confusion_matrix and classification_report

acc_xgb_train = accuracy_score(y_train, xgb.predict(X_train))
acc_xgb_test = accuracy_score(y_test, y_pred)

print(f"Training accuracy of XgBoost is : {acc_xgb_train}")
print(f"Test accuracy of XgBoost is : {acc_xgb_test}")

print(confusion_matrix(y_test, y_pred))
print(classification_report(y_test, y_pred))

输出

9. Cat Boost 分类器

from catboost import CatBoostClassifier

cat = CatBoostClassifier(iterations=10)
cat.fit(X_train, y_train)

输出

# accuracy score, confusion matrix, and classification report of cat boost

acc_cat = accuracy_score(y_test, cat.predict(X_test))

print(f"Training Accuracy of Cat Boost Classifier is {accuracy_score(y_train, cat.predict(X_train))}")
print(f"Test Accuracy of Cat Boost Classifier is {acc_cat} \n")

print(f"Confusion Matrix :- \n{confusion_matrix(y_test, cat.predict(X_test))}\n")
print(f"Classification Report :- \n {classification_report(y_test, cat.predict(X_test))}")

输出

10. 极端随机树分类器

from sklearn.ensemble import ExtraTreesClassifier

etc = ExtraTreesClassifier()
etc.fit(X_train, y_train)

# accuracy score, confusion matrix, and classification report of extra trees classifier

acc_etc = accuracy_score(y_test, etc.predict(X_test))

print(f"Training Accuracy of Extra Trees Classifier is {accuracy_score(y_train, etc.predict(X_train))}")
print(f"Test Accuracy of Extra Trees Classifier is {acc_etc} \n")

print(f"Confusion Matrix :- \n{confusion_matrix(y_test, etc.predict(X_test))}\n")
print(f"Classification Report :- \n {classification_report(y_test, etc.predict(X_test))}")

输出

11. LGBM 分类器

from lightgbm import LGBMClassifier

lgbm = LGBMClassifier(learning_rate = 1)
lgbm.fit(X_train, y_train)

# accuracy score, confusion matrix, and classification report of lgbm classifier

acc_lgbm = accuracy_score(y_test, lgbm.predict(X_test))

print(f"Training Accuracy of LGBM Classifier is {accuracy_score(y_train, lgbm.predict(X_train))}")
print(f"Test Accuracy of LGBM Classifier is {acc_lgbm} \n")

print(f"{confusion_matrix(y_test, lgbm.predict(X_test))}\n")
print(classification_report(y_test, lgbm.predict(X_test)))

输出

12. 投票分类器

from sklearn.ensemble import VotingClassifier

classifiers = [('Support Vector Classifier', svc), ('KNN', knn),  ('Decision Tree', dt), ('Random Forest', rfc),
               ('Ada Boost', ada), ('XGboost', xgb), ('Gradient Boosting Classifier', gb), ('SGB', sgb),
               ('Cat Boost', cat), ('Extra Trees Classifier', etc), ('LGBM', lgbm)]

vc = VotingClassifier(estimators = classifiers)
vc.fit(X_train, y_train)

y_pred = vc.predict(X_test)

输出

# accuracy_score, confusion_matrix and classification_report

vc_train_acc = accuracy_score(y_train, vc.predict(X_train))
vc_test_acc = accuracy_score(y_test, y_pred)

print(f"Training accuracy of Voting Classifier is : {vc_train_acc}")
print(f"Test accuracy of Voting Classifier is : {vc_test_acc}")

print(confusion_matrix(y_test, y_pred))
print(classification_report(y_test, y_pred))

输出

比较模型

我们已经训练和测试了我们的模型，现在是时候进行比较了，以便我们能够找到最适合保险欺诈检测的模型。

models = pd.DataFrame({
    'Model' : ['SVC', 'KNN', 'Decision Tree', 'Random Forest','Ada Boost', 'Gradient Boost', 'SGB', 'Cat Boost', 'Extra Trees', 'LGBM', 'XgBoost', 'Voting Classifier'],
    'Score' : [acc_svc_test, acc_knn_test, acc_dt_test, acc_rfc_test, acc_ada_test, acc_gb, acc_sgb, acc_cat, acc_etc, acc_lgbm, acc_xgb_test, vc_test_acc]
})


models.sort_values(by = 'Score', ascending = False)

输出

决策树分类器的性能最高，为 79%，而随机梯度提升 (SGB) 的性能最低，为 31%。

因此，我们可以说 DTC 是保险欺诈检测的最佳模型之一。

可视化模型比较。

px.bar(data_frame = models, x = 'Score', y = 'Model', color = 'Score', template = 'plotly_dark',
       title = 'Models Comparison')

输出

结论

保险欺诈是一个严重的问题，会对保险公司及其客户产生负面影响。通过定位数据中的模式和异常，可以使用机器学习算法来检测和阻止欺诈。为确保模型的准确性和效率，选择合适的方法并管理数据的不平衡性至关重要。

请记住，我们在选择模型时需要非常谨慎，因为它将对预测产生更大的影响。

下一个主题微分和积分微积分

保险欺诈检测 - 机器学习

机器学习在欺诈检测方面的优势

Python 实现

比较模型

结论

联系信息

关注我们

教程

面试题

在线编译器

Python

Java

.Net Framework

AI, ML and Data Science

Cloud Technology

B.Tech and MCA

Web Technology

PHP

Software Testing

Technical Interview

Java Interview

Python

Web Interview

Database Interview

B.Tech / MCA

Important Interview

Software Testing Interview

Company Interviews

Online Compilers

Multiple Choice Questions

机器学习

监督式学习

分类

杂项

相关教程

面试题

保险欺诈检测 - 机器学习

机器学习在欺诈检测方面的优势

Python 实现

比较模型

结论

相关帖子

机器学习中的提前停止

使用迁移学习进行狗品种分类

机器学习如何与区块链结合使用

机器学习中的概率模型

机器学习中的网络钓鱼网站检测

机器学习很难吗

机器学习中的猫分类

最大似然估计简介

图像分割中的 V-Net

Python 初学者十大机器学习项目

订阅 Tpoint Tech

联系信息

关注我们

教程

面试题

在线编译器