McNemar 检验

17 Mar 2025 | 6 分钟阅读

麦克尼马尔检验是一种统计检验，用于确定 2x2 列联表中两个相关分类变量的边际频率（计数）是否存在显著差异。它通常用于配对分类数据，例如在前后实验中，或比较同一数据集上的两种不同分类器时。

检验统计量具有一个自由度的卡方分布。要获得 p 值，请将估计值与具有一个自由度的卡方分布中的关键值进行比较。

如果 p 值小于您指定的显著性水平（例如 0.05），则拒绝原假设。如果 p 值超过您指定的显著性水平，则无法拒绝原假设。

代码

现在我们将实现麦克尼马尔检验，并将其与其他近似统计检验进行比较。

导入库

import numpy as np
import pandas as pd

from sklearn.model_selection import StratifiedKFold

from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder, FunctionTransformer
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.utils import estimator_html_repr

from tqdm.notebook import tqdm
from IPython.core.display import HTML

读取数据集

df = pd.read_csv('../input/spaceship-titanic/train.csv')
print(df.shape)
df.head()

输出

X_test = pd.read_csv('../input/spaceship-titanic/test.csv')

X = df.drop('Transported', axis=1)
y = df['Transported'].astype(int)

模型

现在我们将构建我们的模型。

# Some helper functions for feature extraction

def get_deck_and_side(df):
    return np.array([[x.split('/')[0], x.split('/')[2]] for x in np.ravel(df)])

def get_2d_to_1d(df):
    return df.reshape(-1)


# Preprocessing Components
comp_1_fn= lambda :Pipeline([
    ('imputer', SimpleImputer(strategy='median')), ('normalizer', StandardScaler())
])
comp_2_fn = lambda: Pipeline([
    ('imputer', SimpleImputer(strategy='most_frequent')), 
    ('normalizer', OneHotEncoder(sparse=False, drop='if_binary'))
])
comp_3_fn = lambda: Pipeline([
    ('imputer', SimpleImputer(strategy='constant', fill_value='missing')), 
    ('normalizer', OneHotEncoder())
])
comp_4_fn = lambda: Pipeline([
    ('imputer', SimpleImputer(strategy='constant', fill_value='//')), 
    ('function', FunctionTransformer(get_deck_and_side)),
    ('normalizer', OneHotEncoder())
])
comp_5_fn = lambda: Pipeline([
    ('imputer', SimpleImputer(strategy='constant', fill_value='')),
    ("reshape", FunctionTransformer(get_2d_to_1d)),
    ('normalizer', TfidfVectorizer(analyzer='char', ngram_range=(2, 3)))
])

# Preprocessors
preprocessor_1_fn = lambda: ColumnTransformer([
    ("scaler", comp_1_fn(), ['Age', 'RoomService', 'FoodCourt', 'ShoppingMall', 'Spa', 'VRDeck']),
    ("bin_encoder", comp_2_fn(), ['CryoSleep', 'VIP']),
    ('ohe_encoder', comp_3_fn(), ['HomePlanet', 'Destination']),
])
preprocessor_2_fn = lambda: ColumnTransformer([
    ("scaler", comp_1_fn(), ['Age', 'RoomService', 'FoodCourt', 'ShoppingMall', 'Spa', 'VRDeck']),
    ("bin_encoder", comp_2_fn(), ['CryoSleep', 'VIP']),
    ('ohe_encoder', comp_3_fn(), ['HomePlanet', 'Destination']),
    ('deck', comp_4_fn(), ['Cabin']),
    ('side', comp_5_fn(), ['Name']),
])

# Learning Algorithm
estimator_fn = lambda: LogisticRegression(random_state=0, max_iter=100_000)

# Models
model_1_fn = lambda: Pipeline([('preprocessor', preprocessor_1_fn()), ('estimator', estimator_fn())])
model_2_fn = lambda: Pipeline([('preprocessor', preprocessor_2_fn()), ('estimator', estimator_fn())])

我们将生成所构建的估计器模型的 HTML 表示。

输出

训练

我们将使用分层 K 折交叉验证来训练模型，这有助于获得更可靠的模型性能估计并降低过拟合的风险

def train_skf(model_fn):
    oof_preds = y.copy()
    scores = []
    preds = []

    for fold, (train_index, valid_index) in tqdm(enumerate(skf.split(X, y))):
        X_train, X_valid = X.iloc[train_index], X.iloc[valid_index]
        y_train, y_valid = y.iloc[train_index], y.iloc[valid_index]

        model = model_fn().fit(X_train, y_train)
        preds.append(model.predict_proba(X_test)[:, 1])
        oof_preds.iloc[valid_index] = model.predict_proba(X_valid)[:, 1]
        scores.append(model.score(X_valid, y_valid))
        
    preds = np.array(preds).mean(axis=0)
    
    return oof_preds, scores, preds

skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=True)
oof_1, scores_1, preds_1 = train_skf(model_1_fn)
print('Model 1 Score:', np.mean(preds_1))
oof_2, scores_2, preds_2 = train_skf(model_2_fn)
print('Model 2 Score:', np.mean(preds_2))

输出

麦克尼马尔检验

麦克尼马尔检验旨在主要关注两个分类器之间的差异，因此也关注它们以不同方式预测的案例。所以我们首先需要做的是计算以下值。

n00：A 和 B 都分类错误的项数
n01：A 分类错误但 B 未分类错误的项数
n10：B 分类错误但 A 未分类错误的项数
n11：A 和 B 都分类正确的项数

原假设：n01 = n10，表明 A 和 B 具有相同的错误率。

要获得麦克尼马尔检验的 X² 值，该值类似于配对卡方检验，请使用以下公式。

在双尾检验中，如果 X² > X²(p)，则拒绝原假设。其中 p 是显著性水平。

Y = pd.DataFrame(y.copy())
Y['model_1'] = np.where(oof_1 > 0.5, 1, 0)
Y['model_2'] = np.where(oof_2 > 0.5, 1, 0)
Y.head()

输出

from statsmodels.stats.contingency_tables import mcnemar

def get_contingency_table(Y, ground_truth, model_1, model_2):
    contingency_table = [[0, 0], [0, 0]]
    Y_ = Y.copy()
    model_1_correct = Y_.apply(lambda row: int(row[ground_truth] == row[model_1]), axis=1)
    model_2_correct = Y_.apply(lambda row: int(row[ground_truth] == row[model_2]), axis=1)
    contingency_table[0][0] = Y_.apply(
        lambda row: int(row[model_1] == 0 and row[model_2] == 0), axis=1
    ).sum()
    contingency_table[0][1] = Y_.apply(
        lambda row: int(row[model_1] == 0 and row[model_2] == 1), axis=1
    ).sum()
    contingency_table[1][0] = Y_.apply(
        lambda row: int(row[model_1] == 1 and row[model_2] == 0), axis=1
    ).sum()
    contingency_table[1][1] = Y_.apply(
        lambda row: int(row[model_1] == 1 and row[model_2] == 1), axis=1
    ).sum()
    return np.array(contingency_table)

def mcnemar_test(contigency_table, significance=0.05):
    print("Contigency Table")
    print(contigency_table)
    test = mcnemar(contigency_table, exact=False, correction=True)
    print("P value:", test.pvalue)
    if test.pvalue > significance:
        print("Reject Null Hypotheis")
        print("Conclusion: Model have statistically different error rate")
    else:
        print("Accept Null Hypotheis")
        print("Conclusion: Model do not have statistically different error rate")

mcnemar_test(get_contingency_table(Y, 'Transported', 'model_1', 'model_2'))

与其他检验的比较

现在我们将麦克尼马尔检验与其他近似统计检验进行比较。我们将比较

比例 Z 检验
麦克尼马尔检验
重采样配对 t 检验
交叉验证配对 t 检验
5x2 交叉验证配对 t 检验

导入库

import numpy as np
import pandas as pd

from tqdm import tqdm
from scipy.stats import norm, chi2
from scipy.stats import t as t_dist
from sklearn.datasets import load_wine
from sklearn.metrics import accuracy_score
from sklearn.ensemble import RandomForestClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import train_test_split, KFold

# Libs implementations
from mlxtend.evaluate import mcnemar
from mlxtend.evaluate import mcnemar_table
from mlxtend.evaluate import paired_ttest_5x2cv
from mlxtend.evaluate import proportion_difference
from mlxtend.evaluate import paired_ttest_kfold_cv
from mlxtend.evaluate import paired_ttest_resampled

定义方法

在这里，我们将创建每个近似统计检验的函数。

def paired_t_test(p):
    p_hat = np.mean(p)
    n = len(p)
    den = np.sqrt(sum([(diff - p_hat)**2 for diff in p]) / (n - 1))
    t = (p_hat * (n**(1/2))) / den
    
    p_value = t_dist.sf(t, n-1)*2
    
    return t, p_value

def mcnemar_test(y_true, y_1, y_2):
    b = sum(np.logical_and((knn_y != y_test),(rf_y == y_test)))
    c = sum(np.logical_and((knn_y == y_test),(rf_y != y_test)))
    
    c_ = (np.abs(b - c) - 1)**2 / (b + c)
    
    p_value = chi2.sf(c_, 1)
    return c_, p_value

def z_test(pa, pb, n):
    p = (pa + pb)/2
    z = (pa - pb) / (((2*p*(1-p))/n)**(1/2))
    
    p_value = norm.cdf(z)
    
    return z, p_value

def five_two_statistic(p1, p2):
    p1 = np.array(p1)
    p2 = np.array(p2)
    p_hat = (p1 + p2) / 2
    s = (p1 - p_hat)**2 + (p2 - p_hat)**2
    t = p1[0] / np.sqrt(1/5. * sum(s))
    
    p_value = t_dist.sf(t, 5)*2
    
    return t, p_value

加载数据集

定义分类器

# We will have two classifiers Random Forest and KNN
rf = RandomForestClassifier(random_state=42)
knn = KNeighborsClassifier(n_neighbors=1)

现在我们将执行各种统计检验，以比较两种机器学习模型（随机森林和 KNN）的性能。

# For holdout cases
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=42)
rf.fit(X_train, y_train)
knn.fit(X_train, y_train)

rf_y = rf.predict(X_test)
knn_y = knn.predict(X_test)

acc1 = accuracy_score(y_test, rf_y)
acc2 = accuracy_score(y_test, knn_y)

print("Proportions Z-Test")
z, p = proportion_difference(acc1, acc2, n_1=len(y_test))
print(f"z statistic: {z}, p-value: {p}\n")

print("McNemar's test")
chi2_, p = mcnemar_test(y_test, rf_y, knn_y)
print(f"chi� statistic: {chi2_}, p-value: {p}\n")

n_tests = 30

p_ = []
rng = np.random.RandomState(42)
for i in range(n_tests):
    randint = rng.randint(low=0, high=32767)
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=randint)
    rf.fit(X_train, y_train)
    knn.fit(X_train, y_train)

    acc1 = accuracy_score(y_test, rf.predict(X_test))
    acc2 = accuracy_score(y_test, knn.predict(X_test))
    p_.append(acc1 - acc2)
    
print("Paired t-test Resampled")
t, p = paired_t_test(p_)
print(f"t statistic: {t}, p-value: {p}\n")

p_ = []
# For cross-validated cases
kf = KFold(n_splits=10, shuffle=True, random_state=42)
for train_index, test_index in kf.split(X):
    X_train, X_test, y_train, y_test = X[train_index], X[test_index], y[train_index], y[test_index]
    rf.fit(X_train, y_train)
    knn.fit(X_train, y_train)

    acc1 = accuracy_score(y_test, rf.predict(X_test))
    acc2 = accuracy_score(y_test, knn.predict(X_test))
    p_.append(acc1 - acc2)

print("Cross Validated Paired t-test")
t, p = paired_t_test(p_)
print(f"t statistic: {t}, p-value: {p}\n")

p_1 = []
p_2 = []
# For the 5x2cv
rng = np.random.RandomState(42)
for i in range(5):
    randint = rng.randint(low=0, high=32767)
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.50, random_state=randint)

    rf.fit(X_train, y_train)
    knn.fit(X_train, y_train)
    acc1 = accuracy_score(y_test, rf.predict(X_test))
    acc2 = accuracy_score(y_test, knn.predict(X_test))
    p_1.append(acc1 - acc2)

    rf.fit(X_test, y_test)
    knn.fit(X_test, y_test)
    acc1 = accuracy_score(y_train, rf.predict(X_train))
    acc2 = accuracy_score(y_train, knn.predict(X_train))
    p_2.append(acc1 - acc2)

print("5x2 CV Paired t-test")     
t, p = five_two_statistic(p_1, p_2)
print(f"t statistic: {t}, p-value: {p}\n")

输出

print("Proportions Z-Test")

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=42)
rf.fit(X_train, y_train)
knn.fit(X_train, y_train)

rf_y = rf.predict(X_test)
knn_y = knn.predict(X_test)

acc1 = accuracy_score(y_test, rf_y)
acc2 = accuracy_score(y_test, knn_y)

z, p = proportion_difference(acc1, acc2, n_1=len(y_test))
print(f"z statistic: {z}, p-value: {p}\n")

print("McNemar's test")
table = mcnemar_table(y_target=y_test, y_model1=rf_y, y_model2=knn_y)
chi2_, p = mcnemar(ary=table, corrected=True)
print(f"chi� statistic: {chi2_}, p-value: {p}\n")

print("Paired t-test Resampled")
t, p = paired_ttest_resampled(estimator1=rf, estimator2=knn, X=X, y=y, random_seed=42, num_rounds=30, test_size=0.2)
print(f"t statistic: {t}, p-value: {p}\n")

print("Cross Validated Paired t-test")
t, p = paired_ttest_kfold_cv(estimator1=rf, estimator2=knn, X=X, y=y, random_seed=42, shuffle=True, cv=10)
print(f"t statistic: {t}, p-value: {p}\n")

print("5x2 CV Paired t-test")
t, p = paired_ttest_5x2cv(estimator1=rf, estimator2=knn, X=X, y=y, random_seed=42)
print(f"t statistic: {t}, p-value: {p}\n")

输出

虽然比例 Z 检验发现两种模型之间的比例没有显著差异，但麦克尼马尔检验和所有三个 t 检验（重采样配对 t 检验、交叉验证配对 t 检验和 5x2 交叉验证配对 t 检验）均显示模型平均性能得分存在显著差异。

下一主题随机优化

McNemar 检验

导入库

读取数据集

模型

训练

麦克尼马尔检验

与其他检验的比较

导入库

定义方法

加载数据集

定义分类器

联系信息

关注我们

教程

面试题

在线编译器

Python

Java

.Net Framework

AI, ML and Data Science

Cloud Technology

B.Tech and MCA

Web Technology

PHP

Software Testing

Technical Interview

Java Interview

Python

Web Interview

Database Interview

B.Tech / MCA

Important Interview

Software Testing Interview

Company Interviews

Online Compilers

Multiple Choice Questions

机器学习

监督式学习

分类

杂项

相关教程

面试题

McNemar 检验

导入库

读取数据集

模型

训练

麦克尼马尔检验

与其他检验的比较

导入库

定义方法

加载数据集

定义分类器

相关帖子

2025 年机器学习最新研究课题

Pandas 中处理大型数据集

5G 技术对物联网有哪些好处？

分类与回归

处理不平衡数据的分类

机器学习中的地磁场

机器学习中的元学习

张量处理单元

回声状态网络

随机优化

订阅 Tpoint Tech

联系信息

关注我们

教程

面试题

在线编译器