2020-10-11 · preprint · Imbalanced Data Classification Techniques · en
highly citedfrom the source record
Background. LLM-as-Judge is increasingly adopted in automated program repair (APR) to assess patch correctness, yet its validity as a measurement instrument remains unaudited: potential benchmark-identifier leakage, evidence-presentation bias, and explanation unreliability have not been systematically examined. Aims. We investigate whether LLM Judge produces unbiased, stable, and well-grounded patch correctness judgments across three validity dimensions: metadata leakage, evidence presentation, and explanation reliability. Method. On a balanced, paired dataset of 326 patches from 163 Defects4J bugs, we evaluate GPT-4o and DeepSeek-V3 across a 12-setting prompt matrix covering benchmark-identifying metadata exposure and evidence presentation variants. We further conduct a manual analysis of 60 GPT-4o explanations using a five-category failure taxonomy and three independent annotators. Results. Under sanitized conditions, GPT-4o achieves an accuracy of 77.3% (MCC = 0.575, FPR = 17.2%), while DeepSeek-V3 achieves 80.4% accuracy (MCC = 0.625, FPR = 22.1%), with no statistically significant difference between the two models. Exposing combined benchmark-identifying metadata yields a suggestive accuracy increase of 4.3 percentage points for GPT-4o and 1.8 percentage points for DeepSeek-V3. Declaring that all tests have passed does not significantly bias either model; in contrast, providing a developer reference patch leads to substantial and statistically significant improvements for both models. Explanation failures are concentrated in misclassified cases, with the false-positive quadrant showing a 93.3% failure rate, primarily driven by hallucinated evidence and missed edge cases. Conclusions. LLM Judge is useful as a triage aid but is insufficient as a standalone correctness oracle. Reliable deployment requires prompt sanitization, explicit false positive rate reporting, clear distinction between reference-assisted and standalone assessment, and treating LLM explanations as investigative hypotheses rather than self-validating justifications.
Claims drawn from cited facts, not live model generation.
This paper focuses on imbalanced data classification techniques and is presented as a preprint. As a preprint, this work has not yet undergone peer review, so its conclusions warrant correspondingly cautious weight.
4 cited facts
This work was produced by a small team of three researchers.
1 cited fact
The work has accumulated 5,439 citations, a figure that marks it as an established reference point in its field. That highly-cited status reflects outsized influence rather than an early or marginal signal of uptake. It draws on a comparatively compact base of 24 referenced works, building its wide reception on a focused set of sources.
4 cited facts
Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation is a preprint, focusing on Imbalanced Data Classification Techniques, published in 2020.
Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation has 5,439 citations in our record, placing it among the highly cited works we track.
Our record lists 3 authors for Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation and references 24 other works. Author names aren't in our data yet — the source link on the page lists the full byline.
Source facts, citations, and refresh stamp for this record.
Sources: openalex_works