Leakage-Aware Multi-Class Email Classification Using Duplicate-Aware Evaluation and Automated Machine Learning
DOI:
https://doi.org/10.65405/9pvbat43Keywords:
Email classification, spam detection, phishing detection, data leakage, duplicate-aware evaluation, TF-IDF, LinearSVC, automated machine learning.Abstract
Reliable evaluation is important in multi-class email classification because aggregated text datasets may contain duplicated messages or overlapping content that can affect separation between model-development and evaluation data. This study presents a leakage-aware framework for classifying emails into Ham, Phish, and Spam. The framework integrates data cleaning, duplicate auditing, removal of conflicting duplicate groups, group-aware partitioning, training-only feature construction, stratified group cross-validation, locked-test evaluation, error analysis, and bootstrap confidence intervals. The downloaded dataset contained 365,448 records; after cleaning and removing conflicting groups, 365,403 remained. A five-fold StratifiedGroupKFold procedure produced a locked test set of 73,038 messages with zero duplicate-group overlap. Word-level TF-IDF with LinearSVC was used as the primary classifier, while a complementary H2O AutoML branch used 128-component SVD. LinearSVC achieved 98.00% accuracy and 97.32% macro F1 on the locked test set. The findings emphasize the importance of controlling duplicate structure and information flow when evaluating multi-class email classifiers
Downloads
References
[1] Cohen, W. W. (1996). Learning rules that classify e-mail. In AAAI Spring Symposium on Machine Learning and Information Access.
[2] Salton, G., & Buckley, C. (1988). Term-weighting approaches in automatic text retrieval. Information Processing & Management, 24(5), 513–523. https://doi.org/10.1016/0306-4573(88)90021-0
[3] Joachims, T. (1998). Text categorization with support vector machines: Learning with many relevant features. In Proceedings of the 10th European Conference on Machine Learning (ECML) (Lecture Notes in Computer Science, Vol. 1398, pp. 137–142). Springer. https://doi.org/10.1007/BFb0026683
[4] Das, A., Baki, S., El Aassal, A., Verma, R., & Dunbar, A. (2020). SoK: A comprehensive reexamination of phishing research from the security perspective. IEEE Communications Surveys & Tutorials, 22(1), 671–708. https://doi.org/10.1109/COMST.2019.2957750
[5] Alhogail, A., & Alsabih, A. (2021). Applying machine learning and natural language processing to detect phishing email. Computers & Security, 110, Article 102414. https://doi.org/10.1016/j.cose.2021.102414
[6] Salloum, S. A., Gaber, T., Vadera, S., & Shaalan, K. (2022). A systematic literature review on phishing email detection using natural language processing techniques. IEEE Access, 10, 65703–65727. https://doi.org/10.1109/ACCESS.2022.3183083
[7] Bountakas, P., & Xenakis, C. (2023). HELPHED: Hybrid Ensemble Learning PHishing Email Detection. Journal of Network and Computer Applications, 210, Article 103545. https://doi.org/10.1016/j.jnca.2022.103545
[8] Kaufman, S., Rosset, S., Perlich, C., & Stitelman, O. (2012). Leakage in data mining: Formulation, detection, and avoidance. ACM Transactions on Knowledge Discovery from Data, 6(4), Article 15. https://doi.org/10.1145/2382577.2382579
[9] Kapoor, S., & Narayanan, A. (2023). Leakage and the reproducibility crisis in machine-learning-based science. Patterns, 4(9), Article 100804. https://doi.org/10.1016/j.patter.2023.100804
[10] Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., & Carlini, N. (2022). Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (pp. 8424–8445). Association for Computational Linguistics. https://doi.org/10.18653/v1/2022.acl-long.577
[11] Roberts, D. R., Bahn, V., Ciuti, S., Boyce, M. S., Elith, J., Guillera-Arroita, G., et al. (2017). Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure. Ecography, 40(8), 913–929. https://doi.org/10.1111/ecog.02881
[12] LeDell, E., & Poirier, S. (2020). H2O AutoML: Scalable automatic machine learning. In 7th ICML Workshop on Automated Machine Learning.
[13] Kambhatla, G., Nguyen, T., & Choi, E. (2023). Quantifying train-evaluation overlap with nearest neighbors. In Findings of the Association for Computational Linguistics: ACL 2023 (pp. 2905–2920). Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.findings-acl.183
[14] Saka, T., Vaniea, K., & Kökciyan, N. (2025). SoK: Grouping spam and phishing email threats for smarter security. IEEE Access, 13, 54938–54959. https://doi.org/10.1109/ACCESS.2025.3555157
[15] Sharma, A. (2026). The biggest spam ham phish email dataset (250000+) (Version 1) [Data set]. Kaggle.
[16] Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Vol. 1, pp. 4171–4186). Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1423
[17] J. Doshi, K. Parmar, R. Sanghavi, and N. Shekokar, “A comprehensive dual-layer architecture for phishing and spam email detection,” Computers & Security, vol. 133, Art. no. 103378, 2023, doi: 10.1016/j.cose.2023.103378.
