Naive Evaluation Overstates Re-Identification Risk in Shared Honeypot Threat Intelligence

Abstract

Community-scale threat-intelligence sharing, exemplified by the Honey Community model, extends honeypot-based deception beyond a single organization by having participants share anonymized threat data. This model assumes anonymization prevents shared data from revealing which participant generated it — an assumption raised qualitatively in prior work but not, to our knowledge, tested using an evaluation methodology that distinguishes genuine site-level fingerprinting from mere memorization of previously-seen attackers. Using a real, publicly available multi-site honeypot dataset (MURHCAD) spanning four geographically distinct cloud deployments, we train classifiers to re-identify the source site from two anonymized data tiers of increasing strictness. A naive evaluation protocol — standard random cross-validation without attacker-level grouping — suggests strong re-identification capability (83.1% accuracy with a Random Forest against a 40.1% majority-class baseline). An evaluation protocol that holds out attackers entirely unseen during training, reflecting the realistic scenario for a growing threat-sharing community, shows this naive estimate is substantially inflated regardless of model choice: the naive-to-grouped accuracy gap is 46.8 points for Random Forest and 30.2 points for Gradient Boosting. We trace this gap to memorization of individual attacker identities, most pronounced for Random Forest (99.5% training accuracy vs. 37.1% held-out accuracy), and confirm it is systemic across many repeat attackers rather than dependent on any single high-volume outlier. Notably, the two model families disagree on whether genuine site-level signal exceeds a naive baseline once memorization is controlled for — Random Forest's grouped accuracy falls below the majorityclass baseline (36.3%), while Gradient Boosting's exceeds it (49.0%) — indicating that conclusions about the existence of residual re-identification risk, not just its magnitude, can be model-dependent. These results suggest that evaluation methodology, as much as anonymization technique, determines conclusions drawn about privacy risk in shared security data.

KEYWORDS: honeypot, threat intelligence sharing, privacy, re-identification, deception, community defense, evaluation methodology