Paxmod × AIML

How We Built Grooming Detection with the Australian Institute for Machine Learning

Alan Agon

July 18, 2026 · 6 min read

Grooming is not a bad word. It is a pattern. Most chat moderation looks for a single toxic message: a slur, a threat, a link. Grooming defeats that entirely. A predator's early messages are indistinguishable from ordinary friendly chat. The danger is not any one line, it is the trajectory across dozens of turns, from building rapport, to isolation, to secret-keeping, to escalation. A per-message filter, however good, is structurally blind to it.

That is why we partnered with the Australian Institute for Machine Learning (AIML), through the Industrial AI SME Grant Program, to build a system that reads a conversation the way a trained analyst would: as a whole, not one message at a time.

What we built

Instead of scoring messages one at a time, the system models a conversation as a graph. Each message is a node, enriched with a fine-tuned BERT understanding of the text, and the connections between messages let a Graph Attention Network learn how risk builds across turns. It operates over a sliding window of recent messages, which keeps cost bounded even as conversations grow long enough to run at more than a million messages per day.

Figure 1. Detection pipeline. A sliding window of recent messages is encoded once by a fine-tuned BERT model (served via ONNX / TensorRT), assembled into a conversation graph with temporal edges, and scored by a Graph Attention Network in a single forward pass. The window bounds compute as conversations grow, holding end-to-end latency under 50ms at production volume.

How the attention works

The attention mechanism is the heart of it. For each message, the model learns how much weight to give its neighbours. Phase-advanced messages (“how old r u”, “are u alone?”) draw the attention, while genuinely benign chatter is largely ignored. The result is a per-user risk score that no single message could produce on its own.

Figure 2. Attention update at a single node. The layer learns an attention weight for each edge; the phase-advanced neighbours M2 and M4 draw most of the attention while benign messages M1 and M5 are largely ignored. The node's new embedding is the attention-weighted sum of its neighbours, and the classifier reads from it to produce a predator probability.

The results

Adding conversational structure is what closes the gap. A BERT-only per-message classifier sits near 50 to 60% precision. Adding the Graph Attention Network over the conversation graph lifts precision to ~92% at 81% recall on the PAN12 benchmark. Precision is the operative metric: a system that raises constant false alarms gets switched off.

Figure 3. Adding conversational structure closes the precision gap. A BERT-only per-message baseline near 50 to 60% precision rises to ~92% once the Graph Attention Network reasons over the conversation graph, at 81% recall on the PAN12 benchmark.

Predator-class metrics by graph edge-construction strategy
Edge strategy	Precision	Recall	F1	F2
Sequential	0.92	0.82	0.87	0.84
Cross-speaker	0.87	0.77	0.82	0.79
Full connectivity	0.79	0.75	0.77	0.76

Table 1. Predator-class metrics by graph edge-construction strategy on the PAN12 test set. The simple sequential graph outperforms denser alternatives: added edges introduce more noise than signal.

Why trajectory beats keywords

Grooming escalates through phases, each weighted more heavily than the last. Individually benign messages accumulate into a rising risk signal, and the model flags the user as risk crosses the threshold, before explicitly sexual content ever appears. A per-message filter, seeing each turn alone, never accumulates this signal.

Figure 4. Why trajectory beats keywords. Grooming escalates through phases, each weighted more heavily than the last. Individually benign messages accumulate into a rising risk signal; the model flags the user during exclusivity, before explicitly sexual content appears.

Why this matters now

Regulators have caught up with the threat. The UK Online Safety Act now expects proactive protections against grooming, the EU Digital Services Act and renewed child-protection rules raise the bar, COPPA governs US kids' platforms, and Australia's under-16 measures are in force. “We filter profanity” no longer clears it. This is the research edge we are bringing into Paxmod, so studios get detection built with a world-class AI institute without standing up a trust-and-safety research team of their own.

Book a child-safety walkthrough Get an API key

Built through the Industrial AI SME Grant Program, an AIML initiative supported by the South Australian Government via the Department of State Development's Research and Innovation Fund. Benchmark figures are reported on the PAN12 Sexual Predator Identification corpus.