Auto Moderation

Beyond the rule-based filters, AstralChat scores every public chat message with a remote classifier, notifies staff on a hit, and records what it decided into a flagged_messages table so the decisions can later be labelled by hand and used to train an in-house model.

Everything here is best effort and off the main thread: a classifier outage never takes a chat message down with it, and moderation is handed off by MessageStorageService rather than done inline, so message storage is independent of it too.

Pipeline

message sent
   │
   ├─► filters (filters.yml)  ──► hit ──► AutoModerationService.recordFilterHit(…)
   │                                        source = FILTER, action = BLOCKED | FLAGGED
   │
   └─► MessageStorageService.save(…)
          ├─► INSERT INTO chat_messages
          └─► AutoModerationService.submit(…)
                 └─► ModerationService.analyze(content)   [HTTP, cached]
                        └─► thresholds applied locally
                               ├─► flagged → notify (channel / webhook / console)
                               └─► store decision in flagged_messages (source = CLASSIFIER)

The classifier is only asked about messages that were actually stored — a message a filter blocked never reaches it, which is exactly why filter hits are recorded separately.

`moderation.yml`

enabled: true

url: "https://moderation.astralrealms.gg/v1/moderations"
token: "<YOUR_API_TOKEN>"
webhook: "https://astralrealms.gg/api/v1/webhook/moderation"

thresholds:
  flag: 0.4
  alert: 0.5
  use-service: true
  categories:
    threat: 0.4
    identity_attack: 0.4
    sexual_explicit: 0.3

notifications:
  channel: "staff"
  webhook: true
  log-to-console: true

dataset:
  enabled: true
  clean-sample-rate: 0.02
  store-filter-hits: true
  store-private-messages: false
  max-per-player-per-hour: 60

Every nested section is optional — each accessor falls back to a working default, so an older file keeps loading.

Top level

Key	Type	Default	Description
`enabled`	boolean	enabled	Master switch for the classifier. A missing key counts as enabled; the classifier is only actually called when this is not `false` and `url` is set.
`url`	URI	—	The moderation endpoint. Without it the classifier half of the pipeline is off entirely.
`token`	String	—	Sent as `Authorization: Bearer <token>` when non-blank.
`webhook`	String	—	Discord webhook URL flagged messages are posted to.

`thresholds`

Applied locally, on top of whatever the service decided — tightening moderation is a /chat reload, not a classifier redeploy.

Key	Type	Default	Description
`flag`	double	`0.6`	Score at or above which a message counts as flagged. A value `<= 0` falls back to the default.
`alert`	double	`0.8`	Score at or above which staff are pinged in game. Same fallback.
`use-service`	boolean	`true`	Also trust the `flagged` boolean the service returns, even when no local threshold was crossed.
`categories`	`Map<String, Double>`	`{}`	Per-category overrides of `flag`. Detoxify labels: `toxicity`, `severe_toxicity`, `obscene`, `identity_attack`, `insult`, `threat`, `sexual_explicit`.

A message is flagged when at least one category's score is at or above its threshold, or — with use-service: true — when the service flagged it itself.

`notifications`

Key	Type	Default	Description
`channel`	String	`staff`	Chat channel the in-game staff notification is broadcast to. Empty or absent disables it.
`webhook`	boolean	`true`	Also post flagged messages to the Discord `webhook` URL.
`log-to-console`	boolean	`true`	Write flagged messages to the server log.

The console line and the webhook fire for every flagged message; the in-game notification only fires above the stricter alert threshold, and only when the sender is online on this server. It uses message-moderation-notification-format with %player_…%, %reason%, %score% and %message%.

`dataset`

What gets written to flagged_messages.

Key	Type	Default	Description
`enabled`	boolean	`true`	Master switch for persistence.
`clean-sample-rate`	double	`0.02`	Share of non-flagged messages kept as negative training examples. Clamped to `0.0`–`1.0`. A dataset made only of positives trains a useless classifier.
`store-filter-hits`	boolean	`true`	Also record rule-based filter hits — cheap, high-precision labels covering what the model misses.
`store-private-messages`	boolean	`false`	Record private messages too. Off by default: privacy over data volume.
`max-per-player-per-hour`	int	`60`	Cap on rows per player per hour, `0` for unlimited. Stops one spammer from dominating the dataset.

Flagged messages are always kept (subject to the quota); clean ones are sampled.

What is recorded

One row per recorded decision in flagged_messages:

Column	Meaning
`id`	Row id — the one `/chatmod review` takes.
`message_id`	The `chat_messages` row this scored, `NULL` for messages blocked before they were ever stored.
`sender_id`/`sender_name`	Who sent it. The name is `NULL` when the sender was not resolvable.
`channel`	Channel name, or `private-message` for a PM.
`content`	The raw message, truncated to 512 characters.
`source`	`CLASSIFIER`, `FILTER` or `MANUAL`.
`flagged`	Whether it was considered a hit. `false` on sampled clean rows.
`action`	`NONE` (went through, stored as a negative sample), `FLAGGED` (let through, staff notified) or `BLOCKED` (never reached chat).
`reason`	The triggered categories or filter names, comma-separated.
`categories`	JSON array of what tripped.
`scores`	JSON object `{category: score}` — the raw classifier output, i.e. the features.
`top_category`/`max_score`	The strongest signal.
`model`/`request_id`	Which model answered, and its response id.
`label`/`reviewed_by`/`reviewed_at`	The human verdict — see below.

flagged and action are deliberately separate: a stored row is not necessarily a hit.

Review labels

/chatmod review writes one of three ground-truth labels:

Label	Meaning
`TOXIC`	The message really does break the rules.
`CLEAN`	False positive — the message is fine.
`UNSURE`	The reviewer could not decide; excluded from the training set.

The label is the column to train on — the classifier scores are features, not labels.

Exporting the labelled set

SELECT content, label, source, reason, scores, max_score, created_at
FROM flagged_messages
WHERE label IN ('TOXIC', 'CLEAN')
ORDER BY created_at;

Staff commands

See Commands § /chatmod for the full reference — stats, pending, history and review, all behind chat.command.moderation.

Classifier contract

ModerationService.analyze POSTs {"input": ["<message>"]} to url with a five-second connect and request timeout, and expects an OpenAI-moderation-shaped response:

{
  "id": "modr-…",
  "model": "detoxify-original",
  "results": [
    {
      "flagged": true,
      "categories": { "toxicity": true, "insult": true },
      "category_scores": { "toxicity": 0.93, "insult": 0.81 }
    }
  ]
}

Anything other than HTTP 200 is an error and is logged, not retried. Identical content (lower-cased and stripped) is answered from an in-memory cache of the last 256 responses for 60 seconds — chat is full of repeated lines, and re-scoring them costs a round trip for an answer already known.

All HTTP work, including the blocking Discord webhook calls, runs on a dedicated two-thread pool that is shut down with the plugin.

Turning it off

enabled: false (or no url) — no classifier calls. Filters, notifications for filter hits, and chat_messages storage all keep working.
dataset.enabled: false — nothing is written to flagged_messages; staff notifications still fire.
notifications.channel: "" — no in-game pings.

Astral Realms Documentation Help