Why I built an AT Protocol fairness bot

Disagreement is not the problem

Good-faith disagreement is valuable. It can expose an unstated assumption, bring evidence someone missed, or make the opposing case more precise. A reply does not have to be gentle or agreeable to be worth having. It does need to engage with the point it is answering.

Too many public replies take a shortcut instead. They attack the person, replace the argument with a weaker one, use mockery in place of reasoning, or assert a conclusion without showing how they reached it. The result looks like a conversation while making one harder to have.

That is the small, specific problem I built atproto-fairness-bot to address. It watches direct replies to a configured Bluesky account, but it does not judge every reply. First it asks whether a reply is actually pushing back on, criticising, or disagreeing with the point it answers. Agreement, compliments, jokes, questions, and unrelated banter stop there.

Only a debate response receives a fairness score. If that score falls below a configured threshold, the bot can post a brief, polite response identifying the problem. A reply that makes a real counterargument, even a blunt one, gets no response.

The practical reason for building it is time. Replies that offer little or no value can pull attention away from conversations worth having. Rather than spending that time personally answering the same kinds of bad-faith response, the bot can make a short, consistent callout and leave substantive discussion for a person.

The distinction matters: this is not a truth bot, a moderation system, or an arbiter of which side is right. It is trying to ask two narrower questions: is this actually a disagreement, and if it is, does it engage fairly with the substance of what it answers?

Debate first, then a score

The first question is simple: is the reply a debate response? A friendly reply should not become a fairness test just because it is informal, sarcastic between friends, or not especially rigorous. The bot leaves ordinary conversation alone and records no score for it.

A single blended score has an obvious failure mode: it collapses several different kinds of problem into one number, so a reply that is 90 percent substance with one nasty aside can end up looking identical to a reply that is bad in a completely different way. That is a real objection, not a hypothetical one, so the bot scores three separate axes instead of one:

• rhetoric — tone. Insults, personal attacks, sarcasm, or mockery in place of argument score low here, even if the rest of the reply is sound.
• relevance — whether the reply engages the actual point, or misrepresents it, strawmans it, or answers a different point altogether.
• evidence — whether claims are backed by reasoning or evidence, versus asserted with neither where either would clearly be expected.

Each axis runs 0 to 100 and is judged independently. The gate score used against the publication threshold is the weakest of the three, not their average. A reply that nails relevance and evidence but opens with an insult still gates on the low rhetoric score — averaging it against two strong scores would let the insult through unflagged, which defeats the point of scoring tone at all.

The default publication threshold is 60. A debate reply whose weakest axis falls below that score is a candidate for intervention. That is a setting, not a claim that 60 is a universal moral boundary. It makes the policy visible, adjustable, and testable.

Strong language by itself is not unfairness. Someone can say a claim is wrong, unsupported, harmful, or incomplete; the important part is explaining why. A low-scoring reply is one that instead relies on one of these failure modes — each maps directly to one of the three axes above:

• insults or personal attacks in place of an argument (rhetoric);
• sarcasm or mockery in place of substance (rhetoric);
• a misrepresentation or strawman of the original point (relevance);
• an answer to a different point altogether (relevance); or
• unsupported claims where reasoning or evidence would be expected (evidence).

Even then, the intended response is not a reprimand. It should name the specific issue and point back to the kind of reply that would move the conversation forward. The bot should model the fairness it asks for: no insults, no sarcasm, no superiority.

Why AT Protocol makes this possible

AT Protocol makes a constrained project like this practical. Posts, replies, and identities are interoperable records, rather than private implementation details of one social product. Bluesky’s Jetstream provides a filtered JSON firehose, so the bot can receive new post records as they are created instead of repeatedly polling a timeline.

That does not mean the bot scans everything. Its scope is intentionally much smaller: by default it only considers newly created direct replies whose immediate parent was authored by the target account. A nested reply below somebody else’s comment is ignored, even when it belongs to the same broader thread.

That boundary is both technical and social. It keeps the bot from treating a whole thread as its territory and avoids guessing who a participant is actually responding to. The one deliberate exception — judging the target account’s own outgoing replies, off by default — is covered later in Holding the same standard both ways, because it changes what the bot does on an unfair verdict, not just what it watches.

The event-to-reply pipeline

The implementation is a small Swift command-line process. Its continuous watcher follows a straightforward path:

1. Subscribe to app.bsky.feed.post creation events through Jetstream.
2. Filter to direct, non-self replies to the configured account.
3. Fetch the root post and immediate parent post from Bluesky’s public AppView.
4. Give that context and the new reply to an OpenAI-compatible model.
5. Stop when the reply is general conversation; otherwise assign three sub-scores from 0 to 100 — rhetoric, relevance, evidence — gated by whichever is lowest.
6. Do nothing when the gate score meets the configured threshold.
7. For a gate score below the threshold, optionally ask a second model to review the proposed response — it can approve it, reject the text, or overturn the finding entirely.
8. Publish only an approved, bounded reply, then publish its associated verdict record.

The model receives three pieces of context: the root of the thread, the specific message from the target account being answered, and the new reply. This helps it distinguish genuine disagreement from a reply that is simply reacting to an earlier part of the conversation.

Its response is constrained JSON: an isDebate decision, a score when the reply is debate, short reasoning, and a proposed reply only when a score falls below the configured threshold. Proposed replies are capped at 300 characters and may not start with a name, handle, label, or signature. That keeps the public intervention short and avoids making a response look like a performative verdict.

A second opinion before a public reply

A language model is not a reliable moral or factual authority. Context can be missing, sarcasm can be misunderstood, and the model can confidently make a bad call. For a system that can post publicly, the default should be restraint.

The bot therefore supports an optional second-model gate. The reviewer sees the original context, the first model’s scores and reasoning, and its candidate reply, and makes two independent judgments rather than one:

• Does it agree the original reply was actually unfair? If it disagrees with the first model’s finding — it thinks the reply engaged fairly after all — it overturns the verdict outright, regardless of how good the drafted reply text is.
• If it agrees, is this specific candidate reply fit to post? It can approve the draft with only minimal edits for clarity, proportionality, factual caution, tone, and the character limit, or reject the text itself while still agreeing the underlying finding holds.

Those two outcomes — overturning the finding versus rejecting the phrasing — are logged distinctly, even though both currently result in nothing being posted. Collapsing them into one “approved” flag would have hidden a real asymmetry: a system that can only ever police tone isn’t the same as one that can also say the first model was simply wrong.

Two model calls do not produce certainty. They do support a useful bias: a false positive, where the bot replies when it should have stayed silent, does more harm than letting a questionable reply pass. The review step is there to make publication harder, not more frequent.

A verdict that becomes part of the network

The AT Protocol part of this project is more than its delivery mechanism. When the bot posts a callout, it also writes a dev.maclong.feed.verdict record to its own repository. The record contains strong references to both the judged reply and the bot’s public response, along with the gate score, the three sub-scores behind it, the model’s reasoning, the exact reply text, and a timestamp. An illustrative record looks like this:

{
  "$type": "dev.maclong.feed.verdict",
  "subject": {
    "uri": "at://did:plc:author/app.bsky.feed.post/3kexample",
    "cid": "bafy…subject"
  },
  "reply": {
    "uri": "at://did:web:fairness.example/app.bsky.feed.post/3kreply",
    "cid": "bafy…reply"
  },
  "score": 28,
  "rhetoric": 28,
  "relevance": 55,
  "evidence": 40,
  "reasoning": "The reply offers no supporting reasoning.",
  "replyText": "Please address the evidence behind the claim.",
  "selfAssessment": false,
  "createdAt": "2026-08-05T12:00:00Z"
}

That makes the bot’s interventions inspectable as structured network data, separate from the conversation thread. reply and replyText are optional: they’re present for an external callout like the one above, and absent for a self-assessment (see Holding the same standard both ways), which the selfAssessment flag distinguishes. It publishes a verdict record only when it has actually posted or self-assessed as unfair, not for fair scores, general conversation, or dry-run checks. If the verdict-record write fails, it is logged, but the already-posted reply is not retried. The record documents an action; it is not another chance to amplify it.

Holding the same standard both ways

There’s an asymmetry worth naming directly, because a reader will notice it before I finish explaining anything else: by default this bot can only ever defend the account it watches. It judges replies made to the target, never replies made by the target. Structurally, that means it can call out someone else’s strawman but has nothing to say if the target does the exact same thing to a critic. “Fairness bot” implies a standard applied evenhandedly; what ships by default is closer to an argument-quality filter for one account’s mentions.

I don’t think that’s disqualifying, but it deserved a real answer instead of a hopeful name. So the bot supports an optional SELF_REVIEW_ENABLED mode, off by default, that closes the gap: when enabled, it also judges the target account’s own outgoing replies to other people, through the identical three-axis scoring and optional reviewer gate described above. Nothing about the judgment changes based on who wrote the reply.

What does change is the response to an unfair verdict. When the target is the one who engaged unfairly, the bot does not post a public counter-reply — having it publicly “reply” to its own operator would be a strange, dishonest performance, and it isn’t the bot’s place to stage that conversation. Instead it publishes a dev.maclong.feed.verdict record with selfAssessment: true and no reply or replyText: a public, timestamped acknowledgment that the same standard was applied and the target’s own reply didn’t clear it. The record is the accountability mechanism, not a manufactured exchange.

It’s off by default because turning it on is a real decision about how much scrutiny to invite, not a default anyone should inherit silently. I’ve turned it on for my own account, because a standard I’m not willing to have applied to myself isn’t much of a standard.

Safeguards outside the prompt

Good prompts are not enough. A public bot needs operational safeguards too.

The project includes a one-shot check command for inspecting an existing reply. It fetches the same context and prints FAIR or UNFAIR with a score out of 100. For general conversation, it prints N/A instead because there is no fairness score. It also shows the reasoning and suggested response. The command is dry-run by default; publishing requires an explicit --post flag.

The continuous watcher persists both its Jetstream cursor and the AT URI of every post it has already answered. If the process restarts or reconnects, it can resume from the saved cursor without duplicating an earlier public reply. It also retries lost WebSocket connections with exponential backoff.

Keeping the watcher running with Pitchfork

A reply watcher is only useful while it is running, so the bot uses Pitchfork for process supervision. Its daemon configuration runs the release binary from the repository through Mise. It retries after an exit and is marked for startup at login.

The first run is still done in the foreground. Once the account, model, and responses have been checked, pitchfork boot enable installs Pitchfork’s login-time integration. mise run start then builds the release binary and starts the fairness-bot daemon. mise run logs tails its output, and mise run stop stops it. pitchfork boot status and pitchfork list are useful checks when confirming that the login service and daemon are registered.

Pitchfork is not a replacement for the bot’s own recovery logic. It restarts the process; the saved Jetstream cursor and reply log are what let a restarted process resume without silently skipping retained events or posting a duplicate reply.

The bot account is deliberately separate from the account it watches. At startup it configures its own profile and applies Bluesky’s native bot self-label. Automation that speaks in public should identify itself as automation; a bot should not borrow the voice or identity of the person it is nominally helping.

What success would look like

The best outcome is not that the bot replies a lot. It is that it rarely needs to.

If it does reply, a successful intervention is one that makes the missing step legible: address the claim, explain the reasoning, provide evidence, or state the real disagreement without reducing the other person to a caricature.

The project is an experiment, not a claim that automated fairness enforcement is solved or should be applied broadly. Its scope is one account, direct replies, a configurable scoring rule, public verdict records, and visible automation. It needs monitoring, revision, and a willingness to turn it off if it makes conversations worse.

But the underlying standard seems worth defending: disagreement deserves an argument. Software can help remind us of that only when it is narrow, transparent, and at least as accountable as the people it addresses.