Using LLMs for Prediction Market Signal Generation

LLMs are tempting for trading because they're fluent about everything — and fluency is exactly what makes them dangerous if you wire them up wrong. The right mental model is narrow: an LLM is excellent at turning unstructured information into structured features, and unreliable at producing a calibrated probability. Build your signal pipeline around that distinction and LLMs become a genuine edge. Ignore it — and trust the model's confident "75%" — and you'll trade on noise dressed up as insight.

This is the signal-generation companion to building an AI trading agent with Claude and the overview in AI and automation in prediction markets. For the non-AI fundamentals of what to watch, see prediction market signals: what to watch before you trade.

What a "signal" actually is

A signal is a structured, machine-readable reason to act: a direction, a magnitude, and ideally a confidence, attached to a specific market. "The Fed sounded more hawkish than expected, which lowers the probability of a June cut" is an observation. A signal is that observation converted into {market: KXFED-JUN, direction: NO, strength: high} — something a rules engine can evaluate. The whole game is producing those structured tuples reliably, and that's where an LLM earns its keep.

Where LLMs genuinely add signal

• News synthesis. Reading twenty articles and extracting what actually changed since yesterday is a core LLM strength. This is the highest-value use.
• Event and entity extraction. Pulling the who/what/when out of messy text — "which market does this headline affect, and in which direction?" — is reliable when the task is scoped tightly.
• Tone and sentiment. Judging whether a statement reads as hawkish/dovish, bullish/bearish, escalating/de-escalating, with the nuance keyword matching misses.
• Normalization. Mapping a hundred different phrasings of the same event onto one canonical market.

Every item on that list is a language task. None of them asks the model to predict the future or do arithmetic — and that's precisely why they work.

Where LLMs don't add signal — the calibration trap

The failure mode that costs money is treating an LLM's stated probability as a real one. Ask a model "what's the probability?" and it will give you a confident, fluent number that is not calibrated — when it says 70%, the thing does not happen 70% of the time. Three related limits:

• Probability calibration. Raw LLM confidence is a vibe, not a frequency. Never feed it straight into sizing.
• Arithmetic. Models are confidently wrong on numbers. Fees, edges, and expected value are deterministic-code territory, the same lesson behind why your Kalshi bot's P&L is wrong.
• Recency. A model only knows what's in its prompt. For today's news you must retrieve and supply it — the model won't have it otherwise.

A practical pipeline

The architecture that works keeps the LLM in the middle, doing extraction, with deterministic code on both ends:

  sources (news, filings, data feeds)
        │  retrieve fresh context
        ▼
  [ LLM: extract structured features ]   ← the language task
        │  { affects, direction, tone, magnitude }
        ▼
  [ map features → specific Kalshi market ]
        │
        ▼
  [ rules engine: convert to confidence, gate, size ]  ← the math
        │
        ▼
  signal → (your bot's risk + execution layer)

The LLM step uses tool use so you get structured output instead of prose to regex:

import anthropic

client = anthropic.Anthropic()

EXTRACT = {
    "name": "extract_signal_features",
    "description": "Extract market-relevant features from a news item.",
    "input_schema": {
        "type": "object",
        "properties": {
            "affects_topic": {"type": "string"},
            "direction": {"enum": ["increases", "decreases", "unclear"]},
            "tone": {"enum": ["strong", "moderate", "weak"]},
            "is_new_information": {"type": "boolean"},
        },
        "required": ["affects_topic", "direction", "tone", "is_new_information"],
    },
}

def extract(headline, body):
    msg = client.messages.create(
        model="claude-sonnet-4-6",   # any capable Claude model
        max_tokens=512,
        tools=[EXTRACT],
        tool_choice={"type": "tool", "name": "extract_signal_features"},
        system="Extract only what the text supports. Prefer 'unclear' over guessing.",
        messages=[{"role": "user", "content": f"{headline}\n\n{body}"}],
    )
    return msg.content[0].input

The model returns features, not a trade. Whether those features become a signal — and how strong — is decided by code you control, against rules you can test.

Turning fluent text into a usable probability

Since you can't trust the model's number, derive your own. Two workable approaches:

1. Map features to confidence with your own rules. "New information + strong tone + clear direction" earns higher confidence than "old news, weak tone." You define the mapping and tune it against outcomes.
2. Calibrate empirically. Log every signal and its eventual outcome, then build a reliability curve: when your pipeline said "high confidence," how often was it right? Adjust until your stated confidence matches reality. This is the only way to earn the right to feed a number into sizing.

Either way, the model supplies features; your calibrated logic supplies the probability. That's the line that separates a real signal from a confident hallucination.

A concrete example: a Fed-decision signal

Trace one signal end to end. The Fed releases a statement. Your pipeline:

1. Retrieves the statement text and a handful of reputable summaries — the model can't react to what it isn't given.
2. Extracts features with the LLM: affects = "near-term rate cut," direction = "decreases," tone = "strong," is_new_information = true.
3. Maps those features to the specific Kalshi market on the next rate decision.
4. Scores them with your calibrated rules into a confidence — high here, because it's strong, directional, and genuinely new information.
5. Hands off the signal to your bot's risk and execution layer, which sizes it and places a limit order only if it clears the bar.

At no point did the model output a price or a position size. It translated language into structure; your tested logic did everything that touches money.

Specify the rule before you deploy it, not after the first bad trade

The pipeline above describes how a signal is produced. It does not say what happens when the signal is wrong, late, or malformed — and that gap is where most LLM trading experiments actually break. Before anything goes live, six things need written answers. "If the news is bullish, buy" is not an answer to any of them.

1. Trigger. The exact, machine-detectable event that activates the pipeline. Narrow beats broad: a trigger scoped to FOMC statement releases produces far less noise than one that fires on every governor's speech.
2. Data source. Where the text comes from, in what format, at what latency, and whether secondary commentary is blended in. Mixing social media into the first pass raises confabulation risk; keep it to a later confirmation stage if you want it at all.
3. Target market. The specific ticker or the query that resolves to one. Selecting by category and title substring, then filtering to contracts closing inside a defined window, is reproducible; "the Fed market" is not.
4. Order action. Which side, which order type, and the price band you will accept. A model output of neutral, or a confidence below your threshold, must map to no order — an explicit abstain, not a smaller position.
5. Position limits. Maximum contracts per signal event, exposure ceiling per market family, an account-balance floor below which submission suspends, and a cooldown after any order so a re-fetched feed item cannot stack a second entry.
6. Failure conditions. The states that abort the signal before an order reaches the exchange.

Write the abort list explicitly

Failure conditions deserve their own treatment because they are the part people skip. Each one should log the signal as aborted with the reason recorded, so the log tells you later which conditions are actually firing:

• Model timeout. No response within your budget, abort. A stale signal is worse than a missed one — the market has already moved.
• Schema parse failure. If the response doesn't parse into the expected structure, abort. Never infer a trade from a malformed response.
• Low confidence. Below your threshold, abstain. "Unclear" is a valid and valuable output.
• Market not open. Re-check status at order time, not at signal time.
• Spread too wide. Past a configured maximum, abort — wide spreads mean thin liquidity and poor fills, and the round-trip spread cost will usually dwarf the edge the signal claimed to find.
• Conflicting stance inside a cooldown. If the model returns the opposite of an open position on the same market, abort and flag for review rather than auto-reversing.
• Duplicate trigger. Deduplicate on the feed item's GUID. A re-fetched RSS item is the most mundane way to double-enter a position, and it happens constantly.

The last one generalizes: idempotency is not optional in anything that reads a feed and places orders. Assume every fetch will return items you have already seen.

What our builder actually does here — and what it does not

Worth being exact, because this is a place where automation tools tend to oversell. Bot for Kalshi does not run a language model over news. There is no LLM in the trigger path, no headline summarization, and no model-generated stance feeding an order.

What the engine does have is a keyword-in-feed trigger: you supply an RSS or Atom feed URL and a keyword, and the condition evaluates true when that keyword appears in a recent item's title or description. That is a literal substring match, not comprehension. It will catch "Fed raises rates" and it will miss "the Committee voted to maintain the target range, with one dissent favoring a 25-basis-point increase" — which is precisely the nuance gap that motivates using a model in the first place.

So the honest division of labor is this. If you want LLM-derived signals, you build the extraction pipeline yourself against the public Kalshi API and your own model provider, using the structure described above. Our builder covers the execution side: turning a supported condition into a rule, placing single-market limit orders, and enforcing stop-loss and take-profit levels, position limits, and an account-level daily loss cap that you set before arming anything. It does not validate your signal, supply a pick, or establish that an edge exists. Those caps constrain a rule; they do not prevent losses.

If your workflow is closer to "watch this feed for this phrase, then place a limit order under caps" than to "have a model read and judge the text," the keyword trigger may cover it directly — with the substring limitation understood up front.

Sourcing and freshness: garbage in, garbage out

An LLM signal is only as good as what you feed it. Two things matter most: source quality — reputable primary sources beat aggregators and rumor — and freshness. Because the model only knows what's in the prompt, your retrieval layer is doing the real work of deciding what's current and relevant. This is retrieval-augmented generation in practice: deterministic code fetches the right, recent context, and the model reasons over it. Skimp on retrieval and even a perfect model produces stale or irrelevant signals.

Cost and latency

Don't call a model on every tick. Trigger extraction when new information actually arrives — a fresh headline or a data release — not on a timer, because a thesis changes far less often than a price does. Keep your stable instructions in a cached system prompt so you aren't re-paying for them on every call, and reserve the model for the language step while cheap deterministic code handles the fast path. An always-on pipeline that calls a frontier model indiscriminately gets expensive fast and buys you nothing.

Common LLM-signal mistakes

• Trusting the model's probability. The number is fluent, not calibrated. Derive your own.
• Letting the model do math. Fees, edges, and sizing belong in deterministic code.
• Forgetting retrieval. No fresh context in, no useful signal out.
• Skipping the backtest. An untested signal is a guess with extra steps.
• Acting on low-conviction extractions. "Unclear" is a valid, valuable output — let the pipeline abstain.

Backtest the signal like anything else

An LLM-derived signal is still just a strategy input, and it lies in all the same ways. Test it net of fees, with strict point-in-time discipline so the model never "sees" news that hadn't broken yet, and over enough resolved markets to distinguish edge from luck. The full method is in how to backtest a Kalshi strategy — and lookahead bias is especially easy to introduce with news data, so guard against it deliberately. Net-of-fees is the part that decides thin-edge strategies: we closed our own 15-minute Bitcoin cell in August 2026 after 6,298 settled windows showed a real, positive gross edge of about +0.67¢ per contract against an average taker fee of about 1.55¢: the taker fee is larger than the measured edge, so only resting limit orders (no maker fee on that series) are even structurally viable. A signal can be correct and still lose money once costs are in. See our Kalshi fees guide for the formula.

From signal to trade

A validated, calibrated signal still has to pass through the same guardrails as any other: position sizing from half-Kelly or less, hard risk limits, and limit-order execution to keep fees from eating the edge. The LLM made the language problem tractable; the boring, deterministic layer is still what keeps you solvent.

The bottom line on LLM signals

Used narrowly, an LLM can make the language-shaped part of the problem — reading text and turning it into structured features — easier to operate. That is a workflow advantage, not evidence of trading edge. Let the model translate language into reviewable features, let deterministic and tested code handle math and limits, and treat every signal as a hypothesis. The system still needs point-in-time tests, monitoring, and a stop control.