Using LLMs for Prediction Market Signal Generation

LLMs are tempting for trading because they're fluent about everything — and fluency is exactly what makes them dangerous if you wire them up wrong. The right mental model is narrow: an LLM is excellent at turning unstructured information into structured features, and unreliable at producing a calibrated probability. Build your signal pipeline around that distinction and LLMs become a genuine edge. Ignore it — and trust the model's confident "75%" — and you'll trade on noise dressed up as insight.

This is the signal-generation companion to building an AI trading agent with Claude and the overview in AI and automation in prediction markets. For the non-AI fundamentals of what to watch, see prediction market signals: what to watch before you trade.

What a "signal" actually is

A signal is a structured, machine-readable reason to act: a direction, a magnitude, and ideally a confidence, attached to a specific market. "The Fed sounded more hawkish than expected, which lowers the probability of a June cut" is an observation. A signal is that observation converted into {market: KXFED-JUN, direction: NO, strength: high} — something a rules engine can evaluate. The whole game is producing those structured tuples reliably, and that's where an LLM earns its keep.

Where LLMs genuinely add signal

• News synthesis. Reading twenty articles and extracting what actually changed since yesterday is a core LLM strength. This is the highest-value use.
• Event and entity extraction. Pulling the who/what/when out of messy text — "which market does this headline affect, and in which direction?" — is reliable when the task is scoped tightly.
• Tone and sentiment. Judging whether a statement reads as hawkish/dovish, bullish/bearish, escalating/de-escalating, with the nuance keyword matching misses.
• Normalization. Mapping a hundred different phrasings of the same event onto one canonical market.

Every item on that list is a language task. None of them asks the model to predict the future or do arithmetic — and that's precisely why they work.

Where LLMs don't add signal — the calibration trap

The failure mode that costs money is treating an LLM's stated probability as a real one. Ask a model "what's the probability?" and it will give you a confident, fluent number that is not calibrated — when it says 70%, the thing does not happen 70% of the time. Three related limits:

• Probability calibration. Raw LLM confidence is a vibe, not a frequency. Never feed it straight into sizing.
• Arithmetic. Models are confidently wrong on numbers. Fees, edges, and expected value are deterministic-code territory, the same lesson behind why your Kalshi bot's P&L is wrong.
• Recency. A model only knows what's in its prompt. For today's news you must retrieve and supply it — the model won't have it otherwise.

A practical pipeline

The architecture that works keeps the LLM in the middle, doing extraction, with deterministic code on both ends:

  sources (news, filings, data feeds)
        │  retrieve fresh context
        ▼
  [ LLM: extract structured features ]   ← the language task
        │  { affects, direction, tone, magnitude }
        ▼
  [ map features → specific Kalshi market ]
        │
        ▼
  [ rules engine: convert to confidence, gate, size ]  ← the math
        │
        ▼
  signal → (your bot's risk + execution layer)

The LLM step uses tool use so you get structured output instead of prose to regex:

import anthropic

client = anthropic.Anthropic()

EXTRACT = {
    "name": "extract_signal_features",
    "description": "Extract market-relevant features from a news item.",
    "input_schema": {
        "type": "object",
        "properties": {
            "affects_topic": {"type": "string"},
            "direction": {"enum": ["increases", "decreases", "unclear"]},
            "tone": {"enum": ["strong", "moderate", "weak"]},
            "is_new_information": {"type": "boolean"},
        },
        "required": ["affects_topic", "direction", "tone", "is_new_information"],
    },
}

def extract(headline, body):
    msg = client.messages.create(
        model="claude-sonnet-4-6",   # any capable Claude model
        max_tokens=512,
        tools=[EXTRACT],
        tool_choice={"type": "tool", "name": "extract_signal_features"},
        system="Extract only what the text supports. Prefer 'unclear' over guessing.",
        messages=[{"role": "user", "content": f"{headline}\n\n{body}"}],
    )
    return msg.content[0].input

The model returns features, not a trade. Whether those features become a signal — and how strong — is decided by code you control, against rules you can test.

Turning fluent text into a usable probability

Since you can't trust the model's number, derive your own. Two workable approaches:

1. Map features to confidence with your own rules. "New information + strong tone + clear direction" earns higher confidence than "old news, weak tone." You define the mapping and tune it against outcomes.
2. Calibrate empirically. Log every signal and its eventual outcome, then build a reliability curve: when your pipeline said "high confidence," how often was it right? Adjust until your stated confidence matches reality. This is the only way to earn the right to feed a number into sizing.

Either way, the model supplies features; your calibrated logic supplies the probability. That's the line that separates a real signal from a confident hallucination.

A concrete example: a Fed-decision signal

Trace one signal end to end. The Fed releases a statement. Your pipeline:

1. Retrieves the statement text and a handful of reputable summaries — the model can't react to what it isn't given.
2. Extracts features with the LLM: affects = "near-term rate cut," direction = "decreases," tone = "strong," is_new_information = true.
3. Maps those features to the specific Kalshi market on the next rate decision.
4. Scores them with your calibrated rules into a confidence — high here, because it's strong, directional, and genuinely new information.
5. Hands off the signal to your bot's risk and execution layer, which sizes it and places a limit order only if it clears the bar.

At no point did the model output a price or a position size. It translated language into structure; your tested logic did everything that touches money.

Sourcing and freshness: garbage in, garbage out

An LLM signal is only as good as what you feed it. Two things matter most: source quality — reputable primary sources beat aggregators and rumor — and freshness. Because the model only knows what's in the prompt, your retrieval layer is doing the real work of deciding what's current and relevant. This is retrieval-augmented generation in practice: deterministic code fetches the right, recent context, and the model reasons over it. Skimp on retrieval and even a perfect model produces stale or irrelevant signals.

Cost and latency

Don't call a model on every tick. Trigger extraction when new information actually arrives — a fresh headline or a data release — not on a timer, because a thesis changes far less often than a price does. Keep your stable instructions in a cached system prompt so you aren't re-paying for them on every call, and reserve the model for the language step while cheap deterministic code handles the fast path. An always-on pipeline that calls a frontier model indiscriminately gets expensive fast and buys you nothing.

Common LLM-signal mistakes

• Trusting the model's probability. The number is fluent, not calibrated. Derive your own.
• Letting the model do math. Fees, edges, and sizing belong in deterministic code.
• Forgetting retrieval. No fresh context in, no useful signal out.
• Skipping the backtest. An untested signal is a guess with extra steps.
• Acting on low-conviction extractions. "Unclear" is a valid, valuable output — let the pipeline abstain.

Backtest the signal like anything else

An LLM-derived signal is still just a strategy input, and it lies in all the same ways. Test it net of fees, with strict point-in-time discipline so the model never "sees" news that hadn't broken yet, and over enough resolved markets to distinguish edge from luck. The full method is in how to backtest a Kalshi strategy — and lookahead bias is especially easy to introduce with news data, so guard against it deliberately.

From signal to trade

A validated, calibrated signal still has to pass through the same guardrails as any other: position sizing from half-Kelly or less, hard risk limits, and limit-order execution to keep fees from eating the edge. The LLM made the language problem tractable; the boring, deterministic layer is still what keeps you solvent.

The bottom line on LLM signals

Used narrowly, an LLM can make the language-shaped part of the problem — reading text and turning it into structured features — easier to operate. That is a workflow advantage, not evidence of trading edge. Let the model translate language into reviewable features, let deterministic and tested code handle math and limits, and treat every signal as a hypothesis. The system still needs point-in-time tests, monitoring, and a stop control.