Skip to main content

​
Investigation tool calling

Contributor guide for the investigation ReAct loop: tool schemas, LLM invoke payloads, and conversation messages. Applies to every provider the agent uses (Anthropic, OpenAI-compatible, CLI-backed, Bedrock, and future clients)—not one vendor.

​
Architecture

The investigation agent does not call integration APIs through the LLM. The flow is:
  1. Tools — get_registered_tools("investigation"), filtered with tool.is_available(...).
  2. Schemas — llm.tool_schemas(tools) from get_llm(LLMRole.AGENT) (built in core/llm/client_builders.py; client classes in core/llm/transports/sdk/agent_clients.py). Each client class shapes schemas for its API (function definitions, tool specs, CLI prompt JSON, etc.).
  3. Invoke — llm.invoke(messages, system=..., tools=tool_schemas); the model returns tool calls.
  4. Execute — Tools run locally; results are appended as user/assistant turns the same client can read on the next invoke.
  5. Seed path — Before the loop, _build_seed_calls may inject deterministic tool runs; synthetic assistant + tool-result messages must match the active client (tools/investigation/stages/gather_evidence/agent.py).
investigate/agent.py  →  get_llm(LLMRole.AGENT)  →  *AgentClient.tool_schemas / invoke
                    ↓
              tools/*  (input_schema, extract_params, run)

​
Where code lives

ConcernLocation
Provider routingcore/llm/factory.py (get_llm, resolve_llm_route) and core/llm/client_builders.py
Native SDK clientscore/llm/transports/sdk/agent_clients.py, core/llm/transports/sdk/llm_clients.py
LiteLLM transportcore/llm/transports/litellm/clients.py, core/llm/transports/litellm/routing.py (when OPENSRE_LLM_TRANSPORT=litellm)
Chat / non-agent LLMcore/llm/transports/sdk/llm_clients.py (separate client classes; routing shared via factory.py)
Investigation loop & message dispatchtools/investigation/stages/gather_evidence/ and core/
Provider-specific schema/message helpersNext to the client implementing tool_schemas() (strict normalizers live beside that client)
Tool definitionstools/ (input_schema, public_input_schema)
When adding a provider, implement both tool_schemas() and the message shapes the runtime loop already branches on (or extend those branches). Do not assume one vendor’s JSON tool format works elsewhere.

​
Why bugs are easy to miss

  • JSON Schema draft-07 vs API strictness — Tool authors often use patterns that validate in draft-07 ("type": ["object", "null"], anyOf, nullable, implicit objects, bare items: {}). A given LLM API may require a single string type, explicit items, and a closed set of keys. Unit tests that only check “has properties” miss union type arrays.
  • Many tools in one request — Investigation sends a relevance-selected set of tool schemas in a single invoke (select_investigation_tools in tools/investigation/stages/gather_evidence/tools.py: the planner’s planned_actions when present, otherwise alert-relevant sources first, capped at MAX_AGENT_TOOL_SCHEMAS). It is still many schemas at once, so one invalid schema can fail the whole call (HTTP 400, “invalid tools”, etc.) even when the alert never uses that tool. Tool descriptions and parameters live only in these schemas — the alert-context user message no longer re-lists them.
  • Separate client classes — The non-agent reasoning clients (transports/sdk/llm_clients.py) and the tool-calling agent clients (transports/sdk/agent_clients.py) are distinct; a schema or normalizer fix in one does not apply to the other. Provider-specific normalizers must run in tool_schemas() (or shared helpers the client calls).
  • Contract tests can lag APIs — Registry-wide schema tests must encode the strictest rules your shipped adapters enforce. Extend assertions when production shows a new rejection reason.

​
Tool input_schema (authoring)

When adding or changing tools under tools/:
  • Top-level — Investigation tools use type: object with a properties dict.
  • Single type — Prefer one string per node ("string", "object", "array"). Avoid "type": ["object", "null"]; use optional fields via anyOf/oneOf, omit from required, or document that a provider adapter will normalize (and add adapter + test in the same PR).
  • Arrays — Always set items with an explicit type or properties (never empty {}).
  • Composites — $ref, $defs, allOf, anyOf, oneOf, nullable may need a normalizer in the client adapter; do not add them to public schemas without updating that adapter and tests.
  • Stability — Tool call id values must stay consistent between the assistant turn that requests tools and the following tool-result turn for that provider’s format.
Run tool unit tests under tests/tools/. After schema changes, run the registry strict adapter contract (uses the strictest normalizer currently wired in the repo): 
uv run python -m pytest tests/core/runtime/llm/test_investigation_tool_schemas.py -q
Shared assertions live in tests/core/runtime/llm/investigation_tool_schema_contract.py. When you add a stricter provider adapter, point test_investigation_tool_schemas.py at its normalizer and extend the contract module if the API rejects new patterns. Bedrock-specific unit tests stay in tests/core/runtime/llm/test_bedrock_converse.py (no duplicate registry test there).

​
Provider adapters (transports/sdk/agent_clients.py)

Each *AgentClient should own:
ResponsibilityNotes
tool_schemas(tools)Map RegisteredTool / public_input_schema → API payload. Never pass raw schemas if the API is strict.
invoke(..., tools=...)Attach schemas the API expects; handle retries and map errors to RuntimeError with actionable text.
Message compatibilityInvestigation builds history via MessageFormatter (core.messages) — assistant_from_response, tool_results_from_execution, and synthetic_assistant_tool_call — each must match your invoke parser.
Checklist when adding or changing a client:
  • tool_schemas output matches what invoke sends (no duplicate or divergent normalization).
  • New JSON Schema patterns in tools → update the adapter normalizer and contract tests in the same PR.
  • Serialized payload round-trips like the SDK will send it (e.g. json.dumps on the tools list).
  • Validation errors from the API (“missing field type”, “invalid tools”) → treat as schema/adapter bugs first.
  • Throttling / rate limits: align with existing retry policy in sibling clients.
Provider-specific modules (e.g. strict JSON Schema helpers) stay beside the client; keep investigation logic in investigation.py as dispatch only.

​
LiteLLM transport

Route all API providers through LiteLLM with a global transport switch (no change to LLM_PROVIDER): 
export OPENSRE_LLM_TRANSPORT=litellm
When set to litellm, both investigation (get_llm(LLMRole.AGENT)) and non-agent LLM calls (get_llm(LLMRole.REASONING), get_llm(LLMRole.CLASSIFICATION), get_llm(LLMRole.TOOLCALL)) use core/llm/transports/litellm/clients.py via litellm.completion. Leave unset or set to sdk to use native vendor SDK clients under core/llm/transports/sdk/. Supported providers: anthropic, openai, bedrock, and OpenAI-compatible providers (deepseek, groq, openrouter, gemini, nvidia, minimax, ollama), plus azure-openai (always via LiteLLM). Set the matching API key and model env vars from .env.example as usual. User-facing setup: LLM Providers. CLI-backed providers (codex, claude-code, opencode, kimi, copilot, etc.) always use their subprocess path regardless of this setting.

​
Investigation messages (investigation.py)

  • Same ToolCall.id across synthetic seed assistant message, tool results, and evidence keys.
  • Provider-specific IDs — Use opaque ids only when the client requires them (e.g. length/format); keep stable seed_{tool.name} (or equivalent) where history/tests expect predictable ids.
  • Block vs string content — Some APIs require content as structured blocks, not raw strings (including after guardrails). Match what invoke already produced earlier in the thread.
  • zip(tool_calls, results, strict=True) when pairing calls to results.
Extend tests/agent/test_investigation.py when you add a client branch for synthetic/assistant messages.

​
Verification

Minimum before merging schema or client changes: 
uv run python -m pytest tests/core/runtime/llm/test_investigation_tool_schemas.py -q
uv run python -m pytest tests/core/runtime/llm/test_agent_llm_client.py tests/agent/test_investigation.py -q
When touching a specific provider, also verify end-to-end with that provider configured: 
uv run opensre
# /investigate <fixture.json>   # interactive shell
# or: opensre investigate -i <fixture.json>
Use the same LLM_PROVIDER / model users report in issues; unit tests alone are not enough for adapter strictness gaps.