To resolve empty tool_calls and JSON parsing errors immediately, launch your local inference server with an explicit tool parser and guided decoding:

That stalls the pipeline. Execution halts immediately.

# Launch vLLM with automated tool choice, chat parser, and Outlines grammar masking
vllm serve Qwen/Qwen2.5-Coder-32B-Instruct-AWQ \
  --port 8000 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes \
  --guided-decoding-backend outlines

Open-weight models (such as Qwen 2.5, Llama 3.3, and DeepSeek R1) emit function calls wrapped in custom tokens (like <tool_call> or Hermes XML) rather than OpenAI’s raw wire schema. Without --tool-call-parser, the inference server leaves these raw tags inside message.content and sets tool_calls: None, crashing Pydantic AI, LangGraph, and CrewAI pipelines.

Root Cause Analysis

When client applications query an OpenAI-compatible local server (vLLM, Ollama, SGLang, or LiteLLM) with a tools parameter, they expect a structured ChatCompletionMessageToolCall object.

Instead, local models frequently trigger these runtime exceptions:

pydantic_core._pydantic_core.ValidationError: 1 validation error for ToolCall
arguments
  JSON decode error: Expecting property name enclosed in double quotes [type=json_invalid, input_value='{\n  \'repo\': \'agentic-pulse\',\n}', input_type=str]

Or downstream agent code crashes with:

AttributeError: 'NoneType' object has no attribute 'tool_calls'
json.decoder.JSONDecodeError: Unterminated string starting at line 1 column 42

This failure happens across three distinct layers:

1. Unparsed Chat Template Tags: Foundation models are trained on specific tool formatting templates. Qwen and Hermes models emit <tool_call>{"name": "...", "arguments": {...}}</tool_call>. If the inference server does not parse these tags into OpenAI schema objects, the client receives raw text in message.content while message.tool_calls remains empty.
2. Grammar Violations in Output Tokens: Free-form LLM generation allows illegal JSON tokens, including single-quoted keys ({'query': ...}), unescaped newlines in string arguments, and trailing commas ({"key": 1,}). Python’s strict standard json.loads() rejects all three.
3. Reasoning Token Bleed: Reasoning models output <think>...</think> traces before emitting tool calls. If your client parser uses a naive regex expecting { at index 0, the reasoning block breaks the extractor.

Failure Symptom	Underlying Mechanism	Root Environment	Resolution
tool_calls: None	Model generated <tool_call> XML into raw content	vLLM / SGLang without parser flag	Set --tool-call-parser hermes
JSONDecodeError	Single quotes or trailing commas in argument string	Unconstrained model decoding	Enable --guided-decoding-backend outlines
Empty Args Object {}	Model emitted thoughts inside arguments block	DeepSeek R1 / Reasoning models	Use regex tag stripper before parsing

Step-by-Step Resolution

Step 1: Configure the Correct Server Tool Parser

Inference engines require explicit parser flags matching the model’s base tokenizer and prompt formatting.

For vLLM, select the parser matching your model family:

# For Qwen 2.5 and Hermes-trained models
vllm serve Qwen/Qwen2.5-Coder-32B-Instruct --enable-auto-tool-choice --tool-call-parser hermes

# For Meta Llama 3.1 / 3.2 / 3.3 Instruct
vllm serve meta-llama/Llama-3.3-70B-Instruct --enable-auto-tool-choice --tool-call-parser llama3_json

# For Mistral / Mixtral models
vllm serve mistralai/Mistral-Small-24B-Instruct-2501 --enable-auto-tool-choice --tool-call-parser mistral

For Ollama, verify that your model supports native tool calling by checking its manifest:

ollama show qwen2.5-coder:32b --modelfile | grep -i "tools"

If the Modelfile lacks the {{ if .Tools }} template block, update Ollama to the latest release (ollama 0.5.x+) and pull the official updated tag.

Step 2: Enforce Token-Level JSON Grammar with Outlines

Instead of hoping the model generates valid JSON syntax, force the sampling engine to reject illegal tokens during generation.

vLLM supports Outlines and XGrammar backends to constrain token logits:

# Install outlines dependencies on Ubuntu 24.04
pip install outlines xgrammar

# Start vLLM with grammar constraints
vllm serve Qwen/Qwen2.5-Coder-14B-Instruct \
  --port 8000 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes \
  --guided-decoding-backend outlines

When client requests send a tools parameter containing a JSON schema, Outlines constructs a finite state machine (FSM) over the vocabulary, guaranteeing that every generated token conforms to the schema.

Step 3: Implement Client-Side Fallback Extraction in Python

Network proxies or edge setups may occasionally strip headers. Add a tolerant fallback extractor to your agent’s client loop to catch unparsed tags and malformed quotes:

import re
import json
import ast

def extract_tool_calls_robust(response_message):
    """
    Extracts tool calls from either native OpenAI objects or 
    raw markdown/XML content generated by open-weight models.
    """
    # 1. Check if the server already parsed the tool calls cleanly
    if getattr(response_message, "tool_calls", None):
        valid_calls = []
        for call in response_message.tool_calls:
            args = call.function.arguments
            if isinstance(args, str):
                try:
                    parsed_args = json.loads(args)
                except json.JSONDecodeError:
                    # Tolerant fallback: handle single quotes and trailing commas
                    parsed_args = ast.literal_eval(args)
            else:
                parsed_args = args
            valid_calls.append({"name": call.function.name, "arguments": parsed_args})
        return valid_calls

    # 2. Server failed to parse: inspect raw text content
    content = response_message.content or ""
    
    # Strip reasoning <think> blocks if present
    content = re.sub(r"<think>.*?</think>", "", content, flags=re.DOTALL).strip()
    
    # Look for XML <tool_call> tags (Hermes / Qwen format)
    xml_matches = re.findall(r"<tool_call>(.*?)</tool_call>", content, flags=re.DOTALL)
    if xml_matches:
        recovered = []
        for raw in xml_matches:
            raw = raw.strip()
            try:
                recovered.append(json.loads(raw))
            except json.JSONDecodeError:
                recovered.append(ast.literal_eval(raw))
        return recovered

    # Look for raw markdown ```json blocks
    json_blocks = re.findall(r"```json\s*(\{.*?\})\s*```", content, flags=re.DOTALL)
    if json_blocks:
        return [json.loads(b) for b in json_blocks]

    return []

Step 4: Handle Pydantic v2 Schema Coercion

When using Pydantic AI or custom Pydantic schemas, convert stringified arguments into typed models safely:

from pydantic import BaseModel, ValidationError

class SearchDatabaseArgs(BaseModel):
    query: str
    limit: int = 10
    exact_match: bool = False

def dispatch_tool_call(raw_call: dict):
    tool_name = raw_call.get("name")
    raw_args = raw_call.get("arguments", {})

    if tool_name == "search_database":
        try:
            # Pydantic v2 type coercion handles str-to-int conversion automatically
            validated_args = SearchDatabaseArgs.model_validate(raw_args)
            return execute_database_search(validated_args)
        except ValidationError as e:
            # Return validation feedback directly into agent message history
            return {"error": "Invalid arguments", "details": e.errors()}
            
    return {"error": f"Unknown tool: {tool_name}"}

Verification: End-to-End Function Calling Test

Verify your local vLLM endpoint using an automated Python script:

# test_tool_calling.py
from openai import OpenAI
import json

client = OpenAI(base_url="http://localhost:8000/v1", api_key="local-token")

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_server_load",
            "description": "Get current CPU and memory load for a cluster node.",
            "parameters": {
                "type": "object",
                "properties": {
                    "node_id": {"type": "string"},
                    "include_swap": {"type": "boolean"}
                },
                "required": ["node_id"]
            }
        }
    }
]

response = client.chat.completions.create(
    model="Qwen/Qwen2.5-Coder-32B-Instruct-AWQ",
    messages=[{"role": "user", "content": "Check the CPU load on node-production-04."}],
    tools=tools,
    tool_choice="auto"
)

msg = response.choices[0].message
assert msg.tool_calls is not None, "Error: tool_calls is None! Check --tool-call-parser flag."

call = msg.tool_calls[0]
args = json.loads(call.function.arguments)

print(f"PASS: Function Name -> {call.function.name}")
print(f"PASS: Node ID       -> {args['node_id']}")

Run the validation on your terminal:

python3 test_tool_calling.py

Expected terminal output:

PASS: Function Name -> get_server_load
PASS: Node ID       -> node-production-04

Common Pitfalls & Edge Cases

• Adjacent Unbracketed Tool Calls: Under multi-tool scenarios, some models output multiple JSON objects without an enclosing array ({"name": "t1"}{"name": "t2"}). Standard json.loads fails on the second object. Split on }{ before decoding.
• LiteLLM Parameter Stripping: If using LiteLLM proxy in front of vLLM, ensure drop_params: false is configured in config.yaml. Otherwise, LiteLLM may strip tool_choice or custom chat templates.
• System Prompt Conflicts: Do not manually instruct the model “Output your tool call as JSON” in the system prompt if you are using --tool-call-parser hermes. The model’s built-in chat template already injects the correct system tokens; adding conflicting instructions causes the model to wrap JSON inside markdown blocks twice.

Official Documentation & Primary References

• vLLM Official Documentation: Engine configuration arguments, PagedAttention KV cache sizing, and distributed serving guides.
• vLLM GitHub Repository: Upstream source code, AsyncLLMEngine architecture, and issue tracker.
• SGLang GitHub Repository: RadixAttention serving runtime and FlashInfer kernel integration specs.
• Hugging Face Hub Python Library Documentation: Environment variables, download timeout settings, and cache management on Linux.

More in \"LLM Serving"

How to Fix 'ValueError: The model's max sequence length is larger than the maximum number of tokens in KV cache' in vLLM on Linux

Fix vLLM ValueError: The model's max seq len is larger than the maximum number of tokens in KV cache on Linux. Tune max_model_len, FP8 KV cache, and VRAM sizing.

How to Fix Hugging Face Model Download Timeouts and IncompleteRead on Linux

Resolve Hugging Face Hub download timeouts, IncompleteRead, and Connection reset errors on Linux. Enable hf_transfer, configure resume download, and tune worker threads.

How to Fix 'FlashInfer kernel compilation failed' and Import Errors in SGLang on Linux

Resolve FlashInfer JIT compilation failed, nvcc fatal unsupported gpu architecture, and CUDA mismatch errors in SGLang on Ubuntu Linux.