Qwen3.8-27B Local Agent Setup: vLLM, SGLang & MTP

This guide walks you through setting up Qwen3.8-27B as a local coding agent — from hardware requirements through inference server configuration to agent harness integration. By the end, you'll have a fast, private coding agent running on your own hardware with MTP speculative decoding and properly tuned reasoning effort.

Hardware Requirements

Qwen3.8-27B is a 27B dense model. At 17GB for the GGUF quantized version, it needs meaningful but accessible hardware:

Option 1: vLLM Server

vLLM is the most mature inference framework for Qwen3.8-27B on Linux GPU systems. It provides an OpenAI-compatible API server, PagedAttention for memory efficiency, and native MTP support.

Installation

pip install vllm>=0.8.0

Launch the Server

vllm serve Qwen/Qwen3.8-27B \
  --host 0.0.0.0 \
  --port 8000 \
  --max-model-len 262144 \
  --enable-mtp \
  --num-speculative-tokens 4 \
  --gpu-memory-utilization 0.92 \
  --dtype bfloat16

Key flags:

• --enable-mtp activates Multi-Token Prediction for speculative decoding
• --num-speculative-tokens 4 predicts 4 tokens ahead (good balance of speed vs accuracy)
• --max-model-len 262144 sets the native context length. For YaRN-extended 1M context, see 1M context strategies
• --gpu-memory-utilization 0.92 uses 92% of available VRAM

Verify the Server

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3.8-27B",
    "messages": [{"role": "user", "content": "Hello, are you running?"}],
    "max_tokens": 50
  }'

Option 2: SGLang Server

SGLang excels at structured generation and agent workflows. Its MTP-aware scheduling reduces latency in rapid tool-call chains — exactly the pattern coding agents use.

Installation

pip install sglang[all]>=0.5.0

Launch the Server

python -m sglang.launch_server \
  --model Qwen/Qwen3.8-27B \
  --port 8000 \
  --context-length 262144 \
  --enable-mtp \
  --speculative-num-steps 4 \
  --dtype bfloat16

SGLang's advantage over vLLM for agentic workloads is its scheduling: when the model generates a tool call and pauses for the result, SGLang manages the KV cache efficiently across the pause, reducing re-computation on resume.

Option 3: LM Studio (GGUF)

The zero-friction option. LM Studio provides a GUI for downloading and running GGUF models with an OpenAI-compatible API endpoint.

1. Download LM Studio from lmstudio.ai
2. Search for Qwen3.8-27B in the model browser
3. Download the Q5_K_M quantization (~17GB)
4. Load the model and enable the local API server on port 1234

LM Studio handles MTP configuration automatically on supported hardware. Throughput is lower than vLLM/SGLang but setup takes under 5 minutes.

MTP Configuration

Multi-Token Prediction is the key to fast local inference. Without MTP, you'll see 8-15 tok/s. With MTP enabled, expect 15-30+ tok/s depending on hardware.

How MTP Works

Instead of predicting one token at a time, the model predicts N future tokens in parallel using additional prediction heads. The inference engine then verifies these predictions against the model's autoregressive output. Correct predictions are accepted in batch — effectively multiplying throughput.

Tuning Speculative Tokens

Start with 4 speculative tokens. If you notice output quality issues (rare), drop to 2. If throughput matters more than occasional regeneration, try 8.

Reasoning Effort Configuration

As covered in the agentic coding overview, the default xhigh reasoning effort causes severe overthinking. For agent loops, configure medium as the default.

System Prompt for Agent Mode

Per-Request Effort Override

For most agent actions, use medium. When the agent encounters a complex bug or architectural question, temporarily escalate:

Agent Harness Integration

Claude Code Harness

If you use the Claude Code agent harness, you can point it at your local Qwen3.8-27B server:

export ANTHROPIC_BASE_URL=http://localhost:8000/v1
export ANTHROPIC_API_KEY=not-needed
export ANTHROPIC_MODEL=Qwen/Qwen3.8-27B

The harness expects an OpenAI-compatible API, which both vLLM and SGLang provide.

OpenCode

OpenCode supports local model backends natively. In your opencode.yaml:

model:
  provider: openai-compatible
  base_url: http://localhost:8000/v1
  model: Qwen/Qwen3.8-27B
  api_key: not-needed
reasoning:
  effort: medium
  max_thinking_tokens: 5000
  preserve_thinking: true

Direct API Agent Loop

For custom agent implementations, here's the core loop pattern:

import openai

client = openai.OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-needed"
)

def agent_step(messages, tools, effort="medium"):
    response = client.chat.completions.create(
        model="Qwen/Qwen3.8-27B",
        messages=messages,
        tools=tools,
        reasoning_effort=effort,
        max_tokens=4096
    )
    return response.choices[0].message

# Agent loop
while not done:
    response = agent_step(messages, tools)
    if response.tool_calls:
        # Execute tool calls, append results
        for call in response.tool_calls:
            result = execute_tool(call)
            messages.append({"role": "tool", "content": result})
    else:
        # Agent finished
        done = True

Performance Tuning Checklist

1. Enable MTP with 4 speculative tokens for 2x+ speed improvement
2. Set reasoning_effort: medium as the default for agent loops
3. Budget ~5,000 reasoning tokens — enough for meaningful thought, fast enough for iteration
4. Use preserve_thinking: on during development to debug prompt issues
5. Set --gpu-memory-utilization 0.92 (vLLM) to maximize available context
6. Monitor token generation speed — below 10 tok/s, check MTP configuration and quantization level
7. Use Q5_K_M or higher quantization — lower quantizations noticeably degrade coding ability

For detailed model benchmarks and architecture information, see the Qwen3.8-27B model guide.