serving-llms-vllm

Name: serving-llms-vllm
Availability: InStock
Author: orchestra-research

Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching. Use when deploying production LLM APIs, optimizing inference…

copies: implexa run skills.sh/serving-llms-vllm

view source

installs

--

stars

--

karma

--

SKILL.md

vLLM - High-Performance LLM Serving

Quick start

vLLM achieves 24x higher throughput than standard transformers through PagedAttention (block-based KV cache) and continuous batching (mixing prefill/decode requests).

Installation:

pip install vllm

Basic offline inference:

from vllm import LLM, SamplingParams

llm = LLM(model="meta-llama/Llama-3-8B-Instruct")
sampling = SamplingParams(temperature=0.7, max_tokens=256)

don't have the plugin yet? install it then click "run inline in claude" again.