SeedInfer — Private AI on Verified Hardware

Faza 0 · NVFP4seedinfer/nemotron-lightning-1m1M ctx · 2M KVCUDA 13.3 · driver 580.65+47900:8000 · 47901:3001

SeedInfer Docs — run your own node

seedinfer/nemotron-lightning-1m (NVIDIA Nemotron 3.5 Lightning 30B A3B NVFP4 — 30B/3B MoE+Mamba, 1M ctx) on RTX 5090 32GB (GB202, Blackwell). Earn $0.02 / 1M input + $0.05 / 1M output. Tailscale Headscale + vLLM nightly + heartbeat.

Hardware ↓ One-liner install ↓Verify ↓/provider → full guide

Input

$0.02 / 1M

prompt tokens

Output

$0.05 / 1M

completion tokens

Context

1M · 2M KV

~22-28 GB VRAM

Quick requirements

ModelNVFP4 1M $0.02/$0.05

GPU minRTX 5090 32GB

VRAM22-28GB (16-22+6 KV)

OSUbuntu 24.04 noble

Driver / CUDA580.65+ / 13.3

Ports47900:8000 + 47901:3001

Disk60GB+ free (df -h)

NVFP4 flags host 1:1: marlin + flashinfer + fp8 · 0.93/1048576/128/4096

VRAM math — dlaczego 32GB

Wagi NVFP4

16-22 GB

W4A16 + FP8 via ModelOpt (vs 66GB BF16). On-disk ~20-30GB download z HF.

KV cache 1M

~6 GB

FP8 KV, 1M tokens, --kv-cache-dtype fp8. 2M KV max = ~12GB at full buffer.

Total + headroom

22-28 GB

+ --gpu-memory-utilization 0.93 + --max-num-batched-tokens 4096 → 32GB daje ~4-10GB zapasu. Na 24GB OOM bez downscale.

On OOM: lower VLLM_GPU_MEMORY_UTILIZATION=0.80 + VLLM_MAX_MODEL_LEN=32768 or 131072. Only then is 24GB (3090/4090) stable — see 24GB tier plan below.

GPU Matrix — minimum RTX 5090 32GB (Blackwell)

Minimum RTX 5090 32GB (GB202, Blackwell sm_120, 21760 CUDA, 680 Tensor 5th gen, 32GB GDDR7 ~1.8 TB/s). NVFP4 host env VLLM_ATTENTION_BACKEND=FLASHINFER + VLLM_NVFP4_GEMM_BACKEND=flashinfer-cutlass + VLLM_MOE_BACKEND=marlin + VLLM_MAMBA_BACKEND=flashinfer. A100/H100 auto-fallback humming.

GPU	Arch	VRAM	BW	NVFP4 1M	Est. tput*	Status
RTX 5090 32GB	GB202 sm_120	32GB GDDR7	~1.8 TB/s	✅ ~22-28GB	~120-180 tok/s	minimum
A100 40GB	GA100 sm_80	40GB HBM2e	1.6 TB/s	✅ W4A16	~60-90	welcome
A100 80GB	GA100	80GB HBM2e	2.0 TB/s	✅	~70-100	welcome
H100 80GB	H100 sm_90	80GB HBM3	3.0 TB/s	✅	~150-220	welcome
L40S 48GB	AD102 sm_89	48GB GDDR6	864 GB/s	✅	~80-120	welcome
RTX 6000 Ada 48GB	AD102	48GB GDDR6	960 GB/s	✅	~80-120	welcome
RTX 6000 Pro Blackwell	GB202	96GB GDDR7	~1.8 TB/s+	✅ 96GB	~130-190	welcome
RTX 4500 Blackwell 32GB	GB203	32GB GDDR7	~1.0 TB/s	✅	~90-130	welcome
RTX 5000 Blackwell	GB203	32-48GB GDDR7	~1.2 TB/s	✅	~110-160	welcome
RTX 3090 24GB ⏳	GA102 sm_86	24GB GDDR6X	936 GB/s	⚠️ tight 24GB	~50-80	plan
RTX 4090 24GB ⏳	AD102 sm_89	24GB GDDR6X	1.0 TB/s	⚠️ tight	~70-100	plan

* Est. tput single-user prefill+decode for Nemotron 30B NVFP4 (W4A16+FP8 KV), batch 1, 1k in / 256 out, no prefix cache. Real throughput depends on marlin (Blackwell FP4 + flashinfer-cutlass) / humming (A100 W4A16) + KV hit.
Eventually 3090/4090 (24GB) — planned as "community" tier z auto-downscale VLLM_MAX_MODEL_LEN=131072 + VLLM_GPU_MEMORY_UTILIZATION=0.85. Currently welcome for testing, but the official minimum is 32GB.

OS / Driver / CUDA matrix

Stack	Wersja wymagana	Blackwell GB202	Fallback	Status
OS	Ubuntu 24.04+ (noble) kernel 6.8+	✅	Ubuntu 22.04+	required
Driver + CUDA	580.65+ + CUDA 13.3	✅ native sm_120	570+ (13.2) PTX JIT	native
Driver fallback	570.86+ + CUDA 13.2	⚠️ via PTX JIT	PTX forward compat	fallback
Legacy	550.90+ + CUDA 12.4	⚠️ legacy JIT	no Blackwell nat.	legacy
Docker	24+ + compose plugin	✅	—	required
nvidia-ctk	nvidia-container-toolkit	✅	install auto via install.sh	auto
Tailscale	1.82+	✅	tailnet.seedinfer.com	auto
vLLM	nightly cu12 (PTX JIT)	✅ `pip install --pre vllm --extra-index-url https://wheels.vllm.ai/nightly`	native cu13 when avail.	nightly
Disk	50GB HF cache + 28GB vLLM	60GB+ free (df -h)	`df -h`	50GB+
Ports	47900:8000 (vLLM) + 47901:3001 (agent)	host mapping, `VLLM_PORT/AGENT_PORT` override	41000+ if busy	47900/47901

CUDA PTX JIT: vLLM nightly cu12 wheels run on CUDA 13.3 via forward-compat (driver 580+). No rebuild required. Blackwall sm_120 native PTX only on 580+. Check: nvidia-smi | grep Driver + nvidia-smi --query-gpu=compute_cap --format=csv (12.0 dla 5090). With driver <580: warn, <570: warn, <550: error in install.sh/entrypoint.sh.

Instalacja — one-liner vs manual

Terminal command that sets up the environment + verification. Copy and paste on Linux.

★ Recommended — one command (auto-authkey + prebuild)

curl -fsSL https://seedinfer.com/install.sh | bash

No parameters — install.sh automatically fetches an authkey from /api/v1/auth/request + prebuild ghcr.io/seedinfer/provider:cuda13.3-nvfp4 || https://seedinfer.com/provider-image.tar.gz (Pi) || build. Pi nie buduje CUDA — tylko hostuje.

Advanced — custom authkey / model / gateway (expand)

1 · Z kluczem z Generate invite

curl -fsSL https://seedinfer.com/install.sh | bash -s -- --authkey YOUR_AUTHKEY

2 · Auto-fetch key (jq)

curl -fsSL https://seedinfer.com/install.sh | bash -s -- --authkey $(curl -s https://seedinfer.com/api/v1/auth/request | jq -r .authkey)

3 · Full options + hostname

curl -fsSL https://seedinfer.com/install.sh | bash -s -- --authkey YOUR_AUTHKEY --model seedinfer/nemotron-lightning-1m --gateway https://seedinfer.com --hostname provider-5090

ENV: SEEDINFER_PREBUILD_IMAGE SEEDINFER_PREBUILD_URL SEEDINFER_SKIP_PREBUILD=1

Key YOUR_AUTHKEY → generate at /provider → Generate invite (valid 24h, tag:provider) — now optional, install.sh auto-fetches if missing. Details at /provider.

Steps — what install.sh does

nvidia-smi check — VRAM ≥32GB, driver 580+, ports 47900/47901 free; auto-fetch authkey from /api/v1/auth/request if --authkey is missing
Installs docker 24+ + nvidia-ctk + tailscale if missing
Domyślnie: kontener — jeśli host już w tailscale.com (100.94.x.x) → docker run -d --name tailscale-seedinfer --restart unless-stopped --cap-add=NET_ADMIN --cap-add=NET_RAW --device /dev/net/tun -v tailscale-seedinfer-state:/tailscale -e TS_AUTHKEY -e TS_HOSTNAME -e TS_LOGIN_SERVER -e TS_EXTRA_ARGS="--advertise-tags=tag:provider --accept-routes" tailscale/tailscale:latest (healthcheck, volume, network seedinfer-tailnet). Współistnienie 100.94.x.x (host, tailscale.com) + 100.64.x.x (kontener, Headscale) — nie rozłącza. Provider agent używa kontenera (DNS 100.64.x.x). Opt-in host: --force-host-tailscale (--reset, rozłączy tailscale.com) lub TAILSCALE_USE_CONTAINER=0. Jeśli brak istniejącego tailnetu → host tailscale up --login-server https://tailnet.seedinfer.com --authkey XXX --advertise-tags tag:provider
Clones provider/ → /opt/seedinfer-provider, creates .env (VLLM_MODEL=nvidia/...NVFP4)
Prebuild: docker pull ghcr.io/seedinfer/provider:cuda13.3-nvfp4 → curl https://seedinfer.com/provider-image.tar.gz | docker load (Pi) → docker compose up -d --build (fallback) → vLLM auto-download ~30GB → ./models/cache. Sidecar compose: docker compose --profile tailscale up -d (tailscale service, volume tailscale-seedinfer-state)
Heartbeat every 30s to /api/v1/providers/heartbeat → pending → verifying → verified

Manual (dev)

git clone https://github.com/seedinfer/seedinfer.com.git
cd seedinfer.com
cp provider/.env.example provider/.env
# edytuj TAILSCALE_AUTHKEY, MODEL, SEEDINFER_GATEWAY_URL
docker compose -f provider/docker-compose.yml up -d --build
docker logs -f seedinfer-provider | grep -i download
du -sh ./models/cache
curl -fsS http://127.0.0.1:47901/health | jq

# Tailscale — domyślnie kontener (nie rusza hosta, współistnienie 100.94.x.x + 100.64.x.x):
docker exec tailscale-seedinfer tailscale status  # kontener 100.64.x.x (Headscale)
tailscale status  # host 100.94.x.x (dom, tailscale.com — nienaruszony)
docker exec tailscale-seedinfer tailscale ip -4  # 100.64.x.x
tailscale ip -4  # host 100.94.x.x
ping -c2 gateway.seedinfer.ts.net
# Sidecar compose: docker compose --profile tailscale up -d
# volume: tailscale-seedinfer-state:/tailscale  network: seedinfer-tailnet
# healthcheck: tailscale status  cap_add: [NET_ADMIN, NET_RAW]  device: /dev/net/tun
# TS_EXTRA_ARGS="--advertise-tags=tag:provider --accept-routes"  TS_LOGIN_SERVER=https://tailnet.seedinfer.com
# Opt-in host (rozłączy tailscale.com): curl ... | bash -s -- --force-host-tailscale

/install.sh·/provider.tar.gz·/api/v1/auth/request·/providers → fleet

Verification — health, models, fleet

Lokalnie — vLLM + agent

curl -fsS http://127.0.0.1:47901/health | jq
# {"status":"ok","provider_id":"...","vllm_health":{"status":"ok"},"gpu":{"count":1}}
curl -fsS http://127.0.0.1:47900/v1/models | jq
curl http://127.0.0.1:47901/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"seedinfer/nemotron-lightning-1m","messages":[{"role":"user","content":"Hello"}],"max_tokens":32}'

Host ports: 47900:8000 vLLM, 47901:3001 agent. Env override: VLLM_PORT=41000 AGENT_PORT=41001.

Gateway — heartbeat + verify

# heartbeat co 30s (agent/main.py)
POST https://seedinfer.com/api/v1/providers/heartbeat
# auto-verify po 2 heartbeat (~60s)
GET https://seedinfer.com/api/v1/providers | jq
# verification: pending -> verifying -> verified
# manual verify:
curl -X POST https://seedinfer.com/api/v1/providers/verify \
  -H "Content-Type: application/json" \
  -d '{"provider_id":"provider-5090-xxx"}' | jq

Fleet UI + API

/providers — pending/verifying opacity 60, verified 🟢 opacity 100 (official node).
GET /api/v1/providers?verified=1 — tylko verified.
Pi gateway decyduje verified (nie Headscale ACL).

Endpoints: /api/v1/providers · /api/v1/models · /api/v1/pricing

Pi gateway — Headscale & telemetry

# Control plane: tailnet.seedinfer.com (Headscale)
# Authkey: curl -fsSL https://seedinfer.com/api/v1/auth/request | jq
# Docs: https://seedinfer.com/docs
curl -fsS https://seedinfer.com/api/v1/providers | jq '.data[] | {id,status,verification}'
tailscale status  # na providerze
docker logs -f seedinfer-provider | grep -E "heartbeat|vllm|download"

Troubleshooting

Symptom	Cause	Fix
OOM CUDA	1M ctx + 0.93 on 24GB, full KV	`VLLM_MAX_MODEL_LEN=32768` + `VLLM_GPU_MEMORY_UTILIZATION=0.80` → restart
`nvidia-smi` brak	driver <580, missing module	`sudo apt update && sudo apt install nvidia-driver-580 && sudo reboot` · fallback 570 / 550 legacy · `ubuntu-drivers autoinstall`
docker no nvidia runtime	missing nvidia-ctk	`sudo apt install nvidia-container-toolkit && sudo nvidia-ctk runtime configure --runtime=docker && sudo systemctl restart docker`
port 47900/47901 in use	`ss -tlnp \| grep :4790`	`VLLM_PORT=41000 AGENT_PORT=41001 curl ... \| bash -s -- --authkey XXX` (zakres 479xx wolny, lub 41000+)
disk full / no space	HF cache 60GB + vLLM 28GB	`df -h` · `docker system prune -a` · `rm -rf ./models/cache/.../snapshots` · check 60GB+ free
vLLM down in /health	HF model missing, QUANTIZATION mismatch	`docker logs seedinfer-provider --tail 100` · `VLLM_QUANTIZATION=modelopt` (nie modelopt_fp4) + `--kv-cache-dtype fp8` auto
tailscale invalid authkey	key expired (24h)	New key: `curl -fsSL https://seedinfer.com/api/v1/auth/request \| jq` + `docker exec tailscale-seedinfer tailscale up --authkey NEW` (kontener) lub `tailscale up --authkey NEW` (host) — details /docs
host już w tailscale.com 100.94.x.x	domowy tailnet, nie chcesz rozłączać	Domyślnie: kontener — `install.sh` auto `docker run -d --name tailscale-seedinfer --restart unless-stopped --cap-add=NET_ADMIN --cap-add=NET_RAW --device /dev/net/tun -v tailscale-seedinfer-state:/tailscale -e TS_AUTHKEY -e TS_HOSTNAME -e TS_LOGIN_SERVER -e TS_EXTRA_ARGS="--advertise-tags=tag:provider --accept-routes" tailscale/tailscale:latest` (healthcheck, volume, network). Współistnienie `100.94.x.x` (host) + `100.64.x.x` (kontener) — nie rozłącza. Provider używa kontenera (`docker exec tailscale-seedinfer tailscale status`). Compose: `docker compose --profile tailscale up -d`. Opt-in host: `--force-host-tailscale` (`--reset`, rozłączy) lub `TAILSCALE_USE_CONTAINER=0`
tailscale-seedinfer nie startuje	volume/network/authkey	`docker logs tailscale-seedinfer` · `docker volume create tailscale-seedinfer-state` · `docker network create seedinfer-tailnet` · check `TS_AUTHKEY`/`TS_LOGIN_SERVER` · healthcheck `tailscale status`
heartbeat 401/404	gateway path fallback	Agent fallback na `/api/providers/heartbeat` — log warn, nie krytyczne · check `SEEDINFER_GATEWAY_URL`
chat_template error	jinja missing	HF `tokenizer_config.json` jinja auto z nvidia — nie nadpisuj · fallback `./provider/assets:/qwen_setup:ro` · `VLLM_CHAT_TEMPLATE=/qwen_setup/...`

Logi: docker compose -f provider/docker-compose.yml logs -f · docker exec tailscale-seedinfer tailscale status (kontener 100.64.x.x) · tailscale status (host 100.94.x.x — nienaruszony) · docker logs tailscale-seedinfer · curl -fsS http://127.0.0.1:47901/metrics | jq · nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv · df -h && du -sh ./models/cache

Pliki, env, endpoints

provider.tar.gz

Full provider/ pack — Dockerfile.cuda + compose + agent

https://seedinfer.com/provider.tar.gz →

install.sh

One-liner plug-and-play ~279 linii · NVFP4

https://seedinfer.com/install.sh

Control plane

Headscale tailnet.seedinfer.com (WireGuard) · Gateway https://seedinfer.com

docs.seedinfer.com /docs →

ENV	Default	Opis (host 1:1 RTX 5090 GB202)
VLLM_MODEL	nvidia/...NVFP4	HF repo NVFP4 (plug-and-play auto-download)
MODEL	seedinfer/nemotron-lightning-1m	logiczny alias via `--served-model-name`
VLLM_QUANTIZATION	modelopt	compressed-tensors ModelOpt NVFP4 (nie modelopt_fp4/auto)
VLLM_KV_CACHE_DTYPE	fp8	FP8 KV (nie auto)
VLLM_MOE_BACKEND	marlin	Blackwell FP4 native · A100/H100 → humming
VLLM_MAMBA_BACKEND	flashinfer	host 1:1
VLLM_ATTENTION_BACKEND	FLASHINFER	host 1:1
VLLM_GPU_MEMORY_UTILIZATION	0.93	host 1:1 dla 32GB (nie 0.90)
VLLM_MAX_MODEL_LEN	1048576	1M ctx (2M KV)
VLLM_PORT / AGENT_PORT	47900 / 47901	host→container 8000/3001, env override
TAILSCALE_*	tailnet.seedinfer.com	Headscale login-server + tag:provider

/api/v1/models /api/v1/pricing /api/v1/providers /api/v1/telemetry /install.sh /provider.tar.gz

SeedInfer.com · Docs · Faza 0 CUDA — /install.sh · /provider.tar.gz · /api/v1/auth/request · /api/v1/providers · /providers · /provider · docs.seedinfer.com /docs