iTranslated by AI
Building a Local LLM Environment with RX 7900 XTX, WSL2, ROCm, and vLLM
Introduction
What this article covers
This article is a compilation of my struggles and successes in building a local LLM environment using Windows 11 + WSL2 + ROCm + vLLM with an RX 7900 XTX. This is intended for those who want to try local LLMs on AMD GPUs or want to be liberated from cloud AI restrictions.
After reading this article, you will be able to:
- Comfortably run 7B class models at 270+ tok/s on an RX 7900 XTX
- Use it as an OpenAI-compatible API from other tools
- Fully utilize LLMs without restrictions while maintaining your privacy
Why I wanted to build a local LLM environment
I am a heavy user who utilizes AI on a daily basis. I subscribe to a certain cloud AI's Pro plan and use LLMs for everything from simple searches to deep conversational reasoning, thought expansion, home lab consultation, and writing.
However, starting in 2026, the restrictions on the Pro plan became quite strict. The MAX plan is too expensive, and I found myself increasingly wondering if there was a better way.
That was when I realized: If I build a local LLM environment, wouldn't I be able to use it without limits?
I have an RX 7900 XTX (not NVIDIA)
However, the GPU I own is the AMD Radeon RX 7900 XTX (24GB). Since I didn't have an NVIDIA GPU, which is the mainstream choice in the LLM community, I started my investigation by asking: "Can you even run local LLMs on an AMD GPU?"
When I bought this GPU, the common image was that if you wanted AI performance, you chose NVIDIA, and while AMD might be possible, it wasn't well-suited.
However, as I continued my research, I found information suggesting that AMD GPUs have become quite capable recently. After several days of trial and error, I succeeded in building a practical local LLM environment using WSL2 + ROCm + vLLM.
Can the RX 7900 XTX compete?
The AMD Radeon RX 7900 XTX (24GB) has become a very competitive GPU as of February 2026, thanks to the maturation of ROCm and official support from vLLM.
Here is a rough comparison of GPUs (as of February 2026).
| GPU | VRAM | Price Range (Approx.) | Price per GB (Approx.) | LLM Evaluation |
|---|---|---|---|---|
| RX 7900 XTX | 24GB | ~130,000 JPY | ~5,400 JPY/GB | ★★★★★ |
| RTX 4090 | 24GB | ~280,000 JPY | ~11,600 JPY/GB | ★★★★☆ |
| RX 7900 XT | 20GB | ~100,000 JPY | ~5,000 JPY/GB | ★★★★☆ |
| RTX 4080 SUPER | 16GB | ~180,000 JPY | ~11,200 JPY/GB | ★★★☆☆ |
Strengths of the RX 7900 XTX:
- Equipped with 24GB of high-capacity VRAM
- Cost performance at less than half the price of an RTX 4090
- ROCm + vLLM runs properly (though with some constraints)
- Officially supported by vLLM as a "first-class platform" (announced in January 2026)
Results after building:
- Qwen2.5-7B-Instruct: 272 tok/s
- Mistral-7B-Instruct: 207 tok/s
- At these speeds, it is more than practical (though this may not apply to larger models).
In conclusion, my personal experience after days of trial and error is that "AMD GPUs are definitely competitive."
Of course, there are limitations compared to NVIDIA (e.g., FP8 KV cache quantization is not available). However, if you prioritize cost-performance for home lab use, the RX 7900 XTX is a very strong option.
Environment Information
Hardware Configuration (Author's PC)
- CPU: Intel Core i7-14700KF (20 cores, 28 threads)
- Memory: DDR4 64GB
- GPU: AMD Radeon RX7900XTX (24GB VRAM)
- Storage: NVMe SSD 2TB
- Motherboard: ASUS TUF GAMING Z790-PLUS WIFI D4
- Power Supply: 1200W (80PLUS Gold)
Software Configuration
- OS (Host): Windows 11 25H2
- AMD GPU Driver: AMD Software Adrenalin Edition 26.1.1
-
WSL2: Ubuntu 24.04 LTS
- Allocated Memory: 48GB
- Allocated Processors: 16 cores
- Swap: 16GB
- ROCm: 7.2
- PyTorch: 2.9.1+rocm7.2.0
- vLLM: v0.15.0
- Python: 3.12
.wslconfig Settings
Windows side C:\Users\<Username>\.wslconfig:
[wsl2]
memory=48GB
processors=16
swap=16GB
- After configuring, run
wsl --shutdownin PowerShell and restart WSL.
Minimum Recommended Specs
| Item | Minimum | Recommended |
|---|---|---|
| CPU | Ryzen 5 / Core i5 (6+ cores) | Ryzen 7 / Core i7 (8+ cores) |
| Memory | 32GB (16GB allocated to WSL2) | 64GB (32GB+ allocated to WSL2) |
| GPU | RX 7900 XT (20GB VRAM) | RX 7900 XTX (24GB VRAM) |
| SSD | NVMe 500GB | NVMe 1TB or more |
| PSU | 750W or more | 800-850W or more |
1. Setting up WSL2 and Ubuntu 24.04
1-1. Enabling WSL2
In PowerShell (Admin):
wsl --install
After restarting, verify the version.
wsl --version
1-2. WSL2 Resource Allocation (.wslconfig)
Create the .wslconfig mentioned above.
Use roughly 70-75% of host memory for memory and about 80% of physical cores for processors.
After creating/editing, run in PowerShell:
wsl --shutdown
1-3. Installing Ubuntu 24.04
wsl --install -d Ubuntu-24.04
Set your username/password upon the first launch.
1-4. Installing Basic Packages
Launch Ubuntu in WSL and run:
sudo apt update && sudo apt upgrade -y
sudo apt install -y \
build-essential git wget curl \
python3-pip python3-venv \
ninja-build
2. Setting up AMD GPU Drivers and ROCm
2-1. AMD Drivers on Windows
- Install AMD Software: Adrenalin Edition 26.1.1.
- If you already have a different version installed, it is safer to update to 26.1.1 or later.
(I will omit the detailed steps as it is simply following the official installer.)
2-2. Installing ROCm (on Ubuntu-24.04)
Since ROCm installation methods tend to change with every version, I strongly recommend checking the official instructions first.
- ROCm Official: https://rocm.docs.amd.com/
- Setup for WSL2: Refer to the section for WSL2
wget https://repo.radeon.com/amdgpu-install/7.2/ubuntu/noble/amdgpu-install_7.2.70200-1_all.deb
sudo apt install ./amdgpu-install_7.2.70200-1_all.deb
sudo amdgpu-install -y --usecase=wsl,rocm --no-dkms
2-3. Verifying GPU Recognition (on Ubuntu-24.04)
rocm-smi does not work in WSL, so use rocminfo to verify.
rocminfo | grep "Name:"
If gfx1100 is included, the RX 7900 XTX is correctly recognized.
3. Installing ROCm-supported PyTorch
3-1. Creating a Python Virtual Environment
Create a venv directly under your home directory.
python3 -m venv ~/ai-ws
cd ~/ai-ws
source ~/ai-ws/bin/activate
You can name the directory ai-ws or anything you prefer.
3-2. Installing PyTorch (ROCm build)
pip3 install --upgrade pip wheel
wget https://repo.radeon.com/rocm/manylinux/rocm-rel-7.2/torch-2.9.1%2Brocm7.2.0.lw.git7e1940d4-cp312-cp312-linux_x86_64.whl
wget https://repo.radeon.com/rocm/manylinux/rocm-rel-7.2/torchvision-0.24.0%2Brocm7.2.0.gitb919bd0c-cp312-cp312-linux_x86_64.whl
wget https://repo.radeon.com/rocm/manylinux/rocm-rel-7.2/triton-3.5.1%2Brocm7.2.0.gita272dfa8-cp312-cp312-linux_x86_64.whl
wget https://repo.radeon.com/rocm/manylinux/rocm-rel-7.2/torchaudio-2.9.0%2Brocm7.2.0.gite3c6ee2b-cp312-cp312-linux_x86_64.whl
pip3 uninstall torch torchvision triton torchaudio
pip3 install torch-2.9.1+rocm7.2.0.lw.git7e1940d4-cp312-cp312-linux_x86_64.whl torchvision-0.24.0+rocm7.2.0.gitb919bd0c-cp312-cp312-linux_x86_64.whl torchaudio-2.9.0+rocm7.2.0.gite3c6ee2b-cp312-cp312-linux_x86_64.whl triton-3.5.1+rocm7.2.0.gita272dfa8-cp312-cp312-linux_x86_64.whl
3-3. Verification
python - << 'PY'
import torch
print("CUDA available:", torch.cuda.is_available())
if torch.cuda.is_available():
print("Device count:", torch.cuda.device_count())
print("Device name:", torch.cuda.get_device_name(0))
PY
If it looks like this, you are good to go:
CUDA available: True
Device count: 1
Device name: AMD Radeon RX 7900 XTX
4. Installing vLLM
4-1. Selecting the vLLM Version
At the time of writing (February 2026), the latest version of vLLM is v0.16.0, but v0.15.0 is the stable version, so we will use v0.15.0 in this guide. Here is a summary of the characteristics of each.
Features of v0.15.0
- No MLA support
- Flash Attention is treated as an optional feature
- Simple and stable configuration
Features of v0.16.0
- Added MLA support (for models like DeepSeek v3)
-
Changed to require Flash Attention (this can be bypassed by setting the environment variable
BUILD_FA=0) - Standard attention models (Qwen2.5, Mistral, etc.) work without issues
Notes for RX 7900 XTX
The RX 7900 XTX (RDNA3 / gfx1100) does not support Flash Attention.
When using v0.16.0, please note the following:
- By default, Flash Attention is required.
- You can build without Flash Attention by setting the
BUILD_FA=0environment variable. - Triton Attention will be used, so there is no issue with performance.
- Standard attention models will function normally.
- MLA models (GLM-4.7-Flash, DeepSeek-V2/V3, etc.) cannot be used.
Conclusion
- If you only use standard attention models: Either v0.15.0 or v0.16.0 is fine.
- For simplicity: v0.15.0 is recommended.
-
To try out the latest features: Use v0.16.0 and append
BUILD_FA=0to the environment variables.
4-2. Cloning the Repository
cd ~/ai-ws
git clone https://github.com/vllm-project/vllm.git
cd vllm
git fetch --tags
git checkout v0.15.0
We specify the version mentioned above using git checkout v0.15.0.
4-3. Applying use_existing_torch.py (Important)
vLLM provides a script to build based on an already installed PyTorch.
If you do not apply this, the following will occur:
- The default vLLM build will attempt to automatically install the CUDA version of PyTorch as a dependency.
- The ROCm version of PyTorch (2.9.1+rocm7.2.0) that you carefully installed will be overwritten.
- As a result, the GPU will no longer be recognized.
Therefore, do not forget to apply this.
python use_existing_torch.py
4-4. Installing Dependencies
pip install -r requirements/build.txt
pip install -r requirements/rocm.txt
pip install ninja cmake wheel packaging pybind11
4-5. Setting ROCm Environment Variables
export PYTORCH_ROCM_ARCH="gfx1100"
export HSA_OVERRIDE_GFX_VERSION=11.0.0
If you are using v0.16.0, add the following as well:
export BUILD_FA=0 # Build without Flash Attention
4-6. Building vLLM
pip install -e . --no-build-isolation
It takes about 10 minutes to complete, so please be patient.
4-7. Verification
python - << 'PY'
import vllm
print("vLLM version:", vllm.__version__)
PY
If a string like 0.15.0+... appears, you are successful.
5. [Important] WSL2-Specific Issues and Fixes
This is a part where I struggled quite a bit in my previous scraps. vLLM will not start unless you apply this fix.
5-1. ROCm Platform Not Detected
When you try to start vLLM on WSL2, you will get the following error:
RuntimeError: Device string must not be empty
What is happening?
At first glance, it looks like an error stating the "device string is empty," but the actual cause is further upstream.
vLLM checks whether ROCm can be used at startup using a tool called amdsmi (AMD System Management Interface).
However, since amdsmi does not function in WSL2, the detection of the ROCm platform fails.
Looking at the logs in detail:
ROCm platform is not available because: Error code: 34
As a result:
- ROCm platform detection fails.
- The GPU is not recognized.
- The device string becomes empty.
-
RuntimeError: Device string must not be emptyoccurs.
5-2. Modifying the Code
We will modify the code to detect ROCm using PyTorch instead of amdsmi.
File to modify:
~/ai-ws/vllm/vllm/platforms/__init__.py
nano ~/ai-ws/vllm/vllm/platforms/__init__.py
Modification:
Rewrite the rocm_platform_plugin() function as follows:
def rocm_platform_plugin() -> str | None:
is_rocm = False
logger.debug("Checking if ROCm platform is available.")
try:
# Use PyTorch instead of amdsmi
import torch
if hasattr(torch.version, 'hip') and torch.version.hip is not None:
is_rocm = torch.cuda.is_available() and torch.cuda.device_count() > 0
if is_rocm:
logger.debug("Confirmed ROCm platform is available via torch.version.hip.")
else:
logger.debug("ROCm platform is not available because no GPU is found.")
else:
logger.debug("ROCm platform is not available because torch.version.hip is not found.")
except Exception as e:
logger.debug("ROCm platform is not available because: %s", str(e))
return "vllm.platforms.rocm.RocmPlatform" if is_rocm else None
Original Code (Reference)
def rocm_platform_plugin() -> str | None:
is_rocm = False
logger.debug("Checking if ROCm platform is available.")
try:
import amdsmi # ← This does not work in WSL2
amdsmi.amdsmi_init() # ← Fails with Error code: 34
try:
if len(amdsmi.amdsmi_get_processor_handles()) > 0:
is_rocm = True
logger.debug("Confirmed ROCm platform is available.")
else:
logger.debug("ROCm platform is not available because no GPU is found.")
finally:
amdsmi.amdsmi_shut_down()
except Exception as e:
logger.debug("ROCm platform is not available because: %s", str(e))
return "vllm.platforms.rocm.RocmPlatform" if is_rocm else None
| Item | Before Modification | After Modification |
|---|---|---|
| Detection Method | amdsmi | torch.version.hip |
| GPU Check | amdsmi.amdsmi_get_processor_handles() | torch.cuda.is_available() |
| WSL2 Support | ❌ Does not work | ✅ Works |
Why does it work with PyTorch?
WSL2 Constraints:
Because it cannot manage the GPU directly like native Linux, AMD management tools like rocm-smi and amdsmi cannot be used. Access must go through the Windows host.
PyTorch supports WSL2:
- It can retrieve ROCm version information via
torch.version.hip. - It can confirm the existence of the GPU via
torch.cuda.is_available(). - It possesses mechanisms to function correctly even through the Windows host.
5-3. Verification
After the modification, check with the following command:
python -c "from vllm.platforms import rocm_platform_plugin; print(rocm_platform_plugin())"
Expected output:
/home/username/ai-ws/lib/python3.12/site-packages/torch/cuda/__init__.py:814: UserWarning: Can't initialize amdsmi - Error code: 34
warnings.warn(f"Can't initialize amdsmi - Error code: {e.err_code}")
vllm.platforms.rocm.RocmPlatform
If this is displayed, ROCm platform detection has succeeded. You may see the amdsmi error in the first two lines, but that is not an issue. We can finally run vLLM on WSL2.
6. Persisting Environment Variables (.bashrc)
Running export every time is cumbersome, so let's add them to .bashrc.
nano ~/.bashrc
Add the following to the end:
# ROCm Environment
export PYTORCH_ROCM_ARCH="gfx1100"
export HSA_OVERRIDE_GFX_VERSION=11.0.0
export BUILD_FA=0
export ROCM_LLVM_ENABLE_WAVE_PROFILER=0
# Optional: Automatically activate venv if working only in this environment
source ~/ai-ws/bin/activate
Apply the changes:
source ~/.bashrc
7. Basic vLLM Functionality Verification
Now, let's create a test program and try calling vLLM directly from Python to test it.
nano test_vllm.py
# test_vllm.py
from vllm import LLM, SamplingParams
if __name__ == '__main__':
# Test operation with a lightweight model
llm = LLM(model="facebook/opt-125m")
# Simple prompt
prompts = ["Hello, my name is", "The future of AI is"]
# Generation parameters
sampling_params = SamplingParams(temperature=0.8, max_tokens=50)
# Run inference
outputs = llm.generate(prompts, sampling_params)
# Display results
for output in outputs:
prompt = output.prompt
generated = output.outputs[0].text
print(f"Prompt: {prompt}")
print(f"Output: {generated}")
print("-" * 80)
Run it:
python test_vllm.py
If the output is displayed normally, you have succeeded.
I plan to write a separate article on how to call it as an API from OpenWebUI.
8. Benchmark Results and Practicality
Since we've come this far, I decided to run some benchmarks.
8-1. Measurement Conditions
- GPU: AMD Radeon RX 7900 XTX (24GB)
- ROCm: 7.2
- vLLM: v0.15.0
- PyTorch: 2.9.1+rocm7.2.0
- WSL2: Ubuntu 24.04
-
Test Settings:
- Number of prompts: 10
- Max generation tokens: 512 / output
- Context length: 4096
gpu_memory_utilization=0.9
8-2. Benchmark Script Used
nano benchmark_simple.py
# benchmark_simple.py
import time
from vllm import LLM, SamplingParams
def benchmark_model(model_name, num_prompts=10, max_tokens=512):
print("\n" + "="*60)
print(f"Model: {model_name}")
print("="*60)
print("Loading model...")
t0 = time.time()
llm = LLM(
model=model_name,
trust_remote_code=True,
gpu_memory_utilization=0.9,
max_model_len=4096,
)
load_time = time.time() - t0
print(f"Load time: {load_time:.2f}s")
prompts = [
"Write a short story about artificial intelligence.",
"Explain quantum computing in simple terms.",
"What are the benefits of exercise?",
"Describe the process of photosynthesis.",
"Write a poem about nature.",
] * (num_prompts // 5)
sampling_params = SamplingParams(
temperature=0.8,
top_p=0.95,
max_tokens=max_tokens,
)
# Warmup
_ = llm.generate(prompts[:1], sampling_params)
print(f"\nRunning benchmark ({num_prompts} prompts, max {max_tokens} tokens)...")
t1 = time.time()
outputs = llm.generate(prompts, sampling_params)
elapsed = time.time() - t1
total_tokens = sum(len(output.outputs[0].token_ids) for output in outputs)
avg_tokens = total_tokens / len(outputs)
tps = total_tokens / elapsed
print("\n" + "="*60)
print("Results:")
print(f" Total time: {elapsed:.2f}s")
print(f" Total tokens generated: {total_tokens}")
print(f" Average tokens per output: {avg_tokens:.1f}")
print(f" Throughput: {tps:.2f} tokens/s")
print("="*60)
return {
"model": model_name,
"load_time": load_time,
"total_time": elapsed,
"total_tokens": total_tokens,
"tps": tps,
}
if __name__ == "__main__":
models = [
"Qwen/Qwen2.5-7B-Instruct",
"mistralai/Mistral-7B-Instruct-v0.3",
]
for m in models:
benchmark_model(m, num_prompts=10, max_tokens=512)
input("\nPress Enter to continue...")
Run it:
python benchmark_simple.py
8-3. Results
Qwen/Qwen2.5-7B-Instruct
- VRAM during model loading: 14.5 GB
- VRAM during inference (max): 21.9 GB
- Average generated tokens: approx. 453 tokens
- Throughput: 272.34 tokens/s
mistralai/Mistral-7B-Instruct-v0.3
- VRAM during model loading: 13.8 GB
- VRAM during inference (max): 21.2 GB
- Average generated tokens: approx. 401 tokens
- Throughput: 206.69 tokens/s
8-4. Personal Thoughts
- Qwen2.5-7B has strong Japanese language capabilities and, at over 270 tok/s, is almost zero-wait in terms of user experience.
- Mistral-7B is also very practical at over 200 tok/s.
- VRAM usage during inference is around 21–22GB, leaving 2–3GB of headroom even on 24GB cards.
With this headroom, you can adjust:
- Increasing context length
- Slightly increasing batch size
8-5. Model Size Estimates for 24GB VRAM
Rough estimates:
| Quantization | Model Size | Recommendation | Note |
|---|---|---|---|
| FP16 | 7B | ◎ | Plenty of room (the configuration in this article) |
| FP16 | 10B | △ | Barely fits. Assumes shorter context |
| AWQ | 32B | ○ | Practical with some tuning |
| AWQ | 70B | × | Basically difficult on 24GB VRAM |
Conclusion and Next Steps
In this article, we accomplished the following:
- Built a local LLM environment using RX 7900 XTX + WSL2 + ROCm + vLLM
- Verified performance of 200–270 tok/s with Qwen2.5-7B / Mistral-7B
- Worked around the WSL2-specific
amdsmiissue
Things I am considering as next steps:
-
vLLM + OpenWebUI Integration (upcoming article)
- Launch OpenWebUI from Docker on WSL
- Connect to vLLM's OpenAI-compatible endpoint
- Use local LLMs with a ChatGPT-like UI
-
Running 30B class models using quantized models (Article already written!)
- Running models like Qwen3-Coder-30B-AWQ on the RX 7900 XTX
-
Summary of considerations for using MoE models (Article already written!)
- Discussing why you need to modify some vLLM executables when using MoE models
-
Doubling context length by FP8 quantization of the KV cache (Article already written!)
- ROCm FP8 quantization is now available in vLLM v0.16.0
Thank you for reading this far.
I hope this serves as a helpful reference for anyone else looking to run vLLM on an RX 7900 XTX + WSL2 setup.
Please let me know in the comments if you have any questions or feedback.
Discussion