iTranslated by AI

The content below is an AI-generated translation. This is an experimental feature, and may contain errors. View original article
💻

Building a Local LLM Environment with RX 7900 XTX, WSL2, ROCm, and vLLM

に公開
2026/02/28

Introduction

What this article covers

This article is a compilation of my struggles and successes in building a local LLM environment using Windows 11 + WSL2 + ROCm + vLLM with an RX 7900 XTX. This is intended for those who want to try local LLMs on AMD GPUs or want to be liberated from cloud AI restrictions.

After reading this article, you will be able to:

  • Comfortably run 7B class models at 270+ tok/s on an RX 7900 XTX
  • Use it as an OpenAI-compatible API from other tools
  • Fully utilize LLMs without restrictions while maintaining your privacy

Why I wanted to build a local LLM environment

I am a heavy user who utilizes AI on a daily basis. I subscribe to a certain cloud AI's Pro plan and use LLMs for everything from simple searches to deep conversational reasoning, thought expansion, home lab consultation, and writing.

However, starting in 2026, the restrictions on the Pro plan became quite strict. The MAX plan is too expensive, and I found myself increasingly wondering if there was a better way.

That was when I realized: If I build a local LLM environment, wouldn't I be able to use it without limits?

I have an RX 7900 XTX (not NVIDIA)

However, the GPU I own is the AMD Radeon RX 7900 XTX (24GB). Since I didn't have an NVIDIA GPU, which is the mainstream choice in the LLM community, I started my investigation by asking: "Can you even run local LLMs on an AMD GPU?"

When I bought this GPU, the common image was that if you wanted AI performance, you chose NVIDIA, and while AMD might be possible, it wasn't well-suited.

However, as I continued my research, I found information suggesting that AMD GPUs have become quite capable recently. After several days of trial and error, I succeeded in building a practical local LLM environment using WSL2 + ROCm + vLLM.


Can the RX 7900 XTX compete?

The AMD Radeon RX 7900 XTX (24GB) has become a very competitive GPU as of February 2026, thanks to the maturation of ROCm and official support from vLLM.

Here is a rough comparison of GPUs (as of February 2026).

GPU VRAM Price Range (Approx.) Price per GB (Approx.) LLM Evaluation
RX 7900 XTX 24GB ~130,000 JPY ~5,400 JPY/GB ★★★★★
RTX 4090 24GB ~280,000 JPY ~11,600 JPY/GB ★★★★☆
RX 7900 XT 20GB ~100,000 JPY ~5,000 JPY/GB ★★★★☆
RTX 4080 SUPER 16GB ~180,000 JPY ~11,200 JPY/GB ★★★☆☆

Strengths of the RX 7900 XTX:

  • Equipped with 24GB of high-capacity VRAM
  • Cost performance at less than half the price of an RTX 4090
  • ROCm + vLLM runs properly (though with some constraints)
  • Officially supported by vLLM as a "first-class platform" (announced in January 2026)

Results after building:

  • Qwen2.5-7B-Instruct: 272 tok/s
  • Mistral-7B-Instruct: 207 tok/s
  • At these speeds, it is more than practical (though this may not apply to larger models).

In conclusion, my personal experience after days of trial and error is that "AMD GPUs are definitely competitive."

Of course, there are limitations compared to NVIDIA (e.g., FP8 KV cache quantization is not available). However, if you prioritize cost-performance for home lab use, the RX 7900 XTX is a very strong option.


Environment Information

Hardware Configuration (Author's PC)

  • CPU: Intel Core i7-14700KF (20 cores, 28 threads)
  • Memory: DDR4 64GB
  • GPU: AMD Radeon RX7900XTX (24GB VRAM)
  • Storage: NVMe SSD 2TB
  • Motherboard: ASUS TUF GAMING Z790-PLUS WIFI D4
  • Power Supply: 1200W (80PLUS Gold)

Software Configuration

  • OS (Host): Windows 11 25H2
  • AMD GPU Driver: AMD Software Adrenalin Edition 26.1.1
  • WSL2: Ubuntu 24.04 LTS
    • Allocated Memory: 48GB
    • Allocated Processors: 16 cores
    • Swap: 16GB
  • ROCm: 7.2
  • PyTorch: 2.9.1+rocm7.2.0
  • vLLM: v0.15.0
  • Python: 3.12

.wslconfig Settings

Windows side C:\Users\<Username>\.wslconfig:

[wsl2]
memory=48GB
processors=16
swap=16GB
  • After configuring, run wsl --shutdown in PowerShell and restart WSL.
Item Minimum Recommended
CPU Ryzen 5 / Core i5 (6+ cores) Ryzen 7 / Core i7 (8+ cores)
Memory 32GB (16GB allocated to WSL2) 64GB (32GB+ allocated to WSL2)
GPU RX 7900 XT (20GB VRAM) RX 7900 XTX (24GB VRAM)
SSD NVMe 500GB NVMe 1TB or more
PSU 750W or more 800-850W or more

1. Setting up WSL2 and Ubuntu 24.04

1-1. Enabling WSL2

In PowerShell (Admin):

wsl --install

After restarting, verify the version.

wsl --version

1-2. WSL2 Resource Allocation (.wslconfig)

Create the .wslconfig mentioned above.
Use roughly 70-75% of host memory for memory and about 80% of physical cores for processors.

After creating/editing, run in PowerShell:

wsl --shutdown

1-3. Installing Ubuntu 24.04

wsl --install -d Ubuntu-24.04

Set your username/password upon the first launch.

1-4. Installing Basic Packages

Launch Ubuntu in WSL and run:

sudo apt update && sudo apt upgrade -y
sudo apt install -y \
  build-essential git wget curl \
  python3-pip python3-venv \
  ninja-build

2. Setting up AMD GPU Drivers and ROCm

2-1. AMD Drivers on Windows

  • Install AMD Software: Adrenalin Edition 26.1.1.
  • If you already have a different version installed, it is safer to update to 26.1.1 or later.

(I will omit the detailed steps as it is simply following the official installer.)

2-2. Installing ROCm (on Ubuntu-24.04)

Since ROCm installation methods tend to change with every version, I strongly recommend checking the official instructions first.

wget https://repo.radeon.com/amdgpu-install/7.2/ubuntu/noble/amdgpu-install_7.2.70200-1_all.deb
sudo apt install ./amdgpu-install_7.2.70200-1_all.deb
sudo amdgpu-install -y --usecase=wsl,rocm --no-dkms

2-3. Verifying GPU Recognition (on Ubuntu-24.04)

rocm-smi does not work in WSL, so use rocminfo to verify.

rocminfo | grep "Name:"

If gfx1100 is included, the RX 7900 XTX is correctly recognized.


3. Installing ROCm-supported PyTorch

3-1. Creating a Python Virtual Environment

Create a venv directly under your home directory.

python3 -m venv ~/ai-ws
cd ~/ai-ws
source ~/ai-ws/bin/activate

You can name the directory ai-ws or anything you prefer.

3-2. Installing PyTorch (ROCm build)

pip3 install --upgrade pip wheel

wget https://repo.radeon.com/rocm/manylinux/rocm-rel-7.2/torch-2.9.1%2Brocm7.2.0.lw.git7e1940d4-cp312-cp312-linux_x86_64.whl
wget https://repo.radeon.com/rocm/manylinux/rocm-rel-7.2/torchvision-0.24.0%2Brocm7.2.0.gitb919bd0c-cp312-cp312-linux_x86_64.whl
wget https://repo.radeon.com/rocm/manylinux/rocm-rel-7.2/triton-3.5.1%2Brocm7.2.0.gita272dfa8-cp312-cp312-linux_x86_64.whl
wget https://repo.radeon.com/rocm/manylinux/rocm-rel-7.2/torchaudio-2.9.0%2Brocm7.2.0.gite3c6ee2b-cp312-cp312-linux_x86_64.whl
pip3 uninstall torch torchvision triton torchaudio
pip3 install torch-2.9.1+rocm7.2.0.lw.git7e1940d4-cp312-cp312-linux_x86_64.whl torchvision-0.24.0+rocm7.2.0.gitb919bd0c-cp312-cp312-linux_x86_64.whl torchaudio-2.9.0+rocm7.2.0.gite3c6ee2b-cp312-cp312-linux_x86_64.whl triton-3.5.1+rocm7.2.0.gita272dfa8-cp312-cp312-linux_x86_64.whl

3-3. Verification

python - << 'PY'
import torch
print("CUDA available:", torch.cuda.is_available())
if torch.cuda.is_available():
    print("Device count:", torch.cuda.device_count())
    print("Device name:", torch.cuda.get_device_name(0))
PY

If it looks like this, you are good to go:

CUDA available: True
Device count: 1
Device name: AMD Radeon RX 7900 XTX

4. Installing vLLM

4-1. Selecting the vLLM Version

At the time of writing (February 2026), the latest version of vLLM is v0.16.0, but v0.15.0 is the stable version, so we will use v0.15.0 in this guide. Here is a summary of the characteristics of each.

Features of v0.15.0

  • No MLA support
  • Flash Attention is treated as an optional feature
  • Simple and stable configuration

Features of v0.16.0

  • Added MLA support (for models like DeepSeek v3)
  • Changed to require Flash Attention (this can be bypassed by setting the environment variable BUILD_FA=0)
  • Standard attention models (Qwen2.5, Mistral, etc.) work without issues

Notes for RX 7900 XTX

The RX 7900 XTX (RDNA3 / gfx1100) does not support Flash Attention.

When using v0.16.0, please note the following:

  1. By default, Flash Attention is required.
  2. You can build without Flash Attention by setting the BUILD_FA=0 environment variable.
  3. Triton Attention will be used, so there is no issue with performance.
  4. Standard attention models will function normally.
  5. MLA models (GLM-4.7-Flash, DeepSeek-V2/V3, etc.) cannot be used.

Conclusion

  • If you only use standard attention models: Either v0.15.0 or v0.16.0 is fine.
  • For simplicity: v0.15.0 is recommended.
  • To try out the latest features: Use v0.16.0 and append BUILD_FA=0 to the environment variables.

4-2. Cloning the Repository

cd ~/ai-ws
git clone https://github.com/vllm-project/vllm.git
cd vllm
git fetch --tags
git checkout v0.15.0

We specify the version mentioned above using git checkout v0.15.0.

4-3. Applying use_existing_torch.py (Important)

vLLM provides a script to build based on an already installed PyTorch.

If you do not apply this, the following will occur:

  • The default vLLM build will attempt to automatically install the CUDA version of PyTorch as a dependency.
  • The ROCm version of PyTorch (2.9.1+rocm7.2.0) that you carefully installed will be overwritten.
  • As a result, the GPU will no longer be recognized.

Therefore, do not forget to apply this.

python use_existing_torch.py

4-4. Installing Dependencies

pip install -r requirements/build.txt
pip install -r requirements/rocm.txt
pip install ninja cmake wheel packaging pybind11

4-5. Setting ROCm Environment Variables

export PYTORCH_ROCM_ARCH="gfx1100"
export HSA_OVERRIDE_GFX_VERSION=11.0.0

If you are using v0.16.0, add the following as well:

export BUILD_FA=0  # Build without Flash Attention

4-6. Building vLLM

pip install -e . --no-build-isolation

It takes about 10 minutes to complete, so please be patient.

4-7. Verification

python - << 'PY'
import vllm
print("vLLM version:", vllm.__version__)
PY

If a string like 0.15.0+... appears, you are successful.


5. [Important] WSL2-Specific Issues and Fixes

This is a part where I struggled quite a bit in my previous scraps. vLLM will not start unless you apply this fix.

5-1. ROCm Platform Not Detected

When you try to start vLLM on WSL2, you will get the following error:

RuntimeError: Device string must not be empty

What is happening?

At first glance, it looks like an error stating the "device string is empty," but the actual cause is further upstream.

vLLM checks whether ROCm can be used at startup using a tool called amdsmi (AMD System Management Interface).
However, since amdsmi does not function in WSL2, the detection of the ROCm platform fails.

Looking at the logs in detail:

ROCm platform is not available because: Error code: 34

As a result:

  1. ROCm platform detection fails.
  2. The GPU is not recognized.
  3. The device string becomes empty.
  4. RuntimeError: Device string must not be empty occurs.

5-2. Modifying the Code

We will modify the code to detect ROCm using PyTorch instead of amdsmi.

File to modify:
~/ai-ws/vllm/vllm/platforms/__init__.py

nano ~/ai-ws/vllm/vllm/platforms/__init__.py

Modification:
Rewrite the rocm_platform_plugin() function as follows:

def rocm_platform_plugin() -> str | None:
    is_rocm = False
    logger.debug("Checking if ROCm platform is available.")
    try:
        # Use PyTorch instead of amdsmi
        import torch
        if hasattr(torch.version, 'hip') and torch.version.hip is not None:
            is_rocm = torch.cuda.is_available() and torch.cuda.device_count() > 0
            if is_rocm:
                logger.debug("Confirmed ROCm platform is available via torch.version.hip.")
            else:
                logger.debug("ROCm platform is not available because no GPU is found.")
        else:
            logger.debug("ROCm platform is not available because torch.version.hip is not found.")
    except Exception as e:
        logger.debug("ROCm platform is not available because: %s", str(e))

    return "vllm.platforms.rocm.RocmPlatform" if is_rocm else None
Original Code (Reference)
def rocm_platform_plugin() -> str | None:
    is_rocm = False
    logger.debug("Checking if ROCm platform is available.")
    try:
        import amdsmi  # ← This does not work in WSL2
        amdsmi.amdsmi_init()  # ← Fails with Error code: 34
        try:
            if len(amdsmi.amdsmi_get_processor_handles()) > 0:
                is_rocm = True
                logger.debug("Confirmed ROCm platform is available.")
            else:
                logger.debug("ROCm platform is not available because no GPU is found.")
        finally:
            amdsmi.amdsmi_shut_down()
    except Exception as e:
        logger.debug("ROCm platform is not available because: %s", str(e))
    
    return "vllm.platforms.rocm.RocmPlatform" if is_rocm else None
Item Before Modification After Modification
Detection Method amdsmi torch.version.hip
GPU Check amdsmi.amdsmi_get_processor_handles() torch.cuda.is_available()
WSL2 Support ❌ Does not work ✅ Works

Why does it work with PyTorch?

WSL2 Constraints:
Because it cannot manage the GPU directly like native Linux, AMD management tools like rocm-smi and amdsmi cannot be used. Access must go through the Windows host.

PyTorch supports WSL2:

  1. It can retrieve ROCm version information via torch.version.hip.
  2. It can confirm the existence of the GPU via torch.cuda.is_available().
  3. It possesses mechanisms to function correctly even through the Windows host.

5-3. Verification

After the modification, check with the following command:

python -c "from vllm.platforms import rocm_platform_plugin; print(rocm_platform_plugin())"

Expected output:

/home/username/ai-ws/lib/python3.12/site-packages/torch/cuda/__init__.py:814: UserWarning: Can't initialize amdsmi - Error code: 34
  warnings.warn(f"Can't initialize amdsmi - Error code: {e.err_code}")
vllm.platforms.rocm.RocmPlatform

If this is displayed, ROCm platform detection has succeeded. You may see the amdsmi error in the first two lines, but that is not an issue. We can finally run vLLM on WSL2.


6. Persisting Environment Variables (.bashrc)

Running export every time is cumbersome, so let's add them to .bashrc.

nano ~/.bashrc

Add the following to the end:

# ROCm Environment
export PYTORCH_ROCM_ARCH="gfx1100"
export HSA_OVERRIDE_GFX_VERSION=11.0.0
export BUILD_FA=0
export ROCM_LLVM_ENABLE_WAVE_PROFILER=0

# Optional: Automatically activate venv if working only in this environment
source ~/ai-ws/bin/activate

Apply the changes:

source ~/.bashrc

7. Basic vLLM Functionality Verification

Now, let's create a test program and try calling vLLM directly from Python to test it.

nano test_vllm.py
# test_vllm.py
from vllm import LLM, SamplingParams

if __name__ == '__main__':
    # Test operation with a lightweight model
    llm = LLM(model="facebook/opt-125m")
    
    # Simple prompt
    prompts = ["Hello, my name is", "The future of AI is"]
    
    # Generation parameters
    sampling_params = SamplingParams(temperature=0.8, max_tokens=50)
    
    # Run inference
    outputs = llm.generate(prompts, sampling_params)
    
    # Display results
    for output in outputs:
        prompt = output.prompt
        generated = output.outputs[0].text
        print(f"Prompt: {prompt}")
        print(f"Output: {generated}")
        print("-" * 80)

Run it:

python test_vllm.py

If the output is displayed normally, you have succeeded.

I plan to write a separate article on how to call it as an API from OpenWebUI.


8. Benchmark Results and Practicality

Since we've come this far, I decided to run some benchmarks.

8-1. Measurement Conditions

  • GPU: AMD Radeon RX 7900 XTX (24GB)
  • ROCm: 7.2
  • vLLM: v0.15.0
  • PyTorch: 2.9.1+rocm7.2.0
  • WSL2: Ubuntu 24.04
  • Test Settings:
    • Number of prompts: 10
    • Max generation tokens: 512 / output
    • Context length: 4096
    • gpu_memory_utilization=0.9

8-2. Benchmark Script Used

nano benchmark_simple.py
# benchmark_simple.py
import time
from vllm import LLM, SamplingParams

def benchmark_model(model_name, num_prompts=10, max_tokens=512):
    print("\n" + "="*60)
    print(f"Model: {model_name}")
    print("="*60)

    print("Loading model...")
    t0 = time.time()
    llm = LLM(
        model=model_name,
        trust_remote_code=True,
        gpu_memory_utilization=0.9,
        max_model_len=4096,
    )
    load_time = time.time() - t0
    print(f"Load time: {load_time:.2f}s")

    prompts = [
        "Write a short story about artificial intelligence.",
        "Explain quantum computing in simple terms.",
        "What are the benefits of exercise?",
        "Describe the process of photosynthesis.",
        "Write a poem about nature.",
    ] * (num_prompts // 5)

    sampling_params = SamplingParams(
        temperature=0.8,
        top_p=0.95,
        max_tokens=max_tokens,
    )

    # Warmup
    _ = llm.generate(prompts[:1], sampling_params)

    print(f"\nRunning benchmark ({num_prompts} prompts, max {max_tokens} tokens)...")
    t1 = time.time()
    outputs = llm.generate(prompts, sampling_params)
    elapsed = time.time() - t1

    total_tokens = sum(len(output.outputs[0].token_ids) for output in outputs)
    avg_tokens = total_tokens / len(outputs)
    tps = total_tokens / elapsed

    print("\n" + "="*60)
    print("Results:")
    print(f"  Total time: {elapsed:.2f}s")
    print(f"  Total tokens generated: {total_tokens}")
    print(f"  Average tokens per output: {avg_tokens:.1f}")
    print(f"  Throughput: {tps:.2f} tokens/s")
    print("="*60)

    return {
        "model": model_name,
        "load_time": load_time,
        "total_time": elapsed,
        "total_tokens": total_tokens,
        "tps": tps,
    }

if __name__ == "__main__":
    models = [
        "Qwen/Qwen2.5-7B-Instruct",
        "mistralai/Mistral-7B-Instruct-v0.3",
    ]
    for m in models:
        benchmark_model(m, num_prompts=10, max_tokens=512)
        input("\nPress Enter to continue...")

Run it:

python benchmark_simple.py

8-3. Results

Qwen/Qwen2.5-7B-Instruct

  • VRAM during model loading: 14.5 GB
  • VRAM during inference (max): 21.9 GB
  • Average generated tokens: approx. 453 tokens
  • Throughput: 272.34 tokens/s

mistralai/Mistral-7B-Instruct-v0.3

  • VRAM during model loading: 13.8 GB
  • VRAM during inference (max): 21.2 GB
  • Average generated tokens: approx. 401 tokens
  • Throughput: 206.69 tokens/s

8-4. Personal Thoughts

  • Qwen2.5-7B has strong Japanese language capabilities and, at over 270 tok/s, is almost zero-wait in terms of user experience.
  • Mistral-7B is also very practical at over 200 tok/s.
  • VRAM usage during inference is around 21–22GB, leaving 2–3GB of headroom even on 24GB cards.

With this headroom, you can adjust:

  • Increasing context length
  • Slightly increasing batch size

8-5. Model Size Estimates for 24GB VRAM

Rough estimates:

Quantization Model Size Recommendation Note
FP16 7B ◎ Plenty of room (the configuration in this article)
FP16 10B △ Barely fits. Assumes shorter context
AWQ 32B ○ Practical with some tuning
AWQ 70B × Basically difficult on 24GB VRAM

Conclusion and Next Steps

In this article, we accomplished the following:

  • Built a local LLM environment using RX 7900 XTX + WSL2 + ROCm + vLLM
  • Verified performance of 200–270 tok/s with Qwen2.5-7B / Mistral-7B
  • Worked around the WSL2-specific amdsmi issue

Things I am considering as next steps:

  1. vLLM + OpenWebUI Integration (upcoming article)
    • Launch OpenWebUI from Docker on WSL
    • Connect to vLLM's OpenAI-compatible endpoint
    • Use local LLMs with a ChatGPT-like UI
  2. Running 30B class models using quantized models (Article already written!)
    • Running models like Qwen3-Coder-30B-AWQ on the RX 7900 XTX
  3. Summary of considerations for using MoE models (Article already written!)
    • Discussing why you need to modify some vLLM executables when using MoE models
  4. Doubling context length by FP8 quantization of the KV cache (Article already written!)
    • ROCm FP8 quantization is now available in vLLM v0.16.0

Thank you for reading this far.
I hope this serves as a helpful reference for anyone else looking to run vLLM on an RX 7900 XTX + WSL2 setup.

Please let me know in the comments if you have any questions or feedback.

Discussion