A Complete Guide to KAN: A New Architecture Beyond MLP - From the Kolmogorov-Arnold Representation Theorem to Implementation

What is this article?

"The 60-year dominance of MLPs might be coming to an end."

Announced in April 2024, KAN (Kolmogorov-Arnold Networks) is an innovative architecture that changes the fundamental structure of neural networks.

I previously wrote an article titled TabPFN v2.5 Complete Guide. While TabPFN represented an "innovation in the learning process," KAN represents an "innovation in the neural network structure itself."

In this article, I will explain KAN, from its mathematical foundation, the Kolmogorov-Arnold representation theorem, to its actual usage on Kaggle.

Target Audience

• Those who know the basics of MLP (Multi-Layer Perceptron)
• Those who want to theoretically understand why KAN is revolutionary
• Those looking for a new weapon for Kaggle tabular data competitions

Prerequisites: A level where you can use PyTorch / scikit-learn

---

Chapter 1: Why KAN now?

60 Years of MLP

Since the invention of the perceptron in 1958, the basic structure of neural networks has not changed:

Input → [Linear Transformation → Activation Function] × N layers → Output

MLPs (Multi-Layer Perceptron) feature:

• Fixed activation functions (ReLU, sigmoid, etc.) at nodes (neurons)
• Trainable linear parameters at edges (weights)

This structure is based on the Universal Approximation Theorem, which guarantees that "any function can be approximated with sufficient width and depth."

KAN's Paradigm Shift

KAN reverses this 60-year-old common knowledge:

Feature	MLP	KAN
Activation Location	Nodes (Fixed)	Edges (Trainable)
Linear Weights	Yes	No
Theoretical Foundation	Universal Approximation Theorem	Kolmogorov-Arnold Representation Theorem

# MLP calculation
output = activation(W @ input + b)  # Activation function is fixed

# KAN calculation (conceptual)
output = sum([phi_i(input_i) for i in range(n)])  # Activation function is trainable

---

Chapter 2: The Kolmogorov-Arnold Representation Theorem

Hilbert's 13th Problem

In 1900, mathematician David Hilbert presented 23 unsolved problems at the International Congress of Mathematicians in Paris.

The 13th problem asked:

"Can the solutions of 7th-degree algebraic equations be represented by finite compositions of addition, subtraction, multiplication, division, and roots, or is it possible to represent them via compositions of two-variable functions?"

More generally, the question was: "Can multi-variable functions be expressed solely by the composition of single-variable functions?"

The Theorem (1957)

Between 1956 and 1957, Andrey Kolmogorov and his student Vladimir Arnold solved this problem.

Kolmogorov-Arnold Representation Theorem

Any continuous multi-variable function $f: [0,1]^n \to \mathbb{R}$ can be expressed in the following form:

$$f(x_1, x_2, ..., x_n) = \sum_{q=0}^{2n} \Phi_q \left( \sum_{p=1}^{n} \phi_{q,p}(x_p) \right)$$

Where:

• $\phi_{q,p}: [0,1] \to \mathbb{R}$ are internal functions (single-variable)
• $\Phi_q: \mathbb{R} \to \mathbb{R}$ are external functions (single-variable)

Implications of the Theorem

This theorem demonstrates something quite astonishing:

"The only true multi-variable operation is 'addition.' Everything else can be represented by the composition of single-variable functions."

For example, even multiplication $f(x, y) = x \cdot y$ can be expressed using only single-variable functions and addition.

f(x, y) = x × y
       = Φ(φ₁(x) + φ₂(y))   # Composition of single-variable functions

Why was this not used until now?

Although this theorem was proven in 1957, it was not utilized in machine learning for a long time due to several reasons:

1. Pathological nature of internal functions: $\phi_{q,p}$ could become extremely "rough" functions.
2. Numerical instability: Direct implementation was difficult.
3. Computational efficiency: It was not practical with computers at the time.

The KAN paper resolved these issues using spline functions and deep learning optimization techniques.

---

Chapter 3: KAN Architecture

Visualizing MLP vs. KAN

MLP (Conventional)

Input Layer      Hidden Layer       Output Layer
 x₁ ───W₁₁──→ [σ] ───W'₁──→
     ╲       ╱              ╲
      W₁₂  W₂₁              W'₂
       ╲  ╱                   ╲
 x₂ ───W₂₂──→ [σ] ───W'₃──→  out

[σ] = Fixed activation function (ReLU, etc.)
W   = Learnable linear weights

KAN (New)

Input Layer      Hidden Layer       Output Layer
 x₁ ───[φ₁₁]──→ Σ ───[Φ₁]──→
     ╲         ╱              ╲
      [φ₁₂] [φ₂₁]            [Φ₂]
        ╲   ╱                   ╲
 x₂ ───[φ₂₂]──→ Σ ───[Φ₃]──→  out

[φ] = Learnable activation function (spline)
Σ   = Simple addition

Key differences:

• MLPs use "fixed activations + learnable weights."
• KANs use "learnable activations + fixed addition."

Parameterization via Splines

KAN activation functions are parameterized by B-splines:

$$\phi(x) = \sum_{i} c_i B_i(x)$$

Where:

• $B_i(x)$ are B-spline basis functions (fixed)
• $c_i$ are spline coefficients (learnable)

Why splines?

1. Smoothness: Continuous and differentiable.
2. Locality: Local changes do not affect the whole function.
3. Efficiency: Represent complex functions with few parameters.
4. Interpretability: The function shape can be visualized after training.

# Example of spline basis (conceptual code)
import numpy as np
from scipy.interpolate import BSpline

# Cubic B-spline
knots = np.linspace(0, 1, 10)
coefficients = np.random.randn(len(knots) + 3)  # Learnable parameters
spline = BSpline(knots, coefficients, k=3)

# Smooth output for any input
x = np.linspace(0, 1, 100)
y = spline(x)

Advantages of KANs

1. Parameter Efficiency

KANs often achieve the same accuracy as MLPs with significantly fewer parameters:

Task	MLP	KAN
Function Approximation	~300 params	~30 params
Physics Law Discovery	~10,000 params	~100 params

2. Interpretability

After training, you can visualize the activation function of each edge:

# After training KAN
model.plot()  # Plots the learned function of each edge

This makes it possible for humans to understand:

• Which input features are important.
• What kind of non-linear transformations were learned.
• Correspondences to physical laws.

3. Improved Scaling Laws

KANs achieve "better performance with smaller models":

KANs have faster neural scaling laws than MLPs.
— Liu et al., 2024

---

Chapter 4: How to Implement KAN

Challenges with Tabular Data

Tabular (structured) data is an area where deep learning traditionally struggles:

Data Type	Good at DL?	Reason
Image	◎	Local patterns, convolutions
Text	◎	Sequential structure, Attention
Tabular	△	Diverse relationships between features

Reasons why GBDTs are strong in tabular data:

• Excellent at splitting per feature.
• Natural handling of missing values.
• Robust to hyperparameters.

Why KANs are Suited for Tabular Data

KANs are suitable for tabular data due to the following reasons:

1. Transformation per feature: Learns individual univariate functions for each input feature.
2. Numerical features: Efficiently captures non-linear patterns in continuous values.
3. Interpretability: Feature importance and transformation methods can be visualized.

pykan - Official KAN Implementation

pykan is the official implementation by the KAN paper authors.

Installation

pip install pykan

Basic Usage

from kan import KAN
import torch

# Model definition
model = KAN(
    width=[n_features, 32, n_classes],  # Input → Hidden Layer → Output
    grid=5,      # Spline grid size
    k=3,         # Spline order
    seed=42,
    device="cuda"
)

# Prepare dataset (pykan format)
dataset = {
    "train_input": X_train_tensor,
    "train_label": y_train_tensor,
    "test_input": X_test_tensor,
    "test_label": y_test_tensor,
}

# Training
model.fit(
    dataset,
    opt="Adam",
    lr=0.01,
    steps=100,
    loss_fn=torch.nn.CrossEntropyLoss(),
)

# Inference
with torch.no_grad():
    predictions = model(X_test_tensor).argmax(dim=1)

Visualizing KAN

One of pykan's powerful features is the visualization of learned activation functions:

# Plot the function of each edge after training
model.plot()

This allows you to visually confirm how each feature is being transformed.

Other KAN Implementations

Library	Features
pykan	Official implementation, visualization features
efficient-kan	Faster version
TabKAN	Tabular specific, multiple variations

We use pykan for Kaggle notebooks.
I also tried TabKAN, but it currently causes import errors,
so I chose pykan from the perspective of stability.

---

Chapter 5: Benchmark Comparison

Experimental Setup

We compare the following models:

• KAN: pykan (Original implementation)
• TabPFN: Pre-trained Transformer (Detailed Article)
• GBDT family: LightGBM, XGBoost, CatBoost
• MLP: Traditional Neural Network

Datasets:

• Breast Cancer (569 samples, 30 features)
• Iris (150 samples, 4 features)
• Wine (178 samples, 13 features)
• Digits (1,797 samples, 64 features)

Detailed benchmark results are published in the Kaggle notebook:
KAN vs TabPFN vs GBDT Tabular Benchmark

About TabPFN version

We use TabPFN v1 in the Kaggle notebook for compatibility.
Since v2.5 requires HuggingFace authentication, additional configuration is needed for the Kaggle environment.

For details on TabPFN v2.5 (which is higher performance), refer to TabPFN v2.5 Complete Guide.

Summary of Results

Dataset	KAN	TabPFN	LightGBM	CatBoost	MLP
breast_cancer	97.4% 🏆	96.5%	96.5%	95.6%	94.7%
iris	-	96.7% 🏆	90.0%	93.3%	66.7%
wine	91.7%	94.4%	100% 🏆	100% 🏆	77.8%
digits	-	98.1% 🏆	96.4%	97.8%	95.8%

Note: KAN experienced errors on some datasets due to pykan library limitations.

Heatmap of Accuracy and Execution Time

Left: Accuracy (Green indicates high accuracy), Right: Execution time (Red indicates fast execution)

KAN achieved the highest accuracy on breast_cancer, but the training time is longer than GBDT. TabPFN requires no training and allows for fast inference.

Average Accuracy and Win Count by Model

Left: Average accuracy across all datasets, Right: Number of times each model took 1st place per dataset

• CatBoost: Top average accuracy (96.7%)
• TabPFN: Most wins (2 wins) - iris, digits
• KAN: 1st place in breast_cancer, average 94.5%

Analysis

Key Point

KAN took 1st place in breast_cancer! (97.4%)

• Achieved the highest accuracy among all models
• Outperformed GBDT and TabPFN
• Demonstrated strength on datasets with many numerical features

Cases where KAN is strong

• Small to medium datasets (< 1,000 samples)
• Numerical features are dominant (few categories)
• Situations where interpretability is important
• Complex non-linear relationships between features

Cases where TabPFN is strong

• Small datasets (< 3,000 samples)
• No time for tuning
• Need for a quick baseline
• Achieved highest accuracy on iris and digits

Cases where GBDT is strong

• Medium to large datasets (> 5,000 samples)
• Many categorical features
• Deployment in production environments
• 100% accuracy on wine (LightGBM, CatBoost)

---

Chapter 6: When to use what

Decision Flowchart

What is the dataset size?
    │
    ├─ < 1,000 samples
    │     │
    │     └─ Is interpretability important?
    │           ├─ Yes → KAN
    │           └─ No  → TabPFN
    │
    ├─ 1,000 to 5,000 samples
    │     │
    │     └─ How much tuning time?
    │           ├─ None → TabPFN
    │           └─ Yes  → KAN or GBDT
    │
    └─ > 5,000 samples
          │
          └─ GBDT (LightGBM / XGBoost / CatBoost)

Comparison Table

Perspective	KAN	TabPFN	GBDT
Optimal data size	< 3,000	< 3,000	Arbitrary
Training time	Medium	None	Medium to Long
Interpretability	◎	×	△
Categorical features	△	○	◎
Production deployment	△	○	◎
Tuning requirement	Medium	None	High

My Recommendation

1. First, create a baseline with TabPFN (30 seconds)
2. Explore interpretability with KAN (feature importance, non-linear patterns)
3. Build the final model with GBDT (especially for large-scale data)

---

Chapter 7: Summary

Innovation of KAN

1. Theoretical foundation: Realizing the 1957 Kolmogorov-Arnold representation theorem with modern deep learning.
2. Architectural reversal: From "node activation" to "edge activation".
3. Parameter efficiency: Equivalent performance with fewer parameters.
4. Interpretability: Ability to visualize the learned functions.

2025 Tabular ML Landscape

                    Interpretability
                      │
                 KAN  │
                  *   │
                      │
Small data ───────────┼─────────── Large data
                      │
           TabPFN *   │   * GBDT
                      │
                 Speed/Scalability

Next Steps

• Kaggle notebook: KAN vs TabPFN vs GBDT Tabular Benchmark
• TabPFN details: TabPFN v2.5 Complete Guide
• pykan official documentation: kindxiaoming.github.io/pykan

---

References

[1] Liu, Z., et al. (2024). KAN: Kolmogorov-Arnold Networks. ICLR 2025. arXiv:2404.19756

[2] Eslamian, A., et al. (2025). TabKAN: Advancing Tabular Data Analysis using Kolmogorov-Arnold Network. arXiv:2504.06559

[3] Chen, J., et al. (2024). TabKANet: Tabular Data Modeling with KAN and Transformer. arXiv:2409.08806

[4] Kolmogorov, A. N. (1957). On the representation of continuous functions of many variables by superposition of continuous functions of one variable and addition.

[5] Arnold, V. I. (1957). On functions of three variables. Dokl. Akad. Nauk SSSR 114.

[6] Hollmann, N., et al. (2023). TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second. ICLR 2023.

---

If you found this article helpful, please like and follow!

If you have any questions or feedback, please let me know in the comments.