「Glyph: 視覚-テキスト圧縮によるスケーリングコンテキストウィンドウ」を試す

!pip show transformers

Name: transformers
Version: 4.57.1
Summary: State-of-the-art Machine Learning for JAX, PyTorch and TensorFlow
Home-page: https://github.com/huggingface/transformers
Author: The Hugging Face team (past and future) with the help of all our contributors (https://github.com/huggingface/transformers/graphs/contributors)
Author-email: transformers@huggingface.co
License: Apache 2.0 License
Location: /usr/local/lib/python3.12/dist-packages
Requires: filelock, huggingface-hub, numpy, packaging, pyyaml, regex, requests, safetensors, tokenizers, tqdm
Required-by: peft, sentence-transformers

from transformers import AutoProcessor, AutoModelForImageTextToText
import torch

processor = AutoProcessor.from_pretrained("zai-org/Glyph")
model = AutoModelForImageTextToText.from_pretrained(
    pretrained_model_name_or_path="zai-org/Glyph",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

Wed Oct 29 11:00:40 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.15              Driver Version: 550.54.15      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA L4                      Off |   00000000:00:03.0 Off |                    0 |
| N/A   61C    P0             30W /   72W |   19869MiB /  23034MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

from IPython.display import Image

Image("https://raw.githubusercontent.com/thu-coai/Glyph/main/assets/Little_Red_Riding_Hood.png")

%%time

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "url": "https://raw.githubusercontent.com/thu-coai/Glyph/main/assets/Little_Red_Riding_Hood.png"
            },
            {
                "type": "text",
                "text": "Who pretended to be Little Red Riding Hood's grandmother"
            }
        ],
    }
]

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
    return_tensors="pt"
).to(model.device)
generated_ids = model.generate(**inputs, max_new_tokens=8192)
output_text = processor.decode(generated_ids[0][inputs["input_ids"].shape[1]:], skip_special_tokens=False)
print(output_text)

<think>Got it, let's see. The question is who pretended to be Little Red Riding Hood's grandmother. From the story, the wolf dressed up like Grandma and let her in. So the wolf is the one pretending.</think>The wolf<|user|>
CPU times: user 5.15 s, sys: 327 ms, total: 5.47 s
Wall time: 6.33 s

%%time

text = '''Little Red Riding Hood
Once upon a time, in the middle of a dense forest, there was a small house, where lived a pretty little girl named Little Red Riding Hood. One sunny day, her mother called her into the kitchen of their little house.
"Grandma is sick" she said. "Take her this basket of cakes, but be very careful! Stay on the path through the forest and don't stop! If you walk quickly, no harm will come to you."
"Don't worry", she said. "I will run to Grandma's without stopping." Little Red Riding Hood kissed her mother and left. The little girl started her long trip through the forest.
...But she soon forgot the wise words of her mother.
She saw a patch of strawberries under a tree. "What lovely strawberries!" she exclaimed, laying her basket on the ground. "They are so nice and ripe... and so big! Delicious! I'll maybe just have one more..."
Suddenly Little Red Riding Hood remembered her mother, her promise, her grandmother, and the basket... She hurried back towards the path.
The wood became thicker and thicker. A yellow butterfly flew down through the trees. "Oh, how pretty!" she exclaimed, and she chased the butterfly through the trees. "I'll catch you!"
Then she saw some large daisies in the grass. "Oh, how beautiful!" she exclaimed. "Grandma will love them!" And she picked a huge bunch of flowers.
Meanwhile, two menacing eyes watched her from behind a tree. A strange noise in the woods made Little Red Riding Hood jump with fright, and her heart started to thump. "I must find the path and run away from here, quickly!"
Little Red Riding Hood ran and ran, and finally found the path. But as soon as she started to relax she heard another strange noise behind her...
... A very big voice which said: "Hello, little girl. Why are you walking in the forest all alone? Where are you going?"
"I'm taking some cakes to my grandma. She lives at the end of this path," said Little Riding Hood in a very small voice.
"Does your grandma live alone?" asked the wolf (for it was the Big Bad Wolf she was talking to!)
"Oh, yes," replied Little Red Riding Hood, "and she never opens the door to strangers!"
"You have a very clever grandma", answered the wolf. "Well, it was very nice talking to you. Maybe we'll meet again!"
The wolf left, thinking "Mmmmm, I will first eat the grandma, and then I will wait for the little girl to come."
The wolf arrived at the end of the path, where the little house stood. Knock! Knock! The wolf knocked on the door. 
"Who's there?" called Grandma from her bed.
"It's me, Little Red Riding Hood. I have brought you some cakes because you are sick," answered the wolf, trying his best to hide his big wolf voice.
"Oh, that's marvellous!" said Grandma. She didn't notice anything strange. "Come in!"
Poor Grandma! In less than two seconds the wolf had jumped across the room and swallowed the old lady. Gulp! Then, the wolf put on grandma's dress and hat, and slid into the bed.
Soon, Little Red Riding Hood knocked on the door. "Grandma, it's me. May I come in?"
The wolf tried to imitate Grandma's small voice, and answered "Hello my dear! Come in!"
"What a big voice you have, Grandma!" said the little girl, in surprise.
"It's to greet you better, my dear," said the wolf.
"And what big eyes you have, Grandma..."
"It's to see you better, my dear!" 
"And what big hands you have!" exclaimed Little Red Riding Hood, approaching the bed. "It's to hug you better, my dear!... And what a big mouth you have," murmured the little girl in a feeble voice. "It's better to EAT you with!" growled the wolf, and he leapt out of the bed and swallowed her as well.Then with a full stomach he fell asleep. 
At that moment a hunter emerged from the forest. He saw the house and decided to stop and ask for a glass of water. He was looking for a big wolf who had been terrorising the village. The hunter heard a strange whistling inside the house. He looked through the window and saw the big wolf snoring on Grandma's bed. "The wolf! He won't escape me this time." 
The hunter opened the stomach of the wolf and, to his surprise, out popped Grandma and Little Red Riding Hood, safe and sound. "Oh, thank you!" said the old lady. 
"It's safe to go home now", said the hunter to Little Red Riding Hood. "The big bad wolf is dead, there is no more danger on the path." 
Little Red Riding Hood's mother arrived at Grandma's house when the sun was setting. She had been worried when her daughter hadn't returned home. When she saw Little Red Riding Hood she burst into tears of joy. "Oh, you're safe!"
After having thanked the hunter again, Little Red Riding Hood and her mother walked back home through the forest.
As they were walking quickly under the trees, the little girl said to her mother: "We must always stay on the path and never stop. If we do that, we will stay safe."'''

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "url": text
            },
            {
                "type": "text",
                "text": "Who pretended to be Little Red Riding Hood's grandmother"
            }
        ],
    }
]

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
    return_tensors="pt"
).to(model.device)
generated_ids = model.generate(**inputs, max_new_tokens=8192)
output_text = processor.decode(generated_ids[0][inputs["input_ids"].shape[1]:], skip_special_tokens=False)
print(output_text)

<think>Got it, let's see. The question is who pretended to be Little Red Riding Hood's grandmother. First, I need to remember the story. Oh right, in the classic fairy tale, there's a wolf that eats the grandmother and then pretends to be her. So the wolf is the one. Let me confirm: the wolf lures Little Red Riding Hood in, then eats the grandmother, and when she comes, the wolf acts like the grandma. Yeah, that's it. So the answer should be the wolf.</think>The wolf.<|user|>
CPU times: user 8.98 s, sys: 208 ms, total: 9.18 s
Wall time: 9.81 s

%%time

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "Who pretended to be Little Red Riding Hood's grandmother"
            }
        ],
    }
]

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
    return_tensors="pt"
).to(model.device)
generated_ids = model.generate(**inputs, max_new_tokens=8192)
output_text = processor.decode(generated_ids[0][inputs["input_ids"].shape[1]:], skip_special_tokens=False)
print(output_text)

<think>Got it, let's see. The question is who pretended to be Little Red Riding Hood's grandmother. First, I need to remember the story. Oh right, in the classic fairy tale, there's a wolf that eats the grandmother and then pretends to be her. So the wolf is the one. Let me confirm: the wolf lures Little Red Riding Hood in, then eats the grandmother, and when she comes, the wolf acts like the grandma. Yeah, that's it. So the answer should be the wolf.</think>The wolf.<|user|>
CPU times: user 9.06 s, sys: 200 ms, total: 9.27 s
Wall time: 9.26 s

git clone https://github.com/thu-coai/Glyph && cd Glyph

cd demo

git lfs install
git clone https://huggingface.co/zai-org/Glyph

uv venv -p 3.12 --seed

uv pip install vllm --torch-backend=auto
uv pip install gradio

(snip)
 + vllm==0.11.0
(snip)

 + gradio==5.49.1
 + gradio-client==1.13.3

uv run vllm serve ./Glyph \
    --host 0.0.0.0 \
     --port 5002 \
    --served-model-name glyph \
    --allowed-local-media-path / \
    --media-io-kwargs '{"video": {"num_frames": -1}}'

(EngineCore_DP0 pid=2003802) torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 352.00 MiB. GPU 0 has a total capacity of 23.49 GiB of which 339.81 MiB is free. Including non-PyTorch memory, this process has 22.93 GiB memory in use. Of the allocated memory 22.01 GiB is allocated by PyTorch, and 449.72 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

!wget https://www.city.kobe.lg.jp/documents/15123/r5_doukou.pdf

!apt update && apt install -y poppler-utils

!pip install pdf2image

import os
from pdf2image import convert_from_path
from PIL import Image

kobe_pdf = "r5_doukou.pdf"
kobe_output_dir = "kobe"

def convert_pdf_to_image(pdf_path, output_dir_path, max_dim=1024):
    os.makedirs(output_dir_path, exist_ok=True)

    images = convert_from_path(pdf_path)
    for i, image in enumerate(images):
        # Resize the image
        width, height = image.size
        if max(width, height) > max_dim:
            if width > height:
                new_width = max_dim
                new_height = int(height * (max_dim / width))
            else:
                new_height = max_dim
                new_width = int(width * (max_dim / height))
            image = image.resize((new_width, new_height), Image.LANCZOS)

        output_path = f"{output_dir_path}/page_{i + 1}.png"
        image.save(output_path, "PNG")
        print(f"Saved: {output_path}")

convert_pdf_to_image(kobe_pdf, kobe_output_dir)```

こんな感じで保存される。

```text:出力
Saved: kobe/page_1.png
Saved: kobe/page_2.png
Saved: kobe/page_3.png
Saved: kobe/page_4.png
Saved: kobe/page_5.png
Saved: kobe/page_6.png
Saved: kobe/page_7.png
Saved: kobe/page_8.png
Saved: kobe/page_9.png
Saved: kobe/page_10.png
Saved: kobe/page_11.png
Saved: kobe/page_12.png
Saved: kobe/page_13.png
Saved: kobe/page_14.png
Saved: kobe/page_15.png
Saved: kobe/page_16.png
Saved: kobe/page_17.png
Saved: kobe/page_18.png
Saved: kobe/page_19.png
Saved: kobe/page_20.png
Saved: kobe/page_21.png

%%time
from PIL import Image

image_path = "./kobe/page_2.png"
image = Image.open(image_path)

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": image
            },
            {
                "type": "text",
                "text": "来神客が利用している交通機関をまとめて。回答は日本語で。"
            }
        ],
    }
]

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
    return_tensors="pt"
).to(model.device)
generated_ids = model.generate(**inputs, max_new_tokens=8192)
output_text = processor.decode(generated_ids[0][inputs["input_ids"].shape[1]:], skip_special_tokens=False)
print(output_text)

<think>
嗯，用户要求总结来神客（神户观光游客）使用的交通方式。我需要仔细阅读提供的分析报告，从中提取关于交通方式的部分。

首先看"利用交通機関"这个部分。这里提到神户市内主要使用"車・バイク"（汽车/摩托车）和"鉄道"（铁路），其中汽车占比最高约3割（30%），铁路约2割（20%）。具体到神户市内，汽车占比更高，约3割，而铁路约1割（10%）。

再看地区差异：北野地区汽车使用比例最高（约2割），其他地区较低；而铁路方面，西神地区使用比例最高（约1割），其他地区较低。

另外还提到"有馬"地区汽车使用比例非常高（约9割），而六甲摩耶地区汽车使用比例最低（约4割），铁路使用比例最高（约1割）。

综合这些信息，我可以总结出神户观光游客主要使用的交通方式：汽车/摩托车是绝对主力，尤其在有马地区占比高达90%；铁路使用比例相对较低，但西神地区使用比例较高；其他交通方式如飞机、公交等未在报告中提及。

需要特别注意数据单位：报告中的比例都是"割"（juku），1割=10%，2割=20%，3割=30%，9割=90%。这个单位在总结时要准确转换。

最后组织语言时，要突出汽车/摩托车的主导地位，以及地区差异特点，特别是有马和西神地区的极端情况。
</think>
根据提供的分析报告，神户观光游客（来神客）主要使用的交通方式总结如下：

1. **主要交通方式**：
   - **汽车/摩托车（車・バイク）**：是神户市内及全市最主要的交通方式，占比最高（约3割，即30%）。尤其在**有馬地区**使用比例极高（约9割，即90%）。
   - **铁路（鉄道）**：次主要方式，占比约2割（20%）。其中**西神地区**使用比例最高（约1割，即10%）。

2. **地区差异**：
   - **有馬地区**：汽车/摩托车使用比例显著高于其他地区（约9割）。
   - **西神地区**：铁路使用比例高于其他地区（约1割）。
   - **六甲・摩耶地区**：汽车/摩托车使用比例最低（约4割），铁路使用比例相对较高（约1割）。

3. **其他地区**：
   - 除有馬、西神外，其他地区汽车/摩托车使用比例普遍较低（约2割以下）。
   - 铁路使用比例普遍低于西神地区（约1割以下）。

**结论**：神户观光游客以**汽车/摩托车**为主要交通方式，尤其在有马地区占比高达90%；铁路为次要方式，西神地区使用比例较高（10%）。地区间交通方式偏好存在显著差异。<|user|>
CPU times: user 50.3 s, sys: 88 ms, total: 50.4 s
Wall time: 50.4 s

<think>
ええと、ユーザーは神客（神戸観光客）が利用する交通手段をまとめるよう要求しています。提供された分析レポートを注意深く読み、交通手段に関する部分を抽出する必要があります。

まず「交通機関の利用」の部分を確認します。ここでは神戸市内では主に「車・バイク」と「鉄道」が利用されており、自動車が約3割（30%）で最も高く、鉄道は約2割（20%）となっています。神戸市内限定で見ると、自動車の割合がさらに高く約3割、鉄道は約1割（10%）です。

次に地域差を見ると：北野地区では自動車利用率が最も高く（約2割）、他地区は低め。鉄道利用率では西神地区が最も高く（約1割）、他地区は低め。

さらに「有馬」地区では自動車利用率が非常に高く（約9割）、六甲摩耶地区では自動車利用率が最も低く（約4割）、鉄道利用率が最も高い（約1割）と記載されている。

これらの情報を総合すると、神戸観光客が主に利用する交通手段は次の通りとまとめられる：自動車/オートバイが絶対的主力であり、特に有馬地区では90%に達する。鉄道の利用割合は比較的低いが、西神地区では比較的高い。その他の交通手段（飛行機、バスなど）については報告書に記載されていない。

データ単位に特に注意が必要：報告書の割合は全て「割」（juku）で表記されており、1割=10%、2割=20%、3割=30%、9割=90%となる。要約時にはこの単位を正確に変換すること。

最終的な文章構成では、自動車/バイクの支配的地位と地域差の特徴、特に有馬と西神地区の極端な状況を強調すること。
</think>
提供された分析レポートに基づき、神戸観光訪問者（来神客）の主な交通手段を以下のようにまとめます：

1. **主要交通手段**：
   - **自動車・バイク（車・バイク）**：神戸市内及び全市で最も主要な交通手段であり、割合が最も高い（約3割、すなわち30%）。特に**有馬地区**での利用率が極めて高い（約9割、すなわち90%）。
   - **鉄道**：次主要な手段で、約2割（20%）を占める。このうち**西神地区**での利用率が最も高い（約1割、10%）。

2. **地域差**：
   - **有馬地区**：自動車/バイクの利用率が他地域より著しく高い（約9割）。
   - **西神地区**：鉄道利用率が他地域より高い（約1割）。
   - **六甲・摩耶地区**：自動車/オートバイ利用率が最低（約4割）、鉄道利用率は比較的高い（約1割）。

3. **その他の地域**：
   - 有馬・西神を除く地域では、自動車/オートバイ利用率が全般的に低い （約2割以下）。
   - 鉄道利用率は西神地区より全般的に低い（約1割以下）。

**結論**：神戸観光客の主な交通手段は**自動車/バイク**であり、特に有馬地区では90%に達する。鉄道は二次的な手段で、西神地区の利用率が高い（10%）。地域間で交通手段の選好に顕著な差異が存在する。<|user|>

(snip)
                "text": "Summarize about the transport services used by visitors to Kobe city"
(snip)

<think>Got it, let's tackle this. The user is asking about the transport services used by visitors to Kobe city. First, I need to look at the section titled "II. 利用交通機関" (Utilizing Transportation Facilities).  First, the main transportation methods: For the whole city, the top 4 are car, bicycle, train (JR, etc.), and bus. Then, by region: In "北野" (Kitano), car usage is higher than other regions. In "西神" (Nishinomiya), car and bicycle usage are very high—car is 9 times, bicycle 5 times the city average. Also, in "六甲・摩耶" and "須磨・舞子", car and bicycle are also high.  Wait, but the question is about transport services used by visitors. Let's confirm: The text says "神戸市内の主な利用交通手段としては" (For the main transportation methods within Kobe city), so it's about how visitors use transport to get to Kobe.  So summarizing: The most common transport services for visitors to Kobe are car, bicycle, train (JR, etc.), and bus. Regionally, North Kobe (Kitano) has high car use, West Kobe (Nishinomiya) has extremely high car and bicycle use (car 9x, bicycle 5x city average), while other areas like Rokko (六甲・摩耶) and Suma (須磨・舞子) also have high car and bicycle usage.</think>The transport services most commonly used by visitors to Kobe city are **car, bicycle, train (JR, etc.), and bus**.  

Key details:  
- **Citywide**: Car, bicycle, train (JR, etc.), and bus are the top four methods.  
- **Regional variations**:  
  - In **North Kobe (北野)**, car usage is higher than other regions.  
  - In **West Kobe (西神)**, car and bicycle usage are **significantly higher**—car is 9 times and bicycle 5 times the city average.  
  - In **Rokko (六甲・摩耶)** and **Suma (須磨・舞子)**, car and bicycle are also prominent.  

This data reflects how visitors primarily rely on these modes to access Kobe, with regional differences in preference.<|user|>
CPU times: user 39.4 s, sys: 69.6 ms, total: 39.4 s
Wall time: 39.4 s