例えばgpt-4oやclaude-3.5-sonnetなどのvlmと比較して、どういうところに優位性がある？gpt-4oやclaude-3.5-sonnetなどもOCRは使用しませんが。

!mkdir -p examples
!wget -O examples/docowl2_page0.png https://huggingface.co/mPLUG/DocOwl2/resolve/main/examples/docowl2_page0.png?download=true
!wget -O examples/docowl2_page1.png https://huggingface.co/mPLUG/DocOwl2/resolve/main/examples/docowl2_page1.png?download=true
!wget -O examples/docowl2_page2.png https://huggingface.co/mPLUG/DocOwl2/resolve/main/examples/docowl2_page2.png?download=true
!wget -O examples/docowl2_page3.png https://huggingface.co/mPLUG/DocOwl2/resolve/main/examples/docowl2_page3.png?download=true
!wget -O examples/docowl2_page4.png https://huggingface.co/mPLUG/DocOwl2/resolve/main/examples/docowl2_page4.png?download=true
!wget -O examples/docowl2_page5.png https://huggingface.co/mPLUG/DocOwl2/resolve/main/examples/docowl2_page5.png?download=true

!pip install icecream

!wget https://github.com/kun432/flash-attention-prebuild-wheels/releases/download/v0.0.0-test/flash_attn-2.6.3+cu121torch2.5-cp310-cp310-linux_x86_64.whl
!pip install --no-dependencies --upgrade flash_attn-2.6.3+cu121torch2.5-cp310-cp310-linux_x86_64.whl

import torch
import os
from transformers import AutoTokenizer, AutoModel
from icecream import ic
import time

class DocOwlInfer():
    def __init__(self, ckpt_path):
        self.tokenizer = AutoTokenizer.from_pretrained(ckpt_path, use_fast=False)
        self.model = AutoModel.from_pretrained(ckpt_path, trust_remote_code=True, low_cpu_mem_usage=True, torch_dtype=torch.float16, device_map='auto')
        self.model.init_processor(tokenizer=self.tokenizer, basic_image_size=504, crop_anchors='grid_12')
        
    def inference(self, images, query):
        messages = [{'role': 'USER', 'content': '<|image|>'*len(images)+query}]
        answer = self.model.chat(messages=messages, images=images, tokenizer=self.tokenizer)
        return answer


docowl = DocOwlInfer(ckpt_path='mPLUG/DocOwl2')

images = [
        './examples/docowl2_page0.png',
        './examples/docowl2_page1.png',
        './examples/docowl2_page2.png',
#        './examples/docowl2_page3.png',
#        './examples/docowl2_page4.png',
#        './examples/docowl2_page5.png',
    ]

answer = docowl.inference(images, query='what is this paper about? provide detailed information.')

ic| prompt: ('USER: <img 1><|image|><img 2><|image|><img 3><|image|>what is this paper '
             'about? provide detailed information. ASSISTANT:')
This paper is about OCR-free multi-page document understanding for derstanding.
As shown in the first image, The paper discusses the problem of OCR-free multi-page document understanding, which is the ability to process and extract meaningful information from multi-page documents without relying on optical character recognition (OCR) software. The authors present a high-resolution compression method called MPLUG-DocOwl2 that achieves promising OCR-free Document Understanding performance by considering the cost of generating thorough sands of visual tokens for a single document image, leading to excessive GPU computation time. The paper also provides a detailed explanation of the overall structure parsing of the document.

answer = docowl.inference(images, query='what is the third page about? provide detailed information.')

ic| prompt: ('USER: <img 1><|image|><img 2><|image|><img 3><|image|>what is the third page '
             'about? provide detailed information. ASSISTANT:')
The third page is about related work.
In the third section of the text, the authors discuss related work in the field of OCR-free document understanding. They begin by defining OCR-free document understanding as comprehending images with rich text information, including scans of document pages. They then go on to discuss previous work in this area, including methods for compressing and decompressing text in images, methods for understanding text in images, and methods for recognizing handwritten text. The authors conclude by highlighting their own contributions to this field, specifically their work on compressing and decompressing text in images using neural networks.

answer = docowl.inference(images, query='この論文は何について書かれたものですか？詳細な情報を提供してください。')

ic| prompt: ('USER: <img 1><|image|><img 2><|image|><img '
             '3><|image|>この論文は何について書かれたものですか？詳細な情報を提供してください。 ASSISTANT:')
 OCR-free Multi-page Document Understanding

answer = docowl.inference(images, query='3ページ目はどのような内容ですか？詳細な情報を提供してください。')

ic| prompt: ('USER: <img 1><|image|><img 2><|image|><img '
             '3><|image|>こ3ページ目はどのような内容ですか？詳細な情報を提供してください。 ASSISTANT:')
 3ページ目は単純な図や図解を掲載しているから質問がありますが詳細を教えていただけますか？