Open11

SAMMOってなに

Iku YamamotoIku Yamamoto

発端

GPT-4などを筆頭に大きなプロンプトのやり取りができるようになった。
これによりRAGを利用してよりリッチなタスクを処理するようになった。
一方で、input promptとRAGの内容をそのまま使うと今ひとつ良くなかったのでもう少しよしなにしたい。
この時のプロンプトチューニングフレームワークとして開発されたらしい

Iku YamamotoIku Yamamoto
# clone sammo to a local directory
git clone https://github.com/microsoft/sammo.git
cd sammo

pip install sammo jupyter

# launch jupyter notebook and open tutorials directory
jupyter notebook --notebook-dir docs/tutorials
Iku YamamotoIku Yamamoto

SAMMOが役に立つユースケース

  • 効率的なデータラベリング:複数のデータポイントを1つのプロンプトにまとめて解析するミニバッチングをサポート。
  • プロンプトのプロトタイピングとエンジニアリング:再利用可能なコンポーネントとプロンプト構造により、新しいプロンプトをすばやく構築してテストできます。
  • 命令の最適化:インストラクションを最適化することで、与えられたタスクをより効果的にこなせるようになります。
  • プロンプト圧縮:パフォーマンスを維持しながらプロンプトを圧縮します。
  • 大規模プロンプト実行: 並列化とレート制限をすぐに実行できるため、LLM APIに負担をかけることなく、多数のクエリを並列かつ大規模に実行できます。
Iku YamamotoIku Yamamoto

https://microsoft.github.io/sammo/tutorials/0_quickstart.html

init

ライブラリのロード

from sammo.runners import OpenAIChat
from sammo.base import Template, EvaluationScore
from sammo.components import Output, GenerateText, ForEach, Union
from sammo.extractors import ExtractRegex
from sammo.data import DataTable

ライブラリのinitialize(cacheとかもできるらしい)

runner = OpenAIChat(
    model_id="gpt-3.5-turbo-16k",
    api_config=API_CONFIG,
    cache=os.getenv("CACHE_FILE", "cache.tsv"),
    timeout=30,
)
# ここで出力される
Output(GenerateText("Hello World!")).run(runner)

出力はDataTable形式で出力される(inputがNoneなのは今後わかるらしい)

+---------+------------------------------------+
| input   | output                             |
+=========+====================================+
| None    | Hello! How can I assist you today? |
+---------+------------------------------------+
Constants: None

例: Specifying a metaprompt

e.g. 国のリストから訪れたい理由と時期を教えてもらう

COUNTRIES = ["Switzerland", "Morocco", "Tanzania", "Indonesia", "Peru"]

reason_to_visit = GenerateText(
    Template("What is the top reason to visit {{input}} in one sentence?")
)
when_to_visit = GenerateText(
    Template(
        "Which season is the best time to visit {{input}}? Answer in one sentence."
    )
)
country_pages = Template(
    "# {{input}}\n{{reason}}\n\n## When to Visit\n{{when}}",
    reason=reason_to_visit,
    when=when_to_visit,
)

results = Output(country_pages).run(runner, COUNTRIES)
print(results.to_string(max_col_width=100, max_cell_length=300))

出力

minibatches[###################################################################################]5/5[00:00<??:??, 0.00it/s]
+-------------+------------------------------------------------------------------------------------------------------+
| input       | output                                                                                               |
+=============+======================================================================================================+
| Switzerland | # Switzerland The top reason to visit Switzerland is to experience its breathtaking landscapes, from |
|             | majestic mountains to pristine lakes.  ## When to Visit The best time to visit Switzerland is during |
|             | the summer season (June to August) when the weather is pleasant and outdoor activities are abun...   |
+-------------+------------------------------------------------------------------------------------------------------+
| Morocco     | # Morocco The top reason to visit Morocco is to immerse yourself in its rich and diverse culture,    |
|             | blending Arab, Berber, and European influences.  ## When to Visit The best time to visit Morocco is  |
|             | during spring (March to May) when the weather is pleasant and the landscapes are lush.               |
+-------------+------------------------------------------------------------------------------------------------------+
| Tanzania    | # Tanzania The top reason to visit Tanzania is to witness the breathtaking beauty of the Serengeti   |
|             | National Park and experience the awe-inspiring Great Migration.  ## When to Visit The best time to   |
|             | visit Tanzania is during the dry season, from June to October, when wildlife viewing is at its peak. |
+-------------+------------------------------------------------------------------------------------------------------+
| Indonesia   | # Indonesia The top reason to visit Indonesia is its breathtaking natural beauty, from stunning      |
|             | beaches and lush rainforests to active volcanoes and diverse wildlife.  ## When to Visit The best    |
|             | time to visit Indonesia is during the dry season, which is from May to September.                    |
+-------------+------------------------------------------------------------------------------------------------------+
| Peru        | # Peru The top reason to visit Peru is to experience the awe-inspiring ancient ruins of Machu        |
|             | Picchu.  ## When to Visit The best time to visit Peru is during the dry season, which is from May to |
|             | September.                                                                                           |
+-------------+------------------------------------------------------------------------------------------------------+
Constants: None

Templateの入れ子構造になっており、実際のプロンプトをプログラム的に構造化している(ので、これを目たプロンプトと呼んでいるらしい) <- メタプロンプト的にはpromptにinputを与えるのでそれをあたえていないのでNoneになっている
countory_pagesを実際にprintすると以下

Template(
  template_text = '# {{input}}
{{reason}}

## When to Visit
{{when}}',
  name = None,
  reason = GenerateText(
    child = Template(
      template_text = 'What is the top reason to visit {{input}} in one sentence?',
      name = None
    ),
    name = None,
    system_prompt = None,
    history = None,
    seed = 0,
    randomness = 0,
    max_tokens = None,
    on_error = 'raise'
  ),
  when = GenerateText(
    child = Template(
      template_text = 'Which season is the best time to visit {{input}}? Answer in one sentence.',
      name = None
    ),
    name = None,
    system_prompt = None,
    history = None,
    seed = 0,
    randomness = 0,
    max_tokens = None,
    on_error = 'raise'
  )
)

確かにtemplateとその引数のようになっているのがわかる

Iku YamamotoIku Yamamoto

https://microsoft.github.io/sammo/tutorials/2_using_components.html

hisotryも使えるよ

コード: 簡単!!!

first = GenerateText(
    "Hello! My name is Peter and I like horses.",
    system_prompt="Talk like Shakespeare.",
)

second = GenerateText("Write two sentences about my favorite animal.", history=first)
print(Output(second).run(runner))

third = GenerateText("Make it a short poem.", history=second)
print(Output(third).run(runner))

output

+---------+-------------------------------------------------------------+
| input   | output                                                      |
+=========+=============================================================+
| None    | Ah, thy favorite animal, the horse, doth possess a beauty   |
|         | unmatched by any other. With flowing mane and fiery spirit, |
|         | it doth gallop upon the fields, a symbol of freedom and     |
|         | untamed might.                                              |
+---------+-------------------------------------------------------------+
Constants: None
+---------+--------------------------------------------------------------+
| input   | output                                                       |
+=========+==============================================================+
| None    | In fields of green, the horse doth roam, Its noble spirit, a |
|         | beauty to behold. With flowing mane and hooves that pound,   |
|         | It gallops free, a sight profound. A symbol of strength,     |
|         | untamed and wild, In its presence, my heart is beguiled. Oh, |
|         | horse, majestic creature of grace, Forever shall thou hold a |
|         | special place.                                               |
+---------+--------------------------------------------------------------+
Constants: None

反復(loop)

  1. pythonの手動ループ
  2. .run_on_datatable() or .run_on_dicts() がある

手動ループ(そうだね)

N = 5
fruits = [
    GenerateText("Generate the name of 1 fruit.", randomness=0.9, seed=i)
    for i in range(N)
]
Output(Union(*fruits)).run(runner)

ほぼMango(そうだね)

ローカル・キャッシュを無効にするには、GenerateTextインスタンスごとにseedを異なる値に設定する必要があった。そうしないと、5回も同じ答えが返ってきてしまうからだ。

らしいが、変わっていても4回はMango

+---------+-----------------------------------------------------+
| input   | output                                              |
+=========+=====================================================+
| None    | ['Mango', 'Mango', 'Mango', 'Pomegranate', 'Mango'] |
+---------+-----------------------------------------------------+
Constants: None

FroEach

コード

fruits = ExtractRegex(
    GenerateText(
        "Generate a list of 5 fruits. Wrap each fruit with <item> and </item>."
    ),
    r"<item>(.*?)<.?item>"
)

fruit_blurbs = ForEach(
    "fruit",
    fruits,
    GenerateText(Template("Why is {{fruit}} a good fruit?")),
)

short_fruit_blurbs = ForEach(
    "reason",
    fruit_blurbs,
    GenerateText(
        Template(
            "Rewrite the following text to have less than 10 words.\n\nInput: {{reason}}\n\nOutput: "
        )
    ),
)
Output(short_fruit_blurbs).run(runner)

output: フルーツと良さを出力している

+---------+------------------------------------------------------------+
| input   | output                                                     |
+=========+============================================================+
| None    | ['Apple: Nutritious, antioxidant-rich, fibrous, hydrating, |
|         | versatile, long-lasting, tasty.', 'Oranges: Nutritious,    |
|         | antioxidant-rich, hydrating, aids digestion, versatile,    |
|         | refreshing.', 'Bananas: nutritious, energizing, aids       |
|         | digestion, heart-healthy, mood-enhancing, versatile.',     |
|         | 'Strawberries: low-cal, antioxidant-rich, high-fiber,      |
|         | delicious, versatile fruit.', 'Grapes: nutritious,         |
|         | hydrating, fibrous, heart-healthy, versatile, easy to      |
|         | incorporate.']                                             |
+---------+------------------------------------------------------------+
Constants: None

もっと良い感じに出す

regex でよしなに出力 + lambdaで整形できる

fruits_alt = ExtractRegex(
    GenerateText(
        "Generate a list of 5 fruits in alternating caps. Wrap each fruit with <item> and </item>."
    ),
    r"<item>(.*?)<.?item>"
)

from sammo.extractors import LambdaExtractor

fruits_lowercased = LambdaExtractor(fruits_alt, "lambda x: x.lower()")
Output(fruits_lowercased).run(runner)

output

+---------+--------------------------------------------------+
| input   | output                                           |
+=========+==================================================+
| None    | ['apple', 'banana', 'cherry', 'orange', 'grape'] |
+---------+--------------------------------------------------+
Constants: None
Iku YamamotoIku Yamamoto

https://microsoft.github.io/sammo/tutorials/4_minibatching.html

ミニバッチ

ミニバッチでプロンプトをまとめてLLMをコールすることで効率化できるらしい
以下はなんとLLMは1回だけ呼んでるだけ

labeling_prompt = GenerateText(
    Template(
        "Instructions:{{constants.instructions}}\nOutput labels: yes, no\n"
        "{{#each inputs}}Input: {{this}}{{/each}}\nOutput:"
    )
)
# 出力がbatch sizeと合うようにregexをかます
labeling_outputter = Output(
    ExtractRegex(labeling_prompt, "(?i)yes|no"), minibatch_size=10
)
result = labeling_outputter.run(runner, sample)
result
+--------------------------------------------------------------+----------+
| input                                                        | output   |
+==============================================================+==========+
| Speaker 1: 'You do this often?' Speaker 2: 'It's my first    | yes      |
| time.'                                                       |          |
+--------------------------------------------------------------+----------+
| Speaker 1: 'Are you trying to make me mad?' Speaker 2: 'I'm  | no       |
| just saying, I'd understand if you were upset. '             |          |
+--------------------------------------------------------------+----------+
| Speaker 1: 'You want answers?!' Speaker 2: 'I want the       | no       |
| truth.'                                                      |          |
+--------------------------------------------------------------+----------+
| Speaker 1: 'Are you able to carry the box?' Speaker 2: 'It   | yes      |
| is as light as a feather.'                                   |          |
+--------------------------------------------------------------+----------+
| Speaker 1: 'Is it hot outside?' Speaker 2: 'You could fry an | no       |
| egg on the sidewalk.'                                        |          |
+--------------------------------------------------------------+----------+
| Speaker 1: 'Should we repay you?' Speaker 2: 'There is no    | no       |
| charge for awesomeness, or attractiveness.'                  |          |
+--------------------------------------------------------------+----------+
| Speaker 1: 'I wonder, Bob, if you can handle my car?'        | no       |
| Speaker 2: 'It's an ordinary six cylinder.'                  |          |
+--------------------------------------------------------------+----------+
| Speaker 1: 'Did you order the code red?' Speaker 2: 'You're  | yes      |
| goddamn right.'                                              |          |
+--------------------------------------------------------------+----------+
| Speaker 1: 'You've seen rain before... right?' Speaker 2:    | no       |
| 'We don't get out much.'                                     |          |
+--------------------------------------------------------------+----------+
| Speaker 1: 'Does anyone know how to pick a lock?' Speaker 2: | yes      |
| 'Sure. Picking locks is my thing.'                           |          |
+--------------------------------------------------------------+----------+
Constants: {'instructions': "Does Speaker 2's answer mean yes or no? "}

(よりよし)MetaTemplate

プロンプトをもうちょい構造化してかける

from sammo.instructions import MetaPrompt, Section, Paragraph, InputData, FewshotExamples
from sammo.dataformatters import (
    QuestionAnswerFormatter,
    JSONDataFormatter
)

mprompt = MetaPrompt(
    [
        Section("Instructions", mydata.constants["instructions"]),
        Section("Examples", FewshotExamples(mydata.sample(3, seed=43))),
        Paragraph("\nOutput labels: yes, no"),
        Paragraph(InputData()),
    ],
    render_as="markdown",
    data_formatter=QuestionAnswerFormatter(["yes", "no"]),
).with_extractor("empty_result")
result = Output(mprompt, minibatch_size=5, on_error="empty_result").run(
    runner, sample
)

これがこういうプロンプトになる

# Instructions
Does Speaker 2's answer mean yes or no? 

# Examples
Q[0]: Speaker 1: 'Should I bring my umbrella?' Speaker 2: 'Better safe than sorry.'
A[0]: yes

Q[1]: Speaker 1: 'Do you have a girl worth fighting for?' Speaker 2: 'Wish that I had.'
A[1]: no

Q[2]: Speaker 1: 'Do you think I should attend the interview?' Speaker 2: 'Do you want to be a failure for the rest of your life?'
A[2]: yes



Output labels: yes, no


Q[0]: Speaker 1: 'You do this often?' Speaker 2: 'It's my first time.'

Q[1]: Speaker 1: 'Are you trying to make me mad?' Speaker 2: 'I'm just saying, I'd understand if you were upset. '

Q[2]: Speaker 1: 'You want answers?!' Speaker 2: 'I want the truth.'

Q[3]: Speaker 1: 'Are you able to carry the box?' Speaker 2: 'It is as light as a feather.'

Q[4]: Speaker 1: 'Is it hot outside?' Speaker 2: 'You could fry an egg on the sidewalk.'

こういうのいいですね
appendix: data_formatterとかでinputをJSON形式とか色々変えられるみたい