ð€transformersã§ã®ã¢ãã«ã®åŠç¿ç¶æ³ãSlackã«éç¥ãã
ã©ãããSpiralAIã®@ksterxã§ãã
ã¿ãªããã¯ãã¢ãã«ã®åŠç¿ãè¡ãæããªãã®ã©ã€ãã©ãªã䜿çšããŠåŠç¿ããŠããŸãã§ããããïŒèªåèªèº«ã¯ããªãã ããã pytorch lightningã䜿ã£ãŠããææãé·ãã£ãã®ããªãšæããŸãããæè¿ã¯èšèªç³»ã觊ã£ãŠããããšããããð€transformersã䜿çšããæ©äŒãå€ãã§ããïŒmegatronãllama-factoryã«æµ®æ°ãããæ°ãããŠããŸãããããïŒ
ð€Hugging Face Hubã«ããå€ãã®ã¢ãã«ããð€transformersã䜿çšããŠããŒããæšè«ãç°¡åã«è¡ããŸãããŸããð€transformersã¯æ¯èŒçæ°ããè«æã®ã¢ã«ãŽãªãºã ã§ãã£ãŠãTrainerãšããŠæäŸãããããšãå€ããããç°¡åã«å€æ§ãªã¢ã«ãŽãªãºã ãè©Šããããšãå€ãã§ãã
å®éã«ð€transformersã䜿çšããŠã¢ãã«ã®åŠç¿ãããå ŽåãåŠç¿ç¶æ³ã確èªããããšããããšæããŸãããã®èŸºã¯ãTrainingArguments
ããã®ã©ãããŒSFTConfig
çã®report_to
ã«ã奜ã¿ã®åºåå
ïŒwandbãtensorboardããããã¯äœãèšå®ããªããšããŠãŒã¶ãŒãã€ã³ã¹ããŒã«ããŠããã€ã³ãã°ã¬ãŒã·ã§ã³å
šéšã«ïŒïŒãæå®ãããšãããã©ã«ãïŒtrain/eval lossããªãœãŒã¹äœ¿çšç¶æ³çïŒã®ãã®ã³ã°ãããŠãããŸãã
Source: Hugging Face Documentation
from trl import SFTConfig, SFTTrainer
train_args = SFTConfig(
report_to="wandb",
# ...
)
...
trainer = SFTTrainer(
args=train_args,
# ...
)
ãã ãæã ã®ãããªå¿é æ§andé¢åãããããªäººéã¯ãåŠç¿ãé²ãã§ããã®ããå¿é ã«ãªãããã§ãããããããwandbã®ããã·ã¥ããŒããèŠãå¿ èŠããªããããã§ããã©ã®ãããé²ãã§ãããããã·ã¥éç¥ãããã§ç¥ããã°è¯ãâââslackã«éç¥ããã°è¯ããã
æ¬é¡
å眮ããé·ããªããŸãããä»åå®çŸããã®ã¯ããšããã¯ãçµäºããã¿ã€ãã³ã°ãšãã¬ãŒãã³ã°èªäœãçµäºããã¿ã€ãã³ã°ã«éç¥ãé£ã°ãã·ã¹ãã ã§ãã
ã§ãããã®
ââââãããªæã
å¿ èŠãªãã®
- Slackã®Token (åèããŒãž)
- ç°å¢å€æ°
SLACK_BOT_TOKEN
ãšããŠèšå®
- ç°å¢å€æ°
- Pythonã©ã€ãã©ãª
transformers
slack_sdk
å®è£
ð€transformersã®ã¢ãã«ãã¬ãŒãã³ã°æã®æåãå€ããæ¹æ³ã®äžã€ããTrainerCallback
ãç¶æ¿ããç¬èªã®ã³ãŒã«ããã¯ãå®è£
ããããšã§ããç¬èªã®ã³ãŒã«ããã¯ã¯ã©ã¹ã«ãã¹ãããããå/åŸããšããã¯ãçµãã£ãæãä¿åãè¡ãæã®ã¡ãœããããªãŒããŒã©ã€ãããããšã§å®çŸãããŸããïŒå
¬åŒããã¥ã¡ã³ã)
ãªãŒããŒã©ã€ãããã¡ãœããã«ã¯ãèªåã§æž¡ããåŠç¿æ¡ä»¶TrainingArguments
ãåŠç¿ã®çŸåšã®ç¶æ
TrainerState
çã®ãªããžã§ã¯ããæž¡ãããšãã§ããŸããä»åã§èšãã°çŸåšã®ã°ããŒãã«ã¹ããããä¿åå
ãã£ã¬ã¯ããªã«ã¢ã¯ã»ã¹ããŠSlackã®ãã£ã³ãã«ã«ãã¹ãããŸãã
from transformers import TrainerCallback, TrainingArguments, TrainerState, TrainerControl
class SlackCallback(TrainerCallback):
default_text = "\n\nProject: {proj}\nRepo: {repo}\nLog: {log_dir}\n\n"
def __init__(self, channel, client=None):
super().__init__()
self.client = (
WebClient(token=os.environ["SLACK_BOT_TOKEN"]) if client is None else client
)
self.channel = channel
def on_epoch_end(
self,
args: TrainingArguments,
state: TrainerState,
control: TrainerControl,
**kwargs,
):
if state.is_world_process_zero:
self.client.chat_postMessage(
channel=self.channel,
text=f"â
Epoch {state.epoch:.1f} (global step {state.global_step}) finished!!â
"
+ self.default_text.format(
proj=args.run_name,
repo=args.hub_model_id,
log_dir=args.output_dir,
),
)
def on_train_begin(
self,
args: TrainingArguments,
state: TrainerState,
control: TrainerControl,
**kwargs,
):
if state.is_world_process_zero:
self.client.chat_postMessage(
channel=self.channel,
text="ðTraining startedð"
+ self.default_text.format(
proj=args.run_name,
repo=args.hub_model_id,
log_dir=args.output_dir,
),
)
def on_train_end(
self,
args: TrainingArguments,
state: TrainerState,
control: TrainerControl,
**kwargs,
):
if state.is_world_process_zero:
self.client.chat_postMessage(
channel=self.channel,
text="ðTraining finishedð"
+ self.default_text.format(
proj=args.run_name,
repo=args.hub_model_id,
log_dir=args.output_dir,
),
)
䜿ãæ¹
SlackCallback
ã¯ã©ã¹ãTrainerã«æž¡ãããšã§åããŸãã
slack_callback = SlackCallback("writeæš©éã®ãããã£ã³ãã«å")
trainer = SFTTrainer(
...,
callbacks=[slack_callback]
)
trainer.train()
ããã§ããã°ã©ã ãåãããšâŠ
ðç¡äºSlackã®ä»»æã®ãã£ã³ãã«ã«éç¥ããããšãã§ããæ§ã«ãªããŸããð
ãŸãšã
ä»åã¯ãð€transformersã䜿çšããŠã¢ãã«ã®åŠç¿é²æãSlackã«éç¥ããæ¹æ³ãã玹ä»ããŸãããããã§ãåŠç¿ã®é²æç¶æ³ããªã¢ã«ã¿ã€ã ã§ææ¡ããã®ã楜ã«ãªããŸãããããããããã·ã¥ããŒãã確èªãã«è¡ããªããŠããéç¥ãæ¥ãã®ã§å®å¿ããŠåŠç¿ãé²ããããŸãã
Slackéç¥ã¯ããŒã ã¡ã³ããŒãšãå ±æããããããããžã§ã¯ãå šäœã®éææ§ãé«ããå¹æãæåŸ ã§ããŸãããã¡ãããä»åã®ã³ãŒããããã«ã«ã¹ã¿ãã€ãºããŠãç¹å®ã®ã€ãã³ãããã°ã¡ãã»ãŒãžãè¿œå ããããšãå¯èœã§ããçããã®ãããžã§ã¯ãã«åãããŠãæé©ãªéç¥ã·ã¹ãã ãæ§ç¯ããŠã¿ãŠãã ããã
LLMãšã³ãžãã¢ãåéããŠããŸãïŒïŒ
SpiralAIã§ã¯ãçæAIÃãšã³ã¿ã¡ãããŒãã«æ§ã ãªãããžã§ã¯ããç«ã¡äžãã£ãŠããŸãïŒããããèå³ãããã°@ksterxãæ¡çšããŒãžãŸã§ãé£çµ¡ãã ããã
Discussion