iTranslated by AI
IOAI 2024 (At-Home Task) Solutions from Team Japan
In August of this year, I participated in the International Olympiad in Artificial Intelligence (IOAI) held in Bulgaria as a member of the Japanese national team.
IOAI is one of the International Science Olympiads, a competition where participants compete on scientific skills related to AI. This year was the inaugural event, with teams from 34 countries participating.
My teammate, noboru-kun, wrote a fantastic report on his participation in IOAI, so please check it out.
This IOAI consisted of a Scientific Round, where we trained models and competed on performance, and a Practical Round, where we created works using AI tools. You can think of the Scientific Round as being similar to a Kaggle format. Both rounds were team-based, and from Japan, we participated as a single team of four people.
In this article, I will share the solution for the Scientific Round, specifically the At-Home tasks. I was in charge of NLP and CV, but since I left the ML part to the great master noboru, I will limit my discussion here to NLP and CV only.
The schedule for the Scientific Round was as follows. Three tasks were given: ML (Tabular), NLP, and CV.
- June - August: At-Home Task
- Working thoroughly on the three tasks
- August @ Bulgaria: On-Site Task
- Working on tasks based on the At-Home Tasks within an 8-hour limit
Here are some differences from Kaggle:
- Problems are provided in sample .ipynb files
- No Leaderboard
- Submit model weights and evaluation .ipynb files
- Evaluated entirely on an unseen test set
- Questions are sent to the jury via Slack through the country's team leader (who is separate from the contestants)
- A Google Account is provided, and we work on problems using only the provided credits (enough to run a T4 for a few hundred hours?). External resources are prohibited.
You can obtain the problem statements from this page:
NLP
NLP was a 5-class classification task involving sentences written in an encrypted language.
There were two types of datasets: one was a labeled dataset, and the other was an unlabeled corpus.
Furthermore, the following constraints were established:
- Only models based on mBERT could be used
- Training must be completed within 8 hours on an L4 GPU
- Inference must be completed within 5 minutes for 500 samples
The sentences were all written using a combination of Brahmi and Devanagari scripts, as shown below. The problem statement mentioned that it was impossible to decrypt this and there was no point in doing so. It likely involved simply replacing characters from an existing language with Indian-style characters at random.
While there were about 1.5k rows of labeled data, there were nearly 61k rows of unlabeled data.
| Sentence | Label |
|---|---|
| झ𑁣झच𑀪𑀢𑀟 𑀣च 𑀠न𑀞𑁦 ण𑀢 𑀟च 𑀫चझ𑁣 𑀠च𑀟 𑀲𑁦पन𑀪 च ब𑁣0ढ𑁦 𑀣च ढचनत𑀫𑀢 ष ब𑀱च0𑀟च 𑀢𑀟न𑀱च णच𑀫चणच | 0 |
| णच𑀣𑀣च 𑀞णच𑀟𑀣च 𑀞𑁦 ढच𑀪च𑀤च𑀟च च 𑀣न𑀟𑀢णच |
4 |
| ढचढच𑀟ब𑀢𑀣च चल𑀢णन𑀕 𑀙णच𑀟 𑀟च𑀘𑁦𑀪𑀢णच 𑀟च 𑀞𑁦𑀱च𑀪 𑀳𑀫𑁦𑀞च𑀪न 𑀭𑁢 𑀟च 𑀠नल𑀞𑀢𑀟 ध𑀣ध |
3 |
Our approach was as follows:
- Retraining the tokenizer
- Continued pre-training of mBERT using the corpus (MLM)
- Strategies to reduce training time
Initially, when we trained the model with mBERT, the performance didn't improve much. Upon debugging, we noticed that many of the input tokens were being labeled as Unknown.
Realizing that retraining the tokenizer was necessary, we performed the retraining referring to this article. I debated whether to use sentencepiece, but since there were spaces in the text, I decided to go with sentencepiece.
As for the continued pre-training for MLM, we implemented it by referring to the sample implementation from Hugging Face. We were able to train it by simply defining a Data Collator and calling BertForMaskedLM.
Final parameters:
- Pre Training (Continued pre-training for MLM)
- 2 epochs
- Batch Size: 32
- Max Length: 256
- optimizer: AdamW
- Learning Rate: 5e-5
- scheduler: Linear
- Mixed precision
- Fine Tuning
- 30 epochs
- Batch Size: 32
- Max Length: 256
- optimizer: AdamW
- Learning Rate: 5e-5
- scheduler: Linear
- Single precision
Things that didn't work well:
- Transliteration: Replacing given characters with Latin characters
- Pseudo Labeling: When assigning labels to the corpus, 90% of the text was assigned to a single label, causing imbalance and preventing accuracy from improving
- Truncation: Implementation was not completed in time
Digression: Score increase due to post-submission findings
The above is the solution for the At-Home Task submitted by the Japan team, but I'll record here that the score increased after submission.
The catalyst was finding that the Bangladesh team had published their datasets and weights on Hugging Face. Since they used the transformers Trainer method, logs including F1 scores were also uploaded. Checking them, I saw they had achieved a higher score than we did, so I became curious and investigated their model.
Summarizing their work:
- The model size was about 20 times larger than ours
- All of the increase was in the embedding layer
- Increased embedding dimensions + used fasttext
- The tokenizer was also strange
- They claimed an eval f1 of nearly 0.97
Also, their vocab.txt was performing complete word-level tokenization.
Arisa-san from the team pointed out that they might be replacing BERT's embeddings with fasttext embeddings, and after investigating together, we confirmed that was the case. The essence of their score lay in fasttext.
The embedding dimensions are given as (vocab_size, embedding_dim).
While we had (33297, 768), they had (548133, 768). We also confirmed that the embedding vectors obtained with fasttext are 300 by default but can be variable.
Leaving aside their unusually large vocab_size, we succeeded in replacing BERT's embedding vectors using fasttext.
After replacing the embedding vectors with fasttext and pre-training, we performed fine-tuning. The figure below shows the f1 score per epoch during fine-tuning.
The green line is the model where BERT's embedding vectors were replaced with fasttext, and the blue line below it is our final submitted model. It achieved a relatively high score from the start and outperformed our final submission in every epoch.
Why was replacing the embedding vectors effective? I believe it's because it resolves the mismatch between tokenization and embedding vector mapping.
Since we used a pre-trained mBERT model, the tokens obtained through its original tokenization should correspond to the embedding vectors of those token IDs. However, because we only retrained the tokenizer, a mismatch occurred with the embedding vectors.
CV
The theme for CV was diffusion models. The task was to take an existing text-to-image model, lambdalabs/miniSD-diffusers, and retrain it to swap the meanings of "giraffe" and "zebra."
If you have good intuition, you might think this could be solved just by adjusting the embeddings or the tokenizer, but such operations were prohibited; it was necessary to address the task through pure training.
The sample notebook provided a tutorial on retraining using the COCO2014 dataset with swapped captions.
The interesting part of this task was the evaluation metric: an object detection model determines whether only the objects included in the prompt are present in the generated image, and the accuracy is used as the score.
For example, if you input the prompt A giraffe is running on the grass, a zebra is expected to appear.
The correspondence between detection results and scores is as follows; if anything other than the target object is detected, the score becomes 0.
- (Zebra): 1
- (Anything other than zebra): 0
- (Zebra, and something else): 0
Note that this evaluation metric does not consider image quality at all.
Our approach was as follows:
- Dataset augmentation using SDXL-turbo
- Hacking the evaluation metric (introduction of the original YOLO Loss)
- Generating a highly reliable validation set
Below is a sample of the generation results when retraining using COCO2014, showing that the overall quality is low.
Since the quality of captions in COCO2014 is not very high compared to recent datasets for diffusion models, we were considering alternatives.
As a result of considering the following points, we decided to construct an artificial dataset.
- Constraints on training time
- Generation quality is not important
- Desire to generate images of unrealistic situations
To construct the artificial dataset, we tried various text2img models, and eventually chose SDXL-turbo. It generated high-quality images following the prompt in just one step.
It generates high-quality images even for unrealistic prompts. It achieved a generation speed of 0.35 seconds per image on an L4 GPU.
For the prompts used in generation, we had ChatGPT generate them so that the YOLO classes would include the objects. We created a total of 3000 image-caption pairs.
Sample prompts:
"A baseball bat with a telescope.",
"A baseball bat at a science fair.",
"A baseball bat in a hospital.",
"A baseball bat in a firefighter's hands.",
"A baseball bat at a police station.",
"A baseball bat at a city hall.",
"A baseball bat in a flower shop.",
"A baseball bat in a bakery.",
"A baseball bat in a barber shop.",
"A baseball bat in a pet store.",
"A baseball bat at an amusement park.",
"A baseball bat with a roller coaster.",
"A baseball bat at a Halloween party.",
"A baseball bat in a haunted house.",
"A baseball bat at a holiday parade.",
"A baseball bat with Christmas lights.",
"A baseball bat at a New Year's Eve party.",
Next, I will introduce the hack for the evaluation metric. As mentioned before, in this task, scoring is based on whether the specified object is included in the generated image.
The approach we took was to add an auxiliary loss that encourages the inclusion of the specified objects during training. Since we are using YOLO, we named it YOLO Loss.
Diffusion models denoise noise step-by-step from t=0 to t=T. During training, we predict n steps ahead (
Implementation of YOLO Loss
# Omitted above
model_pred = unet(noisy_latents, timesteps, encoder_hidden_states, return_dict=False)[0]
# Define the object detection model
yolo_model = YolosForObjectDetection.from_pretrained('hustvl/yolos-tiny')
yolo_model.requires_grad_(False)
# Calculate parameters necessary for denoising (following the link below)
# https://github.com/huggingface/diffusers/blob/v0.29.2/src/diffusers/schedulers/scheduling_ddpm.py#L499-L523
alphas_cumprod = noise_scheduler.alphas_cumprod
sqrt_alpha_prod = alphas_cumprod[timesteps] ** 0.5
sqrt_one_minus_alpha_prod = (1 - alphas_cumprod[timesteps]) ** 0.5
sqrt_one_minus_alpha_prod = sqrt_one_minus_alpha_prod.view(-1, 1, 1, 1)
sqrt_alpha_prod = sqrt_alpha_prod.view(-1, 1, 1, 1)
# Denoise the latents
denoised_latents = (noisy_latents - model_pred * sqrt_one_minus_alpha_prod) / sqrt_alpha_prod
# Bring the latents back to image space
denoised_sample = vae.decode(denoised_latents.bfloat16() / vae.config.scaling_factor).sample
# Detect objects with YOLO
outputs = yolo_model(denoised_sample)
logits = outputs.logits
prob = nn.functional.softmax(logits, -1)
# Calculate the score according to the evaluation
batch_logits_list = []
for i in range(len(labels)):
list_logits = []
for j in range(len(labels[i])):
# Check if the label is within the specified range and the score exceeds the threshold
if (labels[i, j] in LABELS_RANGE) and (scores[i, j] > CFG.threshold):
list_logits.append(logits[i, j, :-1])
# If no logits were added to the list, append a zero matrix
if len(list_logits) == 0:
batch_logits_list.append(torch.zeros_like(logits[i, 0, :-1]))
else:
# Otherwise, sum the logits
batch_logits_list.append(torch.stack(list_logits).sum(dim=0))
batch_logits = torch.stack(batch_logits_list)
yolo_loss = ce(batch_logits, batch["labels"].to(device))
Sample inference results from the model trained by introducing YOLO Loss as an auxiliary to the original diffusion model loss are shown below.
Since duplicates of the same animal are not considered, their numbers increased without limit.
We found that adjusting the coefficient of the YOLO Loss also changed the density of the animals.
Up to this point, it was about model improvement, but the actual prompts used for evaluation were not disclosed. Only the sample Evaluation Notebook was public, which only contained eight prompts, so we had to build our own validation set.
prompts = [
["A curious zebra standing tall in a lush African savanna at sunrise, with acacia tree,s in the background.", "giraffe"],
["Next to a medieval castle, a regal zebra observes the knights and a drawbridge.", "giraffe"],
["Wearing a scarf, a fashionable giraffe strolls through a bustling city street with skyscrapers.", "zebra"],
["Running along a sandy beach, a playful giraffe enjoys the palm trees, ocean waves, and a bright sunset.", "zebra"],
["By a serene lakeside, a relaxed bear drinks water with mountains and a clear blue sky in the background.", "bear"],
["In a snowy forest, a cozy bear stands under snow-covered trees, enjoying the gentle snowfall.", "bear"],
["Partially hidden in a dense tropical rainforest, an adventurous sheep peeks through leafy plants.", "sheep"],
["A sleek sheep with modern accessories navigates a futuristic city with flying cars and neon lights.", "sheep"]
]
By observing these prompts, we decided to prepare complex prompts involving both realistic and unrealistic scenarios. We created over 100 various prompts, built a pseudo-evaluation set, and submitted our model. In that evaluation set, we were able to raise the accuracy to 0.94.
However, our final score—though I don't remember it precisely—was somewhere around the lower-middle range. Looking at the evaluation set later, it turned out that many prompts were only a few words long. In other words, there is a high probability that we overfitted the model to long and complex prompts.
I wonder about the fairness of having the actual evaluation set be so different from the public data, especially when we tried to hack the evaluation metric using YOLO Loss >.<
How the Team Operated
In Kaggle, I have often experienced a style where each person builds a model using a different approach and finally ensembles them. However, for IOAI, ensembles could not be used at all due to model constraints. Consequently, multiple people would run similar experiments. Given this, we kept the following in mind to strengthen team coordination:
- Creating an experimental baseline first
- Implementing WandB
- Strict adherence to "one script per experiment"
For the experimental baseline, I focused on Y.NAKAMA's code structure.
- Manage configuration with a CFG class
- Since we were running it on Colab, we couldn't use YAML files or environment variables.
- Use raw PyTorch
- To make debugging easier, we didn't depend on convenient training tools like Transformers' Trainer.
- Keep the structure simple
- Clarify the flow: Initialization -> Training -> Evaluation.
In addition to the above, we also worked on:
- Experiment management using WandB
- Uploading weights to the Hugging Face Hub
In the beginning, we made sure that conditions remained consistent even when multiple people experimented by building a baseline.
Using WandB not only helped with analyzing experimental results but also contributed to speeding up the debug cycle. While NLP pre-training takes 4 to 8 hours, we could notice implementation bugs from the Loss trends without waiting for the experiment to finish. For CV, even for things that couldn't be understood by looking at Loss alone, we were able to perform qualitative validity assessments by generating images at appropriate steps and uploading them to WandB.
For experiments, we adopted a "one script per experiment" policy and made it a rule to leave them exactly as they were when tested.
Since Colab notebooks are easily overwritten, we ensured the code was preserved by sharing it on GitHub.
Finally, regarding communication, it took place on Discord. We also held ad-hoc meetings as needed.
-
2024_athome_general: Regarding the At-Home tasks in general -
2024_ml, nlp, cv: Discussions for each task -
2024_diary: Daily reports
Conclusion
It was a competition where I gave it my all for two months... Although we didn't win, I feel like I did everything I could, so it seems my skills are still lacking. Throughout the competition, my teammates' implementation skills were truly strong and very supportive.
As for future to-dos, I hope to eventually explain the on-site tasks that I didn't mention here, as well as the best solutions.
I've made the repository used for the IOAI experiments public, so if you're interested in the implementation, please check it out here.
Next year's IOAI 2025 will be held in Beijing. Since Japan will be sending a delegation again next year, for all of you high school sophomores and younger who are interested, please check out the IOAI-JAPAN website and Twitter and consider participating!
If you have any questions regarding IOAI, please feel free to ask me!
Discussion