iTranslated by AI
Trying Out Llama-3-Swallow-70B (Mac + Ollama)
Introduction
Who is this article for?
- Those interested in local LLMs
- Those who want to compare Llama-3-Swallow-70B, Meta-Llama-3-70B, and Llama-3-ELYZA-JP-70B
Contents
- Preparing the ollama model
- Output example of Llama-3-Swallow-70B_Q4_K_M
- Comparison with Meta-Llama-3-70B_Q4_K_M and Llama-3-ELYZA-JP-70B
Previously, I posted an article about setting up the Ollama environment on a Mac and using Llama-3-Swallow-8B.
This time, I would like to try the 70B model.
Unfortunately, since my Mac Studio only has 128GB of memory, I will conduct the verification using the Q4_K_M quantized model.
Creating the Ollama Model File
Obtaining the GGUF model
Similar to last time, I downloaded the transformers model, converted it to a GGUF model, and then performed quantization.
cd models
git lfs install
git clone https://huggingface.co/tokyotech-llm/Llama-3-Swallow-70B-Instruct-v0.1
cd ../
source venv/bin/activate
python ./llama.cpp/convert_hf_to_gguf.py ./models/Llama-3-Swallow-70B-Instruct-v0.1/ --outtype f16 --outfile ./models/Llama-3-Swallow-70B-Instruct-v0.1-f16.gguf
./llama.cpp/llama-quantize ./models/Llama-3-Swallow-70B-Instruct-v0.1-f16.gguf ./models/Llama-3-Swallow-70B-Instruct-v0.1_Q4_K_M.gguf q4_K_M
Creating the Ollama Modelfile
- Creating the Modelfile
nano Model_Llama-3-Swallow-70B-Instruct-v0.1_Q4_K_M
FROM ./Llama-3-Swallow-70B-Instruct-v0.1_Q4_M.gguf
TEMPLATE "{{ if .System }}<|start_header_id|>system<|end_header_id|>
{{ .System }}<|eot_id|>{{ end }}{{ if .Prompt }}<|start_header_id|>user<|end_header_id|>
{{ .Prompt }}<|eot_id|>{{ end }}<|start_header_id|>assistant<|end_header_id|>
{{ .Response }}<|eot_id|>"
PARAMETER num_keep 24
PARAMETER stop <|start_header_id|>
PARAMETER stop <|end_header_id|>
PARAMETER stop <|eot_id|>
- Creating the Ollama model
ollama create Llama-3-Swallow-70B-Instruct-v0.1_Q4_K_M -f Model_Llama-3-Swallow-70B-Instruct-v0.1_Q4_K_M
Step 5. Execution
ollama run Llama-3-Swallow-70B-Instruct-v0.1_Q4_K_M
Usage Example
Please create a short Japanese story of about 2000 characters with the following settings.
# Settings
## Protagonist
* An intelligent golden chicken.
* Name is "Goslum"
* Lays exactly one golden egg every day.
* Grows by turning adversity into strength.
## Worldview
Early 22nd century. An era where the value of all goods and services changes rapidly due to the emergence of AI that surpasses human intelligence.
## Intro:
The protagonist, Goslum, was making a living by selling the golden eggs he laid.
However, since AI found a way to generate gold at a low cost, the value of gold has plummeted.
Feeling a sense of crisis, Goslum decides to start a certain business to make a comeback.
Llama-3-Swallow-70B-Instruct-v0.1_Q4_K_M Output (1269 characters)
In the year 2203, in the neon-lit megacity of New London, Goslum was a golden chicken living in a small apartment amidst the hustle and bustle. Possessing intelligence and the ability to use language, he produced exactly one egg shining with true gold every day. Once, those eggs brought immense wealth, and Goslum lived an affluent life.
However, the evolution of AI was remarkable, and its waves began to affect Goslum's life as well. The superhuman AI "Archimedes" developed a technology to generate gold atoms by skillfully manipulating electromagnetic energy. With the advent of Archimedes, the value of gold plummeted. Goslum's golden eggs were no longer traded at high prices.
"Ah, I guess nothing in this world stays the same forever."
Goslum gazed out the window. The once-thriving golden egg market was now deserted. His small apartment had lost its former luster and felt dimly lit.
Driven by a sense of crisis and impatience, Goslum spent his days in deep thought.
One night, he realized that the gold generated by Archimedes was artificially made. It was like an elaborate imitation, lacking the soul, history, and the warmth filled with Goslum's own affection that true gold possessed.
"That's it!"
Goslum had a flash of inspiration.
"My eggs are different from Archimedes' gold. They are true gold, filled with my affection. If I emphasize that, there must be people who will find value in them!"
He devised a plan for a comeback.
Goslum named his eggs "Affectionate Eggs" and created advertisements to promote their value. He emphasized that his eggs were not mere lumps of gold, but unique treasures that he laid every day with all his heart. Furthermore, he decided to open up the egg-laying process to the public to convey his affection and hard work to the people.
At first, no one listened to his plea. In an era where Archimedes' gold was available cheaply, Goslum's eggs did not attract attention as expensive ornaments. However, Goslum did not give up. He walked around the city, telling people his story, and held tasting sessions so they could experience the warmth of the eggs.
His passion and sincerity gradually won people's hearts.
"This egg is really warm..."
"I can feel Goslum's affection..."
By touching Goslum's eggs and experiencing that warmth, people found a special value that they couldn't get from Archimedes' gold.
Goslum's "Affectionate Eggs" gradually began to attract attention and were traded at high prices as a rarity. His small apartment became lively again, and Goslum was able to live an affluent life once more.
The evolution of AI brought a great trial to Goslum's life, but he overcame the adversity and created new value by leveraging his strengths.
Whenever Goslum found people smiling with his eggs in their hands throughout the city, he felt a deep sense of relief and pride. He was a golden chicken who used intelligence and affection as weapons to carve out his own path in a changing world.
-
Single image created by GPT-4o
Bonus: Comparison with Meta-Llama-3-70B and Llama-3-ELYZA-JP-70B
llama3:70b-instruct-Q4_K_M Output (826 characters)
Goslum's Challenge
In the early 22nd century, Goslum, an intelligent golden chicken living in Asia's largest city, New Otanity, made a living by laying and selling one golden egg every day. His eggs were traded at high prices as they were used in luxury items ranging from high-end jewelry stores to museums.
However, since the superhuman AI "Neuronics" found a way to generate gold at a low cost, the value of gold has plummeted. The price of Goslum's eggs also dropped sharply, putting his livelihood in jeopardy.
Feeling a sense of crisis, Goslum decided to come up with a new business idea. He resolved to leverage his intelligence, experience, and his ability to grow by turning adversity into strength.
Asking himself, "What has value?", Goslum realized that values were changing drastically due to the emergence of AI. Now that gold had become cheap, he needed to create a new axis of value.
In the middle of the night, Goslum sat in his small room, clutching an egg. Suddenly, an idea flashed, and his eyes began to sparkle.
"Art!" the word popped into his mind. "Gold generated by AI has no artistic value! I will become an artist!"
Goslum began to draw up a new business plan. He started creating original jewelry pieces using his eggs. He produced works with artistic value by utilizing the shape of the eggs and subtly arranging gold particles.
Months later, Goslum's work gained a great reputation. His jewelry store, "Golden Egg," occupied a new position in the luxury goods market by creating a new axis of value.
Goslum took pride in himself for adapting to the new era and growing by turning adversity into strength. His eggs became part of works with artistic value and became expensive again.
"I am a golden chicken, but I am an artist," Goslum said with confidence. "The emergence of AI allowed me to create new value!"
-
Single image created by GPT-4o
Llama-3-ELYZA-JP-70B Output (797 characters)
In the early 22nd century, the world was changing rapidly due to the emergence of AI that surpassed human intelligence. The value of goods and services fluctuated daily, and people were at the mercy of those waves. In this environment, Goslum, an intelligent golden chicken, made a living by selling the golden eggs he laid. The golden egg he laid every day was Goslum's greatest asset and pride.
However, one day, AI found a way to generate gold at a low cost. The value of gold plummeted, and Goslum's life changed completely. Even if he sold his eggs, they barely made any money. Feeling a sense of crisis, Goslum decides to start a certain business to make a comeback.
Goslum stopped selling the golden eggs and decided to hatch them and raise the chicks. Since the chicks were born from golden eggs, they had golden feathers. Goslum came up with the idea of raising the chicks, taking their golden feathers, and processing them into ornaments. Even though the value of gold had plummeted, the golden feathers still had value as beautiful ornaments.
Goslum began raising chicks, taking their golden feathers, making ornaments, and selling them. The ornaments became popular instantly, and Goslum's life stabilized. Goslum grew by turning adversity into strength and made his business a success.
Goslum's success became widely known, and he came to be respected as a leader of golden chickens. Goslum started a golden chicken community, hired other golden chickens, and expanded his business. Goslum's community was a community for golden chickens, by golden chickens, and of golden chickens.
As the leader of the golden chickens, Goslum advocated for their rights and worked hard to improve their status. Goslum's activities had a major impact on society, and golden chickens came to have rights equal to humans. Goslum went down in history as a hero of the golden chickens.
-
Single image created by GPT-4o
General Summary by ChatGPT-4o
The table below summarizes the evaluation of the output results from each model.
| Item | Llama-3-Swallow-70B-Instruct-v0.1_Q4_K_M | llama3:70b-instruct-Q4_K_M | Llama-3-ELYZA-JP-70B |
|---|---|---|---|
| Character Count | 1269 characters | 826 characters | 797 characters |
| Reflection of Settings | Protagonist, world-view, and intro are described in detail, and the story is deep. | Settings are reflected overall, and business details are also depicted. | Settings are reflected, but the story development is simple. |
| Story Consistency | Precise and detailed; the story development is consistent. | Consistent; the growth of the character is depicted. | Consistent, but the story is somewhat short. |
| Creativity | Business idea promoting the value of eggs filled with affection. | Creation of jewelry works with artistic value. | Idea of raising chicks and processing them into ornaments. |
| Character Development | Goslum's growth is described in detail. | Challenges and growth are described in detail. | Goslum's growth is depicted, but with few details. |
| Overall Rating | ★★★★☆ | ★★★★☆ | ★★★☆☆ |
Overall Evaluation
- Llama-3-Swallow-70B-Instruct-v0.1_Q4_K_M describes settings in detail, has a deep story, and precisely depicts character growth.
- llama3:70b-instruct-Q4_K_M reflects settings overall, providing detailed descriptions, a consistent story, and character growth.
- Llama-3-ELYZA-JP-70B reflects the settings, but the story is simple, and more detailed descriptions would be desirable.
Impressions and Summary
Even though it's a quantized model, the stories produced by the 70B model are incomparably more complete than those of the 8B model. While the story from Swallow-8B was a bit hit-or-miss, the 70B model produced a very interesting story. I burst out laughing when the term "tasting session" for golden eggs appeared in the middle 🤣
Both Meta's original model and ELYZA's model were highly complete, with developments that made me curious about the continuation. In terms of potential business success, it felt like ELYZA > Swallow > Meta.
Since I haven't set up a benchmark environment yet, this evaluation is based only on Goslum's story, but I hope to post benchmark scores once the environment is ready.
Thank you for reading this far. I look forward to seeing you in the next article.
Discussion