iTranslated by AI

The content below is an AI-generated translation. This is an experimental feature, and may contain errors. View original article
💰

Building a Portfolio Generation App Using Local LLMs

に公開

Introduction

Who is this article for?

  • Those interested in Generative AI
  • Those interested in local LLMs
  • Those interested in asset management
Development Environment
Mac Studio (M2 Ultra 128GB) + Claude MCP (filesystem + git)

Overview

I am currently working on a small web app, and as an intermediate result, I created a portfolio generation app using local LLMs.

In this article, I will provide a brief introduction to this app and summarize the knowledge gained during development as a memorandum.

Regarding the "small app," please look forward to it once it's finished. This article focuses on the development leading up to the portfolio generation app.

App Introduction

Before talking about the portfolio generation app, I'll briefly introduce an LLM character management app I made using Claude MCP.

https://x.com/gosrum/status/1863586577739624902

This app is also one of the intermediate products, but the system is very simple as shown below.

Just by entering character settings, the LLM visualizes inferred statuses, so this alone is quite interesting.

This time, I further implemented a part that creates an asset portfolio from the created character's information. Here is the current app screen.

Well, the part implemented this time is just completion and visualization, so what it does is no different from the LLM character management app, but I faced various difficulties and learned a lot during the development process. I will discuss these below.

Difficulties in Development

Now, from here, I will mainly focus on the points I struggled with during development.

Development Environment

First, the development environment was primarily built using Claude Desktop on Mac with Claude MCP + filesystem + git. The reason is simple: I wanted to proceed with development semi-automatically using Claude's power without paying as much in API fees as possible.

As I mentioned in the following article, Claude MCP is honestly inferior to Cline in many respects.

https://zenn.dev/robustonian/articles/claude_mcp_filesystem_git

Furthermore, with recent updates and the use of a fork called Roo-Cline, the level of freedom and completeness has improved even more. On the other hand, Claude Desktop doesn't feel like it's evolving particularly much.

Even so, the reason I stick with Claude MCP is that I can use Claude to its full potential for just the $20 subscription fee.

Difficulties and Solutions

Below is a summary of the difficulties encountered during development through Claude MCP and my own solutions.

Difficulty Solution
Claude's habit of cutting corners
(omitting code)
Address through prompts (makes it somewhat less likely to cut corners)
"Omitting code is a cardinal sin."
"Never omit code; output the complete code."
"Be careful never to lose existing functionality, and split files carefully so that dependencies are clear."
Rate limits Regularly update README.md
Manage it as Knowledge in a Project, and save tokens by starting from a new conversation as much as possible
Breaking changes (occur rarely) Address with git
Incorporating very recent features like LLM-related ones Include specifications and sample code in the prompt

Failure Stories

Breaking Changes

During development, there was a case where the character management screen changed significantly.

Well, this isn't bad in its own way, but personally, I prefer radar charts, so I rejected it. However, with Claude MCP, it is quite difficult to revert changes once they have been made. The reasons are as follows:

  • Claude Desktop cannot handle ESLint errors, making the debugging process tedious.

    • To be precise, it can handle them if you pass the error messages, but when there are large-scale modifications, debugging work caused by a massive amount of ESLint errors occurs.
    • On the other hand, with Cline, perhaps because it uses VS Code or Cursor, it notices and fixes ESLint errors even without being explicitly given the error messages.
  • Even if I tell it to "revert just that part" after various debugging sessions, it can only go back to a halfway point, so I eventually decided to abandon that branch.

The Dilemma: One-shot with Detailed Specs vs. Step-by-Step

Initially, I tried to create everything at once by providing overall specifications detailed down to the specifics.
Then, Claude started generating files one after another.

"New files are being created one after another! This is wonderful!" I thought at first, but what started after a set of files were ready was a long, long debugging session.

To use a concept from Made in Abyss as an example, "many files being generated" = "the layers of the Abyss getting deeper," and "debugging work" = "the Strains of Ascending." It is terrifying that you cannot see the output until debugging is complete and the frontend starts up successfully.

It might be fine when creating from scratch, but if you do this from a halfway-finished project, you end up with a codebase that no longer retains its original form due to the Strains of Ascending.

As a result, this screen was created.

At first glance, it is an exciting and wonderful page, but in reality, only the appearance is done, and nothing is finished inside. In addition to the breaking changes mentioned above, there were mysterious items like skills that were not in the specifications.

Well, certainly, it would be quite interesting if all of this could be implemented, but it is far from what I wanted to create. If this weren't just a mockup but had been generated in a form where an app exceeding my imagination actually worked, it would have been fine... However, if I were to actually implement this, I would need to consider additional specific specifications, and it seems like more than 10 times the debugging work would be required. After thinking about various things, I gave up on continuing development on this branch.

So, after that, I changed my policy to adding functions little by little, step-by-step. At least at my current level, I recognize this as the optimal approach.

Other Observations

  • pydantic_ai can only be used with models supporting Tools

    • Initially, I was using pydantic_ai for learning purposes to generate investment policies, rationales, and portfolios, but I realized it only supports certain models that can use Tools.
    • Therefore, I decided to use Ollama's structured output, which has no model dependency.
  • In backend development, you should use a language you can read and write yourself to some extent

    • Unless you want to study a programming language or deploy on Vercel, it's better to use a language you're somewhat familiar with if you just want to build an app.
    • This is because if you leave everything to Claude, work will stall when Claude can't handle something or when you're in a timeframe where Claude is unavailable.
  • You should align your understanding with Claude before proceeding (especially if things aren't going as specified)

    • For example, if there's a part of the UI or design you don't like, input an image and ask, "Do you know what I don't like about this?" If Claude guesses correctly, say "Yes"; if wrong, say "No" and tell it the correct answer. By doing this, the probability of Claude fixing exactly what I want fixed—rather than what it thinks should be fixed—increased.
  • Tips for structured output

    • This is a basic point, but I'm writing it as a reminder because I failed to do this when I made the LLM Werewolf game.
    • When having an LLM make decisions or evaluations, the accuracy improves if you ask it to state the policy or reasoning first before making the decision (or giving a score).

Summary

In this article, I introduced a portfolio generation app using local LLMs and summarized the lessons learned during development using Claude MCP + filesystem.
I had never really thought about the frontend before, but once I got a taste of it, I started getting particular about it, and my time just melted away (lol).

Well, since the local LLM will mostly play an active role on the backend of the app I'm currently making, I want to proceed with development while being careful not to fall too deep into the frontend swamp. What I really want to build starts from here, so I hope to get it into shape during the year-end/New Year holidays and the three-day weekend following that.

This will be my last article for this year. Happy New Year to everyone!

Thank you for reading until the end. I look forward to your continued support in the next one.

Bonus: Best Portfolios by Model

Here, as a bonus, I had several models create portfolios for a virtual character with a setting called the "Philosopher of Capitalism," who has a complete understanding of capitalism. Not only the portfolios but also the investment rationales show the personality of each model, which is interesting.

In terms of investment philosophy and portfolios, I personally prefer gemma-2-27b-it. Also, it might be due to the prompt influence, but many other models were making meta-statements about their own parameters. Furthermore, for some reason, many LLMs had the idea that a high crime coefficient leads to leveraged investment (though leveraged investment itself is certainly not a crime, lol).

  • gemma-2-27b-it.f16.Q8_0
    alt text

  • qwen2.5:32b-instruct-q4_K_M

alt text

  • phi-4:f16

alt text

  • command-r-plus:104b-08-2024-q4_K_M

alt text

  • Athene-V2-Chat:Q4_K_M
    alt text

Discussion