Makedeco Advent Calendar Special Feature: 2023 Year-end Roundtable

As a special project for the Makedeco Advent Calendar 2023, the Makedeco operation members tomo/yoshiso/UKI/hio held a year-end roundtable discussion. Each member prepared a single slide to look back on this year while viewing them. We hope you enjoy it.

We hope you enjoy it.

Makedeco Event Retrospective

For the first slide, we looked back at the events we held this year as part of our annual review.

About Event Participant Clusters

tomo: Looking at it briefly, what are your first impressions?

UKI: In terms of the audience, the deep tech area is where the most participants gathered. To understand the needs in more detail, should we take some kind of survey?

tomo: We used to take seminar surveys, but the response rate was poor, so we stopped. It's also quite a lot of work to tabulate them.

UKI: It's interesting to know (in terms of the audience layer) whether individual investors are listening or if people actually involved in fund management are coming to listen.

tomo: Following ABCD Forecast/DeepPortfolio, Daken-san's event had the highest attendance with 480 people. My sense is that there are about 300 regulars who always come, and then it fluctuates upwards depending on the theme.

hio: For the ChatGPT event, many people came because it was a trending topic.

tomo: For the ChatGPT event, about 200 people registered all at once on the day it was announced. I felt this was able to reach people in slightly different clusters.

UKI: ABCD Forecast/DeepPortfolio

UKI: I spent a lot of time trying to improve performance with ABCD Forecast, but in the end, it didn't work out and I gave up. I can't reproduce it well. There might be some missing information regarding function F.

tomo: After all is said and done, cutting-edge technology events attract the most people. I think many people are interested. The study session for ABCD Forecast/DeepPortfolio was the most popular one this year.

hio: J-Quants API Release Event

hio: I'm surprised that the J-Quants API release was just in April this year. The hackathon and the options event, as well as other events related to the J-Quants API, were also great.

tomo: Judging the hackathon was fun. In terms of technical skill, many people were touching it for the very first time, but conversely, that was good. Their ideas were really fresh. However, it took a huge amount of time to prepare, yet it had the lowest number of participants (lol).

yoshiso: imos-san AMA

yoshiso: I personally found the imos-san AMA and ABCD Forecast/DeepPortfolio to be the most interesting.

tomo: Since UKI and yoshiso enjoyed those, maybe we should dig deeper into deep tech next year?

UKI: You want to go in that direction?

tomo: This year, I tried things like project management events because I wasn't sure which areas would resonate, so I experimented quite a bit. I wanted to see where the demand lies.

yoshiso: We did cover a lot of different themes this year.

tomo: On the other hand, there might be people who are happy we handle various topics. It feels like we're providing a point of contact for unknown domains.

yoshiso: That's true.

Events That Were Difficult to Prepare

tomo: The event that was most difficult to prepare for was definitely the ABCD Forecast one. Since that required corporate permission, we had to hold preliminary meetings to explain to imos-san and Ito-san's companies how to handle it. We had to explain that Makedeco is a volunteer group and doesn't have a direct business connection.

yoshiso: Doing that kind of groundwork for an event actually adds a lot of value.

tomo's Review

How to Use Generative AI?

tomo: November and December got quite busy, so since it's now December, I first looked back with the goal of hearing various opinions for planning future Makedeco events. Excluding November/December, we planned one event a month, exactly 10 times. I think we had a unique flavor with things like the deep tech series and the AMAs with Daken-san and imos-san.
Also, looking back on this year, I feel that Generative AI was just too incredible. I've touched language models once before for work, but they never really worked properly. For example, even when trying to infer sentiment, they didn't read the context at all and felt more like they were calculating based on word distance. That was fine for what it was.
However, ChatGPT's vanilla performance is so high that in specific domains, it really makes you wonder what all the previous effort and techniques were for. But when it comes to actual fund management, there's the problem that backtesting is too difficult.

UKI: Backtesting is truly difficult.

yoshiso: It's seriously difficult.

tomo: With the latest models like GPT-4 Turbo, the training period has been extended to April 2023, so there's effectively only half a year that the model doesn't know. However, looking at actual surveys from financial institutions, people are starting to use ChatGPT quite a bit for primary screening of disclosure materials—the kind of work usually given to new hires. If further breakthroughs occur next year, it might become smarter than I am. I'd like to know what kind of usage everyone is thinking about for such a time.

UKI: Smarter than yourself?

tomo: Smarter than myself.

UKI's Thoughts on the Use of Generative AI

UKI: In my view, rather than having ChatGPT discover alpha, I want to use it for building bots.

yoshiso: You mean having it write the code?

UKI: Exactly. For example, in arbitrage, if the profit is only a few yen, the cost of manual effort makes it hard to go after. But if generative AI gets smarter and can build bots at an assistant level, we can aggressively pursue those opportunities. The break-even point for profit changes. I think about how to release things as quickly as possible.
Also, this year I took the LLM course held by the Matsuo Lab and tried fine-tuning an LLM. I wanted to teach it bot coding. However, fine-tuning is difficult. I couldn't get it to work well at all. When I took the course, I wanted to create an assistant for bot creation, but I had to give up because fine-tuning was so hard.

tomo: How about you, hio? I assume you're using Copilot; what's it like?

hio: Writing code yourself is tiring, isn't it? It would be great if it could write code for us 24 hours a day. We review the output, fix it a bit, and then the generative AI writes more code. If we can create a cycle like that, productivity would skyrocket.

tomo: What specifically is GitHub Copilot good at?

hio: It autocompletes things very well when creating objects for compatibility with other data or performing numerical calculations for API return values while writing test cases. For those kinds of things, like writing internal formulas or fine-grained function-level tasks, you can rely on it completely. However, it can't handle the overall direction.

tomo: For that, it might be better to ask Copilot Chat.

hio: Personally, I really want to build an analysis platform, and I'd be happy if generative AI could build those parts aggressively.

tomo: What about yoshiso?

yoshiso: I'm also looking forward to using it in ways similar to UKI. I think humans are stronger in specialized knowledge and fields. Rather than coming up with ideas, I'd appreciate support in the execution parts or increasing the number of moves. However, I'm not using it at all right now. Primarily, I don't want to send my code outside. Until a breakthrough occurs in that regard, I don't have much motivation.

tomo: Certainly, from the perspective of protecting the core parts, you wouldn't want data to be sent out.

UKI: I don't really expect investment ideas to come from ChatGPT. Essentially, generative AI is about probability distributions, so it often brings back content it has seen and learned somewhere else, rather than being emergent. Due to those characteristics, while it can be used for support, I don't think it's something that produces definitive ideas.
For example, I tried to have ChatGPT write Bitcoin transaction processing in Python, but it couldn't write it at all, perhaps because there is little literature on it.

yoshiso's Review

About ABCD Forecast

yoshiso: Crypto and Numerai models were good this year. For Numerai, the NRM has roughly doubled.

UKI: I wish NMR itself would go up more...

yoshiso: Exactly, that part is a bit disappointing. For crypto, it's currently around 55% to 60%.

UKI: Excellent, excellent.

yoshiso: Since crypto is already reaching asset capacity limits, I'm trying to implement stock and macro strategies. I've been monitoring a stock long-short model that I've been building since spring, before summer. It performed well in September and October, but I'm struggling a bit since releasing it. I haven't made large profits here yet. I also tested ABCD Forecast, but it seems tough as well.

UKI: ABCD Forecast was impossible.

tomo: So it's impossible for everyone.

yoshiso: Let's talk about this in detail. Probably, the design of the F function there is a specific transformation rule—it's likely a special function. I confirmed that it seems possible to build a portfolio much better than random, but it's nowhere near being an operational Sharpe-based strategy. I feel the issue is how to construct the F function, and it's not working well.
I feel like it won't work without filtering features that are effective for specific pairs. I strongly sense that we need to really master the data variation mechanism and feature filtering, but I couldn't dive that deep and gave up.

UKI: You revert with the inverse matrix, right? I thought there might be a problem where the models correlate with each other at that point, so I gave up there too. Even if you try hard to make the pair features independent, they get messy when reverting with the inverse matrix. I think that's the problem.

yoshiso: Right. Ultimately, if there's a bad model in one of the matrices, there's a possibility that everything breaks. If one bad pair is formed, it's a structure where the impact on all asset direction predictions isn't good. Personally, I thought an architecture that distributes and averages contributions linearly might be better. Like taking the average after many pairs are formed. I thought accuracy would be better if assets other than the pair didn't affect the prediction.

UKI: I gave up there as well.

About Deep Tech

yoshiso: I was trying various things with deep tech, but it takes an enormous amount of computation time, and GPUs are becoming hard to use in the cloud, so I didn't make much progress.

tomo: What new deep learning models did you try? Transformer-based?

yoshiso: I haven't tried Transformer-based ones at all. I'm mainly doing strategies that involve data augmentation. I'm trying various data augmentation strategies we discussed during imos-san's session. However, the data was too heavy, and I needed a massive amount of GPU power for proper evaluation. Just when I designed the architecture and was about to start, GPUs became unavailable due to Generative AI, so I was frustrated this year. I finally bit the bullet and started a company, so I plan to buy GPUs and run them heavily. Next year, I want to try various things using detailed fundamental data from the J-Quants API. I want to study that area of knowledge.

tomo: At yoshiso's scale of operation, it's better to do various things with multi-asset, isn't it?

yoshiso: Yes. I want to improve my Sharpe ratio.

tomo: What about FX?

yoshiso: Just a little bit for FX. I have one running live now and I'm watching it. But I expect the Sharpe ratio for FX to drop, so I think stocks are the top priority for now.

tomo: Since the yen is likely to trend stronger next year, holding exposure to US stocks might be tough. Building a strong model for Japanese stocks seems like the best approach.

yoshiso: Indeed.

UKI: Do you include currency hedging in your strategy? I've thought about it too, but it's difficult.

yoshiso: Not hedging, exactly. But I liquidated all my dollar-denominated assets when it hit 150. Except for what I use for operations, I converted everything to yen and gold. Yen hedging in FX is too painful due to interest rates, so I just moved back to yen.

hio's Review

About Numerai Models

hio: For me, it's just Numerai, but it's reached about 100%. It has doubled.

tomo: Excellent.

hio: For Numerai, I'm using a strategy of "beating it with a bundle of cash." Like running GPUs heavily. It seems no one else is using large models; my model takes about 3 days to train. My approach is to create models for all targets and do a lot of ensembling. It's simple, but it seems not many people go that far, so running GPUs heavily is working.

UKI: Is that a deep learning model?

hio: It's not deep learning, it's LGBM. But I increase the estimators and it takes about 7 minutes on an A100. I also try with about 20 seeds. Since I've built the codebase for Numerai, I can quickly run it again with different features. I'm short on machine power, so lately I've been battling with memory.

yoshiso: I'm using a relatively simple model with constraints.

hio: My regret for this year is that I haven't touched the J-Quants API, so I haven't done any hands-on work there. I want to do some hands-on work next year.

UKI: hio-san, aren't you going to write for the Advent Calendar?

tomo: In fact, you haven't even put out last year's yet.

hio: That's right. I still have last year's, and I haven't touched it. I want to try even larger models next year, so I thought it was great to hear that yoshiso is building a cluster.

uki's Review

uki's Market Analysis

UKI: I wrote here that the market conditions are good, but my own performance hasn't been great. The market itself was actually very good. The Nikkei Average soared, and cryptocurrencies kept rising. However, my Japanese stock long-short model was sluggish. Since I talk about various things at Makedeco, I wondered if my edge had vanished. But actually, my performance was also poor in 2021. So, I don't know if the edge is gone or if it's just a temporary slump. And as I think yoshiso also feels, it seems like the effectiveness of factors has weakened this year. Factor performance has been poor recently.

yoshiso: I feel that very strongly. There were many periods between 2018 and 2020 where the rotation was quite intense, so something like that might be happening.

UKI: I'm proceeding with system upgrades; I started with Lasso, then Ridge, and now it's a linear model. I've created a model that might be the final transformation, and I'm currently deciding when to release it. But if I release it when performance is bad, it's hard to judge whether the results are good or bad. Still, I'm planning to change the model anyway. It feels like the market open/close models are becoming simpler and simpler.

yoshiso: Value stocks kept rising this year, didn't they?

UKI: Exactly. And large-cap stocks also went up, so it was difficult.

yoshiso: Large-cap stocks have had an advantage since 2018, right?

UKI: Since the Nikkei grew, especially in the first half, and large-caps went up, from a factor perspective, I guess it was that kind of year. Crypto continued to rise, so that part was doing well. However, a bot I was running with high leverage blew up twice, and Hoheto-kun told me my leverage was too high. I always mess up. That ate into my profits. It's better not to use too much leverage. If it doesn't blow up, it recovers afterward, so I could end in the green, but once it blows up, it's over. It's better to run it slowly with low leverage.

tomo: How are you splitting your time between the two?

UKI: It depends on the period. I concentrate on either crypto bots or Japanese stocks depending on the time. Also, the topics of interest for next year are the Bitcoin ETF and the halving, wondering if the bubble is really coming. Should we do something for Makedeco?

tomo: If UKI-san and yoshiso did a crypto event, it would probably attract 2,000–3,000 people.

UKI: That might be a bit different from Makedeco's vibe, so I'll leave it to tomo-san. I'm not sure if filling Makedeco with crypto botters is the right move.

tomo: I want to grow the J-Quants API, so I think that's the priority. Especially since Inotoma-san and the others are in NY, I want us to liven things up in Japan.

yoshiso: Apparently some new data is coming out in December.
Note: TOPIX was expanded in December/January: https://qiita.com/j_quants/items/68ffe2383cd6c3b8f6e1

hio: It's index data, right? A PR came to GitHub.

tomo: You're right. A PR arrived today.

UKI: Also, I'm planning to make the Advent Calendar go viral with content even more interesting than last time, so stay tuned. I think it has a good chance.

Talk About Next Year's Events

tomo: Are there any events you want to do next year? Anyone specific you'd like to hear from?

UKI: I'd like to invite people like shinshin-san or katsu-san, who are actively working on stocks. But I don't know them personally.
In the past, stock system trading was divided into those who used commercial tools like Izanami ("system trading crowd") and those who analyzed using R or Python. However, it seems like the number of people in that system trading crowd is decreasing, and I have the impression that many people who were doing crypto bots are now moving slightly toward the stock side ("botter crowd"). If we're going to hear from someone, I feel like someone who wrote for the Advent Calendar would be good. Like botter_01-san. Since "botter" is in his name, he's definitely part of the botter crowd, right?

tomo: Definitely (lol).

hio: Tutorials using notebooks attract people, but they're not that interesting.

tomo: Well, it's because people like UKI-san and yoshiso-kun don't show up to those (lol).

UKI: By the way, tomo-san, what's your motivation for continuing Makedeco? Do you have an ultimate vision?

tomo: Well, I feel like if interesting people gather, that's enough for me.

UKI: Which means gathering around themes that we ourselves enjoy is the way to go.

tomo: Exactly. Like there aren't many opportunities to hear an AMA from imos-san, for example.

UKI: An event where we just gather and talk would be fine too. There might be a need for it. An online roundtable where we pick up comments and just let the conversation flow sounds good.

yoshiso: Sounds good. Like skimming through papers together.

UKI: Sounds good, sounds good.

tomo: Oh, right, let's do an event for the 1st anniversary of the J-Quants API release. I think use cases have accumulated by now, so we can probably invite various people.

hio: I also feel like inviting academic people. It would be interesting if they could explain what's hot in academia right now and which areas they are digging into in an easy-to-understand way.

UKI: I wonder who... in Japan, people from PFN or Nomura AM come to mind first. Although it seems Nakagawa-san has been finding it difficult to make appearances lately, so he might not come. I'm really glad we could do the AMA event last year.

tomo: Indeed.

UKI: Among famous people, maybe Mizuta-san from SPARX? But since his specialty is artificial markets, I can't predict how much the audience would be interested. Personally, I most want to hear about topics like "generating time series and training on them," but that leads back to imos-san.

tomo: I feel like I've gotten some ideas for next year after hearing all this.

UKI: Oh, really?

tomo: Yes, yes. I want to keep things lively next year too. Well, look at the time. I wonder if this can be summarized...

yoshiso: I wonder if it will wrap up nicely...

tomo: Well, we'll manage somehow.

So, did we manage somehow? If there's a lot of good feedback, I'm thinking of planning more things like this, so please let us know what you think on Twitter or Discord.