iTranslated by AI
What happens when PlanetScale reaches its limits
Sudden Anomalies
One day, PlanetScale Insights suddenly began reporting unsettling Anomalies.
Looking at the data, I saw an unsettling spike in load lasting about 5 to 6 minutes at the top and bottom of every hour (:00 and :30)... My intuition was that someone was hitting the API without permission and causing the load.
Investigation
In any case, I decided to investigate to find proof. I thought it might be a batch job, but I couldn't find any batches scheduled to run every 30 minutes.
I recalled that you could see queries in PlanetScale's Anomaly logs, so I checked. ...However, perhaps because the duration was too short, unfortunately, no queries were displayed. I had no choice but to look at the general slow queries and server-side logs.
But even after looking at the server-side request logs, I couldn't find any requests that significantly increased during those specific times. I also checked the slow queries, but they were the usual suspects; I didn't see anything particularly new or strange that increased only during those windows... What exactly is going on?
I had expected to resolve this quickly with some access restrictions, but I was stuck with no clear path to a solution.
When I tried accessing the site during those periods, it felt quite heavy, and reports of slowness started coming in from users as well.
I tried fixing several suspicious-looking areas as a test, but it didn't solve anything at all...
I started to wonder if it might be an issue with PlanetScale itself, wanting to shift the blame.
Upgrade
As a final resort, I began considering a plan upgrade.
Just to be sure, I checked if I could upgrade without downtime.
Changing cluster sizes is a seamless operation that requires no downtime.
I see, it seems fine.
Furthermore, as I read on, I found the following description:
As a rule of thumb, if CPU utilization is consistently at or near 100% over a period of time, increasing the cluster size may be beneficial. Conversely, if CPU utilization stays below 50% for a period, it may be possible to decrease the size. You can monitor CPU utilization by clicking on your database, clicking "Primary" in the architecture diagram, and referring to the charts under "Metrics & Performance."
...Wait a second, now that I think about it, there was a graph that caught my eye while I was looking at the Anomalies.
It's totally hitting 100%.
I checked the status of the Primary as well... and that was also at 100%. Since the upgrade only increased the cost by about 3,000 yen, I upgraded immediately.
As a result, it was resolved.
Summary
Yes, in the end, it was simply a case of the DB server running out of resources. I wonder why it resulted in such mysterious graphs every 30 minutes... If it had been pegged at 100% all the time, I would have noticed it immediately...
In any case, I've decided to work on gradually improving the parts causing the load that I've been neglecting (the next upgrade would cost about 7,000 yen more, so I really don't want to raise it again).
Discussion