How a 'God-Mode' Engineer Crashed a Production Server in 0.1 Seconds

Horror Level: Low
Probability of recurrence: Medium

What is an "Ore TUEEE" (I'm OP) Engineer?

※ This chapter contains a lot of personal bias and prejudice, so please read it as a joke.

An "Ore TUEEE" engineer is an engineer who entered the workforce without being able to discard a certain kind of "Chuunibyou" (eighth-grader syndrome) values, represented by lines like:
"Well, any kind of security has absolutely nothing to do with me" (Source: BLOODY MONDAY)
"Wizard-class hacker" (Source: ne0;lation)

Ore TUEEE engineers are basically vain and arrogant. They are usually intoxicated by their own development environment. They live to increase productivity, speed up tasks, and be looked up to by others.
To put it bluntly, an Ore TUEEE engineer is someone who wants to play the role of a hacker and doesn't want to use the same tools as everyone else.

Ore TUEEE engineers often have their work PC (client PC) fully running on Linux. However, just installing Linux only makes you "Ore TUE" level, so they go with Arch Linux. This elevates them to "Ore TUEE." Furthermore, they set their desktop environment to a tiling window manager like i3 (sway, awesome). Now, they have finally become an Ore TUEEE engineer. That's "Ore TU-Triple-E."
Ore TUEEE engineers basically live inside the terminal. They don't use the mouse much. Their keyboard is usually an HHKB, Kinesis Advantage2, Realforce, a Thinkpad external, or a custom-built one.

Yes, that's me.

Ore TUEEE engineers are always conscious of efficiency, so they handle things with the minimum number of moves.
But even for someone like that, when things go wrong, they really go wrong—it's a scary world.
The failure is right behind you.

Record Leading Up to the Incident

An Ore TUEEE engineer's day begins with package updates. This is because they love new things. Using new things also increases the probability of encountering bugs. I hate bugs I create myself, but I love bugs created by others. When something breaks, they say "Are you serious?" but their eyes are sparkling.

Let's see what's updated today by running the Arch Linux update command: yay -Syyu.
💬 (Oh, looks like there's a kernel update today. Version 5.8 already? Linus is working hard.)

After that, I efficiently knock out work issues. Since there was a task that needed to be done on the production server, I opened a new tmux window and logged in via SSH to start working.
💬 (I want to make the shell startup a bit faster. Oh, and I want to automate this part around here...)

Since the work reached a stopping point for now, I decided to peek at the vim-jp slack.
💬 (They're getting excited over which default maps can be overridden today. These guys really love this stuff.)

💬 (Ah, right, the kernel update came in today. The next task looks like it'll be long, and this is a good break point, so I'll just reboot for now.)

To exit in i3 (the Linux desktop manager), you press Win+Shift+e and then wait for the message to appear:

Press r to reboot.
💬 (That's so tedious!)

Well, I'll just reboot normally from the terminal right there.
sudo reboot and... enter.
Huh? Wait? It's not rebooting??

Yes, the one that rebooted was the production server.

Why did the tragedy occur?

It was a common, silly mistake of simply typing a command in the wrong place. I was trying to be careful, but I got caught out just as my concentration wavered. Although the client prompt and the production prompt looked different, it wasn't prominent enough to prevent the mistake (the command and Enter were already typed with god-speed before I could even read the text).

The causes, I believe, were overconfidence and habituation.
In the case of Arch Linux especially, the kernel is updated frequently, so I reboot my client PC quite often. Moreover, because I performed it at a half-baked timing—just because I felt I reached a good stopping point—I ended up executing it over SSH while the session with the production server was still active.

Recovery Work

Fortunately, the server that I rebooted ran programs that started at fixed intervals, like cron, so it didn't lead to a major disaster.
After confirming the OS reboot and service startup, I wrote a report (well, a letter of apology, so to speak), and the situation settled down.

What should be done to prevent the tragedy from ever happening again?

Change the terminal background color for production servers to make them easily recognizable (Essential)

Pattern: Adding a light configuration on the production server side

To prevent others from making mistakes, I think it's best to add a little extra effort on the server side.
Make sure to do at least one of these.

Make the prompt look flashy

Change the PS1 in ~/.bashrc on the SSH destination to make the prompt clearly understandable. Since you might not notice subtle changes, it's good to make it stand out immensely.

↓ You could make it more stylish, but standing out is what matters!
For how to apply colors, refer to this.

PS1="[\e[37;41;1m\]prrrrrrrrrrrrrrrrrodunction\[\e[37;49m\] \u@\h \W]\$ "

Sample

Automatically change the background

In ~/.bashrc:

echo -e "\033]11;#400000\a"

In ~/.bash_logout:

reset

Set these to forcibly change the terminal background color upon logging in.

Pattern: Taking care of it on your own PC because the production server is a "no-touch" zone

For those living in environments where adding unnecessary(?) configurations on the server side is not permitted! You might not be able to save others, but at least protect yourself.

While there are ways to dynamically change the background color of the terminal you're working in, my personal recommendation is to change the terminal application itself.

How to change the terminal application itself

Change the background settings of a non-main terminal application so that it has a glaringly obvious color.

How to dynamically change the terminal background color

• How to dynamically change pane colors in tmux: https://bacchi.me/linux/change-terminal-bgcolor/
• How to dynamically change it by overriding ssh with a function (please add this to your ~/.bashrc, etc.):

function ssh () {
  echo -e "\033]11;#400000\a"
  command ssh "$@"
  reset
}

How to rewrite PS1 before starting work

I don't highly recommend it, but there is also a way to reset the prompt after SSHing but before starting work. This temporarily changes only your prompt without affecting others. Of course, it will return to its original state once you log out and log back in.

You could save it as a snippet in your OS clipboard. However, you need to be careful because it's meaningless if you forget to type it.

The PS1 setting is the same as in Make the prompt look flashy.

Other things you can do

• When rebooting your client PC, use the GUI features. Don't simply run sudo reboot.
• Minimize the time spent working on production servers. Log out properly once the work is done.
• Understand that there are dangerous commands even among those that aren't as obviously scary as rm or dd.

Things I want you to do even more

I believe it's essential to set up IaC (Infrastructure as Code), CD (Continuous Delivery), and system monitoring environments so that you don't have to SSH into production. If you are SSHing into production right now, you should build a mechanism to move away from that as soon as possible.
However, depending on the size of the company or the environment, these things might not be in place or might be neglected, so in those cases, please be very careful...!

To continue being an "Ore TUEEE" Engineer

It depends on your usual behavior, but I feel like people around me are thinking, "Even though he's 'Ore TUEEE,' he still makes mistakes like this." Since I have a lot of pride, it hurts a lot. If you want to remain an Ore TUEEE engineer, you should refine your work environment so that you never make careless mistakes...

So that you don't end up saying "Did I do something again?" in its original sense...