Developing an OSS with Over 250,000 Downloads Using Vibe Coding: I Did, But... ── Rethinking AI and Human Roles as Collaboration

Introduction

I am @dyoshikawa.

Since around June 2025, I have been working on a project called Rulesync, an OSS tool published using Vibe Coding with Claude Code.

Through this experience, I was able to freely experiment with various techniques using Claude Code, so I would like to share some tips on AI coding in general.

On the other hand, as a downside of Vibe Coding (with too little human intervention), there were times when the codebase and documentation fell into disarray, causing development to stall midway. I will also discuss the lessons learned from this process and how I managed to restore the project to a developable state.

Finally, I will touch on topics such as the role of humans, the role of AI, and the future of software development.

Rulesync: Centralized Management of Configuration Files for Major AI Coding Tools

First, let me briefly introduce the tool I developed and published, Rulesync.

Please take a look at this X post:

https://x.com/suin/status/1943203866117574699

Things like cursor/rules, CLAUDE.md, github/custom-instructions.md… I’m sure many of you have thoughts on these.

As new AI coding tools are announced one after another and trends change frequently, we as users face the pain of having to support various file formats. That’s what leads to meme-style posts like the one above.

Recently, there has been a movement toward standardization called AGENTS.md, which aims to unify files known as Rules or Memories and alleviate this pain.

However, AGENTS.md has not yet been uniformly adopted across all tools, and the situation where files for Custom Slash Commands, Subagents, MCP integration, and Ignore are separated per tool still persists. Furthermore, new categories of files may emerge in the future.

Therefore, I developed Rulesync to attempt to solve this problem.

https://github.com/dyoshikawa/rulesync

With Rulesync, you can define rules, ignore files, MCP, commands, subagents… all in one place, and it can generate everything in the formats required by various AI tools at once. For rules, it supports splitting files and also generating AGENTS.md.

Moreover, regarding commands and subagents implemented in tools like Claude Code—these are extremely useful and, in my opinion, essential features—their implementation status varies across different AI coding tools.
To address this, Rulesync provides a simulation feature for Commands and Subagents. For example, if you usually make full use of commands and subagents in Claude Code, and you want to use GitHub Copilot alongside it, Rulesync aims to recreate a development experience where commands and subagents can be invoked and utilized in the same way.

Rulesync was developed using Claude Code (Max plan) and Cursor editor.

It is written in TypeScript. The choice of language was simply because I am familiar with it, and the Node.js runtime is highly prevalent on developer machines, making it suitable for distribution.

Rulesync received more attention than I expected, with over 300 GitHub stars, a total of 250,000 npm downloads, and 20,000+ weekly downloads.

GitHub Star count

npm total downloads

npm weekly downloads

Originally, I started this project to solve my own pain, and it has also been adopted by the Zenn team.

Feel free to check out the Rulesync repository. If it can help solve your pain points, I’d be happy.

Starting Development with Vibe Coding

Now, let’s move on to the development process using AI.

Development of Rulesync started around June of this year in a style close to the original meaning of Vibe Coding. By “original meaning,” I mean the approach described in the following post:

https://x.com/karpathy/status/1886192184808149383

It’s a style where you develop by simply describing the desired specifications without looking at the code much, and when bugs occur, you just describe the bug’s behavior and let the AI fix it, without verifying the implementation yourself.

I adopted this approach because I started it as a side project without much time to invest, and I was also curious to see how far I could go without actually touching the code.

I tried not to look at the code as much as possible, but I did skim through it to ensure no malicious code was being generated. That's because, in theory, AI data poisoning is a concern.

https://www.cloudflare.com/ja-jp/learning/ai/data-poisoning/

In practice, I haven’t seen any actual cases of this, and I think the probability is extremely low unless there is external influence like prompt injection.

Going as Far as Possible Without Looking at Code

To go as far as possible without looking at the code, I not only used Claude Code but also set up surrounding mechanisms as much as I could.

Here’s a list of what I can think of:

• Static analysis with types and linters
• Automated tests
• o3-search-mcp
• serena mcp
• knip
• Dev Containers + enabling bypass mode 👈
• AI code reviews 👈
• similarity 👈

I leveraged these. In this article, I will focus on the bottom three marked with 👈.

Dev Containers + Bypass Mode

First, the point of having the AI code in bypass mode.

In this article, enabling the --dangerously-skip-permissions option is referred to as bypass mode. It's an option with a name that makes you want to avoid it, but I consider it essential for improving the comfort of Vibe Coding.

Enabling bypass mode allows the AI to work without asking for user approval at all, which greatly improves the comfort of Vibe Coding. However, running it on the host machine is understandably scary. I set up a sandbox environment using Dev Containers, which are available in VSCode and Cursor, and run Claude Code inside that environment.

https://github.com/anthropics/claude-code/tree/main/.devcontainer

Anthropic has published sample Devcontainer configurations for Claude Code at the above link, so it's relatively easy. You can basically copy and paste the settings.

As an original touch, since I didn't want to type that long option every time, I set up an alias in ~/.zshrc so that just typing claude launches it in bypass mode.

.devcontainer/Dockerfile

RUN echo "alias claude=\"claude --dangerously-skip-permissions\"" >> ~/.zshrc

There is also a method to set permissions.defaultMode to bypassPermissions in claude/settings.json, but I chose this approach to avoid accidentally launching Claude in bypass mode when running the claude command on the host machine.

AI Code Reviews

Next, the point of having AI perform code reviews.

I have a Subagent perform the review and report the results. Then, I wanted all steps—deciding whether to merge based on the review results and actually merging—to be handled by Claude Code automatically. So I configured everything to be executed with a single Custom Slash Command.

Since it's in bypass mode, there is no intermediate confirmation or approval. Once the command is run, the human just waits.

For the review, I prepared a code-reviewer subagent like this. Its definition is to perform code reviews from the perspective of code tidiness.

.claude/agents/code-reviewer.md

---
name: code-reviewer
description: >-
  An agent used for comprehensive code reviews.
  Reviews are conducted based on general software engineering principles
  such as DRY, SOLID, maintainability, and best practices.
model: opus
---
Reviews code from a general software engineering perspective.

- Adherence to the DRY principle
- Addition or updating of test code according to feature development
- Compliance with .claude/memories/coding-guides.md

Other general best practices.

I also prepared a separate security-reviewer subagent. Its definition is to perform reviews from a security perspective.

.claude/agents/security-reviewer.md

---
name: security-reviewer
description: >-
  An agent used for code reviews focused on security.
  It aims to detect vulnerabilities and malicious code in particular.
  This agent is only used when explicitly invoked by the user.
model: opus
---
Reviews code to identify vulnerabilities and malicious code.
If a GitHub PR URL is specified, it reviews that PR. If not specified, it reviews the PR associated with the current branch.

Note:
This project is a CLI tool used in the user's local environment. Therefore, the security considerations may differ from those of web applications used by many unspecified users. Please conduct a security review appropriate to the nature of the project.

To orchestrate these Subagents, I defined a judge-pr Command. It is named this way to execute all steps—reviewing, determining whether to merge, and merging—in a single command.

.claude/commands/judge-pr.md

---
description: Review the PR and merge if it passes.
---
target_pr = $ARGUMENTS
If target_pr is not specified, use the PR for the current branch.

First, check the GitHub status of the target PR. If the status is not "success", output an error and exit.
Next, execute the following in parallel:

- Call the code-reviewer subagent to review the code changes in $target_pr
- Call the security-reviewer subagent to review security issues in $target_pr

Combine the results from each subagent and report.
Then, if there are no issues with the code changes, call the pr-merger subagent to merge the PR.

I will omit the details, but I also prepared a separate pr-merger subagent.

Furthermore, I introduced a tool called similarity.

https://github.com/mizchi/similarity

similarity is an OSS tool published by mizchi. For TypeScript, there is an implementation called similarity-ts. By running the similarity-ts command, you can detect code with high similarity. It allows setting a threshold option; for example, threshold 0.85 will detect code with 85% or higher similarity.

By passing these detection results to Claude Code and having it plan and implement commonalization, it becomes possible to some extent to perform "refactoring without looking at the code."

Continuing with this development style, by around August 14th, at version v0.62.0, weekly downloads reached about 15,000, and GitHub stars were steadily increasing.

However... the Codebase Collapsed

For a moment, I thought, "Can we go anywhere with Vibe Coding like this?" But we hit a limit around v0.67.0, August 23rd.

Since the v0.1.0 release was on June 18th, we were able to proceed for about two months without looking at the code much. That itself is impressive. However, at this stage, issues started to arise: feature additions became unstable, bugs couldn't be fixed even when described, changes made things worse, and after hours of work, failing tests couldn't be passed. It got to the point where making changes with Claude Code felt scary.

While the accuracy of the AI model itself and the sophistication of the Agent are remarkable, and by solidifying peripheral tools like MCP and similarity, I was able to push the limits of Vibe Coding further. However, I strongly felt that this doesn't eliminate the limits themselves.

What Went Wrong?

What specifically went wrong?

You probably have a good idea—it essentially comes down to not providing instructions about the design. Without looking at the code, it's difficult to give design instructions.

Of course, this isn't the fault of the AI coding tools. The AI was only told things like "I want to add this feature" or "Fix this bug," and it simply executed those requests. This is the consequence of humans not taking responsibility for the design.

Will even this problem be solved by future AI evolution? One thing I wonder is whether it would be comfortable for users if the AI autonomously decided to refactor code. Having things done that weren't instructed could make the tool less convenient to use.

Also, there is no single correct answer for software design and structure. So from the AI vendor's perspective, training for this is probably more difficult compared to simply writing code that works according to specifications. Therefore, writing "clean code" seems to be a relatively weak area for AI.

Returning to a Developable State with Large-Scale Refactoring

Let me return to the actual development story.

Anyway, I needed to get from a state where development had stalled back to a developable state.

To do that, I decided to make large-scale changes to the design and structure. I did the coding myself, using Cursor Tab as an aid.

Having Claude Code do a large-scale refactoring all at once was difficult. Perhaps because the context was too large, it couldn't change the structure while preserving the specifications. I had also generated documentation, but when I checked it as part of the refactoring, I found descriptions of features that didn't exist and errors scattered throughout, so I rewrote it. This was likely due to a lack of frequent course correction and context management on the human side.

Since I performed a large-scale refactoring all at once, it was painful that the test harness didn't work. Although I had test files prepared, the changes involved deleting target files—and even entire directories—and rebuilding them, making it difficult to effectively use the test harness.

Test files existed, but since the target files (and entire directories) were being deleted...

There were some regressions and breaking changes along the way, but it was completed in about one to two weeks. Needless to say, for a business project, one should allocate more time and plan a gradual codebase migration. I proceeded this way because this is a personal OSS project and the time I could allocate was limited.

I did a lot of rewriting.

Development Style After Refactoring

This is the development style I adopted after the refactoring.

First, I give instructions while keeping the code structure in mind, and I carefully read the changes made by the AI.

Also, in reality, there are many cases where writing the code myself is faster than giving instructions in Japanese, so I mix in my own coding and the Cursor Tab style. When I stopped and thought about it, being fixated on an AI-agent-only development style was putting the cart before the horse—confusing the means with the end. We don't want to use AI; we just want to build good things quickly.

As for 0→1 coding, I of course use AI for this as well, but I feel it's still inconsistent—sometimes it produces more verbose code than if I wrote it myself. So I focused more on asking the AI to implement horizontal expansions of existing implementations.

As a result, there was a swing back toward a more traditional development style.

AI Coding Tips That Became Possible After Refactoring

Here are some AI coding tips I've been using after the refactoring.

Giving Instructions Based on Commit Hashes

A technique particularly useful for horizontally expanding implementations is giving instructions based on commit hashes.

Refer to commit hash xxxxxxxxxx and change path/to/file.ts using the same approach.

This kind of prompt allows the AI to easily and accurately understand the before/after of the part you want to focus on, since it can fetch the diff from the commit hash.

diff-analyzer Subagent

I also frequently define and use a subagent called diff-analyzer.

.claude/agents/diff-analyzer.md

---
name: diff-analyzer
description: Use this agent when you need to analyze the diff between the current branch and origin/main, and summarize the progress of work.
model: sonnet
---
1. Fetch the latest main branch using `git fetch origin/main`.
2. Get the diff between the current branch and main using `git diff origin/main...HEAD`.
3. Get the commit history of the current branch using `git log origin/main..HEAD --oneline`.
4. Summarize the work based on the diff and commit history.

Its content is simple: compare the main branch with the current branch and summarize. This is useful when you want the AI to grasp a broader span of changes beyond a single commit hash.

For example, you can use it with prompts like:

1. Call the diff-analyzer subagent to understand the current changes.
2. Add or modify test code according to the changes you understood.

Using the Claude Code (Agents) SDK

Next, the use of the Claude Code SDK (now renamed to Claude Agents SDK).

Using this makes it easier to balance detailed instructions with more autonomous AI coding.

I use a script like the following:

tasks.ts

export const model: "opus" | "sonnet" = "sonnet";

const task = ({filePath}: {filePath: string}) => {
  return `
Move to the fix-description branch and work there.

Refer to the changes in commit hash ec22e123912bfb3ec30d99abd8d1182fc17c0b62,
and add \`description?: string | undefined\` to ${filePath}, and also add the \`getDescription\` method.

Then, make sure all results of \`pnpm run cicheck\` PASS.
When done, commit and push all changes.
  `
}

export const tasks: string[] = [
  ...([
    "src/subagents/claudecode-subagent.ts",
    "src/subagents/codexcli-subagent.ts",
    "src/subagents/fix-subagent.ts",
    "src/subagents/cursor-subagent.ts",
  ].map((filePath) => task({filePath: filePath ?? ""}))),
  "call pr-handler to create a PR",
];

In tasks.ts, I define an array of target file paths and a task template that can interpolate the file name variable. This exports a list of tasks with the file list injected.

tasks.ts

import { query } from "@anthropic-ai/claude-code";
import { model, tasks } from "../tmp/tasks/tasks.ts";

const runClaudeCode = async (task: string) => {
  console.log("Prompt:", task);
  for await (const message of query({
    prompt: task,
    options: {
      abortController: new AbortController(),
      permissionMode: "bypassPermissions",
      model: model ?? "sonnet",
    },
  })) {
    if (message.type === "assistant") {
      console.log("Claude Code says:", message.message.content[0].text);
    }
  }
};

for (const task of tasks) {
  try {
    await runClaudeCode(task);
  } catch (error) {
    console.error(error);
  }
}

Then run-tasks.ts imports the task list exported from tasks.ts and sequentially kicks off Claude Code task execution in a loop.

This allows you to keep the context independent for each file's work, especially when making similar changes across multiple files. By isolating the context, you can avoid long context windows, resulting in a higher success rate compared to listing all files in a single prompt. It also has the advantage of making the input prompts DRY and eliminating the need to submit the next prompt manually after each task finishes, simply reducing human workload.

Effective AI Coding Requires Detailed Code Knowledge

These tips assume that you have an index of the codebase in your head.

It's also important that the structure is orderly. Keeping the processing in expected places makes it easier to construct prompts.

An engineer developing with many AI tools is more like a "site supervisor" than a single team member. In that sense, the style of letting AI write code without looking at it is like a "site supervisor who never visits the site." To create good products, I believe it's important to be a "site supervisor who is on-site"—one who also moves their own hands and grasps the details.

AI and the Present and Near Future of Software Development

Let me touch on the present and near future of AI and software development. I say "near future" because, honestly, I don't know what lies beyond that.

Take the internet, smartphones, or in the software development domain, public cloud, serverless architecture, declarative UIs like React… Only a few people recognized their value early on. For example, there were people saying React wouldn't become popular because JSX syntax looked weird, or that it was just a path PHP had already taken (I myself had similar thoughts).

On the other hand, there are many technologies that didn't end up as hyped as initially thought. From what I remember, discussions like "AWS will make infrastructure engineers unemployed" or "Firebase will make backend engineers unemployed" come to mind. When ChatGPT first appeared three years ago, there was also talk that engineers' jobs would disappear within a few years.

The Role of Humans, The Role of AI

As for my current thoughts: although people talk about "the role of humans" and "the role of AI," I feel these roles are actually inseparable. Where AI's capabilities end and where only humans can operate… in reality, this boundary is extremely vague.

In the development context, some argue that "humans should handle design and leave the rest to AI," but I think this is very close to the idea of separating design from manufacturing in software development.

Some may argue that source code itself is a design document, and we need to revisit the age-old debate about whether good design is possible without programming.

https://qiita.com/mdstoy/items/5510f94c9ed981cfbb85

For example, in a 2019 article, former Microsoft engineer Satoshi Nakajima stated that you cannot create good design and good specifications without writing programs yourself.

https://note.com/lifeisbeautiful/n/n1358c3529940

I can say this with confidence: there is an absolute rule that "no matter how talented an engineer is, they can never create good detailed specifications without writing programs themselves." All the excellent engineers I know are aware of this and put it into practice. Of course, they do rough design before starting to write programs, but experienced engineers know that what they have at that stage is nothing more than a "tentative design." Therefore, they don't waste time writing detailed design documents at that stage; instead, they immediately start creating programs (or prototypes).
(omitted)
To me, the entire process where "upstream engineers who don't write programs create detailed design documents, and downstream engineers do the coding" seems fundamentally wrong.

Software development involves many uncertain elements, and I personally agree with the idea that we need to actually get our hands dirty and explore while building things.

Furthermore, an engineer at OpenAI, an AI vendor, has also stated that understanding coding is necessary.

https://www.businessinsider.jp/article/2509-openai-engineer-advice-for-high-schoolers/

Based on my own experience, I believe that "effective prompting requires a high-resolution understanding of the codebase." Whether it's formulating specifications that balance implementation cost, maintainability, and richness of experience, or dividing modules through refactoring, an understanding of implementation details is inevitably necessary.

Will Upstream Processes and Communication Remain for Humans?

People often say, "Let AI do more and more work, and let humans do what only humans can do." In this context, discussions commonly suggest that areas like upstream processes, decision-making, and communication will remain as human work. However, I don't think these areas are necessarily safe havens for humans.

In the communication domain, we've seen news about attempts to replace call center workers with AI, and in the political domain, attempts to have AI make decisions.

https://www.nikkei.com/article/DGXZQOUB16BO80W5A610C2000000/

https://www.nikkei.com/article/DGXZQOGR12CYY0S5A910C2000000/

There's hardly any process more upstream than politics, is there?

Conclusion

My current thinking is this: rather than trying too hard to divide roles like "humans design, AI implements," I think it's better to have an awareness of "collaborating with AI" across all processes—specification formulation, communication, design, implementation, testing, and so on.

Furthermore, to collaborate with AI and understand its suggestions, a detailed understanding of the specifics is ultimately necessary. I believe this applies not only to engineers but likely to other professions as well.

In other words, good direction emerges from good details. I think it remains important not to let go of the ability to dive into the details.