Reviewing Zenn RSS Article Feeds with Gemini and Posting to Slack

Classmethod has implemented a system to automatically review DevelopersIO articles.

https://dev.classmethod.jp/articles/devio-autoreview-by-bedrock/

From the writer's perspective, the experience is excellent, and I wanted to implement this with Google Cloud and Gemini. I will review Zenn articles with the Gemini version. As a result, by registering an RSS feed and publishing an article, I was able to receive review results like the following:

I believe the use of generative AI for text review is becoming widespread. This article will introduce the process of subscribing to RSS and posting to Slack, hoping it serves as a helpful sample for integration with Google Cloud.

Architecture

First, the overall architecture is as follows:

RSS Feed Subscription ~ Publish to Topic

Among these, the red-boxed part below:

Please refer to a separate article for this section.

https://zenn.dev/waddy/articles/cloud-run-functions-zenn-rss-to-pubsub

Review with Gemini and Post to Slack

This article describes the red-boxed part above.

Item	Tools Used
Cloud Run function runtime	Node.js 20 (written in TypeScript and transpiled)
Language model for review	gemini-1.5-pro-001

We start from a situation where data in the following format arrives in Cloud Pub/Sub as unread content:

{
  "title": "Reviewing Zenn articles from RSS with Gemini and posting to Slack",
  "url": "https://zenn.dev/waddy/articles/cloud-run-functions-gemini-review-to-slack"
}

Using this data, we retrieve the article body and perform a review. Finally, we post the results to Slack. We will perform the tasks in the following order:

1. Prepare a Slack App
2. Implement a Cloud Run function
3. Prepare environment variables and deploy

Let's dive right in.

Prepare a Slack App

We will enable the Cloud Run function to postMessage review results. Since we'll post via a Slack App, we start by creating the app. Detailed instructions for this are provided in the following article, so please follow the steps there to configure it.

https://zenn.dev/kou_pg_0131/articles/slack-api-post-message

It's sufficient if you can confirm that messages can be sent to a Slack channel via API.

Implement a Cloud Run function

This starts from receiving messages from Cloud Pub/Sub. The Cloud Run function implements the following processes:

• Parse HTML from the URL to get the article body
• Request a review based on media policy prompts (Vertex AI API)
• Request a review based on typo prompts (Vertex AI API)
• Post the results to Slack

index.ts

import { GoogleAuth } from "google-auth-library";
import fetch from "node-fetch";
import { WebClient } from "@slack/web-api";
import * as cheerio from "cheerio";
import { PubsubMessage } from "@google-cloud/pubsub/build/src/publisher/pubsub-message";
import { Firestore } from "@google-cloud/firestore";

const projectId = process.env.PROJECT_ID;
const location = process.env.LOCATION;
const modelId = process.env.MODEL_ID;

// Vertex AI API endpoint URL
const vertexAiEndpoint = `https://${location}-aiplatform.googleapis.com/v1/projects/${projectId}/locations/${location}/publishers/google/models/${modelId}:generateContent`;

// Slack token (from Secret Manager)
const secretSlackToken = process.env.SLACK_TOKEN || "";

// Channel ID for posting
const channelId = process.env.CHANNEL_ID || "";

// Firestore collection name to retrieve prompts
// Ideally, this could be a text file or a fixed string, but
// at the time of writing, I'm loading from Firestore to avoid redeploying
// every time I adjust prompts. Any storage location is fine as long as prompts can be read.
const firestore = new Firestore({
  databaseId: "rss-manager",
});

// Firestore collection name
const collectionName = "prompts";

// Type of message sent from Cloud Pub/Sub
type FeedItemType = {
  title: string;
  link: string;
  pubDate: string;
};
// Pub/Sub subscription. Reviews RSS metadata with Gemini.
export async function geminiReviewer(message: PubsubMessage) {
  const feedItemString = message.data
    ? Buffer.from(message.data as string, "base64").toString()
    : null;

  if (!feedItemString) {
    return;
  }

  const feedItem: FeedItemType = JSON.parse(feedItemString) as FeedItemType;
  const contentUrl = feedItem.link;
  const contentTitle = feedItem.title;
  const bodyHtml = await parseBodyHtml(contentUrl);

  // Data for review requests
  const agenda = await getMediaPolicyPrompt(
    bodyHtml,
    "Provide an overall evaluation against the media policy"
  );
  const fix = await getMediaPolicyPrompt(
    bodyHtml,
    "Point out problematic areas and suggest improvements against the media policy"
  );
  const good = await getMediaPolicyPrompt(
    bodyHtml,
    "Describe the good points in line with the media policy"
  );
  const [agendaBody, fixBody, goodBody] = [agenda, fix, good].map((prompt) => ({
    contents: {
      role: "user",
      parts: [
        {
          text: prompt,
        },
      ],
    },
    generation_config: {
      temperature: 0.7, // Adjust generation diversity (0.0 - 1.0)
      max_output_tokens: 1000, // Maximum number of tokens to generate
      top_p: 0.8, // Adjust token selection diversity
      top_k: 40, // Sample top K tokens
    },
  }));

  const llmRequestTypographyBody = {
    contents: {
      role: "user",
      parts: [
        {
          text: await getTypographyPrompt(bodyHtml),
        },
      ],
    },
    generation_config: {
      temperature: 0.7, // Adjust generation diversity (0.0 - 1.0)
      max_output_tokens: 1000, // Maximum number of tokens to generate
      top_p: 0.8, // Adjust token selection diversity
      top_k: 40, // Sample top K tokens
    },
  };

  // Get authentication token
  const auth = new GoogleAuth({
    scopes: "https://www.googleapis.com/auth/cloud-platform",
  });
  const client = await auth.getClient();
  const accessToken = (await client.getAccessToken()).token || "";

  // Send request to Vertex AI
  const mediaPolicyAgenda = await fetchVertexAi(accessToken, agendaBody);
  const mediaPolicyFix = await fetchVertexAi(accessToken, fixBody);
  const mediaPolicyGood = await fetchVertexAi(accessToken, goodBody);
  const typo = await fetchVertexAi(accessToken, llmRequestTypographyBody);

  // Initialize Slack client
  const slackClient = new WebClient(secretSlackToken);

  // Post message to Slack channel
  await slackClient.chat.postMessage({
    channel: channelId,
    blocks: [
      {
        type: "section",
        text: {
          type: "mrkdwn",
          text: `*Reviewed Article*\n <${contentUrl}|${contentTitle}>`,
        },
      },
      {
        type: "section",
        text: {
          type: "mrkdwn",
          text: `📚 * Media Policy Review by ${modelId}*`,
        },
      },
      {
        type: "section",
        text: {
          type: "mrkdwn",
          text: "*Overall Evaluation*",
        },
      },
      {
        type: "section",
        text: {
          type: "mrkdwn",
          text: mediaPolicyAgenda,
        },
      },
      {
        type: "section",
        text: {
          type: "mrkdwn",
          text: "*Identified Issues and Improvement Suggestions*",
        },
      },
      {
        type: "section",
        text: {
          type: "mrkdwn",
          text: mediaPolicyFix,
        },
      },
      {
        type: "section",
        text: {
          type: "mrkdwn",
          text: "*Good Points in Line with Media Policy*",
        },
      },
      {
        type: "section",
        text: {
          type: "mrkdwn",
          text: mediaPolicyGood,
        },
      },
      {
        type: "section",
        text: {
          type: "mrkdwn",
          text: `📝 * Spell Check by ${modelId}*`,
        },
      },
      {
        type: "section",
        text: {
          type: "mrkdwn",
          text: typo,
        },
      },
    ],
  });
  return;
}

This is a long snippet, so I've only extracted the main processing. It does the following:

• Parses the content of the URL to get the body text
• Requests a review based on media policy prompts (Vertex AI API)
• Requests a review based on typo prompts (Vertex AI API)
• Posts the results to Slack

The prompts may change as needed, but for reference, I've included the current versions at the time of writing.

Media Policy

This media policy is partially inspired by the one used in DevelopersIO's article review system.

You are a reviewer for a corporate blog.

You need to check the blog for inappropriate expressions according to the media policy.
Do not respond with the media policy content as is.
The media policy is the Markdown text enclosed by "~~~".

~~~
### Disparagement

* Including "praising A to disparage B" is not allowed.
* Even if there is clear fault from the other party, explain based on facts + communicate as follows:
    * Methodological suggestion: Introduce workarounds
    * Expressive suggestion: Rephrase into positive opinions (e.g., "Currently, Windows is not supported. It would be great if it were supported in the future from a user's perspective, wouldn't it?")
* When featuring individuals, be careful of the possibility that "you think you are praising them, but they feel uncomfortable."

### Rule Violations

* Reproduction exceeding the scope of citation
* Exposure of confidential information
    * Strings that appear to be secret keys
    * Procedures that might expose secret information
* Violations of the law
    * Cannabis Control Law, etc.: Even overseas experiences can be judged under Japanese law.
    * Specified Commercial Transactions Law, Act against Unjustifiable Premiums and Misleading Representations: Although for BtoC, avoid presenting misleading numerical values altogether.
* Other specific NG items
    * Personal information, information that leads to identification of accounts or IPs
    * Directly reprinting content posted on other sites (detrimental to both parties)

### Speculation and Hearsay

* It is not allowed to **only** cover information with insufficient sources. Factual verification is okay.
* Do not write about things you don't know as if you don't know them.
    * Avoid omitting explanations with **only** links to external sites as much as possible.
    * How much prerequisite information to write should be judged by each person from the perspective of "the article's reader."

### Exaggeration

* If a specific product claims to be "the best in the world," check with the other party for a source. If questionable, rephrase to "top-class" or similar.
* When addressing topics of high social interest, first organize and share objective circumstances and background. Otherwise, your personal opinions may stand out negatively or, at worst, be misinterpreted as disparagement.
* Be careful of cases where "larger subjects" beyond oneself are spoken of as facts or premises, as this can result in a divergence from reality and exaggeration.

### Avoidance of Slang and Memes

Technical blogs aim to deliver knowledge to as many people as possible. Strong word choices aimed at virality in a short period, or too many topical references, can lead to deviations from the main topic. Check for the following NG items:

* Methods where specific characters engage in dialogue are not allowed.
* Internet slang such as "brain-dead," "old fogey," "dead content," "memorial" are not allowed.

~~~

Typos

You are a reviewer for a corporate blog.

Please review the blog from the following perspectives:

- Are there any typos?
- Are there any unnatural Japanese phrasings?
- Is the usage of particles like "te", "ni", "o", "ha" unnatural?
- Finally, please provide an overall impression.

Prepare Environment Variables and Deploy

This time, we have several environment variables and secrets to set up.

Environment Variables:

• PROJECT_ID: Google Cloud project name for the VertexAI API endpoint.
• LOCATION: Region for the VertexAI API endpoint. Set to asia-northeast1.
• MODEL_ID: Model name to use for the VertexAI API endpoint. Set to gemini-1.5-pro-001.
• CHANNEL_ID: Slack Channel ID where review results will be posted.

These are listed in .env.yaml.

.env.yaml (example)

PROJECT_ID: your-project-name
LOCATION: asia-northeast1
MODEL_ID: gemini-1.5-pro-001
CHANNEL_ID: SAMPLEJA23

Secret Manager:

Register the Bot User OAuth Token of the Slack App you created earlier into Secret Manager. The value can be copied from the Installed App Settings screen of the Slack App configuration.

Once you have the value, register it from Secret Manager in the Google Cloud Console. The secret name is SLACK_CHATOPS_BOT_TOKEN.

Preparation is complete. Let's deploy.

Deployment

Deploy with options to set environment variables and secrets.

gcloud functions deploy geminiReviewer \
--region=asia-northeast1 \
--runtime=nodejs20 \
--memory=256 \
--timeout=30s \
--source=. \
--trigger-topic=rss-updates \
--project=your-project-name \
--entry-point=geminiReviewer \
--env-vars-file=.env.yaml \
--set-secrets SLACK_TOKEN=projects/999999999999/secrets/SLACK_CHATOPS_BOT_TOKEN:latest

• --trigger-topic: Specifies the Cloud Pub/Sub topic name where content will flow.
• --env-vars-file: Specifies the .env.yaml you created.
• --set-secrets: Specifies the secret name. This allows the Cloud Run function to retrieve the secret value as an environment variable.

Try Updating RSS

Now, all components are connected, including publishing to Pub/Sub. Let's try writing an article on Zenn and updating the RSS. If the review completes successfully, the results shown at the beginning will be posted to Slack.

With this, we have created a mechanism that checks RSS every 5 minutes, and if there's a new article, it reviews the content with Gemini and posts to Slack. I hope you write many articles and get your own articles reviewed often.

Conclusion

This time, we had Zenn articles reviewed, but if the logic for extracting the body text is determined, articles from any blog site can be reviewed. I'd like to state what I like about this review mechanism and points I want to improve.

What I like

• Since the target is publicly available content, the entire text can be reviewed without hesitation.
• There aren't many opportunities to have my articles reviewed from a media policy perspective, so it serves as an opportunity to re-recognize my tendencies and habits.
• Because I include a prompt asking to "state good points in line with the media policy," I get praised. That's nice.
• It's helpful to see what kind of reviews other people's articles receive.
• Since it's a generative AI review, I can casually look at it, decide whether to act on it myself, and don't have to reply. It's easy.

Points I want to improve

• Since it uses RSS, it only reviews new posts, and automatic re-review of updates after correction is not possible.
• Occasionally, it quotes with HTML tags, so it might be better if the input were Markdown, but that's difficult with the current Zenn system (public content is HTML only).
• DevelopersIO includes textlint results, which I wanted to emulate, but it couldn't be run with Node.js (under investigation).

Please let me know if you have any suggestions on how to improve the prompt specification or other points.

Source Code

https://github.com/cm-wada-yusuke/llm-reviewer/tree/main/gemini-reviewer