Productivity Weekly (April 27, 2022 Issue)

Hello. I am Hirakiba from the Productivity Improvement Team at Cybozu Inc.

Our Productivity Improvement Team holds a meeting called "Productivity Weekly" every Wednesday to share tips discovered during the week regarding developer productivity improvement within the company.
This article is a summary of the topics from that meeting.

This is the 72nd edition. Past articles can be found here.

news 📺

CircleCI's Scheduled Workflows feature is scheduled to be phased out starting June 3, 2022

Last October, a new feature for scheduled CircleCI pipeline execution was released. It was mentioned that existing scheduled workflows would eventually be deprecated.

While the exact deprecation date was unknown, it was announced in February (which I learned about in this Weekly) that deprecation would occur on June 3, 2022.

The deprecation for Scheduled Workflows is June 3rd 2022.
https://discuss.circleci.com/t/scheduled-pipelines-are-here/41684/11

Documentation has been updated, but it uses the term "phased out" instead of "deprecation".

Scheduled workflows will be phased out starting June 3, 2022. Visit the scheduled pipelines migration guide to find out how to migrate existing scheduled workflows to scheduled pipelines, or to set up scheduled pipelines from scratch.
https://circleci.com/docs/2.0/workflows/#scheduling-a-workflow

When it was called "deprecated," I thought scheduled workflows would still be usable for a while, but the documentation phrasing "phased out" sounds like they will gradually become unavailable. I'm not entirely sure what this means yet (if anyone knows, please tell me). Maybe a brownout will start, or they might still work fine for a while after June 3rd.

The documentation explains how to migrate from scheduled workflows to scheduled pipelines, so it's a good idea to refer to that for migration^[1].

• Migrate scheduled workflows to scheduled pipelines

Although how exactly it will be phased out remains a mystery, it would be wise to perform the migration as early as possible. Let's get to migrating.

Organization Profile Updates: Member-only READMEs and pinned private repositories | GitHub Changelog

https://github.blog/changelog/2022-04-20-organization-profile-updates-member-only-readmes-and-pinned-private-repositories/

On GitHub, it's now possible to place private READMEs on Organization profile pages. Previously, only public READMEs could be placed. This update allows you to have a private README separately from the public one.

You can set up a private README by placing profile/README.md in a repository named .github-private. Organization members who want to view it can do so by clicking the View as button on the Org's top page and selecting Member.

The View as button displayed on the Org's top page. If there's no private README, a create a README file link is also shown.

The dialog that appears when you click create a README file in View as. It guides you to the repository creation screen along with instructions on how to create a private README.

The information displayed when trying to create a .github-private repository. It explains that this is a special repository.

Note that for private repositories, profile/README.md appears at the top of the Organization as expected, but unfortunately, it did not appear for internal repositories. Perhaps members in the same Enterprise who are not in the Org cannot view the private README^[2].

By the way, the documentation says to create a public repository, so it seems achievable even with a public one^[3].

If your organization does not already have a .github-private repository, create a public .
https://docs.github.com/en/organizations/collaborating-with-groups-in-organizations/customizing-your-organizations-profile#adding-a-member-only-organization-profile-readme

It's a bit disappointing that the behavior doesn't seem to account much for Enterprise use yet. I hope that improves in the future. Regardless, a README visible only to Organization members is quite valuable. For example, guidance on contribution methods, where to communicate daily, Org rules, and so on. This is a welcome feature.

Private Profiles | GitHub Changelog

https://github.blog/changelog/2022-04-21-private-profiles/

GitHub user profiles can now be made private (public beta). When enabled, others will no longer be able to see that user's activity, stars, etc., at all.

It seems that scope settings cannot be configured finely; for example, it is not possible to hide only stars. It might be a good idea to set this for bot users used in organizations. Since this feature is still in public beta, if you have any concerns, feel free to send feedback.

OpenID Connect Support added to AWS CLI Orb - v3.1 - Announcements - CircleCI Discuss

https://discuss.circleci.com/t/openid-connect-support-added-to-aws-cli-orb-v3-1/43817

The circleci/aws-cli orb now supports authentication via OIDC. This support starts from v3.1.

CircleCI recently supported OIDC, but since the circleci/aws-cli orb did not support it, it was necessary to put effort into shell scripts, such as manually running the AWS CLI to extract tokens (Reference).

With the official CircleCI orb now supporting it, AWS authentication using OIDC has become extremely simple. It's something worth utilizing.

Test Splitting on Self-hosted Runners

https://discuss.circleci.com/t/test-splitting-on-self-hosted-runners/43810

In CircleCI, test splitting and parallel execution are now available when using self-hosted runners.

CircleCI has a feature that automatically splits the test suite and executes each test case in separate containers (broadly speaking). For details, please refer to the following documentation:

• Parallelism and test splitting - CircleCI

This feature allows tests to be allocated based on past test execution results, which is effective for reducing the overall test time.

This test splitting feature is now available for self-hosted runners. It is not possible to run them across both self-hosted runners and CircleCI-hosted runners; you must prepare and specify multiple self-hosted runners.

It's great to see self-hosted runners becoming more like CircleCI-hosted runners. If you were thinking that test splitting wasn't possible, you should give it a try.

CircleCI | Product Roadmap - CircleCI

https://circleci.com/product-roadmap/

CircleCI has started publishing its Product Roadmap. It is divided into categories: Now, Next, Future, and Released, covering topics for both Cloud and Server. Similar to other organizations, GitHub^[4] and Docker also publish their roadmaps.

Personally, I'm particularly interested in the following two topics:

• Talk about removing kots from Server 3.x
  • I'm happy because I don't like kots.
• Talk about allowing users to download audit logs themselves
  • While the Server version allows self-downloading, for the Cloud version, you have to request it from support.
  • More than that, I want them to support streaming.
    • I commented asking for streaming support.

Publishing a Product Roadmap is genuinely exciting. It's also great that users can participate in discussions before implementation. I wish every XaaS would publish a Product Roadmap like this^[5].

Resource Class Insights - Announcements - CircleCI Discuss

https://discuss.circleci.com/t/resource-class-insights/43790

In CircleCI, trends in job resource consumption are now visualized. Previously, you could view the resource consumption graph for a single job execution, but with this change, you can now see how resource consumption trends across multiple job executions.

Example of a cron job for my homepage. It consumes more CPU than I expected lol.

You can see the trends in average and maximum CPU and memory usage for each resource class.

Being able to view these graphs is helpful for selecting the appropriate resource class.

know-how 🎓

Accelerating docker build in CircleCI without using Layer Caching - Google Slides

https://docs.google.com/presentation/d/1UpRyHsdx4bpBWE9P0VWznVCcupmjVWfmEXRoMbcAeIQ/mobilepresent?slide=id.p

These slides examine which methods are most appropriate for accelerating docker build in CircleCI without using Layer Caching.

CircleCI has a feature called Layer Caching that caches Docker image layers, but Layer Caching consumes more credits compared to when it's disabled. Therefore, various methods were tested to speed up docker build on CircleCI without using Layer Caching. Several methods are introduced, such as using various cache functions within Docker itself or not building the application within docker build in the first place.

Furthermore, each method was actually measured and analyzed quantitatively and qualitatively. It's excellent to have quantitative data.

I think these slides are packed with know-how that will be useful not only when doing docker build in CircleCI but also for those who are unfamiliar with recent Docker caching features. Since I only had a vague understanding of recent Docker caching features, I found it very helpful.

Terraform operations at Recruit's AWS infrastructure - Practical efforts and organization building / HashiCorpVirtualStrategyDay_sudo - Speaker Deck

https://speakerdeck.com/recruitengineers/hashicorpvirtualstrategyday-sudo

These are slides from Recruit regarding Terraform operations. They introduce the reasons for adopting Terraform Enterprise (on-premises version of Terraform Cloud) and organizational efforts for Terraform operations.

The slides cover an introduction to the AWS infrastructure "SENTO" and the SENTO team that manages it, the history of Terraform operations, the Terraform Enterprise configuration, the Terraform application flow, the effects after introducing Terraform Enterprise, organization building for Terraform operations (agile process, Terraform code review), and what is important in Terraform operations for enterprise companies.

I also have my own thoughts on Terraform operations, and I'm thinking about various things such as how to perform terraform apply easily and safely, as well as maintenance costs and permissions when having CI services run terraform apply. These slides serve as a helpful example when considering those points. It's also great that both tools and culture are mentioned. Personally, I found the code review methods particularly helpful.

If you're building internal tools in Go, do it like this

https://zenn.dev/eihigh/articles/0774ddadc21eb4

This article introduces practices for adopting Go within a company. It covers repository structure, what to do when using private repositories, and Go education.

Personally, the repository structure of keeping package files flat and separating commands into directories was fresh (I used to divide directories a lot). I found the reasons for doing so quite convincing. Also, the recommended content for learning Go was new to me and very helpful.

Let's run CircleCI Server on-premises (Part 1): Tool installation

https://devops-blog.virtualtech.jp/entry/20220421/1650509039

This is an article about running CircleCI Server (3.x), the on-premises version of CircleCI. It consists of five parts.

• Let's run CircleCI Server on-premises (Part 1): Tool installation
• Let's run CircleCI Server on-premises (Part 2): Preparation
• Let's run CircleCI Server on-premises (Part 3): Core service installation
• Let's run CircleCI Server on-premises (Part 4): Nomad client construction
• Let's run CircleCI Server on-premises (Part 5): VM service construction

Of course, the official CircleCI Server documentation exists and should be referred to, but this series of articles includes actual command inputs/outputs and detailed explanations of what is being done in each step. It's great because you can understand what needs to be done even without actually setting up CircleCI Server^[6].

This should be helpful for those considering introducing CircleCI Server or those who haven't been able to update from version 2.x yet.

Writing Kubernetes manifests comfortably with Jsonnet

https://zenn.dev/atodekangae/articles/435d4d4f3ef55d

This article introduces how to describe k8s manifests using Jsonnet instead of YAML. While kustomize is a famous tool for creating k8s resources for multiple environments, Jsonnet is also presented as an option.

The article covers how to write Jsonnet, a hands-on session for converting actual manifests to Jsonnet to support various environments, and deployment.

It was my first time learning about Jsonnet, and my impression is that it seems to have a higher degree of freedom than kustomize. It's also nice that ArgoCD supports Jsonnet. Those who are dissatisfied with kustomize might want to give it a try.

tool 🔨

[Possible to operate production at a low cost] Comparing the pros and cons of Render.com with Heroku

https://zenn.dev/mc_chinju/articles/compare_render_and_heroku

This is an introduction article for Render.com, a PaaS similar to Heroku. It provides a comparison with Heroku in addition to introducing Render.com.

The article is packed with content, covering an introduction to Render.com, comparisons with Heroku (pricing, regions, DB, IaC, preview environments, Redis, Cron, Worker, CD, autoscaling, etc.), migration methods from Heroku, the utilization of IaC, and the utilization of GitHub Actions.

It seems to have many advantages over Heroku in terms of pricing, and the fact that an Asia region is available (likely) for free is also a plus. The ability to do things in an IaC-like manner is also great. However, since PostgreSQL is only free for 90 days after creation, it seems to fall behind Heroku in that specific aspect (Heroku's PostgreSQL Free plan has no time limit, though it is limited to a maximum of 10,000 rows).

With the recent issues often heard regarding Heroku, many people might be considering migrating to other services (I am one of them). It's worth knowing as one of the options.

koneta 🍘

This is a corner for writing topics that came up in Productivity Weekly that I want to share but don't have the stamina to fully explain, or topics that didn't have much to say, in about a sentence.

• know-how 🎓
  • What is being verified in sigstore's Keyless Signing - sometimes I laugh
    • This article explains what is being verified in sigstore's Keyless Signing, which has been a hot topic recently. It is explained from an attacker's perspective, making it easier to understand than a purely theoretical explanation.
      • Though I haven't been able to read it all yet.
  • Good things to know about S3 download processing in Go - Backend performance improvement
    • An explanation and method for improving performance when downloading something from Amazon S3 in Go.
    • I didn't know such a method existed.
  • "Tao of Node - Design, Architecture & Best Practices" Japanese Translation
    • A Japanese translation of "Tao of Node," a collection of best practices for using Node.js.
    • It covers a wide range of practices from repository structure to performance.
• tool 🔨
  • The official Vitest VSCode plugin has been released | DevelopersIO
    • It seems similar to using something like Jest in VSCode.
    • A JetBrains IDE version is also introduced at the end of the article, but that one seems to still be unofficial.

Afterword

It was Golden Week, wasn't it? I was mainly playing Elden Ring. Since it's Golden Week, the next issue (2022/05/04 issue) will be on hiatus. While I'm happy for long holidays like Golden Week, if the break is too long, it becomes hard to return to work, so personally, I'd prefer a moderate amount of time off. I wish a three-day weekend system would become the norm.

The Productivity Improvement Team will be holding an internship this summer, so please apply if you are interested (Applications start 4/22 → The Productivity Improvement course internship starts from 5/23. Sorry).

• Productivity Improvement - Design & Research - Cybozu Internship & Events 2022 Engineers & Designers | Cybozu Inc.

Cybozu's Productivity Improvement Team is active in improving the development productivity of in-house engineers. If you're interested in such a team, click the link below!
https://note.com/cybozu_dev/n/n1c1b44bf72f6

omake 🃏

This is this week's bonus.

A story about how comfortable net battles might become possible in Mega Man Battle Network

https://zenn.dev/akatsuki/articles/47cc86955acd38

This article explains the history and technology behind playing net battles for the GBA^[7] software Mega Man Battle Network (Rockman EXE)^[8] via an emulator.

Since Mega Man Battle Network was premised on communication battles using a link cable, simply changing the route to play over the internet caused latency issues. Therefore, the article discusses methods for enabling comfortable communication battles even with high latency at a slightly higher layer, along with the history.

It includes several interesting techniques, such as emulating the opponent's save data on one's own machine and transmitting only button inputs, which is genuinely interesting to read. Not limited to this topic, it makes me think that online battles in the world are amazing.

The Mega Man Battle Network series was a very interesting game, but unfortunately, after 6, no sequels^[9] or remakes have been made. Mega Man Battle Network celebrated its 20th anniversary last year, but unfortunately, no good news was announced. Whether it's a remake or a remaster, I want the Battle Network series to be revived in the modern age. I'm counting on you, Capcom!

脚注

1. Since the units for pipelines and workflows are different and the new method doesn't support cron expressions, the migration seems like quite a bit of a hassle.

2. I couldn't verify whether members in the same Enterprise who are not in the Org can view it. If anyone knows, please let me know.

3. I haven't tried with a public repository, so I don't know if it actually works.

4. As a side note, looking at the link https://github.com/orgs/github/projects/4247, GitHub Inc. has created at least 4247 GitHub Projects... they really use their own product's task management tool properly. Admirable (or perhaps expected). Personally, I used to think GitHub Projects was a bit lacking in features (though Beta has become quite usable)...

5. Incidentally, Cybozu does not publish such things (at least I don't think they did), although they do announce the contents of the next release. Too bad.

6. However, because they use eksctl to build Amazon EKS (as recommended by the official documentation), future maintenance might be difficult (honestly, eksctl is hard to use, so building with something like Terraform is easier).

7. Short for Game Boy Advance, a handheld game console released by Nintendo in 2001. By the way, the first game console my parents bought for me was a GBA.

8. The Mega Man Battle Network (Rockman EXE) series is an action RPG set in the modern age. It wasn't just an action game; it also had strategic elements like choosing chips. It was a masterpiece that was fun for both single-player and communication battles. For the record, I played 2, 4, 5, and 6.

9. There is also the Mega Man Star Force (Ryusei no Rockman) series for the Nintendo DS, but the game system and other elements are quite different. That was interesting in its own way.