iTranslated by AI
Introduction
Zenn uses a package called openGraphScraper for embedding link cards that utilize OGP.
Indeed, the GitHub link card above is also powered by openGraphScraper.
Recently, I submitted a pull request to openGraphScraper to address an issue Zenn was facing, and it was successfully merged.
In this article, I will introduce the issue Zenn was experiencing with openGraphScraper and how it was resolved.
The Issue of Garbled Text After Updates
In v5.2.0 of openGraphScraper, the decoding process based on character encoding was removed, causing garbled text on some websites after updating[1].
The garbled text occurs on websites using character encodings other than UTF-8, such as Shift_JIS or EUC-JP.
As an example of a Shift_JIS site, let's try retrieving the OGP information for Hiroshi Abe's Homepage using openGraphScraper (v6.3.4).
import ogs from "open-graph-scraper";
const main = async () => {
const { result } = await ogs({ url: 'http://abehiroshi.la.coocan.jp/' })
console.log(JSON.stringify(result, null, 2))
}
main()
The following output is obtained:
{
"ogTitle": "̃z[y[W",
"charset": "Shift_JIS",
"requestUrl": "http://abehiroshi.la.coocan.jp/",
"success": true
}
As shown, the values are garbled. The `` represents missing information, and it was impossible to restore the original string even by attempting further encoding or decoding.
(often displayed as a black diamond with a white question mark or an empty box) is a symbol assigned to the code point U+FFFD in the Specials block of the Unicode standard, used to indicate problems when a system cannot render a string of characters correctly. It is usually displayed when data is invalid or does not match any character.
(omitted)
Because the replacement is the same for all errors, the original character cannot be restored.
Because of this, we were stuck on a version before 5.2.0 for some time, which became a problem for us.
Discovering the Issue
When I searched through the openGraphScraper issues, I found a relevant one.
To be precise, it was about character encoding detection, but the underlying context was that they also wanted to avoid garbled text, which matched the issue we were facing. Incidentally, Hiroshi Abe's Homepage was also cited as an example here.
Upon reading it, since ogs() allows passing HTML directly instead of just a URL, the issue reporter seemed satisfied with the following workaround:
// Fetch the website yourself and perform appropriate decoding
// ...
// Appropriately decoded HTML information
const html: string = // ...
// Pass the HTML information directly to extract OGP information
const res = await ogs({ html })
We could have adopted this approach, but I thought it would be more convenient if openGraphScraper itself handled the fetching and decoding, reducing the burden on users. Therefore, I decided to submit a pull request linked to the aforementioned issue.
Creating the Pull Request
So, I created a pull request.
Since sites like Shift_JIS will get garbled if decoded directly as UTF-8, I modified it to first retrieve .arrayBuffer() from the fetch response object, identify the character encoding via the HTML header or the chardet library, and then perform decoding accordingly. I added iconv-lite as a dependency for the decoding process.
During this process, the integration tests that scrape the test website hosted on the maintainer's GitHub Pages were failing, so I also submitted a pull request for that repository.
I struggled a bit with the phenomenon where the test site was garbled no matter how I viewed it in the browser, but I wrote about that in a separate article (the conclusion was that it was fine to ignore it).
Successfully Merged 🎉
After various interactions and waiting for verification, it took about two weeks, but it was merged.
Shortly after, v6.4.0, which incorporates the changes from the pull request, was released.
I am grateful to the maintainer for their support.
Verifying v6.4.0
Let's try retrieving the OGP information for Hiroshi Abe's Homepage with v6.4.0.
import ogs from "open-graph-scraper";
const main = async () => {
const { result } = await ogs({ url: 'http://abehiroshi.la.coocan.jp/' })
console.log(JSON.stringify(result, null, 2))
}
main()
The following output is obtained:
{
"ogTitle": "阿部寛のホームページ",
"charset": "Shift_JIS",
"requestUrl": "http://abehiroshi.la.coocan.jp/",
"success": true
}
I was able to confirm that the information could be retrieved as intended without garbled text.
Conclusion
We updated the Zenn codebase to v6.4.0, confirmed that no garbled text occurred, and released it, marking this issue as resolved.
Through this OSS commit, I feel that not only have we solved Zenn's problem, but we have also contributed to other services that might face the same issue in the future.
Also, I feel that my understanding of character encodings, which I had previously treated rather vaguely, has improved a little.
Since there are parts of my commit where the code is not very clean or involves duplication, I intend to propose refactoring when I have more bandwidth.
References
- About the replacement character in Unicode - Programming from 30
- The character with a black diamond and question mark () is a replacement character used when there is no target for conversion during UTF-8 conversion - Code Progress
- Specials (Unicode block) - Wikipedia
Discussion