A Simple Task: Just Fetching Data from a URI

Time for another break between tasks!

I've been writing code to fetch RESTful API data individually across the various tools I've built for my own use. I finally got tired of copy-pasting, so I decided to extract the "fetching" functionality into a standalone package dedicated solely to that purpose.

https://github.com/spiegel-im-spiegel/fetch

For now, I'm focusing only on GET requests. Using this package, you can write something like this:

sample.go

// +build run

package main

import (
    "context"
    "fmt"
    "io"
    "net/http"
    "os"

    "github.com/spiegel-im-spiegel/fetch"
)

func main() {
    githubUser := "spiegel-im-spiegel"
    u, err := fetch.URL("https://api.github.com/users/" + githubUser + "/gpg_keys")
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        return
    }
    resp, err := fetch.New(fetch.WithHTTPClient(&http.Client{})).
        Get(
            u,
            fetch.WithContext(context.Background()),
            fetch.WithRequestHeaderSet("Accept", "application/vnd.github.v3+json"),
        )
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        return
    }
    defer resp.Close()
    if _, err := io.Copy(os.Stdout, resp.Body()); err != nil {
        fmt.Fprintln(os.Stderr, err)
    }
}

Using this:

$ go run sample.go | jq -r '.[]|select(.key_id=="B4DA3BAE7E20B81C")|.raw_key'
-----BEGIN PGP PUBLIC KEY BLOCK-----

mQSuBFF8+hcRDAClgJHjXcpxJhhx2aaxQEF/V6ds8DL358hk2TRXmpcpwBmjSCB4
cW/soISF6vzPQgWNvZjaPwWfrwWRjMsfk/9XfDtibvIgqShtS5Nc/p6/wxOMHh6x
...

See? I was able to extract the OpenPGP public key from the GitHub site.

(For details on what's happening here, please refer to my previous article: "Extracting OpenPGP Public Keys Registered on GitHub".)

It doesn't have authentication features like OAuth, and it's probably too niche to be used as a general-purpose package. However, I've tried to write it in a way that avoids common bugs and vulnerabilities typically encountered when using the net/http package.

cURL as DSL

The sample code above is nearly equivalent to the following curl command line.

$ curl "https://api.github.com/users/spiegel-im-spiegel/gpg_keys" -H "Accept: application/vnd.github.v3+json"

Or rather, there is usually a curl command example provided as an "API specification," and I often write code to match that.

Actually, there is a tool—or rather, a service—literally named "cURL as DSL," which I've been using for a long time. "cURL as DSL" converts curl command lines into code for Go, Python, node.js, Java, PHP, Vim script, and more.

However, it seems that "cURL as DSL" itself is no longer being maintained, and it recommends

https://curl.trillworks.com/

as a similar service. This one supports even more languages.

I started using these kinds of tools after reading the following article by Yoshiki Shibukawa, who released "cURL as DSL" in 2015:

http://blog.shibu.jp/article/115602749.html

Specifically, I found myself in "strong agreement" with this part at the time:

The inspiration for this tool came from the "Copy as cURL" menu in Google Chrome's developer tools. If you can copy in cURL format, many users can easily create cURL commands. I thought it could become a means of communication. Even if it's redundant, being able to auto-generate code is a very welcome option for non-programmers. After all, plenty of non-programmers use Excel VBA or Flash JSFL.

Also, those who have used Python, Ruby, or node.js likely understand the convenience of environments like interactive modes, irb, or REPLs. If cURL spreads and everyone starts writing cURL pseudo-code in web documentation, I believe cURL has the potential to become the REPL of the HTTP world.

To be blunt, if you have a curl command line example and a table of available parameters, you can turn it into code. Even when I created a client-side package for PA-APIv5, there’s no denying that it was much easier to understand because Amazon provided curl sample code (and PHP code too. The Java code details were hidden, and I couldn't make heads or tails of it, haha).

So, if you are planning to release a RESTful API specification to the public, please write it in curl 🙇 Well, I suppose the trend these days is GraphQL though (lol).

Related Links

• Things to watch out for with Go's net/http keep-alive - Carpe Diem

Reference Books

https://www.amazon.co.jp/dp/4908686033

https://www.amazon.co.jp/dp/4873119030