Adding Color to Your Daily Life: nvim-treesitter Configuration Tips

This is the article for day 22 of the Vim advent calendar 2021 Part 2.
Yesterday's article was "Implementing a completion function when editing GitHub issues" by Gorilla.
Tomorrow, the article "Even lighter statusline and tabline" by @seroqn is scheduled to be published.

Introduction

This year saw major changes in Neovim with the releases of v0.5.0 and v0.6.0. In particular, the integration of the tree-sitter library from v0.5.0 has drawn significant attention and has been featured in various articles, including the following:

https://lab.mo-t.com/blog/neovim-v05-introduction-new-features-part-2
https://www.soum.co.jp/misc/vim-advanced/6/
https://zenn.dev/duglaser/articles/c02d6a937a48df

As introduced in these articles, you can leverage the power of tree-sitter by using a plugin called nvim-treesitter. Based on high-speed parsing, it allows you to perform various processes such as syntax highlighting and indentation accurately and flexibly.

However, introducing nvim-treesitter does not necessarily mean it will be easier for all users. Syntax highlighting, in particular, changes significantly when switching to tree-sitter. The more accustomed you are to traditional coloring, the more you might find it harder to read.

Therefore, in this article, I will introduce how to customize the behavior of nvim-treesitter's syntax highlighting to your liking. Without directly modifying the parser, you can bring it closer to your preference just by adding a few settings called queries. If you are hesitating to migrate to tree-sitter because you don't like how the coloring changes, please give this a try.

I only started getting into tree-sitter a few months ago and am still learning. If you find anything strange in the following explanation, I would appreciate it if you could let me know in the comments section.

Practical Examples of Customization

In this article, I will use Markdown coloring as a practical example of customization. Markdown support was recently added to official Neovim (about a week ago) and can now be installed with :TSInstall markdown. It uses the grammar defined in the tree-sitter-markdown repository.

Since I also use Markdown frequently, I tried resetting the Markdown coloring to my liking. Please compare the state before and after the settings. Can you tell what changed?

Before configuration

After configuration

There were 6 changes in total. That's quite a lot.

1. The entire blockquote turned gray.
2. Text in H1 headers now has an underline.
3. The entire syntax for hyperlinks and images is colored.
4. Fenced code blocks without a specified language^[1] are now entirely colored green.
5. In fenced code blocks, specifying the language as the shorthand py instead of python now applies Python syntax highlighting.
6. In fenced code blocks, specifying :filename after the language name python now applies Python syntax highlighting.

5 and 6 currently only support specific language names; if you want to support other languages, you will need to add separate settings. While it's not a universal solution, it shouldn't be a major problem for personal use.

The settings in this article are based on the commit (6c5f4b1) at the time of writing.
tree-sitter-markdown is relatively new and still actively being developed, so default coloring settings and grammatical structures may change. In fact, there have been changes over the last few days.
Depending on the changes, the settings described in the article might not work as intended. Please view this as a "specific example of customization."

The appearance of syntax highlighting naturally depends heavily on the configured colorscheme.
I am currently using a colorscheme called vim-gruvbit, but it may not be possible to reproduce exactly the same look (I also customize the colorscheme in places).
Also, to color code blocks as in the Python example, you need to install the Python parser with :TSInstall python.

How nvim-treesitter Works

Before explaining the configuration method, let me first briefly introduce the steps nvim-treesitter takes to perform syntax highlighting.

Essentially, tree-sitter is a parser generator—that is, "something that generates syntax parsers." Neovim uses parsers for each language created with tree-sitter (e.g., tree-sitter-rust, tree-sitter-python, tree-sitter-markdown) to obtain a syntax tree of the open text and provides convenient features such as syntax highlighting. In short, it provides syntax highlighting through the following steps:

1. Parsing: Parse the text to create a syntax tree.
2. Query Search: Search the syntax tree for parts that match predefined queries (search patterns).
3. Coloring: Apply predefined colors to the matched parts captured in step 2.

Parsing

For example, suppose you have a Markdown file like this:

example.md

# Title

This is a paragraph. You can **emphasize** words in the middle.

- List
- List with **emphasis**

Parsing the above file using tree-sitter-markdown yields a syntax tree like the following:

[Syntax Tree Corresponding to example.md]

atx_heading [0, 0] - [1, 0]
  atx_h1_marker [0, 0] - [0, 1]
  heading_content [0, 1] - [0, 14]
paragraph [2, 0] - [3, 0]
  strong_emphasis [2, 27] - [2, 37]
list [4, 0] - [6, 0]
  list_item [4, 0] - [5, 0]
    list_marker_minus [4, 0] - [4, 2]
    paragraph [4, 2] - [5, 0]
  list_item [5, 0] - [6, 0]
    list_marker_minus [5, 0] - [5, 2]
    paragraph [5, 2] - [6, 0]
      strong_emphasis [5, 2] - [5, 12]

Strings like atx_heading represent node names, and values like [0, 0] - [1, 0] indicate the location of those elements.^[2] You can see that structures such as "Title (atx_heading)", "Emphasis (strong_emphasis)", and "Bullet points (list)" are captured.

Query Search

A query in tree-sitter is like a search pattern for finding desired nodes in a syntax tree. For example, a file called highlights.scm in markdown defines a query like this:

[Example Query 1]

(strong_emphasis) @text.strong

This means "Capture a node named strong_emphasis with the name @text.strong."^[3] In the previous syntax tree example, it matches two places: strong_emphasis at [2, 27] - [2, 37] and [5, 2] - [5, 12].

Let's look at a slightly more complex example. The following query is also defined in highlights.scm:

[Example Query 2]

(atx_heading
 (heading_content) @text.title
 )

This means "When there is a node named atx_heading and it has a child node named heading_content directly below it, capture heading_content as @text.title." In the previous syntax tree example, the atx_heading at [0, 0] - [1, 0] has heading_content at [0, 1] - [0, 14], so it matches this one place.

Coloring

In the previous examples, we found two matches for the first query and one match for the second query. In nvim-treesitter, Neovim syntax highlighting is added according to the captured locations as follows:

• @text.strong → TSStrong
• @text.title → TSTitle

This applies coloring to the parts corresponding to the two strong_emphasis and one heading_content nodes in the Neovim buffer.

There is no consistent rule for the correspondence between capture names (tags starting with @) and Neovim syntax highlight names. I don't know if there is a document clearly describing the correspondence at this point (at least I don't know of one), but the code is summarized here.
By the way, if it is a highlight name starting with a capital letter, you can specify it directly, such as @TSTitle.

Highlight Customization Steps

This is the main topic: "how to customize nvim-treesitter." Based on my experience, following these steps seems to provide a smooth customization process:

1. Prepare a sample file written in the target language (ensure it includes the content for which you want to change the coloring).
2. Examine the syntax tree of the relevant part to understand the structure around the node you want to modify.
3. Add or modify queries so that the color of the targeted node is changed.
4. Reload the sample file and verify if the modified rules are reflected.

Specific Example: Configuration to add an underline to H1 headers

As a relatively simple example, let's add a configuration to underline only H1 headers following the steps above.

Installing nvim-treesitter-playground

First, install the nvim-treesitter-playground plugin, which is essential for tweaking nvim-treesitter settings.

https://github.com/nvim-treesitter/playground#setup

This makes the following two commands available:

• :TSPlaygroundToggle: Displays the syntax tree of the given buffer
• :TSHighlightCapturesUnderCursor: Displays the syntax highlight groups under the cursor

Preparing a Sample File

Prepare a sample file like the following:

sample.md

# tree-sitter-markdown example

## Link syntax

Please support the [Vim advent calendar 2021](https://qiita.com/advent-calendar/2021/vim).

Determining the nodes to change coloring for from the syntax tree

By running :TSPlaygroundToggle while the sample file is open, you can see the syntax tree corresponding to the current window.

:TSPlaygroundToggle execution result

For the text # tree-sitter-markdown example, you can see that the following syntax tree corresponds to it:

• ats_heading (# tree-sitter-markdown example)
  • ats_h1_marker (#)
  • heading_content ( tree-sitter-markdown example)

By the way, you can also see that for the text ## Link syntax, the following syntax tree corresponds:

• ats_heading (## Link syntax)
  • ats_h2_marker (##)
  • heading_content ( Link syntax)

ats_h1_marker is used in H1 headers, and ats_h2_marker is used in H2 headers. This is the key to distinguishing header levels.

Adding a query

Based on the syntax tree we looked at above, if we verbalize what we want to do this time, it would be: "Add an underline only to the heading_content child element of an ats_heading that has an ats_h1_marker child element." Let's add the settings to the query file to achieve this.

To add settings to a query file, use the :TSEditQueryUserAfter command as follows:

:TSEditQueryUserAfter highlights markdown

This will open a file for writing user settings related to syntax highlighting.^[4]

If you use the :TSEditQuery command instead of :TSEditQueryUserAfter, you can see the queries provided by nvim-treesitter by default. While editing is possible, it is recommended to keep it for viewing only as it may be overwritten by updates to the nvim-treesitter plugin itself.

Let's write the following query in the opened buffer and save it.

highlights.scm

(atx_heading
 (atx_h1_marker)
 (heading_content)  @text.underline
 )

It might be hard to understand the meaning until you get used to it, but it might be easier to grasp if you think of it as pattern matching against a syntax tree represented as an S-expression.

• Match when the name is atx_heading and it has child nodes atx_h1_marker and heading_content.
• Capture the heading_content node of the matched part as @text.underline.

It is specified that coloring should be performed with the highlight group TSUnderline for parts captured by @text.underline. Just by writing this setting, you can achieve "underlining only the titles of H1 headers."

If it is a highlight group starting with a capital letter, you can specify the highlight group name directly, such as @Title.

Verifying if the modified highlight rules are reflected

Return to the buffer of the original Markdown file and reload it with :e; the coloring process will run again with the changes reflected. The :TSHighlightCapturesUnderCursor command is useful for checking how coloring is applied under the cursor.

You can see that the coloring by TSUnderline is being applied correctly. The appearance is also as expected!

Final Query Configuration

To wrap up the long explanation, here are the final configuration files.

highlights.scm Settings

Adding the following queries to highlights.scm will achieve points 1 to 4 of the changes shown at the beginning.

highlights.scm

; Color hyperlinks and images with TSAttribute
[
 (inline_link)
 (image)
] @attribute

; Make the entire blockquote the same color as comments
(block_quote) @comment

; Underline titles in H1 headers
(atx_heading
 (atx_h1_marker)
 (heading_content) @text.underline
 )

; Color content of fenced code blocks without a specified language with TSLiteral
(fenced_code_block
 .
 (code_fence_content) @text.literal)

In query files, parts enclosed in [ ] represent alternation, matching any of the elements within the brackets. In other words, the setting:

[ (inline_link) (image) ] @attribute

is equivalent to:

(inline_link) @attribute
(image) @attribute

The . used in the last fenced code block is called an anchor operator. In the example above, it ensures a match only if the first child element of fenced_code_block is code_fence_content. Therefore, it won't match if an element like info_string is included before code_fence_content. If you omit this ., the coloring of fenced code blocks with a specified language will become strange (try it out to see how...).

injections.scm Settings

In tree-sitter, it is possible to "color the content of a Markdown code block with Python grammar." In fact, if you look at the syntax tree with :TSPlaygroundToggle, you can see a Python syntax tree embedded inside while looking at a Markdown file. This uses a feature of tree-sitter called language injection. Although it is inside a Markdown file, it borrows queries for Python coloring.

A Python syntax tree embedded within a Markdown file's syntax tree

injections.scm is where you configure "where in the Markdown file to perform language injection," and you can open it by executing the following command:

:TSEditQueryUserAfter injections markdown

Adding the following query here will achieve points 5 and 6 of the changes shown at the beginning.

injections.scm

(fenced_code_block
 (info_string) @lang
 (code_fence_content) @content
 (#vim-match? @lang "^(py|python)(:.*)?$")
 (#set! language "python")
)

This configuration specifies that Python syntax highlighting should be applied only when the info_string text matches the regular expression ^(py|python)(:.*)?$. You can support languages other than Python by changing the regular expression pattern or the language name.

Conclusion

I have explained how to achieve your preferred syntax highlighting by tweaking tree-sitter queries. Since you are not modifying the parser itself, it's not a "anything goes" solution, but you can achieve a fairly high degree of freedom based on the syntax tree.
I hope everyone gives nvim-treesitter a try and adds some color to their monochrome daily lives!

References

• tree-sitter Official Documentation

  The following descriptions are particularly useful as items related to this article:

  • Using Parsers > Pattern Matching with Queries
  • Syntax Highlighting > Queries

• nvim-treesitter > Contributing > Parser Configurations

脚注

1. Refers to the markup for writing code, surrounded by three backticks.

2. It's a bit confusing that both row and column indices start at 0.

3. In the original tree-sitter documentation, tags like @text.strong are given the name "capture."

4. The file path seems to be (Neovim's config path)/after/queries/markdown/highlights.scm.