Ruby 3.2 - RubyVM::AbstractSyntaxTree

This is an article for Day 21 of the Ruby 3.2 Advent Calendar.

https://qiita.com/advent-calendar/2022/ruby32

---

RubyVM::AbstractSyntaxTree

Added error_tolerant option to RubyVM::AbstractSyntaxTree.parse

Feature #19013: Error Tolerant Parser - Ruby master - Ruby Issue Tracking System

Passing a string with a syntax error to something like RubyVM::AbstractSyntaxTree.parse would normally result in an error.

s = <<EOS
def hoge
  p 1,
end
EOS
RubyVM::AbstractSyntaxTree.parse(s)
#=> SyntaxSuggest: Could not find filename from "syntax error, unexpected `end' (SyntaxError)"
#   <internal:ast>:33:in `parse': syntax error, unexpected `end' (SyntaxError)
#           from a.rb:6:in `<main>'

In Ruby 3.2, by setting the error_tolerant option to true, it is now possible to return an object that includes the error instead of raising an exception.

s = <<EOS
def hoge
  p 1,
end
EOS
RubyVM::AbstractSyntaxTree.parse(s, error_tolerant: true)
#=> (SCOPE@1:0-3:3
#    tbl: []
#    args: nil
#    body:
#      (DEFN@1:0-3:3
#       mid: :hoge
#       body:
#         (SCOPE@1:0-3:3
#          tbl: []
#          args:
#            (ARGS@1:8-1:8
#             pre_num: 0
#             pre_init: nil
#             opt: nil
#             first_post: nil
#             post_num: 0
#             post_init: nil
#             rest: nil
#             kw: nil
#             kwrest: nil
#             block: nil)
#          body: (ERROR@2:2-3:3))))

This might be useful for tools that need to parse code in an incomplete state. I guess.

Added keep_tokens option to RubyVM::AbstractSyntaxTree.parse

Feature #19070: Enhance keep_tokens option for RubyVM::AbstractSyntaxTree parsing methods - Ruby master - Ruby Issue Tracking System

By setting keep_tokens to true, it is now possible to obtain tokens via #tokens.

s = <<EOS
def hoge
  p 1,2,3
end
EOS
ast = RubyVM::AbstractSyntaxTree.parse(s, keep_tokens: true)

pp ast.tokens
# [[0, :keyword_def, "def", [1, 0, 1, 3]],
#  [1, :tSP, " ", [1, 3, 1, 4]],
#  [2, :tIDENTIFIER, "hoge", [1, 4, 1, 8]],
#  [3, :nl, "\n", [1, 8, 1, 9]],
#  [4, :tSP, "  ", [2, 0, 2, 2]],
#  [5, :tIDENTIFIER, "p", [2, 2, 2, 3]],
#  [6, :tSP, " ", [2, 3, 2, 4]],
#  [7, :tINTEGER, "1", [2, 4, 2, 5]],
#  [8, :",", ",", [2, 5, 2, 6]],
#  [9, :tINTEGER, "2", [2, 6, 2, 7]],
#  [10, :",", ",", [2, 7, 2, 8]],
#  [11, :tINTEGER, "3", [2, 8, 2, 9]],
#  [12, :nl, "\n", [2, 9, 2, 10]],
#  [13, :keyword_end, "end", [3, 0, 3, 3]]]

puts ast.tokens.map{_1[2]}.join
# def hoge
#   p 1,2,3
# end