Capturing Screen Recordings for Failed iOS UI Tests in CI Environments

This article is for the 6th day of the Software Testing Advent Calendar 2023.

When automatically running UI tests in a CI environment, it is important to prepare as much information as possible to investigate the cause of test failures.
In particular, screen recordings when a test fails can be a powerful aid.

Platforms like Firebase Test Lab or Bitrise have features to automatically collect screen recordings.
However, if you are using GitHub Actions or CircleCI, you need to build that functionality yourself.

In this article, I will explain how to keep screen recordings that can be used with GitHub Actions, CircleCI, etc.

Prerequisites

I assume a situation where UI test cases are implemented using XCUITest.

Additionally, iOS Simulators provide a feature to collect screen recordings via commands from the Mac where the Simulator is running.

xcrun simctl io booted recordVideo "test-video.mov"

In this article, we will use the above feature to collect screen recordings during test case execution.

Overview of the Approach

On the Mac running the Simulator, we set up a server that has the functionality to start, end, and discard screen recordings upon request.

At the start of a test case, it requests the server to start screen recording.

Also, at the end of a test case, it requests the server to end the screen recording.
At this point, if the test case was successful, it requests the disposal of the screen recording.

Screen recordings that remain without being discarded are kept as artifacts of the CI execution result.

Implementation Details

Set up a server to control screen recordings

You can use any technology for the server, but here is an example script using Ruby.

recording_server.rb

require 'fileutils'
require 'sinatra'

post '/record_video/:udid/:test_name' do
  recordings_dir = 'recording'
  video_base_name = "#{recordings_dir}/#{params['test_name']}"
  recordings = (0..Dir["#{recordings_dir}/*"].length + 1).to_a
  body = JSON.parse(request.body.read)
  FileUtils.mkdir_p(recordings_dir)

  video_file = ''
  if body['delete']
    recordings.reverse_each do |i|
      video_file = "#{video_base_name}_#{i}.mp4"
      break if File.exist?(video_file)
    end
  else
    recordings.each do |i|
      video_file = "#{video_base_name}_#{i}.mp4"
      break unless File.exist?(video_file)
    end
  end

  if body['stop']
    # Stop recording
    simctl_processes = `pgrep simctl`.strip.split("\n")
    simctl_processes.each { |pid| `kill -s SIGINT #{pid}` }

    # If the deletion parameter is included in the request, delete the recording
    File.delete(video_file) if body['delete'] && File.exist?(video_file)
  else
    # Start recording
    puts `xcrun simctl io #{params['udid']} recordVideo --codec h264 --force #{video_file} &`
  end
end

The script above sets up a server that receives HTTP communication from the Simulator.

The server process also executes and stops the xcrun simctl command for the Simulator.

Additionally, it handles the deletion of the recording files.

Request the server at the start and end of test cases

Create a parent class for test cases that inherits from XCUITest as follows.

XCTestCaseWithRecording.swift

import ImageIO
import MobileCoreServices
import XCTest

class XCTestCaseWithRecording: XCTestCase {
    static private let recordingServerOrigin = "http://localhost:4567"

    override func setUpWithError() throws {
        try super.setUpWithError()
        recordVideo()
    }

    override func tearDownWithError() throws {
        recordVideo(stop: true)
        try super.tearDownWithError()
    }

    private func recordVideo(stop: Bool = false) {
        let udid = ProcessInfo.processInfo.environment["SIMULATOR_UDID"] ?? ""

        // The property `name` is in a format like "-[(TestClassName) (TestCaseName)]", so we parse it
        let parts = name.components(separatedBy: " ")
        let testClassName = String(parts[0].dropFirst(2))
        let testCaseName = String(parts[1].dropLast(1))

        let urlString = "\(XCTestCaseWithRecording.recordingServerOrigin)/record_video/\(udid)/\(testClassName).\(testCaseName)"
        guard let url = URL(string: urlString) else { return }

        // If the test case succeeded, delete the screen recording
        let json: [String: Any] = ["delete": !isTestFailed(), "stop": stop]

        var request = URLRequest(url: url)
        request.httpMethod = "POST"
        request.httpBody = try? JSONSerialization.data(withJSONObject: json, options: [])
        URLSession.shared.dataTask(with: request).resume()
    }

    private func isTestFailed() -> Bool {
        if let testRun = testRun {
            let failureCount = testRun.failureCount + testRun.unexpectedExceptionCount
            return failureCount > 0
        }
        return false
    }
}

The code above obtains the current test class name and test case name and sends a request to the server set up earlier.
It requests to start recording before the test case begins and requests to stop recording after it finishes.
Furthermore, if the test succeeds, a deletion parameter is attached to the request to stop recording.

Use the parent test case class created in this way by inheriting it in each test case.

import XCTest

final class AccountTests: XCTestCaseWithRecording {
    func testLogin() throws {
        let app = XCUIApplication()
        app.launch()

        // Implement test processing here
    }
}

Keep the remaining screen recordings as CI artifacts

Run the defined server with the following command:

ruby recording_server.rb

When you run XCUITest in this state, the screen recording files of failed tests will be left in the recording folder.

Keep the recording folder as an artifact of the CI execution result.

References

https://testableapple.com/video-recording-of-failed-tests-on-ios/

https://riccardocipolleschi.medium.com/video-record-your-uitest-4cc5b75079af

https://qiita.com/Cychow/items/02c99b9fe8f8b822cca2