Open9

swiftでmp4→HLS→mp4変換

藤城藤城

iOS上でmp4 or mov動画をHLS変換して、再度mp4に戻したい

藤城藤城

mp4→HLS

mp4 or mov動画をHLS変換はAVAssetReaderで読み込んでAVAssetWriterで書き込み
HLS固有の設定はこのくらい

assetWriter.outputFileTypeProfile = .mpeg4AppleHLS
assetWriter.preferredOutputSegmentInterval = preferredOutputSegmentInterval
assetWriter.delegate = segmentStore
assetWriter.initialSegmentStartTime = .zero

変換全文
Concurrencyを使うとスッキリ！
適当に元動画を読み込んで書き込み先動画の設定
過不足ないかはわかってない

参考
https://developer.apple.com/documentation/avfoundation/media_reading_and_writing/writing_fragmented_mpeg-4_files_for_http_live_streaming

ソースコードはアコーディオンの中に

ソースコード全文
func encodeToHLS(from asset: AVAsset,
                 preferredOutputSegmentInterval: CMTime = CMTime(seconds: 6, preferredTimescale: 1),
                 videoDecompressionSettings: [String: Any] = [
                    kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_32BGRA
                 ],
                 audioDecompressionSettings: [String: Any] = [
                    AVFormatIDKey: kAudioFormatLinearPCM
                 ]) async throws -> [(type: AVAssetSegmentType, data: Data, report: AVAssetSegmentReport?)] {

    // MARK: 元動画の読み込み準備
    let assetReader = try AVAssetReader(asset: asset)
    let videoTrack = try await asset.loadTracks(withMediaType: .video).first!
    let audioTrack = try await asset.loadTracks(withMediaType: .audio).first!

    let videoOutput = AVAssetReaderTrackOutput(track: videoTrack, outputSettings: videoDecompressionSettings)
    assetReader.add(videoOutput)

    let audioOutput = AVAssetReaderTrackOutput(track: audioTrack, outputSettings: audioDecompressionSettings)
    assetReader.add(audioOutput)

    // MARK: 書き込み先動画の準備
    let assetWriter = AVAssetWriter(contentType: .mpeg4Movie)
    // HLS出力の設定
    assetWriter.outputFileTypeProfile = .mpeg4AppleHLS
    assetWriter.preferredOutputSegmentInterval = preferredOutputSegmentInterval
    // 変換したセグメントをためておく
    let segmentStore = SegmentStore()
    assetWriter.delegate = segmentStore

    let videoInput = AVAssetWriterInput(mediaType: .video, outputSettings: [
        AVVideoCodecKey: AVVideoCodecType.h264,
        AVVideoWidthKey: try await videoTrack.load(.naturalSize).width,
        AVVideoHeightKey: try await videoTrack.load(.naturalSize).height,
        AVVideoCompressionPropertiesKey: [
            AVVideoAverageBitRateKey: try await videoTrack.load(.estimatedDataRate),
        ]
    ])
    assetWriter.add(videoInput)

    let desc = try await audioTrack.load(.formatDescriptions).first!
    let audioDesc: AudioStreamBasicDescription = CMAudioFormatDescriptionGetStreamBasicDescription(desc)!.pointee
    let audioInput = AVAssetWriterInput(mediaType: .audio, outputSettings: [
        AVFormatIDKey: audioDesc.mFormatID,
        AVSampleRateKey: audioDesc.mSampleRate,
        AVNumberOfChannelsKey: audioDesc.mChannelsPerFrame,
        AVEncoderBitRateKey: try await audioTrack.load(.estimatedDataRate)
    ])
    assetWriter.add(audioInput)

    // MARK: 変換開始
    assetWriter.initialSegmentStartTime = .zero // HLSの場合、設定必要
    assetReader.startReading()
    assetWriter.startWriting()
    assetWriter.startSession(atSourceTime: .zero)

    async let video: Void = withCheckedContinuation { (continuation: CheckedContinuation<Void, Never>) in
        videoInput.requestMediaDataWhenReady(on: DispatchQueue(label: "videoQueue")) {
            while videoInput.isReadyForMoreMediaData {
                if let sampleBuffer = videoOutput.copyNextSampleBuffer() {
                    videoInput.append(sampleBuffer)
                } else {
                    videoInput.markAsFinished()
                    continuation.resume()
                }
            }
        }
    }
    async let audio: Void = withCheckedContinuation { (continuation: CheckedContinuation<Void, Never>) in
        audioInput.requestMediaDataWhenReady(on: DispatchQueue(label: "audioQueue")) {
            while audioInput.isReadyForMoreMediaData {
                if let sampleBuffer = audioOutput.copyNextSampleBuffer() {
                    audioInput.append(sampleBuffer)
                } else {
                    audioInput.markAsFinished()
                    continuation.resume()
                }
            }
        }
    }
    // 動画と音声の両方が終了するまで待つ
    await _ = (video, audio)

    await assetWriter.finishWriting()

    return segmentStore.segments
}

class SegmentStore: NSObject, AVAssetWriterDelegate {
    var segments: [(type: AVAssetSegmentType, data: Data, report: AVAssetSegmentReport?)] = []

    func assetWriter(_ writer: AVAssetWriter,
                     didOutputSegmentData segmentData: Data,
                     segmentType: AVAssetSegmentType,
                     segmentReport: AVAssetSegmentReport?) {
        segments.append((type: segmentType, data: segmentData, report: segmentReport))
    }
}

HLS → mp4

init.mp4とsegment1.m4s...setmentN.m4sのバイナリを繋げたらいいらしい？

https://stackoverflow.com/questions/23485759/combine-mpeg-dash-segments-ex-init-mp4-segments-m4s-back-to-a-full-source-m

とりあえず、全て繋げてみたら、AVPlayerで再生できた
mp4dumpしてみると、そのまま繋がっており、moofとmdatが複数回登場してる
これでいいのか、ちゃんとわかってない

let asset = AVAsset(url: videoURL)
let segments = try await encodeToHLS(from: asset)
let jointed: Data = Data(segments.map({ $0.data }).joined())

AVAssetExportSessionを使って再書き出ししてみる
この出力結果もAVPlayerで再生できた

最悪、init.mp4+segmentN.m4sをsegmentN.mp4として変換し、mp4をAVMutableCompositionで1つに結合しないとかなと思ってたけど、そこまでする必要はなさそう

let asset = AVAsset(url: jointedVideoURL)
let exportSession = AVAssetExportSession(asset: asset, presetName: AVAssetExportPresetPassthrough)!
exportSession.outputURL = exportedVideoURL
exportSession.outputFileType = .mp4
await exportSession.export()
DuongTHDuongTH

You save my life!
But, How to monitoring encodeToHLS?
I mean how to known how many progress complete.

Thanks!

藤城藤城

最初にAVAssetSegmentType.initializationが1つ、AVAssetSegmentType.separableがN個流れてくる
initializationはinit.mp4、separableがsegmentN.m4sとして保存して、m3u8作ってHLSで再生できることを確認

#EXTM3U
#EXT-X-TARGETDURATION:6
#EXT-X-VERSION:9
#EXT-X-MEDIA-SEQUENCE:1
#EXT-X-PLAYLIST-TYPE:VOD
#EXT-X-INDEPENDENT-SEGMENTS
#EXT-X-MAP:URI="init.mp4"
#EXTINF:6.0,
segment1.m4s
#EXTINF:6.0,
segment2.m4s
...
#EXTINF:6.0,
segmentN.m4s
#EXT-X-ENDLIST
藤城藤城

boxを眺めてみる
box typeの参考
https://hides.yokohama/2021/11/15/post-679/

mdatが入ってない

$ mp4dump init.mp4
[ftyp] size=8+20
  major_brand = iso5
  minor_version = 1
  compatible_brand = isom
  compatible_brand = iso5
  compatible_brand = hlsf
[moov] size=8+1119
  [mvhd] size=12+96
    timescale = 44100
    duration = 0
    duration(ms) = 0
  [trak] size=8+503
    ...
  [trak] size=8+420
    ...
  [mvex] size=8+64
    ...

こっちにはmdatがあるがftypが入ってない

$ mp4dump segment1.m4s
[moof] size=8+3858
  [mfhd] size=12+4
    sequence number = 1
  [traf] size=8+2448
    [tfhd] size=12+16, flags=20038
      track ID = 1
      default sample duration = 20
      default sample size = 48412
      default sample flags = 1010000
    [tfdt] size=12+8, version=1
      base media decode time = 29
    [trun] size=12+188, version=1, flags=e01
      sample count = 15
      data offset = 3874
    (trunの繰り返し)
    [trun] size=12+188, version=1, flags=e01
      sample count = 15
      data offset = 2609948
  [traf] size=8+1378
    [tfhd] size=12+16, flags=2001a
      track ID = 2
      sample description index = 1
      default sample duration = 1024
      default sample size = 6
    [tfdt] size=12+8, version=1
      base media decode time = 0
    [trun] size=12+104, flags=201
      sample count = 24
      data offset = 229452
    (trunの繰り返し)
    [trun] size=12+84, flags=201
      sample count = 19
      data offset = 2790079
[mdat] size=8+2792999
藤城藤城

init.mp4とsegment1.m4s...setmentN.m4sのバイナリを繋げたらいいらしい？

https://stackoverflow.com/questions/23485759/combine-mpeg-dash-segments-ex-init-mp4-segments-m4s-back-to-a-full-source-m

とりあえず、全て繋げてみたら、AVPlayerで再生できた
mp4dumpしてみると、そのまま繋がっており、moofとmdatが複数回登場してる
これでいいのか、ちゃんとわかってない

let asset = AVAsset(url: videoURL)
let segments = try await encodeToHLS(from: asset)
let jointed: Data = Data(segments.map({ $0.data }).joined())
藤城藤城

AVAssetExportSessionを使って再書き出ししてみる
この出力結果もAVPlayerで再生できた

最悪、init.mp4+segmentN.m4sをsegmentN.mp4として変換し、mp4をAVMutableCompositionで1つに結合しないとかなと思ってたけど、そこまでする必要はなさそう

let asset = AVAsset(url: jointedVideoURL)
let exportSession = AVAssetExportSession(asset: asset, presetName: AVAssetExportPresetPassthrough)!
exportSession.outputURL = exportedVideoURL
exportSession.outputFileType = .mp4
await exportSession.export()

ftypと1つmdatになってmoofが消えた
他のデバイスも含めて考えると、これを使う方が無難そう？

$ mp4dump exported.mp4
[ftyp] size=8+20
  major_brand = mp42
  minor_version = 1
  compatible_brand = isom
  compatible_brand = mp41
  compatible_brand = mp42
[mdat] size=16+5512640
[moov] size=8+8373
  [mvhd] size=12+96
    timescale = 44100
    duration = 529200
    duration(ms) = 12000
  [trak] size=8+5343
    ...
  [trak] size=8+2906
    ...
藤城藤城

［備忘録］
swift-nioを使ってサーバを建てる
https://github.com/apple/swift-nio

HLSのURLのAVAsset AVAsset(url: URL(string: "http://localhost:8080/hls.m3u8")!) を直接AVAssetReaderやAVAssetExportSessionに入れられなかった

let group = MultiThreadedEventLoopGroup(numberOfThreads: System.coreCount)
let bootstrap = ServerBootstrap(group: group)
    .serverChannelOption(ChannelOptions.backlog, value: 256)
    .serverChannelOption(ChannelOptions.socket(SocketOptionLevel(SOL_SOCKET), SO_REUSEADDR), value: 1)
    .childChannelInitializer { channel in
        channel.pipeline.configureHTTPServerPipeline(withErrorHandling: true).flatMap {
            channel.pipeline.addHandler(HTTPHandler(dirURL: URL.temporaryDirectory.appending(path: "hls")))
        }
    }
    .childChannelOption(ChannelOptions.socket(IPPROTO_TCP, TCP_NODELAY), value: 1)
    .childChannelOption(ChannelOptions.socket(SocketOptionLevel(SOL_SOCKET), SO_REUSEADDR), value: 1)

let serverChannel = try await bootstrap.bind(host: "127.0.0.1", port: 8080).get()
try await serverChannel.closeFuture.get()

class HTTPHandler: ChannelInboundHandler {
    typealias InboundIn = HTTPServerRequestPart
    typealias OutboundOut = HTTPServerResponsePart

    private let dirURL: URL

    init(dirURL: URL) {
        self.dirURL = dirURL
    }

    func channelRead(context: ChannelHandlerContext, data: NIOAny) {
        let reqPart = self.unwrapInboundIn(data)

        switch reqPart {
        case .head(let request):
            handleRequest(request, context: context)
        case .body, .end:
            break
        }
    }

    private func handleRequest(_ request: HTTPRequestHead, context: ChannelHandlerContext) {
        do {
            let filePath = dirURL.appendingPathComponent(request.uri).path
            let fileData = try Data(contentsOf: URL(fileURLWithPath: filePath))
            let response = HTTPResponseHead(version: .init(major: 1, minor: 1), status: .ok)
            context.write(self.wrapOutboundOut(.head(response)), promise: nil)
            context.write(self.wrapOutboundOut(.body(.byteBuffer(ByteBuffer(bytes: fileData)))), promise: nil)
            context.writeAndFlush(self.wrapOutboundOut(.end(nil)), promise: nil)
        } catch {
            let response = HTTPResponseHead(version: .init(major: 1, minor: 1), status: .notFound)
            context.write(self.wrapOutboundOut(.head(response)), promise: nil)
            context.writeAndFlush(self.wrapOutboundOut(.end(nil)), promise: nil)
        }
    }
}
藤城藤城

HLSをAVAssetDownloadURLSessionでダウンロードしてみる
movpkgとして保存されるっぽい
このAVAssetもAVAssetReaderやAVAssetExportSessionに入れるとエラーになった

https://developer.apple.com/documentation/avfoundation/offline_playback_and_storage/using_avfoundation_to_play_and_persist_http_live_streams

class ContentViewModel: AVAssetDownloadDelegate {
    var downloadSession: AVAssetDownloadURLSession?
    func downloadHLS() async {
        let asset = AVURLAsset(url: URL(string: "http://localhost:8080/hls.m3u8")!)
        let configuration = URLSessionConfiguration.background(withIdentifier: "downloadIdentifier")
        downloadSession = AVAssetDownloadURLSession(configuration: configuration,
                                                        assetDownloadDelegate: self,
                                                        delegateQueue: OperationQueue.main)
        let downloadTask = downloadSession!.makeAssetDownloadTask(asset: asset,
                                                                  assetTitle: "assetTitle",
                                                                  assetArtworkData: nil,
                                                                  options: nil)
        downloadTask?.resume()
    }

    func urlSession(_ session: URLSession, assetDownloadTask: AVAssetDownloadTask, didFinishDownloadingTo location: URL) {
        print("didFinishDownloadingTo location: \(location)")
        let asset = AVAsset(url: location)
    }
}