Open19日前にコメント追加19

NSAdaptiveImageGlyphをハックしてカスタム絵文字を通知欄に出したい

文中からNSAdaptiveImageGlyphを取り出す

let range = NSRange(location: 0, length: textView.attributedText.length)
textView.attributedText.enumerateAttribute(.adaptiveImageGlyph, in: range) { adaptiveImageProvider, range, _ in
　let imageGlyph = (adaptiveImageProvider as! NSAdaptiveImageGlyph)
　print(imageGlyph.contentIdentifier)
　print(imageGlyph.contentDescription)
　print(imageGlyph.imageContent)
}

データサイズ52KB
imageContentの構造

imageContentはheicのメタデータ拡張であると考えられる

let url = Bundle.main.url(forResource: "NSAdaptiveImageGlyphImageContent", withExtension: "heic")!
let data = try! Data(contentsOf: url)
let imageSource = CGImageSourceCreateWithURL(url as CFURL, nil)!
let metadata = CGImageSourceCopyMetadataAtIndex(imageSource, 0, nil)!
print(metadata)

<CGImageMetadata 0x158092440> (
    tiff:DocumentName = CE0AF467-2EB5-481E-9B53-DA745BC26BF20
    tiff:XPosition = 0/1
    tiff:TileWidth = 160
    tiff:YPosition = 0/1
    dc:description = (
            "<CGImageMetadataTag 0x302a08640> dc:[x-default] = fox glasses, Qualifiers = (\n    \"<CGImageMetadataTag 0x302a086e0> xml:lang = x-default\"\n)"
        )
    Iptc4xmpExt:DigitalSourceType = http://cv.iptc.org/newscodes/digitalsourcetype/trainedAlgorithmicMedia
    tiff:TileLength = 160
    photoshop:Credit = Apple Image Playground
    tiff:Orientation = 1
    iio:hasXMP = True
    xmp:CreatorTool = Apple TextKit
)

heicの中に複数のサイズの画像が格納されている

160
40
64
96
320

Iptc4xmpExt:DigitalSourceType = http://cv.iptc.org/newscodes/digitalsourcetype/trainedAlgorithmicMedia

これは生成AIで作成したコンテンツであることを示すタグ

tiff:DocumentName = CE0AF467-2EB5-481E-9B53-DA745BC26BF20

これはNSAdaptiveImageGlyphのcontentIdentifierと一致する

let url = Bundle.main.url(forResource: "NSAdaptiveImageGlyphImageContent", withExtension: "heic")!
let data = try! Data(contentsOf: url)
let adaptiveImageGlyph = NSAdaptiveImageGlyph(imageContent: data)
print(adaptiveImageGlyph.contentIdentifier)

このheicはNSAdaptiveImageGlyphのimageContentとして復元可能

復元したNSAdaptiveImageGlyphは挿入可能

let url = Bundle.main.url(forResource: "NSAdaptiveImageGlyphImageContent", withExtension: "heic")!
let data = try! Data(contentsOf: url)
let adaptiveImageGlyph = NSAdaptiveImageGlyph(imageContent: data)
let attr = NSAttributedString(adaptiveImageGlyph: adaptiveImageGlyph, attributes: [:])
textView.insertAttributedText(attr)

自作のheicでNSAdapativeImageGlyphを追加する

let exportURL = URL(filePath: NSTemporaryDirectory()).appending(component: "new.dat")
try? FileManager.default.removeItem(at: exportURL)
let destination = CGImageDestinationCreateWithURL(
        exportURL as CFURL, UTType.heic.identifier as CFString, 1, nil)!

let cgImage = createCGImage(width: 160, height: 160)!
CGImageDestinationAddImage(destination, cgImage, [:] as CFDictionary)

CGImageDestinationFinalize(destination)

let data = try! Data(contentsOf: exportURL)
let adaptiveImageGlyph = NSAdaptiveImageGlyph(imageContent: data)
let attr = NSAttributedString(adaptiveImageGlyph: adaptiveImageGlyph, attributes: [:])
textView.insertAttributedText(attr)

writeImageAtIndex:1025: ⭕️ ERROR: 'Example' is trying to save an opaque image (160x160) with 'AlphaLast'. This would unnecessarily increase the file size and will double (!!!) the required memory when decoding the image --> ignoring alpha.

↑このエラーはCGImageの生成時のものなので関係ない

認識はしないが、insertすると空白の文字が入る。
NSAdaptiveImageGlyphの時点でimageContentが0なので変なデータを入れるとここで弾かれる

metadataを移植した画像１つのheicはNSAdaptiveImageGlyphとして認識した

let url = Bundle.main.url(forResource: "NSAdaptiveImageGlyphImageContent", withExtension: "heic")!
let data = try! Data(contentsOf: url)
let imageSource = CGImageSourceCreateWithURL(url as CFURL, nil)!
let metadata = CGImageSourceCopyMetadataAtIndex(imageSource, 0, nil)!

let exportURL = URL(filePath: NSTemporaryDirectory()).appending(component: "new.dat")
try? FileManager.default.removeItem(at: exportURL)
let destination = CGImageDestinationCreateWithURL(
        exportURL as CFURL, UTType.heic.identifier as CFString, 1, nil)!

let cgImage = createCGImage(width: 160, height: 160)!
let image = UIImage(cgImage: cgImage)
CGImageDestinationAddImageAndMetadata(destination, cgImage, metadata, nil)
CGImageDestinationFinalize(destination)

let data2 = try! Data(contentsOf: exportURL)
let adaptiveImageGlyph = NSAdaptiveImageGlyph(imageContent: data2)
print(adaptiveImageGlyph.contentIdentifier)
print(adaptiveImageGlyph.contentDescription)
print(adaptiveImageGlyph.imageContent)
let attr = NSAttributedString(adaptiveImageGlyph: adaptiveImageGlyph, attributes: [:])
textView.insertAttributedText(attr)

CE0AF467-2EB5-481E-9B53-DA745BC26BF20
fox glasses
3637 bytes

自前のmetadataでも成功した。
tiff:DocumentNameさえあれば大丈夫ぽい

func makeMetadata() -> CGImageMetadata {
        let mutableMetadata = CGImageMetadataCreateMutable()
        CGImageMetadataSetValueWithPath(
                mutableMetadata,
                nil,
                "tiff:DocumentName" as CFString,
                NSString(string: "CE0AF467-2EB5-481E-9B53-DA745BC26BF20")
        )
        return mutableMetadata
}

png画像からカスタムなNSAdaptiveImageGlyphを作る

let imageContent = NSMutableData()
let destination = CGImageDestinationCreateWithData(
    imageContent,
    UTType.heic.identifier as CFString,
    1,
    nil
)!

let url = Bundle.main.url(forResource: "blobcat", withExtension: "png")!
let image = UIImage(contentsOfFile: url.path())!.cgImage!

let metadata = CGImageMetadataCreateMutable()
CGImageMetadataSetValueWithPath(
    metadata,
    nil,
    "tiff:DocumentName" as CFString,
    NSString(string: "8f0ea1a8-704e-495c-beb9-5eb4dcc26783")
)

CGImageDestinationAddImageAndMetadata(destination, image, metadata, nil)
CGImageDestinationFinalize(destination)

let adaptiveImageGlyph = NSAdaptiveImageGlyph(imageContent: imageContent as Data)
let attr = NSAttributedString(adaptiveImageGlyph: adaptiveImageGlyph, attributes: [:])
textView.insertAttributedText(attr)

通知での活用
INSendMessageIntentでのみ許可されているので、あくまでコミュニケーション用であるっぽい。

UNNotificationAttributedMessageContextがそもそもちゃんと動いていない気がする

壊れてなかった、Communication NotificationのCapabilityが必要でした。

カスタム絵文字を通知に出すことに成功