iTranslated by AI

The content below is an AI-generated translation. This is an experimental feature, and may contain errors. View original article
🎮

Building a Custom UI for Portrait Expression Editing with Next.js and ComfyUI-AdvancedLivePortrait

に公開
2024/09/12

Introduction

In my previous article, I tried the super powerful technology called "ComfyUI-AdvancedLivePortrait".

https://zenn.dev/asap/articles/c0a1b82a10f0e2

This technology serves as the foundation for services like the one shown below.

https://x.com/hirochuu8/status/1830921626533388713

By using this technology, you can freely edit the facial expressions and compositions of AI-generated human images, as well as images you have taken yourself.

In my last article, I made it possible to run "ComfyUI-AdvancedLivePortrait"—originally an extension for the "ComfyUI" image generation AI Web UI—using a standard Python script without depending on the Web UI itself.

Since I had gone through the effort of making it run independently from the Web UI, I wanted to try running it on a front-end I built myself. So, this time, I used Next.js to create a custom UI for editing portrait expressions.

What I Built

Here is the result. Please watch the video below.
https://youtu.be/wQsF2_ucioM

Since my front-end knowledge is still developing, the UI and functionality might feel a bit basic, but I believe it allows for very fast and stress-free image editing.
Although not shown in the video, you can save the created image by right-clicking the preview image and selecting save.

Tech Stack & Operating Environment

Tech Stack

I selected the following:

Front-end: Next.js (App Router)
Back-end: Python

Operating Environment

The operating environment is as follows:

Front-end server: M2 Mac
Back-end server: Ubuntu 20.04 (GPU: RTX 3060 12GB, RAM: 64GB)
(I have also confirmed it works on Google Colab, though it is slightly slower. Additionally, it will run on a CPU, albeit slowly.)

Deliverables

You can find it in the repository below.
https://github.com/personabb/AdvancedLivePortrait_Nextjs

How to Run

Environment Setup

It will work as long as you have an environment where Next.js+pnpm can be used for the front-end, and Python (or Google Colab) can be used for the back-end.
If you already have these available, please skip the steps below.

Environment Setup

Both Front-end and Back-end

If you have git installed, please clone the repository using the following command:

git clone https://github.com/personabb/AdvancedLivePortrait_Nextjs.git



If you do not have git installed, you can download it by clicking the green "Code" button on the following page and then clicking "Download ZIP" near the bottom.
https://github.com/personabb/AdvancedLivePortrait_Nextjs


After this, please execute your commands with this repository as the current directory.
You can place this repository on your Desktop or keep it in your Downloads folder, but make sure to set your terminal's current directory to this specific directory.

Front-end (Next.js Environment Setup)



The front-end uses Next.js (+ pnpm).
Please set up the environment as follows. (This assumes Mac; apologies to Windows users.)
(The materials and screenshots used here are from last year, so they might be a bit outdated. As long as you can eventually use Next.js (+ pnpm), you may use other methods for setup.)

Installing Homebrew (Skip if already installed)



We will install Homebrew, a package manager.
Launch the terminal and run the following command. You will be asked for a password, so please enter it.

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"



You will be asked to press the Enter key at some point. Please press it.


Run the following commands to add the path to your environment variables (execute them line by line):

echo 'eval "$(/opt/homebrew/bin/brew shellenv)"'  >> ~/.zshrc
eval "$(/opt/homebrew/bin/brew shellenv)"
PATH=/usr/local/bin:$PATH
export PATH



Run the following command; if the Homebrew version is displayed, the installation was successful.

 brew -v


Installing Node.js (Installing nodebrew)



Install nodebrew using the following command:

brew install nodebrew



Execute the command shown in the red box of the image below (please check carefully as it may vary from person to person):



If it looks like the image above, use this command:

/opt/homebrew/opt/nodebrew/bin/nodebrew setup_dirs






Next, add the path shown in the red box of the image above to your environment variables.
In the case of the image above, you can add it with the following command:

echo export PATH=$HOME/.nodebrew/current/bin:$PATH >> ~/.zshrc



However, please change the export PATH=$HOME/.nodebrew/current/bin:$PATH part to match your own execution screen (the red box part).


Afterward, update the settings to apply the environment variables:

source ~/.zshrc


Installing Node.js (Installing Node.js)



Use the installed nodebrew to install Node.js version 19.6.1.
(The version you install probably doesn't matter much. I'm writing this because I'm using the environment above, but I generally recommend the latest version.)


Run the following command:

nodebrew install v19.6.1



If "Installed successfully" is displayed, you're good to go.


Next, run the following command to make the installed version 19.6.1 usable:

nodebrew use v19.6.1



If "use v19.6.1" is displayed, it's successful.
By using nodebrew, you can install multiple versions and switch between them depending on your needs.
(Of course, as long as you can eventually use Next.js (+ pnpm), you may install it using other methods.)

Installing pnpm



Install pnpm using the following command:

npm install --global pnpm@8.6.0



With this, you can now use Next.js and pnpm.



Back-end (Using a Local PC)



Execute the following command to set the current directory to ./AdvancedLivePortrait_Nextjs/alp_backend.

./AdvancedLivePortrait_Nextjs/
cd alp_backend


Installing Python



As a prerequisite, we will use Python version 3.10 or 3.11.
We will install Python using pyenv to specify the version.


Python versions are managed with pyenv.
For information on how to install pyenv itself, please refer to the following:
https://qiita.com/koooooo/items/b21d87ffe2b56d0c589b


Once pyenv is installed, you can specify the Python version with the following commands:

./AdvancedLivePortrait_Nextjs/alp_backend/
pyenv install 3.10.14 # or 3.11.9, etc.
pyenv local 3.10.14 # or pyenv global 3.10.14



This sets the Python version.
Use pyenv global when you want to apply this version system-wide.
Use pyenv local when you want to apply this version only within the current directory.


Run the following command to verify that the Python version has been updated:

./AdvancedLivePortrait_Nextjs/alp_backend/
python -V
# Python 3.10.14


Setting up the Python Virtual Environment



Next, we will set up a virtual environment to install the necessary packages.
We will use venv for this.
Since venv is the official Python virtual environment module, it can be used without additional installation if Python is available.

./AdvancedLivePortrait_Nextjs/alp_backend/
python -m venv env
source env/bin/activate



From now on, whenever you run the back-end, please enter this virtual environment.
To enter the virtual environment next time, you only need to run the following command:

./AdvancedLivePortrait_Nextjs/alp_backend/
source env/bin/activate


Back-end (Using Google Colab)



If you are using Google Colab, the environment is already set up, so no special processing is required.
At the very least, please create a Google account.

Execution Preparation

Front-end

The current directory is assumed to be ./AdvancedLivePortrait_Nextjs.

Configure the Back-end URL

Create .env.local in the root of the repository (./) and configure it as follows.
The URL is the back-end server's URL. (The following is an example.)

.env.local
NEXT_PUBLIC_API_BASE_URL=http://192.168.0.xxx:8000

(Please check the private IP address of the back-end server in advance. The port is 8000.)
Specify the back-end server's URL in the http://192.168.0.xxx part.

Installation and Build of Necessary Packages for Next.js

Execute the following commands one by one.

./AdvancedLivePortrait_Nextjs
pnpm i
pnpm build

Back-end (For Local PC)

For back-end preparation, set ./AdvancedLivePortrait_Nextjs/alp_backend as the current directory and execute the subsequent commands.

Installing Python Packages

Install the necessary Python packages. Execute the following commands.

./AdvancedLivePortrait_Nextjs/alp_backend/

# Package installation
git clone https://github.com/PowerHouseMan/ComfyUI-AdvancedLivePortrait.git
pip install fastapi\[all\]
cd ComfyUI-AdvancedLivePortrait
pip install -r requirements.txt
cd ..

Back-end (For Google Colab)

Uploading the Code

First, upload the necessary code to Google Drive.
Place only the alp_backend directory directly under MyDrive on Google Drive.
(Since it's for the back-end, front-end files are not required.)
The directory structure will be as follows:

MyDrive/
    └ alp_backend/

Obtain an ngrok Access Token

ngrok is a service that allows you to easily expose network services running on a local PC to the outside.
We will use this feature to expose the API server started on Google Colab and connect to it from the front-end server.

Please refer to the following article to obtain an access token:
https://zenn.dev/protoout/articles/47-ngrok-setup-2022

Register the Obtained ngrok Access Token in Google Colab Environment Variables

Register the obtained access token in Colab's environment variables with the name NGROK.

It should look like the image above.

The following article provides an easy-to-understand guide on how to register it:
https://note.com/npaka/n/n79bb63e17685

Execution

Front-end

The current directory is assumed to be ./AdvancedLivePortrait_Nextjs.

Starting the Front-end Server

Execute the following command:

./AdvancedLivePortrait_Nextjs
pnpm dev

Once the IP address for connection (only accessible from the same PC) is displayed in the terminal, the preparation for the front-end server is complete.

Connecting from a Device

From the same PC as the front-end server, you can access the Web app by connecting to http://localhost:3000/ through a browser.

Back-end (For Local PC)

For the back-end, set ./AdvancedLivePortrait_Nextjs/alp_backend as the current directory and execute the subsequent commands.

Starting the Back-end Server

Run the following command to start the back-end server:

./alp_backend
python run.py

Once the connection IP address (you cannot connect to this specific address) is displayed in green text in the terminal, the back-end server is ready.

Back-end (For Google Colab)

Starting the Back-end Server

Run all cells in MyDrive/alp_backend/alp_api_backend.ipynb starting from the top.
(Google Drive authentication will be required, so please authorize it.)


It will take some time, but the last cell will produce an output like the one above.
Once this output appears, the back-end server is ready.

Here, a URL like the following will be displayed:
PUBLIC_URL: https://cefb-xx-xxx-xxx-xxx.ngrok-free.app
Please enter this URL into .env.local as follows:

.env.local
NEXT_PUBLIC_API_BASE_URL=https://cefb-xx-xxx-xxx-xxx.ngrok-free.app

Execution

Front-end

The current directory is assumed to be ./AdvancedLivePortrait_Nextjs.

Starting the Front-end Server

Execute the following command:

./AdvancedLivePortrait_Nextjs
pnpm dev

Once the connection IP address (only accessible from the same PC) is displayed in the terminal, the front-end server is ready.

Connecting from a Device

From the same PC as the front-end server, you can access the Web app by connecting to http://localhost:3000/ through a browser.

Back-end (For Local PC)

For the back-end, set ./AdvancedLivePortrait_Nextjs/alp_backend as the current directory and execute the subsequent commands.

Starting the Back-end Server

Run the following command to start the back-end server:

./alp_backend
python run.py

Once the connection IP address (you cannot connect to this specific address) is displayed in green text in the terminal, the back-end server is ready.

Back-end (For Google Colab)

Starting the Back-end Server

Run all cells in MyDrive/alp_backend/alp_api_backend.ipynb starting from the top.
(Google Drive authentication will be required, so please authorize it.)


It will take some time, but the last cell will produce an output like the one above.
Once this output appears, the back-end server is ready.

Here, a URL like the following will be displayed:
PUBLIC_URL: https://cefb-xx-xxx-xxx-xxx.ngrok-free.app
Please enter this URL into .env.local as follows:

.env.local
NEXT_PUBLIC_API_BASE_URL=https://cefb-xx-xxx-xxx-xxx.ngrok-free.app

Usage

Common

When you connect to the front-end server from a tablet or PC, a screen like the one below will be displayed.

You can select an image file by clicking the "Select File" button at the bottom of the screen.
Please select the image file you want to edit from here.

After that, as shown in the video at the beginning of the article, you can edit the image by moving the sliders or directly manipulating the numbers to the right.

Summary

In this article, I created a custom UI for editing portrait expressions using the technology from "ComfyUI-AdvancedLivePortrait".

I explained how to run "ComfyUI-AdvancedLivePortrait" programmatically without using the Web UI in my previous article, so please take a look at that as well.

https://zenn.dev/asap/articles/c0a1b82a10f0e2

Thank you for reading this far!

Acknowledgments

https://github.com/PowerHouseMan/ComfyUI-AdvancedLivePortrait
I have used a significant amount of the implementation code from this repository.
I would like to express my gratitude to the developers.

Code

I'm writing this down as a sort of memorandum.
If you are not interested, you can skip this part.

Full Front-end Code

AdvancedLivePortrait_Nextjs/src/app/page.tsx
AdvancedLivePortrait_Nextjs/src/app/page.tsx
"use client";

import { useState, useEffect, useCallback } from "react";
import debounce from "lodash/debounce";

export default function Home() {
  const [image, setImage] = useState<File | null>(null);
  const [preview, setPreview] = useState<string | null>(null);
  const [loading, setLoading] = useState(false); // Managing loading state
  const [isUpdating, setIsUpdating] = useState(false); // Flag to allow slider operation even during image processing
  const [isInitializing, setIsInitializing] = useState(false); // Managing initialization state

  // Obtain API base URL from environment variables
  const API_BASE_URL = process.env.NEXT_PUBLIC_API_BASE_URL;

  // Managing the state of parameters adjusted by sliders
  const [params, setParams] = useState({
    rotate_pitch: 0,
    rotate_yaw: 0,
    rotate_roll: 0,
    blink: 0,
    eyebrow: 0,
    wink: 0,
    pupil_x: 0,
    pupil_y: 0,
    aaa: 0,
    eee: 0,
    woo: 0,
    smile: 0, // Initial value is 0
  });

  const sliderDescriptions: { [key: string]: string } = {
    rotate_pitch: "Vertical rotation of the face (forward/backward)",
    rotate_yaw: "Horizontal rotation of the face (left/right)",
    rotate_roll: "Tilt of the face",
    blink: "Degree of blinking",
    eyebrow: "Eyebrow movement",
    wink: "Winking of one eye",
    pupil_x: "Horizontal movement of the pupils",
    pupil_y: "Vertical movement of the pupils",
    aaa: "Action of opening the mouth",
    eee: "Action of pronouncing the 'i' sound",
    woo: "Action of pronouncing the 'u' sound",
    smile: "Degree of smiling (0.5 = 50% smile)",
  };

  // Function called when parameters change
  const handleParamChange = (e: React.ChangeEvent<HTMLInputElement>) => {
    const { name, value } = e.target;
    setParams((prevParams) => ({
      ...prevParams,
      [name]: parseFloat(value),
    }));
    console.log(`Updated ${name} to ${value}`);
  };

  // Asynchronous function to send images and parameters to the API
  const sendUpdateToBackend = useCallback(
    debounce(async (newParams) => {
      if (image) {
        setIsUpdating(true); // Indicates that processing has started
        setLoading(true);

        const formData = new FormData();
        formData.append("file", image);
        Object.keys(newParams).forEach((key) => {
          formData.append(key, String(newParams[key as keyof typeof params]));
        });

        try {
          const response = await fetch(`${API_BASE_URL}/edit`, {
            method: "POST",
            body: formData,
          });

          if (!response.ok) {
            throw new Error(`API request failed with status ${response.status}`);
          }

          const blob = await response.blob();
          const editedImageURL = URL.createObjectURL(blob);
          setPreview(editedImageURL);
        } catch (error) {
          console.error("Error while sending API request:", error);
        } finally {
          setLoading(false); // End of loading
          setIsUpdating(false); // Indicates the end of image processing
        }
      }
    }, 200), // Set a 500ms delay
    [image, API_BASE_URL]
  );

  // Call debounced processing when parameters change
  useEffect(() => {
    if (image) {
      sendUpdateToBackend(params);
    }
  }, [params, sendUpdateToBackend]);

  // Added API request function for initialization
  const initializeImage = async (imageFile: File) => {
    const formData = new FormData();
    formData.append("file", imageFile);
  
    try {
      setIsInitializing(true); // Start initialization
      const response = await fetch(`${API_BASE_URL}/initialize`, {
        method: "POST",
        body: formData,
      });
  
      if (!response.ok) {
        throw new Error(`Initialization API request failed with status ${response.status}`);
      }
  
      const result = await response.json();  // Get JSON response
      console.log(result.message);  // Should display "Initialization completed"
    } catch (error) {
      console.error("Error during initialization API request:", error);
    } finally {
      setIsInitializing(false); // End initialization
    }   
  };
  
  // Function to display preview when an image is selected
  const handleImageChange = (e: React.ChangeEvent<HTMLInputElement>) => {
    const file = e.target.files?.[0] || null;
    setImage(file);
    if (file) {
      initializeImage(file);
      const reader = new FileReader();
      reader.onloadend = () => {
        setPreview(reader.result as string);
      };
      reader.readAsDataURL(file);
    }
  };

  // Reset button handler
  const handleReset = () => {
    setParams({
      rotate_pitch: 0,
      rotate_yaw: 0,
      rotate_roll: 0,
      blink: 0,
      eyebrow: 0,
      wink: 0,
      pupil_x: 0,
      pupil_y: 0,
      aaa: 0,
      eee: 0,
      woo: 0,
      smile: 0,
    });
  };

  return (
    <div className="container mx-auto p-4">
      <h1 className="text-2xl font-bold mb-4">Image Editing Tool</h1>

      <div className="flex">
        {/* Image Preview */}
        {preview && (
          <img
            src={preview}
            alt="Preview"
            className="mb-4 h-auto object-contain mr-8" // Add space to the right of the image
          />
        )}

        {/* Arrange sliders vertically */}
        <div className="flex flex-col items-start justify-center w-1/2 space-y-4">
          {Object.keys(params).map((param) => (
            <div key={param} className="flex items-center space-x-4">
              {/* Slider Name */}
              <label className="w-32">{param}</label>
              
              {/* Slider */}
              <input
                type="range"
                name={param}
                min={param === "smile" ? "-2" : "-20"} // Only smile is -2 to 2, others are -20 to 20
                max={param === "smile" ? "2" : "20"}
                step={param === "smile" ? "0.1" : "1"} // Both are 0.1 steps
                value={params[param as keyof typeof params]}
                onChange={handleParamChange}
                className="w-48 h-4" // Wider and slightly thicker slider
                disabled={isInitializing} // Disable slider during initialization
              />

              {/* Numeric Input Field */}
              <input
                type="number"
                name={param}
                min={param === "smile" ? "-2" : "-20"} // Only smile is -2 to 2, others are -20 to 20
                max={param === "smile" ? "2" : "20"}
                step={param === "smile" ? "0.1" : "1"}
                value={params[param as keyof typeof params]}
                onChange={handleParamChange}
                className="w-16 p-1 border rounded"
                disabled={isInitializing} // Disable slider during initialization
              />

              {/* Supplementary explanation */}
              <span className="text-gray-500">{sliderDescriptions[param]}</span>
            </div>
          ))}

          {/* Reset Button */}
          <button
            onClick={handleReset}
            className="bg-blue-500 hover:bg-blue-700 text-white font-bold py-2 px-4 rounded mt-4"
            disabled={isInitializing} // Disable reset button during initialization
          >
            Reset
          </button>
        </div>
      </div>

      {/* Image Upload */}
      <input type="file" accept="image/*" onChange={handleImageChange} className="mt-4" />

      {loading && <p>Processing image...</p>}
      {isUpdating && !loading && <p>Processing new image...</p>}
    </div>
  );
}

On the backend side, time-consuming processes such as downloading models and detecting face areas are handled by initializeImage during initialization (at the time the image is uploaded).

Additionally, moving the sliders results in continuous API requests to the backend. To reduce the load even slightly, I have implemented debounce processing so that a request is sent only once, 200ms after the slider stops moving.

The following article provides a clear explanation of debounce processing:
https://zenn.dev/k_zumi_dev/articles/37aa04f584d7df#デバウンス処理を使って猶予時間を設ける

Regarding the various parameters, the sliders are generally set to move within a range of -20 to 20, but the smile parameter specifically ranges from -2 to 2.

By pressing the reset button, all parameters can be returned to 0 in one go.

Full Back-end Code

AdvancedLivePortrait_Nextjs/alp_backend/run.py
AdvancedLivePortrait_Nextjs/alp_backend/run.py
import sys
sys.path.append("./ComfyUI-AdvancedLivePortrait")

from fastapi import FastAPI, UploadFile,File, Form
from fastapi import HTTPException
from fastapi.middleware.cors import CORSMiddleware
from PIL import Image
import io
import os
from fastapi.responses import FileResponse, JSONResponse, StreamingResponse
from module.advanced_live_portrait import AdvancedLivePortrait_execution_prepare, AdvancedLivePortrait_execution_main

app = FastAPI()

# CORS settings
origins = [
    "*" # Specify the front-end URL
    # Add other origins as needed
]

app.add_middleware(
    CORSMiddleware,
    allow_origins=origins,
    allow_credentials=True,
    allow_methods=["*"],  # Allow all HTTP methods
    allow_headers=["*"],  # Allow all headers
)

editor = None
img_tensor = None
prepared_face = None


@app.post("/initialize")
async def initialize_image_endpoint(file: UploadFile = File(...)):
    global editor, img_tensor, prepared_face
    # Save the uploaded image to a temporary file
    input_image_path = f"./tmp/{file.filename}"
    output_image_path = f"./tmp/initialized_{file.filename}"
    os.makedirs(os.path.dirname(input_image_path), exist_ok=True)
    
    with open(input_image_path, "wb") as f:
        f.write(await file.read())
    
    # Execute initialization process
    editor, img_tensor, prepared_face = AdvancedLivePortrait_execution_prepare(input_image_path, output_image_path)
    
    # Return initialization completion message
    return JSONResponse(content={"message": "Initialization completed", "status": "success"})

# Image editing function (wrapper for AdvancedLivePortrait_execution)
def edit_image(output_path, parameters):
    # Code to call the AdvancedLivePortrait_execution function goes here
    # parameters are parameters sent from Next.js
    edited_image_pil = AdvancedLivePortrait_execution_main(editor, img_tensor, prepared_face, output_path, parameters)
    # Save the image to a memory buffer
    img_byte_arr = io.BytesIO()
    edited_image_pil.save(img_byte_arr, format='PNG')  # Save in PNG format (JPEG is also possible)
    img_byte_arr.seek(0)  # Move the pointer to the beginning of the buffer

    return img_byte_arr

@app.post("/edit")
async def edit_image_endpoint(
    file: UploadFile = File(...),
    rotate_pitch: float = Form(0),
    rotate_yaw: float = Form(0),
    rotate_roll: float = Form(0),
    blink: float = Form(0),
    eyebrow: float = Form(0),
    wink: float = Form(0),
    pupil_x: float = Form(0),
    pupil_y: float = Form(0),
    aaa: float = Form(0),
    eee: float = Form(0),
    woo: float = Form(0),
    smile: float = Form(0.5),
):
    # Save the uploaded image to a temporary file
    input_image_path = f"./tmp/{file.filename}"
    output_image_path = f"./tmp/edited_{file.filename}"

    with open(input_image_path, "wb") as f:
        f.write(await file.read())

    # Pass parameters to AdvancedLivePortrait_execution to perform editing
    parameters = [
        rotate_pitch, rotate_yaw, rotate_roll, blink, eyebrow, wink,
        pupil_x, pupil_y, aaa, eee, woo, smile
    ]

    print(f"Received parameters: {parameters}")
    img_byte_arr = edit_image(output_image_path, parameters)

    # Return the edited image
    #return FileResponse(output_image_path)
    return StreamingResponse(img_byte_arr, media_type="image/png")



if __name__ == "__main__":
    import uvicorn
    uvicorn.run("run:app", host="0.0.0.0", port=8000, reload=True)

I have implemented two APIs: @app.post("/initialize") and @app.post("/edit").

@app.post("/initialize") is called when an image is uploaded on the front-end, and it pre-executes everything from downloading the face detection model to the face detection itself.

Next, when parameters are changed on the front-end, @app.post("/edit") is called to perform the image editing process.
The processed image is sent directly to the front-end.

AdvancedLivePortrait_Nextjs/alp_backend/module/advanced_live_portrait.py
AdvancedLivePortrait_Nextjs/alp_backend/module/advanced_live_portrait.py
import os
import sys
import numpy as np
import torch
import cv2
from PIL import Image
import time
import copy
import dill
import yaml
from ultralytics import YOLO
from module.utils import checkpoint_pickle

current_directory = "./ComfyUI-AdvancedLivePortrait"
models_dir = "./ComfyUI-AdvancedLivePortrait/models"
os.makedirs(models_dir, exist_ok=True)

import math
import struct
import safetensors.torch
import logging
import itertools

def load_torch_file(ckpt, safe_load=False, device=None):
    if device is None:
        device = torch.device("cpu")
    if ckpt.lower().endswith(".safetensors") or ckpt.lower().endswith(".sft"):
        sd = safetensors.torch.load_file(ckpt, device=device.type)
    else:
        if safe_load:
            if not 'weights_only' in torch.load.__code__.co_varnames:
                logging.warning("Warning torch.load doesn't support weights_only on this pytorch version, loading unsafely.")
                safe_load = False
        if safe_load:
            pl_sd = torch.load(ckpt, map_location=device, weights_only=True)
        else:
            pl_sd = torch.load(ckpt, map_location=device, pickle_module=checkpoint_pickle)
        if "global_step" in pl_sd:
            logging.debug(f"Global Step: {pl_sd['global_step']}")
        if "state_dict" in pl_sd:
            sd = pl_sd["state_dict"]
        else:
            sd = pl_sd
    return sd


from LivePortrait.live_portrait_wrapper import LivePortraitWrapper
from LivePortrait.utils.camera import get_rotation_matrix
from LivePortrait.config.inference_config import InferenceConfig

from LivePortrait.modules.spade_generator import SPADEDecoder
from LivePortrait.modules.warping_network import WarpingNetwork
from LivePortrait.modules.motion_extractor import MotionExtractor
from LivePortrait.modules.appearance_feature_extractor import AppearanceFeatureExtractor
from LivePortrait.modules.stitching_retargeting_network import StitchingRetargetingNetwork
from collections import OrderedDict

cur_device = None
def get_device():
    global cur_device
    if cur_device == None:
        if torch.cuda.is_available():
            cur_device = torch.device('cuda')
            print("Uses CUDA device.")
            """elif torch.backends.mps.is_available():
            cur_device = torch.device('mps')
            print("Uses MPS device.")"""
        else:
            cur_device = torch.device('cpu')
            print("Uses CPU device.")
    return cur_device

def tensor2pil(image):
    return Image.fromarray(np.clip(255. * image.cpu().numpy().squeeze(), 0, 255).astype(np.uint8))
def pil2tensor(image):
    return torch.from_numpy(np.array(image).astype(np.float32) / 255.0).unsqueeze(0)
def rgb_crop(rgb, region):
    return rgb[region[1]:region[3], region[0]:region[2]]

def rgb_crop_batch(rgbs, region):
    return rgbs[:, region[1]:region[3], region[0]:region[2]]
def get_rgb_size(rgb):
    return rgb.shape[1], rgb.shape[0]
def create_transform_matrix(x, y, s_x, s_y):
    return np.float32([[s_x, 0, x], [0, s_y, y]])

def get_model_dir(m):
    return os.path.join(models_dir, m)

def calc_crop_limit(center, img_size, crop_size):
    pos = center - crop_size / 2
    if pos < 0:
        crop_size += pos * 2
        pos = 0

    pos2 = pos + crop_size

    if img_size < pos2:
        crop_size -= (pos2 - img_size) * 2
        pos2 = img_size
        pos = pos2 - crop_size

    return pos, pos2, crop_size

def retargeting(delta_out, driving_exp, factor, idxes):
    for idx in idxes:
        #delta_out[0, idx] -= src_exp[0, idx] * factor
        delta_out[0, idx] += driving_exp[0, idx] * factor

class PreparedSrcImg:
    def __init__(self, src_rgb, crop_trans_m, x_s_info, f_s_user, x_s_user, mask_ori):
        self.src_rgb = src_rgb
        self.crop_trans_m = crop_trans_m
        self.x_s_info = x_s_info
        self.f_s_user = f_s_user
        self.x_s_user = x_s_user
        self.mask_ori = mask_ori

import requests
from tqdm import tqdm

class LP_Engine:
    pipeline = None
    detect_model = None
    mask_img = None
    temp_img_idx = 0

    def get_temp_img_name(self):
        self.temp_img_idx += 1
        return "expression_edit_preview" + str(self.temp_img_idx) + ".png"

    def download_model(_, file_path, model_url):
        print('AdvancedLivePortrait: Downloading model...')
        response = requests.get(model_url, stream=True)
        try:
            if response.status_code == 200:
                total_size = int(response.headers.get('content-length', 0))
                block_size = 1024  # 1 Kibibyte

                # tqdm will display a progress bar
                with open(file_path, 'wb') as file, tqdm(
                        desc='Downloading',
                        total=total_size,
                        unit='iB',
                        unit_scale=True,
                        unit_divisor=1024,
                ) as bar:
                    for data in response.iter_content(block_size):
                        bar.update(len(data))
                        file.write(data)

        except requests.exceptions.RequestException as err:
            print('AdvancedLivePortrait: Model download failed: {err}')
            print(f'AdvancedLivePortrait: Download it manually from: {model_url}')
            print(f'AdvancedLivePortrait: And put it in {file_path}')
        except Exception as e:
            print(f'AdvancedLivePortrait: An unexpected error occurred: {e}')

    def remove_ddp_dumplicate_key(_, state_dict):
        state_dict_new = OrderedDict()
        for key in state_dict.keys():
            state_dict_new[key.replace('module.', '')] = state_dict[key]
        return state_dict_new

    def filter_for_model(_, checkpoint, prefix):
        filtered_checkpoint = {key.replace(prefix + "_module.", ""): value for key, value in checkpoint.items() if
                               key.startswith(prefix)}
        return filtered_checkpoint

    def load_model(self, model_config, model_type):

        device = get_device()

        if model_type == 'stitching_retargeting_module':
            ckpt_path = os.path.join(get_model_dir("liveportrait"), "retargeting_models", model_type + ".pth")
        else:
            ckpt_path = os.path.join(get_model_dir("liveportrait"), "base_models", model_type + ".pth")

        is_safetensors = None
        if os.path.isfile(ckpt_path) == False:
            is_safetensors = True
            ckpt_path = os.path.join(get_model_dir("liveportrait"), model_type + ".safetensors")
            if os.path.isfile(ckpt_path) == False:
                self.download_model(ckpt_path,
                "https://huggingface.co/Kijai/LivePortrait_safetensors/resolve/main/" + model_type + ".safetensors")
        model_params = model_config['model_params'][f'{model_type}_params']
        if model_type == 'appearance_feature_extractor':
            model = AppearanceFeatureExtractor(**model_params).to(device)
        elif model_type == 'motion_extractor':
            model = MotionExtractor(**model_params).to(device)
        elif model_type == 'warping_module':
            model = WarpingNetwork(**model_params).to(device)
        elif model_type == 'spade_generator':
            model = SPADEDecoder(**model_params).to(device)
        elif model_type == 'stitching_retargeting_module':
            # Special handling for stitching and retargeting module
            config = model_config['model_params']['stitching_retargeting_module_params']
            checkpoint = load_torch_file(ckpt_path)

            stitcher = StitchingRetargetingNetwork(**config.get('stitching'))
            if is_safetensors:
                stitcher.load_state_dict(self.filter_for_model(checkpoint, 'retarget_shoulder'))
            else:
                stitcher.load_state_dict(self.remove_ddp_dumplicate_key(checkpoint['retarget_shoulder']))
            stitcher = stitcher.to(device)
            stitcher.eval()

            return {
                'stitching': stitcher,
            }
        else:
            raise ValueError(f"Unknown model type: {model_type}")


        model.load_state_dict(load_torch_file(ckpt_path))
        model.eval()
        return model

    def load_models(self):
        model_path = get_model_dir("liveportrait")
        if not os.path.exists(model_path):
            os.mkdir(model_path)

        model_config_path = os.path.join(current_directory, 'LivePortrait', 'config', 'models.yaml')
        model_config = yaml.safe_load(open(model_config_path, 'r'))

        appearance_feature_extractor = self.load_model(model_config, 'appearance_feature_extractor')
        motion_extractor = self.load_model(model_config, 'motion_extractor')
        warping_module = self.load_model(model_config, 'warping_module')
        spade_generator = self.load_model(model_config, 'spade_generator')
        stitching_retargeting_module = self.load_model(model_config, 'stitching_retargeting_module')

        self.pipeline = LivePortraitWrapper(InferenceConfig(), appearance_feature_extractor, motion_extractor, warping_module, spade_generator, stitching_retargeting_module)

    def get_detect_model(self):
        if self.detect_model == None:
            model_dir = get_model_dir("ultralytics")
            if not os.path.exists(model_dir): os.mkdir(model_dir)
            model_path = os.path.join(model_dir, "face_yolov8n.pt")
            if not os.path.exists(model_path):
                self.download_model(model_path, "https://huggingface.co/Bingsu/adetailer/resolve/main/face_yolov8n.pt")
            self.detect_model = YOLO(model_path)

        return self.detect_model

    def resize_image_to_stride(self,image_tensor, stride=32):
        # Get image height and width
        _, _, h, w = image_tensor.shape

        # Resize to dimensions divisible by the stride
        new_h = (h // stride) * stride
        new_w = (w // stride) * stride

        # Convert tensor to NumPy and resize
        image_np = image_tensor.squeeze(0).permute(1, 2, 0).cpu().numpy()  # (B, C, H, W) -> (H, W, C)
        resized_image_np = cv2.resize(image_np, (new_w, new_h), interpolation=cv2.INTER_LINEAR)

        # Convert resized image back to tensor
        resized_image_tensor = torch.from_numpy(resized_image_np).permute(2, 0, 1).unsqueeze(0).float()  # (H, W, C) -> (B, C, H, W)

        return resized_image_tensor

    def get_face_bboxes(self, image_rgb):
        detect_model = self.get_detect_model()

        # Check if the image is loaded correctly
        if image_rgb is None or image_rgb.size == 0:
            raise ValueError("The input image is empty or invalid.")

        # Ensure the image is 3-channel (RGB)
        if len(image_rgb.shape) == 2:  # If grayscale, convert to 3 channels
            image_rgb = cv2.cvtColor(image_rgb, cv2.COLOR_GRAY2RGB)

        elif len(image_rgb.shape) == 3 and image_rgb.shape[2] == 1:  # Also convert 1-channel mono to 3 channels
            image_rgb = cv2.cvtColor(image_rgb, cv2.COLOR_GRAY2RGB)

        # Verify dimensions and expand to 4D (batch format expected by YOLO)
        if len(image_rgb.shape) == 3:
            image_rgb = np.expand_dims(image_rgb, axis=0)  # (height, width, 3) -> (1, height, width, 3)

        # Change dimensions to (batch_size, channels, height, width) format
        image_rgb = torch.from_numpy(image_rgb).permute(0, 3, 1, 2).float()  # (1, height, width, 3) -> (1, 3, height, width)

        # Normalize pixel values from 0-255 to 0-1
        image_rgb /= 255.0

        # Check size for debugging
        print(f"Image dimensions before YOLO: {image_rgb.shape}")

        image_rgb = self.resize_image_to_stride(image_rgb, stride=32)


        pred = detect_model(image_rgb, conf=0.7, device="")  # Input to YOLO model
        return pred[0].boxes.xyxy.cpu().numpy()  # Return detected bounding boxes

    def detect_face(self, image_rgb, crop_factor, sort = True):
        bboxes = self.get_face_bboxes(image_rgb)
        w, h = get_rgb_size(image_rgb)

        print(f"w, h:{w, h}")

        cx = w / 2
        min_diff = w
        best_box = None
        for x1, y1, x2, y2 in bboxes:
            bbox_w = x2 - x1
            if bbox_w < 30: continue
            diff = abs(cx - (x1 + bbox_w / 2))
            if diff < min_diff:
                best_box = [x1, y1, x2, y2]
                print(f"diff, min_diff, best_box:{diff, min_diff, best_box}")
                min_diff = diff

        if best_box == None:
            print("Failed to detect face!!")
            return [0, 0, w, h]

        x1, y1, x2, y2 = best_box

        #for x1, y1, x2, y2 in bboxes:
        bbox_w = x2 - x1
        bbox_h = y2 - y1

        crop_w = bbox_w * crop_factor
        crop_h = bbox_h * crop_factor

        crop_w = max(crop_h, crop_w)
        crop_h = crop_w

        kernel_x = int(x1 + bbox_w / 2)
        kernel_y = int(y1 + bbox_h / 2)

        new_x1 = int(kernel_x - crop_w / 2)
        new_x2 = int(kernel_x + crop_w / 2)
        new_y1 = int(kernel_y - crop_h / 2)
        new_y2 = int(kernel_y + crop_h / 2)

        if not sort:
            return [int(new_x1), int(new_y1), int(new_x2), int(new_y2)]

        if new_x1 < 0:
            new_x2 -= new_x1
            new_x1 = 0
        elif w < new_x2:
            new_x1 -= (new_x2 - w)
            new_x2 = w
            if new_x1 < 0:
                new_x2 -= new_x1
                new_x1 = 0

        if new_y1 < 0:
            new_y2 -= new_y1
            new_y1 = 0
        elif h < new_y2:
            new_y1 -= (new_y2 - h)
            new_y2 = h
            if new_y1 < 0:
                new_y2 -= new_y1
                new_y1 = 0

        if w < new_x2 and h < new_y2:
            over_x = new_x2 - w
            over_y = new_y2 - h
            over_min = min(over_x, over_y)
            new_x2 -= over_min
            new_y2 -= over_min

        return [int(new_x1), int(new_y1), int(new_x2), int(new_y2)]


    def calc_face_region(self, square, dsize):
        region = copy.deepcopy(square)
        is_changed = False
        if dsize[0] < region[2]:
            region[2] = dsize[0]
            is_changed = True
        if dsize[1] < region[3]:
            region[3] = dsize[1]
            is_changed = True

        return region, is_changed

    def expand_img(self, rgb_img, square):
        crop_trans_m = create_transform_matrix(max(-square[0], 0), max(-square[1], 0), 1, 1)
        new_img = cv2.warpAffine(rgb_img, crop_trans_m, (square[2] - square[0], square[3] - square[1]),
                                        cv2.INTER_LINEAR)
        return new_img

    def get_pipeline(self):
        if self.pipeline == None:
            print("Load pipeline...")
            self.load_models()

        return self.pipeline

    def prepare_src_image(self, img):
        h, w = img.shape[:2]
        input_shape = [256,256]
        if h != input_shape[0] or w != input_shape[1]:
            if 256 < h: interpolation = cv2.INTER_AREA
            else: interpolation = cv2.INTER_LINEAR
            x = cv2.resize(img, (input_shape[0], input_shape[1]), interpolation = interpolation)
        else:
            x = img.copy()

        if x.ndim == 3:
            x = x[np.newaxis].astype(np.float32) / 255.  # HxWx3 -> 1xHxWx3, normalized to 0~1
        elif x.ndim == 4:
            x = x.astype(np.float32) / 255.  # BxHxWx3, normalized to 0~1
        else:
            raise ValueError(f'img ndim should be 3 or 4: {x.ndim}')
        x = np.clip(x, 0, 1)  # clip to 0~1
        x = torch.from_numpy(x).permute(0, 3, 1, 2)  # 1xHxWx3 -> 1x3xHxW
        x = x.to(get_device())
        return x

    def GetMaskImg(self):
        if self.mask_img is None:
            path = os.path.join(current_directory, "LivePortrait/utils/resources/mask_template.png")
            self.mask_img = cv2.imread(path, cv2.IMREAD_COLOR)
            if self.mask_img is None:
                raise FileNotFoundError(f"Mask image not found at path: {path}")
        return self.mask_img


    def crop_face(self, img_rgb, crop_factor):
        crop_region = self.detect_face(img_rgb, crop_factor)
        face_region, is_changed = self.calc_face_region(crop_region, get_rgb_size(img_rgb))
        face_img = rgb_crop(img_rgb, face_region)
        if is_changed: face_img = self.expand_img(face_img, crop_region)
        return face_img

    def prepare_source(self, source_image, crop_factor, is_video = False, tracking = False):
        print("Prepare source...")
        engine = self.get_pipeline()
        source_image_np = (source_image * 255).byte().cpu().numpy()
        img_rgb = source_image_np[0]

        psi_list = []
        for img_rgb in source_image_np:
            if tracking or len(psi_list) == 0:
                crop_region = self.detect_face(img_rgb, crop_factor)
                face_region, is_changed = self.calc_face_region(crop_region, get_rgb_size(img_rgb))

                s_x = (face_region[2] - face_region[0]) / 512.
                s_y = (face_region[3] - face_region[1]) / 512.
                crop_trans_m = create_transform_matrix(crop_region[0], crop_region[1], s_x, s_y)
                mask_ori = cv2.warpAffine(self.GetMaskImg(), crop_trans_m, get_rgb_size(img_rgb), cv2.INTER_LINEAR)
                mask_ori = mask_ori.astype(np.float32) / 255.

                if is_changed:
                    s = (crop_region[2] - crop_region[0]) / 512.
                    crop_trans_m = create_transform_matrix(crop_region[0], crop_region[1], s, s)

            face_img = rgb_crop(img_rgb, face_region)
            if is_changed: face_img = self.expand_img(face_img, crop_region)
            i_s = self.prepare_src_image(face_img)
            x_s_info = engine.get_kp_info(i_s)
            f_s_user = engine.extract_feature_3d(i_s)
            x_s_user = engine.transform_keypoint(x_s_info)
            psi = PreparedSrcImg(img_rgb, crop_trans_m, x_s_info, f_s_user, x_s_user, mask_ori)
            if is_video == False:
                return psi
            psi_list.append(psi)

        return psi_list

    def prepare_driving_video(self, face_images):
        print("Prepare driving video...")
        pipeline = self.get_pipeline()
        f_img_np = (face_images * 255).byte().numpy()

        out_list = []
        for f_img in f_img_np:
            i_d = self.prepare_src_image(f_img)
            d_info = pipeline.get_kp_info(i_d)
            out_list.append(d_info)

        return out_list

    def calc_fe(_, x_d_new, eyes, eyebrow, wink, pupil_x, pupil_y, mouth, eee, woo, smile,
                rotate_pitch, rotate_yaw, rotate_roll):

        x_d_new[0, 20, 1] += smile * -0.01
        x_d_new[0, 14, 1] += smile * -0.02
        x_d_new[0, 17, 1] += smile * 0.0065
        x_d_new[0, 17, 2] += smile * 0.003
        x_d_new[0, 13, 1] += smile * -0.00275
        x_d_new[0, 16, 1] += smile * -0.00275
        x_d_new[0, 3, 1] += smile * -0.0035
        x_d_new[0, 7, 1] += smile * -0.0035

        x_d_new[0, 19, 1] += mouth * 0.001
        x_d_new[0, 19, 2] += mouth * 0.0001
        x_d_new[0, 17, 1] += mouth * -0.0001
        rotate_pitch -= mouth * 0.05

        x_d_new[0, 20, 2] += eee * -0.001
        x_d_new[0, 20, 1] += eee * -0.001
        #x_d_new[0, 19, 1] += eee * 0.0006
        x_d_new[0, 14, 1] += eee * -0.001

        x_d_new[0, 14, 1] += woo * 0.001
        x_d_new[0, 3, 1] += woo * -0.0005
        x_d_new[0, 7, 1] += woo * -0.0005
        x_d_new[0, 17, 2] += woo * -0.0005

        x_d_new[0, 11, 1] += wink * 0.001
        x_d_new[0, 13, 1] += wink * -0.0003
        x_d_new[0, 17, 0] += wink * 0.0003
        x_d_new[0, 17, 1] += wink * 0.0003
        x_d_new[0, 3, 1] += wink * -0.0003
        rotate_roll -= wink * 0.1
        rotate_yaw -= wink * 0.1

        if 0 < pupil_x:
            x_d_new[0, 11, 0] += pupil_x * 0.0007
            x_d_new[0, 15, 0] += pupil_x * 0.001
        else:
            x_d_new[0, 11, 0] += pupil_x * 0.001
            x_d_new[0, 15, 0] += pupil_x * 0.0007

        x_d_new[0, 11, 1] += pupil_y * -0.001
        x_d_new[0, 15, 1] += pupil_y * -0.001
        eyes -= pupil_y / 2.

        x_d_new[0, 11, 1] += eyes * -0.001
        x_d_new[0, 13, 1] += eyes * 0.0003
        x_d_new[0, 15, 1] += eyes * -0.001
        x_d_new[0, 16, 1] += eyes * 0.0003
        x_d_new[0, 1, 1] += eyes * -0.00025
        x_d_new[0, 2, 1] += eyes * 0.00025


        if 0 < eyebrow:
            x_d_new[0, 1, 1] += eyebrow * 0.001
            x_d_new[0, 2, 1] += eyebrow * -0.001
        else:
            x_d_new[0, 1, 0] += eyebrow * -0.001
            x_d_new[0, 2, 0] += eyebrow * 0.001
            x_d_new[0, 1, 1] += eyebrow * 0.0003
            x_d_new[0, 2, 1] += eyebrow * -0.0003


        return torch.Tensor([rotate_pitch, rotate_yaw, rotate_roll])
g_engine = LP_Engine()

class ExpressionSet:
    def __init__(self, erst = None, es = None):
        if es != None:
            self.e = copy.deepcopy(es.e)  # [:, :, :]
            self.r = copy.deepcopy(es.r)  # [:]
            self.s = copy.deepcopy(es.s)
            self.t = copy.deepcopy(es.t)
        elif erst != None:
            self.e = erst[0]
            self.r = erst[1]
            self.s = erst[2]
            self.t = erst[3]
        else:
            self.e = torch.from_numpy(np.zeros((1, 21, 3))).float().to(get_device())
            self.r = torch.Tensor([0, 0, 0])
            self.s = 0
            self.t = 0
    def div(self, value):
        self.e /= value
        self.r /= value
        self.s /= value
        self.t /= value
    def add(self, other):
        self.e += other.e
        self.r += other.r
        self.s += other.s
        self.t += other.t
    def sub(self, other):
        self.e -= other.e
        self.r -= other.r
        self.s -= other.s
        self.t -= other.t
    def mul(self, value):
        self.e *= value
        self.r *= value
        self.s *= value
        self.t *= value

    #def apply_ratio(self, ratio):        self.exp *= ratio

def logging_time(original_fn):
    def wrapper_fn(*args, **kwargs):
        start_time = time.time()
        result = original_fn(*args, **kwargs)
        end_time = time.time()
        print("WorkingTime[{}]: {} sec".format(original_fn.__name__, end_time - start_time))
        return result

    return wrapper_fn


class Command:
    def __init__(self, es, change, keep):
        self.es:ExpressionSet = es
        self.change = change
        self.keep = keep

crop_factor_default = 1.7
crop_factor_min = 1.5
crop_factor_max = 2.5

class AdvancedLivePortrait:
    def __init__(self):
        self.src_images = None
        self.driving_images = None
        self.crop_factor = None

    def parsing_command(self, command, motoin_link):
        command.replace(' ', '')
        # if command == '': return
        lines = command.split('\n')

        cmd_list = []

        total_length = 0

        i = 0
        #old_es = None
        for line in lines:
            i += 1
            if line == '': continue
            try:
                cmds = line.split('=')
                idx = int(cmds[0])
                if idx == 0: es = ExpressionSet()
                else: es = ExpressionSet(es = motoin_link[idx])
                cmds = cmds[1].split(':')
                change = int(cmds[0])
                keep = int(cmds[1])
            except:
                assert False, f"(AdvancedLivePortrait) Command Err Line {i}: {line}"


                return None, None

            total_length += change + keep
            es.div(change)
            cmd_list.append(Command(es, change, keep))

        return cmd_list, total_length


    def run(self, retargeting_eyes, retargeting_mouth, turn_on, tracking_src_vid, animate_without_vid, command, crop_factor,
            src_images=None, driving_images=None, motion_link=None):
        if turn_on == False: return (None,None)
        src_length = 1

        if src_images == None:
            if motion_link != None:
                self.psi_list = [motion_link[0]]
            else: return (None,None)

        if src_images != None:
            src_length = len(src_images)
            if id(src_images) != id(self.src_images) or self.crop_factor != crop_factor:
                self.crop_factor = crop_factor
                self.src_images = src_images
                if 1 < src_length:
                    self.psi_list = g_engine.prepare_source(src_images, crop_factor, True, tracking_src_vid)
                else:
                    self.psi_list = [g_engine.prepare_source(src_images, crop_factor)]


        cmd_list, cmd_length = self.parsing_command(command, motion_link)
        if cmd_list == None: return (None,None)
        cmd_idx = 0

        driving_length = 0
        if driving_images is not None:
            if id(driving_images) != id(self.driving_images):
                self.driving_images = driving_images
                self.driving_values = g_engine.prepare_driving_video(driving_images)
            driving_length = len(self.driving_values)

        total_length = max(driving_length, src_length)

        if animate_without_vid:
            total_length = max(total_length, cmd_length)

        c_i_es = ExpressionSet()
        c_o_es = ExpressionSet()
        d_0_es = None
        out_list = []

        psi = None
        pipeline = g_engine.get_pipeline()
        for i in range(total_length):

            if i < src_length:
                psi = self.psi_list[i]
                s_info = psi.x_s_info
                s_es = ExpressionSet(erst=(s_info['kp'] + s_info['exp'], torch.Tensor([0, 0, 0]), s_info['scale'], s_info['t']))

            new_es = ExpressionSet(es = s_es)

            if i < cmd_length:
                cmd = cmd_list[cmd_idx]
                if 0 < cmd.change:
                    cmd.change -= 1
                    c_i_es.add(cmd.es)
                    c_i_es.sub(c_o_es)
                elif 0 < cmd.keep:
                    cmd.keep -= 1

                new_es.add(c_i_es)

                if cmd.change == 0 and cmd.keep == 0:
                    cmd_idx += 1
                    if cmd_idx < len(cmd_list):
                        c_o_es = ExpressionSet(es = c_i_es)
                        cmd = cmd_list[cmd_idx]
                        c_o_es.div(cmd.change)
            elif 0 < cmd_length:
                new_es.add(c_i_es)

            if i < driving_length:
                d_i_info = self.driving_values[i]
                d_i_r = torch.Tensor([d_i_info['pitch'], d_i_info['yaw'], d_i_info['roll']])#.float().to(device="cuda:0")

                if d_0_es is None:
                    d_0_es = ExpressionSet(erst = (d_i_info['exp'], d_i_r, d_i_info['scale'], d_i_info['t']))

                    retargeting(s_es.e, d_0_es.e, retargeting_eyes, (11, 13, 15, 16))
                    retargeting(s_es.e, d_0_es.e, retargeting_mouth, (14, 17, 19, 20))

                new_es.e += d_i_info['exp'] - d_0_es.e
                new_es.r += d_i_r - d_0_es.r
                new_es.t += d_i_info['t'] - d_0_es.t

            r_new = get_rotation_matrix(
                s_info['pitch'] + new_es.r[0], s_info['yaw'] + new_es.r[1], s_info['roll'] + new_es.r[2])
            d_new = new_es.s * (new_es.e @ r_new) + new_es.t
            d_new = pipeline.stitching(psi.x_s_user, d_new)
            crop_out = pipeline.warp_decode(psi.f_s_user, psi.x_s_user, d_new)
            crop_out = pipeline.parse_output(crop_out['out'])[0]

            crop_with_fullsize = cv2.warpAffine(crop_out, psi.crop_trans_m, get_rgb_size(psi.src_rgb),
                                                cv2.INTER_LINEAR)
            out = np.clip(psi.mask_ori * crop_with_fullsize + (1 - psi.mask_ori) * psi.src_rgb, 0, 255).astype(
                np.uint8)
            out_list.append(out)

        if len(out_list) == 0: return (None,)

        out_imgs = torch.cat([pil2tensor(img_rgb) for img_rgb in out_list])
        return (out_imgs,)


class ExpressionEditor_modify:
    def __init__(self):
        self.sample_image = None
        self.src_image = None
        self.crop_factor = None

    def run(self, prepared_face, rotate_pitch, rotate_yaw, rotate_roll, blink, eyebrow, wink, pupil_x, pupil_y, aaa, eee, woo, smile,
            src_ratio, sample_ratio, sample_parts, crop_factor, src_image=None, sample_image=None, motion_link=None, add_exp=None):
        rotate_yaw = -rotate_yaw
        
        self.psi = prepared_face
        self.crop_factor = crop_factor
        self.src_image = src_image
        new_editor_link = []
        new_editor_link.append(self.psi)

        pipeline = g_engine.get_pipeline()

        psi = self.psi
        s_info = psi.x_s_info
        #delta_new = copy.deepcopy()
        s_exp = s_info['exp'] * src_ratio
        s_exp[0, 5] = s_info['exp'][0, 5]
        s_exp += s_info['kp']

        es = ExpressionSet()

        if sample_image != None:
            if id(self.sample_image) != id(sample_image):
                self.sample_image = sample_image
                d_image_np = (sample_image * 255).byte().numpy()
                d_face = g_engine.crop_face(d_image_np[0], 1.7)
                i_d = g_engine.prepare_src_image(d_face)
                self.d_info = pipeline.get_kp_info(i_d)
                self.d_info['exp'][0, 5, 0] = 0
                self.d_info['exp'][0, 5, 1] = 0

            # "OnlyExpression", "OnlyRotation", "OnlyMouth", "OnlyEyes", "All"
            if sample_parts == "OnlyExpression" or sample_parts == "All":
                es.e += self.d_info['exp'] * sample_ratio
            if sample_parts == "OnlyRotation" or sample_parts == "All":
                rotate_pitch += self.d_info['pitch'] * sample_ratio
                rotate_yaw += self.d_info['yaw'] * sample_ratio
                rotate_roll += self.d_info['roll'] * sample_ratio
            elif sample_parts == "OnlyMouth":
                retargeting(es.e, self.d_info['exp'], sample_ratio, (14, 17, 19, 20))
            elif sample_parts == "OnlyEyes":
                retargeting(es.e, self.d_info['exp'], sample_ratio, (1, 2, 11, 13, 15, 16))

        es.r = g_engine.calc_fe(es.e, blink, eyebrow, wink, pupil_x, pupil_y, aaa, eee, woo, smile,
                                  rotate_pitch, rotate_yaw, rotate_roll)

        if add_exp != None:
            es.add(add_exp)

        new_rotate = get_rotation_matrix(s_info['pitch'] + es.r[0], s_info['yaw'] + es.r[1],
                                         s_info['roll'] + es.r[2])
        x_d_new = (s_info['scale'] * (1 + es.s)) * ((s_exp + es.e) @ new_rotate) + s_info['t']

        x_d_new = pipeline.stitching(psi.x_s_user, x_d_new)

        crop_out = pipeline.warp_decode(psi.f_s_user, psi.x_s_user, x_d_new)
        crop_out = pipeline.parse_output(crop_out['out'])[0]

        crop_with_fullsize = cv2.warpAffine(crop_out, psi.crop_trans_m, get_rgb_size(psi.src_rgb), cv2.INTER_LINEAR)
        out = np.clip(psi.mask_ori * crop_with_fullsize + (1 - psi.mask_ori) * psi.src_rgb, 0, 255).astype(np.uint8)

        out_img = pil2tensor(out)

        filename = g_engine.get_temp_img_name() #"fe_edit_preview.png"
        results = list()
        results.append({"filename": filename, "type": "temp"})

        new_editor_link.append(es)

        return {"result": (out_img, new_editor_link, es)}

def AdvancedLivePortrait_execution_prepare(input_image_path, output_image_path):
    # Specify the path for the input image

    if not os.path.exists(os.path.dirname(output_image_path)):
        os.makedirs(os.path.dirname(output_image_path))

    # Load image containing a face
    image = Image.open(input_image_path)
    image = image.convert("RGB")
    print(image.size)
    img_tensor = pil2tensor(image).to(get_device())  # Convert image to tensor and transfer to GPU

    print(img_tensor.shape)

    print("load image")
    # Create an instance of the LP_Engine class
    engine = LP_Engine()

    print("load LP_Engine")
    # Detect and prepare face
    crop_factor = 1.7  # Crop size for the face
    prepared_face = engine.prepare_source(img_tensor, crop_factor)  # Prepare face

    # Create an instance of the ExpressionEditor class to edit expressions
    editor = ExpressionEditor_modify()

    print("load ExpressionEditor")
    
    return editor, img_tensor, prepared_face
    

    

def AdvancedLivePortrait_execution_main(editor, img_tensor, prepared_face, output_image_path, parameters):

    # Settings for changing expressions
    rotate_pitch = parameters[0]  # Vertical rotation of the face (forward/backward)
    rotate_yaw = parameters[1]    # Horizontal rotation of the face (left/right)
    rotate_roll = parameters[2]   # Face tilt
    blink = parameters[3]         # Blink degree
    eyebrow = parameters[4]       # Eyebrow movement
    wink = parameters[5]          # Single eye wink
    pupil_x = parameters[6]       # Horizontal pupil movement
    pupil_y = parameters[7]       # Vertical pupil movement
    aaa = parameters[8]           # Opening the mouth
    eee = parameters[9]           # Pronouncing 'ee'
    woo = parameters[10]          # Pronouncing 'oo'
    smile = parameters[11]        # Smile degree (0.5 = 50% smile)

    print("start running")
    mainprocess_time = time.time()
    # Edit expression and get resulting image
    result = editor.run(prepared_face,
        rotate_pitch, rotate_yaw, rotate_roll,
        blink, eyebrow, wink, pupil_x, pupil_y,
        aaa, eee, woo, smile,
        src_ratio=1, sample_ratio=1,
        sample_parts="All",  # Edit overall expression
        crop_factor=1.7,
        src_image=img_tensor  # Specify the face image prepared earlier
    )
    print("メイン処理実行時間:", time.time() - mainprocess_time)

    # Unpack results
    edited_img, motion_link, expression_data = result["result"]

    print("finish running")

    # Save the resulting image
    edited_image_pil = tensor2pil(edited_img)  # Convert tensor to PIL image
    #edited_image_pil.save(output_image_path)
    #print(f"Edited image saved at: {output_image_path}")

    return edited_image_pil



Basically, it is the same as the previous article. However, to speed up processing, I split the face detection part and the face editing part into two functions: AdvancedLivePortrait_execution_prepare and AdvancedLivePortrait_execution_main.

Discussion