Enhancing Identity Preservation in Portrait Generation via Reward Optimization

Recent advancements in tailored image synthesis have highlighted the remarkable capabilities of pre-trained text-to-image frameworks in encapsulating individual identity traits from a collection of portrait photographs. However, these solutions may not accurately reflect the key characteristics of the input, leading to a loss of essential identity traits. To alleviate this issue, our study introduces a new framework for personalized portrait generation. This framework leverages reward optimization to refine the generation process, integrating a face recognition model into the reward function. It assesses the similarity between user-provided images and synthetic portraits to determine rewards. We utilize a pathwise estimator for gradient estimation, employing the Gumbel-Softmax technique to fulfill the differentiability requirement and incorporating a KL divergence regularizer to mitigate the risk of overfitting on reward. Our showcases indicate a marked improvement in preserving human identity in the generated portraits.

Download paper here

Under Review