I'm interested in computer vision, machine learning, image processing, and computational photography. Much of my research is about human faces and pose estimation.
Video-based Multi-person Pose Estimation and Tracking
Songyao Jiang, and Yun Fu
Current Work, 2019
Video-based Multi-person Pose Estimation and Tracking. Under development and construction.
Inferencing model provided on GitHub.
Face Recognition and Verification in Low-light Condition
Songyao Jiang, Yue Wu, Zhengming Ding, and Yun Fu
This project tends to solve the problem of recognizing people in low light condition,
which is quite useful in security. In low-light condition, we usually utilize near IR,
mid-range IR and long-range IR to obtain the portrait images of the target person.
However, those IR images are very different than the visible images that we used to
train our deep face recognition and verification methods. To utilize the knowledge we learned from visible
images and apply it on IR images, we developed a semi-supervised and an unsupervised
transfer learning methods to transfer the knowledge we learned from visible spectrum to
IR spectrum. Based on which, we developed our low-light face recognition and verification system.
Spatially Constrained Generative Adversarial Networks for Conditional Image Generation
Songyao Jiang, Hongfu Liu, Yue Wu and Yun Fu
Submitted to a Springer Journal (under review), 2018
Image generation has raised tremendous attention
in both academic and industrial areas, especially
for criminal portrait and fashion design.
The current studies always focus on
class labels as the condition where spatial contents are
randomly generated. The edge details
and spatial information is usually blurred and difficult
to preserve. In light of this, we propose a novel
Spatially Constrained Generative Adversarial Network
, which decouples the spatial constraints from
the latent vector and makes them feasible as
additional controllable signals. Experimentally, we provide
both visual and quantitive results, and demonstrate that the proposed SCGAN
is very effective in controlling the spatial
contents as well as generating high-quality images.
Segmentation Guided Image-to-Image Translation with Adversarial Networks
Songyao Jiang, Zhiqiang Tao and Yun Fu
IEEE International Conference on Automatic Face & Gesture Recognition (FG), 2019
Recently image-to-image translation methods neglect to
utilize higher-level and instance-specific
information to guide the training process, leading to a great
deal of unrealistic generated images of low quality. Existing
methods also lack of spatial controllability during translation.
To address these challenge, we propose a novel Segmentation
Guided Generative Adversarial Networks, which
leverages semantic segmentation to further boost the generation
performance and provide spatial mapping. Experimental results on multi-domain
face image translation task empirically demonstrate our ability
of the spatial modification and our superiority in image quality
over several state-of-the-art methods.
Rule-Based Facial Makeup Recommendation System
Taleb Alashkar, Songyao Jiang and Yun Fu
IEEE International Conference on Automatic Face & Gesture Recognition (FG), 2017
Facial makeup style plays a key role in the facial appearance making it
more beautiful and attractive. Choosing the best makeup style for a certain face
to fit a certain occasion is a full art. To solve this problem computationally,
an automatic and smart facial makeup recommendation
and synthesis system is proposed in this paper. Additionally, an automatic facial
makeup synthesis system is developed to apply the recommended style
on the facial image as well. To this end, a new dataset with 961 different females photos
collected and labeled.
Examples-Rules Guided Deep Neural Network for Makeup Recommendation
Taleb Alashkar, Songyao Jiang, Shuyang Wang and Yun Fu
AAAI Conference on Artificial Intelligence (AAAI), 2017
We consider a fully automatic makeup recommendation system and propose a novel
examples-rules guided deep neural network approach. The framework consists of
three stages. First, makeup-related facial traits are classified into structured
coding. Second, these facial traits are fed in- to examples-rules guided deep
neural recommendation model which makes use of the pairwise of Before-After images
and the makeup artist knowledge jointly. Finally, to visualize the recommended makeup
style, an automatic makeup synthesis system is developed as well.
This website is generated using source code from Jon Barron.