We have been telling them stories for years to help them learn about the world and to strengthen their character, without putting their lives at risk. Books are a great way to tell stories. The reader can identify with characters and feel their emotions through books. The goal is for the reader to feel connected to the book.
We also want the stories and illustrations to be true to you. Imagine a book with a main character who looks exactly like you. We aim to achieve this. Illustration of a drawing. All images in this article were taken from stock images repositories. They are used only for demonstration purposes.
Personalized books exist already.
What’s the difference between personalized books and regular ones? You’ve probably been familiar with literature for children for some time. Personalized books are a common thing. So what’s the difference? These books let you choose from a variety of options for avatar features, such as blonde hair or brown hair. Are you looking for green or blue eyes? In our case, however, the face of the protagonist is actually the same as the reader’s. This was achieved by making a faceswap.
You can also customize the text.
The book will not only include your name, but the character of the protagonist who tackles the problems. Will he be successful because of his extraordinary strength or clever intellect? The book’s ordering process incorporates a few personality characteristics that can affect how the story is told. This article focuses on how to create a program that can position the face and adjust it on a drawing.
Before purchasing the book, clients must be able see the illustration previews. Our implementation should produce results fast and efficiently. What’s a face swap? Faceswap refers to the act of changing faces in an image, usually a photo or video. Images are row and column of pixels and matrices representing color and shape.
Computer Vision, a subfield within AI and Machine Learning, focuses on how to make machines see images in the same way that we do. In our case, this means understanding what pixel values correspond to a face. After the first stage of facial detection is completed, the program must swap pixels between two targets. Then, it will adapt the colors and shapes of both the faces to produce a smoother final result.
Although Deepfakes, and other facial-swapping apps for real images, have been in the limelight due to their amazing results, little has been done to consider different contexts such as our task. Will technologies that work on photos also work on drawings? Let’s see what happens. Differential faceswaps between picture and photo. We started by trying out various face-swapping methods to find out what issues might arise when using drawings rather than pictures.
The face detection method used by dlib is widely accepted. It identifies the facial keypoints to allow for faces to be swapped between photos later. Machine Learning methods (Haar Cascades and Linear SVM detectors) are used to identify faces. However, we noticed that some drawings were more difficult for the computer to recognise. If the details of the face were too small or complex, it could need some assistance.
Drawings where dlib failed to locate faces.
Examples of drawings in which dlib succeeded despite having closed eyes and a minimalist design. Although it is time-consuming, this will ensure that the face detection process works properly. If done correctly, the user won’t notice the result.
Quick manual editing can be done to create a real-looking face that can later be swapped. Reflect’s FaceSwapping was the first application that we tested to see if it had any issues. We found that it varied in effectiveness depending on how detailed the drawing was. Drawings with too many details resulted often in poor performance. Reflect achieved a great result with a very detailed drawing, but it failed to perform well on a simple one.
DeepFaceLab was also tested to get a baseline.
DeepFaceLab has become a well-known name in DeepFake’s industry. Its main purpose is swapping faces in video, which is not our objective. It is designed to learn shadows and angles using many frames. We found that it does not seem to be able to handle drawings well, or requires excessive hyperparameter tuning in order to achieve meaningful results. The program took too long to accomplish our goals. Customers should have the ability to see a preview of their edited images in just seconds to minutes.
To see if a Neural Stil Transfer might be worth the effort, we also attempted to shape the face in the same style as the illustrations with Deep Dream Generator. However, the results weren’t encouraging. Perhaps there wasn’t enough detail to properly implement the style.
DeepDream Generator results
These results were not only difficult to crop and apply the faceswap late on, but also required additional computational and time resources. The solution We compared the results and realized that photo-realism was not an issue. We don’t require a realistic result, but we do need results that are quick and easy to deliver to potential customers.
We have only one image and several drawings that we can feed our algorithm. We also need reliable results.
It is impossible to fine tune parameters or eliminate bad results. In green: Facial landmarks found by dlib’s facial detector. It should be easy to use. Green: Facial landmarks discovered by dlib’s facial detector. Next, it calculates the affine transform matrix for each triangle vertex. This is done from the “mask”, which is the image of the face. It then uses Poisson Blending, to overlay the mask and also applies bilinear interpolation in order to smoothen the shapes.
