Data-Driven Manga Designer
There are some design tools available to facilitate manga creation. However, as these tools mainly provide some basic facilities to assist users to create their designs, they expect the users to be manga artists, who are already familiar with the manga design rules. In this project, we go one step further. We try to create design tools to help novice users, i.e., users who are not familiar with manga, to create their own manga. Our idea is to apply machine learning techniques to capture the hidden/implicit design rules from existing manga. Given the necessary input content from the user, our design tools would apply the learned rules to produce new designs. At this point, we have developed the following tools to allow novice users to create their own manga: - Manga Layout Tool: Given a sequence of input artworks, and their semantics, including importance values and optional panel relationship, this tool layouts the sequence of input artworks into manga pages automatically. - Manga Composition Tool: Given a storyboard comprising input subjects, their dialogues, shot type and motion state of each panel, and the interaction type between any two subjects, this tool automatically composites them into panels. - Manga Animation Tool: Given a manga page, this tool automatically extracts the elements (panels, comic characters and speech balloons) and infers motion and emotion states of each panel. It then generates 2D camera movements of varying speeds across the page to produce an animation video that can effectively tell the story. We have demonstrated that this tool can be used to generate comic trailers and to present manga on mobile devices.
	Xin Yang, Zongliang Ma, Letian Yu, Ying Cao, Baochai Yin, Xiaopeng Wei, and Rynson Lau, "Automatic Comic Generation with Stylistic Multi-page Layouts and Emotion-driven Text Balloon Generation," ACM Trans. on Multimedia Computing, Communications, and Applications (accepted). [paper]
Abstract: In this paper, we propose a fully automatic system for generating comic books from videos without any human intervention. Given an input video along with its subtitles, our approach first extracts informative keyframes by analyzing the subtitles, and stylizes keyframes into comic-style images. Then, we propose a novel automatic multi-page layout framework, which can allocate the images across multiple pages and synthesize visually interesting layouts based on the rich semantics of the images (e.g., importance and interimage relation). Finally, as opposed to using the same type of balloon as in previous works, we propose an emotion-aware balloon generation method to create different types of word balloons by analyzing the emotion of subtitles and audios. Our method is able to vary balloon shapes and word sizes in balloons in response to different emotions, leading to more enriched reading experience. Once the balloons are generated, they are placed adjacent to their corresponding speakers via speaker detection. Our results show that our method, without requiring any user inputs, can generate high-quality comic pages with visually rich layouts and balloons. Our user studies also demonstrate that users prefer our generated results over those by state-of-the-art comic generation systems.
	Ying Cao, Xufang Pang, Antoni Chan, and Rynson Lau, "DynamicManga: Animating Still Manga via Camera Movement," IEEE Trans. on Multimedia (accepted). [paper] [video] [suppl] [trailer by our method]
Abstract: We propose a method for animating still manga imagery through camera movements. Given a series of existing manga pages, we start by automatically extracting panels, comic characters and balloons from the manga pages. Then, we use a data-driven graphical model to infer per-panel motion and emotion states from low-level visual patterns. Finally, by combining domain knowledge of film production and characteristics of manga, we simulate camera movements over the manga pages, yielding an animation. The results augment the still manga contents with animated motion that reveals the mood and tension of the story, while maintaining the original narrative. We have tested our method on manga series of different genres, and demonstrated that our method can generate animations that are more effective in storytelling and pacing, with less human efforts, as compared with prior works. We also show two applications of our method, mobile comic reading and comic trailer generation.
	Xufang Pang, Ying Cao, Rynson Lau, and Antoni Chan, "A Robust Panel Extraction Method for Manga," Proc. ACM Multimedia, pp. 1125-1128, Nov. 2014. [paper] [code]
Abstract: Automatically extracting frames/panels from digital comic pages is crucial for techniques that facilitate comic reading on mobile devices with limited display areas. However, automatic panel extraction for manga, i.e., Japanese comics, can be especially challenging, largely because of its complex panel layout design mixed with various visual symbols throughout the page. In this paper, we propose a robust method for automatically extracting panels from digital manga pages. Our method first extracts the panel block by closing open panels and identifying a page background mask. It then performs a recursive binary splitting to partition the panel block into a set of sub-blocks, where an optimal splitting line at each recursive level is determined adaptively. Finally, it recovers accurate panel shapes from the computed sub-blocks. Our experiments show that the proposed method can robustly segment panels on the manga pages with various styles.
	Ying Cao, Rynson Lau, and Antoni Chan, "Look Over Here: Attention-Directing Composition of Manga Elements," ACM Trans. on Graphics (SIGGRAPH 2014), 33(4), Article 94, Aug. 2014. [paper] [suppl] [video]
Abstract: Picture subjects and text balloons are basic elements in comics, working together to propel the story forward. Japanese comics artists often leverage a carefully designed composition of subjects and balloons (generally referred to as panel elements) to provide a continuous and fluid reading experience. However, such a composition is hard to produce for people without the required experience and knowledge. In this paper, we propose an approach for novices to synthesize a composition of panel elements that can effectively guide the reader's attention to convey the story. Our primary contribution is a probabilistic graphical model that describes the relationships among the artist's guiding path, the panel elements, and the viewer attention, which can be effectively learned from a small set of existing manga pages. We show that the proposed approach can measurably improve the readability, visual appeal, and communication of the story of the resulting pages, as compared to an existing method. We also demonstrate that the proposed approach enables novice users to create higher-quality compositions with less time, compared with commercially available programs.
	Ying Cao, Antoni Chan, and Rynson Lau, "Automatic Stylistic Manga Layout," ACM Trans. on Graphics (SIGGRAPH Asia 2012), 31(6), Article 141, Nov. 2012. [paper] [video] [more results]
Abstract: Manga layout is a core component in manga production, characterized by its unique styles. However, stylistic manga layouts are difficult for novices to produce as it requires hands-on experience and domain knowledge. In this paper, we propose an approach to automatically generate a stylistic manga layout from a set of input artworks with user-specified semantics, thus allowing less-experienced users to create high-quality manga layouts with minimal efforts. We first introduce three parametric style models that encode the unique stylistic aspects of manga layouts, including layout structure, panel importance, and panel shape. Next, we propose a two-stage approach to generate a manga layout: 1) an initial layout is created that best fits the input artworks and layout structure model, according to a generative probabilistic framework; 2) the layout and artwork geometries are jointly refined using an efficient optimization procedure, resulting in a professional-looking manga layout. Through a user study, we demonstrate that our approach enables novice users to easily and quickly produce higher-quality layouts that exhibit realistic manga styles, when compared to a commercially-available manual layout tool.

Last updated in January 2021