Blog¶

Welcome to my technical blog where I share insights about:

AI System Implementation
Technical Architecture Design
Performance Optimization
Team Enablement

2024/12/12
in Novel Minds
4 min read

Solve Your Image Generation Accuracy with VLM Rejection Sampling

The Genesis of an Idea

Book pages transforming into illustrations Traditional books transforming into illustrated experiences

It was a hot morning towards the end of the summer when we met at Mozart's in Austin, Texas to catch up and share ideas. Over coffee, we discussed our shared love of reading and pondered: wouldn't it be cool to read a book and see the image of what we were reading? While e-readers have transformed where we read, the fundamental reading experience hasn't evolved much. With LLMs becoming multi-modal and image generation improving daily, we saw an opportunity.

We wondered: Do people read anymore?

The statistics were concerning. Literacy rates in the United States had declined over the past five years, complicated by the pandemic and increasing competition from visual media and games. We asked ourselves:

What if our beloved classics could come to life with illustrations?
Could we do this easily for any book?
What might we learn in the process?

From Concept to MVP

Initial Prototyping

After determining that a full-fledged custom EPUB reader with advanced connectivity for on-demand image generation might be too ambitious for our MVP, we narrowed our focus. We decided to start with public domain classics, ultimately selecting:

Around the World in 80 Days by Jules Verne
Through the Looking Glass by Lewis Carroll
Pride and Prejudice by Jane Austen

The choice of Around the World in 80 Days was particularly compelling due to its era-specific atmosphere, world adventures, and our fond memories of reading it in our childhood.

Early Challenges

Our first breakthrough came quickly - successfully injecting a single image into an EPUB. The next day, Jason expanded this dramatically, producing 21 versions of Around the World in 80 Days using various models:

Stability (1.0, 3.0, Core, Core with styles) * DALLE-2

Refining the Process

Identifying Problems

While reviewing the variations, several issues became apparent:

Inconsistent era representation
Incorrect character renderings
Fluctuating visual styles
Reader immersion breaking

Solutions Implemented

To improve the reading experience, we focused on three critical areas:

1. Historical Accuracy

Eliminated anachronistic elements like modern vehicles in 1800s scenes
Maintained period-appropriate details in clothing, architecture, and technology
Ensured all visual elements matched the book's time period

2. Character Consistency

Leveraged ChatGPT 4o to create detailed character profiles
Developed systematic approaches for character appearance continuity
Maintained visual consistency across all illustrations

3. Visual Style

Implemented clean line-art style for:
Consistent rendering across all devices
Better display on e-ink screens
Reduced generation costs with Stability 1XL

Technical Implementation

Quality Control System

VLM rejection sampling process visualization Our VLM-based quality control was key

During our iteration phase with Through the Looking Glass, The Great Gatsby, and The Metamorphosis, we encountered various rendering challenges: