Multi-Modal Networks
This bonus lesson studies CLIP, VQGAN+CLIP, and DALL·E 1 and 2 as historical milestones in multimodal learning. It explains the ideas rather than serving as a guide to current products or model catalogs.
Contrastive Image Pre-Training (CLIP)
The main idea of CLIP is to be able to compare text prompts with an image and determine how well the image corresponds to the prompt.
The model is trained on images obtained from the Internet and their captions. For each batch, we take N pairs of (image, text), and convert them to some vector representations I1,..., IN / T1, ..., TN. Those representations are then matched together. The loss function is defined to maximize the cosine similarity between vectors corresponding to one pair (eg. Ii and Ti), and minimize cosine similarity between all other pairs. That is the reason this approach is called contrastive.
CLIP model/library is available from OpenAI GitHub. The approach is described in this blog post, and in more detail in this paper.
After pre-training, CLIP returns similarity scores for batches of images and text prompts. Those scores become probabilities only after normalization within a specified classification setup. CLIP can be used for several tasks:
Image Classification
Suppose we need to classify images between, say, cats, dogs and humans. In this case, we can give the model an image, and a series of text prompts: "a picture of a cat", "a picture of a dog", "a picture of a human". In the resulting vector of 3 probabilities we just need to select the index with a highest value.
Figure description: CLIP for Image Classification
Picture from this blog post
Text-Based Image Search
We can also do the opposite. If we have a collection of images, we can pass this collection to the model, and a text prompt - this will give us the image that is most similar to a given prompt.
Open the Clip.ipynb notebook to see CLIP in action.
Image Generation with VQGAN+ CLIP
VQGAN represents an image with discrete latent tokens learned by an encoder and decoder with perceptual and adversarial losses. In the Taming Transformers approach, a separate autoregressive transformer models sequences of those tokens; the transformer is not the definition of VQGAN itself.
VQGAN+CLIP later emerged as an experimental workflow: start with an optimizable image or latent representation and use CLIP similarity as a signal that steers the result toward a text prompt. It is a useful historical example of multimodal guidance, not a general description of how current image-generation systems work.
Experiment only with self-authored or clearly licensed assets, and evaluate prompt sensitivity and bias rather than relying on one appealing sample.
DALL-E
DALL·E 1 demonstrated text-to-image generation with an autoregressive transformer over a shared sequence of text and image tokens.
DALL·E 2 used CLIP representations in a hierarchical image-generation system. Both models are presented here as technical history; consult the documentation for any current system to understand its interface, capabilities, and limitations.
References
