Introduction

With the Release of Dall-E 2, Google's Imagen, Stable Diffusion, and Midjourney, diffusion models have taken the world by storm, inspiring creativity and pushing the boundaries of machine learning.

With written suggestions, these models may produce almost any kind of image, from the fanciful to the futuristic to the photo-realistic and of course the cute.

With these superpowers, we can create nearly any image we can think of, redefining what it means for humans to engage with silicon. Diffusion models do have some limitations, despite their sophisticated capabilities, which we will discuss later in the guide. But as these models advance further or are replaced by a new generative paradigm, people will be able to use their minds to create immersive images, films, and other media.

We examine diffusion models in this guide, including their operation, real-world uses, and potential future developments.

What are Diffusion Models?

A subclass of machine learning models known as generative models has the ability to produce new data from training data. Flow-based models, Variational Autoencoders (VAEs), and Generative Adversarial Networks (GANs) are examples of further generative models. All of them are less effective than diffusion models due to their restrictions, even if they can all produce images of a high caliber.

Diffusion models function, in essence, by first injecting noise to destroy training data and then learning to recover the data by undoing this noise. Put differently, diffusion models have the ability to produce coherent images out of noise.

In order for diffusion models to learn how to remove noise, noise is first added to images during training. In order to produce realistic images, the model then applies this denoising technique to random seeds.

Combined with text-to-image guidance, these models can be used to create a near-infinite variety of images from text alone by conditioning the image generation process. Inputs from embeddings like CLIP can guide the seeds to provide powerful text-to-image capabilities.

Diffusion models can complete various tasks, including image generation, image denoising, inpainting, outpainting, and bit diffusion.

• Dall-E 2: When Dall-E 2 was unveiled in April 2022, it produced photos with even more realism and better megapixels than the first Dall-E. Dall-E 2 is now accessible to the general public on the OpenAI website as of September 28, 2022. A small selection of photographs are offered for free, and more can be purchased.
• Imagen is Google's May 2022, version of a text-to-image diffusion model, which is not available to the public.
• Stable Diffusion: In August 2022, Stability AI released Stable Diffusion, an open-source Diffusion model similar to Dall-E 2 and Imagen. Stability AI's released open source code and model weights, opening up the models to the entire AI community. Stable Diffusion was trained on an open dataset, using the 2 billion English label subset of the CLIP-filtered image-text pairs open dataset LAION 5b, a general crawl of the internet created by the German charity LAION.
• Midjourney is another diffusion model released in July 2022 and available via API and a discord bot.

Simply put, Diffusion models are generative tools that enable users to create almost any image they can imagine.

Diffusion Models: Why Are They Important?

The pinnacle of generating capacities at this time is represented by diffusion models. But these models are not without their heroes; they have been made possible by more than ten years of progress in machine learning methods, the ubiquitous availability of enormous amounts of visual data, and better technology.

For some context, below is a brief outline of significant machine learning developments.

• The groundbreaking Imagenet article and dataset, which included over 14 million manually annotated photos, were made available at CVPR in 2009. This enormous dataset from back then is still useful for scholars and companies who are developing models today.
• Ian Goodfellow created GANs in 2014, bringing strong generating capabilities to machine learning models.
• With the publication of the initial GPT in 2018, LLMs made their debut. Shortly after, its GPT-2 and current GPT-3 versions, which include text generation, came out.
• In 2020, NeRFs allowed the world to produce 3D objects from a series of images, and known camera poses.
• Diffusion models have carried on this development over the last few years, providing us with even more potent generative powers.

What about diffusion models makes them so strikingly different from their predecessors? The most apparent answer is their ability to generate highly realistic imagery and match the distribution of real images better than GANs. Also, diffusion models are more stable than GANs, which are subject to mode collapse, where, following training, they merely depict a small number of modes of the actual data distribution. Though the issue is not quite as severe in practice, this mode collapse means that in the worst situation, only one image would be returned for any query. Since the diffusion process evens out the distribution, diffusion models sidestep the issue and have a wider range of pictures than GANs.

A wide range of inputs can also be used to condition diffusion models, including text for text-to-image creation, bounding boxes for layout-to-image generation, masked images for inpainting, and lower-resolution images for super-resolution.

Diffusion models have a wide range of applications, and their actual applications are continually developing. These models will have a significant impact on marketing, social media, AR/VR, retail and eCommerce, and more.

Getting Started with Diffusion Models

Web applications such Open AI's Dall-E 2 and Stable Diffusion's DreamStudio make diffusion models readily available. With the help of these tools, novices may quickly and easily begin working with diffusion models. You can create images by following prompts and carry out inpainting and outpainting. DreamStudio has greater control over the output parameters, but Dall-E 2 has a more straightforward interface with less features. New users on each platform receive free credits; however, after those credits are used up, a usage fee will apply.

• DreamStudio: Users can quickly enjoy stable diffusion with DreamStudio from Stability AI without having to worry about the specifics of the infrastructure. Tools are available for creating images as well as for inpainting and outpainting. Specially, the interface allows the user to set a random seed, meaning that one can navigate the latent space with a fixed prompt (more on this later). 200 free credits are given to new users.
• Dall-E 2: Dall-E 2 is now generally available to all users, according to OpenAI, having exited its restricted beta. Dall-E 2 offers a straightforward, uncomplicated user interface for picture generation, inpainting, and outpainting.
• Local Installation:
  • Stability AI broke headlines when it announced that it was open-sourcing both the model weights and source code for its Diffusion model Stable Diffusion.
  • You can download and install Stable Diffusion on your local computer and integrate its capabilities into applications and workflows
  • Other models, such as Dall-E 2, are currently only available via API or web app as their models are not open-source like Stable Diffusion.

Additionally, aggregation sites such as Lexica.art offer a vast collection of carefully selected photos for search, making it even simpler to get started, gain inspiration from the work of the community, and learn how to create better prompts.

Diffusion Model Prompt Engineering

For Diffusion models, you can manipulate the outputs using prompts. Verbose diffusion models receive two main inputs, which are translated into a text prompt, a seed integer, and a fixed point in the model's latent space. The user enters the text prompt, and the seed integer is often created automatically. To achieve the ideal results, prompt engineering requires constant experimenting. In order to assist you shape the images you wish to generate, we examined Dall-E 2 and Stable Diffusion. Based on our findings, we have compiled our best advice on how to make the most of your prompts, including prompt length, creative style, and essential phrases.

5. Additional Tips

A few additional items are worth mentioning.

Placing the primary subject of the image closer to the beginning of the prompt tends to ensure that subject is included in the image. For instance, compare the two prompts

• "A city street with a black velvet couch" at times will miss the intent of the prompt entirely and the resulting image will not include a couch.
• By rearranging the prompt to have the keyword "couch" closer to the beginning of the prompt, the resulting images will almost always contain a couch.

Certain subject and place combinations usually don't work well. For example, "A black velvet couch on the surface of the moon" produces erratic results, with couches completely absent and other backdrops. Nonetheless, a related prompt like "A black velvet couch in a desert" tends to better capture the essence of the challenge by precisely depicting the black hue, the velvet substance, and the scene's features. The model is presumably better at producing coherent landscapes for deserts than for the moon because there are presumably more desert photographs in the training set.

The field of prompt engineering is constantly growing, with new techniques being discovered on a daily basis. A new kind of job called "Prompt Engineer" is probably going to arise as more companies realize how effective diffusion models can be in solving their problems.

Diffusion Model Limitations

Despite their great potency, diffusion models have many drawbacks, some of which we will discuss below. Disclaimer: these limitations are effective as of October 2022, due to the quick speed of growth.

• Face Distortion: When the sample size is greater than three, faces get significantly warped. The faces become significantly deformed in pictures like "a family of six in a conversation at a cafe looking at each other and holding coffee cups, a park in the background across the street leica sl2 50mm, vivid color, high quality, high textured, real life."
• But as the number of participants in the prompt increases, the faces significantly deform. The new prompt, "a family of six in a conversation at a cafe looking at each other and holding coffee cups, with a park in the background across the street and a Leica SL2 50mm camera, produces strikingly realistic color images that are high quality and highly textured."
• Limited Prompt Understanding: These models can be less effective as productivity tools, but they still increase productivity overall. For some photographs, you will need to manipulate the prompt a lot to achieve the desired result.

Diffusion Models: Additional Capabilities and Tooling

Diffusion models' flexibility gives them more capabilities than just pure image generation.

• With the use of an image editing tool called inpainting, users can alter specific areas of an image and replace them with content that is created by the diffusion model. To make sure the generations fit inside the original image's context, the model makes reference to nearby pixels. A lot of programs allow you to alter photos (generated or real) by "erasing" or masking a certain area of the image and then instructing the model to fill it in with new information. You begin with a created or real-world image when using inpainting. Here, the picture shows a model in a contemporary metropolis leaning on a wall.
  • You then apply a "mask" to the areas of the image you would like to replace, similar to erasing areas of the image. First, we will replace the jacket that the model is wearing.
  • Then, we will generate new clothing for the model to wear, in this case a beautiful fancy gown shiny with pink highlights.
  • Or a bright purple and pink wool jacket with orange highlights and highly detailed.
  • The images above maintain the original pose, but these models can also suggest new poses, in this case, moving the arm down to the side, as seen in this example, "A glowing neon jacket purple and pink"
  • Diversity of materials is demonstrated as well with "A leather jacket with gold studs"
  • And also with "A shiny translucent coat neon"
  • Inpainting can also be used to create new backgrounds that weren't in the original image, or it can be used to replace background pieces by just removing them and filling in the backdrop. First, start by hiding the background poles.
  • To generate a new background, apply a mask to and generate new scenes, in this case "a futuristic cityscape with flying cars."

  As you can see, inpainting is a very effective way to swiftly modify photographs and create dynamically generated new scenes.

  Anticipate even more powerful features from these tools in the future, freeing the user from editing masks. All you need to do is specify the alterations you want, such "Replace the background with a futuristic cityscape," and the image will be modified without requiring any mouse clicks or keystrokes.

• Outpainting

  By adding visual elements in the same style or reimagining a story using natural language description, outpainting allows users to express their creativity beyond the confines of the original image.

  You can expand an image, whether it is produced or real-world, beyond its initial bounds to create a more expansive and well-composed scene.

  • Real-world image outpainting
  • Start by uploading an image and selecting a region on the outside border where you would like to extend the image.
  • Similar to inpainting, you then generate prompts that generate coherent scenes that extend the original image.
  • Generated Image Inpainting works in much the same way. Start by generating a scene that will serve as the seed. Outside scenes lend themselves to more expansive outpainting, so we will start with “central park looking at the skyline with people on the grass in impressionist style."
  • Now add in a man playing frisbee with his dog.
  • It should be noted that the prompt makes no mention of maintaining the original image's skyline consistency. You can concentrate on adding the elements you wish to see, since the model takes care of extending and maintaining the background design automatically.
  • Finally, we will add in a family having a picnic and we have our finished image.

  Outpainting requires an extra layer of prompt refinement in order to generate coherent scenes, but enables you to quickly create large images that would take significantly longer to create with traditional methods.

  

  Impressive levels of imagination and the capacity to construct expansive settings that maintain coherence in theme and style are made possible by outpainting. As with inpainting, this capacity can be further enhanced by making it even easier to generate the desired scenes—just give one prompt—and obtain the precise image you want.

• Diffusion for Video Generation

  As we've seen, creating static pictures is fun and has a lot of real-world uses. A number of recently released models enhance the diffusion models' capabilities to produce videos. Although a larger audience has not yet had access to these capabilities, they will very soon.

  • Meta's Make-A-Video is a new AI system that lets people turn text prompts into brief, high-quality video clips.
  • Google's Imagen Video generates approximately 5 second videos in a similar fashion.
• Diffusion Model Image Curation Sites

  Highly curated collections of generated images and the questions used to make them can be found on curation websites such as Lexica.art. Since millions of photos are indexed, it's likely that the image you initially believed you would need to create already exists and is only a short search away. Low latency searches yield results very instantaneously, and you don't have to wait a minute or two for a diffusion model to generate photos. This is excellent for trying new things, looking for specific kinds of pictures, or investigating ideas. Another excellent resource for learning how to prompt to obtain the desired results is Lexica (provide a few examples here).

Diffusion Models: Practical Applications for today and tomorrow

Diffusion models' apparent use is in design tools, where they enable artists to be even more productive and imaginative. The initial set of these tools has actually already been revealed, and among them is Microsoft Designer, which incorporates Dall-E 2 into its toolkit. With generative product designs, fully generated catalogs, alternate viewpoint generation, and much more, there are a lot of prospects in the retail and eCommerce market.

Product designers will have access to potent new tools that will boost their imagination and enable them to visualize items in various settings, such as homes and offices. 3D diffusion has advanced to the point that products may now be fully rendered in 3D with the touch of a button. Taking this to the extreme, these 3D renders can then be printed as a 3D model and come to life in the real world.

Advertising will become more effective because to the capacity to test various creatives and develop ad creative dynamically, which will result in significant efficiency gains.

Diffusion models will be included by the entertainment sector into special effects tooling, allowing for quicker and more affordable productions. More imaginative and outrageous entertainment ideas—which are currently limited by expensive production costs—will result from this. Similar to this, the models' near-real-time content generating capabilities will enhance the experiences of augmented and virtual reality. With only their voice, users will be able to change their surroundings whenever they choose.

A new generation of tooling is being developed around these models, which will unlock a wide range of capabilities.

Conclusion

We still don't fully understand the depth of Diffusion models' limits, despite their amazing range of possibilities.

The capabilities of foundation models will inevitably grow over time, and advancement is happening very quickly. The way humans engage with robots will shift significantly as these models advance. Text is the Universal Interface, as Roon wrote in his blog, "soon, prompting may not look like "engineering" at all but a simple dialogue with the machine."

There are many potential to advance business, art, and society; however, in order to reap these benefits, technology must be swiftly adopted. Companies that don't utilize this new feature run the danger of falling far behind. We envision a time where human creativity and production will be unrestricted, and humans will only need a prompt to create everything they can think of. Now is the ideal time to begin this adventure, and we hope that this book provides a solid starting point.