Generate Meme Variations Using OpenAI’s DALL-E 2

Exploring OpenAI Dall-E 2’s edit image mode for memes

In this blog, we will see an interesting application by OpenAI’s Dall-E 2 where you can use existing memes and create more meme template variations by using the edit image feature of Dall-E 2.

So why do we need to create multiple variations of memes in the first place?

We are building an AI SaaS app Supermeme.ai — An AI-powered meme generator to generate original memes. Users can search for relevant meme templates for their emotions or actions using our advanced meme search and can pick a template to generate meme text with AI.

Since the number of meme templates in our existing database is limited, it would be ideal if we create more templates from existing meme templates to offer to the users. As we recently got access to Dall-E-2 we explored the possibility of generating meme template variations from existing meme templates. And the results have been interesting!

If you got access to Dall-E 2, visit https://labs.openai.com/, and you will land on the Dall-E 2 home page. To edit images, you can upload your own images by clicking on the “upload an image” link.

Once uploaded, you need to select an area to erase part of the image, and then you need to describe the new image in the text box provided.

1. Chainsaw Bear Meme — Replace with a sword

I uploaded the chainsaw bear meme, selected the mask area as shown below, and wrote: “replace with a sword” as the edit text to replace the chainsaw with a sword.

And among the generations by Dall-E 2, I picked the one that matched best with my prompt. You can see that although I typed replace incorrectly in the prompt it still yielded good results.

2. Change my mind Meme — Replace the text

The second meme I tried was the “change my mind” meme where I tried to mask the text and replace it with “I am awesome”.

But OpenAI threw an error that said “Uploads with realistic faces are not allowed at this time”. I saw the announcement about this very recently but saw it in action this time during my trials. So I couldn’t proceed ahead with this meme template.

3. “What the hell is this?” meme — Add a new animal

The next meme I tried was “what the hell is this?” where I tried to add a lion illustration beside the penguin. But interestingly, OpenAI threw an error saying “This request may not follow our content policy”.

Personally, I didn’t see anything wrong with this meme. Probably it was an error on OpenAI’s part and there was no way for me to report it so I ignored it and continued my exploration.

4. “Left exit 12 off-ramp” meme — Replace the car with a motorbike

The next meme template that I tried was “Left exit 12 off-ramp”. I masked the area with a car and gave the prompt “Add a motorcycle exiting the highway”.

DallE-2 generated several variations but most images looked unnatural and blurry. The closest one I got for usability was this.

5. “Unfinished horse drawing” meme — Replace the horse with an elephant using recursive editing

The next meme template that I tried was “Unfinished horse drawing”. This time I tried to do double edits, one for the back area and one for the front area to convert it into an unfinished elephant drawing.

First I masked the back of the horse and gave a prompt “Add an unfinished back of the elephant pencil drawing”.

The best generation from Dall-E 2 was this —

This time I wanted to do recursive editing so I took the generation from above and masked the front and gave the prompt “Add an unfinished front of the elephant line drawing”.

And finally got this output of unfinished elephant—

6. “Epic handshake” meme — Add a tattoo that reads “power”

Finally, I tried the “epic handshake” meme to add a tattoo to the bicep that reads “community is power”. But Dall-E was bad at generating long text on the arm. It was a bit random. Hence I later changed the text to “power” and it was able to generate something useful.

Conclusion

In general, Dall-E 2’s edit image feature has quite a few limitations.

1. It doesn’t take the erased context into account so you cannot do complex edits. It only tries to do infilling of the masked area based on the surrounding context of the image and prompt.
2. It can only take square images or a square crop of the image to edit.
3. It doesn’t generate photorealistic edits most of the time so the edited image looks unnatural at times.

The other option is to upload your meme templates and use the Dall-E 2’s Generate variations options to let Dall-E generate variations by itself. The only con is that you cannot give any prompt instructions to direct the output.

But it was an interesting exploration and when Dall-E 2 API becomes commercially available we can see a few use-cases with the edit mode. With a human-in-the-loop system, we can expand the existing template library in creative ways.

If you liked the content, feel free to stay connected on Twitter.