Visualizing Poetry using DALL-E
Over the weekend I ran an experiment to use DALL-E 2 to automatically illustrate poems. I had been playing with this thing a few months ago, but was using a different image generator and though it was interesting, it never came out well. The new results are more promising.
We take each poem and offer its individual lines to DALL-E to generate an image for them. In order to make them come out more consistent and in line with their time, we also pick a visual artist in whose style the illustrations should be rendered.
Take for example Robert Frost’s The Road Not Taken. Robert Frost was a contemporary of Edward Hopper and according to at least one source they did hang out. So we kick off DALL-E with the following prompt: “a painting in the style of edward hopper depicting Two roads diverged in a yellow wood” and after putting that line over the picture we get:
This works quite nicely. It’s a manual process though and so I only did 6 poems. They are all originally English except for one which is a translation from Dutch and one of the more famous poems (well, in the Netherlands that is).
See individual poems below or jump straight to the project.
Harlem
This poem by the Harlem renaissance poet Langston Hughes speaks of what happens to a dream postponed — the American dream presumably. It’s 70 years old, but rather relevant.
The poem is very visual and seemed like a good match for this approach. I paired it with the painter Jacob Lawrence, who in his extensive work painted many Harlem scenes.
Memories of Holland
This poem by Hendrik Marsman reaches back to a version of the Netherlands that we’ve mostly forgotten about. It paints a picture of a rural, conservative country that is flat and always threatened by the sea. In these times of climate change weirdly relevant of course.
The poem has very short lines which makes it not always easy for the AI to pick up on the meaning. It mostly seems to get it right though. I considered pairing it with Vincent van Gogh, but that felt too wild, so settled for just generic Expressionism.
Shall I compare thee to a summers day?
Probably Shakespeare’s most famous sonnet and it felt wrong to do something about English literature without including something from England’s national poet. I have to say it seem he’s a bit full of himself.
I paired it with Rubens. Not the same country of course, but roughly the same period of time and a lot of the imagery in the poem does feel vaguely Flemish if not always specifically Rubens like.
There’s one line in there:
“Nor shall death brag thou wand’rest in his shade,”
DALL-E refused action and said it did not think it passed the content guide lines. I think it is just because it contains the word death? Seems like an AI company could do better than that. I just replaced the word with “grim reaper” and it worked. A nice skeleton showed up.
She walks in beauty
The number one romantic British poet, Lord Byron can’t be left out in a project like this. The title of this work alone immediately conjures up images of a mysterious lady and the rest of the poem continues with good imagery. The AI mostly played along, except arguably for:
And on that cheek, and o’er that brow
Which it turned into a skull.
I ended up asking DALL-E to generate paintings in the style of Francisco Goya for this poem. It’s not the right country, but it has the right feel and is roughly from the same period. Contemporary British painters all seemed maybe a bit more subdued compared to what Byron projects
The Lake Isle of Innisfree
This fine poem appears in Irish passports and describes Yeats’ longing to escape the city for a lake and an island. DALL-E plays along very nicely, showing the island and lake and then suddenly switching briefly to the urban grayness.
The pictures are done in the style of Claude Monet. Not very Keltic Revival, but good for painting lakes and bees. Also Monet is just pretty and recognizable.
The Road Not Taken
Robert Frost’s first line “Two roads diverged in a yellow wood” begs for visualization of this poem. The more contemplative bits are harder for the AI to work with, but I think it does ok. Humans seem to have trouble understanding it too.
We talked about in the intro how Edward Hopper is a good match. Hopper’s work is more urban though than this particular poem. On some occasions I swapped Frost’s “road” for “path” to avoid unexpected buildings on the side.
Resources
You can find a javascript animation version of this at douwe.com. I also created a youtube playlists with the animations as videos.
