Roxy Music lead singer Bryan Ferry started writing material for Avalon while staying at Crumlin Lodge on the west coast of Ireland. Ferry was there with his girlfriend, Lucy Helmore, who would become his wife in 1983. The album cover artwork featured the same lough (lake) that can be seen from the lodge. Ferry said of the album:
I've often thought I should do an album where the songs are all bound together in the style of West Side Story, but it's always seemed like too much bother to work that way. So instead, I have these 10 poems, or short stories, that could, with a bit more work, be fashioned into a novel. Avalon is part of the King Arthur legend and is a very romantic thing, when King Arthur dies, the Queens ferry him off to Avalon, which is sort of an enchanted island. It's the ultimate romantic fantasy place.
Guitarist Phil Manzanera has said about the making of the record: "By the time you get to Avalon, 90 per cent of it was being written in the studio. That album was a product of completely changing our working methods," adding "for the last three albums, quite frankly, there were a lot more drugs around as well, which was good and bad. It created a lot of paranoia and a lot of spaced-out stuff." Production began at Manzanera's Gallery Studios in Chertsey, where they worked on song ideas in the studio and developed backing tracks to a Linn LM-1 drum machine. After a month of refining the tracks at Compass Point Studios, the sessions relocated to the Power Station in New York City for final overdubs (including drums) and mixing with Bob Clearmountain. "To Turn You On" was previously released as a B-side to "Jealous Guy" in 1981, and was an outtake from Manifesto as per Davies - only a new mix was made for Avalon.
Manzanera said of the title track, "Avalon", "When we were recording the third or fourth album in London we'd often be working in the same studio as Bob Marley, who'd be downstairs doing all of those famous albums. It just had to rub off somewhere." Rhett Davies recounted the story of how the song got made:
[While on a coffee break in the studio], Bryan and I could hear this girl from the Haitian band next door singing, and we thought, "Wow! What a voice! We've got to get her singing some backing vocals on 'Avalon.'" That was Yanick Étienne, who didn't speak a word of English. She came in with her boyfriend/manager and we described to him what we wanted and she sort of sang the choruses and the [word] "Avalon" – the great sound that is on there. Then we said, "Can she try and do something free at the end?" and we ran the end of the track and she did absolutely nothing. So I said, "No, we want her to sing anything that she would want to sing, totally free." So the second time we ran the tape, she sang exactly what you hear on the record at the end.
Bryan then went straight out and re-sang his vocal properly, because he was so inspired by Yanick's singing. I remember Bryan's manager walked in the room and Bryan was just finishing his vocal. We were doing the playback and I'd never seen the look on his eyes before. He went, "Jesus fucking Christ! That is incredible!" Well, we knew it was a really high point of the evening. I remember going, "Wow! We have really created something special here." That is how I felt. Then we mixed it the next day with Bob [Clearmountain].
It was one of those turnaround things, where the original track was just about to be thrown in the can. And then suddenly, we did a completely different version of the song that just made the record for me ... I thought, "That's it. That completes the record!" I remember we had dinner a couple nights later, and I asked Bryan, "What are you going to call the album?" and he said, "I'm going to call it Avalon," and I thought, "Yeah. Of course."