Despite the title, cover, Swedish alternative titles, and the humorous mini-essay about "The Swedish conspiracy" in the liner notes (written by Paul Lukas, though he was only credited a year later in the liner notes to the band's next release Nothing for Juice), none of the lyrics are explicitly about Sweden itself. Various other locations, such as Seoul, Korea, California, Queens, New York City, Bolivia and Denmark are however all mentioned in the songs. "Duke Ellington" (which appears on the rarities compilation Protein Source of the Future...Now! and the Harriet records compilation The Long Secret), which does mention Sweden, is described as "one of two pieces written for the song-cycle Sweden and intentionally left off of the album". The unreleased "I've Got the Sex" is the other song intended for Sweden that was left off of the album. According to Darnielle this song was originally supposed to be the first track of Sweden, but was left off due to Darnielle accidentally forgetting its tape at home during the mastering process.
In addition, there exists a sequel album to Sweden, entitled Hail and Farewell, Gothenburg, but it was never released. Furthermore, at a December 2010 concert at the Castro Theater in San Francisco, California, Darnielle unveiled 6 more songs from the "Sweden notebooks".
The coda of "Cold Milk Bottle" is taken directly from "Mean to Me," a popular song from 1929, written by Fred E. Ahlert and Roy Turk and performed by artists including Billie Holliday, Sarah Vaughan, Judy Garland, Frank Sinatra, Rosemary Clooney, and Ella Fitzgerald.