Winter Women and Holy Ghost Language School

Winter Women – 62:44

1. "Under the Hood at the Paradise Garage" – 4:06
2. "The Pennsylvania Rock Oil Co. Resignation Letter" – 3:43
3. "Up the River" – 4:29
4. "Ruth vs. Richard" – 2:51
5. "Her Chinese Typewriter" – 4:21
6. "Big Bill Crib and His Ladies of the Desert" – 3:21
7. "Don't You Remember?" – 3:45
8. "Becha Don’t" – 2:54
9. "P.S. 213 Mini"-School – 3:54
10. "Theme from Never Going Home Again" – 2:44
11. "Motorman" – 5:58
12. "Quick as Cupid" – 3:47
13. "I Love You Cedric" – 4:39
14. "Servant in Distress" – 3:28
15. "Hialeah" – 2:56
16. "Wisconsin River Blues" – 5:40

Bonus Tracks from 2009 rerelease-

1. "Rio Tinto" - 3:50
2. "260 Northern Blvd." - 3:02

Holy Ghost Language School – 46:09

1. Seventh Loop Highway – 3:57
2. Holy Ghost Language School – 4:13
3. The Cross and the Switchblade – 3:54
4. I Started Using Alcohol at the Age of Eleven – 3:57
5. Do You Like Blondes? – 2:50
6. Azusa St. – 4:24
7. Topeka and San Antonio – 3:40
8. A Mystical Preparative to Lewdness – 2:47
9. Ship Scrap Beach Business – 2:18
10. First Day of School – 2:50
11. Things Were Going So Well – 4:07
12. All in Vain or the Opposite – 5:05
13. Moral and Epilogue – 2:00

Bonus Tracks from 2009 rerelease-

1. "Rain" - 12:43
2. "Porcupine Posed a Prickly Problem" - 11:53

Despite the solo albums The Fiery Furnaces will continue releasing new material as a group. When asked by Pitchfork about the reason for them being recorded/released separately Matthew, who writes most of the music and lyrics for The Fiery Furnaces responded with:

"Eleanor's on vacation, We have a logjam of records, so [this is] the only way to have something come out relatively quickly. And I don't have to bother Eleanor to get her to sing all this crap, so...that's the advantage. It's an act of mercy."

Winter Women is also stated in the liner notes as being dedicated to Eleanor.

While Winter Women is a more conventional collection of songs, the second record Holy Ghost Language School is a narrative-based concept album, although it has significantly fewer lyrics.

Matthew Friedberger played the majority of the instruments on the records and did most of the production work but John McEntire from Tortoise and Josh Johannpeter from Mahjongg helped out by providing some of the non-synthesized drums.