Brown was recruited out of high school by Washington State, Clemson, Idaho State, UCLA, and Miami (Ohio), before he ultimately committed to attend Brigham Young University.[1] He later said he believed his experience at a private high school prepared him for the culture shock of the overwhelmingly Latter-day Saint, predominately white campus at Brigham Young, but was surprised at the difficulty of the adjustment.[2] During his freshman year at BYU, he considered transferring to another school.[3]
In February 2004, Brown was baptized and confirmed as a member of the Church of Jesus Christ of Latter-day Saints. His mother, who like the rest of his family is not a Latter-day Saint, expressed happiness for Brown's decision and appreciation for the support of the BYU community.[4] Brown played alongside running back Fahu Tahi, whom he nicknamed "Pops" for his extended tenure with the Cougars.[5] During Brown's time at BYU, he was mentored by former record-setting running back Luke Staley.[6]
As a true freshman during the 2002 season, he saw action in all twelve games. Against Utah State, Brown replaced injured starter Marcus Whalen, and rushed for 217 yards and three touchdowns, and made four receptions for 48 yards. Brown became the fourth BYU player to run for more than 200 yards in a single game. The performance helped the Cougars recover from a deficit to win, 35–34, and earned him honors as the Mountain West Conference Offensive Player of the Week.[7] Brown sat out the 2003 season as a redshirt.[1]
In 2004, Brown played in all eleven games and was named to the All-Mountain West Conference second-team.[1] Against UNLV, BYU lost, 24–20. In the game, Brown received only eight carries, but ran for 102 yards, and "the Rebels were seemingly helpless to stop" him.[8] The Salt Lake City Tribune noted that he refused to criticize the playcalling of head coach Gary Crowton and instead replied, "I just do my thing."[9] Wyoming head coach Joe Glenn said of Brown, "He's got some strong legs. Las Vegas put some great shots on him and he ran through some tackles. You've really got to bring your feet when you tackle this guy."[8] Against San Diego State, Brown recorded 106 rushing yards on 16 carries, which made him the first BYU player since 2001 to post four consecutive 100-yard performances. He gave credit for his play to the BYU offensive line.[10] Brown pushed his college career mark past 1,000 rushing yards in the game against New Mexico, which made him only the fourth BYU player to surpass that number as a sophomore.[1]
As a junior in 2005, Brown recorded 210 carries for 1,123 rushing yards and 53 receptions for 454 receiving yards. He was named to the All-Mountain West Conference first-team, and was twice named the conference's Offensive Player of the Week for his efforts against Air Force and Wyoming.[1]
In his final season in 2006, he repeated as an All-Mountain West Conference first-team selection and recorded 1,010 rushing yards. John Beck and Brown became the first BYU quarterback-tailback tandem to twice record 3,000 passing yards and 1,000 rushing yards, respectively, in the same season.[1] Brown became the all-time rushing leader at BYU during his senior year. From 2002 to 2006, he accumulated 3,218 career yards, which eclipsed the previous mark set by Jamal Willis.[11] After breaking the rushing record, Brown praised the play of the offensive line rather than take credit for the accomplishment himself.[3] He averaged over five yards per carry during his college tenure.[12] Brown's record was in turn surpassed by Harvey Unga on November 21, 2009.[13]
After college, Brown was signed by the Cincinnati Bengals and played on the team during the 2007 preseason. He saw action in all four games and recorded stats against Detroit, New Orleans, and Indianapolis.[14] Ultimately, he did not make the team's final roster.