The castle was located on the north-west side of the present town of Kington above the Back Brook, in Castle Hill. Around the end of the 11th century William Rufus granted the estates in and around Kington to Adam Port and it is likely that he built the first castle here. One of his descendants, also called Adam Port, was in dispute with Henry II and as a consequence of this the castle was forfeited to the crown.
In the late 12th century, William Braose held the castle for many years in his capacity as High Sheriff of Herefordshire. In 1201, King John granted the castle and barony to him as his favourite Marcher Baron. However, another dispute arose, and the castle returned to the crown again, later to be granted to King John's supporter Roger Clifford in 1213. In 1215, members of the de Braose family attacked the castle, and in an act of vengeance, King John destroyed it. No further fortifications have been made on the same site.
