Richmond Castle was built on land gifted to Alan Rufus, a kinsman of William the Conqueror, soon after the Norman Conquest of 1066. The square Great Tower, built over the original gatehouse in the 12th century, is of a typical Norman design with square turrets, a flat roof and shallow buttressing.

The castle was declared a ruin in 1538 which probably saved it from being slighted after the English Civil War. In the Victorian era, the castle became the headquarters of the North York Militia. During WW1 it was occupied by the Northern Non-Combatant Corps, a unit of men seeking exemption from military service.