A classic Norman motte-and-bailey castle built in the 1070s by William de Warenne, who fought alongside William the Conqueror at the Battle of Hastings, during the Norman Conquest. The family developed the original fortified house, founded the surrounding village, and also the nearby Castle Acre Priory.

The castle was reported as derelict by 1397 with the ruins serving as a quarry and stockade for cattle for several centuries.