With origins as a 13th century manor house, the defences of Edlingham Castle were strengthened by the addition of a gate tower and stone curtain wall during the Anglo-Scottish wars.

By the mid-17th century it had become abandoned and was being quarried for building stone, leaving only traces of a hall and courtyard with a kitchen, brewhouse and bakehouse. The most complete remaining part of the complex is the three-storey solar tower.