The ruins of Rufus Castle sit high up on a rocky promontory on the Isle of Portland. Much of what remains dates from the late 15th century although the site, which overlooks favourable landing grounds for enemy raiders, was also once a Saxon defensive position and that of an earlier 11th century fortification from which it takes its name.

William II, King of England was also known as William Rufus due to his red hair. One of the castle's few records of action took place in 1142, when during the civil war known as The Anarchy it was captured from King Stephen on behalf of Empress Maud.

The ruins are privately owned but are not open to the public as the site requires major restoration and repairs.