During the Middle Ages Bishop's Waltham Palace was one of the finest residences of the richest churchmen in Europe – the Bishops of Winchester. It was originally built as a fortified castle by Bishop Henry of Bois, the younger brother of King Stephen, in the 1130s. In the late 14th century it was redeveloped into a palatial luxury residence, intended to impress, and could accommodate both the bishop's household, and the king and his court. It was frequented by a number of English monarchs including Henry II, Richard I, Henry V, Henry VIII, Mary I and Elizabeth I.

The palace was abandoned after being extensively damaged in 1644 during the English Civil War. The ruins were purchased by the renowned physician Sir William Jenner in 1889 and were later home to WW2 Admiral and First Sea Lord, Sir Andrew Cunningham. In 1952 it passed to the Ministry of Works and is now under the guardianship of English Heritage.