Walmer Castle is one of a trio of artillery forts built along the Kent coast by King Henry VIII in 1539-40. It later became the official residence of the Lord Warden of the Cinque Ports. Famous holders of the position include the Prime Minister, William Pitt and The Duke of Wellington.

'The Governor's Garden' was established around 1725. Four box-edged beds were made to grow a range of ornamental fruit and vegetables, along with traditional Kentish varieties of apples and pears. The long range of glasshouses were first constructed in 1898 and expanded between 1936-41 to house a vinery, tomato house and small fernery.