The Rectory behind the church was demolished on completion of the Castle. The little township of dwellings belonging to the masons and labourers which had grown up on the bank side behind the church was possibly knocked down too, and the site turned over to gardens and orchards. The home farm would have occupied the flat ground beyond the wood at the rear of the castle. The gardens enjoy fine views and, unlike the castle itself, with its steep spiral stairways, are accessible, with help, for visitors in wheelchairs.
The original garden covered an area of nearly 20 acres, stretching across to the shelter belt in the west and down to Scrogs Wood to the south. To the west, in now what is described as the moat, was a formal arrangement of flower beds, paths and arbours and one raised mound is thought to cover the remains of a fountain.
To the south, archaeologists differ on their interpretation but we think that the recently renewed garden laid out on artificially flat ground stands on the site of either a bowling green or possibly another geometrically designed formal garden. All agree that there was a summer house or pavilion and possibly a dovecote in the wildflower garden below. (Dovecotes were a feature of many castles – their tenants being a useful source of food).