After the Norman Conquest, William I (William the Conqueror) divided the lands of England up amongst his followers. Aubrey de Vere was given an estate that covered Castle Camps and Nosterfield and several other parishes in Cambridgeshire including Abingdon and Hildersham.
He chose the site at Castle Camps for his castle to function as the administrative centre of his Cambridgeshire estate as it lay half way between Cambridge and his caput (major centre) at Castle Hedingham. The castle would continue to be held by the de Vere family for over 500 years, apart from a couple of brief periods when it was confiscated by the Crown.
The castle was begun about the year 1100 and was of a motte and bailey design built of wood. The motte covers about two acres and is surrounded by a wet moat. Originally there was a small bailey to the north-west of the motte (in which the church now sits). It has been suggested a larger bailey was added to the castle in the late 13th Century.
The estate around the castle consisted of a deer park, fishery and windmill.
It seems around the year 1500 the castle no longer existed on the site in its original form and a house with a large brick tower was recorded as occupying the site of the castle. Several houses would later be built on the motte during different periods.
The castle motte has now been much lowered and is occupied by a farm. You can view the motte, moat and earthworks from a public footpath.