It is unclear who specifically built Walden Castle, though it is widely attributed to being constructed on the order of Geoffrey de Mandeville, the Earl of Essex during the Anarchy between 1141-1143. De Mandeville was captured by King Stephen in 1143 and ordered to relinquish control of his castles. He initially agreed to this but then on the intervention of the Empress Matilda and being offered the castle at Bishop Stortford, rebelled against King Stephen, leading a major insurrection against King Stephen in the East of England.
De Mandeville never regained control of the castle having relinquished it to the king. He died in 1144 having attacked Burwell Castle in Cambridgeshire and being mortally wounded.
The castle at Saffron Walden is believed to have never been completed and King Henry II ordered that the castle be torn down in 1157.
Only the stone walls of the keep remain with much of the stone being robbed away.
The castle is located directly next door to the Saffron Walden Museum in the centre of Saffron Walden.