A castle/manor house has stood on the site of Walworth Castle since the late 12th century, with the majority of the present Walworth Castle dating back to the 16th century.

1150 - 1563

The original castle was built around 1150 by the Hansard family but, following the Black Death, the title passed onto the House of Neville in 1349. Walworth was reclaimed by Robert Hansard in 1391. The castle then passed through the hands of various members of the Hansard family until 1539, when Elizabeth Hansard married Sir Francis Ayscough. The castle then passed to the Ayscough family. In 1563 the castle passed to her son William Ayscough, but as there were no further heirs the castle was sold.

1564 - 1758

In approximately 1579 Thomas Jennison, Auditor General of Ireland, purchased the Manor of High Walworth from the Ascough family. He demolished the majority of the manor, apart from the Medieval south−west tower, and built the castle as we know it today.

In 1586 Jennison died and his widow Elizabeth, née Birch, inherited the castle. It was during her ownership that King James VI of Scotland is believed to have rested here, on 14 May 1603, while travelling to his coronation as King of England.

In 1605 Elizebeth Jennison died, and her son William Jennison inherited the castle. The Jennisons were a strongly Catholic family and, in 1610, William was imprisoned for his Roman Catholic beliefs. During his imprisonment Walworth fell into disrepair.

In 1679 Francis Jennison sold the estate and emigrated to Europe. This is possibly because in 1678 Thomas Jennison was accused of involvement in the Popish Plot to assassinate Charles II, arrested by Titus Oates and thrown into Newgate Prison.

In 1681 the castle was divided from the rest of the estate and awarded by Chancery to Robert Jenison. In 1687 the castle was reunited with its estate when Ralph Jenison bought the whole estate for £6,205. The castle was searched for arms in 1689 in response to suspicions of a potential rebellion against the Protestants William III and Mary II.

Ralph Jenison inherited the castle at the age of 10 years in 1704. He later renovated the castle at great expense, and died in debt, at which point the castle was again sold.

1759 – 1980

The castle was sold for £16,000 in 1759 to wine merchant Matthew Stephenson, and then again to Newcastle merchant John Harrison in 1775. His daughter, Ann, married Arthur Aylmer, a British Army officer of the 68th Regiment of Foot, later promoted to Lieutenant-General; and so it passed to the Aylmer family.

After General Aylmer was buried in Heighington in 1831, John Harrison Aylmer inherited the castle. He repaired the roof and replaced previous soldier statues on the towers with pillars topped with balls, to look like the statues. However, in 1868 he, his wife and eldest son were killed in the Abergele rail disaster. His sons Vivian and Edmund, aged 12 and 9 respectively, inherited the castle. Vivian became High Sheriff of Durham and a big game hunter who explored Somaliland and crossed the Horn of Africa in 1885. He died in 1931, and he and his brother were buried in Caerleon.

The castle was then sold to General Aylmer's descendants, Neville and Charles Eade. In World War II, during the Eades' ownership, the castle was used as a prisoner-of-war camp for 200 men, including German and Italian officers, under the command of Major Rollin Holmes. In 1950 Durham County Council bought the castle and it became a girls' boarding school.

1981 - Present

In 1981 the local council sold the castle to John and Jennifer Wain. Following extensive renovations, the castle was opened as a hotel. In 2000 the castle was purchased by Rachel and Chris Swain, who have since invested heavily in the property.