Taunton Castle Hotel stands as part of Taunton Castle.

Taunton Castle was a stronghold and administrative centre of the Bishops of Winchester. Built in the early 12th century beside the River Tow and near the Saxon minister church, it included a large keep for soldiers' quarters and occasionally for prisoners. Beside it stood a hall and a chamber over a vaulted undercroft. The base of the keep survives in The Castle Hotel garden.

In the 1240's Bishop William Raleigh improved the hall and built a new chamber and a chapel, which still survive. There were also kitchens and other offices, and a gateway (altered by Bishop Langton in 1496) which housed the treasury and exchequer. Together these buildings, surrounded by a moat, formed the inner ward.

To the south was a moated outer ward, entered by gates from east and west (Castle Bow). The Castle Bow also included quarters for the Constubte. In the outer ward stood St Peter's Chapel, lodgings and farm buildings. The last Medieval addition was the school house, built by Bishop Fox in 1521-2.

During the Civil War the castle, then recently fortified, was defended by Robert Blake for Parliament, but the keep was probably distmantled soon after the Restoration. The great hall was used for Assizes, and rebels were tried here by Judge Jefferys after Monmouth's Rebellion in 1685. Much of the building was restored in the 1780's by Sir Benjamin Hammet. In 1874 the castle was bought by the Somerset Archaeological and Natural History Society to prevent its demolition, and the building now houses the Society, the County Museum, and the County Local History Library.