The ancient Castle of Fernie was first recorded in 1353 when it belonged to the Earl of Fife - Duncan the 13th. The family of Fernie held the lands as early as the 15th century - Walter Fernie being dead before1496. In 1527 Andrew Fernie obtained a charter erecting Fernie into a free barony and he conferred the life-rent on his wife, Barbara Logane. Their son succeeded in 1551 and he sold it to William Fernie of Foxtoun 1582.

Sir Michael Balfour of Balfour or Burleigh died in 1619, leaving an only daughter, Margaret, who married Robert Arnot, younger of Newton. He assumed the name of Balfour, and sat in Parliament as second Lord Balfour of Burleigh.
His youngest daughter married her cousin, the last Arnot of Fernie, and on extinction of that line the property of Wester Fernie fell to Lady Arnot's eldest brother, John, who succeeded as third Lord Balfour. His estate was forfeited to the Crown because his second son was concerned in the Rebellion of 1715.

His eldest son, Arthur Balfour, remained attached to the house of Harrower, and George 1 granted to him and his five brothers the lands and barony of Western Fernie in 1720. Arthur died in 1746 leaving three sons who succeeded to the estate consecutively. John Balfour of Fernie became heir male in 1757 of the Lords Balfour of Burleigh, but he died without issue in 1795. He was succeeded by his only surviving brother, Francis, who died in 1818. In 1854 the grandson, Major Francis Balfour, claimed the title of Lord Balfour of Burleigh, but the
House of Lords, on the reveal of attainder 1869, ajudged that dignity to Alexander Hugh Bruce of Kennet, who was the descendant in the female line of Robert, fourth Lord Balfour. The duties of Forester of Falkland and Constable of Cupar were associated with the Barony of Fernie.