Patrick Whytelaw, son of Lord Ruthven, constructed the current building around 1550, a typical fortified tower of the time. But in 1587, ownership of the tower was forfeited to Sir John Carmichael. No doubt Fenton Tower was the reward for being the Scottish Ambassador to Denmark and a loyal subject of the crown.
In 1591, King James VI of Scotland, son of Mary Queen of Scots and Lord Henry Darnley, was surrounded by a rebel army in Fife, north of Edinburgh. Fortunately the local townspeople rallied to his support and helped him to escape. James took the ferry across the Firth of Forth to North Berwick and took refuge with the Carmichaels at Fenton Tower.
James VI later became James I of England and Scotland and, upon the death of Carmichael, he granted Fenton Tower to Sir Thomas Erskine, who became Lord Dirleton, Viscount Fenton and later Earl of Kellie.
In 1631, the tower passed to Sir John Maxwell of Innerwick, who became Earl of Dirleton in 1646. However, his enjoyment of the castle was shortlived. In 1650, Oliver Cromwell invaded Scotland and he sacked Fenton Tower along with the nearby Dirleton Castle. With Fenton Tower now just a ruin, the surrounding land passed to Sir John Nisbet in 1663.