Perhaps the greatest volume of information about the Barholm branch of the McCulloch family is present in the manuscript "A History of the Galloway Families of McCulloch" (Walter Jameson McCulloch) which is available in the Ewart Library, Dumfries. However, as pointed out by W. J. McCulloch, "For reasons best known to themselves, successive representatives of the branch have firmly and repeatedly refused to allow the family documents and papers to be examined." What is known is mainly derived from written legal records, but the lack of detailed records limits what can be said about the building, its structural changes and its occupancy. The designation "Barholm" often refers more to the branch of the family and a geographical area, rather than to a specific place.

The lands of Barholm passed (back) into the hands of the McCulloch family early in the 16th century, but a tower house was probably on the present site by late in the 15th century. A building designated Barholm Castle was clearly established on this site at the time of the survey carried out between 1584 and 1596 by Timothy Pont, and published in the 17th century Blaeu maps. While this was probably substantially the present structure, it cannot be said with certainty that the stair tower was in place at this time.