The building of this stronghold was started by Richard Nugent, 12th Baron of Delvin, in the year 1533. The tower was completed by him in 1537. The great hall and further extensions were finished by his grandson and successor, Richard, the 13th Baron, by 1539.

The original de Nogents left their native France to join their fortunes with those of Norman William in the invasion of England in 1066.

Among the names of the principal men who took part in the Norman-Welsh invasion of Ireland (1169-1172) are to be found those of Gilbert de Nogent.

The celebrated Hugh de Lacy, who had been granted the greater part of the County of Meath, had become to rely so much on the young Gilbert, that he gave him the hand of his only sister, Rose, in marriage. As a dowry he bestowed upon them the Barony of Delvin.

De Lacy encouraged the building of strong castles for the purposes of defense against the native owners from whom the these lands had been taken.

In the centuries following, the de Nogents were always "King's Men". They never would forget that all they owned had been granted to them by the power of the larger island across the Irish sea. They did their utmost to maintain the English foothold in the new colony, and while this was no simple task, they spread out and added to their own territory. By the 15th century they were in possession of lands as far north as the shores of Lough Sheelin on the Breffni border. The Nugents adhered strictly.

Built in the 16th century, the tower of Ross Castle was used by the celebrated Myles O'Reilly, otherwise known as 'The Slasher', the night before being killed by Cromwell's troops at the Battle of the Bridge of Finea in 1644.

The castle was also the scene of the tragic 'love story of Orwin and Sabrina', recorded in a book by a late local poet. The castle later fell into disrepair and was restored in 1864 by Anna Maria O'Reilly, a lineal descendant of Myles O'Reilly.