Lying close to St.Mary`s Church on the western edge of Hay-on-Wye is a small but well-preserved motte. The site overlooks a gorge and small stream, locally known as The Loggin Brook, that flows into the River Wye, which was undoubtedly one reason for the construction of a motte and bailey castle there. A recently levelled platform under the car park to the north east may have once have housed the castle's bailey. This little fortress was probably the work of William Revel, a knight of Bernard de Neufmarche who is usually referred to as Bernard Newmarch, and may later have been the seat for the manor or commote of Melinog. Other than this the motte has no further recorded history.
The stone castle
The main fortress within Hay-on-Wye was situated on the great site commanding the town and river under the current ruins of the castle and mansion. This was undoubtedly the 'castello de haia' handed to Miles de Gloucester, 1st Earl of Hereford in 1121 with the daughter of Bernard de Neufmarché. It is most likely that the keep stood by this time. It is therefore possible that this is the oldest Norman tower in Wales, dating to the onslaught of William Fitz Osbern in 1070. During the anarchy (1136-54) in the reign of King Stephen a series of charters were passed by the Gloucesters concerning the castle. In 1165 the last of Miles de Gloucester's male descendants was killed at nearby Bronllys Castle and Hay-on-Wye Castle passed into the hands of William de Braose, 3rd Lord of Bramber and of New Radnor and Buellt. The de Braose dynasty were energetic lords and probably built the core of the gatehouse which now stands besides the keep. In the summer of 1198 a major English army formed here before marching off to victory at the Battle of Painscastle some four miles to the north.
In 1230 the last de Braose of Brecon, William de Braose was hanged by Prince Llywelyn ab Iorwerth and Brecon lordship with Hay-on-Wye passed into the hands of the de Bohuns. Taking advantage of this in 1231, Prince Llywelyn ravaged the lands of his de Bohun in-laws during which Hay-on-Wye town was burnt, although the castle survived the onslaught. The castle saw service in the Barons' War of 1263 to 1266, changing hands three times, once being surrendered to the great Simon de Montfort, 6th Earl of Leicester. With the conquest of Wales by King Edward I Longshanks life became more peaceful in this Marcher town.
Around 1401 both town and castle suffered damage by the forces of Owain Glyndwr, although the castle was listed as defensible against the Welsh in 1403. The fortress later passed to the earls of Stafford, who were to become the unlucky dukes of Buckingham during the Wars of the Roses. The castle was repaired during the conflicts of the 1460s, although its military use would have been somewhat dubious against cannon.
In the 1660s, James Boyle of Hereford built a new mansion on the north side of the castle, while most of the curtain wall was demolished to improve the views. The mansion is now used for second-hand bookselling.