The Kerrs are traditionally said to be of Anglo-Norman descent settling in the Scottish Borders in the 14th century, however the first of the name appears to be John Ker hunter of Swinhope. The two main rival branches of the Kerrs descended from two brothers, Ralph and John who were living near Jedburgh in the 14th century; the Kerrs of Ferniehurst were descended from the eldest and the Kers of Cessford from John. Although the two families were constantly in bitter conflict the descendants of both these houses were appointed Wardens of the Middle Marches; Sir Andrew of Ferniehurst in 1502 and Sir Andrew of Cessford after the Battle of Flodden.

The Kerrs continued to oppose one another during the 16th century and on the death of James IV, when his widow Margaret Tudor remarried the Douglas Earl of Angus, the Kers of Cessford supported the English Queen-mother and the Kerrs of Ferniehurst the young King, James V. Cessford was forced to flee to England when Angus was exiled only to return on the death of James V in 1542 when Sir John Kerr of Ferniehurst lost his castle. The castle was recaptured in 1549 and the English who had repeatedly raped the Kerr women, rather than being killed, were captured and horribly tortured. The rivalry continued when Sir Thomas of Ferniehurst fought for Mary Queen of Scots at Langside and Sir Walter Cessford on the side of James VI. The feud was resolved on the political level by the Union of the Crown and by the marriage of Anne Ker of Cessford to William Kerr of Ferniehurst.