The name was originally perhaps a nickname, Skirmisher. meaning 'hardy fighter', or more likely a version of escrimeur, French for swordsman. The first bearer of the name was one Alexander Carron, who received the hereditary title of Royal Standard (or Banner) Bearer of Scotland after an exceptionally brave action in the presence of the then King of Scots whose army was crossing the river Spey. Carron's descendent, Alexander Scrymgeour, was an adherent of Sir William Wallace, and the only Charter known to have been granted by Wallace as 'Guardian of the Realm' confirmed him as Royal Banner Bearer, granted him lands in Angus, and made him 'Constable of the Castle of Dundee'. After Wallace's death Scrymgeour supported King Robert the Bruce, was captured by the English in Bruce's defeat at Methven in 1306 and hanged at Newcastle-upon-Tyne. This was not the only time that bearing the Royal Banner proved a hazard as well as an honour, as several Chiefs of the Clan died in battle. At Flodden. the Clan Chief being too young to bear arms. his uncle, acting as Banner Bearer, died of wounds.

The Scrimgeours were for many years closely connected with Dundee; until the 17th Century they retained the title of 'Constable of Dundee and many Provosts of that City were Scrimgeours. The town residence, Dudhope Castle, was wrongfully taken from the lawful owners in the 17th Century (see below) and never recovered. They also had extensive lands in Argyll which justifies the Scrimgeour claim (approved by the Lyon Court) to be a 'Highland Clan'.

In their day a number of the Scrimgeour Chiefs were knighted, and in the reign of James VI the then Chief was created Viscount of Dudhope. His grandson, the third Viscount who had been a companion of Charles II during that King's 'exile' in Scotland, was created Earl of Dundee after the Restoration. On the death of this, the first Earl. it was necessary to go back many generations to find a male heir entitled to succeed and King Charles was persuaded by his favourite, the notorious Duke of Lauderdale, that the line was extinct, and the lands and honours associated with it should be granted to his brother Charles Maitland of Hatton. This the King agreed to, although there were two Scrimgeour claimants (one of whom was found more than two centuries later to have had a valid claim). The weakness of Hatton's position was such that he seized all the Scrimgeour documents from Dudhope Castle.