The rules we are using are a heavily modified version of third edition RuneQuest (RQ), by Steve Perrin, Greg Stafford, Steve Henderson, and Lynn Willis, published by Avalon Hill in 1984. I chose this as the base, from which I made a number of modifications. I considered a variety of games as I was preparing the campaign, including a variant of Dungeons & Dragons, 3rd edition (aka "D20"); or a variant of Hero Wars. I chose RQ for a number of reasons:
To characterize what I would like combat to be like, I include a sample from the Laxdaela Saga describing the fight which resulted in the death of Kjartan. The following is from a public domain translation by Muriel Press (The Temple Classics, London, 1899). cf. the electronic edition edited, proofed, and prepared by Douglas B. Killings.
Kjartan at once sprung off his horse and turned upon the sons of Osvif. There stood near by a great stone, against which Kjartan ordered they should wait the onset (he and his). Before they met Kjartan flung his spear, and it struck through Thorolf's shield above the handle, so that therewith the shield was pressed against him, the spear piercing the shield and the arm above the elbow, where it sundered the main muscle, Thorolf dropping the shield, and his arm being of no avail to him through the day. Thereupon Kjartan drew his sword, but he held not the "King's-gift". The sons of Thorhalla went at Thorarin, for that was the task allotted to them. That outset was a hard one, for Thorarin was mightily strong, and it was hard to tell which would outlast the other. Osvif's sons and Gudlaug set on Kjartan, they being five together, and Kjartan and An but two. An warded himself valiantly, and would ever be going in front of Kjartan. Bolli stood aloof with Footbiter. Kjartan smote hard, but his sword was of little avail (and bent so), he often had to straighten it under his foot. In this attack both the sons of Osvif and An were wounded, but Kjartan had no wound as yet. Kjartan fought so swiftly and dauntlessly that Osvif's sons recoiled and turned to where An was. At that moment An fell, having fought for some time, with his inwards coming out. In this attack Kjartan cut off one leg of Gudlaug above the knee, and that hurt was enough to cause death. Then the four sons of Osvif made an onset on Kjartan, but he warded himself so bravely that in no way did he give them the chance of any advantage. Then spake Kjartan, "Kinsman Bolli, why did you leave home if you meant quietly to stand by? Now the choice lies before you, to help one side or the other, and try now how Footbiter will do."
Bolli made as if he did not hear. And when Ospak saw that they would no how bear Kjartan over, he egged on Bolli in every way, and said he surely would not wish that shame to follow after him, to have promised them his aid in this fight and not to grant it now. "Why, heavy enough in dealings with us was Kjartan then, when by none so big a deed as this we had offended him; but if Kjartan is now to get away from us, then for you, Bolli, as even for us, the way to exceeding hardships will be equally short."
Then Bolli drew Footbiter, and now turned upon Kjartan. Then Kjartan said to Bolli, "Surely thou art minded now, my kinsman, to do a dastard's deed; but oh, my kinsman, I am much more fain to take my death from you than to cause the same to you myself."
Then Kjartan flung away his weapons and would defend himself no longer; yet he was but slightly wounded, though very tired with fighting. Bolli gave no answer to Kjartan's words, but all the same he dealt him his death-wound. And straightway Bolli sat down under the shoulders of him, and Kjartan breathed his last in the lap of Bolli. Bolli rued at once his deed, and declared the manslaughter due to his hand.
I think this demonstrates fairly well how combat goes. It is not very cinematic and extremely bloody. The progress on the combat is marked by who receives what wounds: Thorolf's arm-crippling, the sons of Osvif and An wounds, An's disemboweling, and Gudlaug's fatal leg-severing. I would note that An survived his disemboweling and came to be known as "An Brushwood-Belly" from a dream he had while recovering. Note particularly the prominence of serious wounds on the limbs, a notable feature of RuneQuest which contrasts with how swordfighting is usually portrayed in modern books and films.
However, I had a number of differences with the rules from the start, and introduced further corrections over the course of the campaign. The third edition in particular is much more rules-heavy than previous RQ editions, or other implementation of the Basic Role-playing (BRP) mechanics. I discarded the vast majority of the specific rules over BRP. The more radical change was my use of Whimsy Cards -- which were introduced after roughly a dozen sessions into the campaign.
I considered two main alternatives to RuneQuest as the base ruleset for Vinland.
D&D3 is not a good fit in many ways to what I wanted. The core system highly emphasized larger-than-life fantasy as well as specialist PCs. The combat is abstract rather than graphic. However, since it has a lot of support including "open-source" rules text, I considered making a D20 variant which would be compatible with the hordes of D&D support available, but also providing the rules focus I wanted. However, after reflection it seemed like too much work.
Hero Wars, by Robin Laws, was the much-later successor to RuneQuest set in the world of Glorantha which RQ was originally developed for. Rather than the fixed utilitarian skills, it has freeform traits based on culture and occupation. It also has abstract conflict resolution rules rather than a combat system. Neither of these appealed to me for the setting. Hero Wars discusses its combat system by saying:
The Hero Wars rules model combat as it is almost invariably portrayed in popular fictional sources. You rarely see or read about the combatants in a fight delivering a succession of permanent wounds to each other until one of them finally keels over. Instead, the combatants jockey for a favorable position, ducking, dodging, knocking each other over, tossing each other around, and smashing up the furniture. With the exception of the final blow, when the combatants do make contact with each other, they generally deal out minor bruises and cuts.While I think this is accurate to pop fiction sources and movies, this description is totally unlike the combat of the Icelandic sagas. Saga combat is much more brutal and is much more like what HW contrasts itself with, as combatants deal permanent wounds to each other. I also wanted the system to illustrate subtle distinctions among the characters rather than qualitative ones. Most PCs would have all the same culture and occupation, and differ in subtleties rather than broad strokes. Hero Wars was also more heroic, in the sense that there was emphasis on mastery (the equivalent of skill over 100%).
The initial rules changes to RuneQuest were:
The power level is slightly higher than standard RQ3, representing characters who were heroic within the limited scope of Vinlanders (population in few thousands). In basic RQ3, characters may spread 80 points among characteristics, and get an average of 210 percentile points in skills (30 points for every year of age over 15). In my campaign, player characters had 86 points among characteristics and 450 points in skills (the equivalent of a 30 year-old in standard RQ3). On the other hand, RQ3 assumes some magic for most characters -- while in my system magic skills must be paid for separately out of skill points.
During the three-year course of the campaign, there were several more rules changes which I adopted. In order, these were:
Much play is relatively diceless, consisting of extended narration of what all the PCs are doing in different locations. This is interspersed with simple skill rolls. Over time we developed an understanding of the levels of success.
Combat is the notable exception -- which was rules-driven but still not overly complex. For example, a rulebook was never opened. This worked very well during the action-oriented times (i.e. when we had several sessions in a row documenting a summer of raiding). However, players would often get out of the habit of combat during fall, winter, and spring. So if combat occurred, it was often clumsy as they re-learned the rules. I have a few quibbles with combat even with my modifications, but I certainly wouldn't want to replace it with Amber-style combat. Historical saga combat should have a random feel, in my opinion, of hacked limbs and giving oneself over to fate. RQ is very visceral and mundane, and overall I thought it was excellent for this purpose.
My main wish would be that there should be a real mass combat system to handle the large battles where the PCs were among dozens or hundreds of combatants. My usual route was to ask each player to roll three attack rolls and three defense rolls. On the basis of these, I hand-waved what happened to them overall in the battle.