The battle of Portland has much to offer to the 17th Century naval wargamer. It was typical of the early battles that were fought before the line of battle was adopted.
At the battle of Portland, the English frigates were able to take a number of Dutch merchant ships, as well as some warships. The 1653 volume of the Hollandsche Mercurius, on pages 16 and 17 list the ships taken to Dover and Portsmouth after they were captured.
28 ships were taken to Portsmouth and 18 were taken to Dover.
Ships at Portland seem to have generally maneuvered individually.
I believe that the Battle of Portland was a running battle, especially after the first day.
On the first day, there were dismasted ships, including Michiel De Ruyter's ship, the Witte Lam. He had to be towed for the rest of the battle by Jan Duym, in the Zon.
As for communication, they called to each other, as ships were often within shouting distance. That was true of the Dutch, at least. They also would row to each other.
The larger English ships were pretty much immune to being sunk by Dutch ships with 12pdrs. That was particularly true of the 2nd and 3rd Rates. The smaller 4th Rates were probably light enough to be vulnerable. The only English ship sunk at Portland was the Dutch prize, the Sampson, 26 guns. The Sampson had been captured in July 1652, while on fishery protection duties. The Dutch prize, Oak, was taken back the the Dutch, but was recaptured at the same time as the Prosperous, a hired merchantman.
I would recommend two secondary sources, as they tell a good story:
Michael Baumber's book, General-at-Sea and Vol. I of Peter Padfield's book, Tide of Empires. I would forget Carl Ballhausen's book as there are too many errors.
A good, but obscure sources, is a book from 1654 by Jodocus Hondius: Onstelde-Zee, Oft Zee-Daden. There is a chapter or section on Portland, and it seems to have some information that you would not be able to find anywhere else. I don't own the book, but I have a poor photocopy that I have scanned.
I like www.british-civil-wars.co.uk, but the numbers it lists are often not accurate.
The convoy were Straatsvaarders as well as fluits, I would think.
Frank Fox thinks that the hired Dutch ships were largely Straatsvaarders and West Indiamen. I am sure that some of convoy were armed with more guns, while most were probably lightly armed. The Mediterranean was a dangerous place, so probably all had guns. I don't expect that they would have the large Retourschepen. They were reserved for the trip to Batavia, and parts East.
Low firing is at the "round timbers". The English sunk some Dutch ships doing that on the first day. High firing is mostly at the masts, sails, and rigging.
I'm not sure that there is a good description of how the boarding took place. I think that the Dutch ships would attempt to grapple, once they got close enough, where both ships were at low speed. I am not absolutely sure, but both sides included axes in their equipment. The English, in at least one case, repelled boarders with axes. Also, as a last resort, both sides would blow up decks, to enable them to overcome boarders. They also would use the quarterdeck and and forecastle as redoubts, to provide cover to fight off boarders.
I suspect that they would attempt to board when they felt like the enemy's fire was slackening, and they might have taken many casualties.