My current kayak, which weighs 75 pounds is not that heavy once on the water...but if I had to make a lot of frequent turns, overcoming the extra inertia would quickly become a drag.

I've done comparison tests between my 75 pound Great Auk with other kayaks, seeing how far after we stopped paddling they would coast before stopping. I've never found any momentum advantage from the extra (excess?) weight.
In a race between friends, there's probably not much advantage or disadvantage from weight. At elite level there must be, probably greatest in the initial acceleration.

And why aren't skin boats really fast? Sure the coverings flex, but... well, why not find a solution?
Watching John Peterson move his beautiful SOF yaks around with such ease they seem a perfect design solution. Could a baidarka be faster if it had more chines?

Will