If you're very athletic and plan to develop, a high degree of paddling skill, there are strong arguments against using a rudder. A kayak without a rudder forces you to develop better skills, may allow better control in a crisis, rolls a lot better, saves weight and saves money. (The kayak also looks better, in my opinion.)

If you're a schmuck like me, the rudder is a Godsend. If we paddle for an hour with a mild quartering wind, the person without a rudder is constantly annoyed and wastes a lot of energy and aggrivation staying on course. It's just a lot more pleasant, paddling with a rudder.

(I paddle a Chesapeake 17', which is fine without a rudder, but I still like having one.)

If I were an athletic, accomplished paddler, I doubt I would use a rudder. But then, I would probably build a greenland kayak like CLC's North Bay. (See link.))