No, you're not crazy. At least not in my opinion. I tend to agree with you about most of the table saw guards I've seen an used. They do serve a purpose in keeping your tie out of the saw if you're stupid enough to wear one while working the saw (don't laugh, I've got old manuals showing people dressed that way). They also keep your fingers and wood out if you're into waving them around over a running saw. But for much of the actual work, it's more a hinderance then a help, imo.

About kickback, it doesn't always happen straight back. Off hand, I'd say the biggest cause of kickback is the use of a fence. When you finish the cut, you've got a piece caught between the fence and the spinning blade. Let it jiggle, and bind, and it'll explode out of there in a rearward direction, but not necessarily straight back.

For the record, I've had ribs fractured by one kickback of a small piece of wood. Broke a windshield with another kickback, this time of plastic. Worked with a guy who nearly castrated himself from kickback of a metal plate.