Looking through the surface of the water, whether it's water to air, or water to plastic or glass, is effectively the same. Which ever side is lighter. If you're under water, where it's darker you can look up through the surface (but you can't look to the side at much of an angle because the refraction of the water will reflect off the surface no matter what difference in light.

We've all seen the clear poly carbonate boat (with the girl in the bikini in it). It's usually shot from a very high angle, in very clear tropical water. The high angle is why it looks like you'd be able to see the fish. But this clear boat would only be clear in very clear water, and only from a high angle.

A viewing box works because the window at the bottom reflects the dark interior of the box. Your face in the opening cuts most of the daylight from getting in, making it very effective.
So a viewing box might be a fun easy to make solution. The issue with a kayak is leaning over the side to look. Maybe a little ballast on the bottom, a float collar (pool noodle?) wrapped around the outside, and a way to hook it onto the coaming to make it work like an outrigger. Be really neat if there was some wide angle lens, or maybe a convex mirror so you could see more than a narrow angle straight down.

What I've done to peek under the waves, is hold my waterproof Kodak Sport over the side and under and shoot some video. Whenever I've done this where sea lions are hanging out, I've always captured one or two swimming below me looking up.