It may be 'natural' but it's also wrong and a violation of our society's rules about privacy and boundaries.
If your own personal medical file were accidentally left on a cabinet, or knocked onto the floor of your doctor's office, how would you want his bookkeeper (your neighbor) to react when he happens to see it the following morning? Put it back on top of the appropriate cabinet right away, correct?
Most women wouldn't want him to casually open it up and peruse everything in there about their various abortions, STDs, anti-depressant prescriptions, etc, etc.

I just try to give others the same respect I woud want from them.