It seems that many instances of ghosts speaking to human beings takes place when the person least expects it. Anything like that would scare anyone. Usually, it seems that when a ghost is making noises or a being shows up, the person is waking up from a sleep. In a groggy state, anyone would be concerned and confused about what they were seeing if a ghost showed up. Then they would get scared since they would realize that this is not a person in their room, but a ghost.
For me, I would be scared if a ghost talked to me. The only way I would not be scared of seeing a ghost is if I was prepared for seeing a ghost.