Yes, a star can be considered as a celestial body. By definition, celestial bodies is any object that exist outside the earth's atmosphere.
Stars are large balls of hot gases, they produce their own light and exist outside the earth's atmosphere, this make them celestial bodies. Galaxies which are parts of the celestial bodies are large collections of stars held together by gravity.