Yes, Hajj is only performed by Muslim communities. It should be remembered that this is a pilgrimage to what is considered to be the holiest land for Muslims in Saudi Arabia. The country is so strict in implementing this that they do not allow non-Muslims to enter what they feel is sacred land.
Some people have tried their best to pretend that they are Muslims so that they can take part in the Hajj, but so far, not everyone is successful in doing this. This has been done, which means that it is not impossible but is there some thrill in doing something that has no spiritual significance to you?