For an average English speaker, the most difficult languages are Arabic, Chinese, Korean and Japanese. It takes a very long time for an average English speaker to learn any of these languages.
Arabic has very few words, written Arabic has very few vowels which makes difficult for people trying to read the language.
To learn Japanese one has to memorize thousands of characters, different writing systems and different syllabary systems. All of these add up to why the Japanese is hard.
Chinese has so many tones and thousands of characters, these make learning the language difficult for English speakers.
Korean has different sentence structure, syntax and verb conjugations this make it hard for average English speaker to learn the language.