Back in my dim and distant university days, I recall the professor in one of my linguistics classes discussing this.
It’s difficult to decide what language is “the hardest.” Hardest for whom? How many languages do you consider? Few people would be willing to say that Mandarin is “the hardest language in the world.”
More people would likely be willing to say Mandarin is one of the hardest languages with a large speaking population for adult learners who grew up speaking a Western atonal language. One of the things that makes Mandarin difficult is it’s tonal; if you’re an adult and you grew up only speaking an atonal language, then tonal languages are harder for you than atonal languages.
On the other hand, if you’re an adult who grew up speaking a tonal language, Mandarin is a lot less difficult.
Some languages are difficult not because the sounds are difficult, but because they divide the world into semantic chunks in ways quite different for Romance language speakers. Hopi, for example, is a direct-object-prioritizing language. Whereas romance languages prioritize the subject, and prioritizing the direct object is awkward (you can say “the store was run to by me” rather than “I ran to the store,” sure, but it’s awkward and clumsy and passive), Hopi prioritizes the direct object; “the store was run to by me” is the natural construct, and “I ran to the store” is awkward.
In fact, Hopi is so weird in the way it does semantic chunking we spent quite a lot of time on it in my psycholinguistics course.
But Hopi doesn’t have nearly as many speakers as Mandarin. Ask a linguist and she’ll probably tell you that a native English speaker will likely struggle with Hopi more than Mandarin, though neither language will be nearly as easy to learn for a native English speaker as, say, Spanish.