Characters are for writing, not speaking. I have an acquaintance who is functionally illiterate in Chinese but, according to the Chinese people I’ve asked, he speaks a very fluent, very sophisticated Mandarin (don’t ask me how he acquired such a high level without being able to read or write!).
Apparently, he listens to a lot of lectures and audiobooks and watches a lot of documentaries in Chinese. He lived there for 10 years or so and was married to a Chinese woman for a while. He only writes pinyin and can’t read or write more than a few characters apoarently.