Only because companies are trying to do this human verification on the cheap. SIM swapping-alike attacks aren't a problem with institutions like banks where they keep ID on file and you can visit in person to prove your identity.