Just say you find the speaker distracting. Generally, people won't inquire any further. Coming to work to do work is not out of the ordinary so they'll see the speaker as the problem before you.
You can also wear noise cancelling headphones rather than earplugs as if someone wants your attention they will make themselves visible rather than audible and this applies to those wearing all headphones so you won't stick out if that is your concern.
As a personal addition, I work at a tech startup and the guy across from me wears earplugs in an open office that's generally silent but for the sound of clicking keys and no-one has ever asked him why. People generally don't pry into that sort of thing in my experience, especially as there are many medical conditions that necessitate earplugs (tinnitus being an example I suffer from) and that is considered too personal for most workplaces.