I think this varies greatly by situation.
I work in engineering and we are well stocked with toys - fidget or otherwise. I have Engineering Barbie, a beanie baby platypus, a stuffed beaver finger puppet, mind-game construction type blocks, and as I type this, my esteemed colleague is sitting on an urgent system issue conference bridge playing with a finger puppet. We both hold the title Senior Manager, so theoretically we're grown ups.
That said - this style would not have flown in many of my old offices. I always keep a certain few weird things about, and it seems to work with my personal brand - which includes being "the quirky one"... but that may not be the brand you want to pursue.
So, reasons to avoid bringing toys:
You are in a place with lots of customers or other external stakeholders physically present. While cognitive theory is nice, it's easy for it to look too casual to the uninformed observer.
The toy is annoying - fidget toy makers don't always consider how annoying loud clicking, or bright lights or fast moving parts may be to non-fidgeters. Even spinning a coin like a top or playing with a slinky can be unbearable to someone trying to focus.
You have a job with lots of engagement with others - especially people you don't know well. For sure, fidgeting is still cognitively useful even when you're talking to someone. But it can come off as being disengaged or disinterested. So, the less well you know someone, the less you should consider fidgeting to be a good option.
You have an office culture that thrives on being stodgy and prim and proper. Banking, big government, IBM - you could stick out like a sore thumb and that's not good if the company has a habit of amputating sore spots.
If none of this applies to you - give it a shot. Gauge your coworker's reactions, and react accordingly. If you don't like that they seem to be put off by your fidget toy, bring it home.