I'm from the Netherlands, and I occasionally see the gesture when someone becomes a little agitated. When addressing someone, I habitually mention someone's name, pause a half-second while waiting for eye contact, then I address them directly. As the Dutch (and the British, in my experience) are not inclined to gesticulate, pointing is a distinguishing feature of other cultures that I tend to notice.
Your colleagues are probably just noticing a trait of yours. If you are assigning blame, ridiculing someone, or may cause someone embarrassment, then it is considered rude. The angry finger point is considered aggressive. In another context it will just be 'interesting' or 'different', and may thus lead to a little teasing.
At an early age, many Dutch people are taught not to point (at people). I now understand the convention to be contextual, and I exclusively point when I want to direct someone's attention to an object (or person) that is otherwise hard to notice. I never got into the habit of pointing in everyday conversation, or when addressing a person directly.
Why do I think I grew to be this way?
As a child, you learn through shared attention. When you are observing something in private, your parents may physically point at something that they want to draw your attention to: "Look at the cow, what does the cow say?"
My theory is that Dutch parents do not point if there is a risk that a bystander may think that they're the subject of their attention. As a child we are still learning about social conventions and so we point at everything we want our parents to notice. And many parents in my social circles are quick to correct the behavior if it is likely to cause a stranger embarrassment:
- Pointing at a dog in a field? Fine!
- Pointing at the toy you want? Fine!
- Happily pointing at a family member who walks into the garden? No correction!
- Pointing at the man with the funny hat? Rude!
- Pointing at your grandma, but now that stranger that is 30 meters away looks at papa as if we were pointing at her? Very embarrassing!
I think that inconsistent punishment conditions kids to try other solutions, so pointing as a casual gesture in conversation is discouraged. To me, pointing directly at someone in a conversation is associated with assigning blame, directed anger or ridicule - the few times I have seen someone use it in recent years, was when they were frustrated, angry or malicious. I don't often see someone use it casually. If you were to point resolutely at me when you assign me a task, I might need a second or two to realize that it's all in good fun.