All but one of my coworkers like hot drinks. 
<br>Teas, coffees, hot chocolates etc.

I'm not such a great fan of hot drinks so I never bother with them.

They often have rounds of drinks, where the 'hot-drink-drinkers' go down to the cafeteria downstairs to grab a coffee/ tea etc. and they take it in turns to pay for all of the drinks. The whole process takes like 5-10 minutes and happens at least 6-7 times a day. During this time, some of my coworkers seem to be using their phones etc. - as you would on a regular break.

During my first few months of employment, I used to walk with them downstairs but not purchase anything. This became kind of awkward and inconvenient so I began to remain at my desk when they would go grab a drink.

While they were downstairs, I felt like it was okay to have a break of my own. With not much else to do, I would take out my mobile phone and reply to messages, read the news etc. 

When my coworkers would return, I would put the phone away and resume working. I didn't use my phone at any other time.

I started doing this every tine they would go grab a drink, until it was noted in a recent one-to-one meeting with my manager that I was on my phone a lot. It came as a bit of a shock to me but I didn't question it and just apologised, saying I wasn't aware of this. 

Does this suggest that I shouldn't have a break when my coworkers go downstairs to grab a drink? It doesn't seem very fair of them to take regular breaks throughout the day (excluding the hour we have for lunch as well) where they can (almost) do what they want, check their phones,  reply to messages on a regular basis etc. while I am at my desk and cannot. I'm hesitant to use my phone at work at all for the sole reason that I might be ridiculed for taking a break.

Nonetheless, is it even appropriate to take a break while my coworkers go downstairs to grab a hot drink?