As I was meeting with my manager the other day, he asked me who in the department I looked up to, outside of my mentor. I named several people. Then he asked me if there is someone in the company I would rather not work with, or that I would be happy if they just left. Now admittedly I don't enjoy working with all my coworkers equally, but under that kind of pressure I honestly couldn't think of anyone I wanted to call out. So instead I told him that one person I really did not like working with had already left, which was true. He still seemed to want to know about someone who was still working at the company, though. I'm wondering if it is related to performance evaluations coming up. I wanted to be honest, but I also was hesitant to throw anyone under the bus even though some of my coworkers are less deserving than others. I was also afraid that by criticizing others I could make myself look bad. I have a good relationship with my manager and am confident it was not a trap, but it felt that way.
How can I answer a question like that truthfully but politely (and ethically!), and with a minimum of damage to other coworkers' careers?