TL;DR Talk to them or leave them (for lunch)
Is this an issue you feel comfortable discussing with your manager? What I am trying to hint at with this rhetorical question is that this topic seems rather negligible in a workplace setting. While there possibly are types of managers capable of empathizing with you about this it can easily come of as immature to raise this issue with them.
As was expressed by others in the comments, spoilers happening at the workplace are not that much different if they happen elsewhere. You are basically left with two options.
Option 1: Talk to them.
You recognized this option in your question and Joe Strazzere already gave an example in a comment:
"Please don't talk about Game of Thrones near me since I won't get to
watch it until tonight."
However, you have to acknowledge even taking this route you could end up not adhering to your personal TV schedule from time to time and being confronted with spoilers again. So you are only left with option #2.
Option 2: Leave them – for lunch, that is.
You have no authority one way or another to decide what your coworkers are discussing during their break. You can decide what you do, though.
After all, it comes down to one simple question:
Do you prefer discomforting yourself by not socializing with your colleagues and possibly missing out on other interessting discussions or do you really ascribe more value to not having spoilers for your favorite TV show?
Now, I painted option #2 a bit black here. You might as well have lunch break with different people altogether and maybe you even prefer that. Try it out. Do what is important to you. It's your break.