This is harsh, but:
You need to grow up.
These tasks are not "bad", they are necessary but you simply don't want to do them. That's fine, except someone has to do them eventually, and you're part of a team. A team succeeds or fails together, and if those necessary tasks don't get done because everyone on the team selfishly acts like children and says "I'm not doing that because I don't want to", then the team fails.
Now if you're the only one who ends up doing these tasks, then you do have cause for complaining, because being on a team also means everyone should be sharing the good and bad things equally. If this is the case, you need to bring it up with your manager ASAP, because it indicates that your other team members aren't pulling their weight. It may also point to your team members taking on too much work; ideally every developer should have some slack time at the end of a sprint, exactly for handling these "backburner" tasks. (If you have no time left, you're either really good at planning or, more likely, you've overcommitted.)
You also need to reevaluate your assumption that refactoring legacy code is a negative thing. Failure to address technical debt is the number one killer of software products and teams, and the fact that you're actually getting time budgeted to address it is great. Refactoring old code is a great way to learn about aspects of the system you aren't familiar with, which leads to a better overall understanding of the system as a whole, which makes you a better developer.
In fact, if you are currently the only one handling these tasks, there's an opportunity for you to carve out a niche for yourself in the team as the guy who cleans up after everyone else. Being a "code janitor" may not sound glamorous, but if you're good at it your codebase will be healthier, your team will be more efficient, and together you'll all get more work delivered. Just make sure that your efforts are acknowledged - e.g. in sprint planning, "oh yes, this task involves subsystem which I reworked last sprint, so I estimate it will be story points less as a result" - because if you're a good janitor, it's often easy to vanish into the background.