Wear the tie.
If the environment is such that it is inappropriate, you can always take it off. Indeed, the act of taking it off can even signal that you feel relaxed, and non-verbally cues others to this fact.
If you take the tie, you can remove it. If you don't take the tie, it looks silly to be putting it on if you feel you need it.
On sales calls, even if I suspect the environment is casual, I always aim for one notch above. You never know when you will need it, and you can always kick it back. It is harder to kick it up...