For purposes of this answer, I'm going to interpret "unethically" as "contrary to your employer's policies", otherwise it gets into philosophical debate.
Without knowing where you work, nobody here can say whether your co-worker is right or wrong about expectations at your company. But it's quite possible they're right, because plenty of major employers do have policies along these lines. For instance:
Standards for New South Wales government employees, as set by NSW ICAC, under "grey areas and guiding principles":
Felix attends a work-related conference with the support and approval
of his manager. The focus of the conference is within his agency’s
purview and relevant to Felix’s project. He attends dinner on the
first day, which is included in the overall cost of the ticket.
Unbeknown to Felix, all dinner attendees are automatically entered
into a raffle. He wins third prize and receives two Gold Class movie
The gifts, benefits and hospitality policy at Felix’s agency states
that employees should not accept gifts of more than $25 in value in a
single instance. Combined, the tickets retail for just less than $90.
The tickets remain the property of Felix’s agency because he attended
the conference in an official capacity. He advises his manager, who in
consultation with management, determines the way in which the tickets
will be disposed.
Washington State government:
A state employee/officer may not keep a door prize if the state agency paid for the conference and the employee is attending...on state time. ... If you purchase a raffle ticket at a conference or other state event...it would not be considered a gift and you may keep the item.
US National Institutes of Health:
Example 1: At a conference which you attend as part of your official
duties, all attendees are automatically entered into a drawing. The
prize you win from that drawing belongs to the Government because your
attendance was in an official capacity.
Example 2: At a different conference open to the general public, there
is an opportunity to purchase raffle tickets to win a television.
Anyone attending can purchase a ticket. You buy a ticket and win. You
may keep the television because you personally purchased a raffle
ticket to be part of the drawing. Entry into the drawing was not
automatic based on your official attendance.
Government agencies tend to have the most detailed and publicly accessible policies on such things, but plenty of non-government policies have restrictions on accepting "gifts" which might cover such a situation:
Acceptance of individual gifts greater than US$75, or
multiple gifts in one year from the same Third Party totaling
greater than US$75, must be approved by your company’s
General Counsel. Gifts of cash or cash equivalents (such as gift cards, gift certificates or “red packets” commonly offered in Asia)
must never be accepted.
Never offer or give anyone, or accept from anyone, anything of value that is, or could be viewed as, a bribe or kickback or an attempt to influence that person’s or entity’s relationship with IBM...
Immediately return any gift of cash or cash equivalent that you receive from a third party and notify Legal that this has happened
The "why" has already been covered in other answers - tax implications, and the possibility that such prizes might induce employees to waste employer money on unnecessary trips or to favour the prize-giver in business decisions. In some cases even the perception of undue influence can be a problem regardless of whether the gift actually influenced the recipient.
As seen in these examples and already mentioned in Julie's answer, some organisations make distinctions based on the value and nature of the gift. A T-shirt or a small Lego set might well fall under the threshold for such policies. A $200 gift card (aka "cash equivalent") will not.
You might argue that you "paid" for your ticket via the time you spent visiting the sponsor booths, but I wouldn't lean heavily on that argument. If you were on the clock when you visited the booths, then you were already working for your employer and getting paid for your time - however you chose to spend it - in which case the raffle ticket is still a freebie to you and likely to fall under "gift".
If you weren't on the clock (and acknowledging that "on the clock" gets into grey areas when conference lunches etc. are also treated as opportunities for work-related networking), then there's a question of whether the time taken to visit all those booths was in proportion to the value of the benefit. If it's not a large number of booths, your accounting department might suspect that this requirement is just a ruse to get around corporate policies on raffles. Nobody gives away $200 without expecting something in return that they wouldn't otherwise have gotten.
You should check corporate policy, see whether it covers this case, and if not talk to Accounts or your manager.