Yes, the co-workers are jealous. When you return, unless you bring really good swag, they will not want to hear one word about your trip. Spoil sports.
The best strategy is to pick out of the ordinary destinations. When I tell them where I am going and the response is " Where ??? What ? Where is that ? ", I know that I am on the right track.