I'd be irritated the co worker didnt tell me, but also have a bit of a giggle at a whole lot of people dressed in semi formal clothing at a BBQ. like wth? who does that?
I'd try and have a good time regardless of what I was wearing. It really depends how well you get along with your co-worker, if you get along really well and they're fun people then I'd probably forget I was dressed differently, but anyone holding a semi formal bbq sounds like a jerk, so I'd leave early, LOL
anyway, to be honest i always dress on the more dressy side of casual - I usually dress up more than my friends and as long as I felt good in what I was wearing then I guess it wouldnt matter..