I am a PR currently living outside Canada. For most part of 2004 I was outside Canada. Now I am filing for IT return & my accountant suggests that I should be filing as a non resident. My concern is If I file declaring as a non resident will it affect my immigration status. I am under the 3 out of 5 years stay in Canada clause.

I dont think so as to you have already stayed in Canada for 3 years out of 5 year period and then left Canada , whereas the Residency Obligation says 2 years out of 5 year.
You should be fine !