Thousands of angry public servants from one of the biggest Commonwealth departments will refuse to wear their uniform in protests over a pay dispute.

Centrelink, Medicare and Child Support staff from the Department of Human Services will start with low-level industrial action on Thursday over new staff agreements they say attack their pay and conditions.

Besides the uniform ban, the 15,000 workers will relay messages of concern to clients about changes to their working hours and leave cuts.

Their union has threatened further action, which could escalate into a major workplace dispute unless the government improves its offer.

But Employment Minister Eric Abetz says times are tough and pay increases would lead to job losses or increased government debt.

© AAP 2014