Category restriction based on user

I’m not sure if I’m just missing something, but I can’t seem to find a good way to limit the category in which a user can post.

Basically I have a blog with several “departments”. I want to restrict users to only be able to post in particular categories.

Using the Author/Contributor roles won’t really work, since I have several departments.

Thanks for any help!