Restrict Acces by Role bug?

I used the Members plugin to define two new user roles where I want role 1 to have some extra fields in the user profile (field 1 & field 2) and Role 2 to have some other fields (field 3 and 4). I then added two new field groups to the users:

Group 1
field 1
field 2

and

Group 2
field 3
field 4

and restricted the access for the group 1 to user role 1 and group 2 to user role 2.
If I log in as a user in Role 1 I still see the fields from Group 2 and vice versa. I.e. I see both groups with all fields.

If I on the other hand retrict the individual fields within Group 1 to Role 1 and the same for group 2 I now just see the group field label and not the fields. E.g. Log in as a Role 1 member and I see:
Group 1
field 1
field 2

Group 2

Bug? Or, am I doing something wrong?