The default Manager group would take over a custom permission group, so I'd suggest first disabling this on the default Manager's group and then giving the permissions on a custom group for managers who will have permission to view their employees' salaries.
1.) Turn off the permissions in the Payroll category for all users in the Managers group.
2.) Create a new custom permission group and include all managers except the managers who should not see the salaries of their team:
3.) Select the managers who will be able to view salaries under "Group members".
4.) In People’s data > People > Payroll, toggle ON the permissions to view the Payroll category.
5.) In the custom group, on the "Access rights” section, select condition - Reports to indirect to Viewer. This will allow them to view salaries for direct and indirect reports. If you want to allow only for direct reports, the condition will be "Reports to is viewer".


