Could not you just say to those few: 'you can't because I do not trust you'? You are the manager after all, your job is not to make them feel good but to make them work.
I don't think "some people on the team have privileges and others don't based on the manager's discretion" would be healthy in the long run either. Can you imagine interviewing for a team, asking about the PTO policy, and finding out that it varied like that? It would look pretty indistinguishable from "the people who that manager likes have special treatment" to me. You could hide it from prospective employees, but not knowing about it beforehand and then finding out from one of my teammates that the manager revoked their privileges (who presumably would have a chip on their shoulder about it and present the info with their own biases) would make me concerned that there was a bait-and-switch and now I'm stuck on a toxic team.
Yeah, I understand but on other hand you can't reward everyone with the same thing for different outcome. This is exactly what is happening with they pay, some people earns more, some less. People complain about it too. Do you think it is toxic too?
We people being people, and being manager when there is no outcome when everyone is happy, this is why I am not going to be manager. I just wanted to know honest opinion about how to solve it from the OP, or even if this is solvable.
A healthy company already needs to have processes for dealing with employees that aren't meeting expectations that don't involve revoking benefits like PTO. Those should be suitable for issues like this rather than crafting punishments specific to the nature of what specifically is going wrong.
An example of how a healthy company would deal with an employee who isn't meeting expectations? I honestly didn't think that needed an explanation, but okay:
The manager should be giving feedback to the employee, ideally as close to the moment when the expectations are not met (e.g. someone acting poorly in a meeting, take them aside after the meeting and explain what they did that was bad and what would have been a better way to ask). The manager should offer help or resources if appropriate. If the issue persists even after it's addressed a couple times, they should bring it up in their 1:1s with the employee and mention that it's been a recurring problem and try to understand why it's happening and see if there's a way to avoid the issue entirely. I've never been a manager, so I don't know exactly when and how this sort of thing needs to involve record-keeping with HR, but at the very least there should be some form of meeting notes already been kept for their conversations in their 1:1s with employees to track things like this (and plenty of other things; maybe the employee has given feedback to them about other teammates, non-teammate coworkers, or even the manager themself; all of this is important to keep track of for accountability purposes). The manager should make it clear that the issue persisting might affect their ability to remain employed, and if it continues to happen, termination for cause is a last resort.
It do not need an explanation, I am curious about your opinion about it. Since you are not OP of this thread it does not matter that much bu I am curious why he or she did not do just that instead of revoking remote work for everyone... does not makes, does it?
