In general, from a smart contracts perspective, it is very expensive to do any individual tracking of the delegation pool. To the point where I am about 95% sure we should not implement it on L1 Ethereum. Meaning this conversation would almost certainly be ported into Delegation 2.0
With Delegation we have two upgrades planned, which we are currently calling Delegation 1.x and Delegation 2.0.
Delegation 2.0 , has good chances to be on an L2. Therefore I think this thread is great for discussion on how to improve delegation, but keep in mind implementation is probably far away.
To answer paynes question, the answer is that the 0.5% fee mainly exists to prevent the delegation front running. There could be lots of front running if you could just move around stake when inside the protocol in the current implementation. And it will be unlikely that the implementation can get changed enough in 1.x.
The broader question of your point - would it be technically possible to allow delegations to move around more freely? Yes, if we were able to record more data about them cheaply (i.e. use L2). I think your point is worth exploring for Delegation V2.