Here is the answer.. I found that we can assign the primary user person with a different association first to move. in my case, i moved it to another parent first to move the primary. That allowed me to remove the unwanted boxes... Then i could reesablish the relationship. It was little convoluted way, but it worked.
Hope this helps. Thanks