There is no way to do this. Even if you delete the tree, the people you have already invited will still have access.
The best you can probably do is exporting your tree, importing it as a new tree, and then deleting all of the people from the old tree.