I can't remember if I already mentioned this somewhere, but i'm posting this to see if other users also experience the same problem - I tried it with Beta 2 and the Modern template aswell as my current development build and custom template.
Basically, if you originally made a reply (logged in) and the Comment appears nested (indented) when you have that enabled, go back and edit that nested/indented comment then save it - the nest/indent will disappear and it'll turn into a normal comment.
If someone can confirm it for me so I know its not something wierd with my server please do