There was a bug which fooled our element when child elements inside the dialog fired iron-overlay-closed, this could be stuff like <paper-dropdown-menu> for example. This bug has been fixed now!
🐛 Fixed: fooled by iron-overlay-closed from child elements