and how to load the content using ajax call to the popup
See, for example, how the checkout authentication modal dialog is implemented.
How is the mage.modal jQuery UI widget implemented and used?
mage.modal