As we know that CustomAction can have UrlAction that points to an application page. In SharePoint 2007, whenever the CustomAction is clicked it simply navigates to the application page defined in UrlAction.
OpenPopUpPageWithTitle(url, callback, width, height,title)
The method parameters are self explained. Internally the OpenPopUpPageWithTitle method is calling SP.UI.ModalDialog.showModalDialog(options) where the options are the parameters passed to the method. To see how it works below is my ECB definition.
Basically it is an ECB for a custom list definition, on the UrlAction I use the OpenPopUpPageWithTitle method where the url is a custom application page, the callback is a SharePoint method called RefreshOnDialogClose (this method is defined in core.js, it will refresh the underlying page when the dialog close). Basically the callback can be anything such as display notifaction(SP.UI.Notify). The other parameters specified the width, height and title of the dialog box. Below is the screen shot of the my custom ECB.
Off-course, we can achieve the same result by using SP.UI.ModalDialog directly as shown in this post as OpenPopUpPageWithTitle method internally is using SP.UI.ModalDialog