Which Doctype are you using?
Well If you are using dotnetnuke 4.x and want to implement modal popup, first thing you should check is, which doctype your sking is using. If you don't know that, its really simple to know that. Just run your application in browser, and from View > View Source look at the very first line of browser source, it must be showing a doctype tag.
Implement in HTML4
Implement in XHTML
If you are using XHTML strict or transitional doctype then you can Ajax Control Toolkit's modal popup extender control. If you want to use ajax control toolkit controls in dotnetnuke, you have to learn how to do that, and this article can help you to quickly get familiar with that.
Implement using jQuery
Another way to implement modal popup in dotnetnuke for XHTML doctype is to use jQuery. You can easily get a modal popup plugin written in jQuery which you can use in dotnetnuke also. Here is an example to get it from google search.
If you are having problems referencing jQuery js and other js files, you can use ModulePath and SkinPath variables, to help browsers find right path in dotnetnuke application. If you are adding jQuery at module level and you have placed the jquery js and plugin js in module's directory, then used prefix like following to make sure browser will get the js properly.
and in same way if you using jquery at skin level then you can use SkinPath variable.
Hope you enjoyed the summary about creating a modal popup in dotnetnuke. Please write me your feedback if you need code along with this.
Happy coding :)