I've just experienced a problem similar to this.
In my case the problem was that in IIS 6.0 there's a security issue for which there's an option disabled by default in its configuration that prevents APS scripts to use relative paths ("../dir/...") in functions like MapPath(); the dialogs use them.
All I've done was:
1. open the properties of the web site from the IIS control panel
2. switch to "Home directory" tab" then click "Configuration" button oin the bottom side
3. switch to "Options" tab and check "Enable parent paths" option.
In my case that solved the problem.
I hope it will work for you too.
--------------------
The solution above was for another problem.
I'm sorry but in these days I'm exprienced a lot of problems with the editor and I've to solve them all by myself and that caused a little bit of confusion in my mind.
I had that problem on a page with two editors declared with two instances in asp code.
In my case the problem was that in the first I setted up the CustomCulture property to "It-it" but in the second one I forgot to do that. After debugging the editor code I found that it use the CustomCulture value to build a security string and save it into a cookie.
When you try to open a dialog the editor check that string.
It happens that in the second editor works because it is the last one that set up the security string (and place it in the cookie) and so its security check works; the first one fails that check because, for instance, its security strings contains "it-IT" (default) but the cookie contains "en-US" setted up by the second.
The security strings is built on various properties of the editor; try to check that all are the same for both the editors.
Sorry for my bad english, I hope I was clear enough
Bye
Nicola