Here is more information on recreating the problem:

 

IIS custom 404 needs to be set to URL - NOT FILE. This is an option of the custom 404 setting. Choose URL and then type in name of 404 page (\404.asp). Use a simple html page as the 404.asp

 

After pointing to the correct type of error page, CLEAR IE CACHE and then try to use editor. Like I said, about 1/3 of the buttons throw a syntax error. Try "Insert Hyperlink".

 

Ditto on what rwhawkes said - I'd be happy to help more or show you my site with errors or whatever. What I know is that I have been trying to get this worked out for a MONTH and I need to find a solution. Maybe going back a version??