I find part of the solution.

For IE:
        editor.setFocus();
        var txt = editor.getSelection().htmlText;
        if (txt != "")
            editor.getSelection().pasteHTML("<tag>"+txt+"</tag>");
        else
            editor.getSelection().pasteHTML("<tag>");

For FireFox:
    var sel = editor.frame.contentWindow.getSelection();
    var txt = sel.getRangeAt(0);
    if (txt != "")
        editor.insertHTML("<tag>"+txt+"</tag>");
    else
        editor.insertHTML("<tag>");

I still have two problems:
1. When selection is empty, the editor insert also the closing tag ("</tag>"). But i need just the opening tag, so the following text will effect from the tag. (My tag is <div...> with some properties).
2. I need way to identify the browser, prefer by functionality, not HTTP Header.