Hi Ardalan,
1. To limit the upload file extension, I suggest you set "AllowedFileExtensions".
<ValidateOption AllowedFileExtensions="jpg,gif,txt"/>
2. Please refer to http://www.ajaxuploader.com/document/scr/html/How-to-catch-uploader-error.htm, it shows you how to catch the uploader error then shows you custom error message.
Regards,
Ken