Is that exactly what you put in the custom entry? If so, it is wrong. You do NOT include the closing </virtualhost> at the bottom. It should be:
Sentora will automatically add </virtualhost> at the end.
Code:
</VirtualHost>
<VirtualHost *:80>
ServerAdmin webmaster[at]yoursite.tld
DocumentRoot "/etc/sentora/panel/etc/apps/webmail/"
ServerName webmail.yoursite.tld
AddType application/x-httpd-php .php3 .php
<Directory />
php_admin_value open_basedir /etc/sentora/panel/etc/apps/webmail
Options FollowSymLinks Indexes
AllowOverride All
Order Allow,Deny
Allow from all
</Directory>