You need to tell the viewer the right display number. Normally, display numbers come between 0 and 99. If you specify any number smaller than 99, the viewers add 5900 to get the port number. If you specify a larger number, the viewers take it as a port number directly. So how do you use port numbers lower than 99? You have to specify a negative display number! For example, to connect to a server running on port 80 on machine 'snoopy':
vncviewer snoopy:-5820
because -5820 + 5900 = 80. This may not work with all viewers, but Unix and Windows seem to be fine.