Which is better?
 

I can set up a server outside my Linksys router and use a sofwtare firewall like Norton or put the server behind the router and open the ports to experiment running my own FTP and Mail server.
Which would you recommend and why? :/

Considering you would be doing the same thing twice if you put the server outside the router, the logical thing would be put it behind the router. However, that only holds true if you are only accessing the server from the outside.

If you want to access the server from the LAN and the WAN, adding a firewall would be redundant and might hamper troubleshooting if the need arrises.