I am trying to understand the difference between a full fledged application server (e.g. Weblogic, JBoss etc.) and a servlet container (Tomcat, Jetty etc.).
How do they differ and when to use which?
A servlet-container supports only the servlet API (including JSP, JSTL).
An application server supports the whole JavaEE - EJB, JMS, CDI, JTA, the servlet API (including JSP, JSTL), etc.
It is possible to run most of the JavaEE technologies on a servlet-container, but you have to install a standalone implementation of the particular technology.