.... It looks like x11 is the default window manager.
No, X11 is the entire GUI system, not the window manager/desktop environment. The desktop environment for 5.8 Standard is "xfce" (
http://xfce.org).
How do I go about installing jwm or pekwm
Once you have completed the install you go into Menu, System, GSLAPT, and choose jwm, install. But first of all you probably should read the VL Documentation (pre-loaded on your computer), especially the manual about our packaging system (it is also online here:
http://vectorlinux.osuosl.org/docs/vl58/manuals/vl5_slaptget_en.html )
... should I deselect optional/X11.tlz?
Definetely not unless you want to run that machine in command line mode only (no GUI).