Are you using KDE as your desktop environment? If so, there is a control center applet called kkbswitch that will allow you to switch keyboard layouts quickly. You still have to edit your xorg.conf file so that it has both English and Swedish keyboard layouts. However, kkbswitch will save you the time of going through VASM, etc., to change layouts.
Another option is to use xkeysw, though I am not familiar with it at all - just found it through a Google Linux search.