Ok, I tried fbdev, but that didn't work either. Also, after I tried it, using vasm I accidentally overwrote the xorg.conf-fbdev because xorg.conf linked to it (I was trying generic VGA, which doesn't work either).
Worse, I tried several different settings, so my xorg log file doesn't contain the fbdev error messages anymore. The jist of it was something to do with it not being able to find FBIO..something (useless, I know), but it didn't have a 'caught signal 11'.
Maybe I deleted some library at some point, because x was working straight after install. I could just reinstall, there isn't that much configuration I'd have to do again.