Well, I made that question to myself once, so I can tell you what I found (is not too much).
the only difference in distributions JUST the actual distribution of included software?
The quick answer is no. Several differences can appear. You probably know the gnu/linux systems are based on open source code. You need to compile it (a compiler takes the source code, and translate it to binary code, which is the "language" of your computer). A hole distribution is made of a lot (from a few hundred to several thousands, depends on the size of the distro) of packages and they can be compiled in many ways, including many options, and the most important, they need to work fine together. There is a lot of ways to do this and distros like Slackware, Debian or Gentoo did it in very different way.
Another point is the init scripts. You can take several ways to start the system and again, a lot of distros do it in many ways.
The software included and the services and daemons started can make a big diff. We can stay on this days...
Cant the additional programs be downloaded to any distribution? For instance, Fedora7 comes with a ton of stuff, compared to something small like VectorLinux. But, cant I pretty much make my VectorLinux exactly like Fedora 7 if I just download and install everything Fedora 7 comes with?
Yes, you can install a lot of packages on VL and if there is no package made, you can compile almost anything made for linux in VL. A few exeptions are tools made for special distros and other weird software, but generally speaking, the answer is yes. The difference between VL and Fedora will be still there tho... Shortly, they can do the same, but they do it in a different way.
Here is my small contrib, this is an interesting topic and I hope others could explain us more...

There is a lot of docs on internet if you like to read, a very nice project is
Linux from Scratch, I read their book because you can learn all the internals of linux from there.