Linux all-in-one server

[jduped]

I'm very new to the file sharing in Linux, and it took a while to just get win2k3 well integrated with my Linux boxes, now that I have that figured out, I'd like to go the other way...

have a Linux server setup, it would need to feed both linux and windows clients

The system I'm using to serve with is

amd 4200+
asus motherboard m2n-mx
1 gig ram
Using onboard raid controller, softraid 5 (4 drives sata II 320 gig)
o/s win2k3
additional hard drive IDE 200gig 1 partition runs o/s 2nd partition used for torrent/ftp/news groups/irc downloads

primary function
centralized downloading / uploading
centralized file shares / data back ups.

primary use
via remote desktop,
  -file downloads.
  -file organization.
  -data maintenance/restoration

all networked machines
  -use share points with specific user/pass access (some common shares + user specific shares)

(when we move to a house...where I can add proper power distribution, I would also like the machine to be a print server as well currently the circuit its plugged into is as maxed out as I'll allow it to be.)

What I wanted to do but haven't figured it out, is use a distro like ipcop, or openbsd or if there was one specific distro for these tasks but allowed more versatility then something like freeNAS. So I could have a router/data server/torrents.

Or create something like this product called a chili box except it would encompass all my above info instead of just a single drive for network addressed storage.  also a web interface / remote desktop type of interface would be ideal.

Any ideas?

[nightflier]

What you're describing is much like my central file server. It has some shares that are read-write for everybody, some private read-write, as well as read only ones. In addition, it runs a local web server and a music server. Mine runs Debian. I find it very easy to set up and control remotely, using command line only. The nice thing about command line is that you can use tutorials and just copy/paste the commands.

A note about the RAID drives. Linux will see the drives as individual drives, not as one unit. You can do software raid, but if you have data on there now which is written using the Windows drivers, things will probably get complicated.

Are you planning on keeping W2K3 on the machine and dual boot?
Do you have a lot of data on those RAID'ed drives that you want to transfer to the new setup?

[saulgoode]

I am not sure I understand your goals correctly but it might be easiest to have a dedicated firewall between your network and the Internet. This way, you can have a dedicated file/print server (running Vector) on your internal network which can handle a variety of networking protocols and services (including ones such as Samba and NFS which are generally not secure enough to expose to the Web).

My current setup has as a firewall an old 133MHz Pentium with 96Mb RAM running IPCOP. I have three network cards in this box: one for Internet, one for wireless access point, and one for my internal network. Isolating the wireless access permits an added layer of protection against intruders gaining access to my network while providing my laptops with Web access. The Pentium has no problem handling traffic loads though 96Mb is not enough to run SNORT without swapping and I only have a 100Mbps ethernet -- if you have 1000Mbps then you should consider a faster machine.

I have a centralized file server (running Slackware on a 200MHz PII) which also serves up some torrents (legal ones) using Rtorrent. I haven't put a printer on it but will probably do so at some point. My workstations' printers are all shared. Since your machine is such a high end unit (at least by my standards), you will want to enable X Window logins from your Linux workstations (and even from your Windows boxes, if you install Cygwin or such).

[jduped]

First off legal torrents???  The only things I thought that fell under there were open source apps and what distros...I didn't realize this applied to media. Cool.

Currently I run two boxes 24/7

My router which happens to be IPcop, with a few add-ons, just a red and green interface.  I also wrote some nice rules in to my rc.local file for QoS, and to lag the heck out of anyone scanning me, plus some rules for stealth.  It works great, and requires very little maintenance.

My data server (the win2k3 box) its about half full 500gigs and I have a few drives I could slave in to dump it off temperarely.

When I booted the server in ubuntu back when I was first getting into linux. Ubuntu seen the drive and had it on my desktop looking mounted... When I'd try to open it however it gave me a read error, or no access.  I was using 6.10 at the time, so I'm guessing that ntfs read/write tool hadn't been integrated at that point.

For the time being I was just going to try and clean the disks up as there kind of fragmented from the mass deleted that happened a few days ago.  And was looking into something to do periodic back-ups to a usb drive, similar to ghost, but not those lame ghost images...I just want plain read able files that mirror entirely what is in the drives I'm watching.  Zero redunacy...like ghost just not ghost specific images.  My buddy who still works in the industry said to try out Acronis, I got a hacked copy...haven't tried it yet.

If I could accomplish my tasks in linux, believe me, I WOULD LOVE THAT.

As for the reasoning to have an All-in-one central server/gateway, I have wanted that forever, and in the windows world it can't be done...one is a gateway, one is a server, end of story.  I noted right away in linux that it could be done, and that why I want to, it frees up a box for experimenting, plus lowers my power consumption, and takes the stress off my ups.

I like where this topic is going, I just need more clarity...

If I wanted to accomplish this with vector could it be done and easily?

If I wanted to accomplish this with debian could it be done and easily?

if I use X windows on the server, what am I using to remote into the machine? vnc?  I'm not sure what Cygwin does...I went to the website and it seems unclear.

Another question...with the software in windows I can do upward migration on my raid, in other words, if I want to up my min drive size to 500gig, I can do this by simply adding one drive at a time, allowing the array to rebuild well its still live, and repeat till all the larger drives are migrated in...will this work in linux?  If no would a Hardware Raid Card do the job? I found one that isn't too much and says its supported but an outdated linux kernal.

[nightflier]

You can use one machine for everything. It can be done using Vector, Debian, Slackware, Ubuntu or whatever. As for the easy part, it wouldn't be for me, but it seems like you already are doing advanced stuff if you are writing custom rules for your firewall.

VNC is good for remote control. Vector has a tightvnc package in the repo.

My concern about the RAID stems from experimenting with it a couple of years ago, using W2K. I set up a four disk RAID 0, expecting to see great performance gains, but I could not perceive any change in the system. Seemed to me that all I achieved was to quadruple my risk of data loss from a disk failure. When booting Linux, the four drives showed up as hde through hdh. Of course, I could not read anything on them, as the data was spread across all of them. Some research revealed that the so-called embedded RAID controller is little more than a glorified IDE/SATA controller, with the CPU and the software driver doing all the work. I think any gain from the striping was offset by the increased CPU load, at least on my setup. I still use it, but only for mirroring, accepting a performance hit in exchange for data protection.

A true hardware RAID controller is a whole different story, but those usually cost about $500 new. These have their own processors that do the striping/mirroring, and present themselves to the OS as a single drive. Most of these should work with current Linux kernels.

And yes, there is nothing illegal about bittorrent. Like most tools, it can be used in a variety of ways. Just because whiskey smugglers favored muscle cars that could outrun police cruisers, it didn't make muscle cars illegal. However, it is a lot more tempting to break the speed limit in a -69 Roadrunner with a tuned Hemi, than it is in my stock Civic.   

[rbistolfi]

Please correct me if I am wrong. A RAID system is not a replacement for a backup and not improves performance, since the same data is written in all the disks. It ensures a working system in disk failure scenarios. If one disk goes down, the others keep doing the job and the clients could never notice it. Looks like a must if you need the data available 24/7 with no exception at all.

[easuter]

Well, that all depends on what RAID setup you have. Here is a list of different RAID setups:

http://en.wikipedia.org/wiki/RAID#Standard_levels

[rbistolfi]

Well, that all depends on what RAID setup you have. Here is a list of different RAID setups:

http://en.wikipedia.org/wiki/RAID#Standard_levels

Thanks Easuter, that link is very informative. I had no idea about all the possible configurations.

[jduped]

Well theres different levels of RAID, and different types.

RAID 0 - stripping, you get performance boost and faster read/write times, not secure.

RAID 1 - mirroring, This is a viable way to offer back-ups in my opinion as in that you have 2 identical copy's though you only get 50% of your combined drive space.

RAID 5 - uses distributed parity, so you get some speed out of it, you also have a secured data source, in that pending how many drives you have present you can loose a few before you in any danger of data loss, also you get higher percent of drive space utilization then RAID 1.

RAID 10 (or 0+1) - you get both stripping and mirroring. I've never run this one as you do loose drive space.  And it seems bulky with out a need.

In addition to the Raid 5 server, I have a usb external drive backing up critical info (or that was at least the plan though I haven't found a good automated program to back up and maintain the back ups on a weekly basis...I want something like norton ghost, except it doesn't use images, it just does Mirror + Reverse Incremental in plain non-proprietary file formats.  In that should I want to copy if over I could do it through thunar, or explorer, or what ever...

[jduped]

and here I am just writing out everything like a sucker!

[jduped]

speaking of great cars... my 93 Taurus rocks my socks off.

I was looking at getting a hardware raid instead of using the softraid...my issue is pretty simple, in windows my softraid functions and rebuilt fine on a simulated lost drive, it didn't panic, and fully loaded the drive after being placed back in (after I wiped the extracted drive first, of course)

Last time I checked Linux with my server box as I said it mounted the raid as one drive but couldn't read it.  before I get all worked up, I'm going to try the newest version of ubuntu server see if I get a better response...I can't do this for at least a day as a friend needs some repairs done to his box...and I committed to that.  Well seeing as I don't sleep much (not since I seen that flash in the sky on my birthday) I may get a chance to try it out today to just see if I can read/write the drive.

As far as the distro to make the all in one linux super network box.  I was leaning towards Debian as I liked it as a desktop all be it more sloppy, and slow then vector.

I know what your talking about with bit torrent...its just me personally I've never really used it for ... lets say completely kosher uses.  Only in the last year have I been using it and not breaking any "rules" if you catch my drift.

Back to the server...The issues that come into play with the switch, are one I'm going into that unknown kind of place, and I'm don't want to compromise the safety of my data (obviously I'll have it backed up off the server prior to attempting anything. 

Another problem is I don't know what will happen if I loose a drive in Linux will the array rebuild automatically or with minimal commands?

Would it be worth upgrading to a hardware raid card?

I researched a bit a few months back and found only one sata raid 5 capable expansion card compatable with a 32-bit pci slot.  The product is an Intel RAID Controller its for a 64-bit slot but says it is backward compatible to a 32-bit slot.  Though going from 66Mhz down to 33Mhz may not be too bad as the server isn't really under heavy load, I called the store and asked if anyone had more info than I already knew from my research, I got referred to Intel...which I haven't called yet, as I'm sure they'll give me the brush off as the store did.  My buddy has a hardware specialist he uses for his company (not for Linux though) he said he'd inquire as to how much of a performance read/write I'd loose using this kind of card in a lesser slot.

Ok those being the logistical issues.

How could I configure my distro to give me the best bang for my buck but not open it up to external attack?  Of the ones I could build this with which do you figure is the most locked down?

[saulgoode]

My router which happens to be IPcop, with a few add-ons, just a red and green interface.  I also wrote some nice rules in to my rc.local file for QoS, and to lag the heck out of anyone scanning me, plus some rules for stealth.  It works great, and requires very little maintenance.

Should you wish to combine your server and firewall into one machine, your are going to have to either use IPCOP as your server's base Linux or to write your own firewall and traffic-shaping rules on another distro.

The advantage of the former is the browser-based administration (I am not aware of any packages which add a browser interface access to the configuration and logs, though I have not researched it lately). You would to some degree be limited with regard to the server software you could add (since IPCOP is based on 2.4 kernels); this should not be too much of a problem unless you intend to maintain NTFS partitions. Other than this, you should have little problem installing (or enabling) the necessary server software on your existing IPCOP machine (the GREEN interface should have all ports to the firewall open by default).

Using another distro as a firewall+server would place more of a burden upon you with regard to setting up and ensuring the firewall's integrity. Vector should be a good choice for this approach given the accessibility of its configuration tools from the command line (not to mention its Slackware roots). I have not done anything in this area but foresee no real difficulty if you proceed with caution.

As for the reasoning to have an All-in-one central server/gateway, I have wanted that forever, and in the windows world it can't be done...one is a gateway, one is a server, end of story.  I noted right away in linux that it could be done, and that why I want to, it frees up a box for experimenting, plus lowers my power consumption, and takes the stress off my ups.

Understood. My personal preference is for a separate firewall and for IPCOP in particular (where's the fun in buying a firewall appliance?). I suspect that my IPCOP Pentium costs me a few dollars a month in electricity but it cost me nothing so I equate that to a couple of year's worth of usage. I have been considering purchasing Mini ITX and installing IPCOP on it; which would lower power costs to a near insignificant amount while still providing the advantages of a dedicated firewall.

if I use X windows on the server, what am I using to remote into the machine? vnc?  I'm not sure what Cygwin does...I went to the website and it seems unclear.

If you are using X on your system, access from another machine on your internal network is as simple as changing the server IP when starting X (the details are dependent upon how you invoke your Xserver). The X Window System is inherently network transparent; i.e., it does not matter if your keyboard and display are on a different machine than the programs you are executing. This is effectively the same as having a KVM switch and being able to change machines by using the CTL-ALT-Fn keys -- with the advantage that the machines can be located anywhere and still be logged onto locally (my apologies if I am stating the obvious here).

If by "remote" access you mean running X across the Internet (e.g., connecting from work), even this is doable; but you should ensure authentication and encrypt your data (IPCOP's VPN/DMZ/Pinhole access permit this). I am not familiar with the VNC software but suspect it is basically a wrapper for this combined VPN/X11 functionality.

Cygwin is one way to provide your Windows system with X11 capabilities -- it turns your Windows machine into an X11 "dumb terminal". It is a rather bloated way of accomplishing this (it provides other capabilities as well) and there are likely better approaches currently available (I have not worked with Windows extensively in years).

[jduped]

Should you wish to combine your server and firewall into one machine, your are going to have to either use IPCOP as your server's base Linux or to write your own firewall and traffic-shaping rules on another distro.

The advantage of the former is the browser-based administration (I am not aware of any packages which add a browser interface access to the configuration and logs, though I have not researched it lately). You would to some degree be limited with regard to the server software you could add (since IPCOP is based on 2.4 kernels); this should not be too much of a problem unless you intend to maintain NTFS partitions. Other than this, you should have little problem installing (or enabling) the necessary server software on your existing IPCOP machine (the GREEN interface should have all ports to the firewall open by default).

Using another distro as a firewall+server would place more of a burden upon you with regard to setting up and ensuring the firewall's integrity. Vector should be a good choice for this approach given the accessibility of its configuration tools from the command line (not to mention its Slackware roots). I have not done anything in this area but foresee no real difficulty if you proceed with caution.

I recall a co-worker using something that he just installed...he was using linux I can't recall what distro, I believe the tool was called webmin...its still around.  http://www.webmin.com/  I have had an ipcop box around since late 2003, I just really love the idea of a fully functional all-in-one network appliance.  I think I'm most into it as it something unique to linux when in comparison to mac's or windows.

I would prefer have a locked down modified kernel...I haven't edited the kernel and I may need to add drivers as the nic's I use in all my desktops are dlink DGE-530T (the transfer speeds is double over other gigabit nic's with out any tweaking.  Last time I loaded my ipcop (when I upsized the machine it was using) I tried ti included one of these nic's on the green side, and it didn't have a driver for it.

Quote
As for the reasoning to have an All-in-one central server/gateway, I have wanted that forever, and in the windows world it can't be done...one is a gateway, one is a server, end of story.  I noted right away in linux that it could be done, and that why I want to, it frees up a box for experimenting, plus lowers my power consumption, and takes the stress off my ups.

Understood. My personal preference is for a separate firewall and for IPCOP in particular (where's the fun in buying a firewall appliance?). I suspect that my IPCOP Pentium costs me a few dollars a month in electricity but it cost me nothing so I equate that to a couple of year's worth of usage. I have been considering purchasing Mini ITX and installing IPCOP on it; which would lower power costs to a near insignificant amount while still providing the advantages of a dedicated firewall.

I think the reason power is a concern to me...is I live in alberta, and for some reason were getting jacked on power costs...something sick like 11.0/kwh  I work as an electrician, and I helped build an new power generator, 45 Megawatt gas powered one...basically a massive jet engine...and they have plans to build two more in the same area...and I'm like its just going to keep going higher...so what ever power I can save I'd like to attempt it.  maybe trying a mini ITX for the ipcop might be a better idea, do that, and then just leave it as is.

Quote
if I use X windows on the server, what am I using to remote into the machine? vnc?  I'm not sure what Cygwin does...I went to the website and it seems unclear.

If you are using X on your system, access from another machine on your internal network is as simple as changing the server IP when starting X (the details are dependent upon how you invoke your Xserver). The X Window System is inherently network transparent; i.e., it does not matter if your keyboard and display are on a different machine than the programs you are executing. This is effectively the same as having a KVM switch and being able to change machines by using the CTL-ALT-Fn keys -- with the advantage that the machines can be located anywhere and still be logged onto locally (my apologies if I am stating the obvious here).

If by "remote" access you mean running X across the Internet (e.g., connecting from work), even this is doable; but you should ensure authentication and encrypt your data (IPCOP's VPN/DMZ/Pinhole access permit this). I am not familiar with the VNC software but suspect it is basically a wrapper for this combined VPN/X11 functionality.

Cygwin is one way to provide your Windows system with X11 capabilities -- it turns your Windows machine into an X11 "dumb terminal". It is a rather bloated way of accomplishing this (it provides other capabilities as well) and there are likely better approaches currently available (I have not worked with Windows extensively in years).

I got lost on this, in linux, I connect to my 2k3 server via rdesktop.  I was looking for something in windows that I could connect to linux with in the same fashion.  I'm getting more and more use to the terminal but I feel a million times more comfortable in a graphical user interface.  Though I'm slowly being removed from that.  in my windows partion something stops working I'm instinctively seeking the terminal...

This is seeming like a huge project, I think I'll need to do some preparation.   

[nightflier]

Lots of questions and answers in this thread

To isolate one: VNC. It works much like rdesktop, and it is not OS dependent. Windows to Linux or the other way around all works the same. It will also run in a browser java applet, that's how I connect back to my server to get around those "websense" filters. It is easy to set up and use, and works very well.

[nightflier]

RAID: Most controllers have their own internal software for re-building an array after a drive failure. You can access it on boot before the operating system loads.

What is your reasoning for wanting a RAID setup?