VectorLinux

Vectorbie Station => Vectorbie Questions => Topic started by: jduped on December 28, 2007, 12:38:59 pm

Title: Linux all-in-one server
Post by: jduped on December 28, 2007, 12:38:59 pm
I'm very new to the file sharing in Linux, and it took a while to just get win2k3 well integrated with my Linux boxes, now that I have that figured out, I'd like to go the other way...

have a Linux server setup, it would need to feed both linux and windows clients

The system I'm using to serve with is

amd 4200+
asus motherboard m2n-mx
1 gig ram
Using onboard raid controller, softraid 5 (4 drives sata II 320 gig)
o/s win2k3
additional hard drive IDE 200gig 1 partition runs o/s 2nd partition used for torrent/ftp/news groups/irc downloads

primary function
centralized downloading / uploading
centralized file shares / data back ups.

primary use
via remote desktop,
  -file downloads.
  -file organization.
  -data maintenance/restoration

all networked machines
  -use share points with specific user/pass access (some common shares + user specific shares)

(when we move to a house...where I can add proper power distribution, I would also like the machine to be a print server as well currently the circuit its plugged into is as maxed out as I'll allow it to be.)

What I wanted to do but haven't figured it out, is use a distro like ipcop, or openbsd or if there was one specific distro for these tasks but allowed more versatility then something like freeNAS. So I could have a router/data server/torrents.

Or create something like this product called a chili box (http://www.chilisystems.com/index.jsp) except it would encompass all my above info instead of just a single drive for network addressed storage.  also a web interface / remote desktop type of interface would be ideal.

Any ideas?
Title: Re: Linux all-in-one server
Post by: nightflier on December 28, 2007, 05:07:09 pm
What you're describing is much like my central file server. It has some shares that are read-write for everybody, some private read-write, as well as read only ones. In addition, it runs a local web server and a music server. Mine runs Debian. I find it very easy to set up and control remotely, using command line only. The nice thing about command line is that you can use tutorials and just copy/paste the commands.

A note about the RAID drives. Linux will see the drives as individual drives, not as one unit. You can do software raid, but if you have data on there now which is written using the Windows drivers, things will probably get complicated.

Are you planning on keeping W2K3 on the machine and dual boot?
Do you have a lot of data on those RAID'ed drives that you want to transfer to the new setup?
Title: Re: Linux all-in-one server
Post by: saulgoode on December 29, 2007, 12:59:55 am
I am not sure I understand your goals correctly but it might be easiest to have a dedicated firewall between your network and the Internet. This way, you can have a dedicated file/print server (running Vector) on your internal network which can handle a variety of networking protocols and services (including ones such as Samba and NFS which are generally not secure enough to expose to the Web).

My current setup has as a firewall an old 133MHz Pentium with 96Mb RAM running IPCOP. I have three network cards in this box: one for Internet, one for wireless access point, and one for my internal network. Isolating the wireless access permits an added layer of protection against intruders gaining access to my network while providing my laptops with Web access. The Pentium has no problem handling traffic loads though 96Mb is not enough to run SNORT without swapping and I only have a 100Mbps ethernet -- if you have 1000Mbps then you should consider a faster machine.

I have a centralized file server (running Slackware on a 200MHz PII) which also serves up some torrents (legal ones (http://www.publicdomaintorrents.com/)) using Rtorrent. I haven't put a printer on it but will probably do so at some point. My workstations' printers are all shared. Since your machine is such a high end unit (at least by my standards), you will want to enable X Window logins from your Linux workstations (and even from your Windows boxes, if you install Cygwin or such).


Title: Re: Linux all-in-one server
Post by: jduped on December 29, 2007, 02:30:51 am
First off legal torrents???  The only things I thought that fell under there were open source apps and what distros...I didn't realize this applied to media. Cool.

Currently I run two boxes 24/7

My router which happens to be IPcop, with a few add-ons, just a red and green interface.  I also wrote some nice rules in to my rc.local file for QoS, and to lag the heck out of anyone scanning me, plus some rules for stealth.  It works great, and requires very little maintenance.

My data server (the win2k3 box) its about half full 500gigs and I have a few drives I could slave in to dump it off temperarely.

When I booted the server in ubuntu back when I was first getting into linux. Ubuntu seen the drive and had it on my desktop looking mounted... When I'd try to open it however it gave me a read error, or no access.  I was using 6.10 at the time, so I'm guessing that ntfs read/write tool hadn't been integrated at that point.

For the time being I was just going to try and clean the disks up as there kind of fragmented from the mass deleted that happened a few days ago.  And was looking into something to do periodic back-ups to a usb drive, similar to ghost, but not those lame ghost images...I just want plain read able files that mirror entirely what is in the drives I'm watching.  Zero redunacy...like ghost just not ghost specific images.  My buddy who still works in the industry said to try out Acronis, I got a hacked copy...haven't tried it yet.

If I could accomplish my tasks in linux, believe me, I WOULD LOVE THAT.

As for the reasoning to have an All-in-one central server/gateway, I have wanted that forever, and in the windows world it can't be done...one is a gateway, one is a server, end of story.  I noted right away in linux that it could be done, and that why I want to, it frees up a box for experimenting, plus lowers my power consumption, and takes the stress off my ups.

I like where this topic is going, I just need more clarity...

If I wanted to accomplish this with vector could it be done and easily?

If I wanted to accomplish this with debian could it be done and easily?

if I use X windows on the server, what am I using to remote into the machine? vnc?  I'm not sure what Cygwin does...I went to the website and it seems unclear.

Another question...with the software in windows I can do upward migration on my raid, in other words, if I want to up my min drive size to 500gig, I can do this by simply adding one drive at a time, allowing the array to rebuild well its still live, and repeat till all the larger drives are migrated in...will this work in linux?  If no would a Hardware Raid Card do the job? I found one that isn't too much and says its supported but an outdated linux kernal.
Title: Re: Linux all-in-one server
Post by: nightflier on December 29, 2007, 05:56:35 am
You can use one machine for everything. It can be done using Vector, Debian, Slackware, Ubuntu or whatever. As for the easy part, it wouldn't be for me, but it seems like you already are doing advanced stuff if you are writing custom rules for your firewall.

VNC is good for remote control. Vector has a tightvnc package in the repo.

My concern about the RAID stems from experimenting with it a couple of years ago, using W2K. I set up a four disk RAID 0, expecting to see great performance gains, but I could not perceive any change in the system. Seemed to me that all I achieved was to quadruple my risk of data loss from a disk failure. When booting Linux, the four drives showed up as hde through hdh. Of course, I could not read anything on them, as the data was spread across all of them. Some research revealed that the so-called embedded RAID controller is little more than a glorified IDE/SATA controller, with the CPU and the software driver doing all the work. I think any gain from the striping was offset by the increased CPU load, at least on my setup. I still use it, but only for mirroring, accepting a performance hit in exchange for data protection.

A true hardware RAID controller is a whole different story, but those usually cost about $500 new. These have their own processors that do the striping/mirroring, and present themselves to the OS as a single drive. Most of these should work with current Linux kernels.

And yes, there is nothing illegal about bittorrent. Like most tools, it can be used in a variety of ways. Just because whiskey smugglers favored muscle cars that could outrun police cruisers, it didn't make muscle cars illegal. However, it is a lot more tempting to break the speed limit in a -69 Roadrunner with a tuned Hemi, than it is in my stock Civic.   ;D
Title: Re: Linux all-in-one server
Post by: rbistolfi on December 29, 2007, 07:04:49 am
Please correct me if I am wrong. A RAID system is not a replacement for a backup and not improves performance, since the same data is written in all the disks. It ensures a working system in disk failure scenarios. If one disk goes down, the others keep doing the job and the clients could never notice it. Looks like a must if you need the data available 24/7 with no exception at all.
Title: Re: Linux all-in-one server
Post by: easuter on December 29, 2007, 07:46:05 am
Well, that all depends on what RAID setup you have. Here is a list of different RAID setups:

http://en.wikipedia.org/wiki/RAID#Standard_levels
Title: Re: Linux all-in-one server
Post by: rbistolfi on December 29, 2007, 07:50:42 am
Well, that all depends on what RAID setup you have. Here is a list of different RAID setups:

http://en.wikipedia.org/wiki/RAID#Standard_levels


Thanks Easuter, that link is very informative. I had no idea about all the possible configurations.
Title: Re: Linux all-in-one server
Post by: jduped on December 29, 2007, 08:02:35 am
Well theres different levels of RAID, and different types.

RAID 0 - stripping, you get performance boost and faster read/write times, not secure.

RAID 1 - mirroring, This is a viable way to offer back-ups in my opinion as in that you have 2 identical copy's though you only get 50% of your combined drive space.

RAID 5 - uses distributed parity, so you get some speed out of it, you also have a secured data source, in that pending how many drives you have present you can loose a few before you in any danger of data loss, also you get higher percent of drive space utilization then RAID 1.

RAID 10 (or 0+1) - you get both stripping and mirroring. I've never run this one as you do loose drive space.  And it seems bulky with out a need.

In addition to the Raid 5 server, I have a usb external drive backing up critical info (or that was at least the plan though I haven't found a good automated program to back up and maintain the back ups on a weekly basis...I want something like norton ghost, except it doesn't use images, it just does Mirror + Reverse Incremental in plain non-proprietary file formats.  In that should I want to copy if over I could do it through thunar, or explorer, or what ever...

Title: Re: Linux all-in-one server
Post by: jduped on December 29, 2007, 08:05:10 am
and here I am just writing out everything like a sucker!
Title: Re: Linux all-in-one server
Post by: jduped on December 29, 2007, 08:09:32 am
speaking of great cars... my 93 Taurus rocks my socks off.

I was looking at getting a hardware raid instead of using the softraid...my issue is pretty simple, in windows my softraid functions and rebuilt fine on a simulated lost drive, it didn't panic, and fully loaded the drive after being placed back in (after I wiped the extracted drive first, of course)

Last time I checked Linux with my server box as I said it mounted the raid as one drive but couldn't read it.  before I get all worked up, I'm going to try the newest version of ubuntu server see if I get a better response...I can't do this for at least a day as a friend needs some repairs done to his box...and I committed to that.  Well seeing as I don't sleep much (not since I seen that flash in the sky on my birthday) :P I may get a chance to try it out today to just see if I can read/write the drive.

As far as the distro to make the all in one linux super network box.  I was leaning towards Debian as I liked it as a desktop all be it more sloppy, and slow then vector.

I know what your talking about with bit torrent...its just me personally I've never really used it for ... lets say completely kosher uses.  Only in the last year have I been using it and not breaking any "rules" if you catch my drift.

Back to the server...The issues that come into play with the switch, are one I'm going into that unknown kind of place, and I'm don't want to compromise the safety of my data (obviously I'll have it backed up off the server prior to attempting anything. 

Another problem is I don't know what will happen if I loose a drive in Linux will the array rebuild automatically or with minimal commands?

Would it be worth upgrading to a hardware raid card?

I researched a bit a few months back and found only one sata raid 5 capable expansion card compatable with a 32-bit pci slot.  The product is an Intel RAID Controller (http://www.memoryexpress.com/Products/PID-MX3430(ME).aspx) its for a 64-bit slot but says it is backward compatible to a 32-bit slot.  Though going from 66Mhz down to 33Mhz may not be too bad as the server isn't really under heavy load, I called the store and asked if anyone had more info than I already knew from my research, I got referred to Intel...which I haven't called yet, as I'm sure they'll give me the brush off as the store did.  My buddy has a hardware specialist he uses for his company (not for Linux though) he said he'd inquire as to how much of a performance read/write I'd loose using this kind of card in a lesser slot.

Ok those being the logistical issues.

How could I configure my distro to give me the best bang for my buck but not open it up to external attack?  Of the ones I could build this with which do you figure is the most locked down?


Title: Re: Linux all-in-one server
Post by: saulgoode on December 29, 2007, 08:30:55 am
My router which happens to be IPcop, with a few add-ons, just a red and green interface.  I also wrote some nice rules in to my rc.local file for QoS, and to lag the heck out of anyone scanning me, plus some rules for stealth.  It works great, and requires very little maintenance.

Should you wish to combine your server and firewall into one machine, your are going to have to either use IPCOP as your server's base Linux or to write your own firewall and traffic-shaping rules on another distro.

The advantage of the former is the browser-based administration (I am not aware of any packages which add a browser interface access to the configuration and logs, though I have not researched it lately). You would to some degree be limited with regard to the server software you could add (since IPCOP is based on 2.4 kernels); this should not be too much of a problem unless you intend to maintain NTFS partitions. Other than this, you should have little problem installing (or enabling) the necessary server software on your existing IPCOP machine (the GREEN interface should have all ports to the firewall open by default).

Using another distro as a firewall+server would place more of a burden upon you with regard to setting up and ensuring the firewall's integrity. Vector should be a good choice for this approach given the accessibility of its configuration tools from the command line (not to mention its Slackware roots). I have not done anything in this area but foresee no real difficulty if you proceed with caution.

Quote
As for the reasoning to have an All-in-one central server/gateway, I have wanted that forever, and in the windows world it can't be done...one is a gateway, one is a server, end of story.  I noted right away in linux that it could be done, and that why I want to, it frees up a box for experimenting, plus lowers my power consumption, and takes the stress off my ups.

Understood. My personal preference is for a separate firewall and for IPCOP in particular (where's the fun in buying a firewall appliance?). I suspect that my IPCOP Pentium costs me a few dollars a month in electricity but it cost me nothing so I equate that to a couple of year's worth of usage. I have been considering purchasing Mini ITX (http://www.newegg.com/Product/Product.aspx?Item=N82E16813185103) and installing IPCOP on it; which would lower power costs to a near insignificant amount while still providing the advantages of a dedicated firewall.

Quote
if I use X windows on the server, what am I using to remote into the machine? vnc?  I'm not sure what Cygwin does...I went to the website and it seems unclear.

If you are using X on your system, access from another machine on your internal network is as simple as changing the server IP when starting X (the details are dependent upon how you invoke your Xserver). The X Window System is inherently network transparent; i.e., it does not matter if your keyboard and display are on a different machine than the programs you are executing. This is effectively the same as having a KVM switch and being able to change machines by using the CTL-ALT-Fn keys -- with the advantage that the machines can be located anywhere and still be logged onto locally (my apologies if I am stating the obvious here).

If by "remote" access you mean running X across the Internet (e.g., connecting from work), even this is doable; but you should ensure authentication and encrypt your data (IPCOP's VPN/DMZ/Pinhole access permit this). I am not familiar with the VNC software but suspect it is basically a wrapper for this combined VPN/X11 functionality.

Cygwin is one way to provide your Windows system with X11 capabilities -- it turns your Windows machine into an X11 "dumb terminal". It is a rather bloated way of accomplishing this (it provides other capabilities as well) and there are likely better approaches currently available (I have not worked with Windows extensively in years).
Title: Re: Linux all-in-one server
Post by: jduped on December 29, 2007, 09:00:52 am
Quote
Should you wish to combine your server and firewall into one machine, your are going to have to either use IPCOP as your server's base Linux or to write your own firewall and traffic-shaping rules on another distro.

The advantage of the former is the browser-based administration (I am not aware of any packages which add a browser interface access to the configuration and logs, though I have not researched it lately). You would to some degree be limited with regard to the server software you could add (since IPCOP is based on 2.4 kernels); this should not be too much of a problem unless you intend to maintain NTFS partitions. Other than this, you should have little problem installing (or enabling) the necessary server software on your existing IPCOP machine (the GREEN interface should have all ports to the firewall open by default).

Using another distro as a firewall+server would place more of a burden upon you with regard to setting up and ensuring the firewall's integrity. Vector should be a good choice for this approach given the accessibility of its configuration tools from the command line (not to mention its Slackware roots). I have not done anything in this area but foresee no real difficulty if you proceed with caution.

I recall a co-worker using something that he just installed...he was using linux I can't recall what distro, I believe the tool was called webmin...its still around.  http://www.webmin.com/ (http://www.webmin.com/)  I have had an ipcop box around since late 2003, I just really love the idea of a fully functional all-in-one network appliance.  I think I'm most into it as it something unique to linux when in comparison to mac's or windows.

I would prefer have a locked down modified kernel...I haven't edited the kernel and I may need to add drivers as the nic's I use in all my desktops are dlink DGE-530T (the transfer speeds is double over other gigabit nic's with out any tweaking.  Last time I loaded my ipcop (when I upsized the machine it was using) I tried ti included one of these nic's on the green side, and it didn't have a driver for it.


Quote
Quote
As for the reasoning to have an All-in-one central server/gateway, I have wanted that forever, and in the windows world it can't be done...one is a gateway, one is a server, end of story.  I noted right away in linux that it could be done, and that why I want to, it frees up a box for experimenting, plus lowers my power consumption, and takes the stress off my ups.

Understood. My personal preference is for a separate firewall and for IPCOP in particular (where's the fun in buying a firewall appliance?). I suspect that my IPCOP Pentium costs me a few dollars a month in electricity but it cost me nothing so I equate that to a couple of year's worth of usage. I have been considering purchasing Mini ITX and installing IPCOP on it; which would lower power costs to a near insignificant amount while still providing the advantages of a dedicated firewall.

I think the reason power is a concern to me...is I live in alberta, and for some reason were getting jacked on power costs...something sick like 11.0/kwh  I work as an electrician, and I helped build an new power generator, 45 Megawatt gas powered one...basically a massive jet engine...and they have plans to build two more in the same area...and I'm like its just going to keep going higher...so what ever power I can save I'd like to attempt it.  maybe trying a mini ITX for the ipcop might be a better idea, do that, and then just leave it as is.

Quote
Quote
if I use X windows on the server, what am I using to remote into the machine? vnc?  I'm not sure what Cygwin does...I went to the website and it seems unclear.

If you are using X on your system, access from another machine on your internal network is as simple as changing the server IP when starting X (the details are dependent upon how you invoke your Xserver). The X Window System is inherently network transparent; i.e., it does not matter if your keyboard and display are on a different machine than the programs you are executing. This is effectively the same as having a KVM switch and being able to change machines by using the CTL-ALT-Fn keys -- with the advantage that the machines can be located anywhere and still be logged onto locally (my apologies if I am stating the obvious here).

If by "remote" access you mean running X across the Internet (e.g., connecting from work), even this is doable; but you should ensure authentication and encrypt your data (IPCOP's VPN/DMZ/Pinhole access permit this). I am not familiar with the VNC software but suspect it is basically a wrapper for this combined VPN/X11 functionality.

Cygwin is one way to provide your Windows system with X11 capabilities -- it turns your Windows machine into an X11 "dumb terminal". It is a rather bloated way of accomplishing this (it provides other capabilities as well) and there are likely better approaches currently available (I have not worked with Windows extensively in years).

I got lost on this, in linux, I connect to my 2k3 server via rdesktop.  I was looking for something in windows that I could connect to linux with in the same fashion.  I'm getting more and more use to the terminal but I feel a million times more comfortable in a graphical user interface.  Though I'm slowly being removed from that.  in my windows partion something stops working I'm instinctively seeking the terminal...

This is seeming like a huge project, I think I'll need to do some preparation.   :o
Title: Re: Linux all-in-one server
Post by: nightflier on December 29, 2007, 01:13:39 pm
Lots of questions and answers in this thread :)

To isolate one: VNC. It works much like rdesktop, and it is not OS dependent. Windows to Linux or the other way around all works the same. It will also run in a browser java applet, that's how I connect back to my server to get around those "websense" filters. It is easy to set up and use, and works very well.
Title: Re: Linux all-in-one server
Post by: nightflier on December 29, 2007, 01:25:02 pm
RAID: Most controllers have their own internal software for re-building an array after a drive failure. You can access it on boot before the operating system loads.

What is your reasoning for wanting a RAID setup?
Title: Re: Linux all-in-one server
Post by: nightflier on December 29, 2007, 01:28:17 pm
Backup: Use rsync to perform automated backups. This can be done from one drive to another, internal or external, or a network drive.
Title: Re: Linux all-in-one server
Post by: jduped on December 29, 2007, 02:23:17 pm
the reason I'm using a raid as opposed to one drive...if your using just one drive, and it craps out unless you can boot it up or access long enough to obtain your data.  I have personally lost 3 single drives for back up and its just one of those its over kind of deals...

I used Raid 1 for my back up purposes from 2003 till the summer of 2007.  I was looking around and for something that can serve files, and act safe place to store data.  I decided that a RAID 5 seemed to be able to fit the bill, plus I got more drive usage then a Raid 1, plus I could lose 1 drive and it would still be accessible. 

Do I have the wrong idea on RAIDs?  I just want a secure redundant place to save and serve my files.  Servers are a new thing to me as I haven't done a lot of data storage / data back ups.  Short of using ghost, and saving them to a central point, and then pulling the images back if need.

I have the hardware right now, As I said is there something better that I could be doing to achieve my goal of having a redundant Networked addressed storage system?
Title: Re: Linux all-in-one server
Post by: nightflier on December 29, 2007, 03:36:41 pm
My opinion is that mirroring (raid 1) is the way to go. It gives you the ability to lose one drive and continue. Also, this way, all your data is one each drive. They can be read using no special drivers or software, from a LiveCD or other OS. This kind of mirroring can be done totally by the OS itself, no hardware controller needed. If you have a raid 5, you depend on the raid controller being operative to get to your data.

Mirroring does not totally protect your data. In case of some mishap (like if you accidentally files ;) ), it is gone from both copies at once. That's where the suggestion of having another drive that gets synced like once a day comes in. If you go "oops", you can always "step back" to the point of last sync operation. This is easily done using a simple script that is called from crontab.
Title: Re: Linux all-in-one server
Post by: bigpaws on December 29, 2007, 06:29:16 pm
Well I just saw this thread. Lots of miss information here.

Linux can handle software raid the same as windows server.

Using any RAID configuration for even a thought of backup is dangerous

Ipcop, Freesco, Smoothwall and other firewall solutions have more done
than a modified kernel.

The kernel does not have alot of effect on the firewall designed OS.

Using a all in one solution is dangerous, data is then open. All that is
needed is to own that box and all bets are off. Using another machine
you might be able to stop an attack or at least create another level for
an attack against you. For every service (like X connections) allows one
more entry point to your server. Learn nmap, learn how attacks are done
and you then can become scared.

No system is secure and thoughts to the other wise is clueless. The
thoughts that there is nothing on my computer worth stealing. Well
your information is not usually important your box and internet connection
is. You an owned box YOU can be held liable for the spam or worse things
that are coming from it.

This subject is big it took me about 3 years to feel comfortable with it.

NONE of my servers have anything more than needed. NO X, nothing
that is needed. Slackware has about 4 gigs if a full install is done. My
server installs are 150 megs. Every piece of software that is not needed
is adding an attack vector.

Look at the open ports on 5.8 SOHO.

I hope my response allows a thorough understanding before just doing
your project. It has been done before but is your network worth the risk
of someone owning your data?

Bigpaws
Title: Re: Linux all-in-one server
Post by: jduped on December 29, 2007, 11:50:04 pm
Big paws I completely appreciate your feedback, and I know nothing is 100% the only thing that claims near perfection when it comes to this and acting as a layer 3 device is OpenBSD, I could never get it going in the way I wanted so I stuck to Ipcop.  I've played with nmap, and I use to be on the insecure.org mailing list.  I know a few guys that are really good at pen-testing and though they were unable to get into my network, they like myself know that a motivated person, or group could.

I was realizing today that yes my AiO server idea maybe a huge security risk waiting to happen.


That said, I'm abandoning that idea...moving towards something more data safe.

I'll keep my ipcop box for the time being as is, and keep the layer in place as I agree the more to go through will slow a motivated person and having it, is a good idea.  (you got my paranoid brain going again.)  My plan would be to switch the computer its using to something that consumes as little power as possible...this being a later project.

Now with the server, and it being a RAID 5, the reasoning behind picking a raid 5...I had a lot of conversations with people as to the most bang for your buck when using a RAID to back up and yes RAID 5 gives you more of your disk space as usable, (in comparison to RAID 1) though you still loose a percentage of space based on how many drives you have in the array. 

I as was said before I use to run a RAID 1 and felt relatively safe with a mirroring situation and in my opinion that would be the best for part of a back up plan, the other which I had setup and was using hence the reason I had a back up to go to when I gooned up my server the other day.  All be it an older copy of data.

What I'm looking for is 1 TB of secure storage (minimal 2 copies of any given data) protected as much as can be from drive failure.  Of whats stored with 250gigs being in that 'mission critical' kind of data.  Being stored periodically onto a slaved drive or something that is connected via usb.

My main goal for the back up system is fairly simple...

I throw the data onto a network share point, its available to users with correct credentials.
Things such as music rips/tv rips/dvd rips are available to all as read-only, with user credentials read/write access is granted.

As for the level of raid I think I'm going to try a raid 1 I have 4 drives all 320, do one array as raid 1 and the other as jbod...get me a 1 TB drive and drop it in a drive enclosure as a 3rd back up for the critical stuff and a second for the none.

I looked at rsync and looks promising I'm just not sure about how to set it up, all I want it something to read the drive that is to be backed up, note the changes, and sync the changes, only.  So you do one big back up, then a bi-weekly sync of only changed files. 

The end resulting image(s) left behind is 100% accurate copy of what the partion/folders had at that time the image of last sync, and could be put on to a new drive as simply as copy and paste.

Is this do-able?  Is there any changes to this plan that any one would suggest?
Title: Re: Linux all-in-one server
Post by: bigpaws on December 30, 2007, 04:35:07 am
The amount of data you are looking at is TB's?

For one way example the external backup rsync is
great, look at this:

http://www.leftmind.net/projects/rbackup/ (http://www.leftmind.net/projects/rbackup/)

For 2 way synch consider Unison for that.

Look at what you are trying to accomplish. A good thought
since disk space does not appear a problem. Use rsync
for once daily backup and Unison for another drive to keep
another backup file in sync.

Data ---> Uninson <backup drive> period of time
                      \
                         Backup Drive --> rsync to external backup drive


This will keep 2 copies of your information and create a better
failover.

I like having more than one backup and has saved the day many
times.
 
The solution needs to fit the need.

Bigpaws
Title: Re: Linux all-in-one server
Post by: jduped on December 30, 2007, 05:54:56 am
rsync will do the job as the way I mount my drives saves me the need of backing up to a back up...if you know what I mean.  I don't keep anything other then settings on my desktops.  I may play with unison, as then I'd have everything on the box directly...

I like your quote "The solution needs to fit the need."  Its like the one I use in my signature "can it do the job you desire, in the way you wish?"  I used that as my model for picking my linux os...ended up on vector.

I've decided to change up the way things are organized and running on my server, all the data recovery software I ran must of fowled up something because my read times, and lag is unbelievable...

so I'll be going from a RAID5 to 1 array being RAID 1 and the 2nd being jbod giving me 320 gigs of mirrored data and 640 gigs of drive space spanned over two disks.  I have a usb 320 gig drive that I'll be using rsync with to send data, and as I get complete series of things, I'll be dumping them off onto hard drives and storing them in electrostatic bags on my parts self so should the jbod array drop I can rebuild from back ups...I have tones of drives kicking around so it'll be fill then file, and if anything happens there good to go, I think the data break down ratio on a hard drive is considerably slower then that of optical burned media...as in the 7 years I've been storing my stuff on cd and dvd I've lost about 5% to corrupted copies, and 10% have seen data degradation.

I think as an os on the server I may switch to windows media center...or debian, I'm leaning towards debian, I just don't want to get stuck when it comes to setting up the samba shares...Though the purpose of this box is data storage, I can't ignore the advantages of the linux file system.  I'm just more comfortable admining on a windows box.
Title: Re: Linux all-in-one server
Post by: jduped on December 30, 2007, 06:01:09 am
heres a crazy idea I was pondering...could you format your drives as ext3 or reiserfs, all except that of your windows partition, and have windows read/write to them, and share them through the windows share system?
Title: Re: Linux all-in-one server
Post by: lagagnon on December 30, 2007, 07:31:53 am
heres a crazy idea I was pondering...could you format your drives as ext3 or reiserfs, all except that of your windows partition, and have windows read/write to them, and share them through the windows share system?
I don't believe Windows is capable of reading or writing to any filesystem other than its own (NTFS or VFAT). Why not instead, create a seperate VFAT or NTFS partition which you then use to share files between the Linux/WIn systems (because Linux can read/write to these). Or you can use Samba to share between the two systems....
Title: Re: Linux all-in-one server
Post by: jduped on December 30, 2007, 10:51:27 am
ok well that helps, thank you.  off to a new topic :)
Title: Re: Linux all-in-one server
Post by: nightflier on December 30, 2007, 12:24:25 pm
I have not used it, but there is a plugin for windows to let you read/write ext2/3:
http://www.fs-driver.org/
Title: Re: Linux all-in-one server
Post by: jduped on December 30, 2007, 01:35:45 pm
Awesome I'll I might try that prior to switching o/s's on the server box. :D