Collecting ideas, rollout VL on 30~80 machines

[Freston]

Hi all!

I haven't been here for a while, 'cuz I switched back to plain old Slackware. Why? Because I like a vanilla distro on my machine to play around with. VL is just to polished, to finished. I still have VL in dual boot though, and it still is my favorite distro after Slackware.

Now I remember a while ago in the Marketing section we where discussing how to bring VL to the masses. I there said I might be able to install VL on a number of discarded machines that my employer will sell to it's employees for a symbolical price. These machines have run XP, but are now killdisked so they are clean. Now you may have thought I wasn't serious back then but the moment is drawing near.




Anyway. There is no way I can install that number of machines manually. I have Linux Networking Cookbook by Carla Schroder and therein she describes hands free Linux installment over a network. Alas she depends on Fedora and Debian based specific tools. I'm looking for something more generic. Most target machines support network boot.

Steps to take:
Client:
I intend to install one machine with VL tuned to perfection. This basically means I install VL and add Dutch localization and OOo FF3.0.1. Then, using dd draw an image from that, and install that image over the network.

Server:
I have one machine already prepped with Slackware 12.1 (only installed A, AP and N packages). That's all I did at the moment.
It will need (probably) to become both a DHCP server and a FTP server.

Network
The network wont be the companies network, obviously. I'll make an ad-hoc network with the server hooked up to a switch.

Target machines
Most between 1.6 and 2Ghz machines, with ~512 RAM.
40~80GB hard drive (unpartitioned, just one long string of 0's)
Only 25% has a CDROM drive
roughly 20% has a bad 3.5' drive, the rest are ok
Most if not all of them are capable of booting of USB and network
VL runs perfectly on them 



What I still need:
Input from other people.
There are howto's on the net of course, but as I'm still preparing I don't have a very clear picture of what problems I will face. I'd like some human feedback.



I think the best way to approach this is either to create a boot disk [3.5' | CDROM | USB ] ; then
Insert the boot medium. This should then boot the machine to the point where it can 1) contact the DHCP server, 2) draw the image through FTP and 3) write it to the hard drive.
Or, with network boot, boot over the network to the point where it can do this without ever inserting a physical medium. I prefer this because many machines don't have CDROM.

The most beautiful thing would be that the whole process would take nothing more than a little fiddling with the BIOS to draw the correct boot image from the network and then do all this stuff fully automated.

Now I still have a lots of questions and some loose ends. But I'll leave it at this for now as my post is long enough 


---
Now I know my way around a Linux machine. I know how to set up a FTP server. I think I will be able to set up a DHCP server. But for all intends and purposes, I have never ever done anything remotely like what I described above. I have to rely mostly on guesswork while preparing for this all. Which is fun of course, I adopted this little project mainly because I think it's fun and promises results in Real World terms. I come here before the fact because I hope someone knows a little more about this than I do and is willing to share their knowledge. I understand if no one does or isn't 

Also, I understand that from a strategic perspective 30~80 machines isn't a structural widening of VL's user base. So you may not feel it's worth the effort. But on a fun sidenote (we need a little bit of fun in a long post like this)... one well known OS vendor was literally screaming on their website that they rescued (sic!) a school (180 machines and a couple of servers) in my home town from Linux. But failed to mention that that same year the municipality of that city (3000+ machines) had moved away from them and switched to Linux. That's marketing!

Pray bear with me, this is a non-commercial initiative from my side. The company I work for is a non-profit organization in the health care sector. The employees buying the machines will mostly be tremendously underpayed health care workers. It is either this or nothing.

[M0E-lnx]

What about partitioning?
I've never done disk imaging myself, so I'm really asking... when you do an image, will the partition table roll out with it? or do you need to pre-partition everythng to match your image?

[bigpaws]

This project sounds like fun. In the first hand getting access
to lots of hardware, then tearing your hair out accomplishing
the goal.

However in the end the satisfaction of learning (at least for )
is the real reward.

Have a look a clonezilla, which does network cloning. Setup
a small drive as the source and then let it go.

HTH

Bigpaws

[Freston]

What about partitioning?
I've never done disk imaging myself, so I'm really asking... when you do an image, will the partition table roll out with it? or do you need to pre-partition everythng to match your image?

Now there's a good question!
I was able to turn 1.44MB images into bootable 3.5' disks on a FreeBSD machine using nothing but dd. But when I tried the same with a USB stick it failed and did nothing with the partitioning table. I have to look into that, also to see how it behaves with variable disk sizes. But I had the rough idea that I'd use fdisk --all_necessary_options and/or mkreiserfs for the non-interactive building of a file system on the target machine. Better stuck with it then stuck without it, if I may try my hand at translating a Dutch expression.

I have this idea that the target machine boots up using a boot image transported over a network, and that this will end in running the code that writes the main image to the disk. If this idea is correct, then in the code part there will be room for custom commands. I hope I can wiggle building a file system in there somewhere.

This project sounds like fun. In the first hand getting access
to lots of hardware, then tearing your hair out accomplishing
the goal.

However in the end the satisfaction of learning (at least for )
is the real reward.

Yeah! That's the thought!
It's always more fun learning stuff that is 1) much to complicated and 2) will end up with something useful.

I wish I had a picture to post of the pile of machines. It's way over a cubic meter of hardware. It's an entire wall, maybe 2.3 meter high and 1.8 meter wide behind a 1.8 meter by 1.8 meter wall. And it's still growing 

Have a look a clonezilla, which does network cloning. Setup
a small drive as the source and then let it go.

http://www.clonezilla.org/ looks promising. Thanks!


I was afraid that prepping the target machines would be the bottleneck in the operation. Clonezilla also needs some prepping of the target machines, but I may find a way around that, as it has to do with recognizing individual machines. This is a dedicated network where each machine is part of only once. DHCP will be enough to distinguish between individual machines. I'll look into it. On the other hand, it seems that Clonezilla can do as much as 40 machines in one pass. I can get my hands on a 16 or 24 port switch, so there'll be no capacity problems.

[hata_ph]

I use BartPE with norton ghost to network clone all the windows PC at my office. If norton ghost can clone linux OS (I never try it before) then it should work as my windows clients.

1. Setup a file server to store the GHO image.
2. Boot with the BartPE, map the network folder that contain the GHO file.
3. Restore the GHO file.

http://service1.symantec.com/SUPPORT/ghost.nsf/e57b04140f9db1e785256b200056c22a/6a5df3b25c59eb9b8825671d00613c5b?OpenDocument

[newt]

Just a thought here on a one small step within the entire process: My inkling would be to create your partitions (/, /home, swap) at the minimum size possible based upon your smalled hard drive size, and after all workstations are installed rollout an automated script to resize /home to fill in the remaining hard drive space.  I believe this will require /home to be the last partiton on the drive.

For example, given a smallest hard drive of 15 gb: / = 8gb primary (hda1),  swap = 516mb logical (hda5), /home = 6.5gb logical (hda6).  After all systems are installed, run an automated network script to expand hda6 (/home) to remaining available space.  Something along those lines.

Again, this is just a thought on one small step of your journey.  Sounds like fun and if you can figure this out then it seems it would be a great HOWTO since VL was "not designed to be rolled-out across a network" as you're proposing (at least, that's what I recall reading in various posts throughout the past).

Good luck!!

[Freston]

Thanks for your responses!

It's going pretty well. Or at least, it was. Setting up DHCP and TFTP with Dnsmasq was very easy. Setting up a PXE environment was a bit more complex, but once I started to understand the logic behind it I was able to trial&error my way into a network boot server. Right now the server has a boot loader and a complete file system to push over the network with TFTP and NFS respectively.

I also found the BIOS of my main machine supports network boot, so I was able to test my configurations. Right now I'm at the point where the client gets an IP from the server and then the server pushes a kernel over the network to the client. The client starts to load this kernel and all is well until the client crashes 'kernel panic, not syncing'.

What happens? The client is told to have /dev/nfs as root. But the client doesn't agree with that and stalls. After trying several hacks around this that I found (none worked) I have come to understand that it is a kernel compilation time option to enable or disable root=/dev/nfs; which is simple syntax telling the kernel to take the file system it's offered over NFS as it's /

So recompiling the kernel seems the only option at this point. But for the life of me I can't find anything remotely similar to 'nfsroot' or 'allow nfs as root file system' or something like that in my `make menuconfig`. I decided to give it a rest and try again fresh after work today.

Sounds like fun and if you can figure this out then it seems it would be a great HOWTO since VL was "not designed to be rolled-out across a network" as you're proposing (at least, that's what I recall reading in various posts throughout the past).

Well I keep notes, so writing a howto is no problem. There are a lot of howto's on the matter all ready, but I found non describe this exact setup. So I had to take a little bit of this, a little bit of that, and add some of my own recipe. When I've passed the main hurdle atm, the kernel, then the fun stuff begins. Writing the script that partitions, builds the file system and writes the image to disk.



EDIT: I figured it out, the kernel options that is. I needed kernel autoconfig enabled from the networking section. And then NFS filesystem support needed to be promoted from module to build-in et voila! Done

[Freston]

I'm still experimenting and designing. It's difficult, because I have to make a couple of assumptions.

Assumption #1
I can build my own custom initrd for the clients to load. Well, I can, but I ran into some problems adding 'more advanced' networking options to this initrd. Somehow this made the initrd loose the ability to detect and manipulate the clients hardware. It seems I broke something, but I don't know what. Yet. Unresolved.

Assumption #2
The initrd will put the clients in a 'listening state'. The clients will receive command to partition their HD (~8GB root, 500MB swap, rest=/home), this will probably be done through SSH. Depends on #1.

Assumption #3
The clients will fire up a netcat pipe into dd listening for inbound data streams. Now I need a way to have one outgoing data stream being received by multiple clients simultaneously. I don't know if this is possible. The alternative is to target each client individually, which means all outgoing data has to be send multiple times, putting a lot of load on the ad hoc network. Unknown.

Assumption #4
The process is reliable. Untested.

Assumption #5
The whole process is forgiving for minor variations in hardware. Untested.


I'm tackling these 'problems' one at a time. But there is a lot of reading involved... and testing 
And there is always the design of the interface.



---


As you see, I'm being stubborn and use a home brew system rather than any of the tools you provided me. That's because it's more fun this way. It's a personal learning project as much as anything else. I kind of like having to go into corners of the system where I normally have no business. And dissecting functionality normally taken for granted, such as the initrd. I never wondered what that was, and all of a sudden everything falls or stands with my ability to create one from scratch 

[Freston]

UPDATE: progress! 

Assumption #1
I can build my own custom initrd for the clients to load. Well, I can, but I ran into some problems adding 'more advanced' networking options to this initrd. Somehow this made the initrd loose the ability to detect and manipulate the clients hardware. It seems I broke something, but I don't know what. Yet. Unresolved.

Started from scratch. Got it right the first time. Dunno why. Resolved. Tested. Confirmed.

Assumption #2
The initrd will put the clients in a 'listening state'. The clients will receive command to partition their HD (~8GB root, 500MB swap, rest=/home), this will probably be done through SSH. Depends on #1.

Jep. Bin there, done that. Just not through SSH. There's no end to what you can achieve locally with a proper initrd.

Assumption #3
The clients will fire up a netcat pipe into dd listening for inbound data streams. Now I need a way to have one outgoing data stream being received by multiple clients simultaneously. I don't know if this is possible. The alternative is to target each client individually, which means all outgoing data has to be send multiple times, putting a lot of load on the ad hoc network. Unknown.

This is the one causing headaches. As long as I target one system at a time there's nothing wrong. I can adjust to target multiple systems individually, and I hope, think, know that will go right. It's just slower.

Assumption #4
The process is reliable. Untested.

For one machine, the procedure is tested. Reliable I think. I got some errors, but they didn't seem to have any ill effects.

Assumption #5
The whole process is forgiving for minor variations in hardware. Untested.

I found a way to cope with different sizes of hard drives. Differences in soundcard and video will be a problem. Configuring this manually wont be a problem though. It's only a handful of machines to which this applies.

----


Right now all steps of building file system and pushing the image on that are manual. I'm running a couple of tests on the target system before I start automating the whole thing. I'm keeping notes on this, so I can write a howto about mass rollout of images if anyone is interested.

The whole thing isn't even that difficult, it's just that some of us (like me) are not used to -for example- build a partitioning scheme non-interactively with sfdisk. Or pipe dd into nc to reach remote machines.