Help me with scripting this...

[Dweeberkitty]

Hey!

I'm planning on really working on the multimedia disc this next week. I'm going to have some days off. I've got all my scripting stuff done but I want to tie it all together. Here's the problem: I need to be able to (re)compile all of my programs with one script. But first, I need to know the order that they should be compiled. That's where I need help. Let's say that I've already compiled every program individually and have the slack-required file there. What I think needs to happen is this:

1. Search for all the slack-required files (They will be named like the name of the program i.e. ardour-slack-required)

2. Go through the dependencies of those files one by one and check and see if I have a $NAME.SlackBuild available. If there is one, then put that the name of the dependency above the name of the program who's slack-required file we are searching.

3. Output it all to a file

Basically I want to be able to run this script and it would search through all the slack-required files, ordering the programs that I have by dependency and outputs it to a file.

Here's an example:
Say I have just sox and libsndfile and libmad and libA in my compiling folder. The script would read the sox slack-required and see that libsndfile and libmad are both in there. So, it would put them before sox. The resulting "order" file would look like this:

libsndfile
libmad
sox

Now say that it reads the libmad slack-required file and finds that it depends on libA. The script searches my compileing folder and finds that the libA.SlackBuild is there. It would then put libA before libmad:

libsndfile
libA
libmad
sox

Does this make sense? Could I get some help with the technical details of writing this script? Is there a better way of resolving all these dependencies and compiling them all? Thanks in advance!

[Triarius Fidelis]

1. Search for all the slack-required files (They will be named like the name of the program i.e. ardour-slack-required)

Presume that all slack-required files are somewhere under the current directory, or take a directory on the cli?

2. Go through the dependencies of those files one by one and check and see if I have a $NAME.SlackBuild available. If there is one, then put that the name of the dependency above the name of the program who's slack-required file we are searching.

Where would I search for the SlackBuild files? Is case sensitivity important? (I've seen .slackbuild also.)

What happens if I don't find a SlackBuild?

Does this make sense? Could I get some help with the technical details of writing this script? Is there a better way of resolving all these dependencies and compiling them all? Thanks in advance!

My idea would be to have an array of dependency tiers and a tabela associativa (rbistolfi would know ) indicating the tier level for each dependency. Assume that there is only one tier to begin with. Every program added without any dependencies is on tier 0 and recorded as such. Then dependencies come along for programs in tier 0 and are added at tier 1 below. If it so happens that something on tier 0 is required by something, a new tier 0 is formed and the old one gets shifted down with entries in the associative array updated accordingly; if something on tier n has dependencies, they are recorded on tier n+1, etc...

A crappy algorithm indeed, but maybe I'll be able to think of something better when I complete the next semester.

(I assume it wouldn't be necessary to detect cyclic dependencies because your packages aren't RPM after all. )

[rbistolfi]

My idea would be to have an array of dependency tiers and a tabela associativa (rbistolfi would know )

http://www.lua.org/portugues.html

[Dweeberkitty]

Hanumizzle,

Take a look at my vectorcontrib/Dweeberkitty/BonusDisc/. You will see what I'm talking about. I have an organized set of folders with a standard naming scheme across all of them. Also, the slackbuilds are all named "NAME.SlackBuild" there aren't any that are lower case. Here is what folders and files I have:

NAME/
NAME/VERSION
NAME/VERSION/slack-desc
NAME/VERSION/slack-required
NAME/VERSION/NAME-VERSION-ARCH-BUILD.tlz
NAME/VERSION/src/NAME.SlackBuild

Every single package that I have built follows this standard and it is uniform enough that the algorithm used does not have to be written in a broad manner to account for several different scenarios but can be specific for this standard.

If you look in my BonusDisc/Configurations/ORDER file, you'll see why I want to be able to automate this. It is quite daunting to keep track of all dependencies to ensure that a complete rebuild of everything on a clean machine will be successful later down the road.

EDIT: Here's my ORDER file

Code:

libmad
soundtouch
scons
cvs
imaging
dvd-slideshow
fftw
fltk
libgsasl
gst-python
wxGTK
scrollkeeper
libavc1394
libiec61883
liblo
ilmbase
openexr
libsndfile
libsamplerate
twolame
ladspa
VCO-plugins
REV-plugins
MCP-plugins
FIL-plugins
AMB-plugins
caps-plugins
swh-plugins
tap-plugins
jack
dssi
portaudio
jackbeat
bitmeter
libquicktime
mjpegtools
transcode
sox
libgnomecanvas
ffmpeg2theora
raptor
liblrdf
dvgrab
dvdauthor
denemo
lash
gtick
kino
ardour
mandvd
terminatorX
audacity
muse
mixxx
jackEQ
jdelay
jamin
jack-rack
qjackctl
vkeybd
hydrogen
libgig
linuxsampler
liblscp
qsampler
manslide
avidemux
blop
tovid
omins
libpano13
libpano12
glew
libxmi
freeglut
enblend
cinepaint
sweep
mhwaveedit
gc
pigment
python-setuptools
qtractor
lives
fluidsynth
swami
csound
qsynth
TiMidity++
guile
fontforge
autotrace
potrace
t1utils
mftrace
tetex
qdvdauthor
XML-Twig
rosegarden
libatomic_ops
pulseaudio
tapiir
libsigc++
yafray
blender
tse3
ttf2pt1
lilypond
fluidsynth-dssi
lmms
gavl
gmerlin
openmovieeditor
speex
hugin
k9copy
oggconvert

This is what the final product should look like. Only the process of getting to that should be automated. Anyway, take a look in my contrib and see if that helps clear things up. Oh, also take a look at my CompileAll.sh script. That might give you some ideas.

[Triarius Fidelis]

Would it be necessary to detect dependencies of dependencies? If not, then my task is greatly simplified.

[Dweeberkitty]

In a way, yes. The first thing that the script will have to do is search for all the .SlackBuilds that exist. After that is done we will have a list of all packages that then needs to be ordered. Whatever the process that is used to generate the order, at the end of the day, it must be able to work much the same way as Gslapt and make sure that each dependency is met. However, unlike Gslapt where the dependency of the dependency order doesn't matter because it is all recompiled, we are compiling the dependencies which in turn possibly require other programs to be present which should have been compiled before them. So to answer your question, yes, it does need to detect dependencies of dependencies. Sorry! 

[Triarius Fidelis]

There's a topological sort module for Python, that might help me. (Yikes!)

[Triarius Fidelis]

Merry Christmas, Dweeberkitty

http://www.sendspace.com/file/nzu496

You unpack this guy somewhere, cd gen-order, then run python gen-order.py with the root sources directory as the only argument. The default source directory is the current directory, but that would make no sense for this distribution. At the moment, I don't have it looking for SlackBuild files, only printing out the order. Should I raise an exception or something if none is found?? Not sure.

[Dweeberkitty]

Thanks hanumizzle! I'll give this a try real soon. Merry Christmas to you too!

[Triarius Fidelis]

Thanks hanumizzle! I'll give this a try real soon. Merry Christmas to you too!

Err, it might be in reverse order. In which case, pipe the output through perl -we 'print sort { $b cmp $a } <>'

[Dweeberkitty]

Hanumizzle,

I was just going to use your script but I get

Sorry, the file you requested is not available.

from the link. Could you either reupload it or send it to the email address in my profile. Thanks!

[Witek Mozga]

Hanumizzle,

I was just going to use your script but I get

Sorry, the file you requested is not available.

from the link. Could you either reupload it or send it to the email address in my profile. Thanks!

If hanumizzle made a script for you already that`s OK. But if not then I could try. I don`t promise it will be fast as I constantly lack time but I`ll try to do my best. I don`t know where to look for vectorcontrib/Dweeberkitty/BonusDisc/ so if you could specify where to exactly find those example files, that would be nice. Technical note: I will use Tcl (interpreter for this language is present in Vector, it`s tclsh)

[Triarius Fidelis]

Hanumizzle,

I was just going to use your script but I get

Sorry, the file you requested is not available.

from the link. Could you either reupload it or send it to the email address in my profile. Thanks!

Second attempt:

http://www.sendspace.com/file/gdnpeq

[Dweeberkitty]

Cool. It works! However, I find that it produces dependencies that the packager doesn't care about because they are not compiling it. In other words, the script needs to search to see if the .SlackBuild for the dep exists before adding it to the list. Could you do this? Thanks!

~Dweeberkitty

[Triarius Fidelis]

http://pastebin.ca/845915

Try replacing gen-order.py with the file above.