Association Subscribers Manager and Open source stuffs

Hi !

Today I'm introducing a new kind of post : the slack-get functionality highlights. What is that ? Well once a week (or more often maybe) I will introduce and explain a specific functionality of slack-get.
It can be in sg_daemon, or slack-get tools (CLI or GUI). It can also be some nice features of one of the slack-get libraries (slackget10 Perl module or the upcoming C++ one).

Enough introduction ! Today's highlight is about the media management system. Before everything, please note that all the examples in this diary will work on all slack-get version >= 1.0.0_pre-alpha1

How do slack-get manage its media and how the daemon choose to download a package from a server or another ? This is indeed a good question because there is very few documentation about this feature.... my bad
So.. slack-get use a configuration file called medias.xml where all medias are listed. This file is a XML one (as suggested by its extension...), and look like that :
<?xml version="1.0" encoding="ISO-8859-1" standalone="yes"?>
<medialist>
    <media id="media1">
       // lets forget this part for now
    </media>
    <media id="media2">
       // ...
    </media>
</medialist>


Quite easy to understand isn't it ? there is a media list which contains a list of media... fair enough
Now what about the <media></media> entity ? It's basically THE basic entity for all media you want to use as an install source.
As I write this, you can have 3 types of media :

• http ones. For all repositories you access via the http protocol ;
• ftp ones. For... yes : all the repositories you can access via the ftp protocol ;
• file ones. That's the general one. You can use it for installing Slackware packages from your Slackware DVD, from a NFS drive or any other repository which can be seen as "local filesystem" by your system.

The structure of the XML entity <media></media> look like that :
<media id="a_media_id">
    <files>
        <filelist>FILELIST.TXT</filelist>
        <checksums>CHECKSUMS.md5</checksums>
        <packages>PACKAGES.TXT.gz</packages>
    </files>
    <update-repository>
        <faster><!-- an URL --></faster>
       <fast>
          <li><!-- an URL --></li>
          <li><!-- another URL --></li>
       </fast>
       <slow>
          <li><!-- an URL --></li>
          <li><!-- another URL --></li>
       </slow>
    </update-repository>
    <description>The description of this media.</description>
    <web-link>http://www.mysite.org</web-site>
</media>

The <files></files> entity gives the name of the files which actually contains the wanted informations. there is 3 informations wanted :

1. the list of the repository files ( usually in the FILELIST.TXT file ) ;
2. the list of the files checksums ( usually in the CHECKSUMS.md5 file ) ;
3. the detailled list of packages ( usually in the PACKAGES.txt file ).

the slackget10 Perl module compile all those informations in a single file ( called installed.xml, we will discuss about it in another "slack-get functionality highlights" issue ).
You can say here if you prefer to download the compressed version if exists and if the repository is not really standard (file name in lower case for example) you also have more flexibility here.
I got emails from a bunch of peoples who even used the old slack-get daemon (slack-getd) to install red hat packages ! Which means one thing : you can do more than I coded with slack-get :D
Actually, it's not hard to modify the initial way-of-doing-things of slack-get, you just have to replace some of the Perl modules.

Let's see a simple example : adding the Slackware 12.0 DVD to the media list.
First open the medias.xml file with your favorite editor, then add the following :
    <media id="SlackDvd12.0">
        <files>
            <filelist>FILELIST.TXT</filelist>
            <checksums>CHECKSUMS.md5</checksums>
            <packages>PACKAGES.TXT</packages>
        </files>
        <update-repository>
            <faster>file:///mnt/cdrom/</faster>
        </update-repository>
        <description>The official 12.0 cd-rom directory.</description>
    </media>

The first thing you can note is that we didn't used all the possible XML entity in this example. Indeed all fields are not mandatories. The only mandatories "fields" (attribute or tag) are :

• id : the media id attribute ;
• the <update-repository></update-repository> tag ;
• the <faster></faster> tag.

So we added a new media, its id is "SlackDvd12.0", it use the standard files (so we can simplify this entry and remove the <files></files> section), and the only repository we have is in the /mnt/cdrom/ and have a small description.
We can simplify this media section (without loosing any informations), like that :
    <media id="SlackDvd12.0">
        <update-repository>
            <faster>file:///mnt/cdrom/</faster>
        </update-repository>
        <description>The official 12.0 cd-rom directory.</description>
    </media>

This example, is simple and quickly added, let's have a look at a more complete example, by adding the LinuxPackages repository to our medias list :
    <media id="linuxpackages">
        <files>
            <filelist>FILELIST.TXT</filelist>
            <checksums>CHECKSUMS.md5.gz</checksums>
            <packages>PACKAGES.TXT.gz</packages>
        </files>
        <update-repository>
            <faster>http://opensys.linuxpackages.net/Slackware-11.0/</faster>
            <fast>
                <li>http://www.nymphomatic.org/mirror/linuxpackages/Slackware-12.0/</li>
                <li>http://linuxpackages.inode.at/Slackware-12.0/</li>
                <li>http://ftp.scarlet.be/pub/linuxpackages/Slackware-12.0/</li>
                <li>http://www2.linuxpackages.net/packages/Slackware-12.0/</li>
                <li>http://mirror.etf.bg.ac.yu/linuxpackages/Slackware-12.0/</li>
                <li>http://linuxpackages.slackwaresupport.com/Slackware-12.0/</li>
            </fast>
            <slow>
                <li>ftp://linuxpackages.inode.at/Slackware-12.0/</li>
                <li>ftp://ftp.scarlet.be/pub/linuxpackages/Slackware-12.0/</li>
                <li>http://linuxpackages.cgucccc.org/Slackware-12.0/</li>
                <li>ftp://ftp3.linuxpackages.net/pub/Slackware-12.0/</li>
                <li>ftp://mirror.etf.bg.ac.yu/linuxpackages/Slackware-12.0/</li>
                <li>ftp://ftp.slackware.hu/linuxpackages/Slackware-12.0/</li>
                <li>ftp://opensys.linuxpackages.net/pub/Slackware-12.0/</li>
                <li>ftp://ftp.nymphomatic.org/linuxpackages/Slackware-12.0/</li>
            </slow>
        </update-repository>
        <description>Slackware resources to help install and configure the Linux slackware distribution, Email list, Discussion Board, Howtos, Contributed packages, and much more</description>
        <web-link>http://www.linuxpackages.net</web-link>
    </media>

Here we specify compressed files in the <files></files> section. This is possible because the slackget10::File class can load both of compressed and uncompressed files.
Then we have a list of repositories. The <faster></faster> one is like your preferred mirror, it will always be used unless it is not reachable. If it's not, servers in the <fast></fast> section are tried, then come the ones in the <slow></slow> section.
As you can see we have mixed type of mirrors : some are HTTP ones while other are FTP ones. That is not a problem, you can mix sources of all kinds. The only limitation is that slack-get must have a driver for the specified protocol (for the moment you can use only, http://, ftp:// and file://).
You can also see that we have set a web site for this media. This information is use in many way, for example in the search page of the slack-get site, or in the GUI to provide a link to the repository maintener site.

In real world, slack-get can test all the repositories and classify them by their answer time. Like in many other fields, I tried to make the slack-get's parts as clever as possible and do the right thing with the right informations. I hope it works

Now for the unreleased things and the unbelievably new informations : I will add the support of variables in repository URLs... this will allow you to write things like that in your medias.xml :
<update-repository param="$SLACKWARE_VERSION=12.0">
    <faster>http://opensys.linuxpackages.net/Slackware-$SLACKWARE_VERSION/</faster>
</update-repository>

Nice isn't it ? I'm still not sure about the final shape of this feature but I think it'll be like I just showed you.

To conclude, you are now more informed on the media management system of slack-get and you can easily manually add new repositories to your medias.xml file.
In the next version of the GUI you will be able to edit this file with a nice window.
I hope this article helped you to better understand some of the slack-get's black magic
Please, feel free to react, comment and propose improvements.

Arnaud Dupuis