Association Subscribers Manager and Open source stuffs

Hi,
The Slackware::Slackget api is available on CPAN, but the latest documenation is now available in the SVN tree (viewable on Sourceforge's viewsvn service) and on my own server.
The two last ones will be more up-to-date than the CPAN one.

I hope that it will helps contributors !

Arnaud Dupuis.

Hi,
I just added a new feature to the slack-get media system. It is something I wanted to add for a long time now but I never had the time.
But since I am still stuck at home because of this stupid pneumonia, I now have the time...
What I wanted is a variables system in the medias.xml syntax. The direct benefit is that it is much more easier to modify the medias file after an upgrade of your Slackware box.
Indeed, if it is nice to be able to maintain multiple sources of packages, it is sometime a pain in the ass to update this file after an upgrade... When all URL of mirrors must be changed...
The Slackware version number is, indeed, the first "official" variable.
This system is very easy to use : you just have to declare an attribute in the <medialist> tag and it will be automatically converted to a variable by the Slackware::Slackget::MediaList and Slackware::Slackget::Media classes. You can use the variable in any tag content by using the special syntax ${<variable name>}.
Here is a simple example :

<?xml version="1.0" encoding="ISO-8859-1" standalone="yes"?>
<medialist slackware-version="12.1">
<media id="slackware-official">
<files>
<filelist>FILELIST.TXT</filelist>
<checksums>CHECKSUMS.md5</checksums>
<packages>PACKAGES.TXT</packages>
</files>
<download-signature>1</download-signature>
<update-repository>
<faster>http://ftp.lip6.fr/pub/linux/distributions/slackware/slackware-${slackware-version}/</faster>
<fast>
<li>http://ftp.bit.nl/mirror/slackware/slackware-${slackware-version}/</li>
<li>http://ftp.belnet.be/packages/slackware/slackware-${slackware-version}/</li>
</fast>
</update-repository>
<description>The official Slackware web site</description>
<web-link>http://www.slackware.com/</web-link>
</media>
</medialist>

As you can see it is both convinient and easy to implement. I am quite sure it will be very usefull when we will upgrade from Slackware 12.1 to 13.0

Arnaud Dupuis.

Back from a long silence period, here it is : the 6th opus of the "slack-get development diary" !
There is a lot of new things that I need to talk about !
First the bad news : I saw on CPAN reports that the "fix" I made for Slackware::Slackget to properly test on Solaris OS is not working.
That is a bad thing but... I don't really see the purpose on trying to fix an issue on an OS which is not a Slackware based one and not even a GNU/Linux OS ! The Slackware::Slackget module will never be usefull on this OS, so since I have no Solaris to test I will just forget about it (unless somebody provides me with a patch for this system.

Now for all the good news :

• slack-get suite (sg_daemon + slack-get) is now able to install, upgrade and remove packages

That's the first point, and I think it's a pretty important one. I fixed all the daemon and the (cli) client to make them able to perform packages operations without any problems.
So from now on, you can start a daemon and do a "slack-get install flightgear" for example
That's a very important point but it's nothing worth talking about without the dependencies tracking system.
Talking about that...

• the dependencies tracking system is now fully functionnal

Woohooo ! This part was certainly one of the most painfull, but I finally manage to get it working the right way.
This part still need to be tested and there is no guarantees, so far, that it is working properly in all cases.
But so far my tests where very promisefull ! I add no problems and for the moment I have not yet discovered any bugs in this feature.
Again, that does not means that I will not uncover bugs later.

• add support for automatic GPG key import

I added the possibility for the sg_daemon to download and import a GPG key. For the moment it only import Slackware Project's key and it is almost hardcoded but there is no reasons to extend this feature to all medias sooner or later.
This feature works well and is totally automatic : if sg_daemon do not find the Slackware Project's key in the user's GPG keyring, it download it and import it. Nice and easy.

• add new feature to slack-get (cli client)

I added a new command to slack-get, it is called "info". You can use it the exact same way than the "search" command (this is the exact same code which is processing it).
The goal of this command is to provide more informations about a package. Here is an usage example :
$ ./slack-get info flightgear --media=slacky
Package: flightgear-1.0.0-i686-1as
Size (compressed): 3158 KB
Version: 1.0.0
Source: slacky
Description: The FlightGear flight simulator open-source project.The  goal  of  the  FlightGear project is to create a
sophisticated flight simulator framework for  use  in  research  or  academic environments,  for the development and pursuit
of other interesting flight simulation ideas, and as an end-user application
http://www.flightgear.org/ WWW.SLACKY.IT PackaGer Gohanz.

This little example, allow me to introduce the new command line option : --media. This one allow you to restrict the "search" or "info" commandes to the choosen media (in the previous example I wanted only results from the Slacky.it website).

Last about the slack-get cli client, I fixed lots of "non closing bugs". The non closing bugs are an annoying problem of slack-get cli client.
While the whole system became multitasks, it's becomming difficult to keep track of what the client asked by itself (particularly in a multi-administrators context), and after asking a sg_daemon to perform some tasks (and after the tasks are finished) the client does not end.
I corrected a lot of thoses problems.
A good example of this issue is with the "slack-get update" command. This one scheduled a tasks to upgrade all packages which have new ones in the patches/ subtree of the official-media (see config.xml). After the update, the client was not quitting. It is now corrected.

• dynamic network backends negociation

Some of you may have noticed that sg_daemon should be able to dynamically negociate what network backends the client and it should use to understand each others. Until this morning it was not working, and it is now !
If sg_daemon supports XML and Base64 backends, and if the slack-get client supports XML and Gzip backends ; they will both agree on using the biggest common denominator (in this case they will use XML only).
I am very happy (and quite proud) of this mechanism. It allows tons of new development and many plugins to come !

• lots of updates in the Slackware::Slackget Perl module

Last but not least I made tons of modifications in the Slackware::Slackget module. I added constants, generative methods to Slackware::Slackget::Media, add codes to Slackware::Slackget::GPG to support new import features, add method to Slackware::Slackget to check host's Slackware's version, and so on and so forth !

As a conclusion, I will just say that there is still some works to do but it's becomming to be a very usable tool. I'm using it every day now.
I hope all the work I made on slack-get will be of interets for you all !
As a bonus track I give you the new slack-get logo (that I made myself... so be kind )


Enjoy !

Arnaud Dupuis

Hi,
It looks like that the latest released version of the Slackware::Slackget Perl module is quite incompatible with the Solaris operating system.
I was expecting this kind of problem with this release because I added more tests to the test suite. And Slackware::Slackget, as its name said, is designed to run on Slackware GNU/Linux based distributions.
So before the 0.15 final release, in order to provide a very usefull test suite, I will add some more tests.
And unfortunately, I expect more issues... The more the test suite is accurate, the more it will point the problem with Slackware incompatible systems.

The problem with the Solaris systems is that the `file` command do not support the -b switch. I use the file command instead of a CPAN module to reduce the amount of dependencies of Slackware::Slackget but I will look at this and try to make the Slackware::Slackget::File class use a CPAN module if installed (like File::Type).

Arnaud Dupuis

Hi,

Back from holidays, I started to work again on slack-get by a huge code review and documentation updates of the API.
My current contract had an impact on this process since it consists in extending the NMIS tool. This tool is really great and useful like no other ones when it comes to network monitoring tasks.
But this tool also have an important flaw : its API is a real mess. It consist of a main script of 7000 code lines and few modules (around 10) totally undocumented !
My job is currently to run into this code, understand it (hopefully it's not too hard to guess) and find a way to add something like a plug-in system to it. Unfortunately, all NMIS modules exports all their functions in the calling script's namespace...
So I choose to have a different naming convention than the one in NMIS for plug-in's methods name... And their is no naming convention in NMIS...

And I sadly saw that it was exactly the same  with slack-get Perl API... So starting today I will change that and tag along with the following naming convention :

1. a method name consist of English words where space are replaced with underscores ;
2. if the name is conflicting with Perl or C++ built-in function or language syntax, change the case of the first letter to upper  case.
3. if a conflict was solved according to rule 2), and if the name do not conflict with the other language, add a wrapper method around the renamed one, in order to comply with the rule 1).

Boring work but I deserve to do it...

Arnaud Dupuis

Hi,
There is a lot to say for this development diary #4 !

Perl module: Slackware::Slackget vs slackget10 :

First of all, the Perl part of the project (currently the biggest part). As you may all know now : the slack-get Perl module is now listed in the official Perl Module list on CPAN in the Operating System Interface category. And for this big promotion the module changed its name from slackget10 to Slackware::Slackget.
If some of you still wondered : slackget10 is officially abandoned, and today, completely outdated.
I added a lot of features to this module like the stackable backends. I also released on CPAN a debug module which add the support for a debug:// network protocol. This is a totally fake protocol which help development by outputing loads of informations on the standard output. The module name is Slackware::Slackget::Network::Connection::DEBUG. It's not a part of the Slackware::Slackget distribution, it's even not in the SVN trunk but in branch/perl-modules.
One of the big improvement of the Slackware::Slackget module is its test suite : the latest release on CPAN (0.14) haven't encounter even 1 fail !
I also tried a lot to make the module platform independent, and more generic. I have rewrite lots of code in order to take all the interdependent code out of the modules. It's now partially done and the next module which will feel my scalpel is Slackware::Slackget::PkgTools... I saw some horrible things inside of this one !

The slack-get suite programs : sg_daemon and slack-get :

Everything have changed ! Beginning with slack-get : this program wasn't available 2 weeks ago. It's a command line client for the sg_daemon it allow you to query the daemon. At the moment I write this development diary (svn revision 110) you can perform the following actions with slack-get : search a package (based on it's name, or anything in the description of this package), ask the daemon to rebuild the installed packages cache, ask the daemon to reload it's media list, ask the daemon to rebuild the update list. Well... now it only lack the possibility to install, upgrade and remove packages to be released... I'm already on the remove part (the easiest ^^).
Let's show you some search request performed on a cache which contains the official Slackware current repository, Linuxpackages repository and my own perl-modules repository (totally outdated) :

This is a big improvement since it actually do something... You may wonder how it is possible for me to implements 3 or 4 possible actions (features) for slack-get and have wait for so long before doing it. In fact it's now very easy and quick to add features to sg_daemon and slack-get because the base architecture is done coding (and well designed :D).
From now on, the remaining features I will include before releasing a first version (without any GUI but with the CLI client) are :

• install package
• upgrade package
• remove package
• run background for the daemon

Once it'll be done I will bundle it and release the 1.0.0-alpha2 version. Obviously, it will be a testing version.
In the real coding side, the last SVN revision of sg_daemon include tons of improvements like : automatic message formating, a first version of the backend stack negotiation system (between the daemon and the client), a real and simple network protocol, a powerfull and working base architecture and a bullet proof internal communication ! (with that if I don't attract some geek devs I don't know how to do it ).
The next big part of the work will be the implementation of collaborative part. But fortunately, a huge amount of this code was already wrote in slack-get-1.0.0-alpha1. To give you an example, all the code related to the master/slave mechanism is already written and slack-getd is able to work with many other daemons withtout to much pain in the configuration side (assuming you talk XML...).
But in addition to the existing code (that I will still need to port to the new event-based architecture), I want to add an "auto discover" feature to allow many sg_daemon(s) to work together without any configuration or human supervision. I want the daemons able to  elect a master, change their mode between master and slave, and many other distributed stuffs like that, based on some simple rules (what's the sysload, the average sysload, etc.) in total autonomy (if the sysadmin allow it !).
In addition to all of this development, I will publish a debug backend for Slackware::Slackget::Network to allow developers to track the state of a network message during the encoding and decoding process.

Some news from the GUI point :

I made a graphical interface to get the list of official Slackware mirrors (from the "Get Slack" page of the official website). It's still not finished (it cannot update the medias.xml file yet), but it's already working to get the list and writting it down to the hard disk. I think this dialog will be part of the final slack-get GUI.
And since it's the first screenshot I can show, here it is :

Well... nothing to climb to the roof but it's a beginning
All the testing GUI are available from the SVN branch : branch/gui_test.
The GUI is the next main development priority after getting the daemon working.

I think I'm done with this week (and a half...) diary, I'll keep you informed !

Cya

Arnaud Dupuis.

Hi,
Like I announced on the developers' mailing list, the slackget10 was about to die. It's now done. The PAUSE (Perl Authors Upload SErver) admins sent me an email confirming the creation of the Slackware::Slackget registered namespace on PAUSE/CPAN.
I will modify all the library to fit the new namespace and release the changes as soon as possible.
One more thing : I created a new branch on the SVN for the GUI tests

Cya,

Arnaud Dupuis