Association Subscribers Manager and Open source stuffs

To content | To menu | To search

slack-get development diary

slack-get is a package manager for Slackware Linux distributions.

"slack-get development diary" is a diary (surprised ?) where slack-get developers write about the coding progress of the week.

Entries feed - Comments feed

Wednesday 3 September 2008

slack-get development diary #7

By Arnaud Dupuis on Wednesday 3 September 2008, 20:00

Hi,
Here is the 7th opus of the slack-get development diary. There is a lot to say because I made lots of changes since the last diary.

1) The C++ API's first parts :
Since the last post where I was talking about this I added to the existing Slackget::Package and Slackget::PackageList 3 new classes : Slackget::Config, Slackget::Utils and Slackget::QSimpleUpdate.
Slackget::Config is used to represent slack-get's XML configuration file and Slackget::Utils contains some static utilitary methods.
Slackget::QSimpleUpdate is a Qt 4 widget. Its goal is to present a list of packages to users in the most simple way. It behave very simply by showing a list of items wich contains an image, a package name and its version. More informations about a package are available via the Qt's traditionnal tooltip system.

I plan in renaming Slackget::Utils to simply Slackget but I am not yet sure. I am not sure because I want to re-organize the C++ api's tree.
There is currently a trunk/src/ directory which contains QSimpleUpdate sources, and a trunk/lib/Cpp/ which contains the other classes.
More specifically, it contains the SlackgetCore module. I plan on modify this tree the following way :
trunk/lib/Cpp/SlackgetCore/src/ -> contains all source files for the SlackgetCore modules (currently : Slackget::Package, Slackget::PackageList, Slackget::Config and Slackget::Utils)
trunk/lib/Cpp/SlackgetCore/include/ -> contains all headers file for the SlackgetCore modules
trunk/lib/Cpp/SlackgetGui/src/ -> contains all source files for the SlackgetGui modules (currently : Slackget::QSimpleUpdate)... This module actually do not exists at the time I am writting this article :)
trunk/lib/Cpp/SlackgetGui/include/ -> contains all headers file for the SlackgetCore modules
And of course I want to delete the trunk/src/ directory.
I am indeed a big fan of the Qt 4 modular architecture. Talking about Qt 4, all parts of the slack-get C++ api links against Qt 4. SlackgetCore links against QtCore (for all the super handy data types) and, of course, SlackgetGui will links against QtGui.

2) The Perl API and sg_daemon :
Well... It changed a lot so I made a new release of Slackware::Slackget on CPAN ... And again it changed so much since then that I could make another release !
The medias variables system is one of the good examples. But the huge bugs in sg_daemon which was not managing media's host efficiently is the main cause of changes in the Perl API.
This bug (or bugs) was preventing the sg_daemon to switch from a media's host to the next one. I corrected this issue and it is now working as it should do.
The main changes in the Slackware::Slackget Perl module are :
  * I changed Slackware::Slackget::Network::Connection behavior by replacing the host's constructor parameter by a media one (see modules documentation). I also added a next_host() method to this module. The goal is to take care of operations needed by the Connection.pm module (like parsing the URL and loading drivers).
  * I fixed a little bug in Slackware::Slackget::Network::Connection::HTTP.pm (the file was moved to its final destination before all tests).
  * I fixed a bug in Slackware::Slackget::Base and Slackware::Slackget::MediaList. The bug was very simple : I forgot to tell XML::Simple's parser to force array on all <li> elements. This result in a bug when there was only one element in a list.
  * I also spent some time cleanning lots of code (removed unneccessary comments, comments unnecessary code, etc.).
  * All this things result in a massive documentation's update.
  * I finally made all needed changes in the sg_daemon (which is not part of the Perl API...) which is now handling well dead hosts and host changing.

The next focus points in the development are :
     1 - Improve the C++ api (and particularly develop all classes needed to connect to a sg_daemon)
     2 - Port the plugin system to the new daemon.
     3 - Develop a real working GUI.
     4 - Start coding the multi-daemon cooperative work.

Still a lot to do !

Enjoy !

Arnaud Dupuis

no comment no trackback

Wednesday 13 August 2008

slack-get development diary #6

By Arnaud Dupuis on Wednesday 13 August 2008, 11:35

Back from a long silence period, here it is : the 6th opus of the "slack-get development diary" !
There is a lot of new things that I need to talk about !
First the bad news : I saw on CPAN reports that the "fix" I made for Slackware::Slackget to properly test on Solaris OS is not working.
That is a bad thing but... I don't really see the purpose on trying to fix an issue on an OS which is not a Slackware based one and not even a GNU/Linux OS ! The Slackware::Slackget module will never be usefull on this OS, so since I have no Solaris to test I will just forget about it (unless somebody provides me with a patch for this system.

Now for all the good news :
  • slack-get suite (sg_daemon + slack-get) is now able to install, upgrade and remove packages
That's the first point, and I think it's a pretty important one. I fixed all the daemon and the (cli) client to make them able to perform packages operations without any problems.
So from now on, you can start a daemon and do a "slack-get install flightgear" for example :)
That's a very important point but it's nothing worth talking about without the dependencies tracking system.
Talking about that...
  • the dependencies tracking system is now fully functionnal
Woohooo ! This part was certainly one of the most painfull, but I finally manage to get it working the right way.
This part still need to be tested and there is no guarantees, so far, that it is working properly in all cases.
But so far my tests where very promisefull ! I add no problems and for the moment I have not yet discovered any bugs in this feature.
Again, that does not means that I will not uncover bugs later.
  • add support for automatic GPG key import
I added the possibility for the sg_daemon to download and import a GPG key. For the moment it only import Slackware Project's key and it is almost hardcoded but there is no reasons to extend this feature to all medias sooner or later.
This feature works well and is totally automatic : if sg_daemon do not find the Slackware Project's key in the user's GPG keyring, it download it and import it. Nice and easy.
  • add new feature to slack-get (cli client)
I added a new command to slack-get, it is called "info". You can use it the exact same way than the "search" command (this is the exact same code which is processing it).
The goal of this command is to provide more informations about a package. Here is an usage example :
$ ./slack-get info flightgear --media=slacky
Package: flightgear-1.0.0-i686-1as
Size (compressed): 3158 KB
Version: 1.0.0
Source: slacky
Description: The FlightGear flight simulator open-source project.The  goal  of  the  FlightGear project is to create a
sophisticated flight simulator framework for  use  in  research  or  academic environments,  for the development and pursuit
of other interesting flight simulation ideas, and as an end-user application
http://www.flightgear.org/ WWW.SLACKY.IT PackaGer Gohanz.

This little example, allow me to introduce the new command line option : --media. This one allow you to restrict the "search" or "info" commandes to the choosen media (in the previous example I wanted only results from the Slacky.it website).

Last about the slack-get cli client, I fixed lots of "non closing bugs". The non closing bugs are an annoying problem of slack-get cli client.
While the whole system became multitasks, it's becomming difficult to keep track of what the client asked by itself (particularly in a multi-administrators context), and after asking a sg_daemon to perform some tasks (and after the tasks are finished) the client does not end.
I corrected a lot of thoses problems.
A good example of this issue is with the "slack-get update" command. This one scheduled a tasks to upgrade all packages which have new ones in the patches/ subtree of the official-media (see config.xml). After the update, the client was not quitting. It is now corrected.
  • dynamic network backends negociation
Some of you may have noticed that sg_daemon should be able to dynamically negociate what network backends the client and it should use to understand each others. Until this morning it was not working, and it is now !
If sg_daemon supports XML and Base64 backends, and if the slack-get client supports XML and Gzip backends ; they will both agree on using the biggest common denominator (in this case they will use XML only).
I am very happy (and quite proud) of this mechanism. It allows tons of new development and many plugins to come !
  • lots of updates in the Slackware::Slackget Perl module
Last but not least I made tons of modifications in the Slackware::Slackget module. I added constants, generative methods to Slackware::Slackget::Media, add codes to Slackware::Slackget::GPG to support new import features, add method to Slackware::Slackget to check host's Slackware's version, and so on and so forth !

As a conclusion, I will just say that there is still some works to do but it's becomming to be a very usable tool. I'm using it every day now.
I hope all the work I made on slack-get will be of interets for you all !
As a bonus track I give you the new slack-get logo (that I made myself... so be kind ;-) )


Enjoy !

Arnaud Dupuis


no comment one trackback

Tuesday 1 January 2008

slack-get development diary #5

By Arnaud Dupuis on Tuesday 1 January 2008, 09:56

Hi !

First of everything : I wish you all an happy new year. May all your desires come to reality !

My own desire is to release a first version of the new slack-get daemon quickly as possible. I'm coding the real work for the upgrade and install actions (update, remove, search and the others are already working but the remove one was still not test on a real case).
Some things actually happened on slack-get development since the last diary.

Bugfixes

I fix a lots of (very old) bugs. For example, in the Slackware::Slackget::Base class (which is untouched since months), there was a bug and the FILELIST: tag from the packages text file was not removed correctly and was still present in the XML. This is now fixed.
I also improve a lot the way metadata are parsed by classes like Slackware::Slackget::Package by simplifying the parsing regular expressions.
Without turning this diary in a bugfixe list, I also modify some XML parsing behaviors in order to make the resulting data structure consistent over time. I modified the Slackware::Slackget::Package class quite a lot and it now parse itself the dependency fields in the packages metadata and translates them into XML. The impact of this is : a bigger XML file (packages.xml), and an increased loading time. But in the other end, all data are now completely parsed in the XML file and there is no extra work to do after parsing, and the dependency tracking is a lot easier (and actually quicker ^^).

Remaining bugs

There is still a bug in Slackware::Slackget::Network::Backend::Gzip. The backend which is in charge to compress and uncompress the network messages... Well it's not working at all, and worst i don't know why. But in the other hand, i have not tried a lot to fix this module. I focus on sg_daemon and slack-get for the moment (and the involved vital modules).
The Base64 and XML backends are still working fine.

Other stuff

You should also be aware that I'm trying to help as much as I can for the KDE 4.0 release, so until the 11th of January the slack-get's development will certainly be slowed down by my involvement in KDE 4.0 development.
And on the 11th of January i'm going to snowboarding in the Alps until the 20th of January ! So I will not be available at all during this time... Off course, because I'll be too busy trying to break some of my bones ;-)

no comment

Monday 10 December 2007

slack-get development diary #4

By Arnaud Dupuis on Monday 10 December 2007, 22:00

Hi,
There is a lot to say for this development diary #4 !

Perl module: Slackware::Slackget vs slackget10 :

First of all, the Perl part of the project (currently the biggest part). As you may all know now : the slack-get Perl module is now listed in the official Perl Module list on CPAN in the Operating System Interface category. And for this big promotion the module changed its name from slackget10 to Slackware::Slackget.
If some of you still wondered : slackget10 is officially abandoned, and today, completely outdated.
I added a lot of features to this module like the stackable backends. I also released on CPAN a debug module which add the support for a debug:// network protocol. This is a totally fake protocol which help development by outputing loads of informations on the standard output. The module name is Slackware::Slackget::Network::Connection::DEBUG. It's not a part of the Slackware::Slackget distribution, it's even not in the SVN trunk but in branch/perl-modules.
One of the big improvement of the Slackware::Slackget module is its test suite : the latest release on CPAN (0.14) haven't encounter even 1 fail !
I also tried a lot to make the module platform independent, and more generic. I have rewrite lots of code in order to take all the interdependent code out of the modules. It's now partially done and the next module which will feel my scalpel is Slackware::Slackget::PkgTools... I saw some horrible things inside of this one !

The slack-get suite programs : sg_daemon and slack-get :

Everything have changed ! Beginning with slack-get : this program wasn't available 2 weeks ago. It's a command line client for the sg_daemon it allow you to query the daemon. At the moment I write this development diary (svn revision 110) you can perform the following actions with slack-get : search a package (based on it's name, or anything in the description of this package), ask the daemon to rebuild the installed packages cache, ask the daemon to reload it's media list, ask the daemon to rebuild the update list. Well... now it only lack the possibility to install, upgrade and remove packages to be released... I'm already on the remove part (the easiest ^^).
Let's show you some search request performed on a cache which contains the official Slackware current repository, Linuxpackages repository and my own perl-modules repository (totally outdated) :

This is a big improvement since it actually do something... You may wonder how it is possible for me to implements 3 or 4 possible actions (features) for slack-get and have wait for so long before doing it. In fact it's now very easy and quick to add features to sg_daemon and slack-get because the base architecture is done coding (and well designed :D).
From now on, the remaining features I will include before releasing a first version (without any GUI but with the CLI client) are :
  • install package
  • upgrade package
  • remove package
  • run background for the daemon
Once it'll be done I will bundle it and release the 1.0.0-alpha2 version. Obviously, it will be a testing version.
In the real coding side, the last SVN revision of sg_daemon include tons of improvements like : automatic message formating, a first version of the backend stack negotiation system (between the daemon and the client), a real and simple network protocol, a powerfull and working base architecture and a bullet proof internal communication ! (with that if I don't attract some geek devs I don't know how to do it ;-) ).
The next big part of the work will be the implementation of collaborative part. But fortunately, a huge amount of this code was already wrote in slack-get-1.0.0-alpha1. To give you an example, all the code related to the master/slave mechanism is already written and slack-getd is able to work with many other daemons withtout to much pain in the configuration side (assuming you talk XML...).
But in addition to the existing code (that I will still need to port to the new event-based architecture), I want to add an "auto discover" feature to allow many sg_daemon(s) to work together without any configuration or human supervision. I want the daemons able to  elect a master, change their mode between master and slave, and many other distributed stuffs like that, based on some simple rules (what's the sysload, the average sysload, etc.) in total autonomy (if the sysadmin allow it !).
In addition to all of this development, I will publish a debug backend for Slackware::Slackget::Network to allow developers to track the state of a network message during the encoding and decoding process.

Some news from the GUI point :

I made a graphical interface to get the list of official Slackware mirrors (from the "Get Slack" page of the official website). It's still not finished (it cannot update the medias.xml file yet), but it's already working to get the list and writting it down to the hard disk. I think this dialog will be part of the final slack-get GUI.
And since it's the first screenshot I can show, here it is :

Well... nothing to climb to the roof but it's a beginning :)
All the testing GUI are available from the SVN branch : branch/gui_test.
The GUI is the next main development priority after getting the daemon working.

I think I'm done with this week (and a half...) diary, I'll keep you informed !

Cya

Arnaud Dupuis.

no comment

Monday 26 November 2007

slack-get development diary #3

By Arnaud Dupuis on Monday 26 November 2007, 16:19

Hi,

I was a bit lazy about blogging the slack-get development progression... Me bad, please accept my deepest apologies.
In the other hand, where do you think I spent all the time I didn't on blog ? Yes, you are right : playing SSX Blur on my Wii :)
More seriously, I made some nice improvements in sg_daemon :
  • The file downloading code is now working and use a lot POE. If you checkout the code on Sourceforge's SVN and try out sg_daemon you will see that it is downloading all files to build the updates cache ! I'm quite happy to have this functionality working. Although it's not perfect and I'll make the slackget10::Network::Connection* drivers using the POE way of passing parameters. This will become very handy when I will update the drivers to a trully POE based architecture.
  • The other big progress is about the network protocol. I think that I finally managed to define the needs of slack-get . The keys points are : flexibility and powerfullness. The XML based protocol I defined should meet those requirements. So I defined a base data structure that all network messages must respect, and after that I defined a way to serialize those data. So basically you just need the right serialization component to change the way sg_daemon and slack-get client talks. For the moment I will only develop the XML one but, in order to make easy the creation of other components, when a client connects to sg_daemon a quick handshake is made (sg_daemon send a message with the protocol version, client send a message with its protocol version and his supported serialisation method.
  • Following the move slackget10::Network.pm was almost entirely rewrote to only format the network messages. A new class appeared : slackget10::Network::Message wich is an abstraction of a network message (unbelievable isn't it ?), an object of this type must now be give as parameter to all sg10::Network methods. Those changes are all due to the redefinition of the network protocol and to the rewriting of sg_daemon.
Last but not least : I promise to be more active on this blog ;-)

See ya next time !

Arnaud Dupuis

no comment

Tuesday 6 November 2007

slack-get development diary #2

By Arnaud Dupuis on Tuesday 6 November 2007, 07:23

Hi,


First let me tell you some things about me : I've got a new job recently. I'm now working in a IT consulting company in their Open Source business unit. That's great !

I also marry my fiancée in June... So you can tell that I have less time to code ;-)

That was for the explanation why I haven't code a lot this week :)

But the slack-get development is still moving quite nicely, even if I had loads of problems this week. I focused on turning all the network handling code in sg_daemon POE compliant. So I looked at a POE Component (PoCo) wich implements the HTTP protocol.. Great the is one who manage parallel downloads !! I tried it and it is just what I need. So I began to search the same things but for other protocols sg_daemon support... and here are the problems.

I found nothing for the FILE protocol and the PoCo for FTP is still young and do not provide the same API than PoCo::Client::HTTP. So I ended to reverse the changes I already made to slackget10::Network::Connection::HTTP and decided to change the API to emulate the POE way of doing things. So the slackget10::Network::Connection object must, now, be constructed with an argument InlineState => {}which contains 3 events : progress, download_finished, and download_error. You have to associates a handler to each of this events.

Talking about this class, the internal architecture have changed and it is now way easier to add drivers for slackget10::Network::Connection. You just have to create a class wich implements some methods. It will be the feature highlights of the week.

I have not much to say considering that all my family stuff for my weddings and all the problems I met with the "POE-ification" of the network code slowed me down a lot.

Let's hope for next week :-)

Enjoy !

Arnaud Dupuis

no comment no trackback

Saturday 27 October 2007

slack-get development diary #1

By Arnaud Dupuis on Saturday 27 October 2007, 21:41

Hi !

I've got a lot to say today !
First of all I was surprised by the number of failed test in my last release of the slackget10 Perl module. After some work on this, and with the great help of David Cantrell, we managed to find to work out the problem.
It is in the test suite of XML::Simple (used by slackget10::Base and other slackget10::* classes) : if XML::Parser failed to install the XML::Simple's test suite do not chain fail, and the installation is not done well, but it's done.
So if you try to use XML::Simple with the directive :
$XML::Simple::PREFERRED_PARSER='XML::Parser';
slackget10::Base just crash. David submitted a patch which is very simple but also very efficient :
eval 'use XML::Parser';
if($@) {
    warn("XML::Parser is not installed. XML processing operations will be very slow.\n");
} else {
    $XML::Simple::PREFERRED_PARSER='XML::Parser' ;
}

It fixed the problem, and I have integrated it directly to the trunk of slack-get SVN repository.

The second point is the slack-get communication protocol draft. Things are going well, I think I finished to determine at least 70% of the communication needs.
If you are willing to give me an advice on this draft do not hesitate ! Go checkout the SVN version and have a look in the trunk/devel/ directory.

About sg_daemon now : I've got the internal scheduler working ! yeehaa !
I've finish the porting of the "installed packages list building" functionality  and it's now working with the internal scheduler of sg_daemon.
The great thing is : that was the hardest part :)
Porting the other functionalities will be easy... or not... indeed some internal functions of sg_daemon are not adapted to the POE architecture and will take some time to port.

Last but not least, I've come with a new icon for slack-get, feel free to give your opinion :

no comment