SàT PubSub 0.3.0 est disponible

goffi 21/08/2019, 07:00 jabber-xmpp SàT SàT Pubsub project libre release planet-libre

La version 0.3.0 de SàT Pubsub est disponible.

SàT PubSub est un service XMPP PEP/PubSub indépendant du serveur, qui vise à être complet et universel.

Ce projet est né du constat qu'il est difficile d'avoir des services PEP/PubSub complets sur tous les serveurs XMPP, et que même si c'est le cas ils peuvent tarder à implémenter de nouvelles fonctionnalités ou être difficiles à gérer.

Le projet « Salut à Toi » faisant un usage intensif des fonctionnalités PubSub de XMPP, 2 solutions étaient envisageables :

  • se concentrer sur un serveur XMPP particulier et le recommander, voire le faire évoluer dans la direction voulue. Ceci reviendrait à se bloquer sur une implémentation particulière d'un serveur XMPP et risquer de voir des fonctionnalités fonctionner incorrectement ou pas du tout sur les autres.

  • faire un composant indépendant du serveur, en utilisant des extensions XMPP pour avoir un accès privilégié au serveur

C'est la deuxième option qui a été retenue, SàT PubSub utilise les XEPs (extensions XMPP) Namespace Delegation (XEP-0355) et Privileged Entity (XEP-0356) pour pouvoir fournir des fonctionnalités avancés et PEP.

Bien que « SàT » soit utilisé dans le nom (par son origine), ce composant ne nécessite pas d'utiliser Salut à Toi et peut-être utilisé par tout logiciel XMPP.

SàT PubSub implémentait déjà MAM et RSM (permettant de faire des recherches dans les archives et d'utiliser la pagination pour retrouver les éléments), la version 0.3.0 apporte également :

  • le modèle d'accès presence
  • la gestion de +notify
  • la notion d'administrateurs, qui sont des identifiants (« JID ») avec des accès privilégiés
  • les affiliations
  • la possibilité de retrouver les éléments classés par date de création ou modification avec Order-By (XEP-0413)
  • la fonctionnalité expérimentale « Node Schema », pour associer des types de données à un nœud via les Data Forms (XEP-0004). Cette fonctionnalité est notamment utilisée dans Salut à Toi pour les tickets.
  • la fonctionnalité expérimentale « Serial IDs » qui permet d'attribuer des identifiants d'éléments utilisant des incréments (1, 2, 3, etc.) au lieu de valeurs aléatoires. Ceci est notamment utile pour les tickets.
  • la fonctionnalité expérimentale « PubSub Admin », qui permet aux administrateurs de publier des éléments en spécifiant un autre éditeur (« publisher »). Peut-être utilisé pour restaurer une sauvegarde.
  • la fonctionnalité expérimentale « consistent publisher » qui, une fois activée sur un nœud, garde l'éditeur (« publisher ») original quand un élément est modifié par le propriétaire du nœud ou un administrateur. Ceci permet de modifier l'élément sans empêcher l'éditeur original de le modifier à nouveau lui-même (par exemple en éditant un commentaire de blog, ou en changeant le statut d'un ticket).
  • un fichier de configuration peut désormais être utilisé avec SàT PubSub, évitant d'avoir à spécifier les options – dont le mot de passe – entièrement sur la ligne de commande. Le même fichier que pour Salut à Toi (sat.conf) est utilisé, les options de SàT PubSub doivent être mises dans la section [pubsub]).
  • une nouvelle documentation

Vous trouverez plus de détails dans le CHANGELOG.

Le développement sur la version 0.4 a déjà commencé avec le port pour Python 3 qui est fonctionnel et complète le reste de l'écosystème Salut à Toi.

Pour installer SàT PubSub, il vous suffit d'entrer pip install sat_pubsub dans un environement virtuel Python 2, référez-vous à la documentation pour plus de détails.

E

errormovim 07/10/2019, 22:43

SàT progress note 2019-W33

goffi 18/08/2019, 19:22 jabber-xmpp-en SàT project libre SàT progress

Hi there,

after the release I've got some vacations and I've made a break, that's why I've skipped the last few weeks progress notes, but now I'm back.

The release was a big thing, after 3 years of developments, and many new exiting stuff to show (Cagou, which is also running on Android, the advanced file sharing, photo albums, events, etc.), it is also the first "general audience" version, meaning that it's the first one which is usable for everybody, not only people with technical background. Even if there is still work to do on UX and stability, we're on the right track.

The Android version of Cagou is not stable yet, but there are several things which were not fixable easily under Python 2 (there is notably a bug on file opening with Python for Android which is not reproducible on Python 3). I hope to have feedbacks to make next version really enjoyable.

I was really happy to finally do this release, and now I can move forward and in particular do the Python 3 port.

During my holidays I've spend some time doing the first steps of the port (just a few hours here and there), and I could quite quickly get the backend and frontends running. Once back, this week, I've reviewed it and done some polishing before committing, and I'm happy to say that the development version of the backend and all the frontends now run on Python 3. Some features are still not working, but most things are here and running.

This unlock many things, and I'm very looking forward to used them : asyncio with async/await syntax, Brython, Transcrypt, and many libraries which are Python 3 only.

Thanks to better error handling, I could also already fix some issues not seen with Python 2, and we can already appreciate a performance boost (specially visible with tickets).

Once that done, I've released SàT Pubsub 0.3.0 (a release note will come soon about this), and started the developments of 0.4.0 with… Python 3 port : dev version of SàT Pubsub is now also running on Python 3. That means that all SàT ecosystem is now Python 3 only.

Beside that, I've also made 3 pull requests to see Cagou, Primitivus and jp on Flathub, but there are some modifications to do there before they can be merged.

That's all, see you next week.

E

errormovim 07/10/2019, 22:43

Salut à Toi v0.7 « La Commune »

goffi 24/07/2019, 22:50 jabber-xmpp-en SàT libre project release

Salut to you!

It is with a big pleasure and a sort of relief that I announce you the release of "Salut à Toi" v0.7.0 (La Commune).

To remind you "Salut à Toi" (or SàT) is an ecosystem of decentralised communication, based on "XMPP" established standard.
It features many functionalities (instant chatting, file sharing, blogging/microblogging, events organisation, forums, etc.) and has the particularity of being multi-interfaced (different "frontends" are available for the web, desktop, mobile devices or even the terminal and command line).

The preparation of this version has taken 3 years. This is the first version we can call "general audience", in other words it is approachable also by those without technical background . Though, there are still some improvements to be done in the frontend and "user experience" level.

I shall not enumerate again all the functionalities with screenshots, you can consult the announcement of the alpha version to do so by yourself. I'd rather explain some major updates, which has been done:

Cagou, the desktop/mobile interface

As promised by our modest crowdfunding, Cagou is the name of the new desktop/mobile frontend (Android only at this moment). To share a little story with you, it is a reference to a beautiful bird, which does not fly, but barks, and is endemically present on the no less beautiful island of New-Caledonia. It is also a nod to Kivy, the framework we are using and from which the name and logo may remind you of Kiwi of New-Zeland.

So, this interface is multi-platforms, not focusing only on instant messaging: you can use it for file-sharing (e.g. the videos/photos between your computer and phone), or as a remote-control for your media player. It is of course planned that it will be able to cope with blogging in the close future.

Cagou is thought to be usable on a little screen, as well in full-screen mode of desktop, and allows the screen division into zones - to, for instance, follow several discussions at the same time.

On android, the application has still some problems with the reactivity and bugs time to time. Lot of problems will be corrected by with Python 3 port. Consider this version as the first one to have you commentaries et propositions.

Cagou sur Android

End-to-end encryption

Of course, SàT has been featuring end-to-end encryption for several years already through "OTR", however this version witnesses the arrival of "OMEMO", an algorithm correcting the issues of previous one (specially allowing sending of offline encrypted messages, or to display them on multi-devices). OMEMO is implemented only for simple conversation ("1:1", between 2 people) at the moment, but the next version will surely cope with end-to-end-encrypted conversations within groups.

Events, photo-albums, forums, ticket-handling, merge-requests

Numerous new functionalities has appeared in this version. You can now create and manage events (e.g. for the family) with the classic list of invited persons et the replies of type "RSVP" (attending, not attending, maybe attending). It is possible to invite people, even though they don't have XMPP account - using the "invitation" accounts which are automatically generated and sent to their emails.

You can create and share the photo albums, a specialisation of file-sharing. The possibility to create one is not yet available from graphic interface, however, this is planned to be improved soon. To consult the photos is simple from Libervia (web interface).

The forum is also in the party, as well as the tickets handling and merge requests. These two last functionalities have been implemented for the project needs, but they are very flexible and can be (in the next version) easily used in a everyday life (e.g. as a shopping lists, to do lists etc.)

All of this take profit from XMPP "PubSub" capabilities, and can benefit of its permission system (we can imagine the shopping list being shared between the family members indicating who bought what).

Remark - to actually benefit from all this it is necessary to use the "SàT Pubsub" service, the project made for the needs of Salut à Toi (but which can be used by all XMPP based programs)

un blog sur Libervia

album photo sur Libervia

Web Framework

The development of Libervia, the web frontend of SàT, has evolved to make it become a web framework. The reason of this evolution is the need of a very flexible interface, one which would allow implementation and testing of new ideas and functionalities easily. The goal is to have a naturally decentralised and federative framework (thanks to XMPP), which integrates simply to the ecosystem. I will speak only briefly about the technical details, but the point is to join Jinja2 with SàT and use PubSub as the database. Also CLI frontend (jp) allows to use the same models to make a static rendering (for instance to generate a static blog or chat archives).

It is with this framework that these above mentioned features were developed, they are organised in "pages" which are supposed to be simple to use and can work without JavaScript (when possible, it is not the case for the chat). The new official website works thanks to the new framework, you'll find there an introductory documentation.

Even More

I won't get too much into details as the updates are too numerous, but though it it worth mentioning that SàT can be also used to store your folders on the server (it can be used as a "component"), or jp, the command line frontend allows manage a lot of things (find or publish an article or blog, send encrypted or non-encrypted message, find someone's avatar, etc.)

Installation

Salut à Toi is available on Debian and its derivates, but attention! It is only the case for the backend, the console interface and the command line. By the way, a help to make the Cagou package land there too would be much appreciated.

It is also available on the Arch Linux's AUR repository, and you'll also find there the development versions.

The Flatpak packages allows the easy installation of Cagou, Primitivus (terminal) and jp (command line) on most of GNU/Linux distributions, you can find the links on the main page of the site.
You can of course, use as well pip, the Python packages manager. A simple pip2 install --user sat followed by pip2 install --user cagou and pip2 install --user libervia should be enough. The instructions are available in the site's documentations.

For Android, you can find a APK on this link (unsigned, a definitive version will follow in a couple of days). After the release, I will make steps to make it available on F-Droid, and eventually the "play store".

Even though SàT should technically be working on Mac OS X, Windows and *BSD, however on these platforms it is not yet tested (as I do not have any of this devices). I have had some feed-backs concerning Mac use. If you are interested, help with testing and packaging would be nice.

Future

The principal planned development of 0.8 is the port to Python 3, which is now finally possible, as no more dependency is blocking, and it is the only thing promised for the next version. Lot of big steps are to follow, and it will be inevitable to make some choices. Don't hesitate to give me opinions/feedback in the commentaries.

Video conference

With Jingle already implemented, the video conference has been planned for a long time. It shouldn't be too difficult to be implement it into the web interface (thanks to WebRTC), but certainly would need a lot of work on desktop/Android (to evaluate different options, GSTreamer is the hot candidate at the moment, and integrate with Cagou). This is an important work.

Improving the file sharing

SàT already allows advanced file sharing (more than a simple sending of the files to the server), including a server component. It is very possible, that the future developments will continue in this direction.

share a directory with Cagou

iOS Version

It would be technically possible to use Cagou on iPhone. Though, there are several obstacles to cope with - especially the legal ones (Apple store is not compatible with the current licence, AGPL 3+). Lot of time and investments would be needed. This is clearly not the priority, but keep in mind that an iOS version is doable.

ActivityPub Gateway

With this version, SàT can be used as a "component", i.e. a server service. This possibility can be used to build an ActivityPub gateway, that would allow 2 ways communication with projects using this protocol. There is already somebody working on a similar gateway for Prosody, , so I will certainly wait to see how it evolves before starting it by myself.

Chat Evolution

Though its already functional, the chat can become very complete if we take the time for it. Here, we speak of adding some missing features (like reactions or editing of the last sent message), improving the file handling, copy/pasting of code, end-to-end group encryption, etc.
All this is on the roadmap, the question is whether this is a priority or not.

Improvement of User Experience

This is probably the project which will have the priority, once Python 3 port is done. SàT has already lot of features, but the interfaces need some work to become really user-friendly. I would specially like to work on launch screen and contacts discovery. There are as well lots of "small details " which all together take lots of time, but in total make the user-experience much more pleasant: integration to desktop, file-sharing actions, easier file selection, etc.

Well, so much for the future steps, all of this would take time, but again - your feedbacks are highly welcomed. There are other ideas to think about, but one has to choose her priorities.

But…

Salut à Toi is an huge project with a potential - and it is currently developed by a single person. I have currently only one day a week dedicated to SàT (besides mornings, evenings and nights), the rhythm is very difficult to cope with. In the next months I will seriously study the projects financing possibilities. It will be necessary that I find a way to keep the project on track.

Useful Links

  • official website: https://salut-a-toi.org
  • documentation: https://salut-a-toi.org/documentation
  • my blog: https://www.goffi.org (I publish there weekly progress notes)
E

edhelas 24/07/2019, 23:00

debacle 30/07/2019, 12:41

Salut à Toi v0.7 « La Commune »

goffi 24/07/2019, 22:49 jabber-xmpp SàT project libre release planet-libre seenthis

Salut à vous,

c'est avec plaisir et un certain soulagement que je vous annonce la sortie de « Salut à Toi » v0.7 (La Commune).

Pour mémoire « Salut à Toi » (ou SàT) est un écosystème de communication décentralisé, basé sur le standard établi « XMPP ».
Il vous permet de faire de nombreuses choses (messagerie instantanée, partage de fichiers, blogage/microblogage, événements, forums, etc.) et a la particularité d'être multi-interfaces (différents « frontaux » sont disponibles pour le web, le bureau, les appareils mobiles, ou encore le terminal et la ligne de commande).

La préparation de cette version aura pris 3 ans, c'est la première version dite « grand public », autrement dit qui est utilisable par un public qui n'est pas forcément technique, même s'il y a encore des améliorations à faire au niveau de l'interface et l'« expérience utilisateur ».

Il n'est pas question de refaire une longue énumération de fonctionnalités avec captures d'écrans, vous pouvez vous référer à l'annonce de la version alpha pour ceci. Je vais plutôt expliquer certaines nouveautés majeures :

Cagou, l'interface bureau/appareils mobiles

Promis suite à notre modeste campagne de financement participatif, Cagou est le nom du nouveau frontal bureau/appareils mobiles (Android uniquement pour le moment). Pour la petite histoire, c'est une référence à un superbe oiseau qui ne vole pas et aboie, endémique à la non moins superbe Nouvelle-Calédonie, mais aussi un clin d'œil à Kivy, le cadriciel que nous utilisons et dont le nom et logo rappellent le Kiwi de la Nouvelle-Zélande.

Cette interface est donc multi-plateformes, et ne se contente pas de la messagerie instantanée : vous pouvez vous en servir pour transférer des fichiers (par exemple vos films/photos entre votre téléphone et votre ordinateur de bureau), ou comme télécommande pour votre lecteur de médias. Il est bien entendu prévu qu'il gère également le blogage dans un avenir proche.

Cagou est pensé pour être utilisable aussi bien sur un petit écran qu'en plein écran sur un grand moniteur de bureau, et permet de diviser facilement l'écran en zones pour, par exemple, suivre plusieurs salons de discussion en même temps.

Sur Android l'application souffre encore de quelques problèmes de lenteur et même parfois de plantages. Plusieurs problèmes seront corrigés avec le port sur Python 3, aussi considérez cette version comme un premier jet qui permet d'avoir vos commentaires et suggestions.

Cagou sur Android

Chiffrement de bout en bout

Bien que SàT gère le chiffrement de bout en bout depuis plusieurs années via « OTR », cette version voit l'arrivée d'« OMEMO », un algorithme palliant des défauts du précédent (il permet notamment d'envoyer des messages chiffrés hors ligne, ou de les afficher sur plusieurs appareils). OMEMO n'est implémenté qu'en conversation simple (dites « 1:1 », entre 2 personnes) pour le moment, mais la prochaine version verra certainement la gestion des conversations de groupes chiffrées de bout en bout.

Événements, album photos, forums, gestion de tickets, requêtes de fusion

De nombreuses fonctionnalités font leur apparition. Vous pouvez désormais créer et gérer des événements (familiaux par exemples), avec une classique liste d'invités et de réponses types « RSVP » (je viens, je ne viens pas, je viens peut-être). Il est possible d'inviter des gens même s'ils ne sont pas sur le réseau XMPP en utilisant des comptes « invités » générés automatiquement et envoyés à des adresses de courriel.

Vous pouvez créer et partager des albums photos, une spécialisation du partage de fichiers. La création n'est pas encore disponible depuis les interfaces graphiques, c'est une amélioration à prévoir rapidement, la consultation par contre est simple depuis Libervia (l'interface web).

Un forum basique est également de la partie, ainsi qu'une gestion des tickets et requêtes de fusion (« merge requests »). Ces 2 dernières fonctionnalités ont été implémentées pour les besoins du projet, mais elles sont très souples et pourront (dans une prochaine version) facilement être utilisée dans la vie courante (par exemple pour des listes de courses ou de choses à faire).

Tout ceci tire parti des capacités de publication/abonnement (ou « PubSub ») de XMPP, et peuvent profiter de son système de permissions (on peut ainsi imaginer une liste de courses partagée entre les membres de la famille, pour indiquer qui a pris quoi).

À noter que pour en profiter il est actuellement nécessaire d'utiliser le service PubSub « SàT Pubsub », un project fait pour les besoins de Salut à Toi (mais utilisable par tout logiciel XMPP).

un blog sur Libervia

album photo sur Libervia

Cadriciel (framework) web

Le développement de Libervia, le frontal web, l'a mené à devenir un cadriciel (ou « framework ») web. La raison de cette évolution est qu'il fallait une interface très souple, permettant d'implémenter et tester facilement de nouvelles idées ou fonctionnalités. Le but est d'avoir un cadriciel naturellement décentralisé et fédéré (parce que basé sur XMPP), qui s'intègre simplement dans l'écosystème. Je passe rapidement sur les détails techniques, mais il s'agit de lier le moteur de rendu Jinja2 avec SàT et d'utiliser PubSub comme base de données. En outre, le frontal en ligne de commande (jp), permet d'utiliser les même modèles pour faire un rendu statique (par exemple pour générer un blog statique ou des archives d'un salon de discussion).

C'est avec ce cadriciel qu'ont été développées les fonctionnalités mentionnées ci-dessus, elles sont organisées en « pages » qui se veulent simple d'utilisation et peuvent fonctionner sans javascript (quand c'est possible, ce qui n'est pas le cas pour le « chat »). C'est grâce à lui que fonctionne le nouveau site officiel où vous trouverez une documentation d'introduction.

mais encore

Je ne vais pas trop m'étendre, les nouveautés sont trop nombreuses, mais il vaut tout de même la peine de mentionner que SàT peut également être utilisé pour stocker vos fichiers sur le serveur (il peut être utilisé en tant que « composant »), ou que jp, le frontal en ligne de commande permet de gérer beaucoup de choses (retrouver ou publier un article de blog, envoyer un message chiffré ou non, retrouver l'avatar de quelq'un, etc.).

Installation

Salut à Toi est disponible dans Debian et ses dérivées, mais attention c'est le cas seulement pour le backend, et les interfaces console et ligne de commandes, de l'aide serait d'ailleurs apprécié pour empaqueter Cagou (interface bureau/appareils mobiles) et Libervia (interface web).

Il est également disponible sur les dépôts AUR d'Arch Linux, et sur cette distribution il existe des paquets pour installer les versions de développement.

Des paquets Flatpak permettent une installation aisée de Cagou, Primitivus (console) et jp (ligne de commande) sur la plupart des distributions GNU/Linux, vous trouverez les liens sur la page principale du site.
Vous pouvez bien sûr aussi utiliser pip, le gestionnaire de paquets de Python. Un simple pip2 install --user sat suivit d'un pip2 install --user cagou voire d'un pip2 install --user libervia devrait suffire. Les instructions sont disponibles dans la documentation du site.

Pour Android, vous trouverez un Apk sur ce lien. Je vais faire les démarche pour le rentre disponible sur F-Droid, et éventuellement le « play store ».

Bien que SàT devrait techniquement fonctionner sur Mac OS X, Windows et les *BSD, il n'est pas testé sur ces plateformes (car je ne les ai pas à disposition). J'ai eu quelques retours sur Mac. Si des personnes sont intéressées, une aide pour tester et empaqueter serait plus que bienvenue.

Avenir

La principale chose attendue pour la 0.8 est le port sur Python 3, il est enfin possible maintenant que plus aucune dépendance ne bloque, et c'est la seule chose certaine pour la version à venir. Plusieurs grosses pistes sont envisagées ensuite et il va falloir faire des choix, n'hésitez pas à donner votre avis/préférence en commentaire.

Visioconférence

Jingle étant déjà implémentée, la visioconférence est prévue de longue date. Elle ne devrait pas être trop difficile à implémenter pour l'interface web (grâce à WebRTC), mais demandera certainement plus de travail sur bureau/Android (évaluer les différentes options, GSTreamer étant le candidat envisagé pour le moment, et intégrer ça à Cagou). C'est un chantier important.

Améliorer le partage de fichiers

SàT propose déjà un partage de fichiers avancé (plus qu'un simple envoi de fichier sur le serveur), incluant un composant serveur. Il est fort possible que des développements continuent dans cette direction.

partager un répertoire avec Cagou

Version iOS

Il serait techniquement possible de porter Cagou sur iPhone. Il y a plusieurs obstacles à cela, notamment juridique (l'Apple store n'est pas compatible avec la licence actuelle, AGPL 3+). IL faudrait également du matériel dédié, et du temps. Ce n'est clairement pas la priorité, mais sachez qu'une version iOS est envisageable.

Passerelle ActivityPub

Avec cette version, SàT peut désormais être utilisé en « composant », c'est-à-dire un service côté serveur. Cette possibilité peut être mise à profit pour créer une passerelle ActivityPub, ce qui permettrait de communiquer dans les 2 sens avec les projets utilisant ce protocole. Il y a déjà un projet en cours de passerelle XMPP pour Prosody, aussi je vais certainement attendre de voir son évolution avant de me lancer moi-même.

Évolution de la messagerie instantanée

Bien que déjà fonctionnelle, la messagerie instantanée peut devenir très complète si on y consacre du temps. Il s'agirait d'ajouter les quelques fonctionnalités manquantes (réactions, correction du dernier message), d'améliorer la gestion des fichiers, des copier-collers des morceaux de code, d'implémenter le chiffrement de bout en bout en groupe, etc. Tout ceci est prévu, la question est de savoir si on en fait la priorité ou pas.

Amélioration de l'utilisabilité

C'est probablement le chantier qui sera prioritaire une fois le port Python 3 effectué. SàT est déjà bien fourni en fonctionnalité, mais les interfaces ont besoin de travail pour être facilement utilisable. J'aimerais en particulier travailler sur le lancement de l'application et la découverte de contacts. Il y a également beaucoup de « petits détails » qui accumulés prennent du temps mais rendent le logiciel beaucoup plus agréable : intégration au bureau, aux actions de partage, une sélection plus agréable des fichiers, etc.

Voilà pour les pistes envisagées, chacune demande du temps et encore une fois vos commentaires/avis sont les bienvenus. Il y a d'autres idées qui traînent, aussi il va falloir bien choisir les priorités.

Mais…

Salut à Toi est un gros projet qui a un potentiel, mais il est pour le moment développé par une seule personne. Je n'ai qu'un jour par semaine consacré à SàT, et le rythme est très difficile à tenir. Dans les mois qui viennent je vais étudier les options de financement du projet. Il va être nécessaire que je trouve une solution à moyen terme pour que ce projet reste sur les rails.

Liens utiles

  • site officiels: https://salut-a-toi.org
  • documentation: https://salut-a-toi.org/documentation
  • notes de progression (en anglais): https://www.goffi.org/tag/S%C3%A0T%20progress
E

edhelas 25/07/2019, 06:06

E

errormovim 07/10/2019, 22:43

SàT progress note 2019-W28

goffi 14/07/2019, 13:21 jabber-xmpp-en SàT project libre SàT progress

Hello,

remember last week when I was mentioning external things that I've put on hold? There was a Twisted ticket that I've opened months ago, I've made a pull request for that, but there was still some changes to do according to review.

Unfortunately, between my paid job, work on SàT, and real life things I haven't had the time to update it, and little after sending my last week note, I've got an email notification about the ticket being fixed (with a patch replacing mine). I was feeling a bit bad to let this for months and not finding the time to update it myself (in 8 months). Fortunately Twisted folks are nice and understand well ("No need to apologize. We're all volunteers here :)"), and it's a good thing that they managed to fix this issue.

I'm telling this because people don't always understand why things take time (why the release it taking so long? Why nobody did review my merge request?), or may be forgotten. It's just that people can be really really busy, and in free software communities, many of them work on their free time.

During this week, again, it was debugging: really useful for the project, but not so much to tell or explain. The blocker list is getting really low, and I feel fairly optimistic that release can be done next week.

There is maybe one issue I've come through that may worth a mention : while the new chat page of Libervia was working fine during my tests on localhost, it was not working at all once deployed. This is a really simple chat for now, just a proof of concept before adding advanced feature in next version. Libervia pages which come with the new web framework are currently mostly static (i.e. there is no advanced javascript to handle complex changes, this will come in next version), and the chat works by establishing a websocket with the pages, and just sending new messages to display. The web framework is supposed to make life of developer easier, and you only have to declare a dynamic = True variable to activate the websocket. In the background, Libervia create the websocket, associate the page with it, and also the request which is an object containing many useful data, including the session with the profile of the user.

While investigating, I've realised that the issue was happening only in HTTPS, so it was not visible while working on http://localhost. Happily, I'm now working mostly in HTTPS even during local development, this make the detection of this kind of issues easier. I've been using mkcert which makes the process of adding personal certificate painless, and thanks to it I can work easily in HTTPS on my own private and local domain.

This led to an issue with the request object I was storing. This Twisted object is not made to be stored this way, but I was doing it to keep the useful data it contains. Once the HTTP GET request used to retrieve the chat page is finished, the channel is closed, and it is used to test if the request is secure or not (the Twisted session is not stored in the same object or using the same cookie if the request is secured or not). As a result I was getting an empty session, like if the user was not connected, and the chat could not work. I could easily work around this issue, and I've reworked this in a cleaner way; it's working fine now.

The release is not totally ready yet, but I'm already thinking about what's coming next (beside Python 3 port), and there are several big options in my mind, it will be important to choose carefully the priorities. I'll probably focus soon on user experience (UX), as current UI (specially for Cagou), is not that intuitive and accessible for newcomers. I'll write more about this for the release note (which may replace next week progress note).

debacle 15/07/2019, 17:01

E

errormovim 14/10/2019, 21:06

SàT progress note 2019-W27

goffi 05/07/2019, 10:16 jabber-xmpp-en SàT project libre SàT progress

Hello,

this week note will be short, first because I've mainly fixed bugs, second because I've pain in my wrist and it's not comfortable to write.

The most visible part of this week work is the addition of some installation instructions on the front page of https://salut-a-toi.org, which render like this:

capture of installation icons from https://salut-a-toi.org

(this is a screenshot, this image is not clickage, check https://salut-a-toi.org to see the real clickable version)

Thanks to the work explained in previous notes, we're getting closer to an easy installation in a few clicks. We're not exactly there yet though, as Flatpak still needs to be installed in most distributions (I think, I don't know exactly where it's installed by default), Libervia (web frontend) is not installed this way (but I'll update the Docker image at some point which is also easy to install/run), and there are maybe other tweaks to do (for instance on Android you must activate the installation from external sources). Please note than when SàT is available natively in the distribution, it's the recommended way to install it.

Also the many ways to install + different frontends/versions (and I've not put yet Libervia + packages for others platforms like Windows or Mac OS) may be confusing. Maybe a platform detection could help to highlight the most probable package in the future.

Another thing which can cause trouble with SàT specific architecture is the backend: for Flatpak I've made a wrapper to launch it automatically if it's not already, otherwise the running backend is used. It works but there is a risk of troubles when the package will be updated and the running backend version may be older than the expected one. I'll have to modify the backend to check that at some point (probably in 0.8).

An other necessary improvement for 0.8 will be the login screen: for now it's really confusing for somebody not used to XMPP (you have to create a profile with a jid and a password, then click on it, then you land to widget selector). I have several ideas on how to change that, but I'll work on it only after release and python 3 port.

That's all for this week note, as I've said in the first paragraph I've mainly done debugging which is not really interesting to explain here. I'm a bit annoyed to lack the time to do more, as I've put aside some external things that I need to finish: I have 2 XEPs to update, and a pull request for Twisted that I've opened months ago and I need to update too. I haven't had much time either to follow standard mailing list, which is a pity because some really interesting things are happening there like the long awaited full stanza encryption.

I hope to be able to fix enough bugs in the incoming week to finally release the 0.7, I'm looking forward for it, and I'm sorry that the release will happen during the summer when many people will be on holidays (well I'm happy for people being on holidays, it's just that it's not the best time to have the release being noticed).

E

edhelas 05/07/2019, 10:26

SàT progress note 2019-W26

goffi 28/06/2019, 07:28 jabber-xmpp-en SàT project libre SàT progress

Hello everybody,

I was targeting end of June for the release, it seems that I'll have to postpone: it's not ready yet, there are still a couple of annoying bugs. That's a really boring part from a developer point of view, you have to take care of polishing, and things around the code (like the website, documentation or packaging), and you don't see your project evolving much. It's far more exciting to work on new features or experiments, but you need to go through polishing if you want something usable.

I've been talking about Flatpak last week, I still had to do some work on it.

After getting working packages for Cagou, Primitivus and Jp, I've created a repository for that (https://salut-a-toi.org/flatpak/sat-repo) and the flatpakref files which are metadata telling where to find and how to install the packages. I was happy to finally have a simple way to install Cagou so people could test it and give feedback, so I've sent a short notice on Diaspora and ActivityPub. I've indeed quickly got feeback reporting the outdated runtime (1.6 while 18.8 is now available, I've just missed it while working with my Manifest from last year), then I had to update the package again.

Once again, feedback has been useful (thanks JBB), so please if you see something buggy or you would like to see something (new feature or anything), let me know!

During my own test I realised that the MIME type for the flatpak refs files was wrong too (it was text/html so the browser was displaying them instead of downloading them). That was a bad default value used in Libervia, and it's now fixed.

After updating the runtime, testing it successfully without installation (using flatpak-builder --run build-dir org.gimp.GIMP.json sat_wrapper) and uploading it, I've had the bad surprise to realise that python2 was missing when trying to run the package normally. It is actually available in the SDK so it was working while testing with flatpak-builder, but not anymore in the runtime (which make sense as Python 2 support will be stopped next year, but still SàT is using it and will move to Python 3 for 0.8 release). So I had to compile Python 2 myself, and make several try to make it work correctly (and that's only after it worked that I have discovered that there was a pre-made module offered by Flatpak teams).

All of that to get this thing:

Cagou flatpak package seen on KDE Discover

Integration in the package manager (here Discover from KDE) is nice to have, and I hope that it will be the case by default in most distributions in some time. I'll propose the package on Flathub (the central repository of Flatpak) once the stable release is out.

There are still some little issues on the package/metadata (like the screenshot which are not at the recommended 16:9 format), but I'll take care of them later, I have already spent enough time there and I need to move forward. Furthermore, the screenshots seem to be displayed correctly beside their format on KDE Discover and GNOME Software.

Far more serious, there are some crashes on the Flatpak version of Cagou (I've had a infinite recursion crash when clicking on bookmarks) that I don't have on normal packages, so I have to debug that but that's part of the general debugging phase I'm now in.

To install dev preview packages of Cagou, Primitivus or Jp from Flatpak, just enter the following commands:

  • flatpak install --user https://salut-a-toi.org/flatpak/org.salutatoi.Cagou_dev.flatpakref
  • flatpak install --user https://salut-a-toi.org/flatpak/org.salutatoi.Primitivus_dev.flatpakref
  • flatpak install --user https://salut-a-toi.org/flatpak/org.salutatoi.Jp_dev.flatpakref

And later to update them:

  • flatpak update

Or use your graphical software manager. I'll add the links to the website soon, so it should be installable in a few clicks.

Remember that it's dev version, so expect bugs, and in addition there are still some issues specific to the Flatpak version that I'll try to fix today.

That's really a lot of time spent on those packages, but I now have scripts which will help to updates everything easily. Imagine if I had to do all packages myself for every GNU/Linux distributions + Docker + Flatpak + AppImage + Snap + YunoHost + Android + *BSD + Windows + Mac OS + … . I'm already doing several on this list, but I can't do everything alone, so if you feel you can help on packaging, please contact me (see address at the bottom of https://salut-a-toi.org or join us in our official room at sat@chat.jabberfr.org or via web). Note that there are already people working on Arch packages and on Debian and derivatives (but they need help).

So now it's all about debugging and documentation until the release.

E

edhelas 28/06/2019, 07:29

SàT progress note 2019-W25

goffi 20/06/2019, 15:58 jabber-xmpp-en SàT project libre SàT progress

Hello everybody, time for a new progress note.

This week I've been working with Flatpak. Flatpak is a new way to package applications using containerization, which has pros and cons: for a low overhead the applications can be run on most GNU/Linux distributions, they run in the same reproducible environment, and they have limited access to outside. I prefer using software packaged in my distribution when they're available (better integration, no overhead at all), but Flatpak is a nice way to try quickly something or to have access to software not packaged in our distribution.

About SàT, the main goal is to have super simple installation so people can try Cagou (desktop/mobile interface) easily. I'm also working on packages for other frontends mainly jp (CLI) and Primitivus (TUI), so they can be tested quickly. I hope this can encourage people to try them, and help gain some visibility.

Let's talk technical now. I have already worked on Flatpak nearly 1 year ago, and could launch several frontends using hand made manifests (the files instructing Flatpak how to build your application). I didn't found the time to polish it and it has never been made public.

This time, I wanted to have something as automated as possible: there are several frontends, and dependencies may change, I don't want to have to do everything by hand each time I update the packages.

The main difficulty when building flatpak, is that you have to deal with what is available in the "runtime" (the base of the distribution you'll use), and without network (you can't make internet connection during building process). In the case of Python, that means that you have to download every dependencies (including their own dependencies, etc.) and then install them in the right order (if you have wrong order, a module installation will fail because of missing dependency). This task is more complicated that it seems.

Flatpak ecosystem propose a script to make this task easier: Flatpak PIP Generator. It retrieves dependencies of a Python package, download them, generate the hash and retrieve download URL to prepare a list of "modules" usable in the manifest. The problem is that the list is not ordered according to dependencies, and the JSON produced is not directly usable. I quickly moved to make my own script specially tailored for SàT needs.

The first issue with dependencies is that there is no good way to find them. They are normally declared in the setup.py script which is the standard way to install a Python package, but you can't import this script to analyze it (it would execute its code and the usual if __name__ == '__main__' way to avoid that is never used with setup.py).

Well you have PyPI doing the analyze for you, and if you use its JSON API you can retrieve the mandatory requirements + optional ones using the ['info']['requires_dist'] field of the metadata object. Except that this field is far from always being complete, for instance it's not set for Twisted (I have not investigated for the reasons, maybe because Twisted is using an other file dynamically read). Twisted is far from being the only package without this data, so it's unusable in my case.

I've searched a bit on Internet and made different tries, and at the end the best I've found to reliably get the dependencies used by a Python project is to install it in a virtual environment and to parse the output of pip show <package_name>.
This command output a text with a Requires: line showing the actual requirements used in this project. Thanks to that I could solve the dependencies hell problem and my script is able to order correctly and automatically the modules needed in my manifest. The only modules not managed are setup requirements (requirements needed only for building the modules), but there are only a few of them and it's easy to specify them by hand.

I'm actually doing it in two times: first I download (using pip download) the requirements of the Python package to get the list and version needed, then I install the package in a virtual environment to analyze the output of pip show (pip handle a cache, so packages are not downloaded twice).

So my manifest generation script which I was expecting to do quickly took actually most of my free time this week and it's still not finished. I'm adding other options and I have to faces troubles likes Mercurial not being managed by Flatpak (for this case I have to retrieve myself the repository and upload it manually). I'm also sparing you other troubles (like how to organize the packages, SàT being a specific architecture I've first made my own runtime before being discouraged of doing this on the Flatpak mailing list).

At the end it's a lot of time spent just to make installation easier, and I have to do everything myself. I hope this kind of stories I'm explaining in my notes helps people to understand why it takes so long to develop a software, specially when you're on your free time. I hope also this work will help to popularize the project, and, maybe, bring some helping hands :)

See you soon.

SàT progress note 2019-W24

goffi 13/06/2019, 07:07 jabber-xmpp-en SàT project libre SàT progress

As mentioned in my note 2 weeks ago, I was in holidays and skipped last week note.
It has been a really good thing, a relief, to be out of the screen for a couple of days, I recommend!

So I'm now back to work in my paid job, and working again on SàT on my free time. We are already in the middle of June, and the release is still not done; it's time for the home straight. As you can see if you follow progress notes, I'm working regularly, and moving forward each week, but there is so much to do. Anyway things are getting better now, the new website is online with documentation, the tickets system is working even if still buggy (it's getting better with time), and I start to have feedbacks, meaning that people are testing.

During my holidays I've got a report on ActivityPub about an issue with <link> elements, followed by a bug report. With those informations, I could spot the issue and fix it (it was a problem with caching, the data used in template were not deeply copied, and the list containing links was reused and growing each time the page was accessed). Thanks to Rydroid for reporting the issue.

When you test and see something wrong, the right thing to do is to contact me, and if possible report a bug on https://bugs.goffi.org. If it's easier for you, you can contact me by other means, but the ticket creation is the better way to be sure it's not forgotten.

I've finished and deployed the pubsub experimental option I have talked about in a previous note. I've called it consistent publisher and the goal is to keep original publisher if a node owner or an admin modify an item. The main reason is to maintain permission to edit the ticket for the original publisher, and thanks to that I can now change status of a ticket without replacing publisher.

Overwriting an item is not yet ideal in pubsub: either you follow XEP-0060 and any publisher can overwrite any other publisher item (not great in case of tickets or blog comments), or you restrict overwriting to your own published items but you need to do a workaround like this. I have ideas on how to improve the situation with node options and I'll probably propose a XEP at some point after SàT release.

As you know, I'm focusing on debugging, and I've spotted a couple of annoying bugs that I'll fix in the incoming week. My target is a release before the end of June. I have started work on packaging Cagou for desktop with Flatpak, so it should be easy to install and test on most GNU/Linux distributions, I'll probably talk about that in next week note.

E

edhelas 13/06/2019, 07:07

debacle 14/06/2019, 12:03

SàT progress note 2019-W22

goffi 31/05/2019, 15:06 jabber-xmpp-en SàT project libre SàT progress

Hello everybody,

during this week I've put online the new official website at https://salut-a-toi.org.

This is an important step for several reasons, let me explain.

The former site (which is currently still available at https://old.salut-a-toi.org, not sure yet how long it will stay here, but it won't be maintained anymore) was made with Django + bootstrap. It was working and not so bad, but outdated (with a really old screenshot on the front page), and it was a pain to maintain it in parallel of the work on SàT
Django is great, powerful, with a big community, but using it only for our presentation website means that we have to follow its evolution, and we have to solve issues from time to time, e.g. when we update the version. Django template engine is similar to Jinja2 that we use, but it's different enough to have to check documentation to maintain it, not a big deal, but some work that we could avoid.

By moving the website to our own Libervia framework, we stay on the exact same stack as SàT itself, so we don't have to follow other frameworks/libraries, it's easier to maintain. Also it's a good use case to test the framework, and make it evolve according to real life use case.
Furthermore, it simplifies our life for some features, like the news available at https://salut-a-toi.org/news, which is an XMPP blog. With Django we were using a iframe displaying a Libervia instance, while now we can directly show the blog.

With the new website I have added a "documentation" section. So far the documentation was mostly on the wiki, but it was not well maintained and a bit messy. Now it's inside the project itself, and written using reStructuredText which is the markup used most of time with Python project. Also we use Sphinx to render it nicely. As a result, we have a better rendered documentation, which is closer to the project (we can even use source code itself and its docstrings) so it should be up-to-date, and we can use the same source to produce things like man pages or a PDF manual.
Having to render the documentation with Sphinx, was the main reason to create our task manager in Libervia Web Framework.

The documentation is far from being complete, but I'm working on it. You can already see how to use jp for blogging or sending end-to-end encrypted message.

Now the TODO list for the release is done. What is left is debugging, writing documentation and packaging… And some rest! I'm currently in holidays so there will probably not be any progress note next week.

debacle 11/06/2019, 07:36

D

nik 12/06/2019, 07:18

Side note: You can simply switch Django to Jinja2 with one setting.

goffi 13/06/2019, 06:21

Nik: indeed, but I would still have had to convert existing templates, and keep up-to-date with Django evolutions.