SàT progress note 2020-W53

goffi pred 4 mesiacmi jabber-xmpp-en SàT project libre SàT progress

Happy New Year!

It's been a long time since the last progress note, and things have been moving.

First my apologizes for not releasing SàT 0.8 at the end of 2020 like I was expecting: I'm willing to polish some features before releasing, notably in Libervia (web frontend), and I thought that it was best to wait a few more months. I'll try to release more often in the future.

Let's have a high level overview of what has been done since last progress note.

invitations

I've been working on Libervia UX for invitations. SàT implements an invitation system to easily share an activity (photo album, event, or anything). Here "sharing" means sending a notification, and giving access to somebody. The invitation can be sent to somebody in your contact list by entering some letter of his/her name, somebody external by providing the full jid, or somebody without XMPP account by sending an invitation by email. It's only a few click, and access can be removed just by clicking on the deletion cross. I believe that this is a major feature to use easily this as a familial social network.

Photo album invitation in Libervia

Docker integration

You're probably wondering what Docker has to do with an XMPP client (installation put aside). Well, the are several reasons why it's really useful, the first one is to integrate third party software into frontends like Libervia.

As you may know, we are dogfooding most of the development tools with SàT and XMPP (things like tickets/bugs reports are managed with it). So far, there was no translation application used, and translations were done using desktop apps like Gtranslator, which is good but not so easy to install for contributors. With the new Docker integration, Weblate can be added in Libervia with just the following setting in sat.conf:

menu_extra_json = [
    "sat-app:weblate"
  ]

With this simple setting, a new "translate" menu will appear in Libervia and Weblate will be there:

Weblate integrated in Libervia via Docker

The application needs to be integrated with SàT, this is done in a YAML file: here the official Weblate Docker image is used, and SàT settings are re-used when possible to make the process as easy as possible (for instance the configuration to send email is re-used automatically). Note that Docker is currently used, but the plugin managing this is made in such a way that other tools can be used in the future (maybe LXC, systemd-nspawn, Python's virtual envs or something else).

For now it is "weak" integration, Weblate has its own accounts/login. If one day we have the resources, it would be great to work on deeper integration with single log-in (and maybe contributing XMPP login upstream), and a theme adapted to Libervia's one.

An other application I plan to integrate this way is Jitsi Meet: at the moment I have not the time to work on video conference, and Jitsi is really good, handles multi users calls, and uses XMPP too. Integration of Jitsi is a good way to have video conferences quickly in Libervia, until it's possible to find the time to do a native implementation.

End to end tests

The second reason why Docker integration was useful is for end-to-end tests. Historically SàT has been tested with Twisted's trial tools, and there was a Buildbot installation. But with time and lack of resource, this has been unmaintained, and it would have been lot of work to put that back in shape.

Thus I've restarted fresh and I'm now using the popular pytest framework. I've decided to focus on end-to-end tests as it's a way to check the whole ecosystem (including SàT PubSub). For now, tests are done with "jp" (the CLI frontend) and Libervia. For the former the sh module is used with the help of pytest's fixtures, this way tests are really easy to write. For Libervia, I've gone with Helium which is a high level module to use Selenium. The only worry I have is that I'm not sure if Helium will be maintained on the long run, but it should not be a big deal to switch if necessary, and it makes some things easier (notably drag and drop simulation).

Above that there is a little script to make it easy to run the tests, with an option to launch a VNC viewer to follow Libervia tests in real time (check documentation for details). I have long term plan to integrate that in Libervia next the to the source code, in a way similar as what you may see on popular code forges (like Gitlab).

That was also the occasion to rework SàT and Libervia's Docker images. There is still some polishing and documentation to do, but it should be fine for the release.

Full-Text Search for PubSub

One of the last major feature I wanted to implement before the release is Full-Text Search in PubSub. This is using the namespace specified in XEP-0431 and implementation is done in SàT PubSub (the generic PubSub component developed in parallel of SàT to be sure to have all needed features). You can see the result on this blog with the new "search" bar.

the new search box use PubSub Full Text Search

PostgreSQL's FTS engine is used, and it is possible to specify the language of the content. For instance if a blog is known to be in English, you can set the fts_language setting of the node to english. This is needed for improved results because it lets PostgreSQL use the right canonical form of the word. The canonical form of a word is a common root used for variations (like singular/plural, or verb conjugation). Thanks to this, the search will return articles which contain progress note in this blog, even if you search for progresses in plural form.

By default a generic language is used (which correspond to PostgreSQL's simple dictionary), which works with everything but has lower results.

The implementation has been thought to make it possible to override the language used per PubSub item (but it is not yet used). This will be useful for multilingual pubsub nodes like this blog (which is written in French and in English).

The neat thing with having that in PubSub is that it is now available out of the box for all PubSub based feature. So Full-Text Search is now also available in ticket handlers too (like SàT's bug tracker).

SàT advanced features are now usable with generic PubSub services

For features like tickets handlers or merge requests, a SàT PubSub specific implementation has been initially done to manage "Node Schema", i.e. a way to attach a data form template to a node (to indicate how data are organised, and to reject invalid items).

After announcing the ticket feature years ago, I've got a message mentioning XEP-0346 which I've missed at the time. This XEP has the advantages to work without modification on a generic PubSub service. It is now implemented in SàT and SàT PubSub, and thus all features should work with any standard implementation of PubSub. SàT PubSub is still recommended though, as it implements features like Order-By or serial ids which improve user experience and are rare or nonexistent at the moment in other implementations.

many other things

I won't list all the things done since last progress note since there are too many. But in brief, I can tell that XEP-0353: Jingle Message Initiation has been implemented, this is useful to send a file using a bare jid and improves UX a lot, thumbnails of videos are generated (notably useful for photo album), a loading screen is now shown in Libervia to avoid unresponsive actions if JavaScript is not fully loaded, there are new jp commands, bug fixes, etc.

Anyway it's enough for this time, see you soon.

pred 4 mesiacmi