A new print-css.rocks release is available with support for some new tools:

• Typeset.sh
• Vivliostyle
• PagedJS

Unser befreundeter XML-Dienstleister C-REX.net, der für uns in Publishing Lösungen das XML Processing übernimmt, hat einen schönen Artikel über unser Hauptprojekt Onkopedia aus XML Sicht veröffentlicht.

Publikation wichtiger medizinischer Leitlinien auf onkopedia

Umsetzung eines durchgängigen xml-basierten Publikationsprozesses

crexnetc-rex.net

We are building web applications since the mid-90s with Python.

Historically, we have used a bunch of related technologies and frameworks like

• CGI
• FastCGI
• Pyramid
• Flask
• Sanic
• Plone (of course, for CMS-ish applications)

All technologies and web frameworks had their time, their pros and cons (which I do not want to discuss here). Now, we are moving on to FastAPI for new web applications.

FastAPI is one of the most modern Python web frameworks for building applications and services. The primary focus of FastAPI is on micro-services or REST services, but you can of course plug-in your own template language for generating HTML and serving static content.

The reasons for picking up FastAPI are these:

• modern code base (based on the Starlette framework for Python)
• native support for asynchronous operations
• schemas for incoming and outgoing data can be defined using Python type annotations or using the pydantic package for complex data models
• validation of incoming and outgoing data according to their schema definition
• OpenAPI/Swagger integration out-of-the-box
• support for websockets
• good documentation and tutorials
• vibrant developer community
• fully tested
• optional support for GraphQL via graphene

FastAPI can be deployed with any modern AWSGI server like uvicorn. Our personal preference however is hypercorn.

Comparison with Flask? Flask had its time. It is still a good web framework, but you feel its age at all corners (still true with Flask 2). It is in general not a huge problem to move a web application from one framework to another (if it was written properly). I clearly like the built-in features like strong validation of incoming and outgoing data and the OpenAPI support which makes FastAPI the best available option for building micro-services.

Further readings:

• FastAPI: The Right Replacement For Flask?
• Moving from Flask to FastAPI
• Flask vs FastAPI first impressions
• Awesome FastAPI

Today, I released a new version of our Produce & Publish server. The Produce & Publish Server is a REST server that provides a unified API for PDF generation using the PrintCSS approach for a variety of converter tools.

The old codebase has its origins in the years before 2010, based on the Python Pyramid web framework. For several reasons, it was time to trash Pyramid for something better.

The new implementation is based on the FastAPI web framework for Python which gives us out of the box support for building nice REST APIs, data validation, full Python async support and OpenAPI integration.

FastAPI has become meanwhile our primary choice for building REST services on top of Python...and if you are still using Flask for building REST service, then you are doing something wrong.

pp.server

pp.server - Produce & Publish Server

PyPI

pp.client-python

Produce & Publish Python Client

PyPI

Ein Tätigkeitsschwerpunkt von ZOPYX wird in 2021 im Bereich medizinischer Applikationen, insb. im Bereich  medizinischer Publishing Systeme liegen. Als Architekt und technischer Dienstleister der Leitlinien Plattform Onkopedia und anderer Systeme haben wir in den vergangenen Jahre eine umfangreiche Expertise im Bereich Medizin aufgebaut, um auch als kleiner Player im Markt der medizinischen Dienstleistungen (und in dessen Umfeld) erfolgreich agieren zu können.

Im Rahmen dieser Fokussierung und Profilschärfung von ZOPYX haben wir die neue Website Medical Publishing Solutions, die unsere Dienstleistungen und Expertise speziell im medizinischen Sektor bündelt und besser präsentiert.

Unsere medizinischen Publishing-Lösungen erlauben u.a.

• Verlustfreie Konvertierung von DOCX nach XML und zurück
• Konvertierung von XML in alle gängigen Zielformate
• instantanes Publizieren

Durch unsere Publishing-Lösungen versetzen wir unsere Kunden ihre Informationen (z.B. medizinische Leitlinien) in Eigenregie unter Beibehaltung der Autonomie über Daten und Inhalte unabhängig von externen Dienstleistern und Verlagen zu veröffentlichen. Einstellen von Inhalten, Konvertierung und Publizieren erfolgt i.A. als Self-Service Portal ohne weitere Involvierung von externen Dienstleistern oder Verlagen.

Kern-Komponente unserer Lösungen sind das CMS Plone und das eigene XML-Director Framework.

Kontakt
Andreas Jung (info@zopyx.com)
Medical Publishing Solutions

I am moving corporate sites to Wordpress  🤡 -t his is not an April 1st joke!!!

I am currently in progress moving some of my corporate sites and some project sites from Plone to Wordpress. I struggled a long with the decision what to do with my almost ten year old corporate sites. I spend almost two year into static site generators, custom HTML page builders, frameworks like Tailwind UI kit and many more.

I always had an eye on Wordpress of the last years but stability never convinced me. In fact, I could break a fresh Wordpress within minutes...but this has not happened yet with Wordpress 5.6.

As a proof of concept and for getting deeper into Wordpress, I created a new project site (not online so far) for checking various site builder options, extensions and add-ons and deployment options.

Here is my current stack:

• Wordpress 5.6 (installed and maintained through the wp command line utility)
• Porto theme
• Elementor Pro as site builder (feels supernatural compared to WPbakery)
• Smart Slider 3
• UpdraftPlus for backup and restore (stores all data on Dropbox)
• WP Mail SMTP for mail
• Caldera Forms - a super intuitive form builder
• miniOrange 2 for 2FA
• FileBird Lite - for folder support under "media"
• DSGVO AIO for imprint/privacy and cookie alerts
• WPPaint - for manipulating images
• Duplicator - for cloning a WP site
• Matomo Analytics#
• Sucuri Security Suite
• WP Fastest Cache

My current Wordpress 5.6 setup appear rock solid and it's just doing the thing. "The thing" is creating (more or less) shiny websites and focus on content. The available theme for Wordpress leave a lot of room for your own corporate identity. I do not care so much about tiny design details and I don't want to invest a lot of time and money into my "own" theme.

In the end, the decision was based on "getting things done" in a reasonable timeframe and for a reasonable investment.  Of course, Wordpress leaves too many styling options for content creators but that's exactly what Wordpress is for in the first place: building shiny corporate sites, landing pages etc. This is not the primary scope of Plone and the primary scope of Plone can usually not be covered by Plone. So there is no contradiction here using Wordpress as a Plone solution provider.

In the last blog post, I described how to export a Plone site using collective.jsonify to the filesystem as directory of JSON data. This blog post will explain briefly to import the JSON data into ArangoDB as interim migration database

Why ArangoDB? We are using ArangoDB as a document database for storing the JSON data. This allows us to introspect and query the data to be migrated and imported. This is often necessary in complex migration with unknown data and unknown object relations. Using grep or similar tools on a huge JSON dump is unlikely the right approach. Using ArangoDB also allows us to run partial migrations for testing purposes (e.g. for testing the migration on a particular folder or a particular content type).

ArangoDB is multi-model database (key-value store, document store, graph database) and is available as community edition, an enterprise edition and as cloud edition (ArangoDB Oasis). The community edition is sufficient for migration projects. There are various installations options for all operating systems, all Linux distributions and through Docker.

After the installation of ArangoDB, you need to create a dedicated user account within ArangoDB and a dedicated database. This can be easily achieved through the ArangoDB web UI.  We assume for the following steps that you have created a database named plone and an ArangoDB user account plone with password secret.

Preparing your target Plone site

We assume that the target Plone site is running on Plone 5.2 and Python 3. The core functionality is integrated in the collective.plone5migration add-on for Plone. Prepare your buildout like this:

[buildout]
extends = buildout.cfg

auto-checkout +=
    collective.plone5migration     

sources = sources

[sources]
collective.plone5migration = git git@github.com:collective/collective.plone5migration.git


[instance]
eggs +=
    collective.plone5migration

Importing your JSON data into ArangoDB

After running buildout with the configuration given above, you will find this generated import script:

 bin/import-jsondump-into-arangodb --help
usage: import-jsondump-into-arangodb [-h] [-d DATABASE] [-c COLLECTION] [-url CONNECTION_URL] [-u USERNAME] [-p PASSWORD] [-i IMPORT_DIRECTORY] [-x]

optional arguments:
  -h, --help            show this help message and exit
  -d DATABASE, --database DATABASE
                        ArangoDB database
  -c COLLECTION, --collection COLLECTION
                        ArangoDB collection
  -url CONNECTION_URL, --connection-url CONNECTION_URL
                        ArangoDB connection URL
  -u USERNAME, --username USERNAME
                        ArangoDB username
  -p PASSWORD, --password PASSWORD
                        ArangoDB password
  -i IMPORT_DIRECTORY, --import-directory IMPORT_DIRECTORY
                        Import directory with JSON files
  -x, --drop-collection
                        Drop collection

In the former blog post we created a JSON export in I/tmp/content_test_2020-11-25-15-40-59.

You can import this export directory into ArangoDB using:

bin/import-jsondump-into-arangodb -i /tmp/content_test_2020-11-25-15-40-59 -x -u plone -p secret -d plone

Output:

connection=http://localhost:8529
username=root
database=ugent
collection=import
import directory=/tmp/content_test_2020-11-25-15-40-59
truncating existing collection
truncating existing collection...DONE
......

A nice progressbar will should you the progress of the import operation. The import speed depends on your local system. A typical import of 100.000 JSON files (100.000 Plone objects) with a total size of 50 GB takes about 45 to 60 minutes (largely dependent on the IO speed of your disk).

collective.plone5migration has been developed by Andreas Jung as part of a customer project with the University Ghent (migration of the ugent.be site to Plone 5).

In this blog post, I describe to how to use collective.jsonify for exporting a Plone to JSON.

Buildout preparation

• install collective.json (we are using our ugent-fixes branch. This branch will be merged back into the master branch soon)

[buildout]
sources = sources
auto-checkout =  
   collective.jsonify

[instance]

eggs +=
   collective.jsonify

[sources]
collective.jsonify = git git@github.com:collective/collective.jsonify.git branch=ugent-fixes  

Installation of collective.jsonify inside your Zope/Plone installation

bin/instance run src/collective.jsonify/collective/jsonify/scripts/install_migration.py

Run the migration export

bin/instance run src/collective.jsonify/collective/jsonify/scripts/migration_export.py <plone site id>

You will see some output similar to

2020-11-25 15:40:59 WARNING collective.jsonify export >> Directory /tmp/content_test_2020-11-25-15-40-59

Depending on the site of your Plone site and the number of Plone object, the export may take a while. An export on a larger customer site with 100.000 objects and 60 GB data, an export took about 90 minutes...these numbers may vary depending on disk and CPU speed. The export will generate an export directory in /tmp (by default). The directory name is based on the Plone site ID and a timestamp. You can specify a different export directory by setting the $EXPORT_DIR directory.

The next blog post will cover the aspect of setting up ArangoDB as migration data and importing the exported JSON data into ArangoDB.

This is the first of a series of blog posts that deals with Plone migrations. I am trying to tell the story about how we are doing large-scale Plone migrations which usually involves migration of code, switching add-ons and  reorganizing content. This is usually accomplished in our migration through an export of the original Plone site and "magic" importer script on the target system. This migration approach evolved over the years. Our current approach is based on the experience and the work over the last year when we worked on a large-scale migration for the University of Ghent (90.000 content objects, 50 GB of data) and some sites for an industrial customer in Stuttgart.

Migration approaches

In-place Plone migration

Plone 5.2 supports a so-called “in-place” migration which is usually the right choice for simple sites with simple content-types and standard add-ons like PloneFormGen. The database file of Plone 4.3 is migrated in-place and transformed into a Plone 5.2 compliant database file. This is the approach in theory. It is often necessary to perform additional work and additional migration steps to migrate an old Plone database to Python 3 format - in particular when the Plone site has a certain age and history (which often implies a lot of cruft and old configurations that can break or interfere with the standard Plone migration). We use this migration approach only for small and simple sites.

Export/Import using Transmogrifier

The export/import approach allows us to perform a clean export of the content from the old site and to perform a clean import into a fresh Plone 5.2 site. So the new site will not contain any cruft content or obsolete configurations etc. The export of the is usually accomplished using the collective.jsonify add-on for Plone which basically exports all content objects and their metadata as JSON - one JSON file per content object. We also have supplementary code to export additional Plone settings or e.g. the Plone users and groups (if need, unless users and groups are maintained in LDAP or Active Directory or something similar). The exported JSON files are the foundation for a fresh import using the Plone Transmogrifier tool. Transmogrifier is basically a processing pipeline for piping the JSON data into Plone. The import can be customized using so-called “blueprints” in order to inject project-dependent dependencies, filtering steps, etc.

Export/Import using a custom migration pipeline

In our larger Plone migration projects, we often use a variation of the Transmogrifier approach. The export of the data and content from the original site is also done using the collective.jsonify add-on. The exported JSON files are then being imported into a migration database (we use ArangoDB). The migration database allows us to perform a partial import, query, and inspect the migration data in order to work on migration aspects and problems more easily than searching to Gigabyte of JSON files on the filesystem with operating system related tools like “grep”). The import is done using a custom migration package (which is to 80% generic for most Plone migrations, the remaining 20% are project-specific adjustments or additions because every migration is different). Our migration package allows us to run partial migration in order to test migrations more flexible. E.g. the migration package allows us to import a subset of content (either one or more folder or only specific content-types). This is very handy and time-saving. Migrating a large Plone site with a lot of content can take many hours. So testing changes on a particular feature with a subset of the data in advance is always better than having a failure on the full migration set after some hours.

Today, we released some small Plone add-ons as we developed and used over the last years for customer projects. In many cases we duplicated the same over and over again...so it was time for some refactoring and moving the code into dedicated packages.

zopyx.plone.persistentlogger

zopyx.plone.persistentlogger supports persistent logging where the log data is stored on an arbitrary persistent Plone object (as annotation). Typical usecases are application specific logging e.g. for logging a history per content object directly in Plone rather then having a huge common log on the filesystem. The log entries are stored using object annotations.

zopyx.plone.persistentlogger

Persistent logging for Plone objects

PyPI

collective.metadataaudit

collective.metaaaudit logs metadata changes to Plone content objects transparently (for all content types). This add-on in in particular useful when you have to answer the question “who was doing what and when”.

collective.metadataaudit

Audit metadata changes

PyPI

collective.debug

This add-on provides a @@debug browser view that returns some debugging information on the current context object, the current user and its roles, the instance dict of the current context object and request information.

collective.debug

A Plone debug view for introspecting the current context, user, roles and instance dict

PyPI

collective.contentsync2

Content sync between Plone sites over plone.restapi

Features

• sync content folders or individual content objects to a remote Plone site
• full sync for initial sync
• incremental sync for content updates
• configurable sync behavior per content type
• trigger immediate sync upon create or update operations through content rules

collective.contentsync2

Content sync between Plone sites

PyPI

Earlier this month, I released an update of my PrintCSS Portal print-css.rocks.

Thiis update includes support for new tools like

• WeasyPrint
• Typeset.sh
• PagedJS

I also want to mention the portal printcss.live by Andreas Zettl. printcss.live is a very nice and supplementary addition to the PrintCSS community because it allows you to test tools directly within the browser. Andreas Zettl also maintains a list a PrintCSS articles on medium.com.

After almost 12 years I moved my personal blog blog.zopyx.com from Plone 4 to the Ghost blogging platform. You may ask why? First, my old Plone 4 installation reached end of lifetime a long time ago and it was time for moving the content to a different system. I investigated a lot of time into options like static site generators like Eleventy, Huge and others but I was not very much happy with them. I just wanted a simple system where I can add and publish content easily.

The migration from Plone 4 to Ghost was a bit painful. First, I tried to use plone.restapi or collective.jsonify but the Plone 4 site gave me a bunch of headaches and errors. So I decided to create a static copy of the original bog using httrack. With some Python code I was able to extract the content part from the HTML markup of each document and some necessary metadata like publication date, title, description etc. All extracted data was written to a JSON file.

Ghost itself provides some JSON format as import/export format. Due to limited resources I hired a Ghost developer (for a very small budget) that converted my JSON format into the Ghost JSON format.

Then setting up Ghost (running in a Docker container) behind Apache was more or less straight-forward. As with every new system: you need to learn and try things out.

For the moment, I am happy with the result and outcome. There are clearly some issues like a better theme or some CSS fixes to be implemented.

I hope to blog again more in the future.

These days are tough and challenging for all of us. COVID-19 affects everyone across the globe. The Coronavirus hits every country differently. Fortunately, Germany seems to be better prepared than other countries. However, there will be a huge fallout from this crisis and many things will not be the same as they were for a while. So, please #stayathome and stay safe.

But let me tell you a small success story around Plone and COVID-19.

We are running the portal Onkopedia (www.onkopedia.com) for over a decade where the medical guidelines for oncology and hematology for Germany, Austria and Switzerland are published. The portal contains treatment guidelines for over 60 diseases which also includes medical studies, protocols, certifications and information about individuals drugs and active components. With the growing Corona crisis in Europe, high-risk patients with pre-existing conditions like cancer came into the focus. There was an urgent need for providing COVID-19 specific information for most diseases to the public.

The Onkopedia portal was always running and powered by Plone (nowadays running on Plone 5.2, Python 3). Within two or three days we had to extend our data model for COVID-19 in order to provide the functionality required to publish the COVID-19 information specific for each disease. With the help of the whole Onkopedia development team, we were able to to roll out the necessary Onkopedia update within a very short time from of three days. The required changes affected the Plone data model for the guidelines and a bunch of UI aspects like a new COVID-19 section and individual markers in guideline documents for referencing the related COVID-19 information in the COVID-19 master guideline.

As a result, we managed to roll out the new COVID-19 section on onkopedia.com (German only) yesterday with individual COVID-19 content for over 40 different diseases.

Key success factors: Plone and a great team (Abstract Technology, Practice Innovation, AppWeeve)

Thank you all for making this happen and for contributing in the fight with COVID-19.