Midgard RFCs

Midgard D-Bus identifiers

Wed, 30 Apr 2008 09:51:50 +0000

Revision history

2008-04-30, created by Piotr Pokora
2008-05-26, clarified message content ( Piotr Pokora )

Background

As D-Bus allows to create self, freely defined paths for objects, it might be confusing to send messages to other objects without any knowledge about their particular paths. This document defines D-Bus messages and objects' paths, at which instances has been created.

'midgard_article' class name is used in every example below.

Main Bus name

org.midgardproject - the main Midgard D-Bus service name
/org/midgardproject - the main Midgard D-Bus service path

Services

Services objects can be created by any application as long as they conform to this mRFC.

Clients

Clients objects can be created by any application as long as they conform to this mRFC. For asynchronous massages, Midgard API shall provide automatic messages for main I/0 operations.

Database identifier

To limit possible collisions and provide clear and safe services' and clients' identifiers, Midgard configuration name is appended to main service's path. Midgard API implementation should not require to append this name by application, and path with configuration name should be created automatically.

For example, if application has been initialized with default 'midgard' configuration, the base path for services and clients shall look like this:

/org/midgardproject/midgard

Of course for configuration named 'myconfig', the base path shall look like this:

/org/midgardproject/myconfig

Midgard application should not be aware of configuration name as both service and client will be created for the same base path. Midgard API for D-Bus should require only module or class names.

Module or class identifier

Services' paths should be created with classes' names, and if possible with module's or feature's names to separate services objects. For example all user defined MgdSchema classes should be created for 'MgdSchema' path. Class method should be appended to path to finally identify service's name.

This document must define clear path's method name, if class' real method name can not be appended to path or any other name suits better.

MgdSchema

Services for MgdSchema should be created at path with following tokens:

mgdschema
class name
method name

Every mgdschema class' service or client shall be initialized with the same convention.

Create

'create' shall be appended to object's path.

/mgdschema/midgard_article/create

On success, Midgard API shall send asynchronous message to object at path.

Update

'update' shall be appended to object's path

/mgdschema/midgard_article/update

On success, Midgard API shall send asynchronous message to object at path.

Delete

'delete' shall be appended to object's path

/mgdschema/midgard_article/delete

On success, Midgard API shall send asynchronous message to object at path.

Purge

'purge' shall be appended to object's path

/mgdschema/midgard_article/purge

On success, Midgard API shall send asynchronous message to object at path.

Get

'get' shall be appended to object's path for one of these methods:

constructor
get_by_guid
get_by_id
get_by_path
get_parent

/mgdschema/midgard_article/get

Midgard API implementation will send asynchronous messages in above cases, only if object is successfully retrieved from database.

Replication

Services for replication should be created at path with following tokens:

replication
class name
operation

Import

/replication/midgard_article/import

Export

/replication/midgard_article/export

Messages

Messages sent to Midgard D-Bus services should contain:

Serialized object as xml data

References

http://dbus.freedesktop.org/doc/dbus-tutorial.html

Name uniqueness

Wed, 05 Dec 2007 15:50:36 +0000

Revision history

2007-12-05 Created by Piotr Pokora

Background

This document describes Midgard objects' name uniqueness, scope of uniqueness and configuration possibilities. It is also part of Midgard Tree functionality.

Terminology

Whenever name word is used in this document it refers to object property registered with MgdSchema and configured to be unique in particular scope.

Name uniqueness

Unique name should be a property of any type, however it's recommended to use string type with unique names. Name should be configured in Midgard Schema xml file and there should be only one unique name defined per class ( type ). Name should be configured only for classes which may be represented in Midgard Tree. For any other class, unique name should be ignored.

Reserved configuration attribute which defines unique name itself, doesn't qualifies a class ( for which name belongs to) as Midgard Tree enabled.

Name uniqueness should not be supported by underlying database storage provider, unless such functionality would not break sitegroup or Midgard Tree specifications.

Configuration

Unique name should be configured only with reserved unique attribute(s) in xml file. A value for the given attribute should define 3 different scopes of uniquness:

parent

It guarantees a name is unique in scope of parent type object's identifier.

self

It guarantees a name is unique in scope of the same type object's identifier.

both

It guarantees a name is unique in scope of the same and parent type object's identifier.

Examples

<property name="title" unique="both">
<property name="name" unique="parent">

OR Unique name should be configured only with reserved unique and scope attribute(s) in xml file.

Examples

<property name="title" unique="yes" unique_scope="both">
<property name="name" unique="yes" unique_scope="self">

TODO

Describe parent name uniqueness enforce when value is set to 'both'.

References:

MgdSchema documentation

Administrative modules for Asgard

Tue, 24 Jul 2007 12:34:56 +0000

Asgard is the administrative interface used for Midgard CMS. By default it provides a comprehensive object browser and management tool.

It is possible to extend the Asgard functionality by adding other modules to the system.

Request handler plugin setup

Components wishing to operate as Asgard plugins must have a class that registers plugin handler classes to the request switch. This is defined in component manifest via:

'customdata' => array
(
    'asgard_plugin' => array
    (
        'class' => 'net_example_adminmodule_request',
        'src' => 'file:/net/example/adminmodule/request.php',
        'name' => 'Example admin',
        'config' => null,
    ),
),

The actual class must have a method get_plugin_handlers() that returns a request switch array and initiates the Asgard library:

function get_plugin_handlers()
{
    $_MIDCOM->load_library('midgard.admin.asgard');
    $_MIDCOM->auth->require_valid_user();
    return Array
    (
        'index' => Array
        (
            'handler' => array('net_example_adminmodule_handler_main', 'main'),
        ),
    );
}

Note: There must be handler for a view without arguments as this will be the screen users enter when choosing the module from Asgard navigation.

Handler class

The handler class must populate some data used by Asgard.

Handle phase

During the handle phase the plugin must register itself to Asgard:

$my_title = $_MIDCOM->i18n->get_string('example admin plugin', 'net.example.adminplugin');
midgard_admin_asgard_plugin::prepare_plugin($my_title, &$data);

Show phase

During show phase the element must show Asgard's elements around its own content:

midcom_show_style('midgard_admin_asgard_header');
midcom_show_style('midgard_admin_asgard_middle');

// Show plugin's own output

midcom_show_style('midgard_admin_asgard_footer');

On-demand service loading for MidCOM

Thu, 19 Jul 2007 21:24:20 +0000

This is a proposal for moving service and library loading infrastructure in MidCOM 3 to a more on-demand service oriented model.

Background

Currently MidCOM loads a huge number of libraries and services for each request, requiring large amounts of file accesses and PHP parsing. Most of the services loaded are however not really used in a typical request, and so can be seen as being loaded "just in case".

The belief is that this makes the MidCOM environment heavier and more difficult to understand. This proposal outlines how MidCOM could migrate from the current model to a model where services would be loaded when they are used for the first time.

MidCOM service loader

A new class midcom_helper_serviceloader needs to be defined to act as the access point to the MidCOM service system. It will be instantiated as $_MIDCOM->serviceloader at MidCOM start-up.

The service loader should provide two public methods:

boolean can_load($service)

Method for checking if a particular service is available on the system. This would enable components to deal with missing requirements or other situations where some functionality is not available more cleanly.

object load($service)

Method for instantiating an implementation of a particular service.

Inversion of control

An additional benefit of the service loader model is that MidCOM's services can start following the Inversion of Control design pattern.

This means that services are defined by their interface classes, but actual implementation of a service can be chosen by site administrator. This will enable us to easily try and swap between various ways of doing things.

Example: URL name generation can be moved to a midcom_core_service_urlgenerator service. Since the default URL name generator shipping with MidCOM does not deal particularly well with the Georgian language, a web developer deploying Midgard in Georgia can write their own service implementation and just swap that to be used in site configuration.

$GLOBALS['midcom_config_local']['service_midcom_core_service_urlgenerator'] = 'ge_convert_urlgenerator';

Performance

The two-tiered approach of having interface and implementation separately means that two files have to be loaded for each service. This causes a slight performance hit when compared to current approach of one file per service.

However, it is believed that the benefit of loading services only when they're needed and being able to change service implementations on configuration level easily offsets this.

Related tasks

The actual service loader and an URL generator service have already been implemented as an experiment in MidCOM trunk. If this proposal is accepted, all services and "helper libraries" in MidCOM need to be refactored to the new model and components switched to use them.

As this is a big task, the proposal is to switch services to the new model one-by-one, starting from libraries that are used in only few places.

As of 2007-07-19, MidCOM loads 57 files in the midcom.php, with many of them potentially loading sub-dependencies. Most of these could be moved to on-demand handling or removed by the separately proposed DBA rewrite.

General Membership

Thu, 05 Jul 2007 11:33:37 +0000

Revision history

2007-07-05 Created by Piotr Pokora

Background

Generic membership describes the basic way to group any MidgardSchema ( MgdSchema ) type ( class ) as an member of any other type registered in Midgard application. This mRFC also describes known Midgard functionality to group midgard_persons objects as a members of midgard_group one, and capabilities of membership introspection.

Member owner type

The type which groups other types. One type ( class ) can not group objects of the same type.

Member type

The type which belongs to owner group. One type ( class ) can not belongs to group of the same type.

Membership type

The type which holds required information ( like owner or member identifier ) for members and owners.

MidgardSchema definition

Membership information shall be defined in xml schema file as a type properties and attributes and particular introspection capabilities shall be implemented on API level. Membership shall be always defined for member type and never for owner type.

Minimal information should contain:

Member owner type name
Membership type name
Property and (or) field names which should be used for membership links

Example

<type name="midgard_person" table="person" parentfield="creator">
    <membership name="midgard_member" owner="midgard_group" >

Membership introspection

Membership implementation should allow to retrieve following information:

Member owner type

If type has members
What type name(s) are members of owner type
Fetch all members

Member type

If type is a member
What type is an owner of member type's

Member creation and deletion

In general, members should be created with particular membership type implementation, to avoid direct and random database access. Membership type should always create or delete member for given member and owner type.

Midgard Object serialization

Mon, 18 Sep 2006 13:16:29 +0000

Revision history

2006-09-22 Updated by Piotr Pokora, added fixed 'midgard_object' root element to xml schema

Background

This mRFC describes xml file format which should be used for objects' replication, and for data exchange between Midgard and other systems. It is impossible to define xml file format for every type registered in Midgard type system , however this mRFC focuses on minimal xml schema (2) file which should describe serialized Midgard object. Xml file with serialized Midgard Object should contain Midgard namespace, Midgard Object type name, optionally its guid and lang which identify object and common metadata.

Namespace

Midgard namespace should be defined as: __http://www.midgard-project.org/midgard_object/1.8__

( Optionally namespace could define Midgard version for which given xml file can be used )

MidgardObject

Midgard type should be defined as complexType identified by name of string type. Optionally 'guid' and 'lang' atributes could be used to identify Midgard Object. Guid attribute if used should be at least 28 and at most 80 characters long (1). Lang (3) attribute should be 2 letter language code name and define available languages enumeration.

For better understanding and connectivity , midgard-core should have implemented functionality to create xml schema file for any type registered in Midgard Schema. Additionally this implementation should create languages enumeration for all available languages (3) in Midgard environment. Every Midgard object's property should be defined as element type with optional attributes.

Multilingual data

If serialized object is multilingual Midgard type and more than one language is used for such object , all language content objects should be serialized and stored in one xml file. In such case deserialization implementation should return array of multilingual objects, and lang attribute's value should be mandatory set in xml file.

Parameters and attachments

Optionally , object's paramaters ( registered as standalone midgard_parameter type ) and object's attachments ( registered as standalone midgard_attachment type ) could be serialized and stored in object's xml file.

Minimal XML schema file:

Example:

 <?xml version="1.0" encoding="UTF-8"?>
 <midgard_object xmlns="http://www.midgard-project.org/midgard_object/1.8">
   <midgard_article guid="26b621b938c4aea0e021d4baae69d6ab" lang="en">
     <metadata>
       <creator></creator>
       <created>1999-05-07 18:04:04</created>
       <revisor></revisor>
       <revised>2006-09-10 09:28:08</revised>
       <revision>1</revision>
       <locker></locker>
       <locked>1999-08-17 16:01:32</locked>
       <approver></approver>
       <approved></approved>
       <authors>f6b665f1984503790ed91f39b11b5392</authors>
       <owner></owner>
       <schedulestart></schedulestart>
       <scheduleend></scheduleend>
       <hidden>0</hidden>
       <navnoentry>0</navnoentry>
       <size>481</size>
       <published>1999-05-07 18:04:04</published>
       <score>0</score>
       <imported></imported>
       <exported>2006-09-14 12:37:40+0000</exported>
       <deleted>0</deleted>
     </metadata>
     <action></action>
     <id>2</id>
     <created>1999-05-07 18:04:04</created>
     <revisor>f6b665f1984503790ed91f39b11b5392</revisor>
     <score>0</score>
     <title>Host Administration</title>
     <abstract>Midgard host and directory management
 </abstract>
     <locker>f6b665f1984503790ed91f39b11b5392</locker>
     <caldays>0</caldays>
     <approved></approved>
     <calstart>0000-00-00</calstart>
     <content><p>
 Use the host administration tool to create and maintain the       structure and functionality of your web sites. Use the layout adminstration tool for controlling the appearance of the site and the content administration tool for the dynamic content of the site.   </content>
     <icon>0</icon>
     <type>0</type>
     <creator>f6b665f1984503790ed91f39b11b53922</creator>
     <revised></revised>
     <approver>0</approver>
     <extra1></extra1>
     <name></name>
     <author>f6b665f1984503790ed91f39b11b5392</author>
     <url></url>
     <contentauthor>1</contentauthor>
     <contentcreated>1999-05-07 18:04:04</contentcreated>
     <print>0</print>
     <extra2></extra2>
     <view>0</view>
     <extra3></extra3>
     <revision>0</revision>
     <sid>2</sid>
     <locked>1999-08-17 16:01:32</locked>
     <up>d0099a92c1018461bc6e33cd0a1c435b</up>
   </midgard_article>
 </midgard_object>

References:

http://www.midgard-project.org/development/mrfc/0018.html
http://www.w3.org/TR/xmlschema-0/
http://www.loc.gov/standards/iso639-2/index.html

Multilingual MidCOM

Wed, 26 Jul 2006 14:55:44 +0000

This proposal outlines how multilingual sites should be managed in the MidCOM framework utilizing the MultiLang features of Midgard 1.8. This mRFC been submitted to the Midgard Community for discussion and approval under the Creative Commons Attribution-ShareAlike license.

Multilingual sites

Midgard's base of power is Europe where there are many languages and many organizations operating across language barriers. This makes it very important to support easy management of multilingual sites.

Traditionally there have been two different types of multilingual sites:

Sites in multiple languages: These are sites that contain approximately same information in each language, either mandated by law or by organization's desire to reach different language audiences
Multiple language versions of site: These sites can contain very different content structures targeted at domestic and foreign markets, or based on some other differentiation

This proposal focuses on the first of these two. Multiple language versions of a site are better managed using separate content trees for each site than using MultiLang features.

MultiLang background

Right from the beginnings of the project, Midgard has had a strong international focus. Multibyte character set support landed in Midgard already in October 1999 in order to support managing sites in languages like Russian and Chinese. UTF-8 was made the default character set for Midgard in the 1.6 series.

However, in addition to character sets, also translation of actual content was needed. MultiLingual Midgard was presented as a patch by David Schmitter from DataFlow in Switzerland in May 2003, and entered Midgard proper for the 1.5.0 release.

This made the Midgard content formats support different translations of content, but besides DataFlow's proprietary WebInOne publishing tool, no Midgard authoring interface has actually added MultiLang support.

The reason for not supporting MultiLang has mostly been the difficulty of integrating good translation workflow, and unclear PHP-level programming APIs for it. The APIs have now however matured enough with integration to MgdSchema and Query Builder tools so that supporting MultiLingual content cleanly is finally possible.

Implementation into MidCOM

Here is a description on how MultiLang should be implemented into the Midgard Component Framework. The aim here is to make creating and maintaining multilingual sites easy without having to impose changes into MidCOM components.

Choosing languages used on site

Webite's language selections (languages available for the site and the default language) are stored in the MidCOM configuration array.

The language selections can be made using Site Wizard for better convenience

Language selection for content

Languages would be selected using the $argv[0] prefix of the site, i.e. Finnish version of "About us" would reside in /fi/about. This can be implemented in a centralized fashion by modifying MidCOM's URL parser and making it call the corresponding MidCOM i18n service set_language method for changing language.

In addition to the per-language URLs, there would be the "language neutral" URL for each object. This would redirect user to appropriate URL based on their browser language selection, i.e. /about would redirect me to /de/about for German language users.

Navigation building and object instantiation

Language versions of site should obviously show only the contents translated to their languages in navigation and content listings (like news lists for instance). Since MidCOM has a centralized wrapper for Query Builder, the DBA can easily add the language constraint into all DB queries transparently. This means components don't have to do any language-based processing to show only Norwegian news items.

Similarly, when an object is instantiated (get_by_guid etc), DBA should check if the $object->lang is the current language and if not fail the instantiation.

MidCOM should contain a specific "translation to this language not found" error message that could be triggered if requested object is not available in current language. This error message could contain links to the existing language versions.

Content editing

Since editing MidCOM content now happens on-site and there are language prefixes available, switching editing views between languages is just a matter of changing the site prefix.

However, there should be some facilities for showing to translators what has been changes in the "default language" contents of an object so they know what to translate. This relates to RCS version control services in MidCOM.

Content language vs. UI language

MidCOM's i18n service has already been modified so that user interface language can be different from actual content language. Many multilingual organizations prefer to run their software in a common language to ease documentation and training.

Different language versions

The information on different language versions of an object must be available so that MidCOM's metadata system can populate links to the different translations.

API-level requirements

Here is a list of requirements from the MultiLang implementation in Midgard Core that are needed for this mRFC:

The midgard_language objects must be available for regular Query Builder operations, and must contain a locale field with the appropriate UNIX system locale
When mgd_set_lang(X) has been called, all object update/create/delete operations must apply to the content of that particular language
$object->get_languages() must return list of languages the object is available for
Each language version must contain a revision timestamp to help with translation workflow
When set_lang has been called for Query Builder, it must only return objects that are in that particular language, no lang0 objects

MidgardApacheCache

Wed, 14 Jun 2006 16:46:23 +0000

MidgardApacheCache

Revision history

2006-06-14 Created by Piotr Pokora

Background

Cache approaches:

data reusability
limit the need to query the same objects with every request
cache objects which are queried with the same data with every request

Implementation

MidgardApacheCache should extend base MidgardCache class, and should be optimized for any midgard extension which is used in Apache module environment. MidgardCache base class should be assigned as private member of MidgardConnection and thus connection handler object could be responsible for managing internal's core objects references counts.

MidgardApacheCache when instanciated should return UUID which should be available during the whole application runtime. For Apache module, it means - as long as httpd server is running.

There shouldn't be any method or function in implementation which should allow list all cache entries ( MidgardApacheCache instances ) available.

Retrieving MidgardApacheCache instance should be possible only with UUID representing particular object instance and application which use cache object should be responsible to store this UUID.

Example

<?php

$application_cache_guid = "guid_string"; 

$cache = new midgard_apache_cache($application_cache_guid);
if(!$cache->exists())
    $application_cache_guid = $cache->create();
else 
    $cache->get();
?>

The main purpose of MidgardApacheCache should be possibility to store objects , however other types should be supported as well. Objects should be identified by guids in the same way as are identified and retrieved in typical Midgard applications. Other types like arrays should be registered with unique names.

Example

<?php

$object = $cache->get_type_by_guid("object_guid_string");
$object->register("my_constant_array", $array_type);
$array = $object->get_type_by_name("my_constant_array");

?>

MidgardApacheCache should implement such features:

initialize cache
get cache object instance identified by UUID
add type to cache
update type in cache
get type from cache
remove type from cache

Midgard object replication

Fri, 02 Jun 2006 17:47:52 +0000

Revision history

2006-06-02 Created by Piotr Pokora
2006-06-17 Updated: possibility to delete and undelete objects
2006-06-19 (torben) Commented the MidCOM section, reset the mRFC to draft status due to this.

Background

This mRFC is a proposal for repligard functionality replacement , supported by midgard-core and by any language for which Midgard language bindings exist. As repligard supported Midgard database replication without any possibility to change its behaviour a lot , this mRFC focuses on object's records replication , object exporting and object importing to or from any database storage. This mRFC also describes possibility to undelete and purge object's record(s).

Examples in this mRFC use PHP as scripting language.

Example: exporting all newest objects example.

<?php  

$exported_objects = array();
foreach ($_MIDGARD['types'] as $type => $type_id) {
    $qb = new midgardquerybuilder($type);
    $qb->add_constraint("metadata.created", ">", $yesterday); 
$retval = $qb->execute();
    $exported_objects = array_merge($exported_objects, $retval);
}

foreach($exported_objects as $object){
    $object->export();
}   
?>

Midgard object metadata

Every Midgard object's metadata class should have 'exported' and 'imported' members assigned as properties of midgard_metadata type. These two properties's values represented by corresponding database storage values should be settable only by midgard-core with particular methods. Both properties must be MGD_TYPE_DATETIME type.

Midgard object methods

Midgard-core must support export and import methods for object's replication. Object's replication related metadata should be managed by basic methods: create, update, delete, undelete , purge, import, export.

Create

created , current datetime value must be set
revised, current datetime value should be set
imported, empty value must be set
exported, empty value must be set

Update

created, value can not be set
revised, current datetime value must be set
imported, empty value must be set
exported, empty value must be set

Import

created, value can not be set if object's record exists in database in any other case object's record must be created and metadata created with current datetime value must be set
revised, value shouldn't be set if object's record doesn't exists in database in any other case metadata revised must be set with value of imported object's metadata revised value
imported, current datetime value should be set
exported, empty value must be set

Object can be imported to database only if imported metadata property value of the object for which record in database exists is empty and its metadata revised property value is not newer than the value of the same property of object which is to be imported.

Export

created, value can not be set
revised, value can not be set
imported, value can not be set
exported, current datetime value must be set

Object can be exported from database without any restriction. Application which exports objects defines what constraints are used to export object records.

Example: exporting visible midgard_article records

This example demonstrates how to export newest and not hidden midgard_article objects

<?php

$qb = new midgardquerybuilder("midgard_article");
$qb->add_constraint("metadata.created", ">", $yesterday);
$qb->add_constraint("metadata.hidden", "=", FALSE);

/* Do not export objects which were created and exported for the last 24 hours */
$qb->add_constraint("metadata.exported", "=", "");

$retval = $qb->execute();

foreach($retval as $object){
    $object->export();
}   
?>

Delete

created, value can not be set
revised, current datetime value must be set
imported, value can not be set
exported, value can not be set
metadata.deleted, value must be set.

Object's record(s) can not be deleted from database when delete method is invoked. Instead , object's metadata delete property should be explicitly updated with correct delete value. For performance reason this value should be an integer type. This metadata property should be also used by midgard core as mandatory Midgard Query Builder's constraint added internally by Query Builder implementation.

Undelete

created, value can not be set
revised, current datetime value must be set
imported, value can not be set
exported, value can not be set
metadata.deleted, value must be (re)set to initial default state.

Purge

When this method is invoked, midgard core should delete object's record(s) from database and should update repligard table. Following values should be set for corresponding record in repligard table:

object's class name
object's guid
purge action should be set to TRUE value

Repligard table

Repligard table must containt only object's guid and typename ( classname ) for which new object instance with particular guid can be created. Additionaly repligard table should contain information if object's record(s) was purged.

Query deleted objects

Midgard core should implement functionality which allows to query only deleted objects. This functionality could be implemented with new Midgard Query Builder method or with new Midgard type.

Staging/live in MidCOM

Missing topics: Data model, general service architecture, including the layers of the communication infrastucture.

Torben

Since replication interfaces are exposed to PHP level, the staging/live process can be handled in MidCOM space. This should make the simple replication scenarios like an article that is approved much faster, the system more fault tolerant due to the queue system, and finally the replication logic easier to tweak because it is entirely in PHP level.

There will be a midcom.helper.staginglive purecode component to handle the replication process.

This will have to be midcom.services.replication.

Torben

Invoking replication

The replication component will register UPDATE and DELETE watchers for midcom_core_dbobject. Since approvals are stored into Midgard metadata fields using update() method approval should be carried to this watcher.

Note, that you cannot register watches for midcom_core_dbobject, as this class is not a base class for other DBA classes. It is more of a "mix-in" due to Limitations of the PHP OOP API. The correct definition would be adding watches to all defined DBA classes by not limiting the watch to any class.

Torben

To catch objects updated or deleted outside MidCOM DBA the system will also provide a cron entry that will query through all MgdSchema types to see if there is something to replicate. This can be run relatively seldom, for example once per day.

This is not very precise. The criteria after which objects are marked for replication should be outlined.

Torben

Export process

The export watchers will execute an exporting method in the midcom.helper.staginglive interface class and provide it with the affected object as argument.

If approvals or scheduling are being used, the system will first check if the object itself is set to be visible. Otherwise visibility will be assumed.

If the object is visible, the same check will be done for its parent.

Note, that this can prohibit replication under certain circumstances: Assume that a topic and one of its articles are both changed. The article gets approved, the topic not. Replication of the article will then be delayed.

Note as well, that there is another case, where you actually need to be careful if you change the behavoir to cover the above special case: If the topic in question was newly created, it won't be available on the target server.

Normally, the latter should be no problem if and only if only GUIDs are used for linking and, as such, ID mappings are not required. The legacy data structures do not yet do this. I very strongly suggest that this will be changed prior implementing this, even if this means that existing code (mainly QB lookups) have to be adapted to it.

Torben

If the parent is visible, the object will be marked for exporting. If the parent is not visible export is aborted and UI message "Object not exported because parent is not visible" sent to the user.

What happens later, when the parent gets cleared? Is the delayed object queued somewhere? This needs much more detail.

Torben

Then the method will query the object's children and call the same method for them.

I assume that this is the fix for the above problem I mentioned. The problem I see here is that you will often get large trees here, especially if you are at the top of the tree. Also you will have difficulties as you have to know all types which potentially have the object in question as a parent:

Especially topics are tricky here, as several of the components I wrote lately assign their data to topics, without actually using midgard_article. But groups or persons can easily reach a similar spread.

Torben

TODO: Pseudocode for the exporting decision

I'd like to see this before +1'ing anything.

Torben

Finally the exported objects will be stored into replication queue and an at entry registered for running the replication next minute.

Is this wise? Or would it be more appropriate to have a cron job doing it every X minutes to optimize / batch the changes.

Torben

UI message "3 articles and 1 topic exported for replication" is sent to the user.

Subscribers and replication queue

Subscribers (Midgard sites to replicate to) are stored in the database. The subscriber entry contains the replication method used for that subscriber and address and authentication information required for the method.

When objects are exported, the export XML is stored into a midcom_helper_staginglive_queue_entry object for each subscriber.

Data duplication, the data model should be revised here.

Torben

Replication is launched by the at entry registered at export phase. The replication process goes through all subscribers, and tries to send the items of their queue to them using the defined method.

There can be multiple replication methods supported by the system. At first we will only implement a simple HTTP PUT (or POST) method sent to a MidCOM exec handler of the midcom.helper.staginglive component, authenticated using HTTP Basic Auth. Other methods could include Jabber/XMPP, DBE or even email.

Wouldn't some XML-RPC or other standardized remoting mechanism be more appropriate then some proprietary solution?

Torben

Email could provide a high security replication solution where the exporting end would GPG encrypt and sign the email, then send it out as an email. The importing end would then read the email from the server, check its signature, decrypt it and import. This way the importing end could even be a mostly offline computer that would only connect to internet to receive the emails once per day (this is how FSF Europe's member registry works to protect privacy).

Consider using regular X.509 based encryption / signing instead of PGP. More standardized (almost as easy to use) and integrated into existing PKI infrastructures. This should be available completly independant of the actual transport layer used, btw.

Torben

If the replication is successful, the queue entry is deleted. If not, the replication will be attempted again every hour using a cron entry.

Provide some fallback mechanism, similar to the SMTP system. There should be a fixed number of retries, maybe with increased waiting periods after certain escalation levels.

Also, it is recommended to flag entries as failed after a certain amount of retries have been reached.

Failed connections to a subscriber should escalate as well, disabling the subscriber until the situation is resolved. Otherwise, there would be an easy DOS scenario when too many replications pile up for a given subscriber.

Torben

Subscription types

In the initial implementation every object is replicated to every subscriber. At a later stage we can implement different subscription type handlers that will enable limiting the subscription to particular MgdSchema types, object trees or even particular query constraints.

Examples of subscription types would be "content in tree", "my tasks" or "all forum posts".

This is rather vague, especially since the neccessary basic infrastructure to provide something like this is never mentioned above. Goes into the chapter "missing service architecture".

Torben

Import process

Import process will simply call the Midgard import method for each object in the imported XML file.

No sanity checks, nothing? I think there has to be a high level of bulletproofing when letting such replications in.

Torben

MidgardCollector

Thu, 18 May 2006 12:30:50 +0000

Revision history

2006-05-18 Created by Piotr Pokora
2006-05-18 Added use case examples by Piotr Pokora
2006-06-14 Constraint specifications and returned data changes by Piotr Pokora

Background

The main idea of MidgardCollector type is code , resources and data reusability for all data values which are not objects itself and as part of any object could be retrieved from Midgard database without any need to retrieve full objects' records. MidgardCollector is special limited resource data handler and is optimized for performance and data accessibility. The most important approach is to reuse the same functionality for Midgard features like style engine, objects' parameters and object's attachments.

However MidgardCollector should be usable for any Midgard type defined for Midgard application.

Implementation details

MidgardCollector should be implemented as MidgardQueryBuilder wrapper with extended constraints management and extended caching possibilities. Following Midgard objects parameters' convention , MidgardCollector should use 3 constraints to query records drom database:

domain
- property name and its value to limit selected records
name
- property name ( and optionally its value ) to limit selected records
- property name which value should be returned as part of a result
value
- property name which value should be returned as part of a result

Additionally another type should be settable as a parent of the type already initialized for MidgardCollector. In such case type initialized for MidgardCollector should have defined reserved database storage columns not accesible by application. Column names should be prefixed with 'midgard_' namespace like word.

Data retrieving

MidgardCollector should be initialized only for one particular type. This type's property names should be set as domain , name and value constraints.

Returned type should be an hash array with collector's name assigned as key and value as key's value. Additionally guid of an record should be assigned as another array key.

name -
      |-value -> variable
      |-guid -> variable

These hash array should be always set as returned resultset and number of returned records should be limited by domain's value and optionally by name's value. Additionally MidgardQueryBuilder methods like limit or add_constraint or add_order could be used to limit or to order selected records. Parent type assigned to MidgardCollector should be also used to limit selected records , limiting returned results by its type name and guid.

Returned array should be available during the whole application's runtime unless explicitly destroyed by application. If and when returned object should be destroyed depends on application's implementation and its not part of this mRFC.

Data overwriting

MidgardCollector should be able to overwrite data in existing array by destroying already existing key value pair and replacing it with new name and value. Such overwrite should be done with existing collector's instance , without any need to create new instance of object.

Methods

MidgardCollector should have implemented methods which allow:

Set domain and its value
Set name its value
Get resultset by domain
Get resultset by domain and name
Get resultset limited to parent type
Destroy name and value
Destroy the whole collection
Overwrite value for existing name

Examples

Style elements

Select all style elements' values and render one of them:

$mc = new midgard_collector("midgard_element");
$mc->add_domain("style", $_MIDGARD->page);
$mc->add_name("name");
$mc->add_value("value");

$mc->execute();

$style_engine->render($mc->resultset['ROOT']);

Parameters

Select paremeters by domain, update parameter and collector.

$mc = new midgard_collector("midgard_parameter");
$mc->add_domain("domain", "midcom");
$mc->add_name("name");
$mc->add_value("value");

$mc->add_parent_type("midgard_topic"); 
$mc->add_parent_guid("fkj34h68d3f1js0du347fh4j4df"); 

$mc->execute();

$component = $mc->resultset['component'];

/* update parameter */

$topic = new midgard_topic("fkj34h68d3f1js0du347fh4j4df");
$topic->paramater("midcom", "component", "net.nehmer.static");
$mc->add("component", "net.nehmer.static");

$component = $mc->resultset['component'];

Attachments

Select only image/jpg attachments.

$mc = new midgard_collector("midgard_blob");
$mc->add_domain("mimetype", "image/jpg");
$mc->add_name("name");
$mc->add_value("content");

$mc->add_parent_type("midgard_article"); 
$mc->add_parent_guid("fkj34h68d3f1js0du347fh4j4df"); 

$mc->execute();

$retval = $mc->resultset;

SVN+PEAR for MidCOM

Mon, 08 May 2006 20:56:06 +0000

This mRFC is a part of the MidCOM 2.6 release process. It outlines the changes neccessary to make 2.6 source management and PEAR packaging easy to use.

2006-05-19: Accepted according to the lazy consensus rules.

Reasons to switch to SVN

The main reason for me why I want SVN is the ability for atomic commits on several files. The ability to track several changes belonging together has become increasingly difficult with CVS in the past, especially since the MidCOM codebase has increased so much in size.

Since this means greater changes in any way, it is the best point to restructure the repository according to the requirements of easy PEAR packaging:

New repository structure

The basic idea behind the new structure is separating the source tree into their corresponding PEAR packages.

The source files of each package will be in their own dir named after the package (to make packaging easy) in a central src directory:

/src/midcom.core
/src/midcom.admin.content
...

Note that the main package will be renamed to midcom.core. I recommend this to free the package name "midcom" as a meta-package which collects the core and all packages we deem as "required" to get a basic midcom running (f.x. to include required admin components).

Static files will move to the components static directory, for example /src/midcom.helper.datamanager2/static/.

PEAR packaging

Since SVN now uses one directory per package, the packaging scripts themselves should be easy to simplify.

To ger MidCOM PEAR releases as frequent as possible, especially in the case of Bug Fixes, a few enhancements have to be made to the packaging script:

Automatic uploads should be possible.
The Version-Number should be settable (and svn committed) from the command line. Ideally we should have an auto-increment method which increases the current patch-level by one (2.6.0 to 2.6.1 etc.)
The full pear packaging process should be hidden.
All temporary files should go to a temporary dir outside SVN (/tmp perhaps).
The new version number should be tagged in SVN implicitly.

Example session:

cd src/midcom.core
midcom-package 2.6.0
# Package is built to src/midcom.core-2.6.0.tgz
midcom-upload ../midcom.core-2.6.0.tgz

Session with implicit upload:

cd src/midcom.core
midcom-package -u 2.6.0

Session with auto-increment:

cd src/midcom.core
midcom-package -u

Releasenotes

Ideally, the new PEAR Package Tool can generate Package release notes automatically from the CHANGES file in the documentation directory.

It should be not to hard to identify the entries since the last release since an entry is always started by a ISO Timestamp.

Running MidCOM from SVN

Obviously, MidCOM can no longer be run from SVN using this structure. Thus, a simple script which creates a symlinked tree of MidCOM must be written (otherwise development will be hell).

All standard components should be linkable with a single symlink to their root / static directories. Special care must only be taken for midcom.core which has to be linked on a file-level hierarchy to allow adding components like midcom.helper.search etc.

Midgard sitegroup authentication and objectcreation.

Fri, 10 Mar 2006 09:56:51 +0000

Background

Sitegroups are logical containers implemented in Midgard to use one database but to separate out different clients by adding an extra sitegroup attribute to all records in the database.

There is also a sitegroup 0 (SG0) that is special in that all items from that sitegroup are read-only from other sitegroups (SGn) and may act as parents to an object (for example a style) in a sitegroup.

There are some usecases that need to be understood with regard to sitegroups. They are described in this document.

The basic information about sitegroups can be found in documentation: http://www.midgard-project.org/midcom-permalink-f624e440f76a466d5870374bca8e1449

Sitegroup ID

Up to and including 1.7 the sitegroup id has been an integer. In the future the attribute may be an integer (id) or string ( guid or uuid ) which identifies sitegroup by its record in database. The type of identifier depends on implementation.

Sitegroup creation

Sitegroup can be created only by authenticated person which is member of the Administration group in Sitegroup 0 ( SG0 ). These are the only users who can delete or update sitegroups as well.

When a sitegroup is created, it needs to be created with an adminigroup and a administrator.

The following code is a sample of how sitegroup creation should work for midgard 1.8:

// create the sitegroup 
$sitegroup = new midgard_sitegroup();
$sitegroup->name = "Sitegroup";
$sitegroup->create();
// create the admin and the admingroup
$group = new midgard_group();
$group->name = "admingroup";
$group->sitegroup = $sitegroup->id;
$group->create();
// create the admin
$person = new midgard_person();
$person->username = "admin";
$person->sitegroup= $sitegroup->id; 
$person->create();
mgd_update_password($person->id, $person->username, "pwd");
$member = new midgard_member();
$member->uid = $person->id;
$member->gid = $group->id;
$member->sitegroup = $sitegroup->id;
$member->create();
$sitegroup->admingroup = $group->id;
$sitegroup->update();
[NOT TESTED!]

How should this look for 1.7?

Creation of sitegrouped objects from sitegroup 0.

There are some usecases where the SG0 admins want to create objects into a sitegroup.

Creation in 1.7.

To creat an object in midgard 1.x , x<=7, you need to run the following code: $obj = mgd_get_object(); $obj->name = "some name"; // set some other attributes $id = $obj->create(); $obj = mgd_get_object($id); $obj->setsitegroup($sgid);

This will change the objects sitegroup, but not attachments or parameters creatd before changing the objects sitegroup, they will remain in SG0. Also, if you add attachments and parameters to the object later as SG0 these parameters will also stay SG0.

Suggested change to 1.7:

Setsitegroup() should change the sitegroup of the objects attachments and parameters as well.

Creation in 1.8

In 1.8 the following [NOT IMPLEMENTED] rules apply to adding a sitegroup to an object:

1) If user is in SGn (n > 0), create object in SGn
2) If parent object is in SGn (n > 0), create object in SGn
3) If sitegroup property is set to SGn (n > 0), create object in SGn
4) Otherwise create object in SG0

This implies that you cannot change the sitegroup of a created object after it has been created. you cannot change the sitegroup using the update method.

The only way to change the objects sitegroup once it is created, is to export it and import it through replication by another sitegroups person.

NB: $obj->update() should return false when a user tries to reset an objects sitegroup and return the error "Sitegroup cannot be changed after creation.", error number 32.

[NOT IMPLEMENTED]

Updating objects

When a SG0 admin updates an object, different attached records like quota or metadata, may be affected. These records should get the same sitegroup as the object - not the sitegroup of the application.

Application sitegroup

Every Midgard application ( which uses midgard connection and midgard database ) runs in a sitegroup context.

The context of a sitegroup'ed application is identified by sitegroup's identifier assigned as member of the midgard connection. Application's sitegroup can be set ( or even changed ) in one way only - by authentication as a user.

Setting the application sitegroup with no authenticated user.

Today, when no user is logged in, the application runs in anonymous mode. If the application is running through Apache, the application sitegroup is set according to the sitegroup of the current page. [NOT VERIFIED]

Anonymous mode as a user.

It is suggested to change the way the anonymous mode works. To run application in anonymous mode( when the application must be used within particular sitegroup context and no person should be logged in ) anonymous account should be created for particular sitegroup. Access control for the anonymous account should be covered by the ACL mRFC.

The anonymous account may be duplicated ( replicated ) between different sitegropups as long as as can be identified by particular sitegroup, even when record itself is identified by the same guid or uuid. When sitegroup has no anonymous user, the application should run in SG0 context.

Issues

What should be done with available resources when sitegroup context is changed when application runs?
What should be done with SG0 and with SGn available resources?

Sitegroups and authentication.

Today login to Midgard happens in two ways:

- Through basic authentication where the user may only enter username and password.
- Through the function mgd_auth_midcom($username, $password, $sendcookie)

Many if not most sites uses the mgd_auth_midcom function.

A problem is that midgard needs to determine the sitegroup the user has before checking the password as a username may exist in more than one sitegroup. This is solved in versions up to 1.8 by using a delimiter and the sitegroup name.

A problem with this approach is that the delimiters may be used in the username. When this happens, logins will fail as Midgard cannot divide the username-sitegroup combo properly.

On-Site Toolbar Management for MidCOM 2.6+

Mon, 27 Feb 2006 16:58:52 +0000

This mRFC is a part of the on-site administration integration process. Its goal is to outline a centralized way to create and handle toolbars displayed on-site.

Implementation Note: The page toolbar has been renamed to the view toolbar (I didn't have the mRFC at hand during my holiday, sorry ;-)).

Requirements

As a part of the on-site adminitration work, it is important to provide the user with an unified way to manage the objects being displayed. The currently used midcom_helper_toolbars is a good start here, but fails to meet a few requirements if it should go into the MidCOM core.

This document will give a complete overview about the required functionality.

What should be managed?

There are several "levels" on which objects should be managed. Generally speaking, there is a node, page and object level. This distinction is based on the scope of the operation being done:

Node operations affect the whole topic. They are not bound to a specific article, leaf or similar, more specific structure. Stuff like "Create new article" will go into this toolbar. Also, which is an important thing, this part of operations will always be open to the MidCOM core to add topic management operations to the system (given the right permissions apply). This represents the original "top" AIS toolbar.
Page operations affect the main object shown in the current output page. What the main object is, is largely dependant on the component. The newsticker frontpage, for example, does not have a single main object, while the taviewer does have a sinlge "standard" article on the frontpage. This especially means, that under normal circumstances only a component can reliably add information to this toolbar. Stuff like "Edit object" will go into this toolbar. This represents the original "lower" AIS toolbar.
Object operations are special toolbars which apply only to a single object being displayed in the current page. If you take cases like the original net.siriux.photos AIS interface you will see that a single page has several objects displayed simultaneously each with its own toolbar. The newsticker could in theory provide a similar functionality. The information in this toolbar always affects exactly a single object, and it is possible to have more then one object toolbar visible at the same time.

This gives us essentially two toolbars which are always present (node and page) and from which we always have exactly one instance per context (see below for context-related problems).In addition we have a variable number of the third kind of toolbars that might be used depending on the component.

page vs. object toolbars

As you might have already seen, page and object toolbars will have many things in common in most cases. Only compontens managing large counts of objects (like net.siriux.photos) will have real use for object toolbars.

For all other cases it makes sense to have a page toolbar which is actually bound to an object thus eliminating the distinction between these two kinds of toolbars.

You might now ask why I'm bothering for this distinction at all if it is really that specific: This is relativly simple: The toolbar system has to be built in a way so that the MidCOM core can add operations to the toolbars.

MidCOM only knows two levels of objects: First, there are the topics, also known as Nodes. They are the main (and only) container element known on the Core level and thus need their own toolbar operations (which is undisputed).

Then, as next level, MidCOM knows only objects, nothing more. In the original design, this was bound to the nap leaves. MidCOM was bound to have an individual leaf for each object. Not only for performance reasons, this link has been removed in 2.5 entirely. Nowadays there are many compontents which do have a broad range of management "possibilities", and active objects are a rather flexible term.

Nevertheless there needs to be a way for MidCOM to add commands applicable to a single object only. This is interesting for stuff like Metadata or permission management for example.

The new toolbar system thus has to cover both cases.

The proposed way to do this is to a) give the page toolbar the optional ability to bind to an object and to b) allow a component author to create an arbitary number of object toolbars independant of the page toolbar.

Context sensitivity aka "dynamic load support"

The MidCOM dynamic load ("dl") facility introduces quite a few complicaitons into this, as it blurs the idea of a single toolbar for a page quite a bit.

If you do not respect contexts, what can happen is a dl'ed subrequest can add its own toolbar options to the toolbars actually created for the main request. (Which can have really funny effects if you have, say, 10 dls on a single page.)

The original implementation in midcom_helper_toolbars is not safe in this respect.

Proposed API

Even though the current midcom_helper_toolbars could be easily adapted to fix the dl problem, I propose to create a new implementation in the midcom_services hirarchy. (We are talking about around 50 lines of code + docs, so this is not that much of an issue.)

This will ease transition to the new system asquite some components are out there still using the old class. Replacing them could have interesting side effects. These components will need some additional changes anyway as the toolbar api will be extended a bit to accomodate the idea of object-specific toolbars.

The `toolbars` Service

What I propose is having a midcom_services_toolbars available as $_MIDCOM->toolbars which serves as a central proxy for all toolbar related operations.

Basic API draft:

class midcom_services_toolbars
{
    midcom_helper_toolbar_node & get_node_toolbar($context = $current_context);
    midcom_helper_toolbar_page & get_page_toolbar($context = $current_context);

    midcom_helper_toolbar_object create_object_toolbar(&object);

    string render_node_toolbar($context = $current_context);
    string render_page_toolbar($context = $current_context);
}

Due to the context sensitivity, it is currently not possible to provide direct references to the object and page toolbar instances; there is no context switching notification to any service in the current core (and it won't be added before the page-transition for reasons not important here right now).

The getters will default to the currently active context, which is useful if you need to add site-local stuff into it.

// some site style element
$toolbar =& $_MIDCOM->get_node_toolbar();
$toolbar->do_something($tm);

Rendering though is even easier:

// some site style element
$_MIDCOM->toolbars->render_node_toolbar();

You will usually only render the current context's toolbars. The $context option should be considered reversed for core usage.

Note, that the render call will automatically add the neccessary CSS classes if neccessary.

The toolbar classes

All toolbar instances used will derive from midcom_helper_toolbar.

The code of this class needs to be cleaned up a bit, especially to allow easier default usage. Instead of the current

$toolbar->add_item(Array(
    MIDCOM_TOOLBAR_URL => $data['delete_url'],
    MIDCOM_TOOLBAR_LABEL => $data['l10n_midcom']->get('delete'),
    MIDCOM_TOOLBAR_HELPTEXT => null,
    MIDCOM_TOOLBAR_ICON => 'stock-icons/16x16/trash.png',
    MIDCOM_TOOLBAR_ENABLED => true,
));

the class should automatically apply the defaults:

$toolbar->add_item(Array(
    MIDCOM_TOOLBAR_URL => $data['delete_url'],
    MIDCOM_TOOLBAR_LABEL => $data['l10n_midcom']->get('delete'),
    MIDCOM_TOOLBAR_ICON => 'stock-icons/16x16/trash.png',
));

Apart from this the page toolbar must have at least one special interface function:

class midcom_helper_toolbar_page extends midcom_helper_toolbar
{
    void bind_to_object(&$object);
}

The bind_to_object call essentially does add the object toolbar specific options to the given page toolbar. This is for the very common use-case where there is just a single object per page.

Depending on the implementation itself, other private helpers and public convenience methods will be added, of course.

Request API integration

The request base class will import references to the toolbars of the current context into private memebers, which will be propagated into handler classes as well.

class midcom_baseclasses_components_request ...
{
    var $_node_toolbar;
    var $_page_toolbar;

    ...

    function initialize(...)
    {
        ...
        $this->_node_toolbar =& $_MIDCOM->toolbars->get_node_toolbar();
        $this->_page_toolbar =& $_MIDCOM->toolbars->get_page_toolbar();

        ...
    }
}

This will be done not only for convenience but also to protect the average user from missing the reference-assignment neccessary.

Open questions / TODOs

Should the toolbar classes be moved into the services hirarchy as well? Example: midcom_helper_toolbars_toolbar instead of midcom_helper_toolbar That way all code would be in the same place using the same convention most new services employ. Typing shouldn't be an issue, as all toolbars are created using the toolbars service factory methods.
There is urgent need for beautification of these toolbars, especially various styles would be good. Some advanced menu style could be of help.
Multi-level toolbars (in conjunciton with a dropdown menu style) would be useful as well.
The browser-driven tooltips should be replaced by some Overlib driven tooltip to work around the limitations of the browsers when rendering the tooltips (and thus also allowing HTML there).

Midcom site startup and configuration.

Sat, 11 Feb 2006 18:25:58 +0000

This is an outline for how future MidCOM Sites should be created and configured.

Background:

The current solution to set up MidCOM sites uses the Midgard Template to create sites. The current template is slow and is hard to modify.

For those who do not know, the following is an example of the setup of a very basic MidCOM site.

Midcom goes through three stages:

initialization and startup
Content output
Ending.

The most basic site template for use in MidCOM is this:

<?php 
$GLOBALS['midcom_config_local']['midcom_root_topic_guid'] =
"860f1b5af98b409abd916a80cdb0d68e";
require 'midcom.php';
$_MIDCOM->codeinit();
?>
<html>
<head>
<?php
echo $_MIDCOM->print_head_elements();

?>
</head>
<body <?php echo $_MIDCOM->print_jsonload(); ?> > 

content(); ?>

</body></html>
finish(); ?>

Site configuration

Today site configuration is contained in parameters to the host. This is fairly inefficient as this generates a large set of queries to the database on every request.

The new system will not use a common rootpage for all sites. Instead common functionality is shared using styles.

Instead of keeping the configuration in parameters it should be kept in a simple pageelement that is included into the request by changing the normal midgard-root.php file to something like this:

<(code-midcom-start)> <-- This element is autogenerated as outlined above
<(code-global)>
<(code-init)>
<(ROOT)> <-- the style calls $_MIDCOM->content()
<(code-finish)>
<(code-midcom-finish)> <-- Calling $_MIDCOM->finish();

The following parameter must be set in the configuration:

$GLOBALS['midcom_config_local']['midcom_root_topic_guid'] =
"860f1b5af98b409abd916a80cdb0d68e";

Versioning

Hosts using this configuration will be defined as using version 1 and have the parameter "midgard", "midcom-configuration-version" set to 1.

$host->parameter("midgard", "midcom-configuration-version", 1 );

Hosts missing this parameter are expected not to run Midcom unless they have set their rootpage to the common MidCOM rootpage used in Midcom-Template today.

Site authentication

Today site authentication is handled by the old Nemein-Auth library. This library is snippetbased and old. The new MidCOM authentication libary exists and should be used instead. Thus there will not be anything authentication related in the simple code-init.

The "midcom_site" array

Today, the template populates the midcom_site array with some objects as well.

This array should only be created in the compability style for the old Midcom-Template.

Site Configuration

Setting of page/host configuration options handled through a datamanager2 schema driven MidCOM.

The editor most be accessable even when the site is down for some reason.

This could be done today using a dummy topic[1] that could be set to run MidCOM with a configuration module, and then run a normal MidCOM session. This is a stopgap measure.

Another option is a very simple stylelement that lets the user edit just the path to midcom and/or the generated pageelement in an atempt to generate a working configuration.

Implementation

The following needs to be done:

A MidCOM for generating the configuration must be created.
Todays M-T must be refactored into two styles:
- one simple style
- one backward compatible style Both styles depend on a common root style.
A midcom for creating hosts must be made.
A midcom for creating Sitegroups must be made.
A midcom that takes the other midcom and makes a Form-wizard out of them must be made.

Please note that many of the implementation details must be left for another mRFC (f.x. how should sitecreation be configured in the future).

[1] Dummy topic Midcom currently needs a topic to be able to run. By creating a set of dummy topics and then setting the $GLOBALS['midcom_config_local']['midcom_root_topic_guid'] it is possible to run a normal midcom component without having to create a local topic for it.

Workflow in Midgard

Wed, 11 Jan 2006 13:39:22 +0000

This proposal describes a workflow engine built for handling tasks, processes and workflows within MidCOM applications. This mRFC been submitted to the Midgard Community for discussion and approval under the Creative Commons Attribution-ShareAlike license.

Workflow

Wikipedia defines workflow as:

Workflow is the operational aspect of a work procedure: how tasks are structured, who performs them, what their relative order is, how they are synchronized, how information flows to support the tasks and how tasks are being tracked. As the dimension of time is considered in Workflow, Workflow considers "throughput" as a distinct measure. Workflow problems can be modeled and analyzed using Petri nets.

Workflow is not for everybody

In the Groupware bad paper, Jamie Zawinski comments that

"Groupware" is all about things like "workflow", which means, "the chairman of the committee has emailed me this checklist, and I'm done with item 3, so I want to check off item 3, so this document must be sent back to my supervisor to approve the fact that item 3 is changing from 'unchecked' to 'checked', and once he does that, it can be directed back to committee for review."

Nobody cares about that shit. Nobody you'd want to talk to, anyway.

While this is true for a lot of use cases, there are still heavily regulated industries and activities where well defined and coordinated workflows are a must. This mRFC is split into two pieces: Task management and Workflow management. While workflows are needed by only some, task management should be useful for almost all Midgard users.

Prototypes and implementations

TODO

Object model

Trail

Trails are the prototypes of processes
Trails have a title and description
Trails may have a field map used if the trail is instantiated by a Spawn step in another process for copying data from that process

Step

Steps have a title and description
Steps have a type telling which workflow phase it is
Steps have an up field telling after which step (predecessor) they begin
- Join and Start steps don't have up field defined, they get their predecessors from deliverables
- Only Decisions and Splits may have multiple successors
Steps are the prototypes of tasks and belong to a trail
Steps are assigned to roles (i.e. midgard_groups)
Steps have two time definitions: start and end (deadline)
- The end definition is some offset from a selected earlier step end
- The start definition is some offset from the step's end
- Both offset selections provide an integer field and pulldown (hours, days, weeks, months) for user input

Process

Trail is the parent of the process
- Process creation is a midgard:create privilege inherited from trail
Process has an owner person, and started and completed timestamps
Process has a start timestamp telling when the Start task has to be created
- There is a daily check in cron service to start processes

Task

Mainly the existing org_openpsa_task implementation
Task has optional process and step link attributes
Tasks are assigned to persons (single or multiple) using member objects
Autocloseable
Skippable

Workflow phases

All workflow phases are modeled as steps, and handled during process

Start (circle)

Start is never instantiated as Task, and is simply used to lead to first actual step

Action (box)

Actions are regular tasks within the workflow
Actions always have a predecessor (up), and a successor (phase which has up pointing to this step)

Decision (diamond)

Decision editor asks for icon and text label of decision per each child step
- The editor must be available both in the decision step, and in each child
- Icons are presented as a selection list populated from component config array

Split (triangle pointing down)

Split is never instantiated as Task, and is simply used for pointing to succeeding steps

Join (triangle pointing up)

Join is a real Auto-closing task pointing to predecessors that must be completed via deliverables
Join is created only after all predecessor tasks exist
Joins don't have assignees in most cases -> rendered only in process view

End (two concentric circles, innermost filled)

End is never instantiated as a Task. Instead, the process is completed

Spawn (octagon)

Starts another process, passing some information to it
Spawn is a "dead end" step that that may not have successors, and does not complete the process

Prototype management

UI description/mock-up here

Task management

Task handling is centered into a Task Inbox where the user can see a list of her current open tasks. The tasks may come from active workflow processes, OpenPsa projects, support tickets, or be simply standalone tasks like "Remember to buy milk".

Deliverables

Deliverables are actual results of the task work. For example, if user is assigned to write a report about something, the deliverable could be a new OpenOffice.org document placed into a directory in the org.openpsa.documents repository, or it could be a wiki page.

As most workflows are related to highly regulated and paper-based processes, the most important deliverable will be "Create document from template", which will present user with a datamanager form and an office document template connected to that particular workflow, and will store the resulting document into a specified directory.

Deliverable Plugin Interface

Pseudocode/stubs from existing implementation.

<?php
/**
 * Deliverable object definition in PHP form (actually this is MgdSchema object)
 */
class org_openpsa_deliverable
{
    var $id"; //Local id of the deliverable
    var $plugin; //Name of the plugin used
    var $task; //Id of the task this deliverable is linked to
    var $target; /* Target of the plugin, the plugin interprets the target as it sees fit
                    but generally this is a GUID */

    /**
     * Instances the deliverable plugin interface for this deliverable object 
     * and checks for status via it.
     */
    function status()
    {
    }
}

/**
 * Wrapper class for deliverables operations
 *
 * This allows us to change the ways the plugins work and are loaded at a later
 * time (in case we wish to move them to the actual target components)
 */
class org_openpsa_projects_deliverables_interface
{
    var $_plugins = array(); //List of plugins available

    /**
     * Searches for plugins and fills $this->_plugins
     */
    function _find_plugins()
    {
    }

    /**
     * Returns an array of available plugins as objects
     */
    function list_plugins()
    {
    }

    /**
     * Checks all deliverables for task returns true if all return true otherwise false
     * @param integer $task_id Identifier of an OpenPsa Projects task
     * @return boolean Whether all deliverables of the task have been completed
     */
    function check_all_deliverables_status($task_id)
    {
    }

    /**
     * Takes plugin name and returns plugins classname for instancing
     */
    function _resolve_plugin($plugin)
    {
    }

    /**
     * Based on deliverable object id/guid returns correct plugin initialized
     */
    function get_deliverable_plugin($identifier)
    {
    }

    /**
     * Returns an empty (not initialized with deliverable object) plugin for plugin name
     */
    function get_plugin($name)
    {
    }
}

/**
 * Baseclass for deliverables plugins
 */
class org_openpsa_projects_deliverables_interface_plugin
{
    var $_deliverable; //A deliverable object
    var $name; //Name of the plugin
    var $description; //Description of the plugin

    /**
     * Returns true or false depending on whether the plugin considers the target
     * "delivered" or not.
     * @return boolean
     */
    function status()
    {
    }

    /**
     * In task view the select plugin(s) row for this
     */
    function render_select_plugin($task_id=NULL)
    {
    }

    /**
     * Creates the deliverable object (NOTE: not the target)
     */
    function handle_select_plugin($task_id)
    {
    }

    /**
     * Depending on deliverable status this will either render a way to
     * create the target or a link to the target
     *
     * There are two modes, 'task' and 'todo' for different renderings.
     */
    function render_deliverable($mode = 'task')
    {
    }

    /**
     * This will create the deliverable target
     *
     * At this time the mode selection has no effect.
     */
    function handle_deliverable($mode = 'task')
    {
    }
}

?>

Task micro-workflow

In addition to the regular workflows, all tasks within Midgard have a micro-workflow related to assigning it between people. This micro-workflow goes in the following way:

Project manager assigns tasks to resources
Resources can be local or remote (DBE) users
Resources can accept or decline task
- Hours can be reported to task after acceptance
When task is finished resources can mark it as completed
- Task's deliverables (document, SVN commit, ...) must be completed first
Project manager must approve the task before it is closed

When user assigns a task to herself the proposed phase is skipped and task is automatically accepted by the user, by same logic if user who is also manager of task marks it as completed the task is approved and closed automatically.

Getting Things Done

Getting Things Done is a popular task management methodology that focuses on storing all tasks, and making them easily filtrable based on the current working mode, or context. An important concept is elimination of "stuff":

Getting Things Done succeeds because it first addresses a critical barrier to completing the atomic tasks that we want to accomplish in a given day. That’s “stuff.” Amorphous, unactionable, flop-sweat-inducing stuff. David says:

Here’s how I define “stuff:” anything you have allowed into your psychological or physical world that doesn’t belong where it is, but for which you haven’t yet determined the desired outcome and the next action step. [pg. 17]

To follow the Getting Things Done model, the Midgard task manager must make addition and timing of new tasks easy. It also must enable marking tasks into specific contexts, and filtering them by those.

Digital Business Ecosystem

Midgard's task management is connected to the Digital Business Ecosystem. With DBE Midgard tasks can be replicated between business partners in a P2P network.

The DBE connection works transparently for the user of the Midgard task manager. she just assigns tasks to remote users (identified by a different icon in user listings), or receives tasks from them. The micro-workflow and handling of deliverables remains the same.

The actual DBE implementation is a Java ServENT (Server-Client) that monitors changes in tasks through Midgard-Java and pushes them to partners via the DBE network.

Synchronization

To support Getting Things Done, Midgard task management should try to reach beyond the web. Lots of the Contexts may be something that must be done away from the desktop, and these should be synchronizable to the smartphone, PDA or whatever.

The main synchronization protocol for mobile devices is SyncML. While a Horde SyncML implementation in PHP exists, it is lacking in client support. Midgard task manager, and other PIM applications should support SyncML through the Java-based Sync4J implementation instead.

In addition to SyncML, the task manager should seek to support subscribe-able vTODO output and possible related Microformats.

Note: According to Tantek from #microformats, the hCalendar microformat can be directly used for vTodo.

Misc

We need tasks that auto-close when all deliverables have been completed - Joins are tasks that have the steps to join as deliverables - Deliverable plugins call a task on_deliverable_completed() method

Idea: "skip step", which opens the next step but keeps current task open as well, steps need to be specifically marked as "skippable" in the workflow.

Idea: inserting user-written entries for process's activity log

Glossary

assignee: person that has been assigned to complete a task
deliverable: desired outcome of a task. e.g. a document written or decision made by assignee. Task may have multiple deliverables
process: an instance of a trail
role: abstraction of assignees based on business role. In Midgard terms, a midgard_group
step: one step in workflow, steps are prototypes of tasks, steps assignees are roles in stead of persons
task: any activity one needs to complete
trail: defines the workflow as a series of steps, trails are prototypes of processes

Related work

OpenFlow - workflow engine for Python
Galaxia - PHP workflow engine
An activity based Workflow Engine for PHP
wftk - Workflow Toolkit for C and SOAP
OpenWFE - Workflow engine for Java, Python and Perl
Workflow patterns research
Workflow for Midgard 1.4 (Catalyst IT, New Zealand)
Building a Smarter To-Do List Part I and Part II (Merlin Mann)