mRFC 0043: Midgard2 deployment model
- Background
- Terminology
- Usage scenarios
- CMS site deployment
- Making updates to customer's CMS-based site
- End-user application distribution
- Team collaboration
- Packaging
- Tools needed
- Potential issues
- Afterword
Background
With Midgard1, especially since the introduction of filesystem-based MidCOM, the deployment model of applications has been quite lacking. In the history of Midgard we have had two deployment models:
Everything served as a repligard XML file, including both content and the application itself (OpenPSA 1.x, Aegir, VMUC, midHoo)
Components installed via PEAR, templates and configuration via FileSync (OpenPSA 2)
Both of these models lack some features we need from a deployment system:
- Versioning of a Midgard application
- Separation of components from configuration to support component reuse
- Easy team collaboration through quick set-up and tear-down of application instances
- Possibility to either include content with the application or keep it separate
Terminology
Application: a Midgard-powered web site (combination of components, a page hierarchy, templates and configuration)
Bucket: a repository containing replication XML files of Midgard data, and a manifest file listing necessary components and their versions. Bucket can be either a Amazon S3 bucket or a git repository
Component: a MidCOM component providing its own manifest, and potentially routes, templates, MgdSchema definitions and Python-based helper processes
Usage scenarios
CMS site deployment
Alice wants to build her new website with Midgard. She has a Mac laptop, and her ISP has set up a Linux virtual server.
To get started, she downloads the Midgard App Builder and runs it. She instructs the App Builder to set up an empty Midgard application, with only the base MidCOM components installed.
Then she starts her work: she installs the components she needs for the site, converts her XHTML templates to TAL and configures the site structure by using Midgard's on-site administrative tools.
She also sets up a git repository and instructs Midgard to synchronize all changes using that repository as the application bucket.
When the site is ready for launch, she logs on to the Linux server and runs midgard2-setup, providing it with the URL of her application bucket. The Midgard setup tool installs the same set of components that she had on her laptop, imports the site data and sets up a virtual host.
The she tests the site on the actual server and launches it. As the server is monitoring changes to the application bucket, she can later update the website either on the server or on her laptop's local installation. This is useful as she often likes to blog while traveling.
Making updates to customer's CMS-based site
Bob has a customer that has been running a Midgard-powered site for two years now. They contacted him and asked for some layout changes to the site. Since editing a live site is a bad idea, Bob decides to do the changes on a test server.
First he checks if the customer already has their site in an application bucket. But as the site is old, they don't. So he creates an empty bucket to the company version control system and instructs the Midgard installation on the live server to export its data there.
When the whole site is in the bucket together with component information he then logs on to a development server. On the server he already has some other Midgard applications running. In order to not disturb them he decides to set up a new database.
To do this he writes a new Midgard configuration file specifying custom locations for the MgdSchemas, and a new database name. Then he runs midgard2-setup, giving it the name of the configuration file and the URL to the customer's application bucket.
midgard2-setup installs a local copy of the site, exactly like the one running on customer's server. If Bob had been in a hurry, he could have told midgard2-setup to skip installing some slower parts, like for example file attachments.
Now he logs to the copy of the website running on the development server and starts to work.
When the layout changes are done, he then instructs Midgard to export the changes to the customer's application bucket. Then he logs to the live server, and there instructs Midgard to import changes from the bucket. That done, the layout changes are live.
End-user application distribution
Charlie has built a simple but powerful web-based CRM solution that can be run on the desktop. He wants to start distributing it to his customers, with Mac OS X as the main target platform.
To do this, he creates a new github repository that will server as the application bucket. Then he exports his local installation of the application there, skipping some particular MgdSchema types like "customer" that contain his test data.
After this is done, he fires up Midgard App Builder and tells it to create a new application based on the bucket stored on github. This will result in a clean installation of Charlie's CRM app.
Now he packages that as a .dmg file, publishes it and starts marketing his application. Users who can install and use it just like they would any native Mac OS X app, even though it actually runs Midgard and MidCOM on the background.
When he later releases a new version of the application, he does this by publishing updated component versions and creating an updated manifest file that he then pushes into the application bucket on github.
Users of the CRM application will get automatically notified of the new version as their installations check the github repository periodically for changes. Updating the application is as simple as either running midgard2-setup on the command line, or clicking a button in the application interface itself.
Team collaboration
Dexter and Emily are both working in a company providing a social website based on Midgard. They're both developers working on different aspects of the web service.
TODO
Packaging
MidCOM components
In Midgard2, MidCOM components are going to be released via git repositories instead of PEAR packaging. A stable release is made by merging changes from development branches into a stable branch, tagging it and pushing to a repository server. In this sense, midcom_core is also a component as it follows exactly the same model.
Component manifest may list components (repository URL and branch or release tag) the component depends on.
Midgard buckets
Midgard applications are stored in "Buckets", which can be either git repositories or S3 buckets (other repositories, like WebDAV shares, may also be supported). The bucket contains a manifest file listing the components used by the application, and a collection of Midgard replication XML files grouped by their schema type.
Example bucket layout:
mybucket/
manifest.yml
midgard_person/
e0d837d091c511dba90a8ba58d0b46d346d3.xml
midgard_page
e27c3c8a91c511dbb4287d806e4281ab81ab.xml
e28b76a091c511dbb4287d806e4281ab81ab.xml
...
Tools needed
midgard2-installer
Tool that handles installation and updating of MidCOM components. When installing a new component, a local git checkout of the component is made (using the given branch or tag), the possible MgdSchemas shipped with the component are installed to Midgard, and the component itself is symlinked to its correct place.
midgard2-setup
Tool that handles setting up a new Midgard application. It checkouts a given Midgard bucket to the local system, installs the components specified in the bucket manifest (using midgard2-installer routines specified above), and then proceeds to import the Midgard replication XML files in the bucket.
Once components are in place and data has been imported, it then sets up an appropriate virtual host for serving the Midgard application. Once this is done, the Midgard application should be up-and-running and accessible via a browser.
midgard2-setup can either use an existing Midgard configuration file, or set up a new one with sane defaults (based on platform).
Midgard App Builder
Midgard App Builder is a tool based on midgard2 and the Fluid site-specific browser. It makes it possible to easily create and install Midgard applications as self-contained Mac OS X .app packages.
When a Midgard application has been created with App Builder, it will resemble a normal OS X application. Starting it by double-clicking the .app file will start midgard2, lighttpd and d-bus on the background and will open the Midgard application inside a Fluid site-specific browser window. Fluid is based on WebKit and provides many additional javascript APIs that enable applications to behave more like native OS X apps. For instance, they can bounce the dock icon or raise Growl notifications.
When creating a new Midgard application, App Builder should do the normal midgard2-setup tasks (installing the base data from a bucket). After this the fully running Midgard application can then be packaged as a .dmg image that any Mac user will be able to install easily. However, the Midgard bucket subscription on the background will still provide easy update of the base data, for when instance a new release of OpenPSA is made.
Similar tools like Midgard App Builder may be created for other platforms when we have more experiences with real-world usage.
Amazon EC2 image
Midgard would be much more cloud-ready if we provided an EC2 image with preinstalled Midgard. When started up, it would connect to a configured bucket and import the application from there.
Then the EC2 node would be completely ready for business. This would make scaling up websites very easy. However, much software needs to be written to make the kind of cloud setup described in an earlier blog post possible:
http://bergie.iki.fi/blog/midgard2-future_in_the_clouds/
Potential issues
Order of importing objects
Midgard's replication tools require dependencies (objects referenced to via linked fields) to exist in database before objects can be imported. This means that the installer tool must construct a dependency chain and install objects in correct order.
One way to do this is to try importing all objects in sequence. If importing an object fails, move it to the end of the queue and try again later. Repeat until queue is empty. But there may be more optimal solutions to this too.
If a dependency fails to be found completely, we could populate a "skeleton object" into the database. The skeleton object would be completely empty but have the correct GUID, allowing it to be "updated" when the correct object is later being imported.
Keeping Midgard itself up-to-date
Packaging and installation of Midgard (and related software like Python, PHP and Lighttpd) is outside the scope of this document.
On Linux systems they can be installed and updated through native package management tools.
Midgard App Builder makes this slightly trickier, as the .app package will contain both Midgard and the actual database. It might be good if Midgard applications generated through App Builder could somehow move their database under user's home directory after initial run. This would make updating Midgard applications on OS X as easy as any native app, as users could just copy a new version of the .app package on top of the existing one and their data would remain intact.
Afterword
The idea here is the result of much frustration in Midgard's current deployment model. It is partly inspired by discussions in recent Gatherings, and partly by Edd Dumbill's We're all ops people now blog post.
The initial version was written on a rather choppy Austrian Arrows flight from Copenhagen to Vienna, while the plane was working its way around some thunderstorms near Berlin.