Midgard3 API proposal
-
Piotr Pokora
Midgard3 API proposal
Mon August 16 2010 16:15:20 UTCHi!
Here's first proposal for Midgard3 API.
## Main goals:
* provide more flexible content repository system
* support SQL, SPARQL or any other query language
* store content not only in SQL database
* allow partial or full implementation using any language
Please note, vala is used as example language. Code hasn't been compiled
so complain if there's any issue. Also vala is used by purpose.
We're "going GIR" so GLib and GObject are important here.
## Basic architecture
http://github.com/midgardproject/proposals/blob/master/Semantic
Data/API/architecture.png
## Volatile objects
Any object (SchemaObject, DBObject, WorkspaceObject) is not tied to
database or connection any longer. As is, can not be saved and stored in
database with known create or update methods. Objects just keep some
values and are valid as long as application exists.
http://github.com/midgardproject/proposals/blob/master/Semantic%20Data/API/midgard_schema_object.vala
## Mappings
SchemaManager and StorageMapper provide reflection and introspection for
registered classes. SchemaManager describes classes, inheritance,
properties types, etc. StorageMapper underlying storage schema. In case
of SQL, it describes tables, fields, field types, default values, etc.
XML implementation might describe directories, file names, xml node
names, etc. With both, you can register classes, introspect and define
how to store them.
The same class (registered with the same SchemaManager), can have
different storage, for which implementation exist. For example XML
storage implementation might perfectly describe xml files with
serialized objects known from Midgard2.
Even more, any application written in particular language might
implement additional Storage and StorageMappers, using routines which
feel best for specific purpose.
There's no distinction if SQL database is used as storage or XML as
replication data. Two applications may replicate data using one, common
SQLite database for example.
http://github.com/midgardproject/proposals/blob/master/Semantic%20Data/API/midgard_schema.vala
## Storage activity (CRUD)
StorageManager is an entry point to underlying storage.
It's responsible to create storage, update it, and provide content
manager. In case of SQL database it replaces MidgardConnection.
StorageContentManager, probably the most important part of new API
stores given objects in underlying storage. Its purpose is to make
volatile SchemaObjects persistent.
It's very important to understand: every object may be stored as many
times as needed by as many content managers as needed. For example,
object may be stored in two different databases (without any need to
create new instance for given connection and storage), xml file or in
any other storage.
There's no limitation how many storage managers might be used by
application. For example, two applications might keep schema information
in SQLite database while storing content in different, dedicated databases.
QueryManager provides query tools to read objects using required
constraints. Very similar to QueryBuilder and MidgardQuerySelect but
provides clear and clean API for different queries like, pure SQL
(DBObject and SchemaObject), Workspace (workspace objects), SPARQL.
http://github.com/midgardproject/proposals/blob/master/Semantic%20Data/API/midgard_storage.vala
http://github.com/midgardproject/proposals/blob/master/Semantic%20Data/API/midgard_query.vala
## Examples
There are two examples, first one shows how to define schema and storage
mapper and second, how to register classes from available schemas.
http://github.com/midgardproject/proposals/blob/master/Semantic%20Data/API/example.vala
http://github.com/midgardproject/proposals/blob/master/Semantic%20Data/API/example2.vala
There's also simple migration file which shows how some functionalities
can be replaced with new API.
http://github.com/midgardproject/proposals/blob/master/Semantic%20Data/API/migration.txt
Please, comment. Also, if API feels uncomfortable or annoying, complain.
Piotras
_______________________________________________
dev mailing list
dev@lists.midgard-project.org
http://lists.midgard-project.org/mailman/listinfo/dev -
-
Re: [midgard-dev] Midgard3 API proposal
Fri August 20 2010 15:05:08 UTCOn Mon, Aug 16, 2010 at 19:11, Piotr Pokora <piotrek.pokora@gmail.com> wrote:
> Hi!
>
> Here's first proposal for Midgard3 API.
So we finally discussed the proposal with Piotras via IM today and
found what actually been keeping me unsure about the proposal (Piotras
is a good psychologist now :).
My main objection was that API use seems to be very repetitive and
somehow does not protect from possible API misuses. To solve that I
have proposed to approach it from a chaining perspective.
We are working with several mapping levels here and almost all of them
are rather uniform in their use cases:
- you construct a structured object which is a composition of smaller
entities (objects);
- you map the structured object to each other;
- you reflect on the structure later at some point.
First use case actually interesting here. This structured object's
construction is not really something that immediately causes
activities in other levels. Instead, you pile up construction until
you think it is done and then you would map (realize) the construction
to a different structured object. This second step and use case
actually can be compared to a COMMIT of a transaction. Everything you
have done before is and can be modified but only mapping brings the
object to life by allowing it to have hard mapping with actual storage
layer.
Reflection part is easy -- when you have structured object and want to
explore its entities, reflection is needed.
So these two distinctive use cases (COMMIT is a single step and can be
ignored for the moment) ask for different APIs.
The construction use case cares about ability to easily build up the
object. For that one common approach is to employ a chaining: an
object is created and then every method that modifies it actually
returns the reference to the object itself so that further
modification can be applied. Note that at this stage we don't need to
validate correctness yet as partially constructed object might be in
incorrect state (conflicting relations between the entities, for
example) but validation would be part of "COMMIT" step.
From other hand, the fact that object's state can be allowed to stay
incomplete until it is realized, gives practical freedom:
object-modifying methods do not need to return errors, they just
return the reference to the modified object. And that is guaranteed to
exist because, after all, it is the object they operate on. This is
basic concept of chaining.
Let's say we have some class Type and it has a method
addProperty(name, ...) that returns the type reference it was called
on. Then new person type may be described as (language is irrelevant
here, so I'm using some abstract programming language here with
"commonly known syntax"):
person = new Type
person.addProperty(new Property("name", ...))
.addProperty(new Property("surname", ...))
.addProperty(new Property("foobar", ...)).commit()
so we have created a type with three properties by chaining calls to
every addProperty and asked for the type validation (reference checks
and so on, for completeness purposes).
If you look at it, type and storage manager association can be handled
in pretty much same way. By removing need to check correctness of an
intermediate type state, we allow for easy readable code. The
verification part happens in commit() method (I'm not tied to the
names, just an example), which can be as complex as needed and might
perform needed validation.
Based on this concept Piotras has built some prototype for the API:
http://friendpaste.com/4av6OtxoOvp4IGlxPFtLPW where naming of methods
is again, a prototype, not a hard defined thing.
For reflection API you don't need chaining in all the cases. However,
construction phase is considered important enough to work on
simplification.
--
/ Alexander Bokovoy
_______________________________________________
dev mailing list
dev@lists.midgard-project.org
http://lists.midgard-project.org/mailman/listinfo/dev -
-
Re: [midgard-dev] Midgard3 API proposal
Wed August 25 2010 09:00:06 UTCHi!
> Based on this concept Piotras has built some prototype for the API:
> http://friendpaste.com/4av6OtxoOvp4IGlxPFtLPW where naming of methods
> is again, a prototype, not a hard defined thing.
I created new API proposal which is built on top of model concept:
http://github.com/midgardproject/proposals/tree/master/Semantic%20Data/API/model/
Basically:
* schemas and mappers are just "some" models
* models can be associated with other models
* particular storage managers can do some actions on given models
(actions are rather preparations instead of actions)
* execution is rather bulk one, but can be done in small steps
API itself doesn't focus on objects (MidgardObject known from Midgard2).
Those are only volatile data holders from API point of view.
Piotras
_______________________________________________
dev mailing list
dev@lists.midgard-project.org
http://lists.midgard-project.org/mailman/listinfo/dev -
-
Re: [midgard-dev] Midgard3 API proposal
Mon August 30 2010 10:15:07 UTCOn Wed, Aug 25, 2010 at 12:52 PM, Piotr Pokora <piotrek.pokora@gmail.com> wrote:
> I created new API proposal which is built on top of model concept:
> http://github.com/midgardproject/proposals/tree/master/Semantic%20Data/API/model/
example looks good, at the first sight, but can you tell more about
Model interface API?
any Model has these methods:
public abstract SchemaModel add_model (SchemaModel model);
public abstract SchemaModel get_model_by_name (string name);
public abstract SchemaModel add_parent_model (SchemaModel model);
public abstract SchemaModel get_parent_model ();
add_model/get_model_by_name are for child-models and add_parent_mode,
get_parent_model are for parent models.
right?
So it is the many-to-many graph. probably even cyclic?
For RDF-ontologies we need to be able to define the following:
1) there are properties
2) property has public name (URI) and local data-type (string, number, etc.)
3) property can be associated to any number of Schemas
object created using such schema will consist of trigraphs: (object +
property_uri + value)
So, how can we do this with proposed API?
--
Alexey Zakhlestin
http://www.milkfarmsoft.com/
_______________________________________________
dev mailing list
dev@lists.midgard-project.org
http://lists.midgard-project.org/mailman/listinfo/dev -
-
Re: [midgard-dev] Midgard3 API proposal
Mon August 30 2010 10:40:05 UTCHi!
> any Model has these methods:
> public abstract SchemaModel add_model (SchemaModel model);
> public abstract SchemaModel get_model_by_name (string name);
> public abstract SchemaModel add_parent_model (SchemaModel model);
> public abstract SchemaModel get_parent_model ();
>
> add_model/get_model_by_name are for child-models and add_parent_mode,
> get_parent_model are for parent models.
> right?
> So it is the many-to-many graph. probably even cyclic?
It depends on implementation actually. Main idea is to work with models
easily and always keep model alive (unless explicitly destroyed of course).
> For RDF-ontologies we need to be able to define the following:
> 1) there are properties
> 2) property has public name (URI) and local data-type (string, number, etc.)
> 3) property can be associated to any number of Schemas
>
> object created using such schema will consist of trigraphs: (object +
> property_uri + value)
>
> So, how can we do this with proposed API?
It depends if you think about SchemaModel or StorageModel. They both
implements model, but Schema one is not tied to any storage by design.
If property can be associated to any number of Schemas, then
* as many SchemaModelProperty must be created as needed
* as many StoragaModelProperty must be created as needed
(if value must be stored, in other case only first model is valid)
In other words, *if* implementation is based on GObject, every property
must be registered explicitly for any class, or may be "work arounded"
by SchemaObject implementation.
If property uses uri namespaces:
There's NamespaceManager:
http://github.com/midgardproject/proposals/blob/master/Semantic%20Data/API/midgard_namespace.vala
It can be tied to StorageManager if needed, but also might be
SchemaObject (Query, CRUD) itself.
SchemaObject (Storable derived):
http://github.com/midgardproject/proposals/blob/master/Semantic%20Data/API/midgard_schema_object.vala
provides methods to set/get/list properties, so its implementation
should be combined NamespaceManager to provide safe way of
reading/writing namespaced properties.
Piotras
_______________________________________________
dev mailing list
dev@lists.midgard-project.org
http://lists.midgard-project.org/mailman/listinfo/dev -
-
Re: [midgard-dev] Midgard3 API proposal
Mon August 30 2010 16:50:06 UTC>> Based on this concept Piotras has built some prototype for the API:
>> http://friendpaste.com/4av6OtxoOvp4IGlxPFtLPW where naming of methods
>> is again, a prototype, not a hard defined thing.
>
> I created new API proposal which is built on top of model concept:
> http://github.com/midgardproject/proposals/tree/master/Semantic%20Data/API/model/
There are two more issues (at least :) I didn't resolve so far.
StorageManager is a (kind of) facade for other storage related classes.
Like ContentManager, ModelManager or QueryManager.
Returned QueryManager could be responsible to create particular Query
class, like Select, Iterator, Data, and so on. I wonder if it makes
sense to provide such functionality (with predefined and limited number
of Query classes) or make them standalone, constructed for given
StorageManager?
Another issue is similar. I am pretty sure, with SQL StorageManager
implementation, we will need more than one StorageManager (different SQL
class instance) objects sharing the same underlying database connection.
Those might be, for example, raw SQL and Workspace managers used by the
same application.
Should we, for such case, provide some adapters for StorageManager or is
it the case needed so seldom, that new explicit connection for new
StorageManager should be made?
Piotras
_______________________________________________
dev mailing list
dev@lists.midgard-project.org
http://lists.midgard-project.org/mailman/listinfo/dev -
-
Re: [midgard-dev] Midgard3 API proposal
Tue August 31 2010 09:00:05 UTCOn 30.08.2010, at 14:36, Piotr Pokora wrote:
> Hi!
>
>> any Model has these methods:
>> public abstract SchemaModel add_model (SchemaModel model);
>> public abstract SchemaModel get_model_by_name (string name);
>> public abstract SchemaModel add_parent_model (SchemaModel model);
>> public abstract SchemaModel get_parent_model ();
>>
>> add_model/get_model_by_name are for child-models and add_parent_mode,
>> get_parent_model are for parent models.
>> right?
>> So it is the many-to-many graph. probably even cyclic?
>
> It depends on implementation actually. Main idea is to work with models
> easily and always keep model alive (unless explicitly destroyed of course).
well, aren't we talking about implementation? ;)
I just try to understand consequences of this API choice and identify edge-cases.
for example: I don't understand why "add_parent_model(model)" and "get_parent_model()" methods are defined this way.
looks like we can add arbitrary number of parents (will those be on the same level or will they represent stack?) but we can get only single parent.
shouldn't we be able to get all of those?
or was intention to set only single parent?
then it should be defined as get/set property.
but, if so, how will we be able to attach single ModelProperty to many models? (and we need this for RDF)
>> For RDF-ontologies we need to be able to define the following:
>> 1) there are properties
>> 2) property has public name (URI) and local data-type (string, number, etc.)
>> 3) property can be associated to any number of Schemas
>>
>> object created using such schema will consist of trigraphs: (object +
>> property_uri + value)
>>
>> So, how can we do this with proposed API?
>
> It depends if you think about SchemaModel or StorageModel. They both
> implements model, but Schema one is not tied to any storage by design.
SchemaModel will be RDF side of property.
StorageModel will be local side of property (linking RDF-URI to SQL datatype)
> If property can be associated to any number of Schemas, then
>
> * as many SchemaModelProperty must be created as needed
> * as many StoragaModelProperty must be created as needed
> (if value must be stored, in other case only first model is valid)
you mean we should just create a lot of SchemaModelProperties with the same URI?
that doesn't sound like a good idea
or did I misunderstand you?
probably we should just talk basing on some specific example.
I will start a separate thread for the use-case.
> In other words, *if* implementation is based on GObject, every property
> must be registered explicitly for any class, or may be "work arounded"
> by SchemaObject implementation.
>
> If property uses uri namespaces:
>
> There's NamespaceManager:
> http://github.com/midgardproject/proposals/blob/master/Semantic%20Data/API/midgard_namespace.vala
>
> It can be tied to StorageManager if needed, but also might be
> SchemaObject (Query, CRUD) itself.
>
> SchemaObject (Storable derived):
> http://github.com/midgardproject/proposals/blob/master/Semantic%20Data/API/midgard_schema_object.vala
>
> provides methods to set/get/list properties, so its implementation
> should be combined NamespaceManager to provide safe way of
> reading/writing namespaced properties.
yes. we need to do this.
on some level, there should be API which will allow us to interchange full (namespaced) and local (aliased) property names freely as was mentioned in Concept/Implementation documents.
_______________________________________________
dev mailing list
dev@lists.midgard-project.org
http://lists.midgard-project.org/mailman/listinfo/dev -
-
Re: [midgard-dev] Midgard3 API proposal
Tue August 31 2010 09:20:05 UTCOn 30.08.2010, at 20:42, Piotr Pokora wrote:
>>> Based on this concept Piotras has built some prototype for the API:
>>> http://friendpaste.com/4av6OtxoOvp4IGlxPFtLPW where naming of methods
>>> is again, a prototype, not a hard defined thing.
>>
>> I created new API proposal which is built on top of model concept:
>> http://github.com/midgardproject/proposals/tree/master/Semantic%20Data/API/model/
>
> There are two more issues (at least :) I didn't resolve so far.
> StorageManager is a (kind of) facade for other storage related classes.
> Like ContentManager, ModelManager or QueryManager.
>
> Returned QueryManager could be responsible to create particular Query
> class, like Select, Iterator, Data, and so on. I wonder if it makes
> sense to provide such functionality (with predefined and limited number
> of Query classes) or make them standalone, constructed for given
> StorageManager?
well, my thoughts were, that exact number of storage-query classes is dependent on storage.
we just need to have:
1) some entry-point, which will be able to convert schema-query to storage-query (and how exactly this storage-query looks like is just "implementation detail" which is not important to application developer)
2) some entry point, which will be able to execute this storage-query (that will be Executable interface, I guess)
as long as these are present Storage can have any set of query-classes (in the edge-case it even can be single object holding sql-string)
am I talking nonsense? ;-)
> Another issue is similar. I am pretty sure, with SQL StorageManager
> implementation, we will need more than one StorageManager (different SQL
> class instance) objects sharing the same underlying database connection.
> Those might be, for example, raw SQL and Workspace managers used by the
> same application.
wellÂ… we can represent it as the system of pipes/transformers, which convert queries to other form.
Something like this:
SchemaQuery => SchemaQuery in Workspace => StorageQuery
^ ^
(WorkspaceMapper) (StorageMapper)
in this case, we don't need to have actual connection for the workspace.
workspace can operate on higher level, above any storage-details.
> Should we, for such case, provide some adapters for StorageManager or is
> it the case needed so seldom, that new explicit connection for new
> StorageManager should be made?
_______________________________________________
dev mailing list
dev@lists.midgard-project.org
http://lists.midgard-project.org/mailman/listinfo/dev -
-
Re: [midgard-dev] Midgard3 API proposal
Tue August 31 2010 09:40:04 UTCHi!
> I just try to understand consequences of this API choice and identify edge-cases.
>
> for example: I don't understand why "add_parent_model(model)" and "get_parent_model()" methods are defined this way.
> looks like we can add arbitrary number of parents (will those be on the same level or will they represent stack?) but we can get only single parent.
> shouldn't we be able to get all of those?
>
> or was intention to set only single parent?
> then it should be defined as get/set property.
Right. get/set is much better here.
The idea is to define parent schama class or parent storage. And such
should be only one.
> but, if so, how will we be able to attach single ModelProperty to many models? (and we need this for RDF)
Implementation might be difficult for such case.
Use case: StorageContentManager saves some object.
It should read properties of an object (either via SchemaModel API or
using GObject API) and then should read all properties available from
StorageModelProperty.
Possible delta case should be computed somehow.
This is the case I a am not happy about. It's not only
StorageModelProperty issue. It's also StorageContentManager and its
familiy ones.
>> If property can be associated to any number of Schemas, then
>>
>> * as many SchemaModelProperty must be created as needed
>> * as many StoragaModelProperty must be created as needed
>> (if value must be stored, in other case only first model is valid)
>
> you mean we should just create a lot of SchemaModelProperties with the same URI?
> that doesn't sound like a good idea
>
> or did I misunderstand you?
No, you didn't. As wrote above. It's also StorageContentManager issue.
If property's URI going to be constant, than it's safe to duplicate it.
It also depends on the fact how such property is propagated during
runtime for an object.
>> SchemaObject (Storable derived):
>> http://github.com/midgardproject/proposals/blob/master/Semantic%20Data/API/midgard_schema_object.vala
>>
>> provides methods to set/get/list properties, so its implementation
>> should be combined NamespaceManager to provide safe way of
>> reading/writing namespaced properties.
>
> yes. we need to do this.
>
> on some level, there should be API which will allow us to interchange full (namespaced) and local (aliased) property names freely as was mentioned in Concept/Implementation documents.
Do you mean this?
MidgardStorage::setParameterAlias('HelloWorld', 'dc:title', 'title');
I think it can be managed via NamespaceManager.
Piotras
_______________________________________________
dev mailing list
dev@lists.midgard-project.org
http://lists.midgard-project.org/mailman/listinfo/dev -
-
Re: [midgard-dev] Midgard3 API proposal
Tue August 31 2010 10:05:05 UTCOn 31.08.2010, at 13:36, Piotr Pokora wrote:
>
> Do you mean this?
> MidgardStorage::setParameterAlias('HelloWorld', 'dc:title', 'title');
>
> I think it can be managed via NamespaceManager.
I mean application developer should be able to use any of these if namespace and parameter-alias are defined:
obj.setParameter('http://dublincore.org/documents/dcmi-namespace/title', 'new_title');
obj.setParameter('dc:title', 'new_title');
obj.setParameter('title', 'new_title');
this, as the bonus, would be nice too:
obj.title = new_title
_______________________________________________
dev mailing list
dev@lists.midgard-project.org
http://lists.midgard-project.org/mailman/listinfo/dev -
