An ontological analysis is, in principle, the evaluation of a selected modelling grammar from the viewpoint of a pre-defined and well-established ontology. The current focus of ontological analyses is on the bi-directional comparison of ontological constructs with the elements of the modelling grammar that is under analysis. Weber (1997) clarifies two major situations that may occur when a grammar is analysed according to an ontology. After a particular grammar has been analysed, an assessment of the modelling strengths and weaknesses of the grammar can be made according to whether some or any of the following situations arise out of the analysis.
Ontological incompleteness (or construct deficit) exists unless there is at least one grammatical construct for each ontological construct.
Ontological clarity is determined by the extent to which the grammar does not exhibit one or more of the following deficiencies:
Construct overload exists in a grammar if one grammatical construct represents more than one ontological construct.
Construct redundancy exists if more than one grammatical construct represents the same ontological construct.
Construct excess exists in a grammar when a grammatical construct is present that does not map to any ontological construct.
Though this type of ontological analysis is widely established, it still has a range of shortcomings. These shortcomings can be categorised into the three main phases of an ontological analysis: preparation of the input data, the process of conducting the analysis, and the evaluation and interpretation of the results. The first two identified shortcomings refer to the quality of the input data.
Most of the ontologies that are currently used for analysing modelling grammars have been specified in formal languages. While such formalisation is beneficial for a complete and precise specification of the ontology, it is not a very natural or intuitive specification. An ontology that is not clear and intuitive can lead to misinterpretations as the involved stakeholders have problems with the specifications. Furthermore, it forms a hurdle for the application of the ontology as it requires a deep understanding of the formal language in which it is specified.
The specification of an ontology typically requires a formal syntax, which allows the precise specification of the elements and relationships of the ontology. Such specifications are required, but are not necessarily intuitive. Consequently, textual descriptions of the ontology in ‘plain English’ often extend the formal specification. However, even if an ontology is specified in an intuitive and understandable language, the actual comparison with the selected modelling grammar remains a problem. Unless the ontology and the grammar are specified in the same language, it will be up to the coder to ‘mentally convert’ the two specifications into each other for comparison purposes, which adds a subjective element to the analysis. Different languages can also lead to different levels of detail and further complicate the analysis. In any case, they make a more automated comparison practically impossible. This is the typical situation in nearly all previous analyses.
The three further shortcomings identified below are related to the process of the ontological analysis and refer to what should be analysed, how it should be analysed as well as who should conduct the analysis.
The first decision that has to be made in the process of an ontological analysis is on the scope and depth of the analysis. Even though most ontologies have been discussed for many decades, they still undergo modifications and extensions. It is up to the researcher to clearly specify the selected version of the ontology and the scope and level of detail of the analysis. In our work in the area of Web Services, for example, it was often not clear what constructs form the core of the standard and, in fact, two researchers who conducted independent analyses of the same Web Services standard selected a different number of constructs.
Moreover, many ontological analyses focus solely on the constructs of the ontology and the constructs of the grammar but do not sufficiently consider the relationships between these constructs. The difficulty of clearly specifying the boundaries of the analysis, as well as the limited consideration of relationships between the ontological constructs, can lead to a lack of completeness.
After the scope and the level of detail of the analysis have been specified, it is typically up to the coder to decide on the procedure of the analysis, i.e. in what sequence will the ontological constructs and relationships be analysed? Currently, there are hardly any recommendations on where to start the analysis. This lack of procedural clarity underlies most analyses and has two consequences. First, a novice analyst lacks guidance in the process of conducting the ontological evaluation. Second, the procedure of the analysis can potentially have an impact on the results of the analysis. Thus, it is possible that two analyses of the same modelling grammar using the same ontological base, but that follow different processes, may lead to different outcomes.
An ontological analysis of a grammar requires not only detailed knowledge of the selected ontology and grammar, but also a good understanding of the languages in which the ontology and the grammar are specified. This requirement explains why most analyses are carried out by single researchers as opposed to research teams. Consequently, these analyses are based on the individual interpretations of the involved researcher, which adds significant subjectivity to the results. This problem is further compounded by the fact that, unlike other qualitative research projects, ontological analyses typically do not include attempts to further validate the results.
The five shortcomings identified above have a common flavour in that they heavily depend on the researcher conducting the ontological evaluation. Three further shortcomings have been identified, namely lack of result representation, lack of result classification and lack of relevance. These shortcomings are detailed below and refer to the outcomes of the analysis
The results of a complete ontological analysis, i.e. representation mapping and interpretation mapping, are typically summarised in two tables. These tables list all the ontological constructs (first table) and all the grammatical constructs (second table) and the corresponding constructs of the other meta model. Such tables can become quite lengthy and are typically not sorted in any particular order. They don’t provide any insights into the importance of identified deficiencies and they also don’t cluster the findings.
As indicated above, it is common practice to derive ontological deficiencies based on a comparison of the constructs in the ontology and the grammar. Ontological weaknesses are identified when corresponding constructs are missing in the mapping obtained between the ontology and the grammar or one-to-many (or many-to-one or even many-many) relationships exist in the mapping between the two. Such identified deficiencies are the typical starting point for the derivation of propositions and then hypotheses. In general, the ontological analysis does not make any statements regarding the relative importance of these findings in comparison with each other. Though this seems to be the established practice, it lacks more detailed insights into the significance of the results. It is to be expected, however, that missing support for a core construct of an ontology should be rated of higher importance than missing a construct corresponding to a minor ontological construct or a relationship. This lack of a more detailed statement regarding the significance of a potential shortcoming makes it difficult to judge quickly the outcomes of the results of two different sets of analyses (e.g. an ontological analysis of ARIS compared to an ontological analysis of UML).
Finally, the results of an ontological analysis should be perceived as relevant by the related stakeholders. However, if an ontological analysis leads, for example, to the outcome that Entity Relationship Models do not support the description of behaviour then it would hardly be surprising if the IS community developed a rather critical opinion of the worth of the analysis since this is both obvious and well known. It seems that an ontological analysis has to consider the purpose of the grammar as well as the background of the modeller who is applying this grammar. The application of a high-level and generic ontology does not consider this individual context and there is a danger that the outcomes can be perceived as trivial.