=======

Artefacts

Artefact refers to informations related to a AI component in the klarity workflow. When the information refers to the values, whether required or measured, of a component in the klarity workflow, it can be commonly identified as a 'metric'. It is the case, for instance when measuring the precision of a model, the actual coverage of the operational design domain, the quality of a data set, etc.

Yet, some information, such as requirements, observations, data sample... can not be considered as metrics. It is for this reason that we have chosen to use the generic term artefact to encompass all information, whatever its typology. Moreover, in case an artefact is based on metrics, the artefact manages how multiple metrics are organized, compared, adapted to be presented to the stakeholder in order to commit a decision.

The artefacts are intended to be viewed in the KD interface. They are the subject of discussions between the users, who will make proposals for rejection or validation, this process is described in the usage section.

Artefacts of a PCIV are described in the configuration file

Artefacts actual values are contained in a payload file which is sent with each new PCIV.

Artefact Group An artefact is always associated to an artefact group, yet an artefact group can contain only one artefact.

Linked Artefacts : An artefact can be linked to other artefacts, in which case this will be indicated in the corresponding payload.

From the KD point of view, artefacts are immutable values.

Artefact Specification

Configuration

The configuration designates the whole AI-Component life-cycle configuration. It describes, for each stage of the Klarity process, what group is to be displayed and at which abstraction level, what are the artefacts in each of these groups, how and if groups shall be validated, by who, and so on.

Consequently, artefacts are first specificed in a project configuration file where each of them is described as it follows :

{
  "name":
  "scope":
  "type":
  "title":
  "description":
  "tooltip":
  "config": {
  }
},

• name designates the technical name, it works as an id used to manipulate this artefact,
• scope each artefact is associated to at least one scope,
• type via the type of artefact, the rendering will be different depending on the level of abstraction,
• title designates the name of the artefact as it will be displayed on Klarity front-end,
• description clarifies the reading of the artefact and the decision potentially expected,
• tooltip short description displayed when the user holds the cursor over the artefact in the interface,
• config depending on the type a configuration might be expected, to define units or indicate the row and column title of a table, for instance.

Artefacts are then grouped into Artefact Groups.

Artefact Group

Artefact Groups are also specified in the configuration file.

{
  "name": "REQUIREMENTS",
  "title": "Problem definition",
  "metrics": [
    { "name": "ODD_REQ", "scopeId": "component" },
    { "name": "FUNC_REQ" }
  ],
  "config": {},
  "validation": {
    "scoping": "USER_VALIDATION", "development": "NO_VALIDATION", "operation": "NO_VALIDATION"
  }
},

• name designates the technical name of the group. It works as an id and is used to manipulate the group.
• title is the name to be displayed in the interface
• metrics is an array containing the artefacts that fall into this group. An artefact can only be in one group.
  • scopeId is used to link artefacts together, for instance another artefact will be able to refer to this one and to display in its own artefact. View linked metric
• config specific configuration for this metric group
• validation defines the type of validation expected for each stage, if any.

Then, for each Artefact Group, the visibility must be defined.

Group Visibility

Groups visibility refers to the visibility of Artefact Groups depending on both the stage and the abstraction level : every artefact group is not relevant at every level of every stage.

Groups visibility are defined in the configuration file.

Configuration File

The configuration file designates the very file that contains the configuration. In practice, it is broken down into a main configuration file and as many associated files as there are artefacts expected in the life cycle of this project (or more generically, in this blueprint)

The following configuration file example shows the structure of a configuration file :

{
  "name": "",
  "version": "",
  "projectLeader": ""
  "view" :{
    "scoping":{
      "company":{},
      "project":{},
      "indepth":{}
    },
    "development":{
      "company":{},
      "project":{},
      "indepth":{}
    },
    "operation":{
      "company":{},
      "project":{},
      "indepth":{}
    }
  }
  "groups": [],
  "metrics": [],
  "actions": [],
  "validation":{}
},

• name : the name of the configuration file to which each PCIV will refer,
• version : version of the configuration,
• projectLeader: Project leader account or email, to provide project administration rights,
• view : contains the group visibility configuration where the visibiliy of each artefact group can be indicated for each abstraction level of each stage as stated above,
• groups : contains the group configuration,
• metrics: contains the artefacts definition.While this array can directly receive the artefacts as array elements as described here generally it is left empty and the content is divided into as many files as there are artefacts as shown into this example,
• actions: This array is not being used. However, it is planned to use it to define specific actions that can then be assigned to certain artefacts. In the current version (MVP 1.1) all actions are hardcoded.
• validation: No defined or planned use for the moment

Scope

As of current version (MVP V1.1) usage of "Scope" is not clearly defined, yet the general concepts are defined as follows :

• each artefact is associated with at least one scope.
• scope can be seen as "labels" which can be used to identify, across groups, sets of groups dealing with identical themes.
• it is conceivable to use this property to define validation processes (yet, if it is possible to have multiple scopes on a single artefact, there might be conflicts)
• A set of eight scopes are natively defined in Klarity yet they might be extended using extension
  • Project : an artefact related to a specificity of the project
  • Requirement : a component, user, or system requirement that shall be validate and verified
  • Dataset : a metric that enables the analysis / validate content of data used within the project
  • Model : an artefact associated to AI model used during the building of the AI Component
  • VerificationCase : element that require verification and or a validation from users
  • ModelnDataset : an artefact associated to a couple Model + Dataset
  • Sample : an artefact associated to a single sample
  • ModelnSample : an artefact associated to a couple sample + model

Type

An artefact_type designates the display of an information (metric or anything else) in Klarity Dashboard as well as the general behavior depending on the abstraction level.

An artefact_type relies on one to multiple graphical components to handle its display and behaviour on Klarity Dashboard.

An artefact_type can be instanciated in different ways depending on the usage we want to do with it, for instance, a list of elements being associated to a picture viewer will be an artefact_list instanciated as a sample_list, instead of a new and specific artefact type.

Initially, the term ‘metric’ was used instead of ‘artefact’. As a result, this first usage may still be found in various places in the documentation and also in the code (in the configuration file, for example, a ‘metrics’ table expects the list of artefacts).

Defining the type of artefact allows you to specify the tools to be used to manipulate and display the data on Klarity Dashboard.
For instance the "requirements_list" type is used to display data in the form of a list and is more appropriate for listing requirements.
The "html_content" type allow the display of native html content as well as CSS and javascript code.

Instances

• artefact_list
  • sample_list
  • requirements_list
  • requirements_coverage
  • requirements_validation
• confusion_matrix
• image_overlay
• value_ratio
  • list_value_ratio
  • single_value_ratio
• html_content
  • Return on Investment
• ROC_curve

Graphical Component

A graphical component designates a code element that can be used to display an artefact.

Below are the core graphical components being used to display the artefact type integrated into the native version of Klarity Dashboard.

• artefact_list_graphicalComponent
• image_manipulation_graphicalComponent
• matrix_graphicalComponent
• curve_graphicalComponent
• HTML_graphicalComponent
• value_ratio_graphicalComponent
• icon_graphicalComponent

Extension

Klarity Dashboard is built around a modularity strategy that allows the use of extensions and plugins to the core elements of Klarity. Actually, all artefacts types are extensions to the core of Klarity which means it is possible to create and add extensions for a new type of artefact.

Payload

All of the previous elements are related to the configuration of all the artefacts accompanying an AI component throughout its lifecycle. In an optimal case, such a configuration can be used for different AI components dealing with the same type of problem: image classification, anomaly detection in time series, demand forecasting, etc.

In concrete terms, this configuration indicates the nature and form of the artefacts to be produced: requirements in the specification phase, performance in the development phase, bias and drift in the production phase, etc.

This means that this content is produced separately: on the one hand, the configuration which specifies the artefacts to be produced according to the stages and levels of abstraction, and on the other, the content of these artefacts. This content is arranged in what we call payloads.

Structure

First of all, a payload.json file is used for each PCIV. Its layout is as it follows :

{
  "kdConfiguration": "", 
  "projectName": "",
  "prevVersionTags": [""], 
  "versionTag": "",
  "stage": "",
  "metrics": [],
}, 

• kdConfiguration : refers to the "name" of the configuration, must obviously be identical.

• projectName : the name of the project this PCIV belongs to.

• prevVersionTags : an array with the versions tags of the previous PCIV as string : "0.3.2"

• versionTag : version of the PCIV as a string. "0.3.3"

• stage : stage to which the PCIV belongs : scoping, development or operation.

• metrics : an array containing all the artefacts values :
  

  {
    "metricName" : "",
    "scopeId" : "",
    "linkedMetrics" : [
      {
        "type": "",
        "metricName": "",
        "scopeId": "",
      }
    ]
    "values": ""
  },
  

where :

• metricName: is the name of the metric as indicated in the configuration file.
• scopeId : Usage is not defined yet. Can be used to discriminate from multiple data payload for a configured metrics in group.
  • TODO : example
• linkedMetrics : Usage is not defined yet. still it will be used to use the data from another artefact directly in this artefact : for reference, comparison, validation, etc.
  • TODO : example
• values : format will be different depending on the artefact type and instance. Check the artefact type instances section and pages directly for more details.

Artefact Display Specificities

The content of the field "value" is different depending on the artefact type mainly because the rendering, the display of theses values, will be different according to the artefact type instance.

As previously stated an artefact type may have multiple instance type. A single instance can have up to three display style, one for each abstraction level :

• a Company level display
• a Project level display
• an InDepth level display

While it might not be defined for each theses display (mainly because it is not supposed to be displayed, for instance, in company view) a place holder shall be displayed instead.