Changelog
Changelog of what has changed in DCH v2.0 from DCH v1.0
Overview
We have made changes to improve performance and user experience. Click on scopes for more detail.
Building and Site Modelling
Brick schema support has been updated to Brick version 1.3.0 (was version 1.2.1).
DCH uses the SHACL Advanced Features (SHAC-AF) 'rules' distributed in Brick 1.3.0 to perform inference (a key step in publication) over semantic models.
DCH uses the Shapes Constraint Language (SHACL) 'shapes' distributed in Brick 1.3.0 to perform validation over semantic models.
Internally, DCH now handles Site and Building models using graph-based representations for the whole lifecycle. Previously, DCH used a document-like internal storage format.
With models in DCH now validated using SHACL shapes, and with the removal of the JSON interchange format, a number of custom constraints about model composition (which were additional to Brick 1.2.1's constraints) have been removed.
Site and Building models are now stored separately from Data Pool models. This means that all information about 'Point' instances are managed automatically in Data Pools. Information about Points is merged into Site/Building models during publication.
Publication (i.e. inference and validation of semantic models) now occurs across a whole site and its building(s) in a single operation, which will succeed en bloc if validation passes for all constituent models.
The BRIQL query API has received various small but important bug fixes. These include performance improvements, removal of duplicate nodes in solutions, and fixes to text filters.
Benefits of these changes
Simplified onboarding pathways for adding points and their associated timeseries to DCH.
DCH-native APIs for managing points, pools and data sources. Users no longer have to manage data sources via Senaps, nor do they have to manually link timeseries to semantic models.
Two methods now available to find points, via simple REST API query parameters, or via BRIQL.
Users can now manage their points without the concern of having a building or site semantic model registered in the system. This allows system onboarding to be done in separate phases.
Timeseries Data
Abstraction of Senaps
In DCH 1.0, users could only interact with timeseries data via direct access to a 3rd party product called Senaps, and as such required knowledge of the Senaps Sensor API. DCH2.0 has abstracted away the timeseries data storage mechanism (currently Senaps) so that users do not need to understand the technical complexity of the relationship between DCH and Senaps. Previously permissions for users to access data would be defined by roles within Senaps. Instead, DCH permissions were used, managed by the DCH Identity Access Management service. In DCH 1.0, users had to use "data stream" identifiers, that were linked to DCH points, to gather timeseries data. In DCH 2.0, users no longer need to consider these lower-level data stream identifiers, and instead can directly use the DCH point identifiers to request data.
Data Format
How data is represented has changed between DCH 1.0 and DCH 2.0. These design changes were made to overcome limitations with the Senaps data format, namely:
The JSON format is not consistent when requesting data for one data stream versus multiple data streams at a time
The Senaps data format is ambiguous when referring to numeric (scalar) and vector data, as they both share the "v" key to refer to the data
When requesting multiple data streams in CSV format from Senaps, the CSV returned was sparse, leading to unnecessarily large tables of data
A limited amount of return types are permitted by the Senaps API, being JSON and CSV.
DCH 1.0 Data Format
Observations within Senaps Sensor API are structured as a list of results, with a timestamp "t" and a value "v". This value can be one of three types, numeric (i.e. scalar, using the key "v"), vector (using the key "v") or an embedded JSON representation (using the key "j").
DCH 1.0: Senaps Download Data Format - Numeric
{
"results" : [
{
"t" : "2015-11-12T00:00:00.000Z",
"v" : {
"v" : 1.0
}
}
]
}
DCH 1.0: Senaps Download Data Format - Vector
{
"results" : [
{
"t" : "2015-11-12T00:00:00.000Z",
"v" : {
"v" : [1.0, 2.0]
}
}
]
}
DCH 1.0: Senaps Download Data Format - Document
{
"results" : [
{
"t" : "2015-11-12T00:00:00.000Z",
"v" : {
"j" : {"a_key": "a_value"}
}
}
]
}
DCH 2.0 Data Format
DCH 2.0 aims to solve some of the issues with the Senaps data format by making the following changes, as well as adding quality of life features:
Adds ability to request multiple points data and receive in JSON or CSV format
Encodes point ID to reduce size of point references within data section of the response
Uses distinct keys for numeric "n", vector "v" and document "d" data
Adds metadata to describe some aggregate statistics about the data returned, including count of data, minimum and maximum timestamps, point map, data schema version and the URL used to request the data.
DCH 2.0: Point Download Data Format (JSON)
{
"metadata": {
"$schema": "0.1.0",
"startTime": "2021-03-02T15:00:00Z",
"endTime": "2021-03-02T16:00:00Z",
"count": 3,
"points": {
"0": "my_org_id:my_datapool_id:my_point_A",
"1": "my_org_id:my_datapool_id:my_point_B",
"2": "my_org_id:my_datapool_id:my_point_C"
},
"self": "https://beta.dataclearinghouse.org/api/chronos/v1/download?start=2021-03-02T15:00:00Z&end=2021-03-02T16:00:00Z&si=true&ei=true"
},
"data": [
{
"t": "2021-03-02T15:00:00Z",
"p": "0",
"n": 1.2,
"a": "my annotation"
},
{
"t": "2021-03-02T15:00:00Z",
"p": "1",
"v": [
1.2,
2
]
},
{
"t": "2021-03-02T15:00:00Z",
"p": "2",
"d": "{\"my_key\": 1.0}"
}
]
}
DCH 2.0: Point Data Format (CSV)
t,n,v,d,a,point
2023-06-12T02:00:00.000Z,1.0,,,,33b199a8-9eaa-4d75-b4dc-c70aa66a7570
2023-06-12T03:00:00.000Z,10.0,,,,33b199a8-9eaa-4d75-b4dc-c70aa66a7570
2023-12-10T02:00:00.000Z,12.0,,,,33b199a8-9eaa-4d75-b4dc-c70aa66a7570
Benefits of these changes
Users no longer need to understand distinction between Senaps data streams and DCH points. Now only need to consider DCH points.
Data for multiple points can be requested at a time
Non-ambiguous data format when referring to numeric (scalar) and vector data
CSV format is a long format, such that the CSV data is not sparse, reducing the size of the file
Point Management
Points as First Class Resources
In DCH 1.0, points were defined in building and site semantic models. They had references to timeseries stored in the Senaps Sensor Data API, and the person responsible for creating the semantic model was responsible for adding the information about the point and its linked data.
In DCH 2.0, points become first class citizens of DCH. The user can interact with them via the DCH dashboard, and via associated APIs. Points can be created manually, but they are also automatically created for you if you create a data source via DCH's data source API.
Points in Context: Data Pools and Classification
Points do not exist in isolation. They are always defined as existing within a context, and in DCH 2.0 that context is the data pool. Data pools are collections of points and associated point metadata. Each data source has an associated pool where it stores points with references to timeseries, and pools have their own semantic models that can be queried using BRIQL.
Finally, DCH 2.0 adds the ability to classify points. Skilled operators like BMS system integrators (SIs) can define classifier sets that can be re-used to give meaning to points stored in DCH. In this way, SIs can ensure the data in DCH is accurately modelled.
Benefits of these changes
Simplified onboarding pathways for adding points and their associated timeseries to DCH.
DCH-native APIs for managing points, pools and data sources. Users no longer have to manage data sources via Senaps, nor do they have to manually link timeseries to semantic models.
Two methods now available to find points, via simple REST API query parameters, or via BRIQL.
Users can now manage their points without the concern of having a building or site semantic model registered in the system. This allows system onboarding to be done in separate phases.
Application Management
Decoupling from Senaps
In DCH 1.0, the application system was implemented with all application data and mechanics stored and managed via Senaps. All users needed a Senaps account because permissions and authentication were handled via Senaps. Application developers were forced to work, test, and validate through Senaps. In DCH 2.0 Senaps’ role has changed to a deployment platform, used to run the code and scheduling jobs. DCH 2.0 handles the storage of the applications and application metadata. Permissions and authentication is managed by DCH and users no longer need a Senaps account.
Dependency tracking
When an application is installed on a site or building, the system establishes a connection between them. This allows the system to use this information when performing actions like changing the site or buildings.
Application vendor support
In DCH 1.0 the applications had no information about who created the application. In 2.0, applications are associated with vendor profiles and let you set an author for the application.
New data ports
In DCH 2.0, the application developer specifies a BRIQL response port in the data flow, which takes the BRIQL response as a JSON value. A developer can also set a model metadata port, which loads the model metadata of the sites/buildings into the specified port.
Application version support with publish and deprecate
DCH 2.0 provides full semantic versioning support for applications and the operators. A developer can only upload newer versions of applications and operators. In DCH 2.0 applications must be published before they can be used by users. This lets application developers test and verify that the application works before letting users install them. If an application is no longer supported or does not work the vendor can deprecate the application, which will ensure that the application version can no longer be installed. Application versions that have been installed before the developer deprecates the version can still be used, to avoid situations where critical functionality from an application is removed without user consent.
Grafana upgrades and improvements
In DCH 2.0 Grafana has been upgraded from 7 to 10. Furthermore, in DCH 2.0 Grafana no longer relies on Senaps authentication and authorisation, but uses the DCH authentication system. Additionally, whereas DCH 1.0 used Senaps directly, in DCH 2.0 the data source plugins have been rewritten to use the DCH services instead of Senaps.
Improved structure of the packaged application
In DCH 1.0 the application was packaged as a single Json file, which inevitably would get exceptionally large and complicated. In DCH 2.0 the application is organised into separate files and a sub folder for bundled operators.
Benefits of new features
Decoupling from Senaps makes the service faster, improves stability and unlocks opportunities for many useful features. Many of the changes target making it easier to use for application developers and allow more information through to the application from the site and building. Other changes improve usability for the end user with better integrations to Grafana and features to explore, search and filter available applications.
Last updated