Data Logger is a micro service that allows work conducted at a site to be streamed at high frequency and stored in the cloud; regardless of equipment brand or company. Data Logger accepts data over channels. Channels allow data to be multiplexed and distinctly identified and are a function of site and data center or region. This page describes how to interact with the Data Logger.
The DataLogger micro service and client library is used to record high frequency telemetry for a machine on a work site. The DataLogger is flexible in that it allows recording of any binary or text data so long as the following criteria are met.
The DataLogger collects and indexes data payloads spatially and temporally for all machines on a Sitelink3D site. Machine Forward Kinematic (MFK) data, which describes the accurate position and shape of the machine in the local site coordinates is the predominant payload recorded by the Data-Logger service. Regular machine state information such as activity, task and design surface is also recorded.
Aside from just receiving data, DataLogger is capable of forwarding information about the broader context of activity on a site. As such, it is aware of other micro services and communicates with them. An example of this is the Site Owner service which Data Logger notifies under certain circumstances.
Software running on machines will post data to the Data Logger service. This data will vary in terms of frequency and purpose. Positional data is sent at 10Hz whereas periodic data such as the site payload which will be subsequently discussed is sent every fifteen minutes from every machine on site.
For all payloads which are sent periodically sent, the client must ensure that the generated "data" string transmitted is identical.
MFK concepts are discussed in more detail on the Machine Forward Kinematics page. Refer to the code examples page to see the Data Logger payloads discussed here being used in context.
This section provides information required to understand all Data Logger payloads sent over all channels.
The foundation of a Data Logger payload is JSON as represented below.
{
"at": 1448430027995,
"type": "",
"ttl": 900,
"data": {
}
}
All Data Logger payloads have the following fields in common.
Each data payload is submitted to a Data Logger channel path. The data will either be uploaded to the server if there is an internet connection or exported to a log file in the off-line solution. Either way, each data payload is submitted to a channel path. The channel path denotes all the information the server needs to store the data payload in the server database.
The full URL of a DataLogger stream (over MultiStream), which includes the channel path, is as follows.
server/data_logger/v1/publish_live/region/stream/identifier.cellsize
|--------- channel_path ---------|
The definitions for the URL terms are as follows.
api.code.topcon.com/data_logger/v1/publish_live/us/update/67a60dd90642e43a26c4ea254cfd7100734a97b5.medium
This example would be for sending payloads to the the "update" stream at the "us" data centre for site "67a60dd90642e43a26c4ea254cfd7100734a97b5".
The site payload describes a site. This data is created once for a site but is sent to the servers every 15 minutes from every machine working on the site.
The data in the site payload is data that exists in the tp3 file. This data may be imported into an existing tp3 file, exported from a tp3 file, or created (off-line) in a tp3 file.
Site payloads are transmitted periodically to ensure delivery. Data may be lost if a disk gets full for example, or if client software is restarted or data logs cleared. Every machine sends the site payload to ensure that any machine can connect and transmit data or upload a log file to the server first.
The site payload is in fact only needed by the server for sites that were created off-line. It is however sent regardless to ensure consistent behavior between on-line and off-line modes.
{
"at": 1448430027995,
"type": "sitelink::Site",
"ttl": 900,
"data": {
"document_b64": "eyJpZGVudGlmaWVyIjoiNjdhNjBk...",
"algorithm": "SHA256-RSA",
"signature_b64": "cLZdWZTXA4PqZWCYhuXSkJgkAFKt..."
}
}
Note that the base64 data above has been truncated to aid readability.
Site data consists of a base64 encoded document (document_b64). This document is digitally signed by a public private key pair known only to the owner of the site.
This signing verifies the identity of the customer creating the site. Note that the data includes the digital signature (signature_b64) and the algorithm used for signing, in this case SHA256-RSA.
If the site data is created on-line, then it is the Sitelink3D server that performs the digital signing using certificates stored in the Sitelink3D server against the owner. If the site data is created off-line, then the site data is signed by certificates stored on the PC creating the site data. In this latter case, the owner must have uploaded the public key to the Sitelink3D server, and link the public key to their customer account. This operation is to be done in a user freindly way, where the user does not know anything about public or private keys.
{
"identifier": "67a60dd90642e43a26c4ea254cfd7100734a97b5",
"region": "us",
"name": "Honor Pitt Road Upgrade.",
"owner_email": "owner@example.com",
"finger_print": "E8:1E:39:0F:5C:90:8F:EE:66:42:7E:B3:EB:C8:90:15:55:23:19:53",
"finger_print_algorithm": "SHA1",
"burden_of_proof_b64": "R4L6ZFjijRGHsdLp\/ORgKI9b...",
"identifier_algorithm": "SHA1"
}
Note that the base64 data above has been truncated to aid readability.
The site data fields are defined as follows.
The Sitelink3D server uses this data to ensure that the site belongs to the stated owner in a secure way. Once an owner creates a site using a public / private key pair, they can safely give the site document to any third party such as a contractor allowing them to work on the site. The contractor can further give the site data to a sub contractor and so on. The contractors and sub contractors will be publishing data to the site identifier. Regardless of who has access to the site data specified here and who has public knowledge of the site identifier, it is only the owner who can query and take ownership of the data in the Sitelink3D server.
In order to take ownership of the site, the owner must have created the site configuration on-line or alternatively created it off-line with a public private key pair and uploaded the public key to the Sitelink server using the Site Owner micro service.
The asset payload describes an asset such as a client device or a machine. A client device could represent a GX-60 or any other HID. A machine could be an excavator or other plant equipment that contains the client device.
Like the site payload, the asset payload is sent every 15 minutes to ensure delivery to the Sitelink3D server. The asset payload is used by the server to establish trusted data provenance when reporting on the data.
In reality, the asset payload is simply a public key sent to the Sitelink3D server. This public key is used by the server to validate signed documents and thus establish trusted data provenance. Refer to the AssetContext material for more information.
Using public private key pairs is useful to identify assets as we can sign documents to build trusted data provenance and demonstrate that data was indeed created from a particular machine or client. This is useful when Sitelink3D data is being used for invoicing and quality assurance purposes.
{
"at": 1448430027998,
"type": "sitelink::Asset",
"ttl": 900,
"data": {
"uuid": "69240e36-6844-4b05-bee5-4adc9be872e6",
"urn": "urn:X-topcon:machine:Komatsu-Excavator-PC210-SN",
"public_pem_b64": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0t...",
"finger_print": "E1:F6:3F:8A:71:96:28:3A:2A:92:4A:1A:AB:A3:E9:94:21:94:4E:81",
"finger_print_algorithm": "SHA1"
}
}
Note that the base64 data above has been truncated to aid readability.
The asset data fields are defined as follows.
The AssetContext Payload describes the collection of Assets responsible for the generation of data. For example, if a machine position data is published, that machine position data will reference an AssetContext that references a device Asset and a machine Asset. This is due to the fact that both a device and a machine are related to the generation of a machine position data event.
The AssetContext payload is sent every 15 minutes to ensure delivery to the Sitelink3D server. The payload has a lease period of approximately 1 day. When the lease period expires, a new AssetContext is created with a new document and new signatures. This ensures that any malicious party that obtains an AssetContext can only generate malicious data for a client for the remaining duration of the lease. It is difficult in the first place for a malicious party to get hold of an AssetContext, as they are sent over encrypted (SSL) channels.
{
"at": 1448430028006,
"type": "sitelink::AssetContext",
"ttl": 900,
"data": {
"uuid": "3b4404c6-5548-41e9-867b-aac622eb94a8",
"lease_start": 1448430027952,
"lease_ttl": 28800,
"document_b64": "eyJ1dWlkIjoi...",
"signatures": [{
"asset_uuid": "69240e36-6844-4b05-bee5-4adc9be872e6",
"asset_urn": "urn:X-topcon:device:080027DF908E",
"signature_b64": "QgM7cUIGYj...",
"algorithm": "SHA256-RSA"
}, {
"asset_uuid": "fd5b04a0-8beb-4118-ae28-82bed5e0449c",
"asset_urn": "urn:X-topcon:machine:Komatsu-Excavator-PC210-SN",
"signature_b64": "mhZLBNS5kY2M8NoMbyi...",
"algorithm": "SHA256-RSA"
}]
}
}
Note that the base64 data above has been truncated to aid readability.
The AssetContext data fields are defined as follows.
{
"asset_uuid": "fd5b04a0-8beb-4118-ae28-82bed5e0449c",
"asset_urn": "urn:X-topcon:machine:Komatsu-Excavator-PC210-SN",
"signature_b64": "mhZLBNS5kY2M8NoMbyi...",
"algorithm": "SHA256-RSA"
}
The AssetContext has an array of signatures. Each signature is the result of an Asset signing the document in the AssetContext.
Note that the base64 data above has been truncated to aid readability.
The Signature object fields are defined as follows.
The purpose of the MFK ResourceConfiguration Payload is to transfer information about the a machines shape, size, configuration and calibration to the server. This allows the server to rebuild what the machine looks like for the purposes of displaying a machine remotely or for simulating As-Built surfaces.
The MFK ResourceConfiguration Payload is sent every 15 minutes and whenever the configuration of the machine changes. Such a change may result from a machine re-calibration or a change of implement.
The MFK ResourceConfiguration Payload is sent at a low frequency but is referenced by the MFK ReplicateManifest which is sent at a high frequency. This allows changes in the machine shape to be recorded at a high frequency (10Hz) using small ReplicateManifest Payloads.
The MFK ResourceConfiguration references an AssetContext to indicate which device and machine Asset created the ResourceConfiguration.
The MFK ResourceConfiguration Payload contains the MFK library JSON resource configuration file format. Considering DataLogger Payloads cannot exceed 32Kbyte and given that MFK ResourceConfiguration might exceed this limit, the Payload allows a ResourceConfiguration be split into multiple parts. The server must re-assemble the parts into a single ResourceConfiguration.
{
"at": 1448430028009,
"type": "mfk::ResourceConfiguration",
"ttl": 900,
"ac_uuid": "3b4404c6-5548-41e9-867b-aac622eb94a8",
"data": {
"uuid": "bef56036-6e4b-11e5-b0d2-7b31189eae97",
"offset": 0,
"total_size": 13324,
"data_part_b64": "eyJ2ZXJzaW9uIjoiMC4xIiwidX..."
}
}
Note that the base64 data above has been truncated to aid readability. Note also the extra root level ac_uuid field which is a reference to the AssetContext.
The MFK ResourceConfiguration data fields are defined as follows.
The purpose of the ReplicateManifest is to provide the small amount of information needed to a ResourceConfiguration to fully position a machine in 3-space, as well as to provide changing sensor information related to As-Built generation (such as temperature, or compaction values).
The MFK ReplicateManifest references an AssetContext to indicate the Assets involved in the production of the ReplicateManifest. The MFK ReplicateManifest references an MFK ResourceConfiguration. In reality the uuids are able to be sent over the wire at 10Hz as the wire compression algorithms essentially remove them from the data footprint.
{
"at": 1448431637950,
"type": "mfk::ReplicateManifest",
"ttl": 30,
"ac_uuid": "3b4404c6-5548-41e9-867b-aac622eb94a8",
"data": {
"rc_uuid": "bef56036-6e4b-11e5-b0d2-7b31189eae97",
"manifest": ["AAAAAAAAAA==", "AAAAAAAA"]
}
}
Note the extra root level ac_uuid field which is a reference to the AssetContext.
The MFK ReplicateManifest data fields are defined as follows.