iTranslated by AI

The content below is an AI-generated translation. This is an experimental feature, and may contain errors. View original article
🗃️

[NGSI-LD and @Context] Building a Smart City with FIWARE (Knowledge Enhancement Part 2)

に公開
Docker
Linked data
context
FIWARE
ngsi
tech

Overview

This is the third installment in the "Learning FIWARE" article series.
In the previous article, we reviewed the specifications of NGSIv2 and NGSI-LD, which are the format standards for data flowing in FIWARE, through a sample case.

This article aims to provide a thorough understanding of the next-generation NGSI-LD.

For information on the previous generation NGSIv2 and related topics, please refer to the following article:
https://zenn.dev/akitek/articles/570a8bd4c788d9

What is NGSI-LD?

As a review of the previous article, let's look at an overview of NGSI-LD in comparison with NGSIv2.

From the official tutorial

Suppose store context information is defined and stored in a database as shown above.[1]

The characteristics of NGSI-LD are as follows:

  1. All things existing in the world are represented as an Entity, and each Entity has attributes called Property and their values (Value).
  2. Additionally, a Relationship may be defined between multiple Entities, and each Relationship has a Property and its Value, just like an Entity.
  3. Furthermore, a Relationship can have another Relationship. This allows Relationships to be linked together...

These characteristics represent data in a way that did not exist in its predecessor, NGSIv2. As the name Linked Data suggests, NGSI-LD provides a richer representation of connections between Entities.

Let's look at the data representation in the figure above. As we do, be mindful of the following hierarchy that can be seen within the JSON file.

Field Can hold
Entity -> Property
Entity -> Relationship
Property -> Value
Relationship -> Property
Relationship -> Relationship 
{
    "@context": "https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context-v1.3.jsonld",
    "id": "urn:ngsi-ld:Shelf:unit001",
    "type": "https://fiware.github.io/tutorials.Step-by-Step/schema/Shelf",
    "https://fiware.github.io/tutorials.Step-by-Step/schema/locatedIn": {
        "type": "Relationship",
        "object": "urn:ngsi-ld:Building:store001",
        "https://fiware.github.io/tutorials.Step-by-Step/schema/installedBy": {
            "type": "Relationship",
            "object": "urn:ngsi-ld:Person:employee001"
        },
        "https://fiware.github.io/tutorials.Step-by-Step/schema/requestedBy": {
            "type": "Relationship",
            "object": "urn:ngsi-ld:Person:bob-the-manager"
        },
        "https://fiware.github.io/tutorials.Step-by-Step/schema/statusOfWork": {
            "type": "Property",
            "value": "https://fiware.github.io/tutorials.Step-by-Step/schema/completed"
        }
    },
    "https://fiware.github.io/tutorials.Step-by-Step/schema/maxCapacity": {
        "type": "Property",
        "value": 50
    },
    "https://fiware.github.io/tutorials.Step-by-Step/schema/numberOfItems": {
        "type": "Property",
        "value": 50
    },
    "https://fiware.github.io/tutorials.Step-by-Step/schema/stocks": {
        "type": "Relationship",
        "object": "urn:ngsi-ld:Product:001"
    },
    "name": {
        "type": "Property",
        "value": "Corner Unit"
    },
    "location": {
        "type": "GeoProperty",
        "value": {
            "type": "Point",
            "coordinates": [
                13.398611,
                52.554699
            ]
        }
    }
}

Now, there are several points of interest in the sample above.

  1. The presence of @context
  2. The way the type part is written
  3. Parts of Property and Relationship are in URL notation
    3-1. Why are the notation methods different?

These unusual notations are good examples of the characteristics of NGSI-LD... but it's hard to tell what's what! 😅

So, let's take a look at the notation methods in NGSI-LD.

The World of @Context

In this article, we will continue our explanation using the official tutorial.
https://ngsi-ld-tutorials.letsfiware.jp/Understanding-At-Context/

First, the tutorial describes the problems that NGSI-LD aims to solve as follows:

① To create an interoperable system of computer-readable links, we must use a well-defined data format (JSON-LD) and assign unique IDs (URLs or URNs) to both data entities and relationships between entities, so that meaning can be programmatically retrieved from the data itself.
② Furthermore, the use and creation of these unique IDs should be handed over as much as possible so as not to interfere with the processing of the data objects themselves.

Point ① and Point ②... they were a bit hard to grasp, so let's break them down.

JSON-LD

First, how should connections between different media be represented in data? (Like the Relationship between Entities.)

We can look at web services (websites) as a reference.

As you know, a website is a visualization of document-based data composed of HTML and JavaScript. HTML elements have a structure that is very easy for humans to read. On the other hand, data sent through communication (e.g., HTTPS) is in JSON format. Because of its structured and simple format, the JSON format is easy for machines to analyze (parse). Of course, it is also easy for humans to read.

In other words,

  • As a data format for representing data and connections, JSON is human and machine-readable.

Next, let's consider the representation of connections with other websites.
Just like hyperlinks on a website, the URL (location) of other websites must be public and accessible to everyone. If such link information to other websites is embedded in your own site, you will be able to navigate to those other sites.

Let's try writing the link information in JSON like this (key-value information is very easy to read):

{
    "type": "Person",
    "name": "Bob",
    "homepage": "https://xxx.com"
}

Now, here's where the problem arises: problems occur when managing (receiving) transition information to multiple websites.

In other words, in key-value notation using keywords like "type", "name", or "homepage", ambiguity is inevitable.

Let's look at the following example.

// Assume the following json data was received from other sites.
// From Site A
{
    "name": "Bob",
    "homepage": "https://xxx.com"
}
// From Site B
{
    "name": "bboi4", // Login name?
    "homepage": "https://xxx.com" // Wait, it's the same URL as Site A??
}

In this case, the "name" part is different, but the "homepage" part is the same. With this data alone, your own site cannot infer "which link information has what meaning"...

One way to avoid such ambiguity is to write the key clearly. That is, to express the Property more specifically.

{
    "http://ex.com/name": "Bob",
    "http://y.com/homepage": "http://z.com/"
}

If it's written like this, it becomes clear that even if the "name" is the same, its origin is different.

However, as a result, it has become very difficult for humans to read... (Verbose). From a human perspective, something like "name" would be much easier to read.

Therefore, JSON-LD's Context allows for the use of shortened versions of URLs by defining and referencing a @context.

{
    "@context": "http://v.com/myapp.jsonld",
    "name": "Bob",
    "homepage": "X"
}

JSON-LD is a data format that introduced a mechanism for parsing data while considering the context, maintaining data uniqueness while remaining human and machine-readable.

{
    "@context": "http://v.com/myapp.jsonld",
    "@id": "http://y.com/Bob",
    "name": "Bob",
    "homepage": "http://bob.com"
}

For more on JSON-LD, check out the video below. The official website is also important for deepening your understanding.
https://www.youtube.com/watch?v=vioCbTo3C-4
https://json-ld.org/

For RDFa, which appears in the middle of the video, refer to articles like these:
https://qiita.com/maoringo/items/d4bff104815efd98ae29
https://www.w3.org/TR/rdfa-syntax/#s_Syntax_overview

How to Create @context Files

NGSI-LD is an extension of the JSON-LD format, improved in terms of readability and ambiguity for JSON files that include connection expressions.

The core is the @context part, and it's important to appropriately design the model definitions and URLs referenced by the @context.

Now, let's look at how to create a @context file. The general flow is as follows:

  1. Design the data structure within the database.
  2. Select data models, including Entities within the data structure from step 1.
    2-1. Consider whether existing public Data Models can be reused.
    2-2. If reusable: eliminate redundant parts and add necessary components.
    2-3. If not reusable: define a new data model.
    2-4. For both (2-2) and (2-3), define types in .yaml.
  3. Create the @context file from the .yaml file created in (2-4).

Smart Agriculture Example

From the official tutorial

We want to use the data structure shown above. As we can see, the required Entities are Building, Device (TemperatureSensor, FillingLevelSensor), and Person.

We want to create files describing these data definitions... but there's no need to define them from scratch!! Several models are already public and can be reused.

Entity Attribute Property or Relationship type Existing Model available?
Building - - - dataModel.Building -
- id Property URN - -
- name Property text dataModel.Building yes
- address Property address schema.org/address yes
- location Property GeoProperty geojson.org yes
- temperature Property number schema.org/Number no
- fillingLevel Property number schema.org/Number no
- owner Relationship URL - no
Person - - - schema.org/Person -
- id Property URN - -
- name Property Text schema.org/Text yes
Temperature Sensor - - - dataModel.Device -
- id Property URN - -
- temperature Property number schema.org/Number -
FillingLevelSensor - - - dataModel.Device -
- id Property URN - -
- fillingLevel Property number schema.org/Number -

Building can be based on dataModel.Building, and Person can be based on schema.org/Person. As for FillingLevelSensor and Temperature Sensor, while there are no directly related models, we can create new models by referencing dataModel.Device.

Baseline Data Model

Let's start creating the @context file. To create a @context file, you need a data model file. However, creating the final version of the data model file right away can be difficult, so we'll first create a baseline data model. This is the original data model without any redundancy elimination or extensions. For the XXXSensor types, since the original is Device, we'll use Device as the baseline.

baseline.yaml
components:
    schemas:
        # This is the basic definition of a building
        Building:
            $ref: "https://fiware.github.io/tutorials.NGSI-LD/models/building.yaml#/Building"
        # These are all the building categories defined within the
        # Smart City and Smart Agri-Food domains
        BuildingCategory:
            $ref: "https://fiware.github.io/tutorials.NGSI-LD/models/building.yaml#/Categories"

        # This is the basic definition of an IoT device
        Device:
            $ref: "https://fiware.github.io/tutorials.NGSI-LD/models/device.yaml#/Device"
        # This is a complete list of IoT device categories
        DeviceCategory:
            $ref: "https://fiware.github.io/tutorials.NGSI-LD/models/saref-terms.yaml#/Categories"
        # This is a complete list of context attributes measurable by devices
        ControlledProperties:
            $ref: "https://fiware.github.io/tutorials.NGSI-LD/models/saref-terms.yaml#/ControlledProperties"

        # This is the NGSI-LD definition of a person
        # Since schema.org Person is JSON-LD, 
        # additional type and id attributes are required
        Person:
            allOf:
                - $ref: "https://fiware.github.io/tutorials.NGSI-LD/models/ngsi-ld.yaml#/Common"
                - $ref: "https://fiware.github.io/tutorials.NGSI-LD/models/schema.org.yaml#/Person"

Application Data Model

baseline.yaml is merely a basic data model. In practice, there are instances where you need to deal with redundant data definitions or require extensions, so we will create agriculture.yaml to reflect those.

agriculture.yaml
components:
    schemas:
        # This is the basic definition of a building
        Building:
            allOf:
                - $ref: "https://fiware.github.io/tutorials.NGSI-LD/models/building.yaml#/Building"
        
            properties:
                temperature:
                    $ref: https://fiware.github.io/tutorials.NGSI-LD/models/saref-terms.yaml#/temperature
                fillingLevel:
                    $ref: https://fiware.github.io/tutorials.NGSI-LD/models/saref-terms.yaml#/fillingLevel
        # These are all the building categories defined within the
        # Smart City and Smart Agri-Food domains
        BuildingCategory:
            $ref: "https://fiware.github.io/tutorials.NGSI-LD/models/building.yaml#/Categories"
        # Added
        TemperatureSensor:
            type: object
            required:
                - temperature
            allOf:
                - $ref: https://fiware.github.io/tutorials.NGSI-LD/models/device.yaml#/Device
            properties:
                temperature:
                    $ref: https://fiware.github.io/tutorials.NGSI-LD/models/saref-terms.yaml#/temperature
        # Added
        FillingLevelSensor:
            type: object
            required:
                - FillingLevelSensor
            allOf:
                - $ref: https://fiware.github.io/tutorials.NGSI-LD/models/device.yaml#/Device
            properties:
                fillingLevel:
                    $ref: https://fiware.github.io/tutorials.NGSI-LD/models/saref-terms.yaml#/fillingLevel
        # (Unnecessary) This is the basic definition of an IoT device
        # Device:
        #    $ref: "https://fiware.github.io/tutorials.NGSI-LD/models/device.yaml#/Device"
        
        # This is a complete list of IoT device categories
        DeviceCategory:
            $ref: "https://fiware.github.io/tutorials.NGSI-LD/models/saref-terms.yaml#/Categories"
        # This is a complete list of context attributes measurable by devices
        ControlledProperties:
            $ref: "https://fiware.github.io/tutorials.NGSI-LD/models/saref-terms.yaml#/ControlledProperties"

        # This is the NGSI-LD definition of a person
        # Since schema.org Person is JSON-LD, 
        # additional type and id attributes are required
        Person:
            allOf:
                - $ref: "https://fiware.github.io/tutorials.NGSI-LD/models/ngsi-ld.yaml#/Common"
                - $ref: "https://fiware.github.io/tutorials.NGSI-LD/models/schema.org.yaml#/Person"

The file above allows you to create a data definition file for NGSI-LD. Note that the notation refers to Swagger specifications.
https://swagger.io/docs/specification/v3_0/data-models/data-types/

Let's take a look specifically at the TemperatureSensor part.

TemperatureSensor:
    type: object
    required:
        - temperature
    allOf:
        - $ref: https://fiware.github.io/tutorials.NGSI-LD/models/device.yaml#/Device
    properties:
        temperature:
            $ref: https://fiware.github.io/tutorials.NGSI-LD/models/saref-terms.yaml#/temperature
  • type: The data type, which can be Number, String, Array, etc. Here it's an object type (assuming it will contain properties of various types).
  • required: A constraint stating that the specified Property must be provided during registration.
  • allOf: The definition of TemperatureSensor falls entirely under allOf (meaning it is limited to what is in $ref).
  • $ref: The referenced data model definition file (reused if defined elsewhere).
  • properties: Attribute names defined within TemperatureSensor.

saref-related definitions can be found on the following Ontology definition site.
https://ontology.tno.nl/saref/

Automatically Creating NGSI-LD @context Files

Once the data definition file is created, it's finally time to create the @context file.
It's very simple; the NGSI-LD @context file can be generated from the Swagger data model as follows:

$ ./services ngsi [file]
# Creating a NGSI-LD @context file for normalized interactions datamodels.context-ngsi.jsonld created

Looking at the generated file, it looks like this:

{
    "@context": {
        "type": "@type",
        "id": "@id",
        "ngsi-ld": "https://uri.etsi.org/ngsi-ld/",
        "fiware": "https://uri.fiware.org/ns/data-models#",
        "schema": "https://schema.org/",
 ...etc
        "Building": "fiware:Building",
        "FillingLevelSensor": "fiware:FillingLevelSensor",
        "Person": "fiware:Person",
        "TemperatureSensor": "fiware:TemperatureSensor",
 ... etc
        "actuator": "https://w3id.org/saref#actuator",
        "additionalName": "schema:additionalName",
        "address": "schema:address",
        "airPollution": "https://w3id.org/saref#airPollution",
        "atmosphericPressure": "https://w3id.org/saref#atmosphericPressure",
        "barn": "https://wiki.openstreetmap.org/wiki/Tag:building%3Dbarn",
        "batteryLevel": "fiware:batteryLevel",
        "category": "fiware:category",
        "configuration": "fiware:configuration",
        "conservatory": "https://wiki.openstreetmap.org/wiki/Tag:building%3Dconservatory",
        "containedInPlace": "fiware:containedInPlace",
        "controlledAsset": "fiware:controlledAsset",
        "controlledProperty": "fiware:controlledProperty",
        "cowshed": "https://wiki.openstreetmap.org/wiki/Tag:building%3Dcowshed",
...etc
    }
}

In this, the following three elements can be found:

  1. A list of standard terms and abbreviations. This eliminates the need to repeat URIs, reducing the overall file size. (e.g., the prefix "fiware" is defined as "https://uri.fiware.org/ns/data-models#", so "Building" = "fiware:Building" (="https://uri.fiware.org/ns/data-models#Building") can be shortened as such.
  2. A defined set of entity types (such as Building).
  3. A list of attributes (e.g., address) and enumerated values (e.g., barn).

Experiencing NGSI-LD with @context

Now, the created @context file is meaningless unless it is referenced by the database.
Therefore, we will place the @context file and prepare a mechanism so that other services can reference the NGSI-LD data definitions.

To promote interoperability of data exchange, the NGSI-LD Context Broker explicitly exposes the JSON-LD @context file to define the data held within context entities.

Let's prepare for the demo by seeing where to place the created @context file and how to reference it for data exchange.

Architecture

From the official tutorial

Comparing it with the previous article, you might notice a new component (JSON-LD @context). This is exactly where the @context file is stored.

Furthermore, this location needs to be accessible by anyone.
A simple way is to provide the static @context file, which defines the context entities within the system, via an HTTP Web-Server.

Now, let's define the required containers.

docker-compose.yaml
services:
    orion:
        image: fiware/orion-ld
        hostname: orion
        container_name: fiware-orion
        depends_on:
            - mongo-db
        networks:
            - default
        ports:
            - "1026:1026"
        command: -dbhost mongo-db -logLevel DEBUG
        healthcheck:
            test: curl --fail -s http://orion:1026/version || exit 1
    mongo-db:
        image: mongo:4.2
        hostname: mongo-db
        container_name: db-mongo
        expose:
            - "27017"
        ports:
            - "27017:27017"
        networks:
            - default
    ld-context: # HTTP server to host @context files
        image: httpd:alpine
        hostname: context # Accessible via http://context/<filename>
        container_name: fiware-ld-context
        ports:
            - "3004:80" # HTTP port

You can see that we are trying to start a container for an HTTP server to host @context files.

Placing the @context File

...And now, we want to place the created @context on this HTTP server, but it's not there yet.

This is because the @context file is delivered at the same time as the data is POSTed to the Context Broker.

Let's look at a sample:

curl -iX POST 'http://localhost:1026/ngsi-ld/v1/entities/' \
-H 'Content-Type: application/ld+json' \
--data-raw '{
    "id": "urn:ngsi-ld:Building:farm001",
    "type": "Building",
    "category": {
        "type": "Property",
        "value": ["farm"]
    },
    "address": {
        "type": "Property",
        "value": {
            "streetAddress": "Großer Stern 1",
            "addressRegion": "Berlin",
            "addressLocality": "Tiergarten",
            "postalCode": "10557"
        },
        "verified": {
            "type": "Property",
            "value": true
        }
    },
    "location": {
        "type": "GeoProperty",
        "value": {
             "type": "Point",
             "coordinates": [13.3505, 52.5144]
        }
    },
    "name": {
        "type": "Property",
        "value": "Victory Farm"
    },
    "@context": "http://context/ngsi-context.jsonld" # Focus here
}'

Just like with NGSIv2, you can send data to http://localhost:1026/ngsi-ld/v1/entities/ using the POST method.

Please pay attention to the end. It says @context: "http://context/ngsi-context.jsonld". This provides an instruction that "the @context file will be placed (is placed) under http://context/."

Through this initial POST request, the @context file (ngsi-context.jsonld) is placed on the HTTP server, making it accessible to anyone from then on.

Then, in subsequent requests (GET or POST), you can send and receive data using the shortened JSON definition by specifying the URL of this @context file.

curl -iX POST 'http://localhost:1026/ngsi-ld/v1/entities/' \
-H 'Content-Type: application/json' \
-H 'Link: <http://context/ngsi-context.jsonld>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' \ 
--d '{
    "id": "urn:ngsi-ld:Building:barn002",
    "type": "Building",
    "category": {
        "type": "Property",
        "value": ["barn"]
    },
    "address": {
        "type": "Property",
        "value": {
            "streetAddress": "Straße des 17. Juni",
            "addressRegion": "Berlin",
            "addressLocality": "Tiergarten",
            "postalCode": "10557"
        },
        "verified": {
            "type": "Property",
            "value": true
        }
    },
     "location": {
        "type": "GeoProperty",
        "value": {
             "type": "Point",
              "coordinates": [13.3698, 52.5163]
        }
    },
    "name": {
        "type": "Property",
        "value": "Big Red Barn"
    }
}'

Take a look at the header part: Link: <http://context/ngsi-context.jsonld>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json". The Link header explicitly specifies the reference destination for the @context. This allows Key-value pairs that would normally remain ambiguous, such as "address" or "location," to be deciphered by the machine or server.

CRUD Operations with @Context

Finally, let's look at REST operations through NGSI-LD.
Generally, the following request methods are used depending on the data processing you want to perform:

  • GET: Read data
  • POST: Create new entities and attributes
  • PATCH: Modify entities and attributes
  • DELETE: Delete entities and attributes

Additionally, you can specify the response from the Broker.
In the header part:

  • Accept: application/json: Response is in JSON format
  • Accept: application/ld+json: Response is in JSON-LD format
  • Accept: application/geo+json: Response is in GeoJSON or GeoJSON-LD format

Omitting the last one, here are the main differences between JSON and JSON-LD responses:

  • When the JSON-LD format is selected, @context is included as an additional attribute within the response body.
  • When the JSON-only format is used, @context is passed as an additional Link header element and is not represented in the response body.

...Since the response display changes depending on what is written in the header, we will continue to use the following headers so it's clear to everyone:

-H 'Content-Type: application/json' \
-H 'Link: <http://context/ngsi-context.jsonld>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' \
-H 'Accept: application/ld+json' \
Description Method URL Comment
Create a new entity POST /ngsi-ld/v1/entities/ -
Create a new attribute in an existing entity POST /ngsi-ld/v1/entities/<ID>/attrs Only the attribute part (ID and Type not included)
Create multiple entities or attributes at once POST ngsi-ld/v1/entityOperations/create Extension of single creation -> into an array
Overwrite multiple entities or attributes at once POST ngsi-ld/v1/entityOperations/upsert If the entity already exists, the request updates its attributes. If it doesn't exist, a new entity is created.
Read a specific entity GET /ngsi-ld/v1/entities/<ID> -d 'options=keyValues' for simple Key-value display
--> Read by specifying attributes GET + -d 'attrs=<attribute_name>' Comma (,) separated for multiple
--> List specific entities GET + -d 'type=<entity_name>' -
Overwrite an attribute value PATCH /ngsi-ld/v1/entities/<entity-id>/attrs/<attribute> Different from adding an attribute. Only changes the value.
Overwrite multiple attributes PATCH /ngsi-ld/v1/entities/<entity-id>/attrs Just avoid specifying a specific attribute name
Batch update attributes of multiple data entities POST /ngsi-ld/v1/entityOperations/upsert?options=update For updating multiple entities simultaneously
Batch replacement of entity data POST /ngsi-ld/v1/entityOperations/update?options=replace -
Delete an entity DELETE /ngsi-ld/v1/entities/<ID> Deletion of a single entity
Delete an attribute DELETE /ngsi-ld/v1/entities/<ID>/attrs -
Batch deletion of multiple entities POST ngsi-ld/v1/entityOperations/delete -
Batch deletion of multiple attributes from an entity PATCH /ngsi-ld/v1/entities/<entity-id>/attrs -

Summary

In this article, I explained NGSI-LD, the next-generation API and data standard used on FIWARE.
NGSI-LD is an extension of JSON-LD, a data definition that maintains the human and machine readability of JSON files while eliminating ambiguity.
The key is that by creating and referencing @context files, you can exchange data in a manner similar to NGSIv2 based on traditional JSON files.

Furthermore, in FIWARE operations using NGSI-LD, it is necessary to create data definition files in advance and publicly expose the created @context files.

With this article, the discussion on NGSI, the "common language" on FIWARE, concludes.
In the subsequent series, I plan to provide overviews of Open API groups other than Orion and MongoDB and create some small IoT applications.

Finally, it's time for the practical part! 😒

脚注

  1. In FIWARE, "context" is the general term for "information about things." ↩︎

Discussion