Smart API Programmer Guide

Java


Chapter 1. Introduction

1.1. What is the Smart API?

The Smart API is a technology for making better APIs for remote system and device management applications such as various IoT (Internet of Things) solutions. Technically, the Smart API is an object centric, semantics enabled, transaction capable and secure method for transferring and storing linked data. To understand a better what that is and why these features are important, let's open that up a bit:

  • Object centric. Smart API has been designed for transferring remote objects between systems. "Objects" in this case means both the physical devices the objects represent and the programming abstractions software engineers actually use to model those devices. Objects are not only a natural way of representing physical "things" but also directly map into popular object oriented programming languages. Smart API transparently handles the details of transforming an object in memory into a datastructure transmitted over the network and back again, making the whole process easy and fast for engineers to use.
  • Semantics enabled. Smart API builds on the principles of semantic web. The purpose of this technology is to make data exchange unambiguous and suitable for computers to deduct things from the data on behalf of users and programmers. The primary obstacle in most data exchange is that while it may work technically just fine, the data is interpreted incorrectly as different people and organizations use different terms and different measurement units for the same thing. So when for one person "length" is the dimension of a box measured in inches, for the other person the same thing is "depth" measured in centimeters. Common vocabularies of the semantics removes this ambiguity and makes it possible to do for instance unit conversion between inches and centimeters automatically as data is received or sent.
  • Transaction capable. In most applications, there eventually needs to be some way to monetize the operations and data. For a long time it has been common to substitute revenue from the actual data with something else, such as advertising revenue, or apply some fixed pricing model such as a monthly fee on data services. But with the advent of Cloud Computing and Big Data, a more granular pricing model is often desired to pay just for the resources consumed. So you would pay for the amount of bytes transferred, the number of commands sent, or the number of measurements stored. Smart API transaction support is exactly for this purpose. It allows storing such details of data transfers in a way that is cryptographically protected for confidentiality and non-repudiation and serves as a ledger for invoicing.
  • Secure. Smart API is end-to-end crypto enabled. While it is highly recommended and fully supported to use a secure links over technologies such as HTTPS, Smart API messages can further be encrypted and signed per each message. So no matter what data communication links are in between, how the connections terminate and how the data may be stored in intermediate locations, the data remains confidential all the way to the recipient. Smart API also has built in support for authentication using OAuth2 so you get the building blocks of security straight out of the box.
  • Linked data enabled. Connected systems are all about - surprise - connections. Those connections link one thing to another, then to another and another and possibly back again. Such links create graphs. Understanding and being capable of processing graphs is the core of many IoT analysis applications. In Smart API, things can be linked between objects, within messages, across messages, and in any other configuration available and necessary.

So why and when would you use Smart API? The short answer is of course when you need features listed above. The slightly longer answer is that Smart API is a highly recommended technology when you build remote control and measurement applications. True, simple get / put REST APIs over some JSON structure are the norm and in many applications sufficient - at least in the beginning. But when the application grows and it actually needs to be integrated with some larger entities, when security becomes an issue, and when you need to make sure that data is actually correct before making automation decisions, Smart API becomes an essential tool. It saves software engineers from a ton of headaches caused by the tedious process of building data converters, it always stores data in an understandable format, it creates documentation for both the API and the data automatically, and it can handle data ambiguity in a professional manner. And as Smart API is free and pretty much as easy to use as alternative formats, there really is few reasons why not build a system properly from the beginning.

But there already are other API standards and tools such as Swagger/OpenAPI, why another? Well, the simple answer is: Smart API takes many of the best practices of OpenAPI such as declarative specifications and automatic generation of documents and code, but adds essential features needed by modern Big Data applications on top. Where the design of Open API originates from the API itself, Smart API's core is in the data. In a nutshell: OpenAPI is excellent is telling how data is transferred but lacks the functionality to tell what the data is. Smart API fills this gap. It supports vocabularies, links and graphs, something that OpenAPI does not. In SmartAPI the definition of data is made with proper ontologies which can be validated and tested. These are the building block for data accuracy needed in critical systems and scientific computing. And ontology definitions are the core of transforming data into knowledge, an essential process with artificial intelligence applications.

1.2. The Smart API SDK

Smart API SDK is a library available for all major network programming languages, designed to simplify and streamline the development of compatible semantics enabled applications. In a nutshell: if Smart API is a design, the Smart API SDK is the free software and programming tools that help implementing the design.

Smart API SDK hides the details of semantic data completely and offers a familiar programming library approach to the task. By using Smart API SDK, developers can use standard language objects and set their values. The library serializes and deserializes these automatically to the network transfer format.

While the SDK offers methods to perform all tasks required by the SmartAPI standard without any knowledge of the actual details of data transfer, the functionality is extensible and values can be added to the data down to the granularity level of single semantic triples. Implementations may be tested against a test server that rates the modifications and ensures the additions do not break the semantic structure.

The Smart API SDK documentation offers examples on how to perform the most common tasks when creating various applications. While understanding of semantics is not required, it is however useful to understand the overall architecture of a Smart API compatible application in order to grasp which parts of the documentation are important for each task at hand.

Code for the Smart API library can be written by hand using any popular editor or IDE. For a quick startup a visit to http://talk.smart-api.io/develop is recommended as the site contains not only the documentation and tutorials but also an automatic code generator that creates code stubs that can be directly pasted into the application.

The Smart API software is available for multiple languages. Smart API for Java provides a plain old java object (POJO) abstraction to semantic Smart API information. All data transfer takes place through these objects. Objects can be created directly as new instances of the classes or by using the Factory methods. Factories generate empty object stubs and fill in the basic properties of the object with reasonable defaults. For example when a Request object is created, a factory automatically assigns system time into the generatedAt field, as required by the standard.

In Java, there are additional helper classes that simply take in objects and output objects. With these classes the detals of data traffic and formatting are hidden and programmers don't need to learn those to perform common tasks. The Factory module creates objects while the Tools module offers a simple starting point to handling objects, for instance serializing and parsing them. Finally, the Agents can be used to communicate with other systems in an easy, single-line-of-code fashion.

Because Smart API is an object management library at its core, programming with Smart API is all about handling objects. So let's take a look at those more closely next.

Chapter 2. Identifying things and addressing services

There are many ways to point onto something in a remote service. Such pointing can be exact i.e. address exactly something, plural i.e. point to many things, or fuzzy i.e. point to something close to but not necessarily exactly what is described. All these types of addressing a thing are covered by Smart API.

Chapter 4, Object and property identifiers talks in more detail how these three addressing types apply to objects and how to use them. But first we need to outline how they apply to the actual servers and services that operate on those objects.

The method used for identifying thins in Smart API is a URI. That applies to everything, including objects that may be representing some database table or similar. If it is a thing that some external party should somehow read, write, modify, etc, it should have a URI. In many systems only the services or parts of a web application have a URI (say a servlet or bean in a Java application server). The objects just have some internal ID based in some UUID or database table ID. In Smart API these internal formats should be translated into URI format.

How about the services themselves then? What URI's should they have?

The purpose of Smart API is to serve as a standard that helps connecting two previously unknown systems together. In this process there is a problem: is servers i.e. the software have just any URI paths, it is a source of confusion. How can I connect to a service without knowing its path?

The first solution to the problem is the Smart API Find registry, this is where you can publish the URIs and their paths for others to learn. But there is also another solution: standardized paths. Each Smart API server should serve a couple of simple URIs that help others discover more of the system.

The services found behind the standard URIs are called Activities. An Activity is something that the service performs. Each Activity has an identifier and can be addressed with that identifier to perform whatever task it is meant to do. The identifier of an Activity is included in the Smart API message, that is in the payload of the message. In many ways, the identifier replaces the purpose of a URI path. So for instance instead of separating services by paths, like in many webservers, you do the same task with Activity identifiers. For instance, you could have two algorithms, one which performs subtraction and one that does addition. Often you would separate these by their call paths, e.g. http://myservice.org/services/subtraction and http://myservice.org/services/addition. In Smart API, you should have just one URI and path, for example http://myservice.org/smart/v1.0e1.0/access. That path would then be called with two separate activity identifiers e.g. http://myservice.org/subtraction and http://myservice.org/addition.

This separation of URIs and Activity identifiers is in Smart API by design. It helps in addressing services in a multi-protocol environment where IoT devices may be communicating over common IoT protocols such as MQTT and CoAP. While CoAP does have at least a limited support for similar paths as HTTP, MQTT has none. By using the payload, this problem can be overcome.

So instead of having an HTTP path for every distinct service or feature you'd want to implement in a Smart API service, you only have a couple of standard paths and the rest is determined by the payload.

That said, there is a considerable legacy of network server components for HTTP that act as proxies, load balancers, authorization servers etc and most of them operate on paths, not payloads. This is why there are certain standard basic URIs available in Smart API. Otherwise one could ask what is the point of having Activity identifiers in the first place as they could be replaced by request paths or alternatively why not have just one path and put everything into the identifiers.

With the basic paths a compromise between the two approaches is made. The paths allow for enough functionality to for instance create access filtering between anonymous and authorized access to the system at proxy level, creating a security layer in between public and private services.

Important

The basic priciple in the choice between paths and identifiers is that of filtering: paths should function as rough filters of access (i.e. access controlled / not access controlled) and multiplexers of load balancing. Activity identifiers select the actual task to be done.

The six basic paths a server should implement are the following

  • /identify. The simplest path of them all. Because everything in Smart API should have a URI that identifies them, a good practice is to have a method to return this identifier. This path should work over HTTP GET or CoAP GET and return back one URI that is the URI of this particular server. There should be no processing functionality behind this path, just one simple function that returns one identifier.
  • /discover. This is a path for anonymous access to discover what your server does. If there is no previous knowledge of the functionality, other paths served etc, discover can respond with a standard format message to tell what it is and what it can do. Discover should not process anything. It should be just something that returns a directory of accessable features.
  • /authorize. This is a path for gaining access to the system. This is where all initial authorization requests go to. Note that protocols such as OAuth2 use their own standard paths. The /authorize path should be used to initialize the conversation, e.g. to request the creation of an account that later on uses OAuth2.
  • /access. This is the main path to communicate to. Once authorized for access, the rest of communication should go here (and then be recognized by the Activities and their identifiers).
  • /monitor. This is a path for admin systems. The monitor path is for automatic monitoring and may for instance return system load, available disk space etc stats needed by system admins.
  • /configure. Remote configuration path. If /monitor is for read-only monitoring, then /configure is for read-write access for actually remotely administering the system.

Important

Access control and security of standard paths should be as follows:
  • /identify and /discover. No access control. Queries open to anyone to discover what a system is.
  • /authorize. No access control in query. This is the path to be used for requests where a user does not yet have access but would like acquire one.
  • /access. User level access control. Requests in this path should contain credentials that authorize access to the standard features of the system.
  • /monitor and /configure. Admin level access control. These paths are typically filtered at firewall or proxy level so that they are not even visible from a public network but only from a secure source.

Note that you can have other paths also. Technically there is nothing in the Smart API design that would prevent you from adding any number of paths. For example, for load balancing purposes you could for instance have something like /access/cluster1 and /access/cluster2 and then do the rest in the payloads. However, you must then announce those non-standard path somewhere, either in the /discover responses or in Smart API Find (see the discussion on registration later in this manual):.

Chapter 3. Object features

3.1. Basic object features and structure

3.1.1. Constructors

Creating a new Smart API object is as easy as creating any other object when doing object oriented programming. Once you have an object, you can add other objects to it to create a data structure. Some objects, such as Request and Response objects have a standardized structure that all compliant systems should follow. To easily create such structures, Smart API SDK has factory classes that create them for you with a single method call.

When you create an object, you can keep it anonymous or assign an identifier for it. Here's an example on creating a new Device object with an identifier:

Device device = new Device(<Smart API identifier uri>);
				

Similarly, the syntax to create a TemporalContext is exactly the same. In the sample below, it is created without the identifier (it becomes a so called "blank node"):

TemporalContext temporalContext = new TemporalContext();
				

To create an object with a factory, you call the corresponding create method. This is the recommended way to create Message objects, such as Requests, Responses and different kind of error messages, as they form the basis of communication between systems and should be correctly formed to ensure compatibility.

Creating a new Request object with Factory:

Request request = Factory.createRequest(<generatedBy uri>);
				

Creating a new Response object with Factory:

Response response = Factory.createResponse(<generatedBy uri>);
				

3.1.2. Getters

Plain values (literals) and other objects are attached to objects using their properties. To manipulate the properties, the Smart API objects offer corresponding getters and setters. Objects also provide methods to test if value has been set for a specific property. This getter-setter-validator structure is available for all objects with a common naming convention. So for instance if a property is called "size", then the object will have getSize, setSize and hasSize methods.

In the example below, we test whether the object has a quantity and if so, get it:

Evaluation evaluation = <some evaluation object>;
if ( evaluation.hasQuantity() ) {
    String quantity = evaluation.getQuantity();
    ..
} else ..
					

Here's another example: getting the coordinates of a Device object:

Device device = <some device object>;
if ( device.hasCoordinates() ) {
    Coordinates coordinates = device.getCoordinates();
    ..
} else ..
					

In addition to the getters specific to the standardized properties of objects, Smart API objects also feature generic getters and setters, namely get() and add() which allow setting any property value to an object. You could for instance do myObject-> myObject.add("http://my.properties.org/myproperty", 20) to set the value 20 to a property identified as "http://my.properties.org/myproperty". To get a property, you would call get("http://my.properties.org/myproperty").

The generic and standard getters and setters are actually linked. The standard versions use the same property naming schemes you would use with generic getters which means that if you supply the correct property to a generic version, you will get the same result as you would with the standard version. These two are equivalent:

device.add(PROPERTY.SIZE, new Size());
device.setSize(new Size());
					

3.1.3. Setters

Setters work the way you'd expect them to work in any object oriented environment. To set a property, you call the setter with the value you want to set.

Example: set the quantity of an Evaluation object:

Evaluation evaluation = <some evaluation object>;
evaluation.setQuantity(RESOURCE.TEMPERATURE);
				

Another example: set the coordinates of a Device object:

Device device = <some device object>;

Coordinates coordinates = new Coordinates();
coordinates.setLatitude(61.124);
coordinates.setLongitude(24.117);

device.setCoordinates(coordinates);
				

Notice that coordinates are an object of type Coordinates that carries latitude, longitude and altitude as its values. As it the case with many such multidimensional properties, you need to create an object that has the values and then use that as the parameter to the setter. That said, many objects also provide shortcut methods which simply take the most common inputs and create those wrapper objects for you. For example, to set coordinates you could do it like this:

Device device = <some device object>;

device.setCoordinates(61.124, 24.117);
				

Please refer to the API docs for lists of such shortcut methods of each object.

3.1.4. Location geocoding

Some objects also offer features to automatically fill in values from some online data source based on some initial input. An example of this is adding the address data to an object by using reverse geocoding. The Tools module provides geocoding functionality to automatically fill in empty address/coordinate fields. If the coordinates are already filled in, they are used for reverse geocoding, otherwise forward geocoding is used.

Location (reverse) geocoding based on coordinates:

Device device = <some device object>;
String myIdentity = <my Smart API uri>;

device.setCoordinates(60.22, 24.81);
Tools.geoCode(device, myIdentity);
				

Location geocoding based on address:

Device device = <some device object>;
String myIdentity = <my Smart API uri>;

Address address = new Address();
address.setStreetAddress("Betonimiehenkuja 3 A");
address.setCity("Espoo");

device.setAddress(address);
Tools.geoCode(device, myIdentity);
				

3.2. Common methods shared by all: the Obj superclass

3.2.1. Constructor

All regular Smart API objects in the model module extend a super class called Obj. Obj class provides common methods for Smart API identifier URI, object type(s), object name, and object sameAs id uri. Obj also supports methods for getting and setting any type of value to any type of property.

To create a new Obj with Smart API identifier, do this:

Obj obj = new Obj(<Smart API identifier uri>);
				

To create a new Obj without Smart API identifier (blank node), do this:

Obj obj = new Obj();
				

3.2.2. Generic getters and setters

As described previously in this document, the Smart API objects have getters and setters for the properties they own. So for example, to set the weight property of an object, you would call the setWeight method. Manipulating properties this way is the recommended approach as those methods ensure that the datastructure remains correct and compliant and the property objects are of correct type.

But what if you do want to set some other property to an object and that does not have a setter? Let's say that you'd need to record the permeability of a physical object and there is no such thing as setPermeability? Or set up some parameter that is only relevant to your application, let's say myFunctionIterations?

For this purpose, each object inherits from Obj the so called generic getters and setters. They allow you to set the values with some arbitrary string as property identifier. That string can be prefixed with a known domain URL or remain as such.

For example, the VCard namespace contains the standard definitions of the VCard contact information standard. Its URL can be found from the Smart API library headers as NS.VCARD. To set up a property nickname with this namespace, you would do

Device device = <some device object>;
device.add(NS.VCARD + "nickname", "Lampy McLampface");
					

The Smart API library also comes with a large set of predefined property URL's (these are actually the things the ready-made getters and setters use). With them you don't have to worry about figuring out what namespace to use. For example, to set a property that ties one device to another with the controlledBy relation, you would do as follows:

Device device = <some device object>;
Device otherDevice = <some other device object>;

device.add(PROPERTY.ISCONTROLLEDBY, otherDevice);
					

It is recommended to use namespaces and URLs in properties, this makes it possible to link them into ontologies for testing, documenting. Eventually this practice leads into much more standards compliant and understandable communication between systems. However, the generic getters and setters do allow you to set those properties as plain strings without any further info. So for example to set those function iterations mentioned earlier, you can do as follows

Entity myFunction = new Entity();
myFunction.add("myFunctionIterations", 100);
					

As a generic setter let's you set properties, a generic getter allows you to fetch those values back. Note that properties behave like a list. You can set multiple values to a single property and the end result is actually a list of property values. To get a value back, you would use the get method and to get all values, use getAll.

Important

Note that while setting multiple values is possible, the ordering of the value list represented by multiple properties is not guaranteed when transferred over network. So if you do set a value multiple times, note that get may return any of those values.

Multiple property values behave like a list and are trasferable to methods representing list datastructures in the programming language. But in a trict sense they are not a list as the values are not ordered and cannot be traversed in a predetermined manner. If you need to preserve the order of a listing, you need to add an actual list which provides the missing features such as ordering, size and traversal. For more info, refer to the chapter concerning lists and maps in this manual.

Below is an example of getting all the properties of a Device and iterating through their values.

Device device = <some device object>;
ArrayList<Variant> values = device.getAll(<some property>);
for ( Variant valueVariant : values ) {
	if ( valueVariant.isString() ) {
		String value = valueVariant.asString();
		// .. do something ..
	}
	if ( valueVariant.isInteger() ) {
		Integer value = valueVariant.asInt();
		// .. do something ..
	}
	if ( valueVariant.isDouble() ) {
		Double value = valueVariant.asDouble();
		// .. do something ..
	}
	if ( valueVariant.isFloat() ) {
		Float value = valueVariant.asFloat();
		// .. do something ..
	}
	if ( valueVariant.isBoolean() ) {
		Boolean value = valueVariant.asBoolean();
		// .. do something ..
	}
	if ( valueVariant.isDate() ) {
		Date value = valueVariant.asDate();
		// .. do something ..
	}
	if ( valueVariant.isTime() ) {
		Time value = valueVariant.asTime();
		// .. do something ..
	}
	if ( valueVariant.isObj() ) {
		Obj value = valueVariant.asObj();
		// .. do something ..
	}
	if ( valueVariant.isDuration() ) {
		Duration value = valueVariant.asDuration();
		// .. do something ..
	}
	if ( valueVariant.isList() ) {
		List value = valueVariant.asList();
		// .. do something ..
	}
}
				

To iterate the properties with their property identifiers, you can do as follows:

Device device = <some device object>;
HashMap<URL, ArrayList<Variant>> propertiesMap = device.getAllProperties();
// For each property
for ( Map.Entry<URL, ArrayList<Variant>> propertyEntry : propertiesMap.entrySet() ) {
	// Get property URI
	String property = propertyEntry.getKey().toString();
	ArrayList<Variant> values = (ArrayList<Variant>)propertyEntry.getValue();
	// For each value
	for ( Variant valueVariant : values ) {
		if ( valueVariant.isString() ) {
			String value = valueVariant.asString();
			// .. do something ..
		}
		if ( valueVariant.isInteger() ) {
			Integer value = valueVariant.asInt();
			// .. do something ..
		}
		if ( valueVariant.isDouble() ) {
			Double value = valueVariant.asDouble();
			// .. do something ..
		}
		if ( valueVariant.isFloat() ) {
			Float value = valueVariant.asFloat();
			// .. do something ..
		}
		if ( valueVariant.isBoolean() ) {
			Boolean value = valueVariant.asBoolean();
			// .. do something ..
		}
		if ( valueVariant.isDate() ) {
			Date value = valueVariant.asDate();
			// .. do something ..
		}
		if ( valueVariant.isTime() ) {
			Time value = valueVariant.asTime();
			// .. do something ..
		}
		if ( valueVariant.isObj() ) {
			Obj value = valueVariant.asObj();
			// .. do something ..
		}				
		if ( valueVariant.isDuration() ) {
			Duration value = valueVariant.asDuration();
			// .. do something ..
		}				
		if ( valueVariant.isList() ) {
			List value = valueVariant.asList();
			// .. do something ..
		}				
	}
}
					

When you get values from an Obj or any subclass in most cases you'll receive a Variant object which is a common wrapper for all values. It can be used to store values without worrying about typecasting. When you get the value back, the Variant offers methods to cast itself to the desired type. Obj also supports shortcut methods that directly fetch the value and cast it to a type as illustrated by the following examples.

Example 1 - get first value for the property as String:

Device device = <some device object>;

String nickname = device.getFirstAsString(NS.VCARD + "nickname");
// or
String nickname = device.getFirstAsString("vcard:nickname");
					

Example 2 - get first value for the property as Variant:

Device device = <some device object>;

Variant nicknameVariant = device.getFirst(NS.VCARD + "nickname");
// or
Variant nicknameVariant = device.getFirst("vcard:nickname");

if ( nicknameVariant.isString() ) {
	String nicknameUri = nicknameVariant.asString();
	// .. do something ..
}
					

Example 3 - get first value for the property as Object:

Device device = <some device object>;

Object nicknameObject = device.getFirstValue(NS.VCARD + "nickname");
// or
Object nicknameObject = device.getFirstValue("vcard:nickname");

// Have to know the real Object class to be able to cast
String nickname = (String)nicknameObject;
					

Example 4 - get all values and cast to string:

Device device = <some device object>;

ArrayList<Variant> nicknames = device.get(NS.VCARD + "nickname");
// or
ArrayList<Variant> nicknames = device.get("vcard:nickname");

for ( Variant nn : nicknames ) {
	if ( nn.isString() ) {
		String nickname = nn.asString();
		// .. do something ..
	}
}
					

3.3. Automatic object documentation and analysis

While admittedly it always takes time to define data and APIs with a comprehensive design instead of just shoving the values in, once that is done the benefits are many and easily accessible. One of the benefits is that Smart API objects can explain themselves to the engineers without any accompanying documentation. So in essence with Smart API, you can just code and leave the documentation for the software.

During coding the objects offer three helpful methods for debugging and displaying their data. These are especially helpful when you connect to a system previously unknown to you. You don't need to contact the authors of the system to figure out what it does. If the system uses Smart API, the data you receive will explain itself.

3.3.1. Printing objects as RDF

RDF is the data model used to semantically describe data. RDF itself is not a text format, it is a framework within which there are many formats that include for instance JSON-LD, RDF/XML and Turtle. The neat thing is that all those formats are interchangeable. If you receive something in RDF/XML, you can convert it into JSON-LD any time. When Smart API objects are transferred over the network (or serialized), they are expressed in RDF.

It is often useful to see the raw RDF representation of an object to see what it actually contains. To do this, you can use the turtlePrint method. Like so:

Obj o = <some object>;
o.turtlePrint();
					

3.3.2. Printing out the object graph

While RDF is a neat representation of objects once you get used to reading it, just a raw text may become hard to follow if you have a large number of intertwining objects bound together. In this case it would be better to somehow actually "draw" their relationships.

For the purpose of printing out the structure of an object, Smart API objects offer a method called printOut. It will traverse the object and its links and print out the structure and datatypes of each as it goes. Like so:

Obj o = <some object>;
o.printOut();
					

3.3.3. The explain method

Printouts tell you what the objects contain and how they are linked but do not actually explain to you what various properties actually mean. This is where the magic of ontologies come into play. With them, the objects can actually document themselves and explain to you the defined meaning of each property. To make object explain the data, you use the explain method as follows:

Obj o = <some object>;
o.explain();
					

Explaining takes a while longer to perform than just a printout as it downloads the definitions of data from the property URLs during the process. It therefore should not be used in production code. But for debugging and documenting it is an essential tool.

3.3.4. Concept and conformance validation

Validation is the process of testing that the data you have entered is done according to a design. Validation is essential when you make designs for connected systems as it ensures parties understand the messaging properly. The Smart API Ontology Agent provides semantic validation with two methods: quickValidate method for quick concept validation and fullValidate method for complete conformance analysis.

Both validation methods return a Grading object that contains a score of how well your data fits the design as well as a list of issues detected. The Grading object contains the following grades for your data:

  • Grade. This is the average of all other grading values, the "overall score".
  • syntaxGrade. Syntax grade is given based on the quality of the (textual) presentation of the semantic data, i.e., serialization. A serialization gets full 10 points for syntax grade if validation service is able parse in the tested object, and zero points if parsing fails. Note that quickValidate does not calculate the syntax grade because it only tests objects, not their serialization.
  • valueGrade. Value grade is given based on the correctness and the value types of triples. i.e. are property values of correct type or do they violate the type or exceed their range. Note that quickValidate may omit the range checking of some properties in case their definition is not locally available while fullValidate checks the type of all values against the ontology.
  • conceptGrade. Concept grade is given based on checking that the used concepts (URIs) are defined in the ontologies. Both quickValidate and fullValidate check this for all used property URIs and for the values of some specific properties that are expected to have a value that is defined in the ontologies. These properties are type, method, quantitykind, unit, category, distribution, during, and encryptionkeytype.

Concept validation with quickValidate is a tool to test if semantic resource and property identifiers in your object are correctly used. It is particularly useful when new concepts are defined in ontology and then used in the code. Use concept validation as follows:

Obj o = <some object>;
Grading g = OntologyAgent.quickValidate(o);
g.printOut();
					

Conformance analysis with fullValidate is a more comprehensive validation compared to concept validation. It ensures that object structure, as well as object types and property value types conform to the ontologies. So, if ontologies contain vocabulary and grammar for the language that is used to describe the object, then conformance analysis is a "language exam" for that object. Use conformance analysis as follows:

Obj o = <some object>;
Grading g = OntologyAgent.fullValidate(o);
g.printOut();
					

Chapter 4. Object and property identifiers

4.1. URIs

One of the primary ideas behind Smart API is to be able to globally identify connected objects that may be spread across multiple systems and geographies. And so far the best method invented to do that is the good old URI. Every identifier therefore has some http://<domain>/<path> structure. URIs are great because each organization is supposed to have their own domain and that neatly divides the identifiers into unique "namespaces".

In Smart API anything (but not necessarily everything) can have a URI as an identifier. So for instance to command a lamp you could address the lamp and tell it to turn ON or address the switch on the lamp and tell that to go to ON position. Which on that is depends on the design of the application and is also the subject of a somewhat larger topic: how to handle the situations where you'd like to perform an action but don't necessarily know exactly all the details needed.

4.2. Predicates and identifiers

When you identify something in life, that identifier is bound to some concept that is understood by the communicating parties. Two common ways of identifying something are to talk about what that something represents and what that something is. Let's take an example.

Imagine a friend of yours says the following: "I spoke with my mother yesterday." That sentence quite unambiguously defines the person your friend spoke with as it is rare that someone has more than one person he or she refers to as "mother". The word "mother" here not only represents what the person means to your friend but also who that person is. It would be somewhat odd to use the wording "I spoke to my mother Anna today" as again, because it is assumed a person has only one mother, we rarely use the name of that person in conjunction with the family relationship.

Now, consider this: "I spoke with my uncle today." This is trickier. It is not uncommon for a person to have more than one uncle. This is why saying "I spoke with my uncle Tom today" does not sound that strange. When you start speaking about such relatives, it is not uncommon to use two identifying parameters at the same time.

There is a similar distinction in Smart API in defining things. The predicate defines the role and the identifier the identity. Similar to using one or the other or both while speaking about relatives, you can use one or the other or both while defining data in Smart API. Sometimes the choice depends on whether data is available, sometimes it is needed to recognize for instance items in a list.

The only difference between Smart API and natural language is that the identifier is supposed to be unambiguous. In real life, if you say "I spoke to Tom", that might not accurately point to a particular person if there are many people called Tom. But if in Smart API you use the identifier "Tom", it is assumed there won't be more than one Tom. That said, there are plenty of ways to define fuzzy identifiers that may point to multiple objects, more on that later.

So let's as an example model that friend who has mother Anna and two uncles, Tom and Bob.

Person friend = new Person();
Person mom = new Person();
Persom tom = new Person("Tom");
Person bob = new Person("Bob");
friend.add("http://familyrelations.org/ontology#mother", mom);
friend.add("http://familyrelations.org/ontology#uncle", tom);
friend.add("http://familyrelations.org/ontology#uncle", bob);
				

That code will result in the following text representation in Turtle

<Bob> a smartapi:Person .
<Tom> a smartapi:Person .

[] a smartapi:Person ;
    familyrelations:mother [ a smartapi:Person ] ;
    familyrelations:uncle <Bob>, <Tom> .
				

Notice how people are now identified differently in the structure. First, we did not give any information about your friend. Not even the name. And because that person's relationship outwards from this representation (graph) is not defined, we did not actually say that he or she is a friend of anyone. That is why the friend is defined as a so called blank node []. Second, we did not give the name of the mother. This is why we only have the relationship given with the predicate familyrelations:mother and some object that is known to be a Person.

With the uncles, we did provide identifiers: Tom and Bob. This is why those two fellas can be recognized with their identifiets Tom and Bob as well as their family relationship familyrelations:uncle.

That example should clarify the difference between predicates and identifiers. The former tells us what something is while the latter defines who that person is. Now, how should these two be used in code and interpreted? That depends on the situation and how much data is available. If you can provide both, good. If not, usually the predicate is required because with that relationships are formed.

The more complete the data is, the easier it is to process. The data in the example above gives us quite a bit a freedom for for instance data searches. We could query all persons connected to this friend (the query would match all object types Person and return three matches). Or we could find all uncles (returning two objects) or a particular person called Bob.

When data is not complete, most systems that rely on more restricted API models just give up and return an error. But due to the flexibility in defining data, Smart API offers possibilities to create APIs that can intelligently do their best with what is available and provide results even when everything does not match. Managing such uncertainty in data is the topic of the next chapter.

4.3. Managing uncertainty and doing deduction

Controlling and reading things over the Internet typically requires two things: you know exactly the address of the device to connect to and exactly the specs of the data or controls you want to handle. Fair enough, this usually works just nicely when you're dealing with one service or a pool of devices from a couple of manufacturers. But what if you need to send a command to a million different devices from a dozen different manufacturers?

Smart API addresses this problem with three different methods:

  1. Device registry. A registry is a place like a phonebook where things can be sought from. find.smart-api.io is the central registry that Smart API offers for this purpose. When devices are registered into the directory, they send information about their URIs, capabilities, and properties to address. The directory then knows the exact details needed to access that particular device.
  2. Standardized identifiers. Smart API is designed to function as a standard for communication. Therefore it standardizes the names of common properties. So if you have a device somewhere and would like to receive its location, you should ask it with the geo:location property. If all standards compliant devices use this identifier for location, you can ask the location without knowing anything further about the object.
  3. Deduction. Often it is the case that you actually want data that is something like a thing but not necessarily exactly that thing. Say for instance you'd like to extract the temperature from a unit but you don't know how it measures that temperature and what the identifier of a particular temperature sensor might be. The solution to this problem is to address the issue with a quantity and let the recipient deduct and choose which measure best matches that quantity. So in essence you'll be asking "what is the temperature you are measuring?" instead of "what is the value of your temperature sensor?"

The models described above are expressed in Smart API with objects called ValueObjects. You'll learn more about them later in the manual but before a really quick brief and a couple of examples to highlight what the distinction above means in practice.

A ValueObject is simply a structure that describes a value. It has four main components:

  • the value itself (e.g. 36.2),
  • the quantity the value represents (e.g. body temperature),
  • the unit it is measured in (e.g. degrees celsius), and
  • the identifier that can be used to recognize and address the value (e.g. Jason's temperature in the morning).

Any of those four may be omitted depending on how exact the data is desired to be. A ValueObject is then attached to some entity with a property name (a predicate) that creates a relation stating what that ValueObject is to the entity. For example, in this case we could have an entity Person, called Jason, who has two values with predicate bodyTemperature. The first ValueObject has ID MorningTemperature and a value of 36.2 degrees centigrade. The second ValueObject is ID EveningTemperature and value 37.1 degrees centigrade. Both values are of quantity Temperature.

Now, let's make a few requests to get body temperatures from some system where Jason's medical data is stored. To do so, we would create and object and send that to a server that responds to Smart API. In the examples, notice especially how requests are made. To ask for data on an object, you actually send an object to the server. The details you want to receive are the properties of that object without their values. The server should respond back with a similar object but with those fields filled in. It is as if the request you send is a template of an object and you ask the server to fill in the blanks found in that template.

So, let's first do this:

Person jason = new Person("http://www.thejoneses/jason");
ValueObject vo = new ValueObject("http://www.thejoneses/jason/Morningtemperature");
vo.setUnit(RESOURCE.DEGREECELSIUS);
jason.addValueObject(vo);
				

That program tries to find a person with URI http://www.thejoneses/jason. Then it should find a specific temperature, possibly among many temperatures, that is the morning temperature. It is identified by http://www.thejoneses/jason/Morningtemperature.

Let's next tweak that request slightly to show how the tweaks affect the semantics of the search. Here's another request:

Person jason = new Person("http://www.thejoneses/jason");
ValueObject vo = new ValueObject();
vo.setUnit(RESOURCE.DEGREECELSIUS);
jason.add(NS.SMARTAPI + "bodyTemperature", vo);
				

Again, in this example we are looking for something from Jason, but this time we don't identify exactly which value we want. Instead, we want to search the properties based on their relation to Jason. In the example, the values - possibly many - are attached to Jason with the relation (predicate) bodyTemperature. The service should now return the evening and morning temperatures (as opposed to the first search that returned morning only).

Finally, here's the last tweak:

Person jason = new Person("http://www.thejoneses/jason");
ValueObject vo = new ValueObject();
vo.setQuantity(RESOURCE.TEMPERATURE);
jason.addValueObject(vo);
				

Now this one requires some more intelligence from the server. We're again looking for Jason's data but actually don't tell much about what we want. We just want his "temperature". When and what temperature, that is not specified. The system that processes that request is free to choose what measurement of temperature (if any) best matches the request.

What if we would not want to specify anything about the data? The lowest possible granularity allowed is that of the identifier of an object. If the object skeleton sent contains nothing more than just the identifier, then the service should return that object and fill it with all the data it knows about that object. So the structure that basically is the same as "getObject" is in case this:

Person jason = new Person("http://www.thejoneses/jason");
				

At this point, you may ask how in practice would you actually send that object to a server. Clearly if you just create an object it does not mean that it somehow magically finds content for itself. Chapter 9, Requests and Responses will teach you all about that. But as a quick reference here for completeness, this is what an actual network call for Jason's data would look like:

import smartapi.common.HttpClient;
import smartapi.factory.Factory;
import smartapi.model.Person;
import smartapi.model.Request;
import smartapi.model.Response;

HttpClient httpClient = new HttpClient();
Person jason = new Person("http://www.thejoneses/jason");
Request request = Factory.createReadRequest("http://me.example.com", jason);
Response response = httpClient.sendPost(<server uri>, request);
				

Chapter 5. Reading, writing, creating, and deleting objects

5.1. Object templates

Reading and writing data as objects in Smart API is based on the idea of object templates. A template looks like the object you want to manipulate, but is missing values (in case of read) or contains values (in case of write) that you want to read or write. So a read is like saying "I'd like to have objects like this but don't know the details, would you fill them in for me" to the server. A write in a similar manner is "this is what I'd like the object to look like, could you change it for me please".

Whenever an operation (read, write, delete) takes place, it can essentially be split into two distinct stages:

  • First we need to find the objects to operate with. This is called "selecting" and is done by a "selector" process.
  • Next, we actually perform the operation on the found objects.

In simple operations the two stages can be instructed with just one data structure. When the operation gets more complex, they can be split into distinct Activities, each with specific parameters. The selector process then becomes the input to the operation process.

5.2. Reading objects

As a concept, reading an object is simple: tell a system which object you want and expect the system to send you back that object. In Smart API the process is equally simple when you know exactly the object you are interested in: give the target system the identifier of the object and you get that object. Where it gets more complicated is when you actually don't know the object but would rather want to search for it. Luckily Smart API offers an elegant way to do such searches, too.

For an introduction on how the uncertainty related to searching is handled on conceptual level, please read 4.3, “Managing uncertainty and doing deduction”. For practical examples on how it is done, read on.

5.2.1. Reading known objects

Let's start with a basic client side example of reading literal properties (e.g. numbers and strings) and linked objects (e.g. coordinates) of a remote object.

To make a read, we need a Request that addresses our service and an Activity that tells what our service should do. As we are doing a read, the Activity should, unsurprisingly, be a read activity.

Request request = Factory.createRequest("http://my.samplerequester.com");
Activity activity = request.newActivity();
activity.setMethod(RESOURCE.READ);
					

Next, all we need is an Entity that has the identifier of the object. If the identifier in this case would be the URI http://acme.com/o/C35782, then we'd do as follows to add an Entity template with the ID to the Activity:

Entity entity = activity.newEntity("http://acme.com/o/C35782");
					

The request made by our client now contains a template object that corresponds to an object the server side (if found, of course). In the example above, the only matching parameter in the template is the identifier of the entity. As the identifier should be unique per object, the query should return either exactly one object or none if not found.

Let's have all that in a few lines of code once more

Request request = Factory.createRequest("http://my.samplerequester.com");
Activity activity = request.newActivity();
activity.setMethod(RESOURCE.READ);
Entity entity = activity.newEntity("http://acme.com/o/C35782");
					

The resulting data now contains the details that recognize us (http://my.samplerequester.com) as the requester of data, an Activity and its type so that the service knows what type of operation we want it to process, and finally addresses the Entity we are interested in.

As nothing else but an empty template with an identifier is given for the Entity, in Smart API this means "tell me everything about this object". The server should fill in the details of the properties of this object. By convention, just identifier means in Smart API "everything of exactly this one". If instead of leaving the Entity blank, we would add some properties into the Entity, then the convention does not hold. Instead of everything, the server should return only those values of that the included properties represent.

Note that the phrase above says "this object". This is important as it makes a distinction between an identified object and objects linked to that object. If the object links to some other object, the server should not return the details of that object unless specifically told to do so through the process of traversing. So let's look at that next.

5.2.2. Traversing objects

In Smart API, objects representing data can be linked to each other. So for instance, if we have two lamps with a cable connected between them, we can link those two with a relation such as connectedTo. Now, if we read one of the lamps, should the data of the other lamp be returned also i.e. should we follow this link between them?

By default, Smart API server implementations should not follow links. While Smart API automatically handles issues such as self-referencing and circular references to avoid data processing problems with graphs, following links can easily lead into huge dumps of data in case everything is connected together. Instead, the depth of such traversal should be controlled by the requester.

To control the depth of object traversal, requesters can pass traversing and recursion parameters in the request to better define which objects are targeted.

  • A traverse defines which levels read or write considers. In effect a traverse affects how many levels of objects are filled in for objects that are found.
  • A recursion defines how many links the server should follow to find the target objects. In effect, recurse affects how many links are followed to find the objects to fill in.

Examples below show the difference of these two in more detail.

There are two types of traversing: traverseUntil and traverseTo. A traverse of type traverseTo means that we are only interested in the object at the end of the traverse chain. traverseUntil means we are interested in all objects along the path (including the one at the end). In the former case, only the data of the object at the end of the chain would be filled in. In the latter, all data of all objects in between would be filled in and returned.

In read requests, traverseUntil value 1 means that only the root entity object of the request is to be filled. With value 2, also second level objects are filled. And so forth.

Traverse parameters are properties of the Activity we use for reading data. In code, we can manipulate them with the setTraverseUntil and setTraverseTo methods. So let's read our entity again and this time fill in also the first level objects that are linked to it:

Request request = Factory.createRequest("http://my.samplerequester.com");
Activity activity = request.newActivity();
activity.setMethod(RESOURCE.READ);
activity.setTraverseUntil(2);
Entity entity = activity.newEntity("http://acme.com/o/C35782");
					

Now, in the example above, if the server sees that there is an object linked to Entity with ID http://acme.com/o/C35782 it will follow the link to that object and return its data also.

5.2.3. Searching for objects

To search for an object, we do this the same way as when reading a known object, but leave the identifier empty and instead fill in some other properties that recognize it. For instance, the property could be a type. As an example, let's find all objects that are lamps i.e. have their type property set to the URI defining a lamp:

Request request = Factory.createRequest("http://my.samplerequester.com");
Activity activity = request.newActivity();
activity.setMethod(RESOURCE.READ);
Entity entity = activity.newEntity();
entity.addType(RESOURCE.LAMP);
					

Note that the example above will return lamps but only their ID's, not any data of those lamps as we have not defined an ID, which in turn would be a sign for the server to fill in all data. So in essence this is just a list of ID's of data that was found. To get the actual data of the lamps we found, we can do one of the following:

  1. Once we have the list, make a call to each of the entities in the list of returned values by including just the ID. This will return all the data of that entity in each call.
  2. In the search, include a setTraverseUntil parameter which will instruct the Activity to fill in the data to a given level.
  3. In the search, add templates of the properties we want to have filled in.

As the first option is already covered, let's explore the two latter ones:

Request request = Factory.createRequest("http://my.samplerequester.com");
Activity activity = request.newActivity();
activity.setMethod(RESOURCE.READ);
activity.setTraverseUntil(1);
Entity entity = activity.newEntity();
entity.addType(RESOURCE.LAMP);
					

Above we now have a request that searches to all lamps in the system and once found, fills in all the details of the first level properties of those objects and returns everything in the resulting data as a response to the search.

Next, let's do something more fine-grained:

Request request = Factory.createRequest("http://my.samplerequester.com");
Activity activity = request.newActivity();
activity.setMethod(RESOURCE.READ);
Entity entity = activity.newEntity();
entity.addType(RESOURCE.LAMP);
ValueObject vo = entity.newValueObject();
vo.setQuantity(RESOURCE.BRIGHTNESS);
vo.setUnit(RESOURCE.PERCENT);
					

In the example above, we again look for lamps in the system but in this case add a ValueObject into the Entity. This means that we are only interested in the data that this ValueObject represents. In this case brightness.

Important

Note that including a ValueObject in the search also has a side-effect on the search. It acts as a filter condition. As the server cannot return brightness in case it is not defined (returning non-existing properties in search is not allowed), the result will only contain those objects that have a brightness property.

Finally, let's look at a graph search operation that involves parameters for both the selector process and the fill in process. These parameters are the traverse and recurse parameters.

The example below shows a search where we want to search (recurse) the graph to a certain depth and once there, we fill in (traverse) the found objects by following their links to a given depth. In the example we have a set of lamps connected to each other. The connections start from a given lamp (http://acme.com/o/lamp432). This lamp manages the next lamp, which in turn manages the next lamp, etc. We want to follow those manages connections to collect all the lamps in the management ring. For each lamp we then read values, filling in data of connected objects (which may be other things than lamps) to a given depth.

// First, we need an Activity for reading
Request request = Factory.createRequest("http://requester.com");
Activity activity = request.newActivity();
activity.setMethod(RESOURCE.READ);

// Next, find the lamp from which to start searching
Entity entity = activity.newEntity("http://acme.com/o/lamp432");

// Define that we want to follow the MANAGES relation of the lamps
// and include all lamps connected with this relation to our
// pool of found lamps
entity.setRecursionUntil(3);
entity.addRecursionProperty(PROPERTY.MANAGES);

// Finally, tell our Activity to take all the found lamps
// and fill in the details of each and traverse any link to other
// objects each lamp may have, filling in data as it goes
activity.setTraverseUntil(3);
					

5.2.4. Requesting measurement history and other timeseries

The examples and instructions so far have concerned properties and objects in just one point in time, usually simply the latest state. What if we'd like to look at the history and fetch a timeseries of data? This is equally straightforward as the other methods. Because it is the Activity that processes our data, we need to tell that Activity to reach a bit further to the past (or future, if we'd like to get a prediction instead). So, let's do that.

First, as a reminder, here's how one would read one single value from an object, in this case the particular brightness of a lamp:

Activity activity = request.newActivity();
activity.setMethod(RESOURCE.READ);
Entity entity = activity.newEntity("http://acme.com/o/C35782");
ValueObject vo = entity.newValueObject("http://acme.com/o/C35782-lamp5-br");
vo.setUnit(RESOURCE.PERCENT);
					

Next, let's look at how this query would be modified to read multiple values as a timeseries of brightness measurements:

Activity activity = request.newActivity();
activity.setMethod(RESOURCE.READ);
TemporalContext tc = new TemporalContext(Tools.stringToDate("2017-01-01T12:00:00+03:00"), 
		Tools.stringToDate("2017-12-31T12:00:00+03:00"));
activity.setTemporalContext(tc);
Entity entity = activity.newEntity("http://acme.com/o/C35782");
ValueObject vo = entity.newValueObject("http://acme.com/o/C35782-lamp5-br");
vo.setUnit(RESOURCE.PERCENT);
					

In the example above, we create an object of type TemporalContext and give it a starting point and an end point. In this case we'd like to receive one year worth of data in 2017.

The TemporalContext is a very flexible way to define time ranges. You can for instance leave one of the dates empty and just define the granularity of data (e.g. hourly, daily, weekly).

5.3. Writing objects

Writing an object is in all but the simplest cases two phase process. Write queries are therefore a combination of two Activities, one to select, one to update. The simplest cases are where the selection is trivial due to known identifiers. In such cases the two Activities can be combined. Let's start with the simple cases.

5.3.1. Writing objects when identifiers are known

When all identifiers of the write operation are known, then there is no need for a separate selector activity. The following Request sets the brightness of a given lamp to 80 percent by addressing just the write.

Request request = Factory.createRequest("http://my.samplerequester.com");
Activity activity = request.newActivity();
activity.setMethod(RESOURCE.WRITE);
Entity entity = activity.newEntity("http://acme.com/o/C35782");
ValueObject vo = entity.newValueObject("http://acme.com/o/C35782-lamp5-br");
vo.setQuantity(RESOURCE.BRIGHTNESS);
vo.setUnit(RESOURCE.PERCENT);
vo.setValue(80);
					

In the example above, we know the identity of the lamp (http://acme.com/o/C35782) as well as the identity of the property of the lamp we want to write to (http://acme.com/o/C35782-lamp5-br). The rest is therefore straightforward: create an Activity that is of type write and set a value and unit for the ValueObject that represents our brightness value.

5.3.2. Writing objects when objects need to be selected first

If we do not know the identifier or would like to perform a write to multiple objects in one call, the objects need to be selected first, then written. This selector activity is given as an input to the write. The server receiving such a query will first perform a read to find the objects and then apply the object template in the write part onto each one of them.

Let's have an example where we set the brightness value of all lamps to 60 percent

Request request = Factory.createRequest("http://requester.com");

// Create two activities and link them together
Activity updateActivity = request.newActivity();
Activity selectorActivity = new Activity();
updateActivity.setMethod(RESOURCE.WRITE);
selectorActivity.setMethod(RESOURCE.READ);
updateActivity.newInput().addActivity(selectorActivity);

// Define what type of objects we want to select, in this case lamps
Entity readEntity = selectorActivity.newEntity();
readEntity.addType(RESOURCE.LAMP);

// Write a skeleton of the Entity we like to write data with
Entity writeEntity = updateActivity.newEntity();
ValueObject writeVo = writeEntity.newValueObject();
writeVo.setQuantity(RESOURCE.BRIGHTNESS);
writeVo.setUnit(RESOURCE.PERCENT);
writeVo.setValue(60);
					

Quite straightforward, although does involve more code to get things done. Let's filter those objects a bit more. This time we want to again modify lamps but this time only those that have a specific value. In the following example we set the lamps that have brightness at 50 percent to 100 percent.

Request request = Factory.createRequest("http://requester.com");

// Create two activities and link them together
Activity updateActivity = request.newActivity();
Activity selectorActivity = new Activity();
updateActivity.setMethod(RESOURCE.WRITE);
selectorActivity.setMethod(RESOURCE.READ);
updateActivity.newInput().addActivity(selectorActivity);

// Define what type of objects we want to select, in this case lamps
Entity readEntity = selectorActivity.newEntity();
readEntity.addType(RESOURCE.LAMP);
ValueObject readVo = readEntity.newValueObject();
readVo.setQuantity(RESOURCE.BRIGHTNESS);
readVo.setUnit(RESOURCE.PERCENT);
readVo.setValue(50);

// Write a skeleton of the Entity we like to write data with
Entity writeEntity = updateActivity.newEntity();
ValueObject writeVo = writeEntity.newValueObject();
writeVo.setQuantity(RESOURCE.BRIGHTNESS);
writeVo.setUnit(RESOURCE.PERCENT);
writeVo.setValue(100);
					

The code above is almost the same as with setting all lamps but in this case we just add one additional filter condition which is the brightness that needs to be at given value.

5.3.3. Recursive writes

Just like in reading where we may want to follow connected links to a certain depth, in writes the write operation can also be instructed to follow connections to a deeper level. Let's assume we have three thermostats that form a network. The first thermostat manages the second one, the second the third one. Managing in this case does not mean that they sync their values, to set the values we still need to address each of the thermostats. If we only have the ID of the first one, we can tell the write operation to follow a given relation and then apply the write template to whatever is found behind that link. Here's the code that sets each thermostat to 25 degrees centigrade:

Request request = Factory.createRequest("http://requester.com");
Activity activity = request.newActivity();
activity.setMethod(RESOURCE.WRITE);
Entity entity = activity.newEntity("http://acme.com/o/C35782");
ValueObject vo = entity.newValueObject("http://acme.com/o/C35782-thermo-temp");
vo.setQuantity(RESOURCE.TEMPERATURE);
vo.setUnit(RESOURCE.DEGREECELSIUS);
vo.setValue(25);
entity.setRecursionUntil(3);
entity.addRecursionProperty(PROPERTY.MANAGES);
					

The recursive code follows the same familiar pattern as before. The only difference is that in this case we tell it to continue from http://acme.com/o/C35782 three levels down (if levels are found).

5.3.4. Server side processing of read/write operations

As you've probably noticed, Smart API offers a very comprehensive set of selection, graph traversal, and combinatory options. And implementing the full set of them may seem overwhelming at first if you need to create a server that supports the full feature set. Don't be alarmed by this. First, this manual outlines the various possibilities the API offers but is not an spec that requires everything to be implemented. If your server only needs to support selection by simple ID's, then just implement that part. Just because there are roads to all kinds of places in the world does not mean you need to follow all of them. Pick the one that takes you to the destination.

Further, server implementations can follow certain common procedures to implement object oriented request processing. Once you wrap these common procedures into classes or similar structures, they are easy to re-use in various parts of the processing.

First, start with reading the Activity to determine if it is to be processed as read or write. Note that you may specify also other types of Activities, say Big Data analysis activities or estimation and prediction processes.

If the Activity is to read or write objects, next query for the Entities inside the Activity. That object and all objects it contains, also recursively, should be considered as search parameters. Let's refer to it as 'Search Object', short SO. In read requests the SO is a template that the server should fill and return. In basic write requests the SO and any other object in it should have identifiers so that the corresponding object in the system can be found and replaced with the one given in the request. Alternatively, write Activity may have a read Activity as input. In such case, the read Activity is considered as a selector to define the objects that are replaced by the object in the write activity.

5.3.4.1. Processing a search

For the selector Activity, process the Entities as follows

  1. If SO is a ValueObject, data for it is always filled completely for read requests and conversion is done based on the unit if one is provided.
  2. If SO has identifier, server should only consider corresponding object in the system, otherwise all such objects that match the search criteria are considered.
  3. If SO has any properties, then for each of its property P and value V:
    1. If V is a literal, it is considered as a search parameter for matching objects, meaning that you only consider objects that have the given value.
    2. If V is an Object, consider it recursively as a new SO (step 1).
  4. If SO does not have any properties:
    1. If SO has identifier, data for it is completely filled for read requests and for write requests it is considered as a search parameter.
    2. If SO does not have identifier, only return identifiers of corresponding objects in the system, or if the system objects do not have identifiers, then they are returned as filled objects.

5.3.4.2. Processing traversal

A read request Activity may include traverseUntil (TU) or traverseTo (TT) properties to define the depth of objects that are requested to be filled in the response. traverseUntil property means that all levels until the given value are filled. Instead, traverseTo property means that only objects on the given depth are filled. In case of multiple traverseTo property values, objects on all given depths are filled. If both traverseUntil and traverseTo properties are given, objects that are on or lower level than TU as well as objects on the TT levels are filled. For instance, if traverseUntil is 3, and traverseTo has values 6 and 8, then objects on levels 1, 2, 3, 6, and 8 are filled. If neither traverseUntil nor traverseTo property are defined, default value for traverseUntil is 1.

Filling objects traversed to:

  1. If this level is to be filled, fill all literal properties.
  2. If there is a deeper level that is still to be filled based on traverse parameters, move on to fill each object property value. If not, fill each object property with identifier of the object if such exists. If object does not have identifier, fill it recursively.

Chapter 6. Controlling objects

6.1. Basics of sending control commands

The basic concept of controlling something with Smart API is straightforward: the target of the command is represented by and object and the thing to control is a property of that object. So to control the object, you simply change the property. So for example to make a car drive 51 km/h, you set the speed property of the car to 51 km/h. Simple.

Because properties are described in Smart API as semantic values with proper definitions of quantities and units, the commands naturally adapt to changes in the units and the automatic unit conversion features of Smart API help the communicating parties in setting everything right. So, to set the speed to 51 km/h, you could set the value to 51 and unit to km/h or value to 14.1667 and unit to m/s. Both work equally well.

Whether the object at the receiving end actually obeys that command is implementation specific. If you send a speed property to an object that is firmly bolted to the ground and never intends to move, there is not much an API can do about it.

But, assuming that the object is really controllable, let's take that speed as an example. Smart API defines a standard object for speed, so let's do it with that:

PhysicalEntity car = new PhysicalEntity();
Velocity v = new Velocity();
v.setGroundSpeed(new ValueObject(RESOURCE.KILOMETERPERHOUR, 51));
car.setVelocity(v);

HttpClient httpClient =  new HttpClient();
Response resp = httpClient.sendPost(<server uri>, Factory.createWriteRequest(<my identity>, car));
				

6.2. Writing vs recording

By default, Smart API systems should assume that if you write a property, you want that property to result in whatever physical phenomenon is the result of changing the value. Write the speed to make a thing go faster or slower, write a temperature to make the thing hotter or colder.

But what if you don't want to change the phenomenon but simply record what is its state? This process is, not surprisingly, called "recording". Let's take the example of the car again. We'd like it to move with the speed of 51 km/h but it is an autonomous vehicle which will slow down in curves and apply brakes if it sees hazards ahead. We might want to set the average target speed to 51 km/h and then just monitor the values as it goes.

To do this, you change the method of the Activity. In our previous example, we called a factory method createWriteRequest which will create a request that has the activity type as write. To make it just record, use createRecordRequest. So let's record that the speed is now 51 km/h (note how the core inserts the current time with setInstant):

PhysicalEntity car = new PhysicalEntity();
Velocity v = new Velocity();
ValueObject vo = new ValueObject(RESOURCE.KILOMETERPERHOUR, 51);
vo.setInstant(new Date());
v.setGroundSpeed(vo);
car.setVelocity(v);

HttpClient httpClient =  new HttpClient();
Response resp = httpClient.sendPost(<server uri>, Factory.createRecordRequest(myIdentity, car));
				

PhysicalEntity car = new Device();
Velocity v = new Velocity();
ValueObject vo = new ValueObject(NS.UNIT + "KilometerPerHour", 51);
vo.setInstant(DateTime.Now);
v.setGroundSpeed(vo);
car.setVelocity(v);

HttpClient httpClient =  new HttpClient();
Task<Response> resp = httpClient.sendPost(<server uri>, RequestFactory.createRecordRequest(<my identity>, car));
				

car = new PhysicalEntity()
v = Velocity()
vo = new ValueObject(unit=RESOURCE.KILOMETERPERHOUR, value=51)
vo.setInstant(new Date())
v.setGroundSpeed(vo)
car.setVelocity(v)

httpClient =  new HttpClient()
response = client.sendPostWithRequest(server_uri = <server uri>, request_obj = Factory().createRecordRequest(myIdentity, car))
				

6.3. Obtaining feedback

6.3.1. Response data

Servers that adhere to the standard should respond to write requests with a message that repeats the content of the original request but with values filled in with those that match values actually set, possibly in the unit used by the server. So for instance if you set the speed of a car like this:

Velocity v = new Velocity();
ValueObject vo = new ValueObject(RESOURCE.KILOMETERPERHOUR, 51);
v.setGroundSpeed(vo);
					

v = Velocity()
vo = ValueObject(RESOURCE.KILOMETERPERHOUR, 51)
v.setGroundSpeed(vo)
					

Velocity v = new Velocity();
ValueObject vo = new ValueObject(NS.UNIT + "KilometerPerHour", 51);
v.setGroundSpeed(vo);
					

... the server may respond with an object set like this:

Velocity v = new Velocity();
ValueObject vo = new ValueObject(RESOURCE.METERPERSECOND, 10);
v.setGroundSpeed(vo);
					

v = new Velocity()
vo = new ValueObject(unit=RESOURCE.METERPERSECOND, value=10)
v.setGroundSpeed(vo)
					

Velocity v = new Velocity();
ValueObject vo = new ValueObject(NS.UNIT + "MeterPerSecond", 10);
v.setGroundSpeed(vo);
					

Now, 10 m/s is about 36 km/h. What that means is that you tried to set the speed to 51 km/h but it can currently only go 36 km/h. You can use this response data to set the current value in user interfaces, databases, etc. Smart API also makes unit conversion to the desired unit automatically so although the server sends m/s, you will see km/h if so desired.

6.3.2. Subscribing to changes

Taking values from a response is a good way to acknowledge that a command went through but that approach is often insufficient to actually track what is happening. First, many of the phenomenon the IoT systems measure are constantly changing. In our speed example, it is physically impossible for a solid body to reach a certain speed without accelerating first. So if you receive one value, is that the final one or just a snapshot of a trajectory?

Further, more often than not it is desirable to monitor some value passively instead of commanding or polling for it. For this purpose Smart API supports the concept of subscription. Many technical solutions can be used to implement the datastream, in IoT technologies include the pub/sub model of MQTT, the streams of HTTP Push, and the data through WebSockets. These can all be modeled with Smart API.

Subscribing to some Entity is very similar to reading data of that Entity. To subscribe, we create an Entity, feed it to an Activity and set the method of that activity to be subscribe. Here's an example on how to get notifications on the changes in an Entity that is identified by entityId:

Request r = Factory.createRequest(myIdentity);
Activity a = new Activity();
Entity e = new Entity(entityId);
a.addEntity(e);
a.setMethod(RESOURCE.SUBSCRIBE);
r.addActivity(a);
					

r = RequestFactory().create(myIdentity)
a = Activity()
e = Entity(entityId)
a.addEntity(e)
a.setMethod(RESOURCE.SUBSCRIBE)
r.addActivity(a)
					

Request* r = RequestFactory::create(myIdentity);
Activity* a = new Activity();
Entity* e = new Entity(entityId);
a->addEntity(e);
a->setMethod(QUrl(RESOURCE__SUBSCRIBE));
r->addActivity(a);
					

Just like in reading an object, when you subscribe to an object, leaving all other details of that object except the identifier means "everything". So that subscription means "send me notifications of any change in this entity".

If you want to subscribe to a specific property of an object, you fill in the property data of that object just like you would do with a read. Let's track the brightness changes of a lamp for instance:

Activity activity = request.newActivity();
activity.setMethod(RESOURCE.SUBSCRIBE);
Entity entity = activity.newEntity("http://acme.com/o/C35782");
ValueObject vo = entity.newValueObject("http://acme.com/o/C35782-lamp5-br");
vo.setUnit(RESOURCE.PERCENT);
					

activity = request.newActivity()
activity.setMethod(RESOURCE.SUBSCRIBE)
entity = activity.newEntity("http://acme.com/o/C35782")
vo = entity.newValueObject("http://acme.com/o/C35782-lamp5-br")
vo.setUnit(RESOURCE.PERCENT)
					

What the code above means is that we'd like to receive notifications of the changes in lamp http://acme.com/o/C35782 and specifically its property brightness which is identified as http://acme.com/o/C35782-lamp5-br. We'd prefer to receive the data is percentages of max.

Chapter 7. Storing values into Variants

7.1. Using and constructing variants

The Variant is a wrapper class that can carry multiple types of values. The value stored into a Variant can be a string, integer, double, boolean, date, time, or another object or a list of objects. However, if you program with a strongly typed language, it is noteworthy to notice that a Variant itself is not an Obj. The purpose of a Variant is to offer a convenient method to store any type of value to various properties. The value can be set either in the constructor of the Variant or with a setter. The value can be fetched back with a getter. Various types of getters are provided for automatic casting into types.

Below is a set of samples on how to create a Variant and assign it a value in the constructor.

Creating a new Variant from string:

Variant variant = new Variant("Test variant string");
				

Creating a new Variant from double:

Variant variant = new Variant(24.1);
				

Creating a new Variant from Date:

Variant variant = new Variant(Tools.stringToDate("2017-08-25T12:24:11"));
				

Creating a new Variant from Date with current date (and time):

Variant variant = new Variant(new Date());
				

Creating a new Variant from Time:

Variant variant = new Variant(Time.valueOf("08:00:00"));
				

Creating a new Variant from Duration of 1 day and 6 hours:

// using Tools
Variant variant = new Variant(Tools.toDuration("P1DT6H"));
// or using Factory
Variant variant = new Variant(Factory.createDuration(0, 0, 1, 6, 0, 0));
				

Creating a new Variant from List:

List list = List.getDefaultList();
list.addItems(1);
list.addItems(2);
Variant variant = new Variant(list);
				

Creating a new Variant from Obj:

Device device = <some device object>;
Variant variant = new Variant(device);
				

7.2. Getters

Getters give the contained value back, casted in the desired format. If the casting cannot be made, the Variant returns and empty value (a null if supported). For safe programming, the Variant offers methods to test whether casting is possible. The examples below show how this can be used.

Get value of a Variant

Variant variant = <some variant>;
if ( variant.isString() ) {
	String value = variant.asString();
	// .. do something ..
}
if ( variant.isInteger() ) {
	Integer value = variant.asInt();
	// .. do something ..
}
if ( variant.isDouble() ) {
	Double value = variant.asDouble();
	// .. do something ..
}
if ( variant.isFloat() ) {
	Float value = variant.asFloat();
	// .. do something ..
}
if ( variant.isBoolean() ) {
	Boolean value = variant.asBoolean();
	// .. do something ..
}
if ( variant.isDate() ) {
	Date value = variant.asDate();
	// .. do something ..
}
if ( variant.isTime() ) {
	Time value = variant.asTime();
	// .. do something ..
}
if ( variant.isDuration() ) {
	Duration value = variant.asDuration();
	// .. do something ..
}
if ( variant.isList() ) {
	List value = variant.asList();
	// .. do something ..
}
if ( variant.isObj() ) {
	Obj value = variant.asObj();
	// .. do something ..
}				
				

Get value of a Variant as string (assuming this Variant contains a string value)

Variant variant = <some variant>;
String value = variant.getAsString();
// or
String value = variant.toString();
				

7.3. Setters

Variant object automatically sets the type when the value is set. Note that a Variant can only have one value and type at a time.

The following examples set the value of a Variant with values of different types. The resulting Variant eventually carries the value and type of the last setter call.

Variant variant = <some variant>;
variant.set(4);
variant.set("Hello");
variant.set(new Date());
				

7.4. Comparing Variants

Two Variants are defined to be equal when they have the same value and type.

Comparing two Variant:

Variant variant = <some variant>;
Variant otherVariant = <some other variant>;
if ( variant.hasSameValue(otherVariant) ) {
    // .. do something ..
}
				

Chapter 8. Lists and dictionaries

8.1. List types

In most applications with data, lists is something you'll be handling a lot. In essence all large datasets are processed as lists at some point in time. Not surprisingly, Smart API also has comprehensive list support.

The challenge with lists is that there are many possible ways to represent them. They can be linked, indexed, ordered, etc. Each method of representing a list has its pros and cons. One may be fast but non-ordered, the other ordered but large to transfer. What is the most suitable format is largely dependent on application and developer choice.

Smart API offers the following list types:

  • LinkedList
  • OrderedList
  • ItemizedList
  • NudeList

LinkedList stores serialized data items by linking the previous item directly to the next. This creates a hierarchical data structure where the depth of recursion needed to reach the final item is the same as the number of items in the list. LinkedList has the benefit of having a serialization that is 100% compliant with the RDF standard. The downside is poor performance in most platforms, both in terms of processing time and speed, and the risk of overflowing the call stack due to deep recursion. You can use this type of list for small lists containing less than 1000 items. Using a LinkedList for bigger lists results in delays due to slow serialization and parsing, and may fail due overflow caused by the depth of the data hierarchy.

OrderedList stores serialized data items in an indexed array. Because it does not create a deep hierarchical structure, it performs much better with large datasets than LinkedList. OrderedList is recommended to be used with lists that need to be ordered and contain more than 1000 but less than 100000 items.

ItemizedList stores serialized data items in an array. The structure is similar with the OrderedList but the items in the ItemizedList do not have an index number. ItemizedList is recommended to be used with datasets where the order of the items does not matter, and they should work fine even when the dataset is large.

NudeList stores serialized data items in a custom performance-optimized non-semantic format (based on JSON). Because most programming languages and libraries have binary optimized code for processing JSON a NudeList is by far the fastest of the implementations and consumes the least amount of memory. It is optimal for large amounts of data where the data is just literal values (i.e. ints, doubles, dates, strings). The downside of this list is that is performs badly when transferring non-literal values i.e. full objects. NudeList is the recommended format when the data is not objects (just literals), and you need to process efficiently a large amount of data (up to several million entries).

To simplify development, all lists provide an identical API. This essentially means that you can change list type "on the fly" without modifications to the rest of the code. The recipient of data automatically detects the incoming list type and parses it accordingly. When you do change the list type, what you'll see is differences in performance (speed and memory use) and in the size and format of the data that is transferred over network.

8.2. List processing performance

Serialization and parsing speed of various list types can vary considerably. The help in selecting a suitable list type for your application, below are some benchmark measurements done with OpenSuse Leap 42.2 (Quad Core 2.50GHz, 8GB RAM).

In serialization the linked list is the slowest (often by a significant margin). In the benchmark test, serializing of 1000 Evaluation items took ~0.8 seconds for LinkedList, ~0.1 seconds for OrderedList and ItemizedList, and ~0.05 seconds for a NudeList. Increasing the list size further causes LinkedList to take a lot more time compared to other lists and the difference is very significant. Serialization of 50000 items took ~13 minutes for LinkedList, ~3 seconds for OrderedList, ~2 seconds for ItemizedList, and ~0.5 seconds for NudeList. Serialization of 200000 items took ~20 seconds for OrderedList and ~1 second for NudeList. When list size grows even more, the highly structured formats may encounter trouble with memory use and cause programs to crash due to out of memory errors. However, the NudeList still performs just fine and is fast (processing time less than 5 seconds with several millions of items).

If you are working with very large datasets, you should try to minimize repeating data in the list. The solution to the problem offered by Smart API is the so called BaseObject. This object stores the common values that would otherwise be added into the list entries. More on the use of BaseObjects later in this chapter.

To further improve performance, you should try to remove data that could be represented otherwise or implicitly. For instance in timeseries, if the timestamps of values have an equal spacing, there is no need to attach a timestamp to each value. Instead, Smart API timeseries lists represent that spacing with the time step property.

8.3. Creating lists

To create a list of a particular type, simply call its constructor.

Creating a LinkedList:

List list = new LinkedList();
// or
List list = List.getDefaultList();
				

Creating an OrderedList:

List list = new OrderedList();
				

Creating an ItemizedList:

List list = new ItemizedList();
				

Creating a NudeList:

List list = new NudeList();
				

8.4. The list BaseObject

List items may have common properties that are same for each of the items. A standard serialization would include these common properties into each of the list items, basically just duplicating data. In such case, the process can be made more efficient by using the so called BaseObject that carries the values of these common properties. The list then becomes much more compressed and faster to send and process. The items on the actual list then only include properties that differ from an item to another, the common static values are in the BaseObject only.

When Smart API parses the list back into objects, the data in the BaseObject is returned back to the list items themselves. When you read the items from such a parsed list with for example the get(index) method, also the common properties are returned as if they were stored in every item all along.

Let's take an example. In the example below, the list items are of type ValueObject. Each item has its own value but each of the items in the list has the same quantity and unit.

Here's how to create a ValueObject and set it as a base to a list. Note how the value is left empty and only the quantity and unit are filled in.

List list = <some list>;
// ...
ValueObject baseObject = new ValueObject();
baseObject.setQuantity(RESOURCE.ENERGYANDWORK);
baseObject.setUnit(RESOURCE.KILOWATTHOUR);
list.setBaseObject(baseObject);
				

Now, the concept of creating the BaseObject is simple. Leave the value field empty, fill in the rest. But what is the value field i.e. the property that carries the value? When you use a BaseObject, Smart API automatically assumes that the property is rdf:value. This means that if you just make a BaseObject and then let the library give values to it from the list, each item in the resulting list will have the rdf:value property.

What if you don't want to store values rdf:value? What if your values in the list are the speeds of a vehicle and you'd like the property to be smartapi:groundSpeed instead of rdf:value? To do this, you need to define the field with the setBaseObjectProperty like so:

List list = <some list>;
// ...
Velocity baseObject = new Velocity();
list.setBaseObject(baseObject);
list.setBaseObjectProperty(PROPERTY.GROUNDSPEED);
				

The example above would create a list of Velocity objects where the property smartapi:groundSpeed is set to the values found in the list.

8.5. Adding items to a list

Items of a list can be primitives or objects. You can mix and match them, the list parsers will automatically recognize the types and reconstruct the lists to carry objects of the correct type. For clarity, though not mandated, it is however recommended to keep all items of a list of the same type.

Adding primitive items to a list:

List list = <some list>;
// ...
// add string
list.addItems("Test item");
// or integer
list.addItems(5);
// or double
list.addItems(3.4);
// or float
list.addItems((float)3.4);
// or boolean
list.addItems(true);
// or Date
list.addItems(Tools.stringToDate("2017-02-15T12:00:00+03:00"));
// or Time
list.addItems(Time.valueOf("08:00:00"));
// or Duration
list.addItems(Tools.toDuration("P1DT6H"));
				

Adding objects to a list:

List list = <some list>;
// ...
// create object
ValueObject item = new ValueObject();
item.setValue("Test string value");
// add to list
list.addItems(item);
				

Combining two lists:

List list = <some list>;
List otherList = <some other list>;
// ...
// add all items from other list to this list
list.addItems(otherList);
				

8.6. Reading items from lists

Reading items from a list (smartapi.rdf.List) should be done using the get(index) method, as it takes into account the BaseObject and returns a complete object with also the common properties.

Get first item of the list:

List list = <some list>;
// ...
if ( list.size > 0 ) {
    Obj item = (Obj)list.get(0);
}
				

Looping through a list of ValueObjects:

List list = <some list>;
// ...
for ( int i = 0; i < list.size(); i++ ) {
	// get current item
	ValueObject item = (ValueObject)list.get(i);
	// handle this item
	//..
}
				

8.7. Timeseries

Timeseries are an essential part of the Smart API data model as in typical applications the majority of list data is some measurements collected over a given time period. To optimize the presentation of data, TimeSeries class provides methods for directly managing the items of a timeseries and for setting the BaseObject when needed. If the list type for the timeseries is not explicitly set, OrderedList is used by default.

For huge Big Data lists, it is recommended to optimize the list to improve performance. This means that you should use a NudeList as the format and put all the common data into a BaseObject. Further optimization can be achieved by removing explicit timestamps and defining timespan with a step variable. The step defines the timetep between the entries and therefore the timestamps can be reconstructed if you define a start as a datetime for for the dataset. If you issue getListItem(index) for such a list, the list will automatically calculate the correct timestamp and the object structure and return a complete object.

To add items into a TimeSeries, you can use the addListItem method to add a single item or the setList method to set all items in the TimeSeries to match the list:

TimeSeries timeSeries = <some time series>;
List list = <some list>;
// ...
timeSeries.setList(list);
				

To iterate through the values in the TimeSeries, you can either extract the whole list with getList and manipulate that or use the dedicated shortcut methods to access the list items as shown in the example below:

TimeSeries timeSeries = <some time series>;
...
for ( int i = 0; i < timeSeries.getListSize(); i++ ) {
	// assuming that the base object is ValueObject
	ValueObject item = (ValueObject)timeSeries.getListItem(i);
	String timeStamp = Tools.dateToString(item.getInstant());
	String quantity = item.getQuantity();
	String unit = item.getUnit();
	Variant value = item.getValue();
}
				

If you want to change the list type used by the TimeSeries from the default, you need to create a new list instance and replace the existing list in the TimeSeries with the new one right after instantiation.

The following example uses an ItemizedList for list type and a BaseObject to shorten the serialization of the data:

TimeSeries timeSeries = new TimeSeries();
...
// set list type
timeSeries.setList(new ItemizedList());

// create and set base object
ValueObject baseObject = new ValueObject();
baseObject.setQuantity(RESOURCE.ENERGYANDWORK);
baseObject.setUnit(RESOURCE.KILOWATTHOUR);
timeSeries.setBaseObject(baseObject);

// add three items with different time stamps and values
Date now = new Date();
Calendar calendar = Calendar.getInstance();
calendar.setTime(now);
for ( int i = 0; i < 3; i++ ) {
	ValueObject vo = new ValueObject();
	vo.clearTypes();
	vo.setValue(i);
	vo.setInstant(calendar.getTime());
	calendar.add(Calendar.DATE, 1);
	timeSeries.addListItem(vo);
}
				

In the example below, the default list implementation is replaced with a NudeList. Further data optimization is done by defining a BaseObject and describing the list with a start datetime and time step to remove individual timestamps:

TimeSeries timeSeries = new TimeSeries();
// set list type
timeSeries.setList(new NudeList());

// create and set base object
Evaluation baseObject = new Evaluation();
baseObject.setQuantity(RESOURCE.ENERGYANDWORK);
baseObject.setUnit(RESOURCE.KILOWATTHOUR);
timeSeries.setBaseObject(baseObject);

// add start datetime for timeseries
timeSeries.setTemporalContext(new TemporalContext(Tools.stringToDate("2017-08-17T15:30:50.000")));

// add timestep (1 second interval between data items)
timeSeries.setTimeStep("PT1S");

// add three items with different values
for ( int i = 0; i < 3; i++ ) {
	timeSeries.addListItem(i);
}
				

8.8. Dictionaries with the Map object

By design, each Smart API object behaves like a map/dictionary. The properties are map keys and the values map values. This is why there is rarely a need for a dedicated map when you handle standard objects. The convention is very similar to for instance JavaScript where lists have a type (the JS Array) but there is no separate dictionary construct.

In some applications it may however be desirable to have something like a dedicated map, an object that explicitly says this is a map and needs to be a map. This is what the Map object is for. A Map is just like any other object, you fill it, attach it to other objects, serialize it and parse it. But the type information is retained so that when you send a Map to some other system, they will receive a Map, typecasted to that particular type.

The interface offered by a Map resembles that of map/dictionary objects found in many programming languages. To insert a value, you call insert (as opposed to add) and to retrieve a value you call value (as opposed to get). The values are set in specific properties called map entries.

Here's an example on handling a Map:

smartapi.model.Map myMap = new smartapi.model.Map();
myMap.insert("myIntegerValue", 27);
myMap.insert("myDoubleValue", 1.267);
myMap.insert("myStringValue", "The slick fox");

double myDouble = myMap.value("myDoubleValue").asDouble();
				

8.9. Further list and dictionary examples

Creating and printing a timeseries list:

// create time series
TimeSeries timeSeries = new TimeSeries();

// set list type
timeSeries.setList(new OrderedList());

// create and set base object
ValueObject baseObject = new ValueObject();
baseObject.setQuantity(RESOURCE.ENERGYANDWORK);
baseObject.setUnit(RESOURCE.KILOWATTHOUR);
timeSeries.setBaseObject(baseObject);

// add three items with different time stamps and values
Date now = new Date();
Calendar calendar = Calendar.getInstance();
calendar.setTime(now);
for ( int i = 0; i < 3; i++ ) {
	ValueObject vo = new ValueObject();
	vo.clearTypes();
	vo.setValue(i);
	vo.setInstant(calendar.getTime());
	calendar.add(Calendar.DATE, 1);
	timeSeries.addListItem(vo);
}

// print time series items
for ( int i = 0; i < timeSeries.getListSize(); i++ ) {
	// get current item
	ValueObject item = (ValueObject)timeSeries.getListItem(i);
	// print item data
	System.out.println("Item " + i + ":");
	System.out.println("  Timestamp: " + Tools.dateToString(item.getInstant()));
	System.out.println("  Quantity: " + NS.localName(item.getQuantity()));
	System.out.println("  Unit: " + NS.localName(item.getUnit()));
	System.out.println("  Value: " + item.getValueAsString() + "\n");
}
				

Creating and printing a list of random integers:

// create a list
List list = new OrderedList();

// add 10000 items (random integers from 0 to 100)
Random random = new Random();
for ( int i = 0; i < 10000; i++ ) {
	list.addItems(random.nextInt(100));
}

// print 5 first items in the list
for ( int i = 0; i < 5; i++ ) {
	// print item data
	System.out.println("Item " + i + ":");
	System.out.println("  Value: " + list.get(i) + "\n");
}
				

Creating and printing a TimeSeries list of 5 million integers:

// create timeseries
TimeSeries ts = new TimeSeries();

// create nude list for timeseries
List list = new NudeList();
ts.setList(list);

// create base object
Evaluation baseObject = new Evaluation();
baseObject.setQuantity(RESOURCE.ENERGYANDWORK);
baseObject.setUnit(RESOURCE.KILOWATTHOUR);
list.setBaseObject(baseObject);

// add start datetime for timeseries
ts.setTemporalContext(new TemporalContext(Tools.stringToDate("2017-08-17T12:00:00.000")));

// add timestep (1 hour interval between data items)
ts.setTimeStep("PT1H");

// add 5 000 000 items (random integers from 0 to 100)
Random random = new Random();
for ( int i = 0; i < 5000000; i++ ) {
	list.addItems(random.nextInt(100));
}

// print 3 first items in the list
for ( int i = 0; i < 3; i++ ) {
	// print item data
	Evaluation item = (Evaluation)ts.getListItem(i);
	System.out.println("Item " + i + ":");
	System.out.println("  Timestamp: " + Tools.dateToString(item.getInstant()));
	System.out.println("  Quantity: " + NS.localName(item.getQuantity()));
	System.out.println("  Unit: " + NS.localName(item.getUnit()));
	System.out.println("  Value: " + item.getValues().get(0).asString() + "\n");
}
				

Chapter 9. Requests and Responses

Requests and responses are the basic building blocks of network communication. As Smart API has been designed for communication, it also features two dedicated object types, a Request and a Response for this purpose.

The data used by Smart API contains many similar items as you would find in the underlying request/response oriented protocols such as HTTP. At this point you may ask why some of these are duplicated because the same functionality could be achieved with for instance HTTP headers. The reason for the design is that Smart API has been designed as a self-contained messaging solution. It should be as independent as possible from the underlying transport so that the transport can be changed during transit. So for instance you could send a serialization of a Smart API object over an email, which is then sent to another server over HTTP which then makes a notification over MQTT out of it. These transitions should be as easy to make as possible.

Smart API cannot fully function without some interaction with the underlying transport so that the servers can actually parse the data before it is processed by the Smart API library. In most cases you still need to include data on at least the following:

  • Content type
  • Content encoding
  • Content length

9.1. Identifying the recipients with Activities

A request is something you send to a server but data processing is something performed by some service (application) within the server which in turn has many features, one of which you might be interested in. When you send a request to some software, you need to be able to address a particular function of the software to find the correct recipient. In the traditional HTTP world, there is already a straightforward answer to this: the URL. The URL comprises an address that finds the server, a port that finds the application, and a path that determines the function. All good? No, not really.

The trouble with the above is that the addressing is dependent on the features of the protocol, namely HTTP in this case. But what if you need to use some other protocol? MQTT is one of the most commonly used protocols in IoT but it does not have a path, it has a topic. Or if, for some perverted reason, you communicate over email? Or in the worst case have to route between protocols from MQTT to HTTP to CoAP?

Smart API has been designed to be message oriented to avoid such troubles. What it means is that while transferring data will need some URI or topic, their number should be minimized for non-ambiguous access and the payloads should have enough information to address a particular feature of a Smart API enabled application. This addressing information is contained in the Activity objects.

An activity is something a service performs. You could for instance have a software that processes a set of numbers in one of two ways: it either multiplies or divides the numbers. So you'd have two Activities: multiply and divide. An activity has an identifier and that identifier tells to which Activity the incoming data should be passed to.

The Activity is a mandatory component in each Request. In services where there is only one feature available one single Activity should still be included in the Request. In this case that Activity can remain without an ID, it would be picked by default.

So if we would send an object that represents the data of a waste basket to a service that calculates the estimated heat energy obtainable from burning the waste in the basket, we would create a request like this:

import smartapi.factory.Factory;
import smartapi.model.Activity;
import smartapi.model.PhysicalEntity;

PhysicalEntity basket = new PhysicalEntity("http://www.cityofcity.org/wastemgmt/basket_24");
Request request = null;
try {
	request = Factory.createRequest("http://www.cityofcity.org/wastemgmtservices/");
} catch ( Exception e ) {
	e.printStackTrace();
}
Activity a = new Activity("http://www.wasteanalysiscorp.com/energyestimation/");
a.setMethod(RESOURCE.READ);
a.addEntity(basket);
request.addActivity(a);
				

The data of that basket would be now sent to an activity on the server that the server recognizes as http://www.wasteanalysiscorp.com/energyestimation/.

What if we'd like to send the data to two activities simultaneously? Let's say we have the basket data and in addition to the energy from burning, we'd like to get an estimate how much harmful gases and other pollutants the burning process might produce. In this case the request would be formed like this:

import smartapi.factory.Factory;
import smartapi.model.Activity;
import smartapi.model.PhysicalEntity;

PhysicalEntity basket = new PhysicalEntity("http://www.cityofcity.org/wastemgmt/basket_24");
Request request = null;
try {
	request = Factory.createRequest("http://www.cityofcity.org/wastemgmtservices/");
} catch ( Exception e ) {
	e.printStackTrace();
}

Activity energy_activity = new Activity("http://www.wasteanalysiscorp.com/energyestimation/");
energy_activity.setMethod(RESOURCE.READ);
Activity pollutant_activity = new Activity("http://www.wasteanalysiscorp.com/pollutantestimation/");
pollutant_activity.setMethod(RESOURCE.READ);

energy_activity.addEntity(basket);
pollutant_activity.addEntity(basket);
request.addActivity(energy_activity);
request.addActivity(pollutant_activity);
				

9.2. Object oriented requests

Because Smart API is an object transfer library, pretty much all requests you do with it are object oriented. What it means is that you create some object, attach it to a request and receive a response with data for that object.

Object oriented calls are to a most part symmetric when it comes to the structure of a Request and a Response. So when you send a Request with an Activity A and and Entity E, you should expect to get back a Response that has Activity A and Entity E in it, possibly with values filled in.

This symmetrical nature has an important meaning in control and measurement applications related to Internet of Things. The objects carry the state information of physical things and the results of commands. Let's for example assume that we want to set the value of an actuator that controls rotation speed to 110. For example so:

PhysicalEntity actuator = new PhysicalEntity("http://www.control.io/actuators/C1234");
actuator.add("http://www.control.io/actuators/C1234/rotationspeed", new ValueObject(110));
				

In the response, we receive an actuator that has been programmed as follows:

PhysicalEntity actuator = new PhysicalEntity("http://www.control.io/actuators/C1234");
actuator.add("http://www.control.io/actuators/C1234/rotationspeed", new ValueObject(83));
				

Notice the difference in rotation speed. We wanted to set it to 110 but due to whatever reason (a limiter, malfunction, etc) it was actually set to 83.

So as you can see, the objects function as convenient means to confirm what is happening in the remote system.

9.3. Procedural requests

9.3.1. Inputs and Outputs

While object oriented requests are great in most applications, there are situations where you'd just want to perform a good old remote procedure call. Give some inputs, receive some outputs. Smart API Input and Output objects do just that.

To use this functionality, you create either Input or Output objects, attached them to the activity that processes them, and then put the values that are inputs or outputs inside those objects.

Below is an example of a client http://me.com calling a service http://you.com. It passes values to an Activity that calculates the resulting color from a list of input colors. These values are passed in a Map which is added into the input as "callParams". Additionally there is a "callId" parameter.

Activity a = new Activity("http://you.com/colorcalculator");
Input i = new Input();
smartapi.model.Map map = new smartapi.model.Map();
map.insert("dataitems", "color");
map.insert("rgb", Arrays.asList("red", "green", "blue"));
i.addParameter("callId", "1234");
i.addParameter("callParams", map);
a.addInput(i);
a.turtlePrint();
					

Once the result is calculated http://you.com responds with an Output object. Here's what that could look like

Activity a = new Activity("http://you.com/colorcalculator");
Output o = new Output();
o.addParameter("callId", "1234");
o.addParameter("result", "white");
a.addOutput(o);
					

9.4. Asynchronous calls

Asynchronous calls are requests where a response with the eventual response values are not included in the direct response to the original call. This is not to say there is no response, there must always be one to avoid timeout errors, but the data in that response is only preliminary. The actual result is delivered later using some other messaging channel.

To be able to send delayed responses, there needs to be some additional communication channel to deliver the response. This channel is something that has a permanent socket connection that the original requested is guaranteed to listen to. Potential implementation options include

  • MQTT
  • WebSockets
  • Long polling HTTP calls
  • In case the requester is a service, a separate URI the server listens to for notifications

So the first step in the asynchronous calls is that the requester and responder agree on a feedback channel and open it before the first request.

Important

A common caveat is to open the feedback channel after making a request. This is incorrect. There is no guarantee that the response to a request reaches the recipient before the delayed data does. A service could first send a response to a HTTP call and then a notification over MQTT with the actual results. There is nothing that guarantees one reaches the recipient before the other. This is why the feedback channel must always be opened before the request to ensure reliable delivery of results.

To be able to for instance subscribe to an MQTT topic or to open a WebSocket, the caller first needs to know the details of this interface. This is where Smart API Registry comes along. A service can register multiple interfaces into the registry, one of them being the notification channel. This registration data gives the callers the necessary details (topics, ports, etc) to open a channel to listen to.

Finally, the channel needs to remain open and listen to any incoming data. Depending on the programming environment, this may require a separate thread to avoid blocking any other functionality of the call. For convenience, Smart API offers a helper class, the EventAgent that hides the nitty gritty of that process. With the EventAgent, all you need to do is provide a topic to listen to and a callback to call when a notification arrives. The agent handles parsing data for you and the connection management. Unless you tell it to do otherwise, it uses MQTT and the default broker for messages.

Below is an example for listening to results asynchronously using MQTT. The example first opens a connection to the broker for notifications, then calls a service with a request, asking for data for an entity. In this case the data is not ready, it could be for instance that it would be collected from a range of sensors after each request. Once the results are ready, the EventAgent receives the data from the service that was called, parses it into a Notification object and calls the notification callback function which then processes those results.

import smartapi.agents.EventAgent;
import smartapi.agents.EventCallback;
import smartapi.common.HttpClient;
import smartapi.common.Tools;
import smartapi.factory.Factory;
import smartapi.model.Entity;
import smartapi.model.Notification;
import smartapi.model.Request;
import smartapi.model.Response;

public class MyClient implements EventCallback {

	public MyClient() {}

	@Override
	public void onMessage(String publisherUri, String publisherType, 
			String topicUri, String topicType, String message)
	{
		try {
			Notification n = Tools.parseMqttNotification(message);
			Entity e = n.getFirstActivity().getEntities().get(0);
			// do something with the entity data here
		} catch ( Exception e ) {
			e.printStackTrace();
		}
	}
	
	public void sendRequest()
	{
		EventAgent agent = new EventAgent(this);
		agent.subscribe(null, null, "youcom", "calculations");
		HttpClient httpClient = new HttpClient();
		Entity entity = new Entity("http://you.com/devices/C1234");
		Request request = Factory.createReadRequest("http://me.com", entity);
		Response response = httpClient.sendPost(<server uri>, request);
		// note that the response_body in this case would just have some preliminary data
	}
}
			

Chapter 10. Parsing and serializing

Parsing and serializing are processes needed in conversion between objects and a representation that can be transferred over the network. Serializing turns objects into text, parsing turns text into objects. When you use Smart API's network tools, these are done automatically for you. In some integrations you may need to handle network traffic with the network framework that is provided in the system you are developing. In this case it is necessary to know how to perform those two tasks.

10.1. Parsing and serializing with the HttpClient

Client applications can use the Smart API HttpClient that automatically handles serialization process of the Request object into a string as well as parsing process of the received response string back into a Response object.

String serviceUri = "http://service.com/smartapi/v1.0e1.0/access/";
Request request = <some request>;
// ...
HttpClient client = new HttpClient();
Response response = client.sendPost(serviceUri, request);
				

10.2. Parsing and serializing with the Tools module

Smart API's Tools provides methods for parsing and serializing objects. Their use is straightforward: to get the serialized text, you call the appropriate serializer method with an object as argument to get the text and you call a parser metod with text to get an object.

Important

Note that due to the requirements of network traffic, the Request and Response objects have special serialize and parse methods that process additional info such as Content-Type headers. If you are dealing with these two object types, please use the parseRequest, parseResponse, serializeRequest and serializeResponse methods dedicated to them.

10.2.1. Handling Requests and Responses

Incoming messages can be parsed into objects using the parseRequest and parseResponse methods. In addition to the message payload, they take as input the Content-Type header value. You'll find this in the HTTP headers of the data you received.

Request request = Tools.parseRequest(message, contentType);
					

Similarly for Responses:

Response response = Tools.parseResponse(message, contentType);
					

When you serialize Requests and Responses, you'll need to keep track of the content type that is generated. The methods in the Tools module will return this to you when serialization is done. Add this value into the HTTP headers when you send the payload.

Important

It is important to keep track of the content type when you do network traffic, the recipient will need this information to be able to correctly interpret the message. Do not hardcode this value as it may change depending on whether the data is encrypted or uses a different character encoding.

Here's how to correctly serialize a Request object into a string and a content-type value:

Request request = <some request>;
ImmutablePair<String, String> serialized = Tools.serializeRequest(request);
String message = serialized.getLeft();
String contentType = serialized.getRight();
					

Similarly, to serialize a Response object into a string and a content-type value:

Response response = <some response>;
ImmutablePair<String, String> serialized = Tools.serializeResponse(response);
String message = serialized.getLeft();
String contentType = serialized.getRight();
					

10.2.2. Converting other object types

In the majority of cases you you should not have a need to serialize and parse any other types of objects than Requests and Responses. All others will in most cases be contained within those two and get converted as a part of the message. However, if there is a need to do such conversions, the Tools module offers methods to do so as follows:

Serializing a Device object into a string (using default serialization format):

Device device = <some device>;
String serialized = Tools.toString(device);
					

Parsing a string into a Device object:

String serialized = <rdf string>;
Device device = null;
Obj obj = Tools.toObj(serialized);
if ( obj instanceof Device ) {
	device = (Device)obj;
}
					

As RDF supports many serializations, Smart API library will choose one for you when you do the serialization. On the receiving end this serialization is recognized automatically. You also have the option to select the serialization and override the automatic selection, in case you for instance communicate with a system that only supports some given format.

Device device = <some device>;
String serialized = Tools.toString(device, SERIALIZATION.JSON_LD);
					

10.2.3. Supported serialization formats

Smart API supports a range of RDF erialization formats. The selection values are predefined in the library code and are as follows.

  • SERIALIZATION.TURTLE
  • SERIALIZATION.JSON_LD
  • SERIALIZATION.RDF_XML
  • SERIALIZATION.N_TRIPLE
  • SERIALIZATION.RDF_JSON
  • SERIALIZATION.RDF_XML_ABBREV

Chapter 11. Notifications and events

11.1. The publish/subscribe paradigm and notifications

Sometimes communication between entities has asynchronous nature. This means that the response to a request is not immediately available through the same channel the request was made. The reasons for this could be many, maybe some calculation takes a long time to finish, or a change in a state of a sensor needs to be first confirmed. Whetever the reason, such cases are difficult for the traditional request/response calls. It is typical that a response must be received within a set timeframe, often 30 seconds or less, otherwise the call fails with a timeout.

There are also cases where no specific requester can be identified. For instance you might have a sensor that records values and then sends those values to whoever is interested in them. The recipients don't know when new values are available so it makes no sense for them to constantly poll for the values with requests. The sensor could send the value to each recipient with a request, but this also is often not desired. The sensor would need to store the addresses of all recipients and make sure the calls go through firewalls. Usually sensors have low processing power and may prefer to use it sparingly in order to be able to operate on battery for a longer time. Because the sensor gets no additional value from getting a response (after all it is supposed to send, not receive), sending the values using a request/response model is just a waste of resources and an additional headache. A much preferred solution is to send notifications which expect no response. Fire and forget, that's it.

For these types of scenarios, Smart API provides publish/subscribe functionality based on the MQTT protocol. MQTT works through a broker: all parties register themselves to the broker and when something happens (new reading, calculation results ready, etc), a notification is sent through the broker. The broker remains online all the time and has no timeouting requests to fulfill. And as long as everyone can connect to the broker, firewalls are happy.

The way MQTT works is that those who wish to receive data register at the broker with a topic. When someone sends (publishes) data that fits the topic, the broker will forward the data to the subscriber(s).

11.2. Subscribing to notifications

11.2.1. Topics

Subscription to notifications is done based on topics. Smart API standardizes a certain topic format for easier access. The topic is constructed from four parts: sender identifier, sender type, topic identifier, and topic type. If some part of the topic is not defined, it is given an MQTT wildcard (+). This results in a match for that part for all messages, i.e., in receiving messages regardless of what that topic part is in the published message.

11.2.2. Listening to topics and callbacks

To simplify working with notifications, Smart API offers a helper class, the EventAgent. It handles all the details about opening broker connections and parsing the incoming data. All you need to do is have a callback function for the agent to call when something arrives.

In Java, callback functionality is achieved with the help of EventCallback interface. Implement that interface in a class and pass it to EventAgent. It will take care of the rest.

First, define your class to implement EventCallback:

public class MyClass implements EventCallback {
..
					

Then, define an onMessage method in MyClass:

public void onMessage(String publisherUri, String publisherType, String topicUri, String topicType, Notification n)
{
	if (notification != null) {
		// Perform whatever action you want with the notification
	}
}
					

Finally, subscribe to, for instance, state changes of a sensor:

// identifier of the publisher that we subscribe to
String notifierIdentifier = "http://acme.com/service/Cnotifier";
// identifier of the topic which is here identifier of a sensor
String sensorIdentifier = "http://acme.com/sensor/Cabc123";

// create event agent and give this as the callback object (must implement EventCallback) 
EventAgent agent = new EventAgent("TestSubscriber", this);

// subscribe to receive notification from the <notifierIdentifier> related to <sensorIdentifier>
agent.subscribe(notifierIdentifier, null, sensorIdentifier, null);
					

By default the EventAgent will use the main message broker of Smart API for relaying messages. You can change this and other settings of the agent if so desired. See API docs for more details.

11.3. Publishing events

11.3.1. Publishing notifications

Similarly to subscriptions, publishing is done based on topics. If some part of the topic is not applicable, null argument may be given for that part, and it will match to all subscriptions.

Notifying is a straighforward process: create a Notification object and then pass that on to the EventAgent in the publish method.

Here's an example of publishing notifications for sensor readings:

String myIdentifier = "http://acme.com/service/Cnotifier";
String sensorIdentifier = "http://acme.com/sensor/Cabc123";

EventAgent agent = new EventAgent((String)null, "TestNotifier");

Notification n;
try {
	n = Factory.createNotification(myIdentifier);
} catch ( Exception e ) {
	e.printStackTrace();
	return;
}
Activity a = new Activity();
Entity e = new Entity(sensorIdentifier);
ValueObject vo = new ValueObject();

a.addEntity(e);
n.setActivity(a);
vo.setQuantity(RESOURCE.POWER);
vo.setUnit(RESOURCE.WATT);
vo.setValue(new Variant(1012));
e.addValueObject(vo);
agent.publish(myIdentifier, RESOURCE.SERVICE, sensorIdentifier, RESOURCE.DEVICE, n);
					

Chapter 12. Unit conversion

When values are represented with Smart API ValueObjects that carry proper quantity and unit definitions, this brings one important benefit: the automatic conversion on values. Smart API can download unit conversion tables, including rapidly changing values such as currency rates, from the Internet and automatically convert values to the desired units as data is processed.

Performing unit conversion is simple: When you have a ValueObject, call its getValue method with the desired unit. That's it. If a conversion rate cannot be found, the resulting value will be null.

A couple of examples:

// original ValueObject with data in meters
ValueObject length = new ValueObject(RESOURCE.LENGTH, RESOURCE.METER, 2.3);
Variant lengthInKilometers = length.getValue(RESOURCE.KILOMETER);
			

// original data with value in celsius degrees
ValueObject temperature = new ValueObject(RESOURCE.TEMPERATURE, RESOURCE.DEGREECELSIUS, 28.4);
Variant temperatureFahrenheit = temperature.getValue(RESOURCE.DEGREEFAHRENHEIT);
			

ValueObject price = new ValueObject(RESOURCE.CURRENCY, RESOURCE.EURO, 10.5);
Variant priceInPounds = price.getValue(RESOURCE.POUNDSTERLING);
			

Note

Currency related units are a special case in unit conversion. In contrast to general unit conversion, here the result depends on the exchange rate between the currencies at that very moment. Smart API gets the exchange rate from a class that implements CurrencyConverterInterface. By default, this class is FixerIOCurrencyConverter and it uses Fixer.io service API to fetch the rates. If you want to use some other source for currency conversion rates, you can implement your own converter and set it using the setCurrencyConverter method of the SmartAPI class.

Notice that some units are actually a combination of two units. This is often easily recognizedby the common something per something convention. For instance kilometers per hour, ounce per gallon, watt per square meter. Some of these, like kilometers per hour, are so commonly used that the combination itself is the common unit and that's it. But some are more custom in nature or might require a conversion rate that changes dynamically. The latter is true especially for currency related units, say euros per liter.

To enable conversion between custom and changing conversions, Smart API ValueObjects support setting up a so called secondary unit. This is the divider part of a combined unit. So for euros per liter, euro is the unit while liter is the secondary unit. If you tell these to the ValueObject, it can convert between such combined units properly. Like so:

// Converting from from e/kWh to pound sterling/MWh
ValueObject original = new ValueObject(RESOURCE.CURRENCY, RESOURCE.EURO, RESOURCE.ENERGYANDWORK, RESOURCE.KILOWATTHOUR, 0.216);
ValueObject converted = UnitConverter.convert(original, RESOURCE.POUNDSTERLING, RESOURCE.MEGAWATTHOUR);
			

Chapter 13. Security

13.1. Security features of Smart API

Information security is an integral part of the Smart API library. The features supplied in the library include for instance key handling, message signing, message encryption and a notary service. Now, many similar features can be found from the servers that you'd use to serve data using Smart API. For instance, we'd not only expect but actually very much recommend to use HTTPS as the carrier protocol for the data. In fact many of the security features of the library expect you to use those methods as they form the solid basis for many of the features of Smart API security.

But despite the servers having many features already in place, there are important reasons why an additional security layer is needed. While secure transmission can be done with HTTPS, that transmission must always terminate and be decrypted somewhere. Especially when multiple systems are integrated, that decryption takes place in various intermediaries, compromising the system by opening it for eavesdropping at these points. What Smart API offers is message level, end-to-end security. No matter how many "hops" of intermediaries there may be, it is guaranteed that your information stays safe and secure.

13.2. Key management

Smart API relies heavily on public key cryptography. Parties engaged in secure communication should both have a keypair with private and public parts available. The process of creating and distributing these keys is key management.

13.2.1. Generating keys

The first step to security is the generation of keypairs. Once generated, the private key part is stored to a local file. You should always keep this file protected and secret. The public key can be stored locally and uploaded to a keyserver. It is perfectly safe to do so as it is the public part that can be announced to anyone without compromising security. Other parties can fetch the key from the keyserver and use it to send data to you in encrypted format. It is recommended but not mandated to upload the public key. In case the public key is not uploaded to the keyserver, you can deliver it with some other means, say as an email attachement. Keys are stored as text in standard PEM format so any text transfer will do.

Here's how you would generate and store a keypair:

String myIdentifier = <my identifier>;
String privateKeyPath = <path for private key>;
String publicKeyServer = <public key server uri>;
// to get username and password, sign up at
// http://talk.smart-api.io/develop/clientworkbench
String username = <your SmartAPI user name>;
String password = <your SmartAPI password>;

try {
	Crypto.createAndSaveKeyPair(myIdentifier, privateKeyPath, publicKeyServer, username, password);	
} catch ( KeyAlreadyExistsOnServerException keyException ) {
	System.err.println("Key was already found at the server for the given identifier!");
	keyException.printStackTrace();
} catch (  UploadException uploadException ) {
	System.err.println("Exception while uploading public key to server!");
	uploadException.printStackTrace();
}
					

13.2.2. Uploading keys

You can choose to upload the public key to the keyserver directly when you generate the pair or perform the upload separately. Smart API naturally offers methods to do this also.

Important

Key upload are for public keys only. Never publish or otherwise expose your private keys.

To upload public key to a key server, do as follows:

String myIdentifier = <my identifier>
String publicKeyServer = <public key server uri>
String username = <your Smart API user name>
String password = <your Smart API password>

ImmutablePair<String,String> keys = Tools.createCryptoKeys();
String privateKey = keys.getLeft();
String publicKey = keys.getRight();
Crypto.uploadPublicKey(publicKey, myIdentifier, publicKeyServer, username, password);
					

13.2.3. Fetching public keys

Public keys are stored on a public key server and can be fetched from there by anyone. To fetch a key, just invoke the downloadPublicKey method provided in the Smart API library as shown below.

Here's how to fetch a public key:

String myIdentifier = <my identifier>;
String publicKeyServer = <public key server uri>;

PublicKey publicKey = Crypto.downloadPublicKey(myIdentifier, publicKeyServer);
					

Important

Because anyone with a valid usedname and password can upload keys into a keyserver, public key repositories are always susceptible to spooding the identifier i.e. claiming the keys are for someone they are not. To remedy this problem, Smart API provides three methods

  • Key fingerprints
  • A managed PKI and signed certificates as proofs of identity
  • The ability to use any X509 standard compliant external certificates

If you are building a system with any security requirements, you should these methods, at minimum confirming the key fingerprint with the communication party.

13.2.4. Revoking a public key

If the private key is suspected to be compromized, corresponding security measures should be taken and a new key pair has to be generated. Before you can store a new public key to the public key server, the old public key has to be cancelled, i.e. revoked.

To revoke a public key, do as follows:

String myIdentifier = <my identifier>;
String publicKeyServer = <public key server uri>;
String username = <your Smart API user name>;
String password = <your Smart API password>;

if ( !Crypto.revokePublicKey(myIdentifier, publicKeyServer, username, password) ) {
	System.err.println("Revoking public key failed!");
}
					

13.3. The Crypto Key Wallet

When you write code, it may be sometimes tedious to pass around the various keys and slow to load them from disk every time. The Crypto Key Wallet provides a storage for keys to solve this problem. Once the keys are loaded, they are used as default keys in signing and ecryption.

13.3.1. Loading keys to Crypto Key Wallet

Keys are loaded to the wallet with loadPublicKey and loadPrivateKey methods. Both methods store the keys into the wallet and for convenience also return the loaded key for immediate use.

To load a public key:

String keyPath = <path to key>;
PublicKey publicKey = null;

publicKey = CryptoKeyWallet.loadPublicKey(keyPath);
				

To load a private key:

String keyPath = <path to key>;
PrivateKey privateKey = null;

privateKey = CryptoKeyWallet.loadPrivateKey(keyPath);
				

13.3.2. Setting keys manually to Crypto Key Wallet

In addition to loading the keys from disk, you can set them from memory objects. This is done with the setPublicKey and setPrivateKey methods.

Set default public key:

PublicKey publicKey = <my public key>;

CryptoKeyWallet.setPublicKey(publicKey);
				

Set default private key:

PrivateKey privateKey = <my private key>;

CryptoKeyWallet.setPrivateKey(privateKey);
				

13.4. Signing objects

Signing an object ensures that the receiver can trust that the object was sent by the party that is defined as the sender or signer. Signing takes place with the private key of the signer. The signature can then be verified with the public key of the signer.

13.4.1. Sign

With the sign method you can sign any Smart API object. In some cases you may want to sign everything contained within and object or object graph, and in other cases you may only want to sign some smaller part. Signing always applies to the object it is invoked on, inluding any connected objects.

Here's an example on how to sign a Request object:

PrivateKey privateKey = <my private key>;
Request request = <some request>;
request.sign(privateKey);
					

To sign some object, in this case an Input, simply invoke sign on that object:

PrivateKey privateKey = <my private key>;
Request request = <some request>;
Activity activity = new Activity();
request.addActivity(activity);
Input input = new Input();
activity.addInput(input);
// ...
input.sign(privateKey);
					

Note that when serializing signed Request/Response or those Request/Response with signed component, you should use Tools.serializeRequest and Tools.serializeResponse methods, which will generate a MIME multipart message containing the message and signature in seperate parts.

13.4.2. Verify

When you receive a message or an object that is signed, you can verify the signature to be sure that the content is still unmodified and still the same as it was when signed.

Verifying the signature:

String publicKeyServer = <public key server uri>;
Response response = <some response>;
PublicKey senderPublicKey = Crypto.downloadPublicKey(response.getGeneratedBy().getIdentifierUri(), publicKeyServer);

for ( Activity activity : response.getActivities() ) {
	for ( Output output : activity.getOutputs() ) {
		if ( output.isSigned() ) {
			if ( output.verifySignature(senderPublicKey) ) {
			    // signature successfully verfied
			} else {
				System.err.println("Signature verification failed, do not trust the content!");
			}
		}
	}
}
					

13.5. Encrypting objects

Encrypting scrambles the object and ensures confidentiality of transfer. Smart API uses a combination of strong RSA public key cryptography and AES-256 message encryption to achieve this. The resulting content is an encrypted payload in text format that can be decrypted by the recipient using the private key.

13.5.1. Encrypt

To encrypt an object, you invoke the encrypt method on that object. As with signing, you can apply encrypt to any object.

Important

While it is possible to send and receive requests that are completely encypted with Smart API, the recommended convention is not to completely encrypt everything. Instead of encrypting the Request object, you should encrypt the Activity object within the Request. This way some bookkeeping items such as sender identity and timestamp remain in plaintext and are easier to record into logs. Because in the majority of applications even this part is protected by HTTPS in transit, this convention makes it easier to process the data.

Here's how to encrypt the whole Request object with public key:

String receiverIdentifier = <identifier uri of the receiver>;
String publicKeyServer = <public key server uri>;
PublicKey receiverPublicKey = Crypto.downloadPublicKey(receiverIdentifier, publicKeyServer);

Request request = <some request>;
request.encrypt(receiverPublicKey);
					

To encrypt just the Activity, do as follows:

String receiverIdentifier = <identifier uri of the receiver>;
String publicKeyServer = <public key server uri>;
PublicKey receiverPublicKey = Crypto.downloadPublicKey(receiverIdentifier, publicKeyServer);

Request request = <some request>;
Activity activity = new Activity();
request.addActivity(activity);
Input input = new Input();
activity.addInput(input);
activity.encrypt(receiverPublicKey);
					

13.5.2. Decrypt

When you receive a message, you can decrypt the parts that are encrypted. If the receiver follows the common convention of encrypting an Activity within a Request, you'll know by convention which object to decrypt. That said, Smart API messages always carry enough plaintext information to know which objects are encrypted and which are not. The isEncrypted method will simply tell you whether there is a need to perform decryption.

Decrypting a Response with private key:

Response response = <some response>;
PrivateKey privateKey = <my private key>;

if ( response.isReference() ) {
	if ( response.isEncrypted() && response.getEncryptionKeyType().equals(RESOURCE.PUBLICKEY) ) {
		response.decrypt(privateKey);
	}			
}
					

Decrypting all Activity objects inside the Response:

Response response = <some response>;
PrivateKey privateKey = <my private key>;

for ( Activity activity : response.getActivities() ) {
	if (activity.isEncrypted() && activity.getEncryptionKeyType().equals(RESOURCE.PUBLICKEY) ) {
		activity.decrypt(privateKey);
	}
}
					

13.5.3. Ensuring delivery, non-repudiation

Non-repudiation means that in case something has been performed or something exists, a party involved in the process cannot claim the opposite (i.e. that it did not happen or does not exist). The way this is done in Smart API is through a trusted third party, a notary. It is assumed that the notary never lies to either party. So if someone has assured to a notary that something exists, that party cannot go back again and claim that such assurance was never made and the notary is lying.

The notary can also be used to hold parts of messages or keys to the messages. This forces parties of communications to make such assurances to the notary. If data is sent between parties, it can be encrypted and stored at the notary. So unless you contact the notary and do what is told, you'll never get the key and will never be able to access the data you may have received. Such communication is readily included in the methods of Smart API objects. At the data sender end, the method secureKeyAndNotarize takes care of sending a key to the notary. Similarly, at the receiver end the confirmContentValidityAndFetchKey method checks content integrity and requests a key for it from the notary. Let's see how those two work in practice.

Sender end: sending key of the encrypted content to a notary:

String myIdentifier = <my identifier uri>;
String receiverIdentifier = <identifier uri of the receiver>;
String publicKeyServer = <public key server uri>;
PublicKey receiverPublicKey = Crypto.downloadPublicKey(receiverIdentifier, publicKeyServer);
String notaryUri = <notary server uri>;
PrivateKey myPrivateKey = <your private key>;

Response response = <some response>;
Activity activity = new Activity();
response.addActivity(activity);
Output output = new Output();
activity.addOutput(output);
//..
output.encryptAndNotarize(notaryUri, receiverPublicKey, myPrivateKey, myIdentifier);

ImmutablePair<String, String> serialized = Tools.serializeResponse(response);
String message = serialized.getLeft();
String contentType = serialized.getRight();
					

Receiver end: fetching key for the encrypted content from a notary:

String myIdentifier = <my identifier>;
Response response = <some response>;
PrivateKey myPrivateKey = <my private key>;

for ( Activity activity : response.getActivities() ) {
	for ( Output output : activity.getOutputs() ) {
		if ( output.isReference() ) {
			if ( output.isEncrypted() && output.getEncryptionKeyType().equals(RESOURCE.NOTARIZEDSESSIONKEY) ) {
				if ( output.confirmContentValidityAndFetchKey(myIdentifier) ) {
					output.decrypt(myPrivateKey);
				} else {
					System.err.println("Content hash does not match with the content or fetching key failed.");
				}
			}
		}
	}
}
					

13.6. Authentication and authorization

13.6.1. Authenticating with OAuth2

OAuth2 is a standard for access delegation, commonly used in websites that provide logins to users with access credentials from some other service. A typical example is a service to where you can login with your Facebook or Twitter username.

The idea with OAuth2 is that some service can delegate the access management to some other service, removing the need to have yet another username for that particular service. Typically OAuth2 is used for interactive services where there is a user who through an interactive session will during the authentication sequence enter a form to enter the credentials and if this is successful, will be then redirected to the actual service. However, OAuth2 can also be used for non-interactive, machine-to-machine authentication by following the same principles.

OAuth2 has two distinct processes that happen in sequence but should not be confused: authorization and authentication. The authentication part is familiar to everyone, this is where you give your username and password and if ok, get granted access. The authorization part is not for the user, it is for the service or software used by the user. So when you enter some service, it may ask something in the lines of "will you allow application X access your Y". So here the user grants access to certain functionality for reading or modifying data owned by the user.

In Smart API the authorization phase takes place before authentication. It is used to authorize some application and this way works to filter out unwanted applications (which could be say worms, data loggers, spammers, or just unlicensed software instances). To get access, the software first needs to send some access token. What that access token actually is is up to you. In many services it is common to have something like an application key or a developer key or similar. Needless to say, for this token should be long enough and random so that it cannot be guessed if you want this stage to be secure.

The key point to notice here is that this authorization stage is between your service and the client that connects. The request comes first to you, not the authentication server. Once the authorization stage has passed, your service should return a random authorization token. This token is sent to the user and the user uses this token to start authentication. This is the first level of security: in case you don't have the correct application key, you cannot even start guessing the passwords.

Once the authorization key is received, the client software then goes to the authentication server (not your service) and presents a username and password there (not to you). If these are correct, an access token is granted. At this point the authentication server can contact your service and check that the authorization token the client software has presented is correct. If it is not, no access token will be granted.

If all went well, now the client software has both an authorization token for itself and an authentication token for the user. The client should now place the authentication token into the header of each request it makes. The token should be placed in the Authorization header with a "Bearer prefix". So the header should look like this:

Authorization: Bearer <token_content>
					

When your service receives a token, it can contact the authentication service and ask whether the token is valid. If yes, the authentication server will return the authorization code for which it was granted. The service can then check whether such authorization code was given out and whether to grant further access.

On the client side, the Smart API OAuthAgent handles most of the traffic. Below is an example on how to use it. The server end is heavily dependent on your design and which checks are deemed necessary. For more information on that implementation, please refer to the separate OAuth server side programming guide.

import smartapi.agents.OAuthAgent;

public class OAuthSample {

	private static final String clientSecret = "test";
	private static final String serviceAddress = "http://acme.com:8080/service/";
	private static final String securityBrokerAddress = "http://127.0.0.1:3333/oauth2/authorize/";
	private static final String authorizeEndpoint = serviceAddress + "authorize";
	private static final String tokenEndpoint = securityBrokerAddress + "token";
	private static final String CLIENT_ID = "ClienttApp";

	public OAuthSample()
	{
	}

	public void runSample()
	{
		String authorizationCode = OAuthAgent.oauth2AuthorizeApplication(CLIENT_ID,
				"api", authorizeEndpoint);
		String authenticationToken = OAuthAgent.oauth2AuthenticateUser(authorizationCode, 
				CLIENT_ID, clientSecret, "", tokenEndpoint);
	}
	
	public static void main(String[] args)
	{
		OAuthSample sample = new OAuthSample();
		sample.runSample();
	}
}				
				

Important

When dealing with OAuth2, always pay close attention to where messages are actually sent: are they destined to your service or the authentication server and are they about authentication or authorization. Also note that because authorization is an integral part of the login procedure, the authentication server will refuse to grant access unless your service is able to confirm the authorization codes that were presented.

13.6.2. The Smart API PKI

The Smart API PKI (Public Key Infrastructure) is a PKI server that can be used to commonly sign certificates of objects with a trusted Certificate Authority (CA). In most cases organizations already have certificates issued by some commercial CA. However, in smaller projects, lab environments or closed networks, it may be preferable to use a PKI from where you can request certificates on demand.

Chapter 14. Using the Smart API Registry

The Smart API Registry is like a phonebook of things. It serves as a directory of devices, services, algorithms, etc that may want to connect to each other. The Registry only serves as a directory, it does not store, convert or relay any data. All it does is store information about things that might have data and offers search capabilities so that those things can be found by those who need them.

When you have created a Smart API capable service of some kind, it is recommended to register it and the details of data and controls available through it. As said, you never send any actual data to the Registry but the registration includes the ID's, interface addresses and data formats necessary for others to make a connection to your service and get that data.

14.1. Submitting objects to the registry

The easiest way to perform registration is with the Smart API RegistrationAgent. When you submit objects to the agent, it will extract the details needed from them, encode a message ready to be sent to the registry, and finally makes the registration submission.

The RegistrationAgent can be used to register entities, i.e., to add entity descriptions to the registry, and to deregister entities, i.e. remove entity descriptions from the registry. The Smart API Registry stores entity descriptions tagged with the identity of the registrant and degistration of a certain description can only be made by the same registrant that has registrated it.

14.1.1. Performing registration

Making a registration with the RegistrationAgent is straightforward. Create an instance of the agent, when add you devices and services as entities to it and call registrate. Here's an example:

// create entity description
Service service = <some service>;

String myIdentifier = <my identifier>;
RegistrationAgent registrationAgent = new RegistrationAgent(myIdentifier);
registrationAgent.addEntity(service);
Response response = registrationAgent.registrate();
// print out possible errors on the response
Tools.printErrors(response);
// get response status
if ( response.hasStatus() ) {
	Status status = response.getStatus();
	// check if the operation was successful
	if ( status.hasType(RESOURCE.ERROR) ) {
		System.out.println("Registration failed.");
	} else {
		System.out.println("Registration successful.");
	}
}
					

14.1.2. Removing objects from the registry

To remove an entity description from the registry, deregistration has to be made. Deregistration can only be made by the same registrant that has registered it.

Deregistering a service description:

// create entity description
Service service = <some service>;

String myIdentifier = <my identifier>;
RegistrationAgent registrationAgent = new RegistrationAgent(myIdentifier);
registrationAgent.addEntity(service);
Response response = registrationAgent.deRegistrate();
// print out possible errors on the response
Tools.printErrors(response);
// get response status
if ( response.hasStatus() ) {
	Status status = response.getStatus();
	// check if the operation was successful
	if ( status.hasType(RESOURCE.ERROR) ) {
		System.out.println("Deregistration failed.");
	} else {
		System.out.println("Deregistration successful.");
	}
}
					

14.1.3. Debugging the registration process

If you wish to see in a console what is sent to the registration service and what is in the response, you can enable RegistrationAgent's debug mode.

Enabling debug mode:

RegistrationAgent registrationAgent = new RegistrationAgent(<my identifier>);
registrationAgent.debugMode(true);
					

14.2. Searching the Smart API Registry

To browse the data in the Smart API Registry, you need to do a search. The most convenient way of searching is with the help of the SearchAgent. This agent can be used to search for registered entities, such as services or devices. The SearchAgent can be instantiated to perform custom search based on various parameters, or it can be used in a static way to simply make a quick search with commonly used parameters.

14.2.1. Basics of searching

To search with the SearchAgent, you instantiate the agent, add some search parameters execute the search.

Instantiating the agent:

SearchAgent agent = new SearchAgent(<my identifier>);
					

Executing the search:

// search for entities with defined parameters
ArrayList<Entity> entities = agent.search();
					

14.2.2. Search by type

To search by type, set the type for one entity to seek:

// all entities
agent.ofType(RESOURCE.ENTITY);
// ..or, for instance, all services
agent.ofType(RESOURCE.SERVICE);
// ..or, for instance, all devices
agent.ofType(RESOURCE.DEVICE);
					

You can also search for multiple types in the same query:

// all solar panels and hydro generators
String[] types = new String[] {RESOURCE.SOLARPANEL, RESOURCE.HYDROGENERATOR};
agent.anyOfTypes(types);
					

14.2.3. Searching by text content

To search by name (rdfs:label), use the ofName method:

// all entities that have exactly the name "Acme Weather Service"
agent.ofName("Acme Weather Service", true);
					

ofName is a wildcard search so to find items where only a part of the string matches, you use the same syntax:

// all entities that have string "weather" in their name
agent.ofName("weather");
					

You can also include multiple search strings in the same query:

// all entities that have string "weather" or "forecast" in their name
String[] searchStrings = new String[] {"weather", "forecast"};
agent.anyOfNames(searchStrings);
					

Searching by descriptive text (rdfs:comment) is also a typical search type and can be done as follows:

// all entities that have string "energy" in their description
agent.ofDescription("energy");
					

To search based on part of the identifier do as follows:

// all entities that have identifier that contain string "acme.com/service"
agent.ofId("acme.com/service");
					

14.2.4. Time-based search

Search entities that are (re)registered within a given time:

// all entities registered within 2 years
agent.yearsOldData(2);
// ..or all entities registered within 3 months
agent.monthsOldData(3);
// ..or all entities registered within 10 days
agent.daysOldData(10);
// ..or all entities registered within 2 hours
agent.hoursOldData(2);
// ..or all entities registered within 5 minutes
agent.minutesOldData(5);
					

14.2.5. Spatial search

Search entities within a given distance from some point:

// all entities within 3.5 kilometers from the coordinates 60.12 lat, 24.51 lon
agent.pointSearchArea(new Coordinates(60.12, 24.51), 3.5);
					

14.2.6. Search by distance and type

Search quickly for certain type of entities within a distance:

// search for services within 3 km radius from the given point that are registered within the last 5 days
String[] types = new String[] {RESOURCE.SERVICE};
ArrayList<Entity> entities = SearchAgent.searchByPointAndType(<my identifier>, 5, 60.181, 24.818, 3, types);
				

14.2.7. Search by name and type

Search quickly for certain type of entities with search strings for name:

// search for services with name containing string "weather" or "forecast" that are registered within the last 5 days
String[] keywords = new String[] {"weather", "forecast"};
String[] types = new String[] {RESOURCE.SERVICE};
ArrayList<Entity> entities = SearchAgent.searchByNameAndType(<my identifier>, 5, keywords, types);
				

14.2.8. Search by identifier

Search quickly based on identifier search string:

// search for entities that have identifier containing string "acme.com" that are registered within the last 5 days
ArrayList<Entity> entities = SearchAgent.searchById(<my identifier>, 5, "acme.com");
				

14.2.9. Search by description

Search quickly based on search string for description:

// search for entities with description containing string "forecast" that are registered within the last 5 days
ArrayList<Entity> entities = SearchAgent.searchByDescription(<my identifier>, 5, "forecast");
				

14.2.10. Search by exact identifier

Fetch quickly entity description based on the exact identifier:

// fetch entity description for entity "http://acme.com/services/Cweather"
Entity entity = SearchAgent.fetchBySmartAPIId(<my identifier>, "http://acme.com/services/Cweather");
				

14.2.11. Check if entity information has changed in the registry

Sometimes it is handy to check wether the information about some entity is still the same that you have received in an earlier search. To compare the entity you have received with the information on the registry, you can use hasChanged method.

Check quickly if an entity has been changed on the registry:

Entity entity = <entity found earlier on the registry>
boolean hasChanged = SearchAgent.hasChanged(<my identifier>, entity);
				

14.2.12. Debugging registry search

If you wish to see in a console what is sent to the registration service and what is in the response, when you do a search you can enable SearchAgent's debug mode.

Enabling debug mode:

SearchAgent searchAgent = new SearchAgent(<my identifier>);
searchAgent.debugMode(true);
					

Chapter 15. Sharing objects across systems

15.1. The purpose of sharing

One of the key objectives of Smart API is to enable effortless data sharing between different parties. The one that shares the data can decide how, when, and at what price the data is available. Sharing essentially means that you announce the availability of some resource for others i.e. "here is my device, it is available of weekdays for a price of one euro per day".

Sharing is accomplished with the SharingAgent which takes care of the announcement. To define the price of the resource, you attach an Offering object to the announcement. Similarly, other conditions for the availability of the data are described using an Availability object. Availability supports logical conditions and a range of predefined availability factors such as time of day, distance, and geographical area.

Important

Notice that Smart API only defines how availability for sharing is announced. It does not provide functionality on how availability is enforced. Enforcing access is something that is the role of the access control mechanisms of each system and therefore out of the scope of the Smart API library.

15.1.1. Submitting an object as shared

Sharing is achieved by describing Availability of the data and using the agent to associate the availability with the corresponding entity that provides the data. Examples for describing differend kind of availabilities are provided below.

This example defines a service http://acme.com/service/Csmartapi that is available in Spain within certain time of day on weekdays and weekend:

// identifiers
String myIdentity = "http://acme.com/service/Cregistrator";
String serviceIdentity = "http://acme.com/service/Csmartapi";
// create availability for the service
Availability availability = new Availability();
// available from now on for two weeks
Date now = new Date();
availability.addAvailability(now, Tools.add(now, 0, 0, 14, 0, 0, 0));
// available local time from 9 am to 2 pm and 4 pm to 8 pm (local time) on weekdays
availability.addAvailability(Factory.createTimeFromLocalTimeString("09:00:00"), Factory.createTimeFromLocalTimeString("14:00:00"), RESOURCE.WEEKDAY);
availability.addAvailability(Factory.createTimeFromLocalTimeString("16:00:00"), Factory.createTimeFromLocalTimeString("20:00:00"), RESOURCE.WEEKDAY);
// available local time from 11 am to 3 pm (local time) on weekends
availability.addAvailability(Factory.createTimeFromLocalTimeString("11:00:00"), Factory.createTimeFromLocalTimeString("15:00:00"), RESOURCE.WEEKEND);
// available in Spain
availability.addAvailability(new Address("Spain", null, null, null));

// share service with defined availability
SharingAgent sharingAgent = new SharingAgent(myIdentity);
Entity entity = sharingAgent.share(serviceIdentity, availability);
if ( entity == null ) {
    System.out.println("Sharing failed.");
} else {
    System.out.println("Sharing successful. ");
}
					

Here we announce that the service is as available on weekdays within 5 km from a center point:

// identifiers
String myIdentity = "http://acme.com/service/Cregistrator";
String serviceIdentity = "http://acme.com/service/Csmartapi";
// create availability for the service
Availability availability = new Availability();
// available  on weekdays
availability.addAvailability((Time)null, (Time)null, RESOURCE.WEEKDAY);
// available within 5 km from 60.123, 24.123
Ring ring = new Ring();
ring.setMaxRadiusInKilometers(5);
ring.setCoordinates(60.123, 24.123);
availability.addAvailability(ring);

// share service with defined availability
SharingAgent sharingAgent = new SharingAgent(myIdentity);
Entity entity = sharingAgent.share(serviceIdentity, availability);
if ( entity == null ) {
    System.out.println("Sharing failed.");
} else {
    System.out.println("Sharing successful. ");
}
					

15.1.2. Cancelling sharing

Removing the sharing of an entity means removing its availability from the registry. The Availability is always registered and removed as a whole, meaning that if the availability changes, it needs to be completely removed and then reregistered.

Removing an Availability:

boolean successful = sharingAgent.removeSharing(serviceIdentity);
if ( !successful ) {
    System.out.println("Removing sharing failed.");
} else {
    System.out.println("Removing sharing successful.");
}
					

Chapter 16. Making offers and recording transactions

16.1. Introduction

Transactions are cryptographically secure proofs of actions that have been performed by some systems to some other systems. You could consider them as sort of receipts that are the basis of bookkeeping and also invoicing. Transactions support signing, which together with the objects called Offerings form the basis for contracts. So in a nutshell: if you want to make money with your data, transactions are the tool for that.

An Offering is a datastructure that can be attached to any Smart API object. Like its real-life counterpart, an Offering contains the details of an offer: what the price is, how long the offer is in force, is the offer for buying or selling. When an offer is signed, this means that the signer accepts the offer. That creates a transaction that can be stored to a ledger to prove that a deal took place.

The Transaction Server is a trusted third party in a communication between two entities. Its purpose is to record those actions that two communicating parties send to it, timestamp and sign them, and offer a receipt of the action later in case there is a dispute whether something took place or not. The use of the Transaction Server is voluntary, the same data transfer can take place without it, however then with the lacking trust relationship it brings.

16.2. The offering protocol

16.2.1. Attaching offerings to object

The way communication works with Offerings is that one party requests data or action from a counterparty and the counterparty responds, not with the data but an Offering. The Offering is an object that can be attached to any object that is transferred. If the object with an Offering is signed and sent back, the counterparty can then respond with the real deal. Let's see an example.

First, let's request something. In this case an Entity that represents the bank balance of a city.

import smartapi.factory.Factory;
import smartapi.model.Activity;
import smartapi.model.Entity;

Entity balance = new Entity("http://www.cityofcity.org/bankbalance");

Request request;
try {
	request = Factory.createRequest("http://www.cityofcity.org/centralbank/");
} catch ( Exception e ) {
	e.printStackTrace();
	return;
}
Activity a = new Activity();
a.setMethod(RESOURCE.READ);
a.addEntity(balance);
request.addActivity(a);
					

In the example the balance information is not for free. So we respond to the request with the same Entity object but attach the details of payment to it. In this example delivering the bank balance costs 2.30 euros.

Activity a = new Activity();
a.setMethod(RESOURCE.READ);
Entity balance = new Entity("http://www.cityofcity.org/bankbalance");
a.addEntity(balance);
UnitPriceSpecification u = new UnitPriceSpecification();
u.setCurrencyValue(2.30);
u.setQuantity(RESOURCE.CURRENCY);
u.setUnit(RESOURCE.EURO);
u.setCurrency("EUR");
Offering o = new Offering();
o.addPriceSpecification(u);
balance.addOffering(o);
					

The recipient can now examine the Entity that was requested. If it contains an Offering, that Entity must be signed and a request must be resent in order to receive data. The signing is done with the private key of the requester.

Entity balance = response.getActivities().get(0).getEntities().get(0);
if  ( balance.hasOffering() ) {
	balance.sign(myPrivateKey);
		
	Request request = Factory.createRequest("http://www.cityofcity.org/centralbank/");
	Activity a = new Activity();
	a.setMethod(RESOURCE.READ);
	a.addEntity(balance);
	request.addActivity(a);
}
					

Important

A signature should always be applied to the object that contains an Offering, not the Offering object. By signing the whole object, the signer confirms not only the offering but the contents of anything the offering refers to. So if the object is for instance about reading a current bank balance and the Offering states a price for it, signing just the offering would allow parties to change the topic to something completely different, say, a receipt of a balance two months old.

Now the data owner can extract the offering from the next request, verify the signature and see if the correct terms for the deal were signed. If so, it can send data back and record a transaction.

boolean verified = false;
try {
    verified = balance.verifySignature(senderPublicKey);
} catch ( Exception e ) {
	e.printStackTrace();
	System.out.println("Signature verification failed.");
}
if ( verified ) {
	// ... collect and send data here	
}
					

16.2.2. Storing transactions and sending data through a notary

When the data owner sends data to the requester, it can further protect its end of the deal by using the Transaction Server as an intermediary in delivering the data. Before the data is sent, it is encrypted but not with the public key of the receiver. Instead, the sender creates a random symmetric key, encrypts data with it and sends this encrypted data without the key to the receiver. The key is encrypted with the public key of the receiver and sent to the notary. The encryptAndNotarize method takes care of this process.

With this procedure the recipient cannot get the data although it holds it, without first fetching the key from the notary. In doing so the recipient proves that it has received the data.

Let's see what that process might look like at the sender end

String notaryAddress = "http://transact.smart-api.io";
Activity a = new Activity();
a.setMethod(RESOURCE.READ);
Entity balance = new Entity("http://www.cityofcity.org/bankbalance");
balance.add("totalBalance", new ValueObject(null, RESOURCE.EURO, 10017201.29));
SmartAPI.setMyIdentity(myIdentity);
balance.encryptAndNotarize(notaryAddress, recipientsPublicKey, myPrivateKey);
a.addEntity(balance);
					

16.2.3. Decrypting data with keys stored at a notary

When the data sent in the previous step is received, it is encrypted. The recipient does not have the key to unlock it and this is intentional: to get the key the receiver must access the notary with a signed hash of the received message. Because the hash must match the message, this proves that the data was received in its entirety. The signature in the hash proves the content of the data and the identity of the receiver. After this step the receiver can no longer falsely claim that no data was received and this way try to avoid payment. After receiving this confirmation of reception of the data, the notary responds with the enryption key used to encrypt the data. This signing and contacting the notary can be automatically done by Smart API's TransactionAgent.

The recipient must now decrypt the package received from the notary. It contains the secret encyption key. Once done, the recipient has an encryption key that can be used to decrypt the actual data. Note that because this secret key was encrypted with the private key of the recipient the whole time, the notary does not know what the encryption key of the data is and cannot steal the data even if it had eavesdropped the data transmission between the data holder and the recipient. The example code below shows how the process is performed in its entirety.

Entity entity = response.getActivities().get(0).getEntities().get(0);
if ( entity.isEncrypted() ) {
	if ( entity.getEncryptionKeyType().equals(RESOURCE.NOTARIZEDSESSIONKEY) ) {
		String signature = Crypto.sign(myPrivateKey, entity.getHashCode());
		String keyString = TransactionAgent.fetchKeyFromNotary(myIdentity, entity.getIdentifierUri(), entity.getHashCode(), signature, entity.getNotaryAddress(), myPrivateKey);
		SecretKey encryptionKey = SmartAPICrypto.decryptAndDecodeKey(myPrivateKey, keyString);
		entity.decrypt(encryptionKey);
	}
}
					

16.3. Transaction Server functionality

The Smart API Transaction Server was an integral part of the exchange that took place in the example of the previous chapter. But it can be used for other purposes also.

In general the Transaction Server is just a database that stores special objects called Transactions. The server receives a Transaction, examines it, signs it, timestamps it, and stores it. A Transaction on the other hand is a wrapper or carrier of other data. It is a signed object, the server will confirm the identity of the sender before it allows storing anything. In case the signature is good, the data it contains is processed. Note that the data that is sent can be encrypted. The Transaction Server does not need to know nor wants to know what the actual data is. This ensures true end-to-end confidentiality, not even the trusted middleman knows what is actually in the data.

The Transaction Server performs three basic functions

  1. Storage of generic transaction objects. Just check signatures and store, no other processing is done.
  2. Storage of monetary objects. Extract the value of the contracts in the objects and maintain accounting of them. Good for invoicing and accounting for balances.
  3. Storage of session keys for encrypted communication. Ensures non-repudiation between the parties

Storing an item at the notary of the Transaction Server is good for proving that some data did exist at some point in time. The notary creates a timestamp and stores the data, signing both, so as long as the notary - and its clock - are trusted, it can be used as a proof of existence.

Note that this method is still not sufficient to prove that some data did not exist or was held or owned by some party. It does contain a signature but the party whose signature it is can always claim that the signature is forged because their private key was stolen and someone stole the identity.

16.3.1. The Transaction Agent

Communicating with the Transaction Server often requires the exchange of data in a certain sequence to ensure everything is in place correctly. To help in following the sequence, the Smart API library contains a special agent class, the TransactionAgent that performs most of the hard work for you.

The TransactionAgent has an API for each of the three main functionalities of the Transaction Server. Simply invoke the method to perform that action. See API Docs and examples below for further details.

16.3.2. Generic Transaction storage

To store something into the generic storage, simply create an object you want to store and then call the storeObjectToNotary method of the agent. It will return the ID of the Transaction that was generated. This ID can then be used to retrieve the same data back. When you use the TransactionAgent, the ID's are generated with a cryptographic random method, making them impossible to guess and retrieve data without this knowledge. To retrieve data, call the fetchObjectFromNotary method with the ID you got as an argument.

Entity entity = new Entity("http://www.sample.org/myentity");
entity.setGeneratedBy(myIdentity);
String transactionStoredToNotaryId = TransactionAgent.storeObjectToNotary(myIdentity, notaryAddress, entity, myPrivateKey);
					

16.3.3. Transaction ledger

Storing data into the ledger is very similar to storing data into the generic storage. The only difference is that the ledger only stores objects with an offering, therefore this check is made before sending data forward. So create an object you want to store, put an Offering to it and then call the storeObjectToLedger method of the agent. It will return the ID of the Transaction that was generated. Note that as an extra measure of security, retrieving a transaction from the notary requires the knowledge of both this ID and the ID of the object that was stored. fetchObjectFromLedger method will then fetch the data with these parameters.

Entity entity = new Entity("http://www.sample.org/myentity");
entity.setGeneratedBy(myIdentity);
UnitPriceSpecification u = new UnitPriceSpecification();
u.setCurrencyValue(10);
u.setQuantity(RESOURCE.CURRENCY);
u.setUnit(RESOURCE.EURO);
u.setCurrency("EUR");
Offering o = new Offering();
o.addPriceSpecification(u);
entity.addOffering(o);

String transactionStoredToLedgerId = TransactionAgent.storeObjectToLedger(myIdentity, notaryAddress, entity, myPrivateKey);
					

16.3.4. Session key store

Using the session key store is simple and straightforward as the communication with the TransactionAgent is already embedded into the source of the objects themselves. To send a key to the Transaction Server, simply call encryptAndNotarize like so:

Entity entity = new Entity("http://www.sample.org/myentity");
SmartAPI.setMyIdentity(myIdentity);
entity.encryptAndNotarize(notaryAddress, myPublicKey, myPrivateKey);
					

Important

Note that the session key is sent to the notary, when the data is sent to the recipient and not when you call the encryptAndNotarize method. So if you are sniffing the network traffic and debugging the transmissions or printing out object contents, pay attention to timing. To be precise, the key transmission takes place when you serialize the object (or typically the Request containing the object).

Chapter 17. Further examples and best practices

17.1. Create and send request

This example shows how to create a simple request and send it. This complete example combines functionality described in the previous sections.

Request weather forecast:

// Serialization for the RDF data
String serialization = SERIALIZATION.JSON_LD;
// my SmartAPI identifier
String myIdentifier = "http://www.corp.com/activity/Crequester";
// URI of the target service activity
String activityUri = "http://www.weather.org/service/Cforecast";
// URI of the service
String serviceUri = "http://www.weather.org/smartapi/v1.0e1.0/access";
// Smart API method for the HTTP header
String smartAPIMethod = HttpClient.SMARTAPI_METHOD_REQUEST;

Request request = null;
try {
	// Create request
	request = Factory.createRequest(myIdentifier);
	// Create activity for request
	Activity activity = new Activity(activityUri);
	activity.setMethod(RESOURCE.READ);
	request.addActivity(activity);
	// Create input for activity
	Input input = activity.newInput();
	// Set start and end time
	input.setTemporalContext(Tools.stringToDate("2017-12-05T12:00:00+03:00"), 
			Tools.stringToDate("2018-01-05T12:00:00+03:00"));
	// SystemOfInterest
	SystemOfInterest soi = new SystemOfInterest();
	// Location for the requested weather forecast
	soi.setCoordinates(61.3432, 24.2344);
	input.setSystemOfInterest(soi);
	// Set requested quantity and unit
	ValueObject valueObject = new ValueObject();
	valueObject.setQuantity(RESOURCE.CLOUDCOVERINDEX);
	valueObject.setUnit(RESOURCE.PERCENT);
	input.addOutputValues(valueObject);
} catch(Exception e) {
	System.err.println("Exception while creating a Request object");
	e.printStackTrace();
}
// Send request to the service
HttpClient client = new HttpClient();
client.setSmartAPIMethodHeader(smartAPIMethod);
client.setSerialization(serialization);
Response response = null;
try {
	response = client.sendPost(serviceUri, request);
} catch (Exception e) {
	System.err.println("Error while sending request.");
	e.printStackTrace();
}
// print out possible errors on the response
Tools.printErrors(response);
// Get response data
if ( response.hasActivities() && response.getFirstActivity().hasOutput() ) {
	Output o = response.getFirstActivity().getFirstOutput();
	// get data from the output or just print it out
	o.printOut();
}
				

17.2. Interpret request, create and send response

This example shows how to interpret a request, create a response and send it back.

Interpret weather forecast request and respond to it:

private final String myIdentifier = "http://www.weather.org/smartapi/Cservice";
// URI of the forecast service activity
private final String forecastActivityUri = "http://www.weather.org/service/Cforecast";

/**
 * @see HttpServlet#doPost(HttpServletRequest request, HttpServletResponse response)
 */
protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
	PrintWriter out = response.getWriter();
	String contentType = "";
	String serialization = null;
	String reqBody = "";
	String contentTypeHeader = request.getContentType();
	// get content type of the request
	contentType = HttpMessage.getMainRdfContentType(contentTypeHeader);
	// Set response serialization. This is important as RDF can
	// be encoded into many different formats (JSON-LD, Turtle, RDF/XML)
	serialization = SERIALIZATION.fromContentType(contentType);
	if ( serialization.equals(SERIALIZATION.UNKNOWN) ) {
		serialization = SERIALIZATION.DEFAULT;
		contentType = SERIALIZATION.toContentType(serialization);
	}
	response.setContentType(contentType);
	// read request body
	try {
		reqBody = request.getReader().lines().collect(Collectors.joining(System.lineSeparator()));
	} catch (Exception e) {
		out.println(Tools.toString(Factory.createParseErrorResponse(myIdentifier, "Unable to read request body"),
				serialization));
		return;
	}
	// handle request
	Request req;
	try {
		req = Tools.parseRequest(reqBody, contentType);
		Response resp = Factory.createResponse(req, myIdentifier);
		// handle each activity
		for ( Activity a : req.getActivities() ) {
			Activity respActivity = Factory.createResponseActivity(a);
			// handle request based on activity id
			switch (a.getIdentifierUri()) {
			case forecastActivityUri:
				// check request method
				if ( a.hasMethod() && a.getMethod().equals(RESOURCE.READ) ) {
					// handle each input
					for ( Input input : a.getInputs() ) {
						Date start, end;
						double lat,lon;
						List<ValueObject> weatherParams = new ArrayList<ValueObject>();
						// verify that all required parameters are available and read them
						if ( input.hasTemporalContext() && input.getTemporalContext().hasStart() && 
								input.getTemporalContext().hasEnd() ) {
							start = input.getTemporalContext().getStart().asDate();
							end = input.getTemporalContext().getEnd().asDate();
						} else {
							out.println(Tools.toString(
									Factory.createInvalidParamsErrorResponse(myIdentifier, 
											"Found input does not contain start and end timestamps. "),
									serialization));								
						}
						if ( input.hasSystemOfInterest() && input.getSystemOfInterest().hasCoordinates() ) {
							Coordinates c = input.getSystemOfInterest().getCoordinates();
							if ( c.hasLatitude() && c.hasLongitude() ) {
								lat = c.getLatitude();
								lon = c.getLongitude();
							} else {
								out.println(Tools.toString(
										Factory.createInvalidParamsErrorResponse(myIdentifier, 
												"Found input does not contain coordinates. "),
										serialization));
							}
						} else {
							out.println(Tools.toString(
									Factory.createInvalidParamsErrorResponse(myIdentifier, 
											"Found input does not contain coordinates. "),
									serialization));
						}
						weatherParams = input.getOutputValues();
						// get data from your system
						Output output = getForecast(start, end, lat, lon, weatherParams);
						// add forecast data as output to response
						respActivity.addOutput(output);
					}
					resp.addActivity(respActivity);
				}					
			}
		}
		out.println(Tools.serializeResponse(resp, serialization));
	} catch (Exception e) {
		e.printStackTrace();
		out.println(Tools.toString(
				Factory.createServerErrorResponse(myIdentifier, "Error while handling request. " + e.getMessage()),
				serialization));
		return;
	}
}
				

17.3. Utility functions for type conversion

Library provides methods for easily converting Strings to some common objects and back.

Convert between String and Date:

Date date = Tools.stringToDate("2017-08-25T12:24:11");
String dateString = Tools.dateToString(date);
				

Convert between String and Time:

Time time = Tools.createTimeFromLocalTimeString("08:00:00");
// or
Time time = Tools.createTimeFromUTCTimeString("08:00:00");

String timeString = time.toString();
				

Convert between local and UTC Time:

// UTC to local
Time localTime = Tools.toLocal("08:00:00");

// local to UTC
Time utcTime = Tools.toUTC("10:00:00");
				

Convert between String and Duration:

Duration duration = Tools.toDuration("P1DT6H");
String durationString = duration.toString();
				

Convert a Variant to string:

Variant variant = <some variant>;
String value = variant.getAsString();
				

Colophon

<bookinfo>Asema Electronics Ltd<copyright><year>2011-2019</year></copyright><legalnotice>

No part of this publication may be reproduced, published, stored in an electronic database, or transmitted, in any form or by any means, electronic, mechanical, recording, or otherwise, for any purpose, without the prior written permission from Asema Electronics Ltd.

Asema E is a registered trademark of Asema Electronics Ltd.

</legalnotice>
</bookinfo>