See also: [device](device.md), [notifications](notifications.md) # Initialization In order to stream, a client must know the stream names, available formats, etc. The only way for a client to get these is by subscribing to the `notification` topic. When the server detects a subscriber on this topic, it will broadcast a set of initialization messages in the following order: - `device-header` - `device-options` - optional - For each stream: - `stream-header` - `stream-options` These initialization messages should only have effect on devices that are not already initialized. They are expected in the above order. Once all streams have been received, the device is initialized and ready to use. See [Streaming](streaming.md). ## "Atomic" Initialization Initialization may happen interspersed with regular notifications: for example, a server that's already serving data and notifications to an existing client may see another client and broadcast initialization messages. Only one set of initialization messages can be sent at a time. I.e., if a new client is detected while initialization messages for a previous client are still outgoing, a new set of messages must go out but only once the previous set is finished. While a set of initialization messages are outgoing, all other notifications must take a back seat and wait until the set is written out. ## Messages ### `device-header` This is the very first message, for example: ```JSON { "id": "device-header", "n-streams": 5, "extrinsics": [ ["Depth","Gyro",[1.0,0.0,0.0,0.0,1.0,0.0,0.0,0.0,1.0,-0.005520000122487545,0.005100000184029341,0.011739999987185001]] ["Depth","Infrared 2",[1.0,0.0,0.0,0.0,1.0,0.0,0.0,0.0,1.0,-0.04986396059393883,0.0,0.0]] ], "presets": ["Default", "Max Range", "Max Quality"] } ``` - `n-streams` is the number of streams to expect The device will wait for this many stream headers to arrive to finish initialization - `extrinsics` describe world coordinate transformations between any two streams in the device, required for proper translation of pixel coordinates between sensors, such as when a point-cloud is needed - `presets` is an optional array of preset names The presets may then be applied using `change-preset` #### Extrinsics In order to translate from one stream viewpoint to another, a graph needs to be available with nodes for each stream and a directional edge between them (one from A to B; another from B to A). So `extrinsics` is an array of such edges: `[,,...]`, Each `edge` is also an array: `[,,[,,...,,,,]]`. Where `` and `` are stream names, and the array inside contains the 9 rotation values (column-major) followed by 3 translation vector values. The complete graph can be transmitted like this but this is overkill: given at least one edge from each stream to another, the rest can be computed. This is what librealsense does: - Depth to IR = identity - Depth to IR2 = {...} - RGB to Depth = {...} With those 3, one can compute IR2 to RGB, for example. ### `device-options` This is optional: not all devices have options. See [device](device.md). ```JSON { "id": "device-options", "options": [ ["Domain",0,0,232,1,0,"The DDS domain (0-232) in which this device will be discovered"], ["IP Address","1.2.3.4",null,"Which IP address to assign to the device; if empty, DHCP will be used", ["optional","IPv4"]] ] } ``` * `"options"` is an array of options * Each option is an array of `[name, value, range..., default-value, description, [properties...]]`: * The `name` is what will be displayed to the user * The current `value` * An optional `range` of valid values * Numeric options (`float`, `int`), defined by a `minimum`, `maximum`, and `stepping` * I.e., is-valid = one-of( `minimum`, `minimum+1*stepping`, `minimum+2*stepping`, ..., `maximum` ) * Booleans can remove the range, e.g. `["Enabled", true, true, "Description"]` * Booleans can be expressed as a range with `minimum=0`, `maximum=1`, `stepping=1` * Free string options would likewise have no range, e.g. `["Name", "Bob", "", "The customer's name"]` * `"IPv4"` is a string option that conforms to `W.X.Y.Z` (IP address) format * Enum options are strings with an array of choices, e.g. `["Preset", "Maximum Quality", ["Maximum Range", "Maximum Quality", "Maximum Speed"], "Maximum Speed", "Standard preset combination of options"]` * A `default-value` which also adheres to the range * If this and the range are missing, the option is read-only * A user-friendly description that describes the option, to be shown in any tooltip * Additional `properties` describing behavior or nature, as an array of (case-sensitive) strings * `"optional"` to note that it's possible for it to not have a value; lack of a value is denoted as `null` in the JSON * If optional, a type must be deducible or present in the properties * E.g., `["name", null, "description", ["optional", "string"]]` is an optional read-only string value that's currently unset * Enums cannot be optional * `"string"`, `"int"`, `"boolean"`, `"float"`, `"IPv4"`, `"enum"` can (and sometime must) indicate the value type * If missing, the type will be deduced, if possible, from the values * `"read-only"` options are not settable * `set-option` will fail for these, though their value may change on the server side * The device server has final say whether an option value is valid or not, and return an error if `set-option` specifies an unsupported or invalid value based on context Device options will not be shown in the Viewer. ### `stream-header` Information about a specific stream: - `name` is the stream name, e.g. `Color` - This will be shown in the Viewer - This is also the name of the topic containing the stream data - E.g., `rt/_Color` - If it contains a space, the space will be replaced by `_` - e.g., `rt/_Infrared_1` - `profiles` array - Each profile is itself an array of `[frequency, format, width, height]` - Format is a string representation, similar to the image encoding in ROS - `default-profile-index` is the index into the `profiles` for the default profile - `sensor-name` is the name of the sensor - Sometimes, a single sensor may produce multiple streams, e.g. `Stereo Module` produces a `Depth` stream and also `Infrared` - Streaming one stream requires starting the sensor, and so may have effect on the other streams, depending on the server implementation - This allows streams to be grouped by the client and may affect its logic - `type` is one of `ir`, `depth`, `color`, `confidence`, `motion` - similar to the librealsense `rs2_stream` enum - `metadata-enabled` is `true` if a `metadata` topic for the device will be written to ```JSON { "default-profile-index": 29, "id": "stream-header", "metadata-enabled": true, "name": "Color", "profiles": [ [30,"rgb8",1280,800], [30,"BYR2",1280,800], [30,"Y16",1280,800], ... ], "sensor-name": "RGB Camera", "type": "color" } ``` ### `stream-options` - `stream-name` is the name of the stream, same as in `stream-header` - `intrinsics` is: - For video streams, see below - For motion streams, a mapping from either `accel` or `gyro` to an array of float values conforming to: ```C++ struct rs2_motion_device_intrinsic { // Scale X cross axis cross axis Bias X // cross axis Scale Y cross axis Bias Y // cross axis cross axis Scale Z Bias Z float data[3][4]; // Variance of noise for X, Y, and Z axis float noise_variances[3]; // Variance of bias for X, Y, and Z axis float bias_variances[3]; } ``` e.g.: ```JSON "intrinsics": { "accel": [1.0,0.0,0.0,0.0,0.0,1.0,0.0,0.0,0.0,0.0,1.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0], "gyro": [1.0,0.0,0.0,0.0,0.0,1.0,0.0,0.0,0.0,0.0,1.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0] } ``` - `options` is an array of option objects, same as `device-options` above Stream options are shown in the Viewer. #### Video Stream Intrinsics ```JSON { "id": "stream-options", "intrinsics": { "width": 1280, "height": 720, "principal-point": [640.2379150390625,357.3431396484375], "focal-length": [631.3428955078125,631.3428955078125] }, "options": [ ["Backlight Compensation",0.0,0.0,1.0,1.0,0.0,"Enable / disable backlight compensation"], ["Brightness",0.0,-64.0,64.0,1.0,0.0,"UVC image brightness"], ], "stream-name": "Infrared 1" } ``` Like extrinsics are used to communicate translation between different stream viewpoints, the intrinsics serve to transform 2D pixel values to 3D world coordinates. I.e., an RGB pixel has to be converted to a 3D point in space, then mapped to a 3D point from the viewpoint of the Depth stream, then transformed back into a 2D Depth pixel. The intrinsics are communicated in an object, as shown above: - A `width` and `height` are for the native stream resolution, as 16-bit integer values - A `principal-point` defined as `[,]` floating point values - A `focal-length`, also as `[,]` floats A distortion model may be applied: - The `model` would specify which model is to be used, with the default of `brown` - The `coefficients` is an array of floating point values, the number and meaning which depend on the `model` - For `brown`, 5 points [k1, k2, p1, p2, k3] are needed The coefficients are assumed 0 if not there, applying no un/distortion. An additional `force-symmetry` boolean can be applied, and defaults to `false`. The intrinsics are communicated for the native resolution the device chooses. Librealsense, or any client, will need to scale these to a target resolution.