The Seven attributes of an Open Data Format for LiDAR Software

Raw data from LiDAR can be very useful in gathering key data points.

And thanks to processing software like Outsight’s Shift platform they can then be converted into actionable insights so useful applications can make good use of them, in this example precisely detecting and tracking people:

A key advantage of using a LiDAR Processing Software that is agnostic to the sensing LiDAR hardware is the ability to output a unique data format regardless of the input device (and very frequently a combination of sensors from different manufacturers, as required in many projects).

Technical diagram showing how Outsight takes 3D raw data transforming it into spatial intelligence solutions and APIs

LiDAR proprietary data formats and protocols made sense when this emerging technology first appeared, as it shortened the time to market of new hardware products delivering raw data.

The first LiDAR prototypes were not intended to be used for production purposes. The burden of decoding each specific format was on the early users.

However, in order for the technology to reach mainstream use in scalable use cases it is imperative for 3D LiDAR data to be accessible through open data formats.

An open data format is one that has a freely available published specification which places no restrictions, monetary or otherwise, upon its use, thus enabling conformant software to read and write the data in that format can be implemented by others.

The key characteristics of an open LiDAR data format

1. Open

The system needs to be open, in a way that allows it to function without proprietary intellectual property.

This will lower the risk of using the system, as open proprietary formats can be used without restrictions and can support movement from one technical platform or environment, without compromising legal agreements.

Intellectual property is also typically publicly available, which provides greater security and accessibility to a bigger range of softwares and technical platforms.

2. Simple

The processing system also has to be simple to use.

Parsing data on the system must be straightforward and efficient, which is easier to say than to do: more complex formats can be more efficient in some specific use cases, the simplicity is then a trade-off between usability, time to market and performance.

3. Robust

When looking for a LiDAR processing software, one of the key characteristics you should look for is its robustness. For example, data elements should be allowed to be placed in any order inside the message body.

This greatly simplifies decoding drivers, limits bugs and facilitates the following characteristic:

4. Backwards-compatible

Your processing Lidar software should fit your needs at every stage of your data processing process.

Over time, you will most likely find that you have new features and data fields that you would like to add to the system, while safely skipping older or unknown elements, so that they can run seamlessly with existing data fields.

Companies like Outsight releases new versions of the software frequently, that can't always be deployed instantaneously in your whole installed fleet: several versions will likely cohabitate in the same network and system.

5. Efficient

When using a LiDAR point-cloud software, the last thing you want is for it to incur wastage in processing time, thus slowing down your processes and even compromising real-time capabilities.

Therefore, it is key to ensure that overhead processing time is minimised and bandwidth is optimised.

6. Adaptable

Your system should also be able to react to real-time user requests without changing its structure and decoding patterns, so that only relevant data is sent at all times.

In application-agnostic software solutions like Outsight's not every user will need the output of every available feature.

For example, in an infrastructure project (where the LiDARs are placed in a fixed position) is not useful for the Ego-Motion information to be streamed on each frame.

An adaptable solution must allow to only stream relevant information for the use case at hand, optimising bandwidth and decoding time.

7. Flexible

Your 3D processing software should be usable for any volume of data.

There will be times that you will need to make use of simple and small-sized volume of data, for example when using a single and low-resolution LiDAR.

Other times, you will need to take advantage of mass amounts of 3D information to perform more complex tasks or simply cover large areas, with the retransmitted point-cloud and associated attributes.

This includes applications where dozens of LiDAR sensors are fused together to create a single output stream.

Therefore, your system should be flexible and readily available to scale and handle data of any volume without any change in the data format.

The Shift Perception LiDAR Software

Outsight's Shift Perception delivers its output using OSEF [Open SErialisation Format].

It uses TLV-encoding, which is a scheme that is used for optional informational elements and contains code that is associated with time, length, and value.

As seen in the diagram below, at each processed LiDAR Frame - typically at 20 fps - the structure of the OSEF data is a tree that contains the timestamp information and all its attributes, for example Ego-Motion and Tracked Objects' information.

Tree diagram showing the structure of OSEF data

With the Shift Perception software you will be able to make full use of an open and standard data format that is open, simple, robust, backwards-compatible, efficient, adaptable, and flexible.

This ensures that you can take your LiDAR solution into the future, with the confidence that no matter how your needs, data points, and outcomes change over time, the system will be able to support your goals seamlessly.