SAILenv is a Virtual Environment powered by Unity3D. It includes 3 pre-built scenes with full pixel-wise annotations. SAILenv is capable of generating frames at real-time speed, complete with pixel-wise annotations, optical flow and depth.

SAILenv also comes with a Python API, designed to easily integrate with the most common learning frameworks available.

Pixel-wise Annotations

SAILenv yields the classic RGB view, depth information, optical flow, semantic segmentation, instance labeling, as shown in the figure above Each view includes information for all the pixels acquired by the camera. The RGB view is straightforwardly implemented representing each pixel (in each channel) with the classic 8-bit encoding, while depth information is inherited by the Unity engine, through a gray-scale texture representing the distance of the observed rendered objects from the camera view. In the Depth View, lighter pixels indicate elements that are closer to the agent.

Each pixel is fully annotated with a category identifier (semantic labeling) and an instance identifier, that are encoded in the category and instance views, respectively, as shown above. While the instance identifier is implicitly given by the unique Unity identifier of each object in the scene, categories can be added or edited using the Unity editor GUI, without any code-level operations. In particular, categories are represented as Unity objects, and they can be attached to every other object by a drag-and-drop operation. SAILenv also includes a “category holder” to organize sets of categories and to allow the user to quickly add them to the current scene.

Optical Flow Comparison

SAILenv also yields highly precise and dense motion information about the environment. Differently from what is done by the most common optical flow algorithms, the SAILenv optical flow is not due to an estimation obtained by observing consecutive frames, and it is fully computed by the physics engine of Unity.
Unity has access to the information about the motion of the objects in the scene and the agent viewpoint, and it uses them to drive the simulation of physics of the environment. SAILenv inherits such information and adapts it to generate a view that includes the motion vectors for all the pixels of the frame.

In the images above we report three examples of the optical flow computed in a scene populated solely by a rotating cube. The cube has no special textures, and it has a uniform color. This clearly makes it hard to estimate the pixel-level motion using classic algorithms, while SAILenv can correctly capture the rotation of the cube. Widely used implementations, such as the Farneback algorithm implemented in OpenCV4, or modern approaches based on convolutional neural networks (FlowNetLite) fail in correctly capturing the motion, as noticeable in the above figures. Despite its very high precision, the optical flow has almost null computational burden on the Unity server.

Object Library

Relying on the Unity engine to handle the virtual environments allows SAILenv to exploit all the facilities of the powerful 3D editor that comes with Unity.
However, creating new scenes in virtual environments might quickly become a time consuming procedure that requires experience in 3D graphics. This is even more evident when preparing photo-realistic objects, that requires the user to pay attention to a large number of aspects in order to reach a certain target appearance for the object.

In order to partially mitigate these issues, SAILenv comes with more than 65 objects that can be placed in any scene, plus some objects related to the structure of the sample scenes (walls, windows, etc.). See the images above for some examples of available objects. Most object meshes were originally taken from the AI2-THOR project, and strongly re-worked in order to improve the quality of their appearance, reaching a more advanced photo-realistic level.

Ready-Made Scenes

SAILenv currently includes a ready-to-go Unity project with all the photo-realistic elements and 4 sample scenes based on them, meant to demonstrate the capabilities of the framework and to run some experiments in simple contexts. The user can either edit one of these scenes or create a new one either using the SAILenv objects or other 3D elements. The sample scenes are about different rooms, and they are based on a variety of objects, and they also include moving objects to evaluate motion-based algorithms. The agent has a predefined motion pattern such that it automatically moves around the scenes exploring the available areas.

 

Download

SAILenv executables: Version Directory

SAILenv Unity Project: Source Code

SAILenv Python API: Source Code (GitHub) | Pip Package

Python Downloader:

Packages can also be downloaded with the Python API. These instructions will download the zip file on the working directory.

import sailenv.downloader as dwn
dwn.download_executable(version="latest")
dwn.download_source(version="latest")

 

Installation

SAILenv supports all main Operating Systems (Windows, Linux, Mac OSX).

Install on Windows

1. Install the python package through pip or through source code
2. Download the executable through version directory or with Python (see above)
3. Double click on SAILenv.exe
4. Run the Python Code

 

Install on Linux

1. Install the python package through pip or through source code
2. Download the executable through version directory or with Python (see above)
3. If you don’t have a monitor attached to the server, you need to setup a X11 server with virtual monitors.
4. Add execution permissions: chmod +x ./SAILenv.run
5. Run ./SAILenv.run on the chosen display
6. Run the Python Code

 

Install on Mac Os X

1. Install the python package through pip or through source code
2. Download the executable through version directory or with Python (see above)
3. If you don’t have a monitor attached to the server, you need to setup a X11 server with virtual monitors.
4. Add execution permissions: chmod +x SAILenv.app/Contents/MacOS/SAILenv
5. Double Click on SAILenv.app
6. Run the Python Code

ArXiv publication

Link to arXiv page.

Extended Tutorial

You can find the tutorial here here.

 

Further Questions

If you have any unsolved questions or you just want to get in contact with us, please write to any of the following email addresses:

meloni@diism.unisi.it

Sample Script

import time
import numpy as np
import cv2
from random import randint

# Import src

from sailenv.agent import Agent
from sailenv.utilities import draw_flow_map


def decode_image(array: np.ndarray):
    """
    Decode the given numpy array with OpenCV.

    :param array: the numpy array to decode
    :return: the decoded image that can be displayed
    """
    image = cv2.imdecode(array, cv2.IMREAD_COLOR)
    return image

frames: int = 1000
host = "127.0.0.1"
if __name__ == '__main__':
    print("Generating agent...")
    agent = Agent(depth_frame_active=False,
                  flow_frame_active=True,
                  object_frame_active=True,
                  main_frame_active=False,
                  category_frame_active=True, width=256, height=192, host=host, port=8085, gzip=False)
    print("Registering agent on server...")
    agent.register()
    print(f"Agent registered with ID: {agent.id}")
    last_unity_time: float = 0.0

    print(f"Available scenes: {agent.scenes}")

    scene = agent.scenes[1]
    print(f"Changing scene to {scene}")
    agent.change_scene(scene)

    print(f"Available categories: {agent.categories}")
    try:
        print("Press ESC to close")
        while True:
            frame = agent.get_frame()
            
            # getting RGB view
            if frame["main"] is not None:
                main_img = frame["main"]
                cv2.imshow("PBR", main_img)

            # getting instance segmentation view
            if frame["object"] is not None:
                obj_img = decode_image(frame["object"])
                cv2.imshow("Object ID", obj_img)

            # getting optical flow view
            if frame["flow"] is not None:
                flow = frame["flow"]
                # utility for converting to HSV color space
                flow_img = draw_flow_map(flow)
                cv2.imshow("Optical Flow", flow_img)

            # getting depth view
            if frame["depth"] is not None:
                depth = frame["depth"]
                cv2.imshow("Depth", depth)

            key = cv2.waitKey(1)
            if key == 27:  # ESC Pressed
                break
    finally:
        print(f"Closing agent {agent.id}")
        agent.delete()