Metrics module

The metrics module relies on the CARLA recorder to facilitate the easy calculus and monitoring of any kind of parameter. After the scenario is finished, the recorder stores the simulation information. Users can then define their own metrics, and check the corresponding results with no need to play the simulation again and again.

This section covers an explanation of the different elements that form the module, a step by step tutorial of how to create a new metric, and a reference of the functions that query the recording and define the metrics.

Structure of the module
How to use the metrics module
Recording queries reference

Structure of the module

Similarly to the main ScenarioRunner module, the Metrics module is run using a main script, metrics_manager.py, and the rest of the information is contained inside a folder structure. Here is a brief introduction to the main script.

metrics_manager.py — The main script of the module. Run this to show the results of the set of metrics. The script has the usual host and port arguments, and some more to set the metrics and recording to be used.
- host (string) – IP address where a CARLA simulation is running. Default is (127.0.0.1).
- port (int) – TCP port where the CARLA simulation is running. Default are 2000 and 2001.
- metrics — Path to the metrics to be used.
- log — Path to the .log file containing the recording (relative to the environment variable SCENARIO_RUNNER_ROOT).
- criteria (optional) — Path to a JSON file with the criteria of the scenario.

The rest of the elements that shape the module can be found in the srunner/metrics folder. These folder has been divided in three subfolders.

srunner/metrics/data — Stores information about the scenarios. By default, it has six files, which are part of the examples metric.
srunner/metrics/examples — Contains some example metrics, and the metric containing the base class BaseMetric to create new metrics.
- basic_metric.py – Contains the base class BaseMetric. All the metrics are inherited from it.
- criteria_filter.py – Returns a JSON with the most important attributes of the criteria.
- distance_between_vehicles.py – Returns the distance betwen two vehicles. Useful to show how to query the recording.
- distance_to_lane_center.py – Calculates the distance between the vehicle location and the center of the lane. Useful to show how to access the map API information..
srunner/metrics/tools — Contains two key scripts that allow to query the recording.
- metrics_parser.py – Transforms the string provided by the recording to a dictionary.
- metrics_log.py – Provides with several functions to query the dictionary created with metrics_parser.py. These functions are the easiest way to access information of a scenario. They listed in a reference in the last segment of this page.

How to use the metrics module

1. Record a scenario

The metrics module needs for a recording of a simulation in order to work. Otherwise, it has no data to make the calculations of the metrics.

Use the record argument. Add the path where the information should be saved should be saved. The path must be relative to the environment variable SCENARIO_RUNNER_ROOT.

python scenario_runner.py --scenario <scenario_name> --record <path/to/save/the/recorder/file>

By recording the scenario two files will be created in the desired path. These files will later be used as the log and criteria arguments in the metrics_manager.py.
1. A CARLA recording (.log) — Contains simulation data per frame. To know more about this, read the recorder docs.
2. A criteria file (.json) — The criteria of the scenario parsed as a dictionary, and stored in a JSON file. The keys of the dictionary are the names of the criteria. The values are the attributes of each criteria.

Note

Only the JSON serializable attributes will be parsed, the rest will be ignored.

By default, both files are named after the scenario that is run. If the scenario is named example.py, the files will be example.log and example.json.

2. Define the metrics

It is time to create the desired metrics that will be used for the scenario. The possibilities are endless, but for the sake of this tutorial, the metric described in srunner/metrics/examples/distance_between_vehicles.py will be used as an example. Let's dig a little bit into the code itself.

class DistanceBetweenVehicles(BasicMetric):
"""
Metric class DistanceBetweenVehicles
"""

def _create_metric(self, town_map, log, criteria):
    """
    Implementation of the metric. This is an example to show how to use the recorder,
    accessed via the log.
    """

    # Get the ID of the two vehicles
    ego_id = log.get_ego_vehicle_id()
    adv_id = log.get_actor_ids_with_role_name("scenario")[0]  # Could have also used its type_id

    dist_list = []
    frames_list = []

    # Get the frames both actors were alive
    start_ego, end_ego = log.get_actor_alive_frames(ego_id)
    start_adv, end_adv = log.get_actor_alive_frames(adv_id)
    start = max(start_ego, start_adv)
    end = min(end_ego, end_adv)

    # Get the distance between the two
    for i in range(start, end):

        # Get the transforms
        ego_location = log.get_actor_transform(ego_id, i).location
        adv_location = log.get_actor_transform(adv_id, i).location

        # Filter some points for a better graph
        if adv_location.z < -10:
            continue

        dist_v = ego_location - adv_location
        dist = math.sqrt(dist_v.x * dist_v.x + dist_v.y * dist_v.y + dist_v.z * dist_v.z)

        dist_list.append(dist)
        frames_list.append(i)

    # Use matplotlib to show the results
    plt.plot(frames_list, dist_list)
    plt.ylabel('Distance [m]')
    plt.xlabel('Frame number')
    plt.title('Distance between the ego vehicle and the adversary over time')
    plt.show()

2.1. Name the metric. First of all, as it was previously mentioned, all metrics are childs of the BasicMetric class.

    class DistanceBetweenVehicles(BasicMetric):

2.2. Name the main method. A metric only requires one method in order to run, _create_metric(), which has three arguments.

town_map — Instance of the carla.Map() where the scenario took place in.
log — Instance to the MetricsLog class, with all the functions needed to access the recorder dictionary.
criteria — A JSOn with the criteria dictionary. The file provided when recording a scenario that is later provided to the metrics_manager.py.

    def _create_metric(self, town_map, log, criteria):

2.3. Implement the metric. The code will vary depending on the metric itself.

The example metric DistanceBetweenVehicles calculates the distance between the ego vehicle, and the car it follows (adversary). To do so, it needs the id of the two vehicles. These can be retrieved from the log using the get_ego_vehicle_id() and get_actor_ids_with_role_name("scenario")[0] functions.

    # Get the ID of the two vehicles
    ego_id = log.get_ego_vehicle_id()
    adv_id = log.get_actor_ids_with_role_name("scenario")[0]  # Could have also used its type_id

The id are used to retrieve the frames where the vehicles were alive using get_actor_alive_frames(actor_id).

    # Get the frames both actors were alive
    start_ego, end_ego = log.get_actor_alive_frames(ego_id)
    start_adv, end_adv = log.get_actor_alive_frames(adv_id)
    start = max(start_ego, start_adv)
    end = min(end_ego, end_adv)

Now everything is ready to loop through those frames, get their transforms, and calculate the distance. To get the transform, get_actor_transform(actor_id, frame) is used.

    dist_list = []
    frames_list = []

...

    # Get the distance between the two
    for i in range(start, end):

        # Get the transforms
        ego_location = log.get_actor_transform(ego_id, i).location
        adv_location = log.get_actor_transform(adv_id, i).location

        # Filter some points for a better graph
        if adv_location.z < -10:
            continue

        dist_v = ego_location - adv_location
        dist = math.sqrt(dist_v.x * dist_v.x + dist_v.y * dist_v.y + dist_v.z * dist_v.z)

        dist_list.append(dist)

Note

The vertical condition of the adversary is to only take into account the adversary when it is driving normally.

Lastly, use matplotlib to define which should be the output of the metric when running metrics_manager.py.

        # Use matplotlib to show the results
        plt.plot(frames_list, dist_list)
        plt.ylabel('Distance [m]')
        plt.xlabel('Frame number')
        plt.title('Distance between the ego vehicle and the adversary over time')
        plt.show()

3. Run the metrics manager

Finally it is time to used the information retrieve from the scenario, and the metrics that have been defined to return some metrics information. Let's run the module using the example metric that has been used so far, and an example log named after it.

python metrics_manager.py --metric srunner/metrics/examples/distance_between_vehicles.py --log srunner/metrics/data/DistanceBetweenVehicles.log

Warning

A simulation must be running. Otherwise, the module will not be able to acces the map API.

This will create a new window with the results plotted. The script will not finish until the ouput window is closed.

metrics_plot

Recording queries reference

When defining a metric, all the information about the scenario is accessed via the MetricsLog class (the log argument at the _create_metric() function). This class is located at srunner/metrics/tools/metrics_log.py, and this reference is a following of the functions contained in it.

Generic actor data

get_ego_vehicle_id(self)
Returns the id of the ego vehicle.
- Return — int
get_actor_ids_with_role_name(self, role_name)
Returns a list of actor id that match the given role_name.
- Return — list
- Parameters
  - role_name (str) — role_name of the actor.
get_actor_ids_with_type_id(self, type_id)
Returns a list of actor id that match the given type_id, according to fnmatch standard.
- Return — list
- Parameters
  - type_id (str) — type_id of the actor.
get_actor_attributes(self, actor_id)
Returns a dictionary with all the attributes of an actor.
- Return — dict
- Parameters
  - actor_id (int) — id of the actor.
get_actor_bounding_box(self, actor_id)
Returns the bounding box of the specified actor.
- Return — carla.BoundingBox
- Parameters
  - actor_id (int) — id of the actor.
get_traffic_light_trigger_volume(self, traffic_light_id)
Returns the trigger volume of the specified traffic light.
- Return — carla.BoundingBox
- Parameters
  - traffic_light_id (int) — id of the traffic light.
get_actor_alive_frames(self, actor_id)
Returns a tuple with the first and last frame an actor was alive. Note that frames start at 1, not 0.
- Return — tuple
- Parameters
  - actor_id (int) — id of the actor.

Generic simulation data

get_collisions(self,actor_id)
Returns a list of dictionaries with two keys. frame is the frame number of the collision, and other_id, a list of the ids the actor collided with at that frame.
- Return — list
- Parameters
  - actor_id (int) — id of the actor.
get_total_frame_count(self)
Returns an int with the total amount of frames the simulation lasted.
- Return — int
get_elapsed_time(self, frame)
Returns a float with the elapsed time of a specific frame.
- Return — float
- Parameters
  - frame (int) — Frame number.
get_delta_time(self, frame)
Returns an float with the delta time of a specific frame.
- Return — float
- Parameters
  - frame (int) — Frame number.
get_platform_time(self, frame)
Returns a float with the platform time of a specific frame.
- Return — float
- Parameters
  - frame (int) — Frame number.

Actor accelerations

get_actor_acceleration(self, actor_id, frame)
Returns the acceleration of the actor at a given frame. Returns None if the actor id doesn't exist, the actor has no acceleration, or the actor wasn't alive at that frame.
- Return — carla.Vector3D
- Parameters
  - actor_id (int) — id of the actor.
  - frame (int) — Frame number.
get_all_actor_accelerations(self, actor_id, first_frame=None, last_frame=None)
Returns a list with all the accelerations of the actor at the frame interval. By default, the frame interval comprises all the recording.
- Return — list(carla.Vector3D)
- Parameters
  - actor_id (int) — id of the actor.
  - first_frame (int) — Initial frame of the interval. By default, the start of the simulation.
  - last_frame (int) — Last frame of the interval. By default, the end of the simulation.
get_actor_accelerations_at_frame(self, frame, actor_list=None)
Returns a dict where the keys are the frame number, and the values are the carla.Vector3D of the actor at the given frame. By default, all actors are considered but if actor_list is passed, only actors in the list will be checked.
- Return — list(carla.Vector3D)
- Parameters
  - frame (int) — Frame number.
  - actor_list (int) — List of actor id.

Actor angular velocities

get_actor_angular_velocity(self, actor_id, frame)
Returns the angular velocity of the actor at a given frame. Returns None if the actor id doesn't exist, the actor has no angular velocity, or the actor wasn't alive at that frame.
- Return — carla.Vector3D
- Parameters
  - actor_id (int) — id of the actor.
  - frame (int) — Frame number.
get_all_actor_angular_velocities(self, actor_id, first_frame=None, last_frame=None)
Returns a list with all the angular velocities of the actor at the frame interval. By default, the frame interval comprises all the recording.
- Return — list(carla.Vector3D)
- Parameters
  - actor_id (int) — id of the actor.
  - first_frame (int) — Initial frame of the interval. By default, the start of the simulation.
  - last_frame (int) — Last frame of the interval. By default, the end of the simulation.
get_actor_angular_velocities_at_frame(self, frame, actor_list=None)
Returns a dictionary where the keys are the frame number, and the values are the carla.Vector3D of the actor at the given frame. By default, all actors are considered but if actor_list is passed, only actors in the list will be checked.
- Return — list(carla.Vector3D)
- Parameters
  - frame (int) — frame number.
  - actor_list (int) — List of actor ids.

Actor controls

get_vehicle_control(self, vehicle_id, frame)
Returns the control of a vehicle at a given frame. The manual_gear_shift attribute will always be False.
- Return — carla.VehicleControl
- Parameters
  - vehicle_id (int) — id of the vehicle.
  - frame (int) — Frame number.
get_vehicle_physics_control(self, vehicle_id, frame)
Returns the physics control of a vehicle at a given frame.
- Return — carla.VehiclePhysicsControl
- Parameters
  - vehicle_id (int) — id of the vehicle.
  - frame (int) — Frame number.
get_walker_speed(self, walker_id, frame)
Returns the speed of a walker at a given frame.
- Return — carla.Vector3D
- Parameters
  - walker_id (int) — id of the walker.
  - frame (int) — Frame number.

Actor transforms

get_actor_transform(self, actor_id, frame)
Returns the transform of the actor at a given frame. Returns None if the actor id doesn't exist, the actor has no transform, or the actor wasn't alive at that frame.
- Return — carla.Transform
- Parameters
  - actor_id (int) — id of the actor.
  - frame (int) — Frame number.
get_all_actor_transforms(self, actor_id, first_frame=None, last_frame=None)
Returns a list with all the transforms of the actor at the frame interval. By default, the frame interval comprises all the recording.
- Return — list(carla.Transform)
- Parameters
  - actor_id (int) — id of the actor.
  - first_frame (int) — Initial frame of the interval. By default, the start of the simulation.
  - last_frame (int) — Last frame of the interval. By default, the end of the simulation.
get_actor_transforms_at_frame(self, frame, actor_list=None)
Returns a dictionary where the keys are the frame number, and the values are the carla.Transform of the actor at the given frame. By default, all actors are considered but if actor_list is passed, only actors in the list will be checked.
- Return — list(carla.Transform)
- Parameters
  - frame (int) — Frame number.
  - actor_list (int) — List of actor id.

Actor velocities

get_actor_velocity(self, actor_id, frame)
Returns the velocity of the actor at a given frame. Returns None if the actor id doesn't exist, the actor has no velocity, or the actor wasn't alive at that frame.
- Return — carla.Vector3D
- Parameters
  - actor_id (int) — id of the actor.
  - frame (int) — Frame number.
get_all_actor_velocities(self, actor_id, first_frame=None, last_frame=None)
Returns a list with all the velocities of the actor at the frame interval. By default, the frame interval comprises all the recording.
- Return — list(carla.Vector3D)
- Parameters
  - actor_id (int) — id of the actor.
  - first_frame (int) — Initial frame of the interval. By default, the start of the simulation.
  - last_frame (int) — Last frame of the interval. By default, the end of the simulation.
get_actor_velocities_at_frame(self, frame, actor_list=None)
Returns a dict where the keys are the frame number, and the values are the carla.Vector3D of the actor at the given frame. By default, all actors are considered but if actor_list is passed, only actors in the list will be checked.
- Return — list(carla.Vector3D
- Parameters
  - frame (int) — Frame number.
  - actor_list (int) — List of actor id.

Scene lights

get_scene_light_state(self, light, vehicle_id, frame)
Returns the state of a scene light for a given frame. The light state group will always be carla.LightGroup.None.
- Return — carla.LightState
- Parameters
  - light_id (int) — id of the scene light.
  - frame (int) — Frame number.

Traffic lights

get_traffic_light_state(self, traffic_light_id, frame)
Returns the state of a traffic light at a given frame.
- Return — carla.TrafficLightState.
- Parameters
  - traffic_light_id (int) — id of the traffic light.
  - frame (int) — Frame number.
is_traffic_light_frozen(self, traffic_light_id, frame)
Returns whether or not a traffic light is frozen at a given frame.
- Return — bool
- Parameters
  - traffic_light_id (int) — id of the traffic light.
  - frame (int) — Frame number.
get_traffic_light_elapsed_time(self, traffic_light_id, frame)
Returns the elapsed time of a traffic light at a given frame.
- Return — float
- Parameters
  - traffic_light_id (int) — id of the traffic light.
  - frame (int) — Frame number.
get_traffic_light_state_time(self, traffic_light_id, state, frame)
Returns the maximum time of a specific state of a traffic light at a given frame.
- Return — float
- Parameters
  - traffic_light_id (int) — id of the traffic light.
  - state (carla.TrafficLightState) — id of the actor.
  - frame (int) — Frame number.

Vehicle lights

get_vehicle_lights(self, vehicle_id, frame)
Returns a list with the active lights of a specific vehicle at a given frame.
- Return — list(carla.VehicleLightState)
- Parameters
  - vehicle_id (int) — id of the vehicle.
  - frame (int) — Frame number.
is_vehicle_light_active(self, light, vehicle_id, frame)
Checks whether or not a given vehicle light is active for a vehicle at a specific frame.
- Return — bool
- Parameters
  - light (carla.VehicleLightState) — The light state to be compared with the vehicle.
  - vehicle_id (int) — id of the vehicle.
  - frame (int) — Frame number.