TAP as a Service (TAPaaS)

Problem Description

Performing network protocol review, checking firewall rules and / or performing network security audit is part of the daily job of the system administrator. To be able to provide this functionality in the cloud, an idea to create a tap-as-a-service was born. For example when enabled and configured it will allow the system administrator to run a Snort IDS on his tenant network.

In short, we want to create a mechanism to forward traffic for example from one VM to another. When packet will be forwarded, the original value of source and target ip/ports information will not be altered and the system administrator will be able to run, for example tcpdump, on the target VM to trace these packets. The administrator, will have to enable a promiscuous mode on the target VM network card to be able to see these packets. [1]

Currently this document concentrates on tapping traffic coming in and out of VMs.

Additional kinds of ports will not be supported in the first version of the spec.

  1. Tapping DHCP port
  2. Tapping router port
  3. Tapping floating ip traffic

Terminology and API

The API [2] and SPEC [3] define two objects:

  1. TapService
  2. TapFlow

TapService represents the port to which the mirrored traffic is delivered. For example this port can be attached to VM where a Snort IDS running.

'tap_services': {
    'id': {'allow_post': False, 'allow_put': False,
           'validate': {'type:uuid': None}, 'is_visible': True,
           'primary_key': True},
    'tenant_id': {'allow_post': True, 'allow_put': False,
                  'validate': {'type:string': None},
                  'required_by_policy': True, 'is_visible': True},
    'name': {'allow_post': True, 'allow_put': True,
             'validate': {'type:string': None},
             'is_visible': True, 'default': ''},
    'description': {'allow_post': True, 'allow_put': True,
                    'validate': {'type:string': None},
                    'is_visible': True, 'default': ''},
    'port_id': {'allow_post': True, 'allow_put': False,
                'validate': {'type:uuid': None},
                'is_visible': True},
}

TapFlow represents the port from which the traffic needs to be mirrored. It can be a port associated with VM on another CN.

'tap_flows': {
    'id': {'allow_post': False, 'allow_put': False,
           'validate': {'type:uuid': None}, 'is_visible': True,
           'primary_key': True},
    'tenant_id': {'allow_post': True, 'allow_put': False,
                  'validate': {'type:string': None},
                  'required_by_policy': True, 'is_visible': True},
    'name': {'allow_post': True, 'allow_put': True,
             'validate': {'type:string': None},
             'is_visible': True, 'default': ''},
    'description': {'allow_post': True, 'allow_put': True,
                    'validate': {'type:string': None},
                    'is_visible': True, 'default': ''},
    'tap_service_id': {'allow_post': True, 'allow_put': False,
                       'validate': {'type:uuid': None},
                       'required_by_policy': True, 'is_visible': True},
    'source_port': {'allow_post': True, 'allow_put': False,
                    'validate': {'type:uuid': None},
                    'required_by_policy': True, 'is_visible': True},
    'position': {'allow_post': True, 'allow_put': False,
                  'validate': {'type:values': position_enum},
                  'is_visible': True}
    'direction': {'allow_post': True, 'allow_put': False,
                  'validate': {'type:values': direction_enum},
                  'is_visible': True}
}

The DirectionEnum states whether the TAP is attached to the ingress, egress, or both directions of the VM port.

direction_enum = ['IN', 'OUT', 'BOTH']

The PositionEnum states where to place the tapping point. It is defined as follows:

position_enum = ['VNIC', 'PORT']

VNIC specifies that the packet should be intercepted close to VM, while PORT specifies that packet should be intercepted on the network level.

In case we are talking about tapping outgoing packets:

  1. VNIC is before security group firewall.
  2. PORT is after security group firewall.

In case we are talking about tapping incoming packet:

  1. VNIC is after the security group firewall.
  2. PORT is before security group firewall.

Multiple TapFlow instances can be associated with a single TapService instance. I.e. one Snort VM can check traffic of multiple virtual servers.

Tap packet forwarding table

After the packet is mirrored, it will be forwarded to the new packet forwarding table (TAP_FORWARD_TABLE).

Packets can come from local CN and external CN. To minimize number of changes, we propose to create a new table to handle mirrored packets.

This table may not be highly optimized, but improves the modular design.

Each mirrored packet, coming either from the same CN, or from external CN, will have a marked tunnel id.

In case the packet coming from the local or external CN needs to be forwarded locally (to the same CN), the following rules will be applied:

filter: tun_id=DEST_TUN_ID action:output:DEST_LOCAL_PORT

In case the packet is coming from the local CN and should be forwarded to external CN:

filter: tun_id=DEST_TUN_ID action:output:OVERLAY_NET_PORT
Field Name Description
DEST_TUN_ID a tunnel number will specify a destination VM
DEST_LOCAL_PORT destination OVS port number (in case it is on same CN)
OVERLAY_NET_PORT packet will be forwarded to other CN

Assigning tunnel id for each TapService

Each TapService will have a unique id that corresponds to the overlay network tunnel id.

By default, each network has it’s own id called segment id allocated from neutron segment pool. A naive approach will be to assign a unique id to be used for TapService from this pool but we concern that admins setup network vnis and they expects number of network to be supported.

More advance solution will be to create a new segment pool to be used exclusively for TapServices. This new network pool should not coincide with the one used in neutron.

In case we run out of free ids in the new segment pool, as a fallback solution, we will assign segment id from the neutron segment pool.

Packet mirroring

In order to support Tap as a Service, a TapFlow packet mirroring rule can be installed in multiple locations, relative to the port:

  1. Tap rule on output
  2. Tap rule on input
  3. Both

In addition, tapping flows can be installed before and after SG firewall rules.

Tap on the output

Packet can be mirrored before or after the security group firewall check.

In theory, we can add additional table and / or modify existing rules to allow mirroring.

To make the design more modular, it was decided to add new tables instead of altering existing rules.

Tap position is BEFORESG

This configuration mode actually implies that packets will be mirrored without filtering by security group.

Change table=1 (EGRESS_PORT_SECURITY_TABLE) to be table=2 and install our tap rules in table=1.

In new table=1 we will add the following rules:

Filter1:in_port=6 Actions:resubmit(,2),
  $DEST_TUN_ID->tun_id,goto_table:TAP_FORWARDING
Filter2:any Actions:resubmit(,2)

In case, the source port traffic should be mirrored to multiple TapService:

Filter:in_port:6 Actions:resubmit(,2),
  $DEST_TUN_ID1->tun_id,goto_table:TAP_FORWARDING,
  $DEST_TUN_ID2->tun_id,goto_table:TAP_FORWARDING

Tap position is AFTERSG

After packets pass the firewall rules they arrive to the table=9 (SERVICES_CLASSIFICATION_TABLE).

We should move all rules from table=9 to a new table (e.g. table=10).

All other tables IDs should change accordingly.

We will add new rules here:

Filter:in_port:6 Actions:resubmit(,10),
  $DEST_TUN_ID1->tun_id,goto_table:TAP_FORWARDING

In case the source port traffic should be mirrored to multiple TapService:

Filter:in_port:6 Actions:resubmit(,10),
  $DEST_TUN_ID1->tun_id,goto_table:TAP_FORWARDING,
  $DEST_TUN_ID2->tun_id,goto_table:TAP_FORWARDING

Tap on the Input

Tap position is AFTERSG

After passing the firewall, packets are forwarded to table=78 (INGRESS_DISPATCH_TABLE).

We should move all rules from table=78 to a new table (e.g. table=79).

We will add new rules in table=78:

Filter:reg7=0x8 Actions:resubmit(,79),
  $DEST_TUN_ID1->tun_id,goto_table:TAP_FORWARDING

In case, the source port traffic should be mirrored to multiple TapService:

Filter:in_port:6 Actions:resubmit(,79),
  $DEST_TUN_ID1->tun_id,goto_table:TAP_FORWARDING,
  $DEST_TUN_ID2->tun_id,goto_table:TAP_FORWARDING

Tap position is BEFORESG

This configuration mode actually implies that packets will be mirrored without filtering by security group.

Before the packets pass the firewall rules they arrive to the table=77 (INGRESS_SECURITY_GROUP_TABLE).

We should move all rules from table=77 to a new table (e.g. table=78) and all other tables IDs should be updated appropriately.

We will add new rules here (table=77)

Filter:in_port:6 Actions:resubmit(,78),
  $DEST_TUN_ID1->tun_id,goto_table:TAP_FORWARDING

In case, the source port traffic should be mirrored to multiple TapService:

Filter:in_port:6 Actions:resubmit(,78),
  $DEST_TUN_ID1->tun_id,goto_table:TAP_FORWARDING,
  $DEST_TUN_ID2->tun_id,goto_table:TAP_FORWARDING,

Receiving mirrored packets from other CNs

To be able to forward packets received from other CNs on each CN that has a TapService we will add relevant rules to forward rules to a TAP_FORWARDING table.

We will add new rule in table=0:

Filter:tun_id=$DEST_TUN_ID1 Actions:goto_table:TAP_FORWARDING
Filter:tun_id=$DEST_TUN_ID2 Actions:goto_table:TAP_FORWARDING

Database Changes

In order to support TapServices the following table will be added to distributed nosql database.

Attribute Name Description
key record identify
topic tenant ID
port_id port id of the destination VM
segmentation_id overlay network distinguishing tunnel id

The following fields were omitted here:

  • name
  • description

In order to support TapFlows the following table will be added to distributed nosql database.

Attribute Name Description
key record identify
topic tenant ID
tap_service_id id of the destination TapService
source_port_id port id of the tapped machine
position enum [‘VNIC’, ‘PORT’]
direction_enum enum [‘IN’, ‘OUT’, ‘BOTH’]

The following fields were omitted here:

  • name
  • description

List of relevant OpenFlow tables

INGRESS_CLASSIFICATION_DISPATCH_TABLE = 0
EGRESS_PORT_SECURITY_TABLE = 1
SERVICES_CLASSIFICATION_TABLE = 9
INGRESS_SECURITY_GROUP_TABLE = 77
INGRESS_DISPATCH_TABLE = 78