|
The pages in this section show how to launch and use ObjectVideo's Vitual Video Tool.
The mod is designed to be highly flexible, so that users can interact
with the simulated cameras through a variety of mechanisms ranging from
existing media clients (without any additional programming) to complex custom applications.
In addition to the DirectShow filter
and C Library provided with the mod, we also detail
the underlying network protocol
so that users can imlement video clients in most major languages on a variety of platforms.
Half-life 2 uses a client/server architecture, where the server maintains the overall game
state and the clients implement rendering and user interface functionality. The Virtual
Video Mod introduces two mini-servers into the standard player client, as illustrated in
the figure above. Client applications
interact with these servers over a TCP/IP network, and can run on the same machine as the Half Life
engine, or on different machines and even different platforms.
The servers are described below:
- Camera Server: The camera server maintains a set of virtual cameras that render
the world from a particular point of view.
Clients connecting to this server can create new virtual cameras or stream video from existing cameras,
modify camera parameters and set special
modes of operation such as PTZ control and automatic scanning. Video frames are rendered at a user-specified fixed rate,
and the latest rendered frame can be retrieved on demand. The server maintains independent settings for each virtual camera.
- PTZ Server: Each connection to this server can create a new virtual PTZ controller
or connect to an existing controller
to direct the pan, tilt and zoom of active virtual cameras. A virtual camera must enable
PTZ control and specify a controller ID through the Camera Server before it can accept PTZ commands.
Creating a PTZ controller with a matching controller ID then allows the camera to be driven actively.
This architecture allows for multiple independent active cameras.
The reason for having two servers is to simulate the physical distinction between the
camera and the PTZ active head. Both servers support multiple connection from
video and PTZ clients. While there is usually with a one-to-one mapping
between active cameras and PTZ controllers, other mappings are possible to simulate for
example stereo PTZ cameras. In applications with only static cameras, it is not
necessary to make any connection to the PTZ Server. Three mechanisms are provided
to interact with the Camera and PTZ Servers:
- DirectShow Filter: Allows applications to interact
with the Camera Server using the DirectShow API. This is the highest level interface, and can also be used
with existing DirectShow applications such as Windows Media Player. The advantage of this layer is that
applications can easily switch the DirectShow stream from virtual video to a real camera without significant
code changes.
- C Library: A static library providing a set of C functions to interact with both the
Camera Server and PTZ Server. Windows Sockets 2 must be available to use this library.
This is the highest level interface to the PTZ Server.
- Network Protocol: Documentation is provided that describes the underlying network
protocol implemented by the C Library and DirectShow filter. This is the lowest level interface
to the Virtual Video Tool and allows users to write client applications in any language and on any
platform that supports network sockets.
The above figure provides a more detailed illustration of the interaction between clients, cameras and
PTZ controllers. Clients connected to the Camera Server
may create new cameras or stream video from existing cameras. Virtual cameras are identified by a
unique camID. A default camera with camID = 0 is always created when the game client is launched,
and all new connections connect to this virtual camera. Camera clients can switch to a different
camera by sending a new camID to the Camera Server (see the camID option in
DirectShow Filter Usage, or the
CamSetID C function and Set Camera ID network command).
If a camera identified by camID already exists the
client is connected to the existing camera, otherwise a new virtual camera corresponding to the
requested camID is created. Newly created cameras persist until the game client is closed, even
if all clients disconnect from the camera (to minimize processing overheads, only cameras with
active connections will actually render frames). In the above figure, camera client 1 remains connected
to the default camera with camID = 0, while clients 2 and 3 are both connected to the newly created
camera with camID = 6. This camera may have been created by either client 2 or 3 depending on which
connected first, but both clients have equal control over the camera. Changes to the camera
settings imposed by either client will affect the video streams received by both clients.
Similarly, each connection to the PTZ Server can create a new virtual controller or connect to an
existing controller. Clients must assign a unique identifier ptzID to each new controller at the time of creation
(see the ptzID option in the
PtzSetID C function and Set PTZ ID network command).
A controller with ptzID = 0 is always created by the PTZ Server
and is the default controller initially assigned to new client connections.
In the example shown above, PTZ client 1 remains connected to the default controller with ptzID = 0,
while client 2 creates a new controller with the identifier ptzID = 5. Once created, PTZ controllers
exist until the game is closed and all setting persist between client disconnections and reconnections.
To create an active camera, the camera client must enable the PTZ control flag and set a valid ptzID
on the camera (see the ptzcontrol and ptzID option in DirectShow Filter Usage,
or the
CamSetType C function and Set Camera Type network command).
For example, camera client 1 in the above figure has enabled PTZ control and set and ptzID = 5 for
camera 1, so the camera will accept PTZ commands from PTZ client 2. Camera 2 has PTZ control disabled
and so will behave as a static camera. Neither camera specifies ptzID = 0, so the commands sent
from PTZ client 1 have no effect.
In addition to the network communication layers described above, the Virtual Video Mod also introduces
several new commands to the Half-Life 2 developer console. These commands can be used to query and configure the
Camera and PTZ Servers from within the game client, and also to add noise, interference and other realistic effects
to the rendered frames. For more information, see Console Commands.
Finally, Virtual Video has the capability to generate target and pixel-level ground truth for each
frame, as described in Ground Truth.
This data can be used to evaluate the performance of detection, tracking and other computer vision algorithms.
A typical use case for the Virtual Video tool involves the following steps:
- Launch the Virtual Video Mod via Steam.
- Load a virtual environment/scenario (map) within the game.
- Use console commands to configure image quality settings and camera effects.
- Launch your image processing application (on the same or different machine)
and connect to the mod through the DirectShow filter, C Library calls or network socket.
- Create and configure camera(s) to obtain the desired view(s).
- For active vision, create PTZ controller(s) and configure cameras for PTZ control.
- Retrieve and process rendered frames.
- For active vision, send PTZ control commands back to the mod
through the C Library calls or network socket.
The following sections describe the usage of the various components in greater detail:
|
