[ntar-workers] On Markers and Bookmarks

[Christian Kreibich]

On Tue, 2005-07-12 at 11:46 -0700, Christian Kreibich wrote:
>
> > In any case, in order to implement the GoP stuff, we should definitely add a 
> > spec for such blocks in the draft. Christian, would it be possible for you 
> > to write down a sort of summary of the GoP block, so that we can discuss all 
> > the details and refine it?
> 
> Sure thing, let's just make sure the discussion has settled on the same
> thing. You're welcome to send me the sources of the draft, otherwise
> I'll just send you some raw text.

Chema and I just had another discussion about GoP blocks. So it seems
what we all want is a way to group packets and encode metadata about
them, without making the application read all the packets (so that they
could remain compressed as whole, can be skipped efficiently, etc).

We haven't really agreed on what a GoP block looks like and how to
encode its size etc, have we? It clearly brings back memories from the
start of this list but I don't think that discussion converged on a
final verdict.

Self-contained GoP blocks
-------------------------

What Chema and I would really love is this:

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                    Block Type (GoP Block)                     |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                      Block Total Length                       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   /                                                               /
   /             Packet blocks, markers, metadata, etc.            /
   /                                                               /
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                       (Padding block)                         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                      Block Total Length                       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

A nested sequence of packets (not arbitrarily nested -- no GoP blocks
inside GoP blocks) with valid length fields covering the entire
contents. Of course the problem is knowing the length value at the
beginning of the block before writing the packets.

We like the idea of a final padding block. So an app like tcpdump would
do the following:

(1) Create a new blank file (or start with an existing one) by appending
a SHB and interface info etc, anything it needs to write out before
writing actual packets.

(2) Write the beginning of a GoP block with a fixed size N for the
block, say 5MB. I'd say any size within which sequential scanning is
deemed appropriate for lookups is a good value.

(3) Write packet blocks for all the packets as they are captured.

(4) Towards the end of N, optionally write a stats block with additional
info not available before the packets are seen (such as their number).

(5) Pad with a padding block to size N.

(6) Terminate GoP Block.

(7) Start over at (2).

(8) If user Ctrl-C's the process or when max number of packets is
reached, terminate the current GoP block, rewind to the beginning of the
block, fill in correct length, return to end of file, and (pending
further discussion) for example terminate with a Section End Block
containing indexing structures etc.

A trace-reading app would have to be liberal in what it accepts and
essentially any sequence of packet blocks is fine. GoP blocks, if
present, help seeking but if they're not there the trace is still
perfectly valid.

Regarding compression etc: details on the contained sequence of blocks
could be stored in a sequence of options after the initial length field
of the block.

Markers
-------

Here a GoPBlock is a small-size block, just another plain block. 

    +-----+-------+-------+-------+-------+-----+-------+-----+-- 
    | GoP | block | block | block | block | ... | block | GoP |...
    +-----+-------+-------+-------+-------+-----+-------+-----+-- 

Simple markers would effectively represent "soft" SHBs. These would be
more lightweight and could still encode metadata that would be valid for
the future blocks up to the next marker block.

We much prefer the former suggestion. What do you guys think?

Cheers,
Christian.
-- 
________________________________________________________________________
                                          http://www.cl.cam.ac.uk/~cpk25
                                                    http://www.whoop.org