[ntar-workers] On Markers and Bookmarks

[Jose M. Gonzalez]

Just a small detail: 

Christian Kreibich wrote:
> Chema and I just had another discussion about GoP blocks. So it seems
> what we all want is a way to group packets and encode metadata about
> them, without making the application read all the packets (so that they
> could remain compressed as whole, can be skipped efficiently, etc).
> 
> We haven't really agreed on what a GoP block looks like and how to
> encode its size etc, have we? It clearly brings back memories from the
> start of this list but I don't think that discussion converged on a
> final verdict.
> 
> Self-contained GoP blocks
> -------------------------
> 
> What Chema and I would really love is this:
> 
>     0                   1                   2                   3
>     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
>    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>    |                    Block Type (GoP Block)                     |
>    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>    |                      Block Total Length                       |
>    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>    /                                                               /
>    /             Packet blocks, markers, metadata, etc.            /
>    /                                                               /
>    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>    |                       (Padding block)                         |
>    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>    |                      Block Total Length                       |
>    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> 
> A nested sequence of packets (not arbitrarily nested -- no GoP blocks
> inside GoP blocks) with valid length fields covering the entire
> contents. Of course the problem is knowing the length value at the
> beginning of the block before writing the packets.
Actually the word "block" in "Packet block" and "Padding block" is 
misleading. The goal of a GoPBlock is to have the possibility of 
avoiding a per-packet {type, size, ..., type} block structure. This 
is a must if you want to include compressed packets, where 12 bytes 
may be too much, or traces generated with specific hardware that 
cannot be changed to use the mainstream NTAR block structure (I'm 
thinking in DAG cards spitting ERF files). 

There can be different internal formats to organize GoPBlocks (an 
ERF GoPBlock, a GoPBlock following any weird compression system). 


> We like the idea of a final padding block. 
This is also the case outside of a GoPBlock. It's not clear how to write 
the contents of FF markers without the possibility of adding PadBlocks, 
as the distance to a future block is unknown in advance. 

Imagine the following case. I'm writing a trace, and my previous FF 
marker block stats "there is a block beginning in 1000 bytes." I've 
written 900 bytes, so I have only 100 bytes left. The next packet is 
200 bytes long, so I cannot write it right away, as the previous FF 
marker will point to the middle of a packet. The solution is to add 
a 100-byte long PadBlock (actually the inside of the PadBlock is 
100-12 bytes, to make space for the block frame). 

> So an app like tcpdump would
> do the following:
> 
> (1) Create a new blank file (or start with an existing one) by appending
> a SHB and interface info etc, anything it needs to write out before
> writing actual packets.
> 
> (2) Write the beginning of a GoP block with a fixed size N for the
> block, say 5MB. I'd say any size within which sequential scanning is
> deemed appropriate for lookups is a good value.
> 
> (3) Write packet blocks for all the packets as they are captured.
> 
> (4) Towards the end of N, optionally write a stats block with additional
> info not available before the packets are seen (such as their number).
> 
> (5) Pad with a padding block to size N.
> 
> (6) Terminate GoP Block.
> 
> (7) Start over at (2).
> 
> (8) If user Ctrl-C's the process or when max number of packets is
> reached, terminate the current GoP block, rewind to the beginning of the
> block, fill in correct length, return to end of file, and (pending
> further discussion) for example terminate with a Section End Block
> containing indexing structures etc.
> 
> A trace-reading app would have to be liberal in what it accepts and
> essentially any sequence of packet blocks is fine. GoP blocks, if
> present, help seeking but if they're not there the trace is still
> perfectly valid.
> 
> Regarding compression etc: details on the contained sequence of blocks
> could be stored in a sequence of options after the initial length field
> of the block.
> 
> Markers
> -------
> 
> Here a GoPBlock is a small-size block, just another plain block. 
> 
>     +-----+-------+-------+-------+-------+-----+-------+-----+-- 
>     | GoP | block | block | block | block | ... | block | GoP |...
>     +-----+-------+-------+-------+-------+-----+-------+-----+-- 
> 
> Simple markers would effectively represent "soft" SHBs. These would be
> more lightweight and could still encode metadata that would be valid for
> the future blocks up to the next marker block.
> 
> We much prefer the former suggestion. What do you guys think?
> 
> Cheers,
> Christian.
> -- 
> ________________________________________________________________________
>                                           http://www.cl.cam.ac.uk/~cpk25
>                                                     http://www.whoop.org
> 
> 
> _______________________________________________
> ntar-workers mailing list
> ntar-workers at winpcap.org
> https://www.winpcap.org/mailman/listinfo/ntar-workers