[ntar-workers] On Markers and Bookmarks

[Gianluca Varenni]

----- Original Message ----- 
From: "Christian Kreibich" <christian at whoop.org>
To: "NTAR workers" <ntar-workers at winpcap.org>
Sent: Thursday, July 07, 2005 11:06 AM
Subject: Re: [ntar-workers] On Markers and Bookmarks


> Hey Gianluca,
>
> On Mon, 2005-07-04 at 19:21 -0700, Gianluca Varenni wrote:
>>
>> > So what's the verdict on the PacketsBlock/GoPBlock idea?
>>
>> My verdict would be to use markers and sections, and not make use of
>> GoPBlocks. Actually, I'm still thinking of what is the best approach. In
>> general GoPBlocks introduce a real hierarchy of blocks in the file 
>> format,
>> and a hierarchical file format was discarded last year when the pcap-ng 
>> was
>> first discussed.
>
> mhmm unfortunately I cannot recall that discussion any more. Note though
> that I don't mean arbitrary recursive nesting -- just packet blocks
> contained in a grouping block. This was on the assumption that it might
> be useful to have a grouping mechanism for a bunch of packets on a finer
> granularity than sections. So in a sense a GoPBlock is similar to a
> marker block that is valid until the next marker, with GoPBlocks having
> the added benefit that you can potentially have end-of-GoP content more
> naturally.

...and the disadvantage that if the capture file is truncated, the entire 
GoPBlock can be almost unusable (since you don't have the GoPBlock trailer).

>
>> I think that the idea of the indexing block is worth more discussion, 
>> maybe
>> before trying to implement it; moreover, I'm still working quite heavily 
>> on
>> the library to clean it and offer some missing features that are needed 
>> for
>> these new block types (in particular, something to allow random access to
>> the blocks of a file).
>
> Isn't that just the indexing we're discussing here (including Chema's
> navigation blocks proposal)?

Not exactly. It's a (missing) feature by which you can
- ask the offset of a block relative to the section/file
- reuse such offset to open a specified block (something like 
"ntar_open_block_at_offset()")

I'm still thinking on how to implement it with a clean and yet powerful 
inteface (one of the problems is trying to open a block in a section that 
it's different from the one you are actually in. Is it allowed? What are the 
implications?).

>
>> Some random issues/ideas/thoughts that came to my mind in the last days:
>> - do we really need markers? can the indexing block can point to packet
>> blocks?
>
> I believe yes, as long as the indexing structure in the end just points
> to file offsets (relative to section starts), it can point to pretty
> much anything.
>
>> - we should remember that two ntar files can be concatenated using "cat". 
>> As
>> a consequence, the file offsets should be relative.
>
> Yep.
>
>> - can the indexing block index block in multiple sections? There can be 
>> byte
>> order and interface id issues.
>
> I'd really really really keep sections self-contained.
>
>> - Is the timestamp information to seek in the file enough for the 
>> majority
>> of apps?
>
> Imho, yes. If the trace gets big, time becomes the primary mechanism for
> selections. Even if you're indexing semantically different items (say
> markers identifying the beginning of malicious flows, etc) you'll likely
> still use time to navigate among multiple instances. I'd love to hear
> counterexamples though.
>
>> - how is the timestamp represented? pcap-ng does not have a fixed
>> representation for the timestamp precision: the interface description 
>> block
>> contains a specific option that tells the precision of the timestamps for
>> the packets captured on that interface. By default it's microseconds.
>
> I think it doesn't matter too much as long as comparison operations work
> universally.

Yes, provided that the user does not try to compare indexing timestamps with 
the packet timestamps directly...

Have a nice day
GV



>
> Cheers,
> Christian.
> -- 
> ________________________________________________________________________
>                                          http://www.cl.cam.ac.uk/~cpk25
>                                                    http://www.whoop.org
>
>
> _______________________________________________
> ntar-workers mailing list
> ntar-workers at winpcap.org
> https://www.winpcap.org/mailman/listinfo/ntar-workers