[ntar-workers] On Markers and Bookmarks

[Christian Kreibich]

On Mon, 2005-07-11 at 22:43 -0700, Gianluca Varenni wrote:
>
> > Would you feel better about optional end-of-GoP bocks? That might be an
> > option, though I think Section End Blocks should be required.
> 
> Actually, I'm realizing now that maybe we have different concepts of 
> grouping. I imagined a *strict* grouping of packets into GoP (i.e. a packet 
> block cannot live outside a GoP, thus strict nesting). Instead maybe your 
> vision is a more "relaxed" one: a section can groups packets using markers, 
> using a Begin_GoP block and an End_GoP block. A good (i.e. easy to navigate) 
> file will have packets surrounded by Begin/End_GoP blocks every now and 
> then, but a file without Begin/End_GoP blocks, or with packets "floating" 
> outside those blocks, is still well-formed.
>
[snip sketch ]
> 
> Is this what you mean? In any case, I really love this idea.

Yes, exactly. Strict GoP blocks are ideal, but packet blocks without
surrounding GoP blocks would of course still be valid. After all, a
normal tcpdump-like sequential scan would just go through block by block
and report every packet block found. I believe the trace-creating app
could create the "strict" layout without much overhead though?

> > I do agree btw that in-situ file restoration is made harder by end-of-
> > {GoP,Section} blocks. It's still doable though -- you overwrite a bit
> > more at the end with new valid blocks (needn't be more than 12 bytes
> > each -- a type and two null length fields) and mark the last block
> > before that as unusable.
> 
> The only real issue I see in in-situ restoration is alignment. The spec 
> clearly states that every block should be aligned to 32bit boundaries. We 
> can have ntar (or any other implementation of pcap-ng) ignore this detail, 
> create non-aligned "garbage" blocks to fix the truncated captures, and tweak 
> the code to deal with "garbage" blocks differently (by spec, the block 
> header contains the "useful" block length, that can be unaligned to 32 bits, 
> but the actual block size is ALIGN_TO_32BITS(block_unaligned_size)), but in 
> any case you have a file that cannot be made well-formed.

Mhmm that is true. Why exactly is 32bit alignment required? Speedier
mmapped access? 

> > You could wrap the concept of an offset in an opaque handle. In the
> > handle implementation, include a pointer to the section it belongs to,
> > and in the function abort if an offset handle is passed in that doesn't
> > match the session handle. Would that work? Kind of a poor-man's type
> > checking?
> 
> It could work. The advantage of offsets is that they are integral values, 
> usually returned by value (so no dynamic memory allocation is involved), or 
> however allocated by the caller (i.e. you have something like 
> offset=ntar_get_block_offset() or result=ntar_get_block_offset(..., 
> &offset,...)), while an opaque handle (being opaque) is allocated by the 
> callee, so you have something like offset_handle = 
> ntar_get_block_offset_handle() and then 
> ntar_close_offset_handle(offset_handle). If your app maintains several (= 
> possibly thousands) handles, it can be troublesome (lots of mallocs or 
> handle pools, possible leaks because the app forgets to close the handles).

It's a big tradeoff -- if you give users bare values, they can screw
them up. If you give them opaque handles, they can screw up memory
management. I'm personally fine with either, tending to prefer the
offset handles. In my experience it's not that hard to have allocation
and deallocation of such handles together closely in the code.

> In general, even a simple offset can be tested: when you open a section, you 
> know its offset in the file and the length (well, the section length is 
> optional), so you can easily check if
> section_start < requested_block_offset < section_start + section_length.

Yeah. You just don't know if it's an aligned offset.

Cheers,
Christian.
-- 
________________________________________________________________________
                                          http://www.cl.cam.ac.uk/~cpk25
                                                    http://www.whoop.org