[ntar-workers] On Markers and Bookmarks

[Christian Kreibich]

On Fri, 2005-07-08 at 08:57 -0700, Gianluca Varenni wrote:
>
> > mhmm unfortunately I cannot recall that discussion any more. Note though
> > that I don't mean arbitrary recursive nesting -- just packet blocks
> > contained in a grouping block. This was on the assumption that it might
> > be useful to have a grouping mechanism for a bunch of packets on a finer
> > granularity than sections. So in a sense a GoPBlock is similar to a
> > marker block that is valid until the next marker, with GoPBlocks having
> > the added benefit that you can potentially have end-of-GoP content more
> > naturally.
> 
> ...and the disadvantage that if the capture file is truncated, the entire 
> GoPBlock can be almost unusable (since you don't have the GoPBlock trailer).

I think you would be in no worse a situation than without the GoP block
trailer. Corruption yields a broken final block, whatever it is, likely
a packet block. Corrupted blocks will have to be detected at a "low"
level in the library anyway. I wouldn't put vital information in the
trailer, only stats or whatever your app might willingly put there. Same
for the Section End Block, btw. If it has stats and indexing structures,
yay; if not, too bad, you'll have to seek.

Reconstruction of corrupted files will involve a re-scan of the last
section (as the baseline granularity of the file) anyway. This will be
sped up while valid GoP blocks exist because they can be skipped as a
whole, and slower for the last GoP. Packet counters could be re-
established on the fly there.

Would you feel better about optional end-of-GoP bocks? That might be an
option, though I think Section End Blocks should be required.

I do agree btw that in-situ file restoration is made harder by end-of-
{GoP,Section} blocks. It's still doable though -- you overwrite a bit
more at the end with new valid blocks (needn't be more than 12 bytes
each -- a type and two null length fields) and mark the last block
before that as unusable.

> > Isn't that just the indexing we're discussing here (including Chema's
> > navigation blocks proposal)?
> 
> Not exactly. It's a (missing) feature by which you can
> - ask the offset of a block relative to the section/file
> - reuse such offset to open a specified block (something like 
> "ntar_open_block_at_offset()")

Ah, I see. Thanks.

> I'm still thinking on how to implement it with a clean and yet powerful 
> inteface (one of the problems is trying to open a block in a section that 
> it's different from the one you are actually in. Is it allowed? What are the 
> implications?).

You could wrap the concept of an offset in an opaque handle. In the
handle implementation, include a pointer to the section it belongs to,
and in the function abort if an offset handle is passed in that doesn't
match the session handle. Would that work? Kind of a poor-man's type
checking?

> > I think it doesn't matter too much as long as comparison operations work
> > universally.
> 
> Yes, provided that the user does not try to compare indexing timestamps with 
> the packet timestamps directly...

Yeah, that's true.

Cheers,
Christian.
-- 
________________________________________________________________________
                                          http://www.cl.cam.ac.uk/~cpk25
                                                    http://www.whoop.org