[ntar-workers] On Markers and Bookmarks

[Gianluca Varenni]

----- Original Message ----- 
From: "Christian Kreibich" <christian at whoop.org>
To: "NTAR workers" <ntar-workers at winpcap.org>
Sent: Monday, July 11, 2005 6:14 PM
Subject: Re: [ntar-workers] On Markers and Bookmarks


> On Thu, 2005-07-07 at 22:42 -0700, Loris Degioanni wrote:
>>
>> > Isn't that just the indexing we're discussing here (including Chema's
>> > navigation blocks proposal)?
>>
>> No, I think what Gianluca is talking about is really going back and
>> forth in the file: fseek and friends. Implementing them in a portable
>> and 64-bit-friendly way (since the specification requires 64-bit
>> support) is really complicated. Moreover, fseek seems to be very slow
>> (slower than reading everything) on several platforms.
>
> Ewww that sucks. :(

Yes, or better, I think that short seeks (<100bytes) badly affect the 
buffering mechanism of the "FILE" calls (fread/fwrite/fseek) that are 
currently used in ntar. In particular, during the weekend I performed some 
(incomplete) tests to see what happens with fast SCSI disks on windows and 
linux, I plan to post the results in a day or so. It could be a good 
starting point for future optimizations...

>
>> > Imho, yes. If the trace gets big, time becomes the primary mechanism 
>> > for
>> > selections. Even if you're indexing semantically different items (say
>> > markers identifying the beginning of malicious flows, etc) you'll 
>> > likely
>> > still use time to navigate among multiple instances. I'd love to hear
>> > counterexamples though.
>>
>> I kind of like the idea of using time to navigate inside traces, it
>> could have many cool applications. The other important indexing method
>> in my opinion is the number of packets: you probably want to know how
>> many packets you skip if you jump to the next GoPBlock.
>
> Yeah, that'd be nice, also for finding the total number of packets in a
> trace faster than by scanning all the packet blocks.
>
>> I think for
>> example about tools like Ethereal, which could reserve the right space
>> in the list of packets with a quick scan, without even reading the 
>> packets.
>> By the way, would it make sense to report the number of packets of a
>> GoPBlock in the following GoPBlock marker? This would prevent us from
>> having to go back in the file.
>
> I like that. I think I have a preference for end-of-block blocks, just
> like with the Section End Block for sections that I proposed earlier.
> Just end the GoPBlock with a separate block that can contain stats etc.
>
>> Christian, if you are still interested in working on this, we could come
>> up with a more precise definition, and then write it down in the specs.
>> At that point, we would be ready to start the implementation. What do
>> you think?
>
> Sure, my offer of around 1 day/week stands. I've started to look at the
> ntar code.

That's really nice! I would really like to have someone reviewing that code 
(and give me an opinion on the API and the general architecture, I'm not 
sure of some architectural details in that library, especially the plugin 
stuff).

In any case, in order to implement the GoP stuff, we should definitely add a 
spec for such blocks in the draft. Christian, would it be possible for you 
to write down a sort of summary of the GoP block, so that we can discuss all 
the details and refine it?

Have a nice day
GV



>
> Best,
> Christian.
> -- 
> ________________________________________________________________________
>                                          http://www.cl.cam.ac.uk/~cpk25
>                                                    http://www.whoop.org
>
>
> _______________________________________________
> ntar-workers mailing list
> ntar-workers at winpcap.org
> https://www.winpcap.org/mailman/listinfo/ntar-workers