[Winpcap-users] is winpcap appropriate formygigabitethernetdevice?

[Gianluca Varenni]

----- Original Message ----- 
From: "max reason" <maxreason at maxreason.com>
To: <winpcap-users at winpcap.org>
Sent: Friday, September 05, 2008 1:25 AM
Subject: RE: [Winpcap-users] is winpcap appropriate 
formygigabitethernetdevice?


>
> I was almost certain one SATAII disk wouldn't keep up, and almost
> certain two disks would work with just enough margin to be reliable.
> However, you are the second person to mention RAID - I don't understand.
> As long as the computer has two disk controllers (which many do),
> I assumed my application could alternately write to the two drives
> and expect everything to work.  Or is there something special about
> one or more of the RAID modes that I don't know about?
>

Well, one of the reasons why you want to use RAID is that you can create a 
single file, instead of separate ones and then merge them.

> Is your suggestion to "disable file system caching" while dumping
> to disk related to the CPU cache concerns I expressed in my reply
> to Zafer Savas?  Or do you refer to the memory cache on the disk?

It's not specifically related to CPU caching. And neither to the memory 
cache on the disk, which is basically useless if you are dumping a 
continuous stream of packets to disk.
The reason why you want to disable the file system cache when dumping 
packets to disk at this speed is that it doesn't help at all. The file 
system cache helps (a lot) when you are basically accessing the disk in a 
burst mode (for example you are saving or reading a small file). The file 
system cache gives you the illusion that the disk is faster than it really 
is, by caching the write operation at a later point in time. Of course at 
the expense of using your RAM (all of it) for the cache. Try to dump a very 
big file at the fastest possible rate on a disk with file system caching 
enabled (i.e. use the normal fopen fwrite and such). If you look at the task 
manager, you will see that the system cache will grow up to use almost all 
your memory. When you dump in a sequential way hundreds of megabytes to disk 
at gigabit rate, you don't care about caching. You would care only if the 
amount of data to dump on a single file is less than your amount of RAM. In 
that case the cache will be useful as it will store temporarily all your 
data even if the disk cannot keep up. More or less like using a RAMdisk.

GV

>
> In those applications where the speed of the computer system and
> peripherals are pushed to the limit, my $250 ~ $500 device replaces
> $12,000 ~ $38,000 alternatives!  Therefore, it is entirely reasonable
> to *require* the computer is doing nothing else (of significance)
> during high-speed image capture/display/save.  This also means it
> is perfectly okay to put the computer into special configurations
> while the capture process is active.  So, it is perfectly okay to
> lock any quantity of memory that doesn't interfere with the OS or
> peripheral controllers doing their jobs.  Therefore, the kind of
> tips you give are relevant and very appreciated.
>
> Thanks much.  And yes, your and Zafer have made my day nice indeed!
> Well, and winpcap of course!  :-)
>
> Max
>
>
>> To add something to it.  If your main objective is dumping to disk,
>> by far the most important things are:
>>
>>   1. use RAID. A single disk CANNOT keep up.
>>   2. you can also use one single thread and use overlapped i/o to
>>      dump to disk.  And disable file system caching while dumping
>>      to disk (you do that with the flags of CreateFile).
>>
>> Creating big buffers in user mode is not always useful. Unless you
>> lock all that amount of memory so that it's in your working set
>> (i.e. it's not swapped out), the risk is that that big amount of
>> memory gets swapped in/out to disk when you use it (because the OS
>> tries to trim your working set size).
>>
>> I played a lot with these things recently while developing TurboCap,
>> and these were the major bottlenecks when dumping to disk.
>>
>> Have a nice day
>> GV
>
> _______________________________________________
> Winpcap-users mailing list
> Winpcap-users at winpcap.org
> https://www.winpcap.org/mailman/listinfo/winpcap-users