[Winpcap-users] Circular buffer

[Gianluca Varenni]

----- Original Message ----- 
From: "Ioan Popescu" <ipopescu at dataq.com>
To: <winpcap-users at winpcap.org>
Sent: Friday, June 23, 2006 10:09 AM
Subject: Re: [Winpcap-users] Circular buffer


> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
>
> Loris Degioanni wrote:
>> 2. According to the documentation, the circular buffer should
>> overwrite the
>> older packets with the latest "wire" packets while at the same time
>> serving
>> the oldest available packets to the application. Is my understanding
>> correct?
>>
>>> The driver overwrites a packet only after it's been "consumed" by the
>>> application. If the application is stuck, the driver starts dropping
>>> packets when it reaches the tail of the buffer.
>
> So these "purposely" dropped packets are reported in the pcap_stat
> structure? Is there a way to reset these stats? Besides closing and
> reopening the adapter. Although, I could keep track of when I check it and
> simply subtract off what I consider "handled".
>
>> 3. I have noticed something peculiar related to kernel memory size and
>> CPU
>> usage. As I increase the kernel (and user) buffer, the % CPU usage of my
>> test application goes up. I'm testing this using Task Manager. The test
>> application stays the same, it uses a command line argument to set the
>> memory size. It also works on the same set of data. What reason's might
>> there be for such behavior?
>>
>>> How big is the buffer that you set?
>>> The driver allocates this buffer from the nonpaged memory pool, and
>>> setting it to a very big size could impact on the OS performance. Other
>>> than that, the only reason why changing the size of the buffers could
>>> slow your application down is caching issues.
>
>
> The effect is shown over a range of sizes. I've tried values from 100,000 
> to
> 50,000,000. The effect is "skewed" depending on the machine, but it's 
> still
> there. My first thought was that more processing needed to be done to
> "handle" the larger amount of memory, but I've not yet come up with any 
> good
> ideas as to what might cause such a thing.

The driver does not do any more processing for bigger buffers: apart from 
some other details of no importance here, we basically just store a pointer 
to the buffer, and two counters of head and tail in the buffer. It doesn't 
matter much if the buffer is big or small.

As Loris said changing buffer size could cause performance issues because of 
caching. There's also another thing that should be taken into consideration: 
the amount of data transferred from the kernel buffer to the user buffer at 
every ReadFile system call. If the buffer is too small, you will probably 
have a higher packet loss. After a certain threshold, you will probably stop 
losing packets, and if your application processes packets at a quite 
constant rate, the amount of bytes transferred in the read operation is more 
or less constant, so at user level you will always use the same bytes in the 
user buffer. In the kernel, since the buffer is circular, you will continue 
using all the kernel buffer bytes in a sequential way, so in this case 
caching can be the bottleneck. Last but not least, the value of mintocopy 
and timeout you are currently using can affect the overall performance.

This is just to give you an idea of all the possible reasons why changing 
the buffer size can impact on performance. And as you've seen, a bigger 
buffer doesn't mean better performance at all...

Have a nice day
GV


>
> I understand your reasoning for large buffers, but why is the effect
> noticeable even at smaller ranges? I'll try to see if I can come up with a
> good size at which an "outlier" occurs.
>
> Is there any reason for me to test the kernel and user buffers
> independently? Up until now, I've set both of them to the same values.
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.2.2 (MingW32)
>
> iQIVAwUBRJwgLHGK1KGABytRAQqUig/+Jwa2J+satKUvk9prRxIooC0i4ZA71Zmx
> qfD6Ftjr/Lut3BlbJYUoIivvv4ifrvvDRtZOLDMW1ErRX+2e33gfUN1KNSW+PFa4
> jM9r3lTc55DKD+bFqjJExbLA7cV24/V+97Phj7KPu/fTPJb7YcfycCgZgT7u6yqo
> uOZ/9jtx65oAOy/hGugELblYbXJW4TPrNfpzb9zGIpni9QBD2F0ccD+Cv9ad9Wwz
> 34/NmcpTAfwcHooVdO7TWZ79tlQzi5/wXBGN5fmp6Z/Ffpr5mBzxc79HIdcEXk7Q
> ScBK+SDTPSv/ZIzNEnypJX/aZjdzXMmm4N+5SWh5NdmrL3AY6iK9MGQFQ6t0VYpO
> o+GTw6/X1ZLC1AVyNaQ5QAh1ZDPJvPmOGS17PIjyGvgcBisHshRnpsgrSPxFoSGv
> suSfEyWKLXvnmUuSwbF08U5aLGqTRrY9mQ+9UA1IrSkIq1ykxXk1X8Zit8RUhFz7
> H6DkkcHO0wJshqpXqVdpOy5sq53nEUgFyfz3cv1SXrlipd4ReDs8Dela3iTwl9ul
> 4RjlfIu2G4I8Oqw7163VpLkvguldQ38s71c2B74S990LcLBOOHF3qBsa6BQszwVm
> LmGU1ftMdVzTr+GPwpXyDC02CeoYgCY18vthU7L+w/iz8jPUrhYMuFT/QJGmMLML
> u4IT/OnmbIk=
> =OYxb
> -----END PGP SIGNATURE-----
> _______________________________________________
> Winpcap-users mailing list
> Winpcap-users at winpcap.org
> https://www.winpcap.org/mailman/listinfo/winpcap-users