[Winpcap-users] Better Performance
Charu Agrawal
cagrawal at altasens.com
Tue Apr 28 12:02:03 PDT 2009
Hello David,
I figured out the reason for the poor performance.
I was creating a separate thread for the pcap_loop, and the main thread calculated when to timeout.
When I use pcap_loop in the main thread, I get the performance I need.
But the drawback now is, that I cannot get timeout using pcap_loop or pcap_dispatch.
This is what I want.
Suppose I say pcap_loop(handle,20000,....)
I need to capture 20000 packets. But if I capture only 16000 packets in say 10s , I want the program to timeout.
I tried to use pcap_next_ex for this, but it slows down the code. pcap_loop gives me the needed performance, except the timeout ability.
Is there a workaround ?
Thanks
Charu
-----Original Message-----
From: winpcap-users-bounces at winpcap.org [mailto:winpcap-users-bounces at winpcap.org] On Behalf Of David Chang
Sent: Friday, April 24, 2009 1:08 PM
To: winpcap-users at winpcap.org
Subject: Re: [Winpcap-users] Better Performance
Charu,
Can you give me your hardware specs? Also, what was your throughput
with the empty callback function?
Let's do another test...
u_char dummy[2000]; /* I assume your packets (and snaplen) are less than 1500 bytes long */
void packet_handler(u_char *dumpfileHandler, const struct pcap_pkthdr *header, const u_char *pkt_data)
{
if (header->len < sizeof(dummy)+sizeof(struct pcap_pkthdr))
{
memcpy(dummy, header, sizeof(struct pcap_pkthdr));
memcpy(dummy+sizeof(struct pcap_pkthdr), pkt_data, header->len);
}
packetCounter++;
}
How is the throughput with just the memcpy() overwriting a dummy buffer?
Lastly, I would use MUCH less kernel buffer space (40MB is too much). I
would use 8MB - 16MB. For example:
if (pcap_setbuff(adhandle,16777216) < 0)
DC
Gianluca Varenni wrote:
> ----- Original Message -----
> From: "Charu Agrawal" <cagrawal at altasens.com>
> To: <winpcap-users at winpcap.org>
> Sent: Friday, April 24, 2009 10:01 AM
> Subject: Re: [Winpcap-users] Better Performance
>
>
>
>> Hello David,
>>
>> If I change the code as you suggested, I do get a performance
>> improvement. I have now allocated memory outside the loop.
>> But I still need to use the memcpy function to store all packets in an
>> array. That still slows it down.
>>
>> Anything else I can do to improve performance ? Is there a way to change
>> the winpcap source code for better performance ?
>>
>>
>
> WinPcap has been highly optimized over the years for performance. Before
> looking into optimizing the WinPcap code, I would understand exactly what
> the bottleneck is.
> You say that you are copying packets into your allocated memory. Why can't
> you process the packets inline, i.e. when you receive them? Also, how much
> memory did you allocate for the packets? Preallocating several megabytes
> ahead of time for the packets can be as dangerous as malloc'ing for every
> packet. This is because allocating memory just means that you have reserved
> a large chunk of address space in your process, not that you have physically
> reserved all those megabytes in the physical RAM (i.e. not all that memory
> is your working set).
>
> Hope it helps
> GV
>
>
>
>> Thanks
>> Charu
>>
>> -----Original Message-----
>> From: winpcap-users-bounces at winpcap.org
>> [mailto:winpcap-users-bounces at winpcap.org] On Behalf Of David Chang
>> Sent: Thursday, April 23, 2009 7:03 PM
>> To: winpcap-users at winpcap.org
>> Subject: Re: [Winpcap-users] Better Performance
>>
>> Charu,
>>
>> malloc() is VERY expensive; especially for larger buffers. Are you
>> saying that if you change the callback function to this...
>>
>> void packet_handler(u_char *dumpfileHandler, const struct pcap_pkthdr
>> *header, const u_char *pkt_data)
>> {
>> packetCounter++;
>> }
>>
>> you still can't read 20,000 packets / sec? (I assume each packet is
>> full, so this would be close to 30MB/sec)
>>
>> What type of hardware (CPU, RAM, NIC, motherboard, etc) and O.S. do you
>> use?
>>
>> DC
>>
>> Charu Agrawal wrote:
>>
>>> Hi,
>>>
>>> Yeah. I understand that. But I doubt if that effects the speed much.
>>>
>>> I am doing some processing on the array later. But the processing is
>>> not the bottleneck. It is the packet capture (packet_handler) that is
>>> the bottleneck. Even if I remove
>>>
>>> All the code inside the packet_handler the packet capture still does
>>> not meet expectations. We have nearly 20,000 packets. And as I
>>> mentioned before we are looking to achieve a frame rate
>>>
>>> Of 30-60 Mbytes/s.
>>>
>>> Any ideas ?
>>>
>>> Thanks
>>>
>>> Charu
>>>
>>> *From:* winpcap-users-bounces at winpcap.org
>>> [mailto:winpcap-users-bounces at winpcap.org] *On Behalf Of *Gianluca
>>>
>> Varenni
>>
>>> *Sent:* Tuesday, April 21, 2009 3:33 PM
>>> *To:* winpcap-users at winpcap.org
>>> *Subject:* Re: [Winpcap-users] Better Performance
>>>
>>> First of all, allocating some memory at every received packet is a bad
>>>
>>> idea. You should preallocate memory at the very beginning.
>>>
>>> Then, how many packets do you want to capture? What are you doing with
>>>
>>> all those packets in these arrays?
>>>
>>> Have a nice day
>>>
>>> GV
>>>
>>> ----- Original Message -----
>>>
>>> *From:* Charu Agrawal <mailto:cagrawal at altasens.com>
>>>
>>> *To:* winpcap-users at winpcap.org <mailto:winpcap-users at winpcap.org>
>>>
>>> *Sent:* Tuesday, April 21, 2009 2:20 PM
>>>
>>> *Subject:* Re: [Winpcap-users] Better Performance
>>>
>>> Hi,
>>>
>>> This is what the code looks like -
>>>
>>> This is all initialization code -
>>>
>>> if ((adhandle= pcap_open_live(d->name, // name of the device
>>>
>>> 65536, // portion of the packet to capture.
>>>
>>> // 65536 grants that the whole packet will be captured on all the
>>> MACs.
>>>
>>> 1, // promiscuous mode (nonzero means promiscuous)
>>>
>>> 1, // read timeout
>>>
>>> errbuf // error buffer
>>>
>>> )) == NULL)
>>>
>>> {
>>>
>>> printf("\nUnable to open the adapter. %s is not supported by
>>> WinPcap\n", d->name);
>>>
>>> return RETCODE_ERR;
>>>
>>> }
>>>
>>> //Set the filter string to the source ip address
>>>
>>> //Only packets from this source address will be captured.
>>>
>>> //This does slow down the capture a bit.
>>>
>>> NetMask=0xffffff;
>>>
>>> filter = "src 192.168.1.10";
>>>
>>> if(pcap_compile(adhandle, &fcode, filter, 1, NetMask) < 0)
>>>
>>> {
>>>
>>> printf("\nError compiling filter: wrong syntax.\n");
>>>
>>> pcap_close(adhandle);
>>>
>>> return RETCODE_ERR;
>>>
>>> }
>>>
>>> if(pcap_setfilter(adhandle, &fcode)<0)
>>>
>>> {
>>>
>>> printf("\nError setting the filter\n");
>>>
>>> pcap_close(adhandle);
>>>
>>> return RETCODE_ERR;
>>>
>>> }
>>>
>>> //Set the kernel buffer
>>>
>>> if (pcap_setbuff(adhandle,41943040) < 0)
>>>
>>> {
>>>
>>> printf("\n unable to set buffer");
>>>
>>> return RETCODE_ERR;
>>>
>>> }
>>>
>>> Then I start the pcap_loop on a separate thread - This function is
>>> called for each packet capture.
>>>
>>> I copy the data for all packets in a global array - image_buf
>>>
>>> void packet_handler(u_char *dumpfileHandler, const struct
>>> pcap_pkthdr *header, const u_char *pkt_data)
>>>
>>> {
>>>
>>> //copy the packet data to image_buf and also store the packet
>>>
>> length
>>
>>> image_buf[packetCounter] = (unsigned
>>> char*)(malloc(header->caplen*sizeof(unsigned char)));
>>>
>>> if(image_buf[packetCounter] == NULL)
>>>
>>> {
>>>
>>> printf("\n unable to allocate buffer");
>>>
>>> return;
>>>
>>> }
>>>
>>> memcpy(image_buf[packetCounter],pkt_data,header->caplen);
>>>
>>> pkt_length[packetCounter]=header->caplen;
>>>
>>> packetCounter += 1;
>>>
>>> }
>>>
>>> Please let me know if you need more details. I can zip and send
>>> the code file if needed.
>>>
>>> Thanks
>>>
>>> Charu
>>>
>>> *From:* winpcap-users-bounces at winpcap.org
>>> [mailto:winpcap-users-bounces at winpcap.org] *On Behalf Of *Gianluca
>>> Varenni
>>> *Sent:* Tuesday, April 21, 2009 1:51 PM
>>> *To:* winpcap-users at winpcap.org
>>> *Subject:* Re: [Winpcap-users] Better Performance
>>>
>>> Before trying to modify the WinPcap source code, I would try to
>>> understand what the bottlenecks are. What are you doing with the
>>> received packets?
>>>
>>> Can you show us the packet processing loop that you have in your
>>> code (even in form of pseudocode)?
>>>
>>> Have a nice day
>>>
>>> GV
>>>
>>> ----- Original Message -----
>>>
>>> *From:* Charu Agrawal <mailto:cagrawal at altasens.com>
>>>
>>> *To:* winpcap-users at winpcap.org
>>> <mailto:winpcap-users at winpcap.org>
>>>
>>> *Sent:* Monday, April 20, 2009 10:58 AM
>>>
>>> *Subject:* [Winpcap-users] Better Performance
>>>
>>> Hi,
>>>
>>> I am trying to figure out how to capture image frames most
>>> efficiently. Each frame is approximately 20Mbytes.
>>>
>>> I am trying to achieve frame rate of 1-2 frames per second.
>>> The packets are being received over a Gig eth dedicated link.
>>> ( Packets are sent over that link using only one source).
>>>
>>> I am unable to achieve this frame rate using winpcap. Would it
>>> be helpful to modify the Winpcap source code for better
>>> performance ? Has anyone here have any experience in modifying
>>> the winpcap source to achieve higher frame rate capture ?
>>>
>>> Thanks for your help in advance
>>>
>>> Regards
>>>
>>> Charu Agrawal
>>>
>>>
>>>
>> ------------------------------------------------------------------------
>>
>>> _______________________________________________
>>> Winpcap-users mailing list
>>> Winpcap-users at winpcap.org
>>> https://www.winpcap.org/mailman/listinfo/winpcap-users
>>>
>>>
>>>
>> ------------------------------------------------------------------------
>>
>>> _______________________________________________
>>> Winpcap-users mailing list
>>> Winpcap-users at winpcap.org
>>> https://www.winpcap.org/mailman/listinfo/winpcap-users
>>>
>>>
>>>
>> ------------------------------------------------------------------------
>>
>>> _______________________________________________
>>> Winpcap-users mailing list
>>> Winpcap-users at winpcap.org
>>> https://www.winpcap.org/mailman/listinfo/winpcap-users
>>>
>>>
>> _______________________________________________
>> Winpcap-users mailing list
>> Winpcap-users at winpcap.org
>> https://www.winpcap.org/mailman/listinfo/winpcap-users
>> _______________________________________________
>> Winpcap-users mailing list
>> Winpcap-users at winpcap.org
>> https://www.winpcap.org/mailman/listinfo/winpcap-users
>>
>
> _______________________________________________
> Winpcap-users mailing list
> Winpcap-users at winpcap.org
> https://www.winpcap.org/mailman/listinfo/winpcap-users
>
_______________________________________________
Winpcap-users mailing list
Winpcap-users at winpcap.org
https://www.winpcap.org/mailman/listinfo/winpcap-users
More information about the Winpcap-users
mailing list