[Winpcap-users] Better Performance
David Chang
dchang at fsautomation.com
Tue Apr 28 13:12:26 PDT 2009
Charu,
So long as the pcap_dispatch() and callback routines are in one thread, and
the main application (manipulates the array) is in another thread, you would
get the same speed (less the synchronization mechanism). Are you using some
kind of synchronization mechanism that causes the packet capture to slow
down?
DC
----- Original Message -----
From: "Charu Agrawal" <cagrawal at altasens.com>
To: <winpcap-users at winpcap.org>
Sent: Tuesday, April 28, 2009 12:02 PM
Subject: Re: [Winpcap-users] Better Performance
> Hello David,
>
> I figured out the reason for the poor performance.
> I was creating a separate thread for the pcap_loop, and the main thread
> calculated when to timeout.
> When I use pcap_loop in the main thread, I get the performance I need.
>
> But the drawback now is, that I cannot get timeout using pcap_loop or
> pcap_dispatch.
>
> This is what I want.
>
> Suppose I say pcap_loop(handle,20000,....)
> I need to capture 20000 packets. But if I capture only 16000 packets in
> say 10s , I want the program to timeout.
>
> I tried to use pcap_next_ex for this, but it slows down the code.
> pcap_loop gives me the needed performance, except the timeout ability.
>
> Is there a workaround ?
>
> Thanks
> Charu
>
> -----Original Message-----
> From: winpcap-users-bounces at winpcap.org
> [mailto:winpcap-users-bounces at winpcap.org] On Behalf Of David Chang
> Sent: Friday, April 24, 2009 1:08 PM
> To: winpcap-users at winpcap.org
> Subject: Re: [Winpcap-users] Better Performance
>
> Charu,
>
> Can you give me your hardware specs? Also, what was your throughput
> with the empty callback function?
>
> Let's do another test...
>
> u_char dummy[2000]; /* I assume your packets (and snaplen) are less than
> 1500 bytes long */
>
> void packet_handler(u_char *dumpfileHandler, const struct pcap_pkthdr
> *header, const u_char *pkt_data)
> {
> if (header->len < sizeof(dummy)+sizeof(struct pcap_pkthdr))
> {
> memcpy(dummy, header, sizeof(struct pcap_pkthdr));
> memcpy(dummy+sizeof(struct pcap_pkthdr), pkt_data, header->len);
> }
>
> packetCounter++;
> }
>
> How is the throughput with just the memcpy() overwriting a dummy buffer?
>
> Lastly, I would use MUCH less kernel buffer space (40MB is too much). I
> would use 8MB - 16MB. For example:
>
> if (pcap_setbuff(adhandle,16777216) < 0)
>
>
> DC
>
> Gianluca Varenni wrote:
>> ----- Original Message -----
>> From: "Charu Agrawal" <cagrawal at altasens.com>
>> To: <winpcap-users at winpcap.org>
>> Sent: Friday, April 24, 2009 10:01 AM
>> Subject: Re: [Winpcap-users] Better Performance
>>
>>
>>
>>> Hello David,
>>>
>>> If I change the code as you suggested, I do get a performance
>>> improvement. I have now allocated memory outside the loop.
>>> But I still need to use the memcpy function to store all packets in an
>>> array. That still slows it down.
>>>
>>> Anything else I can do to improve performance ? Is there a way to change
>>> the winpcap source code for better performance ?
>>>
>>>
>>
>> WinPcap has been highly optimized over the years for performance. Before
>> looking into optimizing the WinPcap code, I would understand exactly what
>> the bottleneck is.
>> You say that you are copying packets into your allocated memory. Why
>> can't
>> you process the packets inline, i.e. when you receive them? Also, how
>> much
>> memory did you allocate for the packets? Preallocating several megabytes
>> ahead of time for the packets can be as dangerous as malloc'ing for every
>> packet. This is because allocating memory just means that you have
>> reserved
>> a large chunk of address space in your process, not that you have
>> physically
>> reserved all those megabytes in the physical RAM (i.e. not all that
>> memory
>> is your working set).
>>
>> Hope it helps
>> GV
>>
>>
>>
>>> Thanks
>>> Charu
>>>
>>> -----Original Message-----
>>> From: winpcap-users-bounces at winpcap.org
>>> [mailto:winpcap-users-bounces at winpcap.org] On Behalf Of David Chang
>>> Sent: Thursday, April 23, 2009 7:03 PM
>>> To: winpcap-users at winpcap.org
>>> Subject: Re: [Winpcap-users] Better Performance
>>>
>>> Charu,
>>>
>>> malloc() is VERY expensive; especially for larger buffers. Are you
>>> saying that if you change the callback function to this...
>>>
>>> void packet_handler(u_char *dumpfileHandler, const struct pcap_pkthdr
>>> *header, const u_char *pkt_data)
>>> {
>>> packetCounter++;
>>> }
>>>
>>> you still can't read 20,000 packets / sec? (I assume each packet is
>>> full, so this would be close to 30MB/sec)
>>>
>>> What type of hardware (CPU, RAM, NIC, motherboard, etc) and O.S. do you
>>> use?
>>>
>>> DC
>>>
>>> Charu Agrawal wrote:
>>>
>>>> Hi,
>>>>
>>>> Yeah. I understand that. But I doubt if that effects the speed much.
>>>>
>>>> I am doing some processing on the array later. But the processing is
>>>> not the bottleneck. It is the packet capture (packet_handler) that is
>>>> the bottleneck. Even if I remove
>>>>
>>>> All the code inside the packet_handler the packet capture still does
>>>> not meet expectations. We have nearly 20,000 packets. And as I
>>>> mentioned before we are looking to achieve a frame rate
>>>>
>>>> Of 30-60 Mbytes/s.
>>>>
>>>> Any ideas ?
>>>>
>>>> Thanks
>>>>
>>>> Charu
>>>>
>>>> *From:* winpcap-users-bounces at winpcap.org
>>>> [mailto:winpcap-users-bounces at winpcap.org] *On Behalf Of *Gianluca
>>>>
>>> Varenni
>>>
>>>> *Sent:* Tuesday, April 21, 2009 3:33 PM
>>>> *To:* winpcap-users at winpcap.org
>>>> *Subject:* Re: [Winpcap-users] Better Performance
>>>>
>>>> First of all, allocating some memory at every received packet is a bad
>>>>
>>>> idea. You should preallocate memory at the very beginning.
>>>>
>>>> Then, how many packets do you want to capture? What are you doing with
>>>>
>>>> all those packets in these arrays?
>>>>
>>>> Have a nice day
>>>>
>>>> GV
>>>>
>>>> ----- Original Message -----
>>>>
>>>> *From:* Charu Agrawal <mailto:cagrawal at altasens.com>
>>>>
>>>> *To:* winpcap-users at winpcap.org <mailto:winpcap-users at winpcap.org>
>>>>
>>>> *Sent:* Tuesday, April 21, 2009 2:20 PM
>>>>
>>>> *Subject:* Re: [Winpcap-users] Better Performance
>>>>
>>>> Hi,
>>>>
>>>> This is what the code looks like -
>>>>
>>>> This is all initialization code -
>>>>
>>>> if ((adhandle= pcap_open_live(d->name, // name of the device
>>>>
>>>> 65536, // portion of the packet to capture.
>>>>
>>>> // 65536 grants that the whole packet will be captured on all the
>>>> MACs.
>>>>
>>>> 1, // promiscuous mode (nonzero means promiscuous)
>>>>
>>>> 1, // read timeout
>>>>
>>>> errbuf // error buffer
>>>>
>>>> )) == NULL)
>>>>
>>>> {
>>>>
>>>> printf("\nUnable to open the adapter. %s is not supported by
>>>> WinPcap\n", d->name);
>>>>
>>>> return RETCODE_ERR;
>>>>
>>>> }
>>>>
>>>> //Set the filter string to the source ip address
>>>>
>>>> //Only packets from this source address will be captured.
>>>>
>>>> //This does slow down the capture a bit.
>>>>
>>>> NetMask=0xffffff;
>>>>
>>>> filter = "src 192.168.1.10";
>>>>
>>>> if(pcap_compile(adhandle, &fcode, filter, 1, NetMask) < 0)
>>>>
>>>> {
>>>>
>>>> printf("\nError compiling filter: wrong syntax.\n");
>>>>
>>>> pcap_close(adhandle);
>>>>
>>>> return RETCODE_ERR;
>>>>
>>>> }
>>>>
>>>> if(pcap_setfilter(adhandle, &fcode)<0)
>>>>
>>>> {
>>>>
>>>> printf("\nError setting the filter\n");
>>>>
>>>> pcap_close(adhandle);
>>>>
>>>> return RETCODE_ERR;
>>>>
>>>> }
>>>>
>>>> //Set the kernel buffer
>>>>
>>>> if (pcap_setbuff(adhandle,41943040) < 0)
>>>>
>>>> {
>>>>
>>>> printf("\n unable to set buffer");
>>>>
>>>> return RETCODE_ERR;
>>>>
>>>> }
>>>>
>>>> Then I start the pcap_loop on a separate thread - This function is
>>>> called for each packet capture.
>>>>
>>>> I copy the data for all packets in a global array - image_buf
>>>>
>>>> void packet_handler(u_char *dumpfileHandler, const struct
>>>> pcap_pkthdr *header, const u_char *pkt_data)
>>>>
>>>> {
>>>>
>>>> //copy the packet data to image_buf and also store the packet
>>>>
>>> length
>>>
>>>> image_buf[packetCounter] = (unsigned
>>>> char*)(malloc(header->caplen*sizeof(unsigned char)));
>>>>
>>>> if(image_buf[packetCounter] == NULL)
>>>>
>>>> {
>>>>
>>>> printf("\n unable to allocate buffer");
>>>>
>>>> return;
>>>>
>>>> }
>>>>
>>>> memcpy(image_buf[packetCounter],pkt_data,header->caplen);
>>>>
>>>> pkt_length[packetCounter]=header->caplen;
>>>>
>>>> packetCounter += 1;
>>>>
>>>> }
>>>>
>>>> Please let me know if you need more details. I can zip and send
>>>> the code file if needed.
>>>>
>>>> Thanks
>>>>
>>>> Charu
>>>>
>>>> *From:* winpcap-users-bounces at winpcap.org
>>>> [mailto:winpcap-users-bounces at winpcap.org] *On Behalf Of *Gianluca
>>>> Varenni
>>>> *Sent:* Tuesday, April 21, 2009 1:51 PM
>>>> *To:* winpcap-users at winpcap.org
>>>> *Subject:* Re: [Winpcap-users] Better Performance
>>>>
>>>> Before trying to modify the WinPcap source code, I would try to
>>>> understand what the bottlenecks are. What are you doing with the
>>>> received packets?
>>>>
>>>> Can you show us the packet processing loop that you have in your
>>>> code (even in form of pseudocode)?
>>>>
>>>> Have a nice day
>>>>
>>>> GV
>>>>
>>>> ----- Original Message -----
>>>>
>>>> *From:* Charu Agrawal <mailto:cagrawal at altasens.com>
>>>>
>>>> *To:* winpcap-users at winpcap.org
>>>> <mailto:winpcap-users at winpcap.org>
>>>>
>>>> *Sent:* Monday, April 20, 2009 10:58 AM
>>>>
>>>> *Subject:* [Winpcap-users] Better Performance
>>>>
>>>> Hi,
>>>>
>>>> I am trying to figure out how to capture image frames most
>>>> efficiently. Each frame is approximately 20Mbytes.
>>>>
>>>> I am trying to achieve frame rate of 1-2 frames per second.
>>>> The packets are being received over a Gig eth dedicated link.
>>>> ( Packets are sent over that link using only one source).
>>>>
>>>> I am unable to achieve this frame rate using winpcap. Would it
>>>> be helpful to modify the Winpcap source code for better
>>>> performance ? Has anyone here have any experience in modifying
>>>> the winpcap source to achieve higher frame rate capture ?
>>>>
>>>> Thanks for your help in advance
>>>>
>>>> Regards
>>>>
>>>> Charu Agrawal
>>>>
>>>>
>>>>
>>> ------------------------------------------------------------------------
>>>
>>>> _______________________________________________
>>>> Winpcap-users mailing list
>>>> Winpcap-users at winpcap.org
>>>> https://www.winpcap.org/mailman/listinfo/winpcap-users
>>>>
>>>>
>>>>
>>> ------------------------------------------------------------------------
>>>
>>>> _______________________________________________
>>>> Winpcap-users mailing list
>>>> Winpcap-users at winpcap.org
>>>> https://www.winpcap.org/mailman/listinfo/winpcap-users
>>>>
>>>>
>>>>
>>> ------------------------------------------------------------------------
>>>
>>>> _______________________________________________
>>>> Winpcap-users mailing list
>>>> Winpcap-users at winpcap.org
>>>> https://www.winpcap.org/mailman/listinfo/winpcap-users
>>>>
>>>>
>>> _______________________________________________
>>> Winpcap-users mailing list
>>> Winpcap-users at winpcap.org
>>> https://www.winpcap.org/mailman/listinfo/winpcap-users
>>> _______________________________________________
>>> Winpcap-users mailing list
>>> Winpcap-users at winpcap.org
>>> https://www.winpcap.org/mailman/listinfo/winpcap-users
>>>
>>
>> _______________________________________________
>> Winpcap-users mailing list
>> Winpcap-users at winpcap.org
>> https://www.winpcap.org/mailman/listinfo/winpcap-users
>>
>
> _______________________________________________
> Winpcap-users mailing list
> Winpcap-users at winpcap.org
> https://www.winpcap.org/mailman/listinfo/winpcap-users
> _______________________________________________
> Winpcap-users mailing list
> Winpcap-users at winpcap.org
> https://www.winpcap.org/mailman/listinfo/winpcap-users
>
More information about the Winpcap-users
mailing list