[Winpcap-users] is winpcap appropriate for my gigabitethernetdevice?

[Gianluca Varenni]

----- Original Message ----- 
From: "Zafer SAVAS" <zsavas at aselsan.com.tr>
To: <winpcap-users at winpcap.org>; <winpcap-users at winpcap.org>
Sent: Thursday, September 04, 2008 11:06 AM
Subject: RE: [Winpcap-users] is winpcap appropriate for my 
gigabitethernetdevice?


> Hi Max,
>
> I have been working on a similar project like yours. My aim was to 
> transfer LARGE AMOUNT of data (like ~110MBytes/sec) from a FPGA based 
> hardware to a standard PC. My FPGA is a Virtex-5 xilinx FPGA and it has 
> gigabit ethernet core in it. To maximize the speed I am avoiding any 
> header and sending raw ethernet frames (Layer-2) with format
> -6 bytes dest MAC
> -6 bytes source MAC
> -2 bytes type/length
> -x bytes data
> By the way there is a cross link with the PC and the FPGA and no 
> hubs/routers etc...
>
>
> Here are my answers for your questions:
>
> Q:
> So my first question is, "does winpcap do what I need?".
> A:
> Yes, it is DEFINITELY possible. Thats something what winpcap is capable of 
> doing. It can capture/send raw ethernet packets.
>
> Q:
> I hope winpcap can simply "give my application the packet"  and do no 
> [additional] processing whatsoever.
> A:
> Without any processing whole packet can be passed to the user side.
>
> And here are some advices for your application:
> Winpcap can nearly capture gigabit ethernet packets at line speed (which 
> is about 120MBytes/sec) but it is very dependent on your hardware and your 
> software. You can capture packets but if you process and try saving the 
> packets to harddisk with a standard PC YOU WILL DEFINITELY LOSE PACKETS.
>
> To avoid these problems, here are some advices for you:
> - Use Multi-CPU PC
> - Allocate a large amount of kernel and user RAM in your applciation. (For 
> example RAM for 1G packets which requires a large amount of RAM)
> - Copy group of packets.
> - Do not process them after you capture
> - Use RAID disks to speed up your write to HD.

To add something to it. If your main objective is dumping to disk, by far 
the most important things are
1. use RAID. A single disk CANNOT keep up.
2. you can also use one single thread and use overlapped i/o to dump to 
disk. And disable file system caching while dumping to disk (you do that 
with the flags of CreateFile).

Creating big buffers in user mode is not always useful. Unless you lock all 
that amount of memory so that it's in your working set (i.e. it's not 
swapped out), the risk is that that big amount of memory gets swapped in/out 
to disk when you use it (because the OS tries to trim your working set 
size).

I played a lot with these things recently while developing TurboCap, and 
these were the major bottlenecks when dumping to disk.

Have a nice day
GV

>
> As I said I achieved ~110MBytes/sec of data and in my application. There 
> are two threads for achieving this;
> - 1 thread for capturing packets and writing them to user RAM
> - and the other one for copying the packets to Hard-drive.
>
> God bless winpcap and its creators :)
>
> Good luck
> zafer.
>
> Zafer SAVAS
> Senior Expert Test Engineer
> Aselsan Inc.
>
>
>
> -----Original Message-----
> From: winpcap-users-bounces at winpcap.org on behalf of max reason
> Sent: Thu 04.09.2008 12:52
> To: winpcap-users at winpcap.org
> Subject: [Winpcap-users] is winpcap appropriate for my gigabit 
> ethernetdevice?
>
>
> This is my first post, since I just learned about winpcap.
> I suspect my application for winpcap is unusual, so I'd like
> to ask others with winpcap experience whether winpcap will
> do what I need.  So, here goes.
>
> I designed an inexpensive but high-speed, high-resolution
> digital video camera with 5 megapixel [and 8 megapixel] CCDs.
> To keep the per-unit cost low, I had to find some way to get
> 151,165,440 bytes per second from camera to PC - which equals
> roughly 1.2 gigabits per second.  My solution was to base the
> camera circuit on a cyclone3 FPGA and gigabit ethernet PHY.
> Plus I had to concoct a lossless compression system that is
> both brain-dead simple, and simple and fast enough to compress
> 100 million 16-bit pixels per second enough to reduce the
> bandwidth far enough below 1 gigabit per second to squeeze
> the data through the PHY along with ethernet packet overhead.
>
> The bottom line is the following.  For this to work reliably,
> the packets need to be captured, then stored and/or displayed
> with as little overhead as possible.  Thet's probably obvious!
>
> To make this work, the ethernet cable from the camera must plug
> directly into the RJ45 connector on the PC - no routers/hubs/etc.
> The connection must be direct connect, or no way.
>
> My plan has been to send the data in a simple ethernet packet:
>   6-byte target-MAC-address
>   6-byte source-MAC-address
>   2-byte length-field
>   data (2592 pixels)
>   CRC32
>
> I didn't learn until recently that PC operating systems do not
> typically receive or send such fundamental ethernet packets,
> and support might be non-trivial.  I wrote some prototype code
> on Linux with PF_PACKET that appears to provide me a way to
> process these fundamental packets efficiently, and avoid ALL
> subsequent [higher-level protocol] processing by the OS.
>
> But so far, I have not found a similar mechanism in win32.
> However, if I understand the nature of winpcap correctly,
> winpcap appears to provide similar capabilities.
>
> So my first question is, "does winpcap do what I need?".
>
> I think "what I need" is fairly obvious, but I'll rephrase
> it this way.  On the "PC receives data" side, I need a way to
> receive the packet - at least the data payload, but also the
> header if feasible.  I prefer to receive the CRC32 at the end
> of the packet (after the data), but can live without it.
>
> On the "PC sends data" side, I need to send short, simple
> command packets fairly rarely - no more often than one per
> image == 1944 ~ 2448 received packets.  And that packet
> can and should be sent when no data is being received
> (during a short pause between images).  These packets
> should also be plain, ordinary ethernet packets with
> the same fields shown above.
>
> I very much prefer to ***avoid*** IP, UDP, TCP/IP packets!
> It is difficult enough to make the FPGA control and receive
> data from four CCDs simultaneously (?did I mention that?),
> plus perform realtime lossless data compression, plus send
> the compressed data through the gigabit ethernet PHY also
> in realtime (albeit one scanline later == pipelined).
>
> To waste space, bandwidth and complexity in the packet header
> on totally useless NOTHINGNESS (to my application anyway),
> is just stupid.  To create those larger packets just tempts
> the OS to perform some packet processing, which just consumes
> CPU bandwidth that my application need for other work (say,
> saving the data to disk, or blasting it into a texture-map
> in a GPU card, or etc).  It also throws a complexity-wrench
> into my plans for computing the CRC32 in the FPGA.
>
> I hope winpcap can simply "give my application the packet"
> and do no [additional] processing whatsoever.  It is also
> very important that the OS not waste its time trying to
> process the packet too.  I mean, what on earth for?
>
> And no, I can't just slow down the frame rate a little, to
> whatever bandwidth is "feasible".  Some of the applications
> of this device require "super-cinema HDTV" media output,
> which is 2592 x 1080 x 24 FPS --- and the frame rate is
> fixed by convention (and the equipment that processes it).
> And that's not the only application that requires this
> little device crank out data out at nearly 1 gigabit.
> PS:  I suspect the actual data-payload rate can be in
> the neighborhood of 900Mbps (maybe even 850Mbps), but
> it must not "studder or skip" on frames that contain
> "complexity" the lossless compression cannot compress
> as much as "easy" frames.  That's an absolute no-can-do.
>
> I thank whoever decides to answer my question.  I am happy
> to dig-in, read and learn how to hook winpcap into my code.
> I just want to make sure I don't do all that work just to
> discover winpcap can't do what I need.  Thanks, Max.
>
> _______________________________________________
> Winpcap-users mailing list
> Winpcap-users at winpcap.org
> https://www.winpcap.org/mailman/listinfo/winpcap-users
>
> ######################################################################
> Dikkat:
>
> Bu elektronik posta mesaji kisisel ve ozeldir. Eger size
> gonderilmediyse lutfen gondericiyi bilgilendirip mesaji siliniz.
> Firmamiza gelen ve giden mesajlar virus taramasindan gecirilmekte,
> guvenlik nedeni ile kontrol edilerek saklanmaktadir. Mesajdaki
> gorusler ve bakis acisi gondericiye ait olup Aselsan A.S. resmi
> gorusu olmak zorunda degildir.
>
> ######################################################################
> Attention:
>
> This e-mail message is privileged and confidential. If you are
> not the intended recipient please delete the message and notify
> the sender. E-mails to and from the company are monitored for
> operational reasons and in accordance with lawful business practices.
> Any views or opinions presented are solely those of the author and
> do not necessarily represent the views of the company.
>
> ######################################################################
>
>_______________________________________________
> Winpcap-users mailing list
> Winpcap-users at winpcap.org
> https://www.winpcap.org/mailman/listinfo/winpcap-users