[Winpcap-users] is winpcap appropriate for my gigabit ethernet device?

[max reason]

 Much thanks Zafer,

 Your answers and advice are very helpful - and exactly what I was
 hoping to hear!  Yes, in my situation it is perfectly acceptable
 to require a dual-core or quad-core CPU, and PCIe disk controllers.
 In fact, I bought intel PCIe cards with 2 gigabit ethernet interfaces
 thinking they might be better than the on-motherboard 1Ge controller,
 but I have performed that test yet.  My main point is: in my case,
 it is perfectly okay to say "you must have quad-core CPU, 4GB RAM,
 PCIe ethernet controllers, PCIe disk controllers", etc.  I came to
 the same conclusion you did - that a vanilla desktop/notebook PC
 would not work.  In fact, I am not sure any notebook PC will work,
 partly due to the slower disk drives, but also due to the slower
 *everything* in notebook computers.  Oh yes, it is also okay to
 require the computer have 2+ disk drives to save to, and it does
 appear to be very marginal to expect one SATAII drive to consume
 rougly 100+ MBps (800+ Mbps).

 I was planning to implement mine with multiple processes with
 shared memory - instead of multiple threads like you.  However,
 I do not yet understand the detailed, subtle behavior of the
 level 3 cache in Phenom CPUs.  I worry that CPU #0 will pull
 a large block of data into the level #1 cache owned by CPU #0,
 then CPU #1 (or the "save to disk" function) will try to save
 that large block of data to disk - and cause some sort of
 cache-thrashing.  Whereas, if an application could make sure
 these blocks of data never reached level #1 or level #2 caches,
 then no thrashing would occur.  I'm not sure whether the process
 or writing the data to the disk would be faster from main memory
 or from level #3 cache --- and frankly, I'll bet that depends
 upon the disk controller.

 I do hope I can execute my lossless data compression algorithm
 on the CPU.  If all I want to do is save to disk, then I can
 save the compressed data - no problem.  But when I want to
 write the image onto a texture (pixel-array) object in the GPU,
 the data needs to be decompressed.  The algorithm I developed
 does not compress very well, but it is incredibly SIMPLE - so
 simple I can execute it on the fly in the FPGA or [de]compress
 in the CPU with only 5~8 machine instructions per pixel!  :-)
 Now that I think about it, no need to decompress in the CPU!
 I wrote the 3D simulation/graphics/game engine myself, including
 the vertex and pixel shader programs.  So it will be simple AND
 faster to write a decompress shader that runs in the GPU.  Yes!

 BTW, what level did you program at to achieve your throughput?
 Did you program with winpcap functions, packet.dll functions,
 directly to the kernel-mode NPF driver (which I guess we can
 figure out how to do by looking at the packet.dll source-code)?

 Again, thanks for your very help answers.  PS:  I chose the
 cyclone3 FPGA because it is very cheap - only $15 for 1 unit.
 Of course, add another $9 for the marvell PHY, so the equivalent
 cost is about $25.  Still, all the FPGAs fast enough to support
 gigabit ethernet inside the FPGA were far outside my budget!
 Wish I could put the PHY inside the FPGA, but no such luck.

 Max


> Hi Max,
>
> I have been working on a similar project like yours.  My aim was to
> transfer LARGE AMOUNT of data (like ~110MBytes/sec) from a FPGA based
> hardware to a standard PC.  My FPGA is a Virtex-5 xilinx FPGA and it
> has gigabit ethernet core in it.  To maximize the speed I am avoiding
> any header and sending raw ethernet frames (Layer-2) with format
> -6 bytes dest MAC
> -6 bytes source MAC
> -2 bytes type/length
> -x bytes data
> By the way there is a cross link with the PC and the FPGA and
> no hubs/routers etc...
>
> Here are my answers for your questions:
> 
> Q: So my first question is, "does winpcap do what I need?".
> A: Yes, it is DEFINITELY possible.  Thats something what winpcap is
>    capable of doing.  It can capture/send raw ethernet packets.
> 
> Q: I hope winpcap can simply "give my application the packet"
>    and do no [additional] processing whatsoever.
> A: Without any processing whole packet can be passed to user side.
>
> And here are some advices for your application:
>
> Winpcap can nearly capture gigabit ethernet packets at line speed
> (which is about 120MBytes/sec) but it is very dependent on your
> hardware and your software.  You can capture packets but if you
> process and try saving the packets to harddisk with a standard PC
> YOU WILL DEFINITELY LOSE PACKETS.
>
> To avoid these problems, here are some advices for you:
> - Use Multi-CPU PC
> - Allocate a large amount of kernel and user RAM in your applciation.
>   (For example RAM for 1G packets which requires a large amount of RAM)
> - Copy group of packets.
> - Do not process them after you capture
> - Use RAID disks to speed up your write to HD.
>
> As I said I achieved ~110MBytes/sec of data and in my application.
> There are two threads for achieving this:
> - 1 thread for capturing packets and writing them to user RAM 
> - and the other one for copying the packets to Hard-drive.
>
> God bless winpcap and its creators :)
>
> Good luck
> zafer.
>
> Zafer SAVAS
> Senior Expert Test Engineer
> Aselsan Inc.