Working on PacketFu Performance

Here's a baseline of how PacketFu version 0.1.1 handles a set of 5000 packets. This benchmark test takes in a pcap file, then chucks all the processed packets into a Ruby array. The performance is horrid compared to Wireshark, but ignore that for a moment:

(these packets are all normal TCP packets)
(  1000) [17:26:48] {      20s}
(  2000) [17:27:13] {      25s}
(  3000) [17:27:43] {      30s}
(  4000) [17:28:19] {      36s}
(  5000) [17:29:01] {      42s}
2m33s elapsed, parsed 5001 packets.

Eek. So, the more packets I pull in, the slower PacketFu gets. This is pretty disastrous, if you're using PacketFu in offline mode.

So, after poking at PacketFu::Packet.parse() for a bit, I figured out this morning that if I make a good guess at the packet type before testing it for complete correctness, I get a fairly huge bonus in parsing speed. Here's a run with all normal TCP packets:

(  1000) [11:21:48] {      15s}
(  2000) [11:22:03] {      15s}
(  3000) [11:22:17] {      14s}
(  4000) [11:22:32] {      15s}
(  5000) [11:22:47] {      15s}
1m14s elapsed, parsed 5001 packets.

Testing the new and improved version with a mixed bag of packets, which contains ARP, TCP, ICMP, and UDP (and a few IPv6) packets:

(  1000) [12:21:15] {      11s}
(  2000) [12:21:30] {      15s}
(  3000) [12:21:48] {      18s}
(  4000) [12:22:10] {      22s}
(  5000) [12:22:36] {      26s}
1m32s elapsed, parsed 5001 packets.

Unfortunately, my creeping performance problem persists -- at least when I have a whole bunch of dissimilar packet types. But at least it's less pronounced now, and eliminated entirely when dealing with sets of TCP packets (which is going to be the most common use case, I figure).

---

Update: That was completely wrong. The only reason for the performance boost was that PacketFu::Packet.parse() was forgetting to read in the data. The below is even more true -- this is where the problem lies. Darnit! (please make sure to never use PacketFu r66, it's broken!)

---

tmanning has been looking at PacketFu lately as well, and believes that there are some (more) bugs in PacketFu::Packet.read(), mostly revolving around my atrocious design of how read() and parse() interrelate. I suspect this is the source of most of my performance problems as well, so keep an eye out for the next tagged version of PacketFu for some love in that part of the code. Oh, and I'll be fixing up the PacketFu::File.append() function to be a lot more sane, too.