Working on PacketFu Performance

Here's a baseline of how PacketFu version 0.1.1 handles a set of 5000 packets. This benchmark test takes in a pcap file, then chucks all the processed packets into a Ruby array. The performance is horrid compared to Wireshark, but ignore that for a moment:

(these packets are all normal TCP packets)
(  1000) [17:26:48] {      20s}
(  2000) [17:27:13] {      25s}
(  3000) [17:27:43] {      30s}
(  4000) [17:28:19] {      36s}
(  5000) [17:29:01] {      42s}
2m33s elapsed, parsed 5001 packets.

Eek. So, the more packets I pull in, the slower PacketFu gets. This is pretty disastrous, if you're using PacketFu in offline mode.

So, after poking at PacketFu::Packet.parse() for a bit, I figured out this morning that if I make a good guess at the packet type before testing it for complete correctness, I get a fairly huge bonus in parsing speed. Here's a run with all normal TCP packets:

(  1000) [11:21:48] {      15s}
(  2000) [11:22:03] {      15s}
(  3000) [11:22:17] {      14s}
(  4000) [11:22:32] {      15s}
(  5000) [11:22:47] {      15s}
1m14s elapsed, parsed 5001 packets.

Testing the new and improved version with a mixed bag of packets, which contains ARP, TCP, ICMP, and UDP (and a few IPv6) packets:

(  1000) [12:21:15] {      11s}
(  2000) [12:21:30] {      15s}
(  3000) [12:21:48] {      18s}
(  4000) [12:22:10] {      22s}
(  5000) [12:22:36] {      26s}
1m32s elapsed, parsed 5001 packets.

Unfortunately, my creeping performance problem persists -- at least when I have a whole bunch of dissimilar packet types. But at least it's less pronounced now, and eliminated entirely when dealing with sets of TCP packets (which is going to be the most common use case, I figure).

---

Update: That was completely wrong. The only reason for the performance boost was that PacketFu::Packet.parse() was forgetting to read in the data. The below is even more true -- this is where the problem lies. Darnit! (please make sure to never use PacketFu r66, it's broken!)

---

tmanning has been looking at PacketFu lately as well, and believes that there are some (more) bugs in PacketFu::Packet.read(), mostly revolving around my atrocious design of how read() and parse() interrelate. I suspect this is the source of most of my performance problems as well, so keep an eye out for the next tagged version of PacketFu for some love in that part of the code. Oh, and I'll be fixing up the PacketFu::File.append() function to be a lot more sane, too.

I admit it: I love monkeypatching

One of the things I like most about writing in Ruby is that if I find myself reimplimenting the same thing more than two or three times, I have the option of extending Ruby's base classes to incorporate that functionality.

People argue against this (usually by calling it monkeypatching), but when it comes down to it, I think it's a wonderfully fun way to interact with an interpreted language, even if it's not the safest or prudent habit. After all, the more fun something is, the more dangerous it's got to be, right?

An example base extension I use a lot is binarize -- it takes a String or Array and turns it into binary (really, a pack("C*") string). Here it is at pastie. Once implemented, "414243".binarize magically turns into "ABC." Delight all around.

The argument against monkeypatching is that it's not that much more work to create a module with a function of binarize that takes an argument, so you end up with something like NoFunAtAll::binarize("414243"). But really, only squares with C.S. degrees would do that. Devil-may-care types like myself prefer to extend String and Array directly, and damn the maintainability!

PacketFu on Windows

Barely a week after I presented PacketFu, I got an idle question about PacketFu's usefulness on Windows.

Last night, while I was waiting for my brother and his family to drive up from Houston ahead of the Hurricane, I got to messing around with compiling PcapRub on Microsoft Vista and Windows XP. Much to my amazement, my goofy hacks worked. The rest of PacketFu is pure Ruby without C extensions or anything, so cross-platform love is already baked in there.

So, the latest revision of PacketFu seems to work fine on XP (I haven't tested the Vista machine yet). It works so well, in fact, that I duplicated the climax of my Lone Star RubyConf talk in the form of a Flash movie (link downloads the FLV, it's not embedded or anything).

How many other packet libraries have screencasts a week after they're built? ZERO, that's how many!