So there’s this project that requires a ridiculously complicated communication protocol involving lots of byte-wrangling and formatting and weird transports. For the sake of brevity, I’ll only mention one of the endpoints, which requires decoding an email attachment.

Of course, this means a procmail script sending the email to PHP for processing and much ceremonious mucking about. On first go, it was failing. In debug mode, procmail was telling me that permission was denied. But it wasn’t user permissions: the file was owned by the same user as receiving the email and running the procmail script.

Naturally, when faced with cryptic permission failures, the first thing I did was look at /var/log/audit/audit.log and /var/log/messages for SELinux denials. There I found nothing at all. No errors, no warnings, no ugly “avc: denied” splatters.

Finally, this page here explained it to me. Rebuilding the policies with semodule -DB quickly revealed that my problem was, in fact, SELinux (as it all so often is). Once I could see the policies that were marked “dontaudit,” it was just an hour of building more and more complicated policies for procmail before stuff started working.

Once everything was good, happy, and shiny, a simple semodule -B returned the SELinux logging to the previous state, and I could once again spend my time fighting the convoluted bit-twiddling of the communication protocol.

We had a Hack o’ Lantern this year, who responded to tweets, tweeted jokes, and “accidentally” revealed it’s plans for world domination.

Pure pumpkin cyber evil

For the World Domination Plan, I needed some artwork, which I hereby inflict upon the world.

They’re remixes. Tux is courtesy of Larry Ewing, Simon Budig, and Anja Gerwinski.

Based on this tweet, I now have the ultimate JavaScript approximation for Pi (Π), which I think we can all agree is preferable in every way to the outmoded Math.PI:

var pi=((++[+[]][+[]]+[]+ ++[+[]][+[]]+[])* ++[+[]][+[]])*(++[+[]][+[]]+ ++[+[]][+[]])/((+[+[]]+'x'+(![]+[])[[+!+[]+!+[]]*[+!+[]+!+[]]])/(++[+[]][+[]]+ ++[+[]][+[]]));

For a project that communicates over an expensive network connection (i.e., one that charges by the kilobyte), I needed to find out exactly how many bytes a specific process was going to transfer between my source host and a destination machine. For my own nefarious purposes, I need to know how many bytes of payload data I’m sending/receiving, but I also need to know the true total data transfer, including TCP/IP headers, etc.

Over the years, I have accumulated a few tricks for measuring this sort of thing. Usually, though, I’ve had to measure one particular kind of traffic (specifically, HTTP) — in which case, it’s not hard to set up a proxy using nc. In this latest case, however, the process not only uses HTTP/HTTPS, but ssh to issue remote commands, so I need to monitor all TCP/IP traffic between the machines.

There are other tools that are sometimes helpful. For example, to see what’s using up bandwidth at a given moment, a tool like iftop is great. Unfortunately, I need to know the aggregates, and iftop doesn’t log to a file in a way that I can use.

If I were on a pure Linux environment, it looks like IPTraf would do what I want, but I’m using a Mac.

I don’t doubt that there are much better approaches out there¹, but here’s what I used (pretending that the remote host was at IP 192.168.1.100):

sudo tcpdump -e host 192.168.1.100 > net_process_log.txt
perl -p -i.bak -e 's/(.*?)length (\d+):(.*)length (\d+)/$2,$4/g' net_process_log.txt
cut -d , -f 1 net_process_log.txt > actual_size.txt
cut -d , -f 2 net_process_log.txt > data_size.txt
awk '{s+=$1} END {print s}' actual_size.txt
138099
awk '{s+=$1} END {print s}' data_size.txt
102412
So, in my example, I’m using tcpdump to output all traffic between my machine and the host 192.168.1.100. Typical records output from tcpdump looks like:

11:13:23.834080 xx:xx:xx:xx:xx:xx (oui Unknown) > xx:xx:xx:xx:xx:xx (oui Unknown), ethertype IPv4 (0x0800), length 292: dvr.home.http > apotheosis.home.64602: Flags [P.], seq 1:227, ack 468, win 1354, options [nop,nop,TS val 132300159 ecr 1750033024], length 226
11:13:23.834081 xx:xx:xx:xx:xx:xx (oui Unknown) > xx:xx:xx:xx:xx:xx (oui Unknown), ethertype IPv4 (0x0800), length 66: dvr.home.http > apotheosis.home.64602: Flags [.], ack 469, win 1354, options [nop,nop,TS val 132300159 ecr 1750033026], length 0

There are two lengths specified: the first is the actual packet size, and the second is the payload of the packet. As you can see in the second packet, the payload is zero bytes, but the packet length is 66 bytes.

In any case, I use perl to extract the two lengths into a comma-delimited file, cut to split out the columns, and awk to add them up. It’d be trivial to do all these steps together in a short perl program, but I like keeping around tons of obscure text files from forgotten procedures on my hard drive. Well, actually, I did it this way so I could sanity-check the intermediate steps.

So, this taught me that my process transferred 102,412 bytes of payload, and with the TCP/IP packet overhead transmitted a total of 138,099 bytes.

¹ I didn’t discover bwm-ng until after I did my measurement. It looks like it might be a good solution as well. I probably could have used Wireshark too.

… because I wanted to play with HTML 5 Canvas.

This should run on any reasonably sane browser. Play with it yourself here.