Fri, 6 Dec 2013

Another SELinux Lesson

— SjG @ 6:17 pm

So there’s this project that requires a ridiculously complicated communication protocol involving lots of byte-wrangling and formatting and weird transports. For the sake of brevity, I’ll only mention one of the endpoints, which requires decoding an email attachment.

Of course, this means a procmail script sending the email to PHP for processing and much ceremonious mucking about. On first go, it was failing. In debug mode, procmail was telling me that permission was denied. But it wasn’t user permissions: the file was owned by the same user as receiving the email and running the procmail script.

Naturally, when faced with cryptic permission failures, the first thing I did was look at /var/log/audit/audit.log and /var/log/messages for SELinux denials. There I found nothing at all. No errors, no warnings, no ugly “avc: denied” splatters.

Finally, this page here explained it to me. Rebuilding the policies with semodule -DB quickly revealed that my problem was, in fact, SELinux (as it all so often is). Once I could see the policies that were marked “dontaudit,” it was just an hour of building more and more complicated policies for procmail before stuff started working.

Once everything was good, happy, and shiny, a simple semodule -B returned the SELinux logging to the previous state, and I could once again spend my time fighting the convoluted bit-twiddling of the communication protocol.

Sun, 21 Apr 2013

Measuring network traffic between two hosts

— SjG @ 10:28 am

For a project that communicates over an expensive network connection (i.e., one that charges by the kilobyte), I needed to find out exactly how many bytes a specific process was going to transfer between my source host and a destination machine. For my own nefarious purposes, I need to know how many bytes of payload data I’m sending/receiving, but I also need to know the true total data transfer, including TCP/IP headers, etc.

Over the years, I have accumulated a few tricks for measuring this sort of thing. Usually, though, I’ve had to measure one particular kind of traffic (specifically, HTTP) — in which case, it’s not hard to set up a proxy using nc. In this latest case, however, the process not only uses HTTP/HTTPS, but ssh to issue remote commands, so I need to monitor all TCP/IP traffic between the machines.

There are other tools that are sometimes helpful. For example, to see what’s using up bandwidth at a given moment, a tool like iftop is great. Unfortunately, I need to know the aggregates, and iftop doesn’t log to a file in a way that I can use.

If I were on a pure Linux environment, it looks like IPTraf would do what I want, but I’m using a Mac.

I don’t doubt that there are much better approaches out there1, but here’s what I used (pretending that the remote host was at IP

sudo tcpdump -e host > net_process_log.txt
perl -p -i.bak -e 's/(.*?)length (\d+):(.*)length (\d+)/$2,$4/g' net_process_log.txt
cut -d , -f 1 net_process_log.txt > actual_size.txt
cut -d , -f 2 net_process_log.txt > data_size.txt
awk '{s+=$1} END {print s}' actual_size.txt
awk '{s+=$1} END {print s}' data_size.txt

So, in my example, I’m using tcpdump to output all traffic between my machine and the host Typical records output from tcpdump looks like:

11:13:23.834080 xx:xx:xx:xx:xx:xx (oui Unknown) > xx:xx:xx:xx:xx:xx (oui Unknown), ethertype IPv4 (0x0800), length 292: dvr.home.http > apotheosis.home.64602: Flags [P.], seq 1:227, ack 468, win 1354, options [nop,nop,TS val 132300159 ecr 1750033024], length 226
11:13:23.834081 xx:xx:xx:xx:xx:xx (oui Unknown) > xx:xx:xx:xx:xx:xx (oui Unknown), ethertype IPv4 (0x0800), length 66: dvr.home.http > apotheosis.home.64602: Flags [.], ack 469, win 1354, options [nop,nop,TS val 132300159 ecr 1750033026], length 0

There are two lengths specified: the first is the actual packet size, and the second is the payload of the packet. As you can see in the second packet, the payload is zero bytes, but the packet length is 66 bytes.

In any case, I use perl to extract the two lengths into a comma-delimited file, cut to split out the columns, and awk to add them up. It’d be trivial to do all these steps together in a short perl program, but I like keeping around tons of obscure text files from forgotten procedures on my hard drive. Well, actually, I did it this way so I could sanity-check the intermediate steps.

So, this taught me that my process transferred 102,412 bytes of payload, and with the TCP/IP packet overhead transmitted a total of 138,099 bytes.

1 I didn’t discover bwm-ng until after I did my measurement. It looks like it might be a good solution as well. I probably could have used Wireshark too.

Fri, 12 Oct 2012

Resizing LVM partitions

— SjG @ 10:30 am

I ran into a problem where the default installer for RHEL had configured a server’s partitions in a way that was really suboptimal for how the server was being used. It looks like the default was intended for workstation use or one in which a lot of users would keep their stuff in home directories, but this system is being used as a web application server. The root partition filled up, and the database (with its files in /var) got quite unhappy.

[root@epicwin]# df -h
Filesystem Size Used Avail Use% Mounted on
50G 47G 0 100% /
tmpfs 1.9G 76K 1.9G 1% /dev/shm
/dev/sde1 485M 80M 380M 18% /boot
532G 198M 505G 1% /home

So I had to learn some LVM basics, and move the space around. I found various instructions out there, but many of them were missing steps or glossed over whether or not I’d suffer data loss. Here’s how I solved my problem (without data loss!) and got the system back on its feet.

  • Boot into “rescue” mode.
  • Decline to mount the existing system or filesystem for repair, instead go straight to a root shell.
  • Activate the LVM system:
    lvm vgchange -a y
  • Make sure the filesystems are clean:
    e2fsck -f /dev/mapper/vg_epicwin-lv_home
    e2fsck -f /dev/mapper/vg_epicwin-lv_root
  • Reduce the size of the lv_home logical volume from 540G to 40G:
    resize2fs /dev/mapper/vg_epicwin-lv_home 40g
    lvreduce -L 40g /dev/mapper/vg_epicwin-lv_home
  • Expand the size of the lv_root logical volume from 50G to 550G:

    lvextend -L +500g /dev/mapper/vg_epicwin-lv_root

  • Expand the filesystem on the root logical volume:

    resize2fs /dev/mapper/vg_epicwin-lv_root

  • Exit shell / reboot.
  • Watch SELinux freak out on next boot:

    *** Warning -- SELinux targeted policy relabel is required. ***
    *** Relabeling could take a very long time, depending on file ***
    *** system size and speed of hard drives. ***

  • Watch SELinux print asterisks and then the system reboot unexpectedly in a way that looks quite disturbingly bad.
  • System comes up clean!

A few notes:
If /home had a lot of stuff in it, this would obviously be a lot more complicated. This approach works because I had the space and it wasn’t being utilized. if /home had been closer to full, the approach would have been to add a new physical drive, and it to the logical volume set, and then do something similar to the above.

Mon, 6 Aug 2012

Sneaky, sneaky SELinux

— SjG @ 4:38 pm

I was getting a web site up and running on Centos 6.3, and PHP was refusing to talk to the SMTP server on the same box. This, in and of itself, is not unusual. There’s a checklist:

  • Verify SMTP’s running
  • Check that the SMTP server is configured to accept connections from localhost
  • Check the firewall rules
  • Really check the firewall rules by generating an email using telnet localhost 25 or nc localhost 25 from the command line
  • Check that PHP is trying to talk to the SMTP server you think it is
  • Look in the system log for SELinux being a bully and preventing communication

In this case, all of the above checked out cleanly. I was perplexed. But it turns out that, in what is as shocking and surprising as my cat wanting to be fed,
SELinux was indeed the culprit. It just wasn’t logging the problem anywhere.

A quick
setsebool -P httpd_can_network_connect=1

as root solved the problem.

Moral(s) of the Story: The cat is always hungry. SELinux is always fuckin’ shit up.

Tue, 17 Jan 2012

Sign of the Times

— SjG @ 2:58 pm

I think it’s a sad, sad sign of the times that most Linux distros not only omit figlet from their standard installations, but often don’t even offer it in their package managers.

       _           _                   _                      _ 
  __ _| | __ _ ___| |   __ _ _ __   __| | __      _____   ___| |
 / _` | |/ _` / __| |  / _` | '_ \ / _` | \ \ /\ / / _ \ / _ \ |
| (_| | | (_| \__ \_| | (_| | | | | (_| |  \ V  V / (_) |  __/_|
 \__,_|_|\__,_|___(_)  \__,_|_| |_|\__,_|   \_/\_/ \___/ \___(_)