Wed, 17 Sep 2014

PHP 5.5, PHPUnit/Selenium, Yii, and Mac Ports on Mavericks.

— SjG @ 4:07 pm

There is a special place in hell where every programming project creates its own package manager and you have to use it if you want to use the project. Oh, wait. That’s not a special place in hell. That’s here. Crap.

Some days, I shouldn’t bother getting out of bed. On those days, I have that weird broken cognition that I call “user brain” — I don’t read error messages, I don’t stop to think, and instead of looking at problems and trying to solve them, I randomly bounce around and break things.

So I had User Brain. I was setting up a new name-based SSL virtual host in my development environment, and dropped a line from the configuration. Then I didn’t read the Apache error message carefully, and foolishly thought “Oh, I must need to update my certificate trust chain.” This resulted in my updating my environment via mac ports — and when it asked me if I wanted to migrate from php5 to php5.4, I said OK.

When I finished that process, my development environment was completely b0rken, and I hadn’t solved the original problem. Fast forward an hour of questionable language, and I had a new strategy. Install PHP 5.5!

There are several guides out there on how to make all this stuff work, but they didn’t really work for me. What did, was the following. First, I logged in as root. If I’m going to make stupid mistakes, it’s best to do it with unlimited power. Anything else is unsporting.

# port uninstall --follow-dependents php54

That got rid of all the PHP 5.4 stuff left over from my previous debacle. So now it’s time to install PHP 5.5, and the various modules I need:

# port selfupdate
# port upgrade outdated
# port install php55 php55-gd php55-mbstring php55-iconv php55-curl php55-zip php55-soap php55-pcntl php55-xmlrpc php55-openssl php55-sockets php55-imagick php55-mcrypt php55-solr php55-mysql
# port select --set php php55
# port install php55-apache2handler
# cd /opt/local/apache2/modules
# /opt/local/apache2/bin/apxs -a -e -n php5

That’s all pretty straightforward. But my httpd.conf had a few stray lines in it that needed cleaning up. I had to replace LoadModule php5_module modules/
LoadModule php5_module modules/
and change
Include conf/extra/mod_php54.conf


Include conf/extra/mod_php55.conf

Now I could start up Apache and see what kinds of errors I had.

# /opt/local/apache2/bin/apachectl restart

Well, this is where I finally found and fixed the dropped line that got me started on this whole sordid adventure. I also discovered I didn’t have a good php.ini file, so I copied the distribution one and customized it.

# cp /opt/local/etc/php55/php.ini-development /opt/local/etc/php55/php.ini
# vi !$
# /opt/local/apache2/bin/apachectl restart

Now my dev sites were back up, but I couldn’t run tests. The way that phpunit is structured has changed since the days when it was a Pear package. It now uses a phar package and composer. Now, at some point, I’ll restructure my Yii projects the New Right Way and have a phpunit distribution per project, but I need to get back productive, so I set it up globally for the time being.

# cd /opt/local/bin
# curl -sS | php
# mv composer.phar composer
# wget
# chmod +x phpunit.phar
# mv phpunit.phar phpunit
# composer global require 'phpunit/phpunit-selenium=*'
# composer global require 'phpunit/phpunit-story=*'
# composer global require 'phpunit/dbunit=*'
# composer global require 'phpunit/php-invoker=*'

This is where I discovered the php.ini had set the default include path to a not-quite-right location. I ended up changing



and then once again restarted the environment.
# /opt/local/apache2/bin/apachectl restart

Now there were only two things wrong. The current project is based on Yii 1.1.x, and it still tries to include individual PHPUnit files.

Based on numerous forum posts and false starts, I edited framework/test/CtestCase.php, and changed those “requires” up top to:

if (!method_exists('PHPUnit_Runner_Version', 'id') ||
version_compare(PHPUnit_Runner_Version::id(), '4', '< =') ) { require_once('PHPUnit/Runner/Version.php'); require_once('PHPUnit/Autoload.php'); }

PHPUnit unit tests now run, as do Selenium tests, but they do throw an ugly warning on test failures to tell me:

Warning: include(): Failed opening 'PHPUnit_Extensions_Story_TestCase.php' for inclusion (include_path='.:/Users/samuelg/project/X/Y/protected/commands:/Users/samuelg/project/X/Y/protected/behaviors:/Users/samuelg/project/X/Y/protected/components:/Users/samuelg/project/X/Y/protected/nuclear_secrets:/Users/samuelg/project/X/Y/protected/models:/opt/local/lib/php') in /Users/samuelg/project/X/Y/framework/YiiBase.php on line 418

Nor does it give me line numbers of failures in Selenium tests if I enable screenshots. It’s also seemingly taking screen shots of the wrong pages of the site when there’s an error. I don’t yet have a solution to any of these problems.

Lastly, PHP 5.5 is more sensitive about some things than I expected. First, I had to migrate some crappy developer support code from using direct mysql commands (e.g., mysql_connect, mysql_query, etc.) as they’re deprecated. I now use PDO, and that’s all fine.

But beware PHP being stupid about what’s a variable and what’s a reference. In PHP 5.5 strict mode, it whimpers about things like

$foo = end(my_func());

even if my_func() returns an array. Now, in a sane language, I’d be able to do stuff like that, because hey! an array’s an array, even if it’s being returned by a function! I mean, not to devolve into pointless discussion of language implementations and flaws, but flaming owls of imminent death, is PHP a mess!

Anyway, this post is a mostly cleaned-up version of my little adventure. If it spares anyone else some of the same pain, it will be worth while.

No, the project doesn’t really have a nuclear secrets directory. I just felt stupid nerfing the project name in the first place.

Tue, 22 Apr 2014

Annoying Xcode issue and resolution

— SjG @ 3:08 pm

I’d upgraded by home machine to Mavericks fairly soon after the OS was released, but hesitated in upgrading my main work machine. I didn’t want to have extensive downtime while tracking down odd dependencies and incompatibilities.

Well, time came to upgrade. It seemed safe. Everything was fine on my home machine. So I went ahead and upgraded my work machine.

Suddenly, I couldn’t get MacPorts to build MySQL.

I had followed the migration guide carefully. I tried all the usual tricks. In the port /opt/local/var/macports/build/…/config.log, the error was:

ld: library not found for -lcrt1.10.6.o

Google seemed to think this indicated that my Xcode command-line tools were not installed correctly. That library should be installed with all of the Unixy support that comes with Xcode’s command-line tools. Within the Xcode application, it told me that I had the command-line tools 5.1.1 (5B1008) properly installed. When trying various command-line options, the command-line tools were, in fact, installed. For example, xcrun gave the exact results one would expect.

Other tests also made it look like everything was good:
# xcode-select -p

Numerous Googled sites said to use the “xcode-select” command to install the tools if they were not functioning properly. Eventually, I gave in and tried it. Interestingly, this yielded an unexpected result:
# xcode-select --install
xcode-select: error: no developer tools were found, and no install
could be requested (perhaps no UI is present), please install
manually from ''.

Since I had originally installed Xcode under Leopard from a downloaded package rather than recently through the App store, I thought perhaps in the various upgrades something had gotten messed up. I decided to completely uninstalled Xcode:

/Developer/Library/uninstall-devtools --mode=all
I also removed the vestigial /Developer directory, and the /Application/Xcode directory.

I re-installed Xcode from the App store. Everything functioned and/or failed exactly as it had before.

In desperation, I downloaded the Mavericks command line tools package from Apple, and installed it. It should be the same thing as what was installed with Xcode. But it evidently is not, because now I can build the MacPort for MySQL.

edit/update: It may not be clear above, but normally doing the “xcode-select –install” is all that’s needed. It’s also not stated above, but I tried that after re-installing Xcode from scratch, and had the same issue. Evidently, whatever was mis-configured on my machine is quite rare.

Also, don’t trust Xcode when it tells you that it’s installed the command-line tools in the preferences panel like this: Locations_and_Macintosh_HD
It’s probably lying to you – it’s installed stubs, but not the actual tools.

Sun, 21 Apr 2013

Measuring network traffic between two hosts

— SjG @ 10:28 am

For a project that communicates over an expensive network connection (i.e., one that charges by the kilobyte), I needed to find out exactly how many bytes a specific process was going to transfer between my source host and a destination machine. For my own nefarious purposes, I need to know how many bytes of payload data I’m sending/receiving, but I also need to know the true total data transfer, including TCP/IP headers, etc.

Over the years, I have accumulated a few tricks for measuring this sort of thing. Usually, though, I’ve had to measure one particular kind of traffic (specifically, HTTP) — in which case, it’s not hard to set up a proxy using nc. In this latest case, however, the process not only uses HTTP/HTTPS, but ssh to issue remote commands, so I need to monitor all TCP/IP traffic between the machines.

There are other tools that are sometimes helpful. For example, to see what’s using up bandwidth at a given moment, a tool like iftop is great. Unfortunately, I need to know the aggregates, and iftop doesn’t log to a file in a way that I can use.

If I were on a pure Linux environment, it looks like IPTraf would do what I want, but I’m using a Mac.

I don’t doubt that there are much better approaches out there1, but here’s what I used (pretending that the remote host was at IP

sudo tcpdump -e host > net_process_log.txt
perl -p -i.bak -e 's/(.*?)length (\d+):(.*)length (\d+)/$2,$4/g' net_process_log.txt
cut -d , -f 1 net_process_log.txt > actual_size.txt
cut -d , -f 2 net_process_log.txt > data_size.txt
awk '{s+=$1} END {print s}' actual_size.txt
awk '{s+=$1} END {print s}' data_size.txt

So, in my example, I’m using tcpdump to output all traffic between my machine and the host Typical records output from tcpdump looks like:

11:13:23.834080 xx:xx:xx:xx:xx:xx (oui Unknown) > xx:xx:xx:xx:xx:xx (oui Unknown), ethertype IPv4 (0x0800), length 292: dvr.home.http > apotheosis.home.64602: Flags [P.], seq 1:227, ack 468, win 1354, options [nop,nop,TS val 132300159 ecr 1750033024], length 226
11:13:23.834081 xx:xx:xx:xx:xx:xx (oui Unknown) > xx:xx:xx:xx:xx:xx (oui Unknown), ethertype IPv4 (0x0800), length 66: dvr.home.http > apotheosis.home.64602: Flags [.], ack 469, win 1354, options [nop,nop,TS val 132300159 ecr 1750033026], length 0

There are two lengths specified: the first is the actual packet size, and the second is the payload of the packet. As you can see in the second packet, the payload is zero bytes, but the packet length is 66 bytes.

In any case, I use perl to extract the two lengths into a comma-delimited file, cut to split out the columns, and awk to add them up. It’d be trivial to do all these steps together in a short perl program, but I like keeping around tons of obscure text files from forgotten procedures on my hard drive. Well, actually, I did it this way so I could sanity-check the intermediate steps.

So, this taught me that my process transferred 102,412 bytes of payload, and with the TCP/IP packet overhead transmitted a total of 138,099 bytes.

1 I didn’t discover bwm-ng until after I did my measurement. It looks like it might be a good solution as well. I probably could have used Wireshark too.

Wed, 13 Jun 2012

Building Direct IO library for PHP with Mac Ports

— SjG @ 11:47 am

Say you’re developing on a Mac, and want to test some PHP code that has calls to the direct IO library. You may not actually have a physical serial port, but your unit tests will fail in the wrong way if the library isn’t present. You want the unit tests to fail in the right way!

If you do the expected thing, you’ll find that dio is currently beta:

root# pecl install dio
Failed to download pecl/dio within preferred state "stable", latest release is version 0.0.5, stability "beta", use "channel://" to install
install failed

So you have to try to do it the hard way:

samuel:~ root# pecl install channel://
downloading dio-0.0.5.tgz ...

Easy-peasy, eh? Not so fast!

/private/tmp/pear/temp/dio/dio.c: In function 'zif_dio_fdopen':
/private/tmp/pear/temp/dio/dio.c:138: error: 'EBADFD' undeclared (first use in this function)
/private/tmp/pear/temp/dio/dio.c:138: error: (Each undeclared identifier is reported only once
/private/tmp/pear/temp/dio/dio.c:138: error: for each function it appears in.)
make: *** [dio.lo] Error 1
ERROR: `make' failed

Fortunately, via this bug report, we can see what to do:

# pecl download channel://
# tar xzvf dio-0.0.5.tgz
# cd dio-0.0.5
# phpize
# ./configure

Then edit dio.c, and change line 138 to:

if ((fcntl(fd, F_GETFL, 0) == -1) && (errno == EBADF)) {

Then, finish up:

# make
# make install

Then, create a file called “dio.ini” in /opt/local/var/db/php5/ containing:

Now you can run your tests!

Fri, 11 Nov 2011

Imaginary Tech Conversation

— SjG @ 3:14 pm

Coder #1: Man, revision control sucks.
Coder #2: What? Are you insane?
Coder #1: Insane? I don’t think so.
Coder #2: How can you say revision control sucks, then?
Coder #1: Well, I’m stuck using SVN.
Coder #2: Subversion isn’t so bad. Especially with the new merge tracking stuff from version 1.5. It’s pretty easy to use.
Coder #1: Yeah, well, I stand by my assertion. It sucks.
Coder #2: Can you be more specific?
Coder #1: Have you tried using SVN in a mixed environment? Say Windows, Linux, and Mac OS?
Coder #2: It’s not going to be line-ending issues. Oh, I see. You’re talking about case-sensitive versus case-insensitive filesystems.
Coder #1: I wish it was that easy. That’s crappy, but you can work around it.
Coder #2: Then?
Coder #1: A riddle for you. When is UTF-8 not UTF-8?
Coder #2: Huh?
Coder #1: When it’s in a filename.
Coder #2: [clicks, reads]
Coder #1: But I guess if you don’t have any Macs or ZFS around, you’re fine.
Coder #2: Holy Linus on a unicycle! Well, I guess it’s back to dated tar files…
Coder #3: Use GIT, you IDIOT!