Tue, 23 Jun 2015

The File Format Future Problem

— SjG @ 4:26 pm

Trying to find inspiration on a current geometric art project, I went to look at some old work I’d done. Well, or I tried to.

See, back in the 1990s, I spent lots and lots of time doing geometric art using my favorite vector graphics package. You know, MacDraw II. And then, oh happy day, I upgraded to MacDraw Pro, and later yet to Claris Draw. I don’t recall whether the file format changed between MacDraw II and MacDraw Pro, but there were definitely changes when it became Claris Draw.

Claris Draw continued to run on Mac OS until Apple abandoned PowerPC code support back in 10.5.

Now, five years ago, I started looking for a way to use those old files, and discovered Intaglio, which would read the files — mostly. Some of the really large files didn’t work so well. Even though I purchased a license, there was an upgrade to Intaglio that would have required I re-buy it to fix some bugs. It didn’t seem like there were new versions being released, and support was half-hearted at best1. So I gave up.

A year ago, I tried a number of other programs. I need to convert an old architectural diagram, and found that EazyDraw’s Retro version would read the formats. What’s more, they had a neat “rent-for-nine-month” license for just $20. I bought that, converted a few files, planned to convert all my old backlog, and promptly got distracted and let the nine-month license expire before doing anything.

So tonight, I bought another nine-month license, and went through and converted a few hundred files.

The opening of the old files is not perfect. One class of problem is fonts that I no longer have. Some of the problems seem to be positioning of elements (particularly on the oldest MacDraw files), which sometimes all pile up in one corner — but the elements are all there and could be rearranged to restore the original if I wanted to put the work into it. In some cases, it appears that complex elements (like groups) have acquired a background color. That too is easily remedied.

But what to convert to?

My vector graphics program of choice these days is Affinity Designer, and of course I can’t convert directly to their format. SVG worked well, until I discovered that ClarisDraw layers got ignored and left out of the converted file. Also, if you enable SVG Tidy, enough resolution gets removed from points that lines can shift around.

In the end, PDF results in the cleanest transfer. It preserves all the geometry and groupings.

But things do get lost. This is a probably a function of the lack of universal standard for vector images. One program may support the collecting of geometrical objects into layers. Another may not. In my conversion process, I’m losing the original layers. The geometry, however, is still preserved in a way that would let me move things around to different layers in Affinity Designer, so I’m OK with that.

But this is related to a larger problem. Future-proofing is hard. I say this as a fool with boxes of 5.25″ floppies in a DSDD 40-track hard-sector format readable by only a handful of TRS-80s from thirty plus years ago. But even if the media was still good, and even if I could find hardware to read it, what then? What would I do with text files created in Electric Pencil? I have to face the fact that the games I wrote in Z-80 assembler are gone, buried in the sands of time. I mean, maybe I could find a way to read the disks (if they’re not completely faded away), and maybe I could find a TRS-80 simulator that supports some of the hacks I did, and maybe it would all work. But the amount of time required would be substantial. And for what? Revisit some nostalgia of my misspent youth?

But some of my old stuff I’d like to keep around. You could argue that I should have been paying attention and migrating data as I go. Guilty as charged. But it’s hard. And stuff inevitably falls through the cracks. For example, I use Apple’s Aperture for organizing and editing my photography (digital workflow, digital asset management, DAM … whatever you want to call it). Originally, I used a clumsy system of directories. Then I graduated to iView MediaPro. It was the same problem as MacDraw/ClarisDraw — the company behind the software shifted priorities, and support lagged. Then it was acquired by Microsoft, and any future Mac support looked questionable 2. In any case, I went through a painful process to export my edits, labels, keywords, projects from MediaPro into Aperture. But now Apple has ended development of Aperture. There’s a script to migrate to Adobe’s Lightroom. It’ll bring across my keywords and captions and maybe albums or projects, but it doesn’t preserve the nondestructve edits — and how should it, when there’s no real correlation between many of the edits available in the two programs.

I have close to 50,000 pictures, all of which are organized into albums and projects, most of which have keywords, maybe a third of which have edits, and a small set of which are organized into books. So I face a monumental task to migrate, and in the process I lose the “nondestructive” nature of my edits. I’ll have to export an edited version and an original if I think I’ll ever want to re-edit an image. Furthermore, my organizational approach will need to be revisited, and some of the work I’ve put into organizing will vanish. So I procrastinate. Aperture’s still working (for the time being). I’ll wait until I’m forced to do something.

This could turn into a rant supporting RMS’s philosophy of using pure Free / Open Source software. But that’s not really the solution either. I could just stop updating my Mac’s software, and I’d be able to continue to run the application as-is. I’d probably want to disconnect it from the Internet, because a lot of software fixes are security-related. It would be inconvenient, but it’d work. Until I had a hardware failure of some kind. These issues are not exclusive to non-Free software. Free software changes and evolves too. I have a backup server dependent on Fedora Core 6 for one of the drivers. It works, but if I want to do any security patches, I’m on my own. With Free software, I’m guaranteed to be able to maintain a working system, but I still have to be willing to do all the work. There’s no panacea.

And on the pedestal these words appear:
`My name is Ozymandias, King of Kings:
Look on my works, ye mighty, and despair!’
Nothing beside remains.
– Shelley

1 I’m happy to report that today there seems to be new life in Intaglio. There are posts in their support forum, and new versions have been released.
2 Now, it looks like MediaPro’s been spun off from Microsoft again.

Wed, 17 Sep 2014

PHP 5.5, PHPUnit/Selenium, Yii, and Mac Ports on Mavericks.

— SjG @ 4:07 pm

There is a special place in hell where every programming project creates its own package manager and you have to use it if you want to use the project. Oh, wait. That’s not a special place in hell. That’s here. Crap.

Some days, I shouldn’t bother getting out of bed. On those days, I have that weird broken cognition that I call “user brain” — I don’t read error messages, I don’t stop to think, and instead of looking at problems and trying to solve them, I randomly bounce around and break things.

So I had User Brain. I was setting up a new name-based SSL virtual host in my development environment, and dropped a line from the configuration. Then I didn’t read the Apache error message carefully, and foolishly thought “Oh, I must need to update my certificate trust chain.” This resulted in my updating my environment via mac ports — and when it asked me if I wanted to migrate from php5 to php5.4, I said OK.

When I finished that process, my development environment was completely b0rken, and I hadn’t solved the original problem. Fast forward an hour of questionable language, and I had a new strategy. Install PHP 5.5!

There are several guides out there on how to make all this stuff work, but they didn’t really work for me. What did, was the following. First, I logged in as root. If I’m going to make stupid mistakes, it’s best to do it with unlimited power. Anything else is unsporting.

# port uninstall --follow-dependents php54

That got rid of all the PHP 5.4 stuff left over from my previous debacle. So now it’s time to install PHP 5.5, and the various modules I need:

# port selfupdate
# port upgrade outdated
# port install php55 php55-gd php55-mbstring php55-iconv php55-curl php55-zip php55-soap php55-pcntl php55-xmlrpc php55-openssl php55-sockets php55-imagick php55-mcrypt php55-solr php55-mysql
# port select --set php php55
# port install php55-apache2handler
# cd /opt/local/apache2/modules
# /opt/local/apache2/bin/apxs -a -e -n php5

That’s all pretty straightforward. But my httpd.conf had a few stray lines in it that needed cleaning up. I had to replace LoadModule php5_module modules/
LoadModule php5_module modules/
and change
Include conf/extra/mod_php54.conf


Include conf/extra/mod_php55.conf

Now I could start up Apache and see what kinds of errors I had.

# /opt/local/apache2/bin/apachectl restart

Well, this is where I finally found and fixed the dropped line that got me started on this whole sordid adventure. I also discovered I didn’t have a good php.ini file, so I copied the distribution one and customized it.

# cp /opt/local/etc/php55/php.ini-development /opt/local/etc/php55/php.ini
# vi !$
# /opt/local/apache2/bin/apachectl restart

Now my dev sites were back up, but I couldn’t run tests. The way that phpunit is structured has changed since the days when it was a Pear package. It now uses a phar package and composer. Now, at some point, I’ll restructure my Yii projects the New Right Way and have a phpunit distribution per project, but I need to get back productive, so I set it up globally for the time being.

# cd /opt/local/bin
# curl -sS | php
# mv composer.phar composer
# wget
# chmod +x phpunit.phar
# mv phpunit.phar phpunit
# composer global require 'phpunit/phpunit-selenium=*'
# composer global require 'phpunit/phpunit-story=*'
# composer global require 'phpunit/dbunit=*'
# composer global require 'phpunit/php-invoker=*'

This is where I discovered the php.ini had set the default include path to a not-quite-right location. I ended up changing



and then once again restarted the environment.
# /opt/local/apache2/bin/apachectl restart

Now there were only two things wrong. The current project is based on Yii 1.1.x, and it still tries to include individual PHPUnit files.

Based on numerous forum posts and false starts, I edited framework/test/CtestCase.php, and changed those “requires” up top to:

if (!method_exists('PHPUnit_Runner_Version', 'id') ||
version_compare(PHPUnit_Runner_Version::id(), '4', '< =') ) { require_once('PHPUnit/Runner/Version.php'); require_once('PHPUnit/Autoload.php'); }

PHPUnit unit tests now run, as do Selenium tests, but they do throw an ugly warning on test failures to tell me:

Warning: include(): Failed opening 'PHPUnit_Extensions_Story_TestCase.php' for inclusion (include_path='.:/Users/samuelg/project/X/Y/protected/commands:/Users/samuelg/project/X/Y/protected/behaviors:/Users/samuelg/project/X/Y/protected/components:/Users/samuelg/project/X/Y/protected/nuclear_secrets:/Users/samuelg/project/X/Y/protected/models:/opt/local/lib/php') in /Users/samuelg/project/X/Y/framework/YiiBase.php on line 418

Nor does it give me line numbers of failures in Selenium tests if I enable screenshots. It’s also seemingly taking screen shots of the wrong pages of the site when there’s an error. I don’t yet have a solution to any of these problems.

Lastly, PHP 5.5 is more sensitive about some things than I expected. First, I had to migrate some crappy developer support code from using direct mysql commands (e.g., mysql_connect, mysql_query, etc.) as they’re deprecated. I now use PDO, and that’s all fine.

But beware PHP being stupid about what’s a variable and what’s a reference. In PHP 5.5 strict mode, it whimpers about things like

$foo = end(my_func());

even if my_func() returns an array. Now, in a sane language, I’d be able to do stuff like that, because hey! an array’s an array, even if it’s being returned by a function! I mean, not to devolve into pointless discussion of language implementations and flaws, but flaming owls of imminent death, is PHP a mess!

Anyway, this post is a mostly cleaned-up version of my little adventure. If it spares anyone else some of the same pain, it will be worth while.

No, the project doesn’t really have a nuclear secrets directory. I just felt stupid nerfing the project name in the first place.

Tue, 22 Apr 2014

Annoying Xcode issue and resolution

— SjG @ 3:08 pm

I’d upgraded by home machine to Mavericks fairly soon after the OS was released, but hesitated in upgrading my main work machine. I didn’t want to have extensive downtime while tracking down odd dependencies and incompatibilities.

Well, time came to upgrade. It seemed safe. Everything was fine on my home machine. So I went ahead and upgraded my work machine.

Suddenly, I couldn’t get MacPorts to build MySQL.

I had followed the migration guide carefully. I tried all the usual tricks. In the port /opt/local/var/macports/build/…/config.log, the error was:

ld: library not found for -lcrt1.10.6.o

Google seemed to think this indicated that my Xcode command-line tools were not installed correctly. That library should be installed with all of the Unixy support that comes with Xcode’s command-line tools. Within the Xcode application, it told me that I had the command-line tools 5.1.1 (5B1008) properly installed. When trying various command-line options, the command-line tools were, in fact, installed. For example, xcrun gave the exact results one would expect.

Other tests also made it look like everything was good:
# xcode-select -p

Numerous Googled sites said to use the “xcode-select” command to install the tools if they were not functioning properly. Eventually, I gave in and tried it. Interestingly, this yielded an unexpected result:
# xcode-select --install
xcode-select: error: no developer tools were found, and no install
could be requested (perhaps no UI is present), please install
manually from ''.

Since I had originally installed Xcode under Leopard from a downloaded package rather than recently through the App store, I thought perhaps in the various upgrades something had gotten messed up. I decided to completely uninstalled Xcode:

/Developer/Library/uninstall-devtools --mode=all
I also removed the vestigial /Developer directory, and the /Application/Xcode directory.

I re-installed Xcode from the App store. Everything functioned and/or failed exactly as it had before.

In desperation, I downloaded the Mavericks command line tools package from Apple, and installed it. It should be the same thing as what was installed with Xcode. But it evidently is not, because now I can build the MacPort for MySQL.

edit/update: It may not be clear above, but normally doing the “xcode-select –install” is all that’s needed. It’s also not stated above, but I tried that after re-installing Xcode from scratch, and had the same issue. Evidently, whatever was mis-configured on my machine is quite rare.

Also, don’t trust Xcode when it tells you that it’s installed the command-line tools in the preferences panel like this: Locations_and_Macintosh_HD
It’s probably lying to you – it’s installed stubs, but not the actual tools.

Sun, 21 Apr 2013

Measuring network traffic between two hosts

— SjG @ 10:28 am

For a project that communicates over an expensive network connection (i.e., one that charges by the kilobyte), I needed to find out exactly how many bytes a specific process was going to transfer between my source host and a destination machine. For my own nefarious purposes, I need to know how many bytes of payload data I’m sending/receiving, but I also need to know the true total data transfer, including TCP/IP headers, etc.

Over the years, I have accumulated a few tricks for measuring this sort of thing. Usually, though, I’ve had to measure one particular kind of traffic (specifically, HTTP) — in which case, it’s not hard to set up a proxy using nc. In this latest case, however, the process not only uses HTTP/HTTPS, but ssh to issue remote commands, so I need to monitor all TCP/IP traffic between the machines.

There are other tools that are sometimes helpful. For example, to see what’s using up bandwidth at a given moment, a tool like iftop is great. Unfortunately, I need to know the aggregates, and iftop doesn’t log to a file in a way that I can use.

If I were on a pure Linux environment, it looks like IPTraf would do what I want, but I’m using a Mac.

I don’t doubt that there are much better approaches out there1, but here’s what I used (pretending that the remote host was at IP

sudo tcpdump -e host > net_process_log.txt
perl -p -i.bak -e 's/(.*?)length (\d+):(.*)length (\d+)/$2,$4/g' net_process_log.txt
cut -d , -f 1 net_process_log.txt > actual_size.txt
cut -d , -f 2 net_process_log.txt > data_size.txt
awk '{s+=$1} END {print s}' actual_size.txt
awk '{s+=$1} END {print s}' data_size.txt

So, in my example, I’m using tcpdump to output all traffic between my machine and the host Typical records output from tcpdump looks like:

11:13:23.834080 xx:xx:xx:xx:xx:xx (oui Unknown) > xx:xx:xx:xx:xx:xx (oui Unknown), ethertype IPv4 (0x0800), length 292: dvr.home.http > apotheosis.home.64602: Flags [P.], seq 1:227, ack 468, win 1354, options [nop,nop,TS val 132300159 ecr 1750033024], length 226
11:13:23.834081 xx:xx:xx:xx:xx:xx (oui Unknown) > xx:xx:xx:xx:xx:xx (oui Unknown), ethertype IPv4 (0x0800), length 66: dvr.home.http > apotheosis.home.64602: Flags [.], ack 469, win 1354, options [nop,nop,TS val 132300159 ecr 1750033026], length 0

There are two lengths specified: the first is the actual packet size, and the second is the payload of the packet. As you can see in the second packet, the payload is zero bytes, but the packet length is 66 bytes.

In any case, I use perl to extract the two lengths into a comma-delimited file, cut to split out the columns, and awk to add them up. It’d be trivial to do all these steps together in a short perl program, but I like keeping around tons of obscure text files from forgotten procedures on my hard drive. Well, actually, I did it this way so I could sanity-check the intermediate steps.

So, this taught me that my process transferred 102,412 bytes of payload, and with the TCP/IP packet overhead transmitted a total of 138,099 bytes.

1 I didn’t discover bwm-ng until after I did my measurement. It looks like it might be a good solution as well. I probably could have used Wireshark too.

Wed, 13 Jun 2012

Building Direct IO library for PHP with Mac Ports

— SjG @ 11:47 am

Say you’re developing on a Mac, and want to test some PHP code that has calls to the direct IO library. You may not actually have a physical serial port, but your unit tests will fail in the wrong way if the library isn’t present. You want the unit tests to fail in the right way!

If you do the expected thing, you’ll find that dio is currently beta:

root# pecl install dio
Failed to download pecl/dio within preferred state "stable", latest release is version 0.0.5, stability "beta", use "channel://" to install
install failed

So you have to try to do it the hard way:

samuel:~ root# pecl install channel://
downloading dio-0.0.5.tgz ...

Easy-peasy, eh? Not so fast!

/private/tmp/pear/temp/dio/dio.c: In function 'zif_dio_fdopen':
/private/tmp/pear/temp/dio/dio.c:138: error: 'EBADFD' undeclared (first use in this function)
/private/tmp/pear/temp/dio/dio.c:138: error: (Each undeclared identifier is reported only once
/private/tmp/pear/temp/dio/dio.c:138: error: for each function it appears in.)
make: *** [dio.lo] Error 1
ERROR: `make' failed

Fortunately, via this bug report, we can see what to do:

# pecl download channel://
# tar xzvf dio-0.0.5.tgz
# cd dio-0.0.5
# phpize
# ./configure

Then edit dio.c, and change line 138 to:

if ((fcntl(fd, F_GETFL, 0) == -1) && (errno == EBADF)) {

Then, finish up:

# make
# make install

Then, create a file called “dio.ini” in /opt/local/var/db/php5/ containing:

Now you can run your tests!