Page 1 of 612345...Last »

Wed, 11 Nov 2015

Mac Ports and X-Code Command-line tools

— SjG @ 1:59 pm

I was setting up a new Mac with Mac Ports… and ran into a weird problem.

First, installed Xcode from the App store. Took a long time, but was done. I mistakenly thought it had installed the command-line tools, because now I could run commands like “svn” from the command line. The Department of Tautologies reminds you that some command-line tools are command-line tools, and others are not.

So I tracked down my build problem to the fact that command-line tools were not installed. Following every guide everywhere, I typed:
xcode-select --install
But instead of the expected glorious dialog box and installation, I got the virtually-un-Googlable error:
xcode-select: error: no developer tools were found, and no install could be requested (perhaps no UI present), please install manually from ''.

To make a long story short, the problem was that I ran the xcode-select as root. Eventually, I found that running it as an ordinary user worked as expected.

After that, command-line tools are properly installed, and ports work as God intended them to.

There are a lot of deceptive distractions out there.
Typing, for example, xcode-select -p returns the correct /Applications/ path that you’d expect.
Similarly, the error in the port build was that the directory /usr/include does not exist.
Don’t be deceived. These are lies!

Sun, 18 Oct 2015

Yii mystery on Mac

— SjG @ 3:07 pm

I upgraded the Mac to Yosemite a year or so ago. Yesterday, I wanted to do some development on a project that I’d been idly thinking about. Unfortunately, it required a dependency in a package I’d installed via Mac Ports. I tried to upgrade it, but got an error that I was compiling for the wrong Darwin version. This means I haven’t actually updated any of my Ports since upgrading to Yosemite! For shame.

Rather than fix Mac Ports for Yosemite, and then again when I upgrade to El Capitan, I decided it was time to do that upgrade and then fix it. I went through and did so, and upgraded all my ports, and it all seemed to go well. I went from PHP 5.3 to PHP 5.5, and MySQL from 5.1 to 5.5, and it went without a snag — the web server came up and my old configuration was good, databases same. Everything seemed sweet and easy!

Ha! Sweet and easy with software? Not so fast, buckeroo! Working on another project, I found that Yii was crashing — but only from the command line!
exception 'CDbException' with message 'CDbConnection failed to open the DB connection: could not find driver' in /Users/samuelg/project/unicorn_rainbows/framework/db/CDbConnection.php:399

I quickly through a page up with phpinfo(), and saw that all the usual suspects were valid:

  • I was running the PHP I thought I was (Mac Ports version 5.5, not the built-in Mac OS version)
  • It was using the php.ini file I thought it was (in /opt/local/etc/php55/)
  • PDO was installed
  • PDO’s MySQL driver was installed
  • PHP’s configured recognized the drivers
  • Paths to any config files were correct
  • Ports were all normal
  • Default path to MySQL socket file was correct

Of course, this all makes sense, because my web pages that access the database were working.

So why not from Yii’s console program from the command line?

A quick php -version revealed the problem. It wasn’t the database configuration at all! Well, not exactly.
samuelg$ php --version
PHP 5.6.14 (cli) (built: Oct 15 2015 16:20:41)
Copyright (c) 1997-2015 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2015 Zend Technologies

Wait, what? PHP 5.6? But… how?

samuel$ port installed | grep php
php55-mysql @5.5.30_0+mysqlnd (active)
php55-sqlite @5.5.30_0 (active)
php56 @5.6.6_0+libedit
php56 @5.6.14_0+libedit (active)

Yup, somehow, I’d installed some PHP 5.6 packages as well!

To solve the problem quickly, I uninstalled PHP 5.6. I could have upgraded everything to 5.6 (or just inactivated them, I suppose), but I just wanted to work on my original problem, not spend my day on system configuration.

Mon, 29 Jun 2015

Sorting lots of files into directories, by date

— SjG @ 10:32 am

A process handles periodic tasks, and each time it does, it spits out some telemetry. The telemetry gets written off into a file each time. This process could be any of a number of interesting things: a procmail script doing something with incoming email, a Twitter-bot responding to searches, an IRC-bot responding to events, or whatever. The only important thing in this case is that it creates a variable number of log files each day, and the log files all get dumped into a single directory.

Before long, the log directory will be filled with thousands of files, and will be unmanageable. But, for some reason, we want to keep all these logs, and maybe actually use them. So the key is a script to move the logs into sub-directories based on date.

It turns out it’s easy to write a bash script to create directories in a YYYY-MM format, and move the files into them appropriately. The key is in the stat command. Conveniently, the implementation of this command is completely different and incompatible between Mac OS/BSD and Linux. Jesus H. Christ.


for i in *.log
filemonth=`stat --format=%y $i | cut -c 1-7`
mkdir -p $filemonth
mv $i $filemonth/$i

In Linux, the stat command will give you the data in an ISO format, and using the cut command, you can extract YYYY-MM information. The -p flag to mkdir makes it silently exit without complaining if a directory by that name already exists.

MacOS (or presumably other BSDs):

for i in *.log
filemonth=`stat -f%Sm -t %Y-%m $i`
mkdir -p $filemonth
mv $i $filemonth/$i

In this case, we’re telling stat that we want the modified date as a string, and we specify the time format.

Either of these would be easy to modify to single day resolution (changing which columns you cut in the Linux version, or the timestamp format in the Mac version).

Tue, 23 Jun 2015

The File Format Future Problem

— SjG @ 4:26 pm

Trying to find inspiration on a current geometric art project, I went to look at some old work I’d done. Well, or I tried to.

See, back in the 1990s, I spent lots and lots of time doing geometric art using my favorite vector graphics package. You know, MacDraw II. And then, oh happy day, I upgraded to MacDraw Pro, and later yet to Claris Draw. I don’t recall whether the file format changed between MacDraw II and MacDraw Pro, but there were definitely changes when it became Claris Draw.

Claris Draw continued to run on Mac OS until Apple abandoned PowerPC code support back in 10.5.

Now, five years ago, I started looking for a way to use those old files, and discovered Intaglio, which would read the files — mostly. Some of the really large files didn’t work so well. Even though I purchased a license, there was an upgrade to Intaglio that would have required I re-buy it to fix some bugs. It didn’t seem like there were new versions being released, and support was half-hearted at best1. So I gave up.

A year ago, I tried a number of other programs. I need to convert an old architectural diagram, and found that EazyDraw’s Retro version would read the formats. What’s more, they had a neat “rent-for-nine-month” license for just $20. I bought that, converted a few files, planned to convert all my old backlog, and promptly got distracted and let the nine-month license expire before doing anything.

So tonight, I bought another nine-month license, and went through and converted a few hundred files.

The opening of the old files is not perfect. One class of problem is fonts that I no longer have. Some of the problems seem to be positioning of elements (particularly on the oldest MacDraw files), which sometimes all pile up in one corner — but the elements are all there and could be rearranged to restore the original if I wanted to put the work into it. In some cases, it appears that complex elements (like groups) have acquired a background color. That too is easily remedied.

But what to convert to?

My vector graphics program of choice these days is Affinity Designer, and of course I can’t convert directly to their format. SVG worked well, until I discovered that ClarisDraw layers got ignored and left out of the converted file. Also, if you enable SVG Tidy, enough resolution gets removed from points that lines can shift around.

In the end, PDF results in the cleanest transfer. It preserves all the geometry and groupings.

But things do get lost. This is a probably a function of the lack of universal standard for vector images. One program may support the collecting of geometrical objects into layers. Another may not. In my conversion process, I’m losing the original layers. The geometry, however, is still preserved in a way that would let me move things around to different layers in Affinity Designer, so I’m OK with that.

But this is related to a larger problem. Future-proofing is hard. I say this as a fool with boxes of 5.25″ floppies in a DSDD 40-track hard-sector format readable by only a handful of TRS-80s from thirty plus years ago. But even if the media was still good, and even if I could find hardware to read it, what then? What would I do with text files created in Electric Pencil? I have to face the fact that the games I wrote in Z-80 assembler are gone, buried in the sands of time. I mean, maybe I could find a way to read the disks (if they’re not completely faded away), and maybe I could find a TRS-80 simulator that supports some of the hacks I did, and maybe it would all work. But the amount of time required would be substantial. And for what? Revisit some nostalgia of my misspent youth?

But some of my old stuff I’d like to keep around. You could argue that I should have been paying attention and migrating data as I go. Guilty as charged. But it’s hard. And stuff inevitably falls through the cracks. For example, I use Apple’s Aperture for organizing and editing my photography (digital workflow, digital asset management, DAM … whatever you want to call it). Originally, I used a clumsy system of directories. Then I graduated to iView MediaPro. It was the same problem as MacDraw/ClarisDraw — the company behind the software shifted priorities, and support lagged. Then it was acquired by Microsoft, and any future Mac support looked questionable 2. In any case, I went through a painful process to export my edits, labels, keywords, projects from MediaPro into Aperture. But now Apple has ended development of Aperture. There’s a script to migrate to Adobe’s Lightroom. It’ll bring across my keywords and captions and maybe albums or projects, but it doesn’t preserve the nondestructve edits — and how should it, when there’s no real correlation between many of the edits available in the two programs.

I have close to 50,000 pictures, all of which are organized into albums and projects, most of which have keywords, maybe a third of which have edits, and a small set of which are organized into books. So I face a monumental task to migrate, and in the process I lose the “nondestructive” nature of my edits. I’ll have to export an edited version and an original if I think I’ll ever want to re-edit an image. Furthermore, my organizational approach will need to be revisited, and some of the work I’ve put into organizing will vanish. So I procrastinate. Aperture’s still working (for the time being). I’ll wait until I’m forced to do something.

This could turn into a rant supporting RMS’s philosophy of using pure Free / Open Source software. But that’s not really the solution either. I could just stop updating my Mac’s software, and I’d be able to continue to run the application as-is. I’d probably want to disconnect it from the Internet, because a lot of software fixes are security-related. It would be inconvenient, but it’d work. Until I had a hardware failure of some kind. These issues are not exclusive to non-Free software. Free software changes and evolves too. I have a backup server dependent on Fedora Core 6 for one of the drivers. It works, but if I want to do any security patches, I’m on my own. With Free software, I’m guaranteed to be able to maintain a working system, but I still have to be willing to do all the work. There’s no panacea.

And on the pedestal these words appear:
`My name is Ozymandias, King of Kings:
Look on my works, ye mighty, and despair!’
Nothing beside remains.
– Shelley

1 I’m happy to report that today there seems to be new life in Intaglio. There are posts in their support forum, and new versions have been released.
2 Now, it looks like MediaPro’s been spun off from Microsoft again.

Wed, 17 Sep 2014

PHP 5.5, PHPUnit/Selenium, Yii, and Mac Ports on Mavericks.

— SjG @ 4:07 pm

There is a special place in hell where every programming project creates its own package manager and you have to use it if you want to use the project. Oh, wait. That’s not a special place in hell. That’s here. Crap.

Some days, I shouldn’t bother getting out of bed. On those days, I have that weird broken cognition that I call “user brain” — I don’t read error messages, I don’t stop to think, and instead of looking at problems and trying to solve them, I randomly bounce around and break things.

So I had User Brain. I was setting up a new name-based SSL virtual host in my development environment, and dropped a line from the configuration. Then I didn’t read the Apache error message carefully, and foolishly thought “Oh, I must need to update my certificate trust chain.” This resulted in my updating my environment via mac ports — and when it asked me if I wanted to migrate from php5 to php5.4, I said OK.

When I finished that process, my development environment was completely b0rken, and I hadn’t solved the original problem. Fast forward an hour of questionable language, and I had a new strategy. Install PHP 5.5!

There are several guides out there on how to make all this stuff work, but they didn’t really work for me. What did, was the following. First, I logged in as root. If I’m going to make stupid mistakes, it’s best to do it with unlimited power. Anything else is unsporting.

# port uninstall --follow-dependents php54

That got rid of all the PHP 5.4 stuff left over from my previous debacle. So now it’s time to install PHP 5.5, and the various modules I need:

# port selfupdate
# port upgrade outdated
# port install php55 php55-gd php55-mbstring php55-iconv php55-curl php55-zip php55-soap php55-pcntl php55-xmlrpc php55-openssl php55-sockets php55-imagick php55-mcrypt php55-solr php55-mysql
# port select --set php php55
# port install php55-apache2handler
# cd /opt/local/apache2/modules
# /opt/local/apache2/bin/apxs -a -e -n php5

That’s all pretty straightforward. But my httpd.conf had a few stray lines in it that needed cleaning up. I had to replace LoadModule php5_module modules/
LoadModule php5_module modules/
and change
Include conf/extra/mod_php54.conf


Include conf/extra/mod_php55.conf

Now I could start up Apache and see what kinds of errors I had.

# /opt/local/apache2/bin/apachectl restart

Well, this is where I finally found and fixed the dropped line that got me started on this whole sordid adventure. I also discovered I didn’t have a good php.ini file, so I copied the distribution one and customized it.

# cp /opt/local/etc/php55/php.ini-development /opt/local/etc/php55/php.ini
# vi !$
# /opt/local/apache2/bin/apachectl restart

Now my dev sites were back up, but I couldn’t run tests. The way that phpunit is structured has changed since the days when it was a Pear package. It now uses a phar package and composer. Now, at some point, I’ll restructure my Yii projects the New Right Way and have a phpunit distribution per project, but I need to get back productive, so I set it up globally for the time being.

# cd /opt/local/bin
# curl -sS | php
# mv composer.phar composer
# wget
# chmod +x phpunit.phar
# mv phpunit.phar phpunit
# composer global require 'phpunit/phpunit-selenium=*'
# composer global require 'phpunit/phpunit-story=*'
# composer global require 'phpunit/dbunit=*'
# composer global require 'phpunit/php-invoker=*'

This is where I discovered the php.ini had set the default include path to a not-quite-right location. I ended up changing



and then once again restarted the environment.
# /opt/local/apache2/bin/apachectl restart

Now there were only two things wrong. The current project is based on Yii 1.1.x, and it still tries to include individual PHPUnit files.

Based on numerous forum posts and false starts, I edited framework/test/CtestCase.php, and changed those “requires” up top to:

if (!method_exists('PHPUnit_Runner_Version', 'id') ||
version_compare(PHPUnit_Runner_Version::id(), '4', '< =') ) { require_once('PHPUnit/Runner/Version.php'); require_once('PHPUnit/Autoload.php'); }

PHPUnit unit tests now run, as do Selenium tests, but they do throw an ugly warning on test failures to tell me:

Warning: include(): Failed opening 'PHPUnit_Extensions_Story_TestCase.php' for inclusion (include_path='.:/Users/samuelg/project/X/Y/protected/commands:/Users/samuelg/project/X/Y/protected/behaviors:/Users/samuelg/project/X/Y/protected/components:/Users/samuelg/project/X/Y/protected/nuclear_secrets:/Users/samuelg/project/X/Y/protected/models:/opt/local/lib/php') in /Users/samuelg/project/X/Y/framework/YiiBase.php on line 418

Nor does it give me line numbers of failures in Selenium tests if I enable screenshots. It’s also seemingly taking screen shots of the wrong pages of the site when there’s an error. I don’t yet have a solution to any of these problems.

Lastly, PHP 5.5 is more sensitive about some things than I expected. First, I had to migrate some crappy developer support code from using direct mysql commands (e.g., mysql_connect, mysql_query, etc.) as they’re deprecated. I now use PDO, and that’s all fine.

But beware PHP being stupid about what’s a variable and what’s a reference. In PHP 5.5 strict mode, it whimpers about things like

$foo = end(my_func());

even if my_func() returns an array. Now, in a sane language, I’d be able to do stuff like that, because hey! an array’s an array, even if it’s being returned by a function! I mean, not to devolve into pointless discussion of language implementations and flaws, but flaming owls of imminent death, is PHP a mess!

Anyway, this post is a mostly cleaned-up version of my little adventure. If it spares anyone else some of the same pain, it will be worth while.

No, the project doesn’t really have a nuclear secrets directory. I just felt stupid nerfing the project name in the first place.

Page 1 of 612345...Last »