Page 1 of 612345...Last »

Thu, 21 Sep 2017

Time Machine Backups

— SjG @ 3:37 pm

I use Time Machine for my local desktop backups. It’s a nice solution. It sits there quietly backing stuff up, keeping multiple revisions of files, and even keeping it all encrypted so if the external drive gets swiped it’s not going to be easy to get at the data.

Of course, it’s no substitute for a revision control system for code, nor is it good for situations where the office gets annihilated due to stray meteorite or drone strike. It’s not a complete solution, but it’s part of a broader collection of solutions.

Today I was reminded of some of the limitations. I used Time Machine to migrate to a new machine. That’s a pretty sweet process. You wait for a few hours of disk read time, and suddenly a new machine is populated with all your old settings, applications, data, and so on from your old machine.

But I found some things that weren’t quite right. Most of them had to do with processes that keep open files or databases, and don’t get backed up in a clean fashion.

  • Interestingly, Safari didn’t propagate Ublock Origin, which was a manually-added extension. This was the only surprising one of the bunch.
  • MySQL databases. I hadn’t shut down the MySQL server when the backup ran, so the table files from the Time Machine restore were corrupted. I was able to copy the files off the old machine (after shutting down mysqld gracefully), and use those.
  • TimeKeeper‘s datastore files were all corrupted. I had to delete them, and re-export/import the data from the old machine.
  • VMWare Virtual Machines. I knew they’d get corrupted if backed up by TimeMachine for the same reason as the above, but then I forgot that they weren’t backed up. I had to manually copy them off the old machine. This reminded me — if I want backups of my VMs, I need to do it myself!

That’s all thus far. Nothing too surprising, but a good reminder. Just because you’re backing up, doesn’t necessarily mean you’re backing up stuff in a restorable state!

Wed, 11 Nov 2015

Mac Ports and X-Code Command-line tools

— SjG @ 1:59 pm

I was setting up a new Mac with Mac Ports… and ran into a weird problem.

First, installed Xcode from the App store. Took a long time, but was done. I mistakenly thought it had installed the command-line tools, because now I could run commands like “svn” from the command line. The Department of Tautologies reminds you that some command-line tools are command-line tools, and others are not.

So I tracked down my build problem to the fact that command-line tools were not installed. Following every guide everywhere, I typed:
xcode-select --install
But instead of the expected glorious dialog box and installation, I got the virtually-un-Googlable error:
xcode-select: error: no developer tools were found, and no install could be requested (perhaps no UI present), please install manually from ''.

To make a long story short, the problem was that I ran the xcode-select as root. Eventually, I found that running it as an ordinary user worked as expected.

After that, command-line tools are properly installed, and ports work as God intended them to.

There are a lot of deceptive distractions out there.
Typing, for example, xcode-select -p returns the correct /Applications/ path that you’d expect.
Similarly, the error in the port build was that the directory /usr/include does not exist.
Don’t be deceived. These are lies!

Sun, 18 Oct 2015

Yii mystery on Mac

— SjG @ 3:07 pm

I upgraded the Mac to Yosemite a year or so ago. Yesterday, I wanted to do some development on a project that I’d been idly thinking about. Unfortunately, it required a dependency in a package I’d installed via Mac Ports. I tried to upgrade it, but got an error that I was compiling for the wrong Darwin version. This means I haven’t actually updated any of my Ports since upgrading to Yosemite! For shame.

Rather than fix Mac Ports for Yosemite, and then again when I upgrade to El Capitan, I decided it was time to do that upgrade and then fix it. I went through and did so, and upgraded all my ports, and it all seemed to go well. I went from PHP 5.3 to PHP 5.5, and MySQL from 5.1 to 5.5, and it went without a snag — the web server came up and my old configuration was good, databases same. Everything seemed sweet and easy!

Ha! Sweet and easy with software? Not so fast, buckeroo! Working on another project, I found that Yii was crashing — but only from the command line!
exception 'CDbException' with message 'CDbConnection failed to open the DB connection: could not find driver' in /Users/samuelg/project/unicorn_rainbows/framework/db/CDbConnection.php:399

I quickly through a page up with phpinfo(), and saw that all the usual suspects were valid:

  • I was running the PHP I thought I was (Mac Ports version 5.5, not the built-in Mac OS version)
  • It was using the php.ini file I thought it was (in /opt/local/etc/php55/)
  • PDO was installed
  • PDO’s MySQL driver was installed
  • PHP’s configured recognized the drivers
  • Paths to any config files were correct
  • Ports were all normal
  • Default path to MySQL socket file was correct

Of course, this all makes sense, because my web pages that access the database were working.

So why not from Yii’s console program from the command line?

A quick php -version revealed the problem. It wasn’t the database configuration at all! Well, not exactly.
samuelg$ php --version
PHP 5.6.14 (cli) (built: Oct 15 2015 16:20:41)
Copyright (c) 1997-2015 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2015 Zend Technologies

Wait, what? PHP 5.6? But… how?

samuel$ port installed | grep php
php55-mysql @5.5.30_0+mysqlnd (active)
php55-sqlite @5.5.30_0 (active)
php56 @5.6.6_0+libedit
php56 @5.6.14_0+libedit (active)

Yup, somehow, I’d installed some PHP 5.6 packages as well!

To solve the problem quickly, I uninstalled PHP 5.6. I could have upgraded everything to 5.6 (or just inactivated them, I suppose), but I just wanted to work on my original problem, not spend my day on system configuration.

Mon, 29 Jun 2015

Sorting lots of files into directories, by date

— SjG @ 10:32 am

A process handles periodic tasks, and each time it does, it spits out some telemetry. The telemetry gets written off into a file each time. This process could be any of a number of interesting things: a procmail script doing something with incoming email, a Twitter-bot responding to searches, an IRC-bot responding to events, or whatever. The only important thing in this case is that it creates a variable number of log files each day, and the log files all get dumped into a single directory.

Before long, the log directory will be filled with thousands of files, and will be unmanageable. But, for some reason, we want to keep all these logs, and maybe actually use them. So the key is a script to move the logs into sub-directories based on date.

It turns out it’s easy to write a bash script to create directories in a YYYY-MM format, and move the files into them appropriately. The key is in the stat command. Conveniently, the implementation of this command is completely different and incompatible between Mac OS/BSD and Linux. Jesus H. Christ.


for i in *.log
filemonth=`stat --format=%y $i | cut -c 1-7`
mkdir -p $filemonth
mv $i $filemonth/$i

In Linux, the stat command will give you the data in an ISO format, and using the cut command, you can extract YYYY-MM information. The -p flag to mkdir makes it silently exit without complaining if a directory by that name already exists.

MacOS (or presumably other BSDs):

for i in *.log
filemonth=`stat -f%Sm -t %Y-%m $i`
mkdir -p $filemonth
mv $i $filemonth/$i

In this case, we’re telling stat that we want the modified date as a string, and we specify the time format.

Either of these would be easy to modify to single day resolution (changing which columns you cut in the Linux version, or the timestamp format in the Mac version).

Tue, 23 Jun 2015

The File Format Future Problem

— SjG @ 4:26 pm

Trying to find inspiration on a current geometric art project, I went to look at some old work I’d done. Well, or I tried to.

See, back in the 1990s, I spent lots and lots of time doing geometric art using my favorite vector graphics package. You know, MacDraw II. And then, oh happy day, I upgraded to MacDraw Pro, and later yet to Claris Draw. I don’t recall whether the file format changed between MacDraw II and MacDraw Pro, but there were definitely changes when it became Claris Draw.

Claris Draw continued to run on Mac OS until Apple abandoned PowerPC code support back in 10.5.

Now, five years ago, I started looking for a way to use those old files, and discovered Intaglio, which would read the files — mostly. Some of the really large files didn’t work so well. Even though I purchased a license, there was an upgrade to Intaglio that would have required I re-buy it to fix some bugs. It didn’t seem like there were new versions being released, and support was half-hearted at best1. So I gave up.

A year ago, I tried a number of other programs. I need to convert an old architectural diagram, and found that EazyDraw’s Retro version would read the formats. What’s more, they had a neat “rent-for-nine-month” license for just $20. I bought that, converted a few files, planned to convert all my old backlog, and promptly got distracted and let the nine-month license expire before doing anything.

So tonight, I bought another nine-month license, and went through and converted a few hundred files.

The opening of the old files is not perfect. One class of problem is fonts that I no longer have. Some of the problems seem to be positioning of elements (particularly on the oldest MacDraw files), which sometimes all pile up in one corner — but the elements are all there and could be rearranged to restore the original if I wanted to put the work into it. In some cases, it appears that complex elements (like groups) have acquired a background color. That too is easily remedied.

But what to convert to?

My vector graphics program of choice these days is Affinity Designer, and of course I can’t convert directly to their format. SVG worked well, until I discovered that ClarisDraw layers got ignored and left out of the converted file. Also, if you enable SVG Tidy, enough resolution gets removed from points that lines can shift around.

In the end, PDF results in the cleanest transfer. It preserves all the geometry and groupings.

But things do get lost. This is a probably a function of the lack of universal standard for vector images. One program may support the collecting of geometrical objects into layers. Another may not. In my conversion process, I’m losing the original layers. The geometry, however, is still preserved in a way that would let me move things around to different layers in Affinity Designer, so I’m OK with that.

But this is related to a larger problem. Future-proofing is hard. I say this as a fool with boxes of 5.25″ floppies in a DSDD 40-track hard-sector format readable by only a handful of TRS-80s from thirty plus years ago. But even if the media was still good, and even if I could find hardware to read it, what then? What would I do with text files created in Electric Pencil? I have to face the fact that the games I wrote in Z-80 assembler are gone, buried in the sands of time. I mean, maybe I could find a way to read the disks (if they’re not completely faded away), and maybe I could find a TRS-80 simulator that supports some of the hacks I did, and maybe it would all work. But the amount of time required would be substantial. And for what? Revisit some nostalgia of my misspent youth?

But some of my old stuff I’d like to keep around. You could argue that I should have been paying attention and migrating data as I go. Guilty as charged. But it’s hard. And stuff inevitably falls through the cracks. For example, I use Apple’s Aperture for organizing and editing my photography (digital workflow, digital asset management, DAM … whatever you want to call it). Originally, I used a clumsy system of directories. Then I graduated to iView MediaPro. It was the same problem as MacDraw/ClarisDraw — the company behind the software shifted priorities, and support lagged. Then it was acquired by Microsoft, and any future Mac support looked questionable 2. In any case, I went through a painful process to export my edits, labels, keywords, projects from MediaPro into Aperture. But now Apple has ended development of Aperture. There’s a script to migrate to Adobe’s Lightroom. It’ll bring across my keywords and captions and maybe albums or projects, but it doesn’t preserve the nondestructve edits — and how should it, when there’s no real correlation between many of the edits available in the two programs.

I have close to 50,000 pictures, all of which are organized into albums and projects, most of which have keywords, maybe a third of which have edits, and a small set of which are organized into books. So I face a monumental task to migrate, and in the process I lose the “nondestructive” nature of my edits. I’ll have to export an edited version and an original if I think I’ll ever want to re-edit an image. Furthermore, my organizational approach will need to be revisited, and some of the work I’ve put into organizing will vanish. So I procrastinate. Aperture’s still working (for the time being). I’ll wait until I’m forced to do something.

This could turn into a rant supporting RMS’s philosophy of using pure Free / Open Source software. But that’s not really the solution either. I could just stop updating my Mac’s software, and I’d be able to continue to run the application as-is. I’d probably want to disconnect it from the Internet, because a lot of software fixes are security-related. It would be inconvenient, but it’d work. Until I had a hardware failure of some kind. These issues are not exclusive to non-Free software. Free software changes and evolves too. I have a backup server dependent on Fedora Core 6 for one of the drivers. It works, but if I want to do any security patches, I’m on my own. With Free software, I’m guaranteed to be able to maintain a working system, but I still have to be willing to do all the work. There’s no panacea.

And on the pedestal these words appear:
`My name is Ozymandias, King of Kings:
Look on my works, ye mighty, and despair!’
Nothing beside remains.
– Shelley

1 I’m happy to report that today there seems to be new life in Intaglio. There are posts in their support forum, and new versions have been released.
2 Now, it looks like MediaPro’s been spun off from Microsoft again.

Page 1 of 612345...Last »