I was private-messaged this morning. I’ve redacted the nick-name of the person contacting me, and made a few minor typo fixes. Unfortunately, this is not all that unusual…

[09:56] [redacted]: [link to bug report]
[09:57] [redacted]: I need these features.. 🙂
[09:57] _SjG_: hi
[09:57] [redacted]: very good module but not working.. 🙁
[09:58] _SjG_: it’s been a long time since I’ve done any work on the module
[09:58] [redacted]: any other guys?
[09:58] _SjG_: well, I do intend to do some upgrades on it, but paying work keeps getting higher priority
[09:59] [redacted]: i see
[09:59] [redacted]: it is method to get money? 🙂 leave small bugs and do works for money?
[10:00] [redacted]: i see this not first time
[10:00] _SjG_: no, it’s just that I only have so many hours in a day
[10:01] [redacted]: i am not asking you to do this
[10:01] [redacted]: just asking wht to do?
[10:01] [redacted]: its not working..
[10:01] _SjG_: I understand. It’s on my list of things to fix
[10:02] [redacted]: any other solution? other developer who can finish this small thing?
[10:02] _SjG_: you can ask
[10:03] _SjG_: I don’t know if anyone’s working on it
[10:03] _SjG_: Maybe [redacted] — he was doing some FormBuilder work
[10:05] [redacted]: strange..
[10:06] [redacted]: i had so much doubt about choosing this cms..
[10:06] [redacted]: now i see
[10:07] _SjG_: if you’re trying to make me feel guilty for not immediately fixing the problem, you’re not going to succeed. I do this as a volunteer. I have a full-time job in addition.
[10:09] _SjG_: You can try Joomla, and see if they’re more responsive.
[10:09] [redacted]: so other have too 🙂 but they not publishing modules with bugs I spent so much time and now i see that i should do everything with other cms
[10:10] _SjG_: You can always fix it yourself, and be a contributor
[10:10] [redacted]: i am not programmer thanks god 🙂
[10:12] _SjG_: no, you’re just a person with an unreasonable expectation of everybody doing your work for you. Sorry. Try being a contributor (in any way whatsoever), and I’d be more sympathetic.
[10:13] [redacted]: what?
[10:13] _SjG_: but coming in with a sense of entitlement and insults isn’t going to win you much support.
[10:13] [redacted]: i am just person who choose product like idiot
[10:14] [redacted]: i made a research it fits my need or not
[10:14] [redacted]: and that damn module is buged
[10:14] [redacted]: thats it
[10:14] _SjG_: yes, and it’s on the list of things to be fixed. what’s your point?
[10:14] _SjG_: I’m not dropping everything to make you happy?
[10:14] [redacted]: and developer say that he need money for this or i can fuck off
[10:14] _SjG_: That’s actually not what I said
[10:15] [redacted]: ha ha 🙂
[10:15] _SjG_: I said that it’s on my list of things to fix, but paid projects have taken priority
[10:15] _SjG_: I have bills to pay, you know
[10:15] [redacted]: you think other is too stupid?
[10:15] [redacted]: ok
[10:15] [redacted]: now i see situation
[10:16] _SjG_: I have a list of projects, each of which I go through and fix bugs as I have the time
[10:16] [redacted]: i will do some work in forums and so on to clear the situation for other people who can be fished like this
[10:16] _SjG_: I also get hundreds of feature requests
[10:17] _SjG_: and everyone is just like you, thinking that their project more important than my life, so I should just do their work for them and thank them
[10:17] [redacted]: so i am guilty to choose not working module ? 🙂
[10:17] _SjG_: no, you’re guilty of being impatient
[10:18] _SjG_: and insulting me when I try to explain the situation
[10:18] [redacted]: 🙂 ok
[10:18] _SjG_: accusing me of extortion by intentionally leaving in bugs
[10:18] _SjG_: do you know how many hour a week I put into writing this stuff?
[10:18] [redacted]: how much do you want for fixing this?
[10:19] _SjG_: it’s on the list to be fixed. I have several paying jobs I’m working on now. It will be done after I complete them.
[10:20] [redacted]: i talk with you just 10 min, but i understand that it will be fixed in 5 years 🙂
[10:20] _SjG_: well, there’s just one of me
[10:21] _SjG_: I’d work faster, but I do occasionally like to sleep, talk to my wife, etc
[10:22] [redacted]: maybe do the work till the end? 🙂
[10:22] [redacted]: and only then post them?
[10:22] [redacted]: or dont think that other stupid? 🙂
[10:23] [redacted]: ok i will find some person and pay the money
[10:23] [redacted]: but not for you mtf
[10:24] _SjG_: Dude, with your attitude, I would have charged you the “go to hell” price anyway

I regularly end up in the situation where I have to fix a crap PHP application.

The latest one has lots and lots of PHP 3.x-era code, that references hashes without quoting the index, e.g.,


$foo = $bar[baz]


Now, the PHP interpreter understands this in actuality. It figures out that since the constant is not defined, the programmer probably meant that the index should be ‘baz’. It does, however, throw a well-deserved warning.

The code I’m trying to fix throws lots and lots of warnings. Rather than wade through all the warnings to find which ones are important, I started with the following:


find . -name \*.php -exec grep -ne '\[[^\$0-9\'\''\"]' {} \;


The thousands of lines of output convinced me bigger guns were needed.

So we got ugly.

find . -name \*.php -exec perl -p -i.bak -e 's/\[([^\$\d\'\''\"]+)\]/\[\'\''$1\'\''\]/g' {} \;


Note that that’s one line, and that WordPress seems to want to change some single quotes into back-ticks. Don’t be fooled!

All those extra backslashes and single quotes are to allow passing single-quotes within the regex, and not have bash consider them problematic.

Also note that this could be catastrophic if you have regular expressions in the code you’re operating on — do a diff with the backup version, and merge back the regexes.

I’m sure there are far more elegant solutions… primary among them, not using crap PHP apps in the first place!

So it’s a familiar problem, where you’re developing a data-driven application, and you want to optimize the queries that will run against your database (I’ll have more interesting stuff on this later). The problem, of course, is that to really optimize those queries, you need a lot of sample data.

So I needed to do some address lookup code against a huge collection of users. But because there was the possibility of having to demo the prototype, I really didn’t want 100,000 users named “Foo McBar” living at “10101 Binary Place.” So, with the help of the almighty Internet, the all-frobnicating Perl, and the all-knowing US Bureau of the Census, I created a quick, semi-flexible script to generate people with plausible names and addresses that, if not Google-mappable, at least had agreement on city/state/zip. The city/state/zip is a collection of 250 random zip codes. If you have good zip code data, you can easily extend this to be complete! Names are generated from the most popular forenames and surnames, with a probabilistic bias towards the most common ones. The script also allows you to specify “pick one of n item” type fields, pick a number from a range, plausible email addresses, not-very-plausible phone numbers with or without extensions, and the ability to export as CSV or tab-delimited.

In principle, this should be easy to adapt to other countries, although you’ll need lists of common first names, surnames, street names, and a way of mapping cities to regions, states, districts, cantons, or whatever’s appropriate.

You can grab a copy of it here. It requires a Perl interpreter with the Text::CSV and Getopt::Long CPAN modules.

Usage: user-data-maker.pl [OPTIONS]
   -t, --header : header, a colon-delimited list of column headers
   -f, --format : format string, a colon-delimited list of column contents
       data types:
         fn - first name
         ln - last name
         a1 - street address
         a2 - apartment number
         c - city*
         s - state*
         z - zip 5*
         e - email address
         pne - phone (US), no extension
         pwe - phone (US), with extension
         [a,b,c] - one of a, b, or c
         {a,b,c} - one of a, b, or c in decreasing probability
         [x-y] - a number between x and y, inclusive

         * city, state, and zip will be agree to create a valid address
           if you need multiple addresses, use the code ! to reset the
           synch. The reset works on a left-to-right scan of the format string.

   -n, --number : number of records to create

   Flags:
  -c, --csv : output CSV format (otherwise, tab-delimited).
  -v, --(no)verbose : verbose mode (default false)

Example:


Viajante:samuelg$ user-data-maker.pl --header "First:Last:Age:Email" --format "fn:ln:[10-100]:e" -n 5 --c
First,Last,Age,Email
Margot,Sawyer,33,Margot.Sawyer@netscape.com
Francisco,Cantrell,18,Cantrell@sbcglobal.com
Lynetta,Orozco,28,Lynetta@mac.com
Latrice,Dunlap,41,Latrice.Dunlap@sbcglobal.com
Anissa,Fitzgerald,59,Anissa@hotmail.com


or, more exotically:


Viajante:samuelg$ user-data-maker.pl --header "First Name:Last Name:Address:City:State:Zip:Super Power" --format "fn:ln:a1:c:s:z:[Invisibility,Invincibility,X-Ray Vision,Flight,Likes Squirrels]" -n 5 -c
"First Name","Last Name",Address,City,State,Zip,"Super Power"
Roseanna,Best,"8821 7th Str.",Manati,PR,00674,Flight
Euna,Crawford,"8195 Lee Str.","Fort Washington",PA,19034,Invincibility
Ted,Williams,"7140 Birch Ave.",Monroe,CT,06468,Invincibility
Mariano,Miranda,"2657 1st Way",Lyford,TX,78569,Flight
Tammy,Flowers,"2135 Washington Blvd.",Duluth,MN,55806,"Likes Squirrels"


Enjoy!

Well,not really. This is just one example of a bad approach.

The problem: an HTML file is read, but needs to be entity-escaped. However, not all entities need escaping. Specifically, double quotes with anchor tags need to be left alone.

The right solution: process the HTML via a DOM parser, escape nodes that are not anchor tags. Oh, but did I mention these HTML files may be crappy, non-validating files, or even snippets?

The next solution: Use a regular expression. Yes, this is ugly. Yes, it also works 🙂

Originally, I tried using variable-length lookahead, but ran into problems (PHP 4.x). But PHP provides another solution which is perfect for this sort of thing. Here’s the code:

function pre_esc_quotes($inner)
{
return preg_replace('/"/','QUOTE',$inner[0]);
}
function post_esc_quotes($inner)
{
return preg_replace('/QUOTE/','"',$inner[0]);
}
$tmp=preg_replace_callback('/<a([^>]*?)>/s','pre_esc_quotes',$raw_html);
$tmp = html_entities($tmp);
echo preg_replace_callback(('/</a><a([^>]*?)>/s','post_esc_quotes',$tmp);


This, of course, presumes that the string “QUOTE” won’t show up anywhere in your raw html. Consider replacing it with an opaque string (like “JHG54JHGH76699597569” or something creative and long that will choke the interpreter).

This code is furthermore inefficient in a number of ways. It’s not something you should use. But it does show how preg_replace_callback avoids some scary regex work.

The need arose to have a specific email address round-robin (e.g., cycle through a collection of destination email addresses).

A solution was achieved through use of procmail and a little perl script. It probably could be done more easily and/or better, but I figured other people might find this interesting.

So, first, an alias was created in /etc/aliases (used by postfix in this case, but it should work for sendmail, and variants should work for other MTAs):

rrtest:         |"/usr/bin/procmail -m /etc/postfix/roundrobin_procmail.rc"

Then, the following file was saved as /etc/postfix/roundrobin_procmail.rc:

:0 w:/tmp/rrlock
{
        :0
                dest=|/etc/postfix/rr.pl
        :0
                ! ${dest}
}

And then, of course, we need the perl program. Here’s /etc/postfix/rr.pl:

#!/bin/perl
# ----------------------------------------------------------
@recipients = (
'address1@sample.com',
'address2@sample.com',
'address3@sample.com'
);

$index_file = 'rr-index.txt';

# ----------------------------------------------------------

$index_exists = 1;
open(IN,";
        close(IN);
        $index++;
        }
else
        {
        $index = 0;
        }


if ($index > $#recipients)
        {
        $index = 0;
        }
open(OUT,">/tmp/${index_file}");
print OUT "$index\n";
close(OUT);

print STDOUT $recipients[$index];

exit 0;

Elegant? Not really. But it seems to work 🙂