So, it was another plugin-gets-updated-and-the-site-crashes situation. It’s not exactly the fault of the plug-in. It’s WordPress being stupid about security.

As I wrote back in 2019, I have WordPress automatically updating itself and its plugins using a cron job that uses the magical Word Press CLI. Notably, this update process runs as a different user than the web server. This is by design. I want to minimize the number of directories where the web server has write permissions — especially, I don’t want it being able to write in the directories containing code. This is kind of basic stuff. If someone can abuse a bug in the core or a plugin to write a file in the web tree, they could do all sorts of mischief even without escalating privileges. Denying the web server write access to those areas is a simple mitigation that prevents a whole class of attacks.

WordPress, however, was written with the belief that it should be able to write files wherever it damn well pleases. The idea is a naïve user gives WordPress full write access on their server, or their FTP credentials to their host, or their ssh username and password [!!], and then a lot of functionality is simplified. Once the web server has privileges to write everywhere, it’s easy to give the user the ability to install, update, edit, and remove plugins and themes directly from the web interface. Very convenient! Especially if you don’t have trust issues like I do.

Now, because of the way content is uploaded and plugins work, there are always going to need to be directories where WordPress has write access. That’s fine. I can protect some of those from being a problem by setting directives in the web server to prevent code execution.

There’s a lot of infrastructure to support WordPress’ profligate write permissions. One component of this is an internal function WP_Filesystem that creates a global abstraction of the filesystem. Once that function is called, plugins or themes or whatever can call methods on the global $wp_filesystem variable to interact with the filesystem, while behind the scenes these interactions could be directly, over FTP, over ssh, or other protocols, depending on how the system is set up. Instead of calling file_put_contents(...), for example, the plugin author calls $wp_filesystem->put_contents(...), and doesn’t have to worry about the details of which protocol is used.

The WP_Filesystem function works by calling the function get_filesystem_method in wp-admin/includes/file.php, which tests different ways of writing files. Here’s where I got screwed. To see if it can write directly to the filesystem, this method tries to write a temporary file, and if this succeeds, checks the ownership of the created file. It compares that to what it considers the WordPress file owner, which is determined by looking at the ownership of wp-admin/includes/file.php. If that fails, it moves on to the next protocol.

So in my case, get_filesystem_method didn’t think it could access the filesystem directly, because wp-admin/includes/file.php was not owned by the web server user. So it moved on to try to update via FTP, ssh, etc, and all failed. It then gracefully threw an error that took down the whole site.

Now the question is why this plugin update needed write permissions anyway? The files making up the plugin were installed successfully by my upgrade script. It turns out that the plugin had a new stylesheet in an scss file, and on the first run it was trying to compile it. I’ll grant that that’s a reasonable case. But the directory where it wanted to put that compiled css was writable! It just never got to that point, because of the abstraction layer.

The slightly ridiculous solution to this problem was to change the ownership of
wp-admin/includes/file.php to the web server user, load the main page of the web site to let it generate the css, and then change the permissions on that file back. Stupid, stupid, stupid.

A long PDF was to be printed, but only the text was important. As it was full of images, it seemed like removing the images would save a whole lot of ink.

It turns out ghostscript has some very nice filters for removing classes of content from a file. You can very simply remove text, images, or vector objects without changing the rest of the layout.

For example, to strip vector and images from a PDF, you can use:

gs -o text-only.pdf -sDEVICE=pdfwrite -dFILTERVECTOR -dFILTERIMAGE pdf-with-pictures.pdf

If you don’t have ghostscript installed but use Docker, there are containers that make it easy:

docker run --rm -v pwd:/app -w /app minidocks/ghostscript gs -o text-only.pdf -sDEVICE=pdfwrite -dFILTERVECTOR -dFILTERIMAGE pdf-with-pictures.pdf

This is a WordPress site, first created in 2003. Why I created a WordPress blog site when I was involved in the CMS Made Simple project back then, I don’t recall.

I maintain a handful of other sites running on WordPress too. It was a lot of work until I discovered the wp command-line tool (thanks Beau!). I was able to lock down permissions of the WordPress directories, and have a nightly script that would update WordPress and all of the plugins to make sure I was up to date and (moderately) secure.

One of these WordPress sites I maintain is for an organization. The site has gone through a lot of evolution. I originally built it with a Perl-based system called “Scribe” that I wrote in 2002 or thereabouts. It used a markup-based templating system, CVS to maintain versions, and published static HTML from CGI scripts. Later, I was working on the CMS Made Simple project as part of the core team, so I ported this site to CMSMS. Eventually, I left the CMSMS project, and in 2017 I decided to migrate the site to WordPress (based on the fantasy that other people would join me and maintain content on the site). I was busy, and paid an overseas development firm to port the site for $300. The language barrier made the project difficult, and I ended up rewriting a substantial portion of the code.

This morning a security update resulted in one of the site’s plugins throwing fatal errors and taking the site completely offline. I eventually had to manually poke around in the database and modify JSON strings to resolve the problem (and disable the broken plugin).

I started reflecting on the overall functionality of the site and how it’s used. Maybe it’s time for me to move away from WordPress and go back to some static generating strategy. I hear good things about Jekyll. Of course, this would also probably involve a site redesign too. Ugh. Well, more to evaluate and think about.

Everyone loves to dump on Javascript because of stupid things like [] == ![]; // -> true. Some languages are much better at this. But PHP is not among them. I found the following not quite what I would have expected:

boolval("1");   // returns true
boolval("0");   // returns false
boolval(1);     // returns true
boolval(0);     // returns false
boolval("00");  // returns true
boolval("0.0"); // returns true
boolval(0.0);   // returns false

I had a database corruption event happen in a Docker container, one result of which was that the container was completely wedged and unusable. I spent a lot of time removing containers and images and recreating them, only to have the same problem. In my case, the symptom was I’d create a new MySQL database, and operation would be fine, until part way through a bulk data import or heavy query, and the database would crash and the container would get into an unusable state again. Docker didn’t yield anything in the log, but MySQL showed corruption:

2024-11-27T22:35:38.182396Z 8 [ERROR] [MY-013183] [InnoDB] Assertion failure: os0file.cc:2899:dir_fd != -1 thread 140305764447808
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
2024-11-27T22:35:38Z UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
BuildID[sha1]=511ab47ad6d5dfd73d58eea5860b1c9c9a97ce07
Thread pointer: 0x7f9ad4000fc0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f9b7b3febc0 thread_stack 0x100000
/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x41) [0x15af021]
/...

Once this problem occurred in a container, the database wouldn’t be start-able, often complaining about being unable to update permissions on /var/lib/mysql/mysql.sock.

Eventually, with much futile searching, I found this posting and switched Docker Desktop away from using VirtioFS to using gRPC FUSE for sharing the filesystem between the container and the host machine. This seems to have resolved the issue. I can load big databases again, and heavy transactions work in my development environment again.