I manage a very busy Rocky Linux test-server. For one staging environment, cron is already running five or six maintenance scripts every minute. But when the tests run, the system has to do a lot of additional permissions fixes and filesystem adjustments. I’ve started seeing in the logs the following error message:

pam_systemd(sudo:session): Failed to create session: Maximum number of sessions (8192) reached, refusing further sessions

Now, there is a known older problem with systemd and dbus, that comes up when you search for this error message. I couldn’t find any concrete actions I could take to fix the issue. The other major search results are RedHat pages behind their subscription wall, and, at this point I’m apparently too dumb and out of date to even be able to figure out how to pay for a RedHat subscription.

I think I’ve found at least a temporary solution, however. In /etc/systemd/logind.conf there is the SessionsMax field where you can override the default. I doubled it to 16384, then ran systemctl restart systemd-logind

I’ll have to see if that’s a viable long-term fix rather than just treating the symptoms of a bigger issue.

I recently came across this site for creating warning signs of various sorts. It was intended for use by those who operate laboratories and research facilities. It was pointed out to me because it has a random sign generation function, which creates some funny and absurd signs, perfect for your surrealist academies.

My mind, however, went to Shakespeare.

*This is actually on Rocky Linux / CENTOS / RHEL, but will likely work on others.

Logwatch can be a nice tool for keeping an eye on your servers. It goes through your logs and creates a nightly aggregate email containing information to keep you apprised of various important details. It can be good to bring things to the attention of lazy / overwhelmed sysadmins like me.

Where it fails, though, is where it overwhelms you with useless information. There are different output level settings, and if you turn the detail levels down far enough, it helps a lot. However, with certain configs and certain OSes, you still get overwhelmed with non-actionable information. Here’s how to fix a few of those.

Crontabs. In Rocky Linux, cron logs a success message that contains a process number, which means the default log is filled with lots and lots of lines like session-685197.scope: Succeeded.: 1 Time(s) which logwatch happily throws into the nightly email. Most searches tell you to edit your /etc/logwatch/conf/ignore.conf file and add the following line:

session-.*scope: Succeeded

This doesn’t work for me. Further research indicates that the ignore.conf file wants a Perl-style regular expression. The recommendation above is sort-of-Perlish, but what ended up working correctly for me was putting the following line in my ignore.conf:

\s*session-(.*?)\.scope: Succeeded\.(.*)

HTTP. For some reason, someone thought having a long list of hostile IP addresses would be helpful. Maybe to manually block them? Seems like a hopeless task. Check out /usr/share/logwatch/scripts/services/http around line 596… and un-comment out the conditional.

$flag = 1;
foreach my $i (sort keys %ban_ip) {
   if ($flag) {
      print "\nA total of ".scalar(keys %ban_ip)." sites probed the server \n";
      $flag = 0;
   }
   #if ($detail > 4) {
      print "   $i\n";
   #}
} 

sshd. I know there are a lot of hackers, script kiddies, and bots out there. I don’t need to see the long list of people who tried and failed to log in with ssh. Unfortunately, the detail level setting for sshd aren’t very helpful. I ended up editing /usr/share/logwatch/scripts/services/sshd and liberally sprinkling my own if ($Detail > 4) { … } barriers starting around line 500. Hacky, I know. Also will be clobbered with the next logwatch update. Yuck.

Maybe it’s time for me to submit a bunch of pull requests.

Went to update an image gallery with some new pictures. I wrote the gallery generating code four years ago, but the version of PHP on the server has been updated and one of the libraries I use evidently relies on deprecated syntax ($string{$char_index} for the curious, which now would be $string[$char_index]). So I tried to check out the repo from GitHub, only to find they’ve updated their SSH host keys, so I need to fix that before I can fix the code problem. So I tried to update that, but had packed my Yubikey off in a drawer in the office…

Back in the 80s, I worked in Aerospace and had an old engineer tell me “nobody’s really figured out how to write software.” I scoffed. Totally ridiculous, I thought. He was researching a transition to Ada. I was writing tons of C code.

Now, much older, and perhaps somewhat wiser, I see his perspective. We’ve developed a lot of ways to try to deal with software. There are different approaches, with principles of abstraction, structure, and documentation. I tend to believe that most software methodologies are too discrete. Once you’ve maintained an active codebase for a business over a few decades, you discover that software design principles are great for coding but bad for business. It’s very difficult for a coder to understand which aspects of a business are virtually immutable (“the canonical catalog is Bob’s PDF notes from his meetings with manufacturing”) versus things that change (“yeah, we computed sales bonuses this way for the past twenty years, but now we’re changing that on a quarterly basis”). Often, the people running businesses don’t consciously know these unwritten laws themselves, and only reveal them when a change is suggested. Heaven help you if the change was implemented somewhere first.

Methodologies for business and software come and go, whether they’re TQM or Agile or just “move fast and break things.” I definitely see the advantages of the tendency away from front-loaded planning and towards flexibility and accepting change, but it doesn’t do a good job of managing the complexity of history. How do you maintain and document all of the Chesterton’s Fences in a continuously evolving codebase? When new components are bolted on to older systems, you accumulate a lot of weird adapters to make things fit. Sometimes, intermediary components go away, and you’re left with layers of adapters between things. Eventually, the solution might be the same as it would be for an analogous plumbing problem: rip it out, and start over — but, to overextend a metaphor, when the system is a million lines of code instead of a few thousand meters of piping, the solution may end up replacing the entire building so you can install a faucet.