Gotcha of the Day: WordPress 4.6 Crashes with a Segfault when Serving HTTPs Content

See Also: Gotcha of the Day: More WordPress 4.6 Segfault Challenges

Earlier this morning, one of my clients' sites started spitting out Secure Connection Failed messages when attempting to access https pages within WordPress:

PHP files served via https worked fine, and WordPress files served via http worked fine.

Looking in Apache's error log I saw a stream of generic segfault message:

[Wed Aug 17 12:06:19 2016] [notice] child pid 9015 exit signal Segmentation fault (11)
[Wed Aug 17 12:06:22 2016] [notice] child pid 9035 exit signal Segmentation fault (11)
[Wed Aug 17 12:06:25 2016] [notice] child pid 9034 exit signal Segmentation fault (11)
[Wed Aug 17 12:06:33 2016] [notice] child pid 8883 exit signal Segmentation fault (11)
[Wed Aug 17 12:06:42 2016] [notice] child pid 9031 exit signal Segmentation fault (11)
[Wed Aug 17 12:06:52 2016] [notice] child pid 9037 exit signal Segmentation fault (11)
[Wed Aug 17 12:07:04 2016] [notice] child pid 8976 exit signal Segmentation fault (11)
[Wed Aug 17 12:07:11 2016] [notice] child pid 9033 exit signal Segmentation fault (11)
[Wed Aug 17 12:07:19 2016] [notice] child pid 9036 exit signal Segmentation fault (11)

Ugh. That's not good. Or helpful.

In response to the ticket I opened with the hosting company, the support tech suggested I follow the instructions here which outline a method for analyzing the core dump generated during a segfault.

It's been years since I busted out gdb and tried to make sense of a core dump, but I didn't have much choice.

I tried to follow the instructions in the article, but most of steps didn't work. I wasn't able to install the debug symbols, and my attempt to have the core dump written to a special directory failed. But still, setting ulimit -c unlimited was key, and once that was done, Apache was glad to drop 200+ Megabyte core files in my webroot.

Analyzing the core dump was more fruitful than I'd imagined. Here's a relevant snippet:

Core was generated by `/usr/sbin/httpd'.
Program terminated with signal 11, Segmentation fault.
#0  0xb77d0a0a in ?? () from /lib/libpcre.so.0
Missing separate debuginfos, use: debuginfo-install httpd-2.2.15-54.el6.centos.i686
(gdb) bt
#0  0xb77d0a0a in ?? () from /lib/libpcre.so.0
#1  0xb77d33d4 in ?? () from /lib/libpcre.so.0
#2  0xb77dafb4 in pcre_exec () from /lib/libpcre.so.0
#3  0xb6c658ed in php_pcre_split_impl () from /etc/httpd/modules/libphp5.so
#4  0xb6c65ed7 in ?? () from /etc/httpd/modules/libphp5.so
#5  0xb6e18659 in ?? () from /etc/httpd/modules/libphp5.so
#6  0xb6deee7e in execute () from /etc/httpd/modules/libphp5.so
#7  0xb6dbd0fb in zend_call_function () from /etc/httpd/modules/libphp5.so
#8  0xb6d048c6 in ?? () from /etc/httpd/modules/libphp5.so
#9  0xb6e18659 in ?? () from /etc/httpd/modules/libphp5.so
#10 0xb6deee7e in execute () from /etc/httpd/modules/libphp5.so
#11 0xb6dbd0fb in zend_call_function () from /etc/httpd/modules/libphp5.so
#12 0xb6d048c6 in ?? () from /etc/httpd/modules/libphp5.so
#13 0xb6e18659 in ?? () from /etc/httpd/modules/libphp5.so
#14 0xb6deee7e in execute () from /etc/httpd/modules/libphp5.so
#15 0xb6dbd0fb in zend_call_function () from /etc/httpd/modules/libphp5.so
#16 0xb6d048c6 in ?? () from /etc/httpd/modules/libphp5.so
#17 0xb6e18659 in ?? () from /etc/httpd/modules/libphp5.so
#18 0xb6deee7e in execute () from /etc/httpd/modules/libphp5.so
#19 0xb6dbd0fb in zend_call_function () from /etc/httpd/modules/libphp5.so

What I see there is an infinite loop within PHP code. The system is probably crashing due to running out of stack space. As noted above, this is a WordPress and https problem. Given that there's no mention of SSL in the above function calls, this lead me to believe the problem was strictly within PHP.

Then it hit me: earlier in this morning we'd upgraded the site to WordPress 4.6. Could that be the cause? I grabbed WordPress 4.5.3 and ran some diffs. Whoa, there was way too many changes to detect what might have happened in 4.6 to cause this.

I then decided to try rolling back WordPress. I followed the instructions here, which turned out to be surprisingly simple. They boiled down to:

1. Shutting down the web server
2. Moving the existing 4.6 wp-includes, wp-admin away
3. Copying 4.5.3 version of wp-includes and wp-admin into the web root
4. Copying and overwriting top level 4.5.3 *.php files over those that came with 4.6
5. Started up the web server
6. Logged into wp-admin
7. WordPress detected the version change and told me that it would need to "upgrade" the database. I clicked the button to proceed
8. The system was happy again

I have to say, I'm quite impressed that WordPress can so smoothy "upgrade" to a past version.

Most importantly, WordPress + https worked again!

I've got to say, I'm totally floored by this. It appears as though a WordPress update triggered a low level PHP bug which triggered a massive stack overflow and core dump. What are the odds?

A quick check of Twitter showed that nobody else was complaining of this issue, so my guess is that it's a weird mix of the version of PHP and Apache and such that's causing this. More than anything, this is a reminder that under the hood there's serious complexity (WordPress depends on PHP, which depends on Apache, which depends on the OS, which depends on SSL networking code, and so on) and yet most of the time, things Just Work. Remarkable.

See Also: Gotcha of the Day: More WordPress 4.6 Segfault Challenges