Before anything, I would like to thank
@SharedGrid for the colossal investigative effort. All of what I am about to post is the result of Daniel's work, and nearly all of the words come from his messages to me.
After all of the work done to date, we started by looking at WHM, Apache, MySQL and PHP configuration of my VPS, particularly focusing on the performance of the machine itself. We immediately noticed a pretty high CPU usage, at times reaching 100%, most likely during a busier browsing activity. Daniel associated it with a couple of reasons:
1) the CPU clock speed is low at around ~2.1Ghz, WordPress is very much a single threaded bit of software so it’s taking longer for processes to complete their task and exit the CPU. A VPS with less cores but a higher clock speed (Ghz) is more beneficial than more cores and less speed. 2) Apache, MySQL, PHP has no performance optimization which can contribute to high CPU usage.
The VPS is running a default version of MySQL 5.7, which went end of life in October 2023. While initial recommendation was to switch to MariaDB, for now, we just performed optimization.
The Apache configuration was default as cPanel ships it, which is very conservative out of the box. We performed optimization on ServerLimits and RequestWorkers.
We also performed OS and Kernel optimizations for better memory and process handling.
We switched php-cgi to php-fpm, as it's much superior in the way it handles PHP requests.
We updated php.ini settings to better suit WordPress.
These 5 improvements led to a dramatic decrease in server load, so the changes have definitely made a difference. The new averages at the moment were: 1.40, 1.08, 1.14 - whereas they were near between 3-4 before, 4 being the limit.
Query Monitor, installed to see how the site was performing in terms of queries to the database, was now showing that queries were completing in less than 0.1s which is excellent, but the page generation time was still varying between 5-7 seconds. There was also the issue of the other 2 columns taking longer to load in compared to the first.
To be continued.