TeaBreak Server Blues…

TeaBreak.PK website and server is the busiest I have ever managed after the FAST-NU Karachi one. Ayaz, in both cases has been the go-to-guy and the server Admin. He is one Server / Linux guru and is an expert in what he does.

Of late, we had been facing a lot of traffic on our end. We initially started with a shared hosting package which we immediately had to drop with-in a month of TeaBreak’s launch. We had to upgrade – so we gave TeaBreak it’s required room on a CentOS box. It is a matter of time, and now after two months we are still struggling to find an optimal match between the resources and the demand (the traffic).

Our server is an innocent one where our resources are not at leisure. TeaBreak consistently faces a traffic that averages to 12,000 hits a day and about 4500 unique visits every day. That doesn’t include the crawling engines from where we get far more hits. It sums up to roughly 2GB of traffic per day.

There are times when we have about 70-80 concurrent users demanding their request to be furnished in addition to our routine engines running behind. Each page on TeaBreak queries the database heavily and utilizes some resources in parsing the dynamic page. Our resources were being exhausted more often then one would expect!

1-week traffic summary

24-hour traffic summary

Statistics tool: WordPress WassUp

To eradicate the problem we introduced caching engines (xCache, memcached and a customized WordPress WP-Cache). The result was an immediate 60% reduction in resource consumption however the IO rate shooted up. Nevertheless, we could now shell in without facing a lag of 10 seconds between each request/response!! Yayy.

It’s not the traffic that is the cause of Issue. Our cron scripts and engines that are the heart of TeaBreak consume a lot of resources whenthey run. The engines were initially coded to run continuously andconsistently 24/7. However we had to break the processing and had toput in a 10-minutes interval. The engines perform a lot of dataprocessing, tagging, optimization on the fetched data. Plus, the MySQLdatabase is also growing at a brisk pace. If it continues then with-inthe next 6 to 9 months we might need to cluster / multiply it for thesake of speed.

CPU Usage (this is what we are focusing to downsize)

Network consumption (Still a lot of room before it gets saturated)

Statistics tool: System Real-Time Graph (MRTG)

Anyhow, after the caching mechanisms were introduced, still the performance was not satisfactory and it still is not. We are experimenting with Apache and MySQL performance tuning. We do plan on launching our own Load-balancing equation by introducing Lighttpd and a Squid reverse proxy. Now don’t get your hopes up or be excited — like I said, we are “experimenting”! There’s also a possibility of cloud / grid-based hosting.

As of now (and since some days) TeaBreak has faced some outages due to server resource exhaustion. We have done some immediate LAMP tuning however the way I look at it – yes we will be needing some sort of very primitive and basic load-balancing mechanism — very soon!

Having said all that, we have so many more innovative ideas yet to be launched! How are we going to fit them into our current stringent resources — that, my friend, is another challenge!

Yes, we do look for hosting / optimization suggestions and tips. If you got, please share!