This article has all the optimization techniques , Performance tuning tips and guidelines that are required for any larger servers database, big ecommerce stores, Blogs, CMS Systems etc.
When getting ready to deploy and configure the Big shopping carts like Magento , there are some general technical environment components that must be taken into consideration to create an optimized setup. These range from physical hardware selection and network throughput, to the underlying Open Source stack, which are key underpinnings which help drive own configuration components.
Some of the Settings that we applied for some of the carts and blogs and achieved a great performance boost.
This Article is based on Magento Performance White paper :http://www.magentocommerce.com/whitepaper Thanks to Magento Team for publishing this wonderful article
When trying to achieve the optimal setup, please take the following into consideration:
With a large amount of concurrent users the sufficient amount of RAM is highly critical to handle all incoming connections. Faster, modern systems with multi-core CPUs, high front side bus speeds and fast hard drives, preferably at 7200RPM and above, will generally speed up the entire application.
Insufficient network I/O throughput and latencies in the internal network can significantly impact performance of a multi-server setup. Outbound connection latency may hurt the customers browsing the store frontend.
Since Magento is a PHP application that runs on the LAMP stack. Therefore, current, up-to-date and well-configured versions of the Linux Kernel, Apache, MySQL and PHP will provide better performance results. Proper configuration of the web-server, the database server and PHP itself is required in order to achieve optimal performance results. While Upgrading MySQL you need to check the big Bug list for your selected version, so only prefer Stable versions. If that is a Beta version then these may be cases that your server and database will crash.
Magento Configuration Components:
• Proper cache backend
• Handling sessions with fast storage
• Directory structure optimization
• Flat frontend catalog
• Magento Enterprise Edition cron scripts
• Rebuilding indexes
• Admin panel separation
• Proper search type selection
• Frontend layout complexity
• Number of HTTP requests on the page
• Using parallel connections
Proper Database Configuration
Proper MySQL configuration is one of the most important aspects of configuring any application. You will be wondering to know that Optimizing the MySQL configuration can provide up to a 70% performance. An incorrect configuration may result in the web-server spending more time in idle loops waiting to retrieve data from the database.
As an additional note, the default MySQL installation, even in later versions, is configured to use far less resources than average hardware can provide.
Let’s quickly go through the most important directives in the MySQL configuration file, my.cnf, and their recommended values.
The formulas that can be used for all the calculations can be found in the template file bundled within a MySQL distribution package like (my-huge.cnf, my-innodb-heavy-4G.cnf, my-large.cnf, my-medium.cnf and my-small.cnf
MySQL is improving its performance from its newer versions like 5.1 is 15 times Performance compare to 5.0.
So if you are using a MySQL 5.0 or 4.0 then you are wasting your Server resources.
• Available Memory
MySQL have Database Engines like InnoDB, MyISAM, so each has its own requirements, Magento uses InnoDB as its primary table storage engine type. InnoDB, unlike MyISAM, can use the in-memory buffer pool to cache table indexes and data. Less disk I/O is needed to get data from hard drives when the value of the in-memory buffer pool is set higher. A general recommendation is to set this parameter up to 80% of the available RAM for a dedicated database server. In cases where both the web-server and the database are running on the same machine, it is recommended to split the entire memory pool into two parts, each having its own primary assigned portion (e.g. on a single server with 6 GB RAM installed it can be split to have 2-2.5 GB used by MySQL, with the rest left for the web server).
The key parameter in this section is innodb_buffer_pool_size, which should be set to use as much available memory as possible:
Server Type innodb_buffer_pool_size
combined web and DB server, 6 GB RAM 2-3 GB
dedicated database server, 6 GB RAM 5 GB
dedicated database server, 12 GB RAM 10 GB
Today’s servers typically have more than 1 CPU installed, with 2 or more cores each. The InnoDB engine can effectively use multiple threads to serve more concurrent connections.
innodb_thread_concurrency should be set to a value equal or greater than 8, even for a single CPU. The recommended value is calculated with the following equation:
2*[Number of Total CPUs]+2
thread_cache_size allows for the caching of a client’s threads when a client disconnects, and to reuse them when new connections are created. The recommended value is from 8 to 64, and depends on your max_connections number.
thread_concurrency can be simply calculated as [number of CPUs] * multiplier . The multiplier value is between 2 and 4 and should be determined by testing the different values and benchmarking for the best results in your environment.
• Built-in Caching
table_cache( deprecated in MySQL 5.1) or table_open_cache( Available in MySQL 5.1 or greater) is the number of tables that can be simultaneously opened by MySQL. A value of 1024 will be sufficient for most, if not all, Magento Enterprise Edition sites.
Having the query cache enabled may result in significant speed improvements when you have a large amount of identical queries, which is the case for any eCommerce application frontend. The recommended values for a Magento database server are query_cache_size 64M and query_cache_limit 2M
A sort buffer is used for optimization of sorting in ORDER BY and GROUP BY queries. 8M is the recommended value for a Magento Enterprise Edition database.
• Slow queries logging
Logging slow queries might be useful for debugging purposes, but it should be disabled in production use.
• InnoDB storage
The InnoDB engine works with a single data storage file, which usually grows in time. It’s a good idea to have its initial state configured to be at least twice as large as the Magento database size, and innodb_autoextend_increment should be set to a fairly high value in order to avoid frequent data file extending operations.
InnoDB supports transactional operations by using transaction log files. Transaction log files are generally configured in groups of 2. The bigger the size of the transaction log file, the less often it performs I/O operations on primary storage files. However more time will be required to restore a database in the event it would be necessary.
Do not use multiple InnoDB table spaces unless you are sure you know the benefits in your particular hardware setup.
Apache Web Server Configuration
The most commonly used Apache configuration provides PHP support with mod_php. This Apache configuration loads a large number of modules. However, most of these modules are not necessary in order to run Magento. This becomes more relevant in a multiserver setup, where different tasks can be split on different nodes and each node has to be configured to perform its specific task the best.
The minimum required list of Apache modules is:
• mod_expires – generates content expiration and cache control headers
• mod_deflate – compresses content before it is delivered to the client
• mod_mime – associates the requested file with its type and behavior
• mod_dir–serves directory index files
• mod_rewrite–is used to support Search Engine Friendly URL’s
• mod_authz_host–is required to limit access to specific files
• mod_authz_user–might be required in a staging environment to setup password authentication, but on a live site it is not necessary
With all unused Apache modules disabled by commenting out the corresponding ‘LoadModule’ lines in httpd.conf, it is possible to cut memory consumed by Apache, which will allow more concurrent connections to be handled with the same amount of RAM.
Another important component is setting an optimal number of running Apache processes. The best method is to create the required number of Apache processes when the web server is started. This number should be calculated by measuring the memory amount consumed by Apache under the maximum load. This is currently the best threading method as mpm_worker cannot be safely used with PHP, and the process of forking every new Apache child in mod_prefork mode is an expensive operation.
Also note that ServerLimit and MaxClients values should be specified explicitly to prevent running out of physical memory and going into a swap file, causing a sever breakdown of webserver performance. MaxRequestsPerChild can be set to default value (4000).
Under a heavy load keeping persistent connections becomes disadvantageous, thus the KeepAlive directive should always be set to off.
mod_deflate allows to compress the content before sending it to the browser. Magento .htaccess file already includes the necessary settings to enable the compression. Please make sure to uncomment this section in order to decrease the page load time.
Additionally, you can take advantage of eliminating directory structure scans for .htaccess files by moving all .htaccess directives into appropriate <Directory> sections of the main httpd.conf file.
In order to reduce the I/O throughput on Apache web-nodes in a multi-server setup, it is advisable to use a load balancer capable of handling all of the logging activity, instead of having the activity handled by Apache backends.
Apache and MySQL Conclusion: An optimized MySQL and Apache configuration shows 55-70% performance increases on dynamic pages (that is most of the pages in the URL’s list). The homepage results are less affected as the default Magento setup has cache enabled by default
and caches make fewer queries to the database on hitting the homepage. Default Apache and
MySQL configuration is not able to handle higher concurrencies (100 concurrent sessions) resulting in the results for that concurrency varying a lot between tests.
PHP is an interpreted scripting language. The process of running a PHP script includes a few steps – reading a script file from the hard drive, parsing and compiling bytecode, and finally running that bytecode.
Realpath cache configuration
Optimization of file I/O is not only limited to using faster hard drives. It is also highly recommended to increase the default realpath_cache_size and realpath_cache_ttl values in php.ini settings. Based on tests the recommended values are realpath_cache_size=32k and realpath_cache_ttl=7200 on production servers.
The process of reading PHP scripts from disk and compiling them can be eliminated by enabling PHP accelerators. PHP accelerators cache compiled bytecode, resulting in less file and system I/O. Well known PHP accelerators eAccelerator and APC are tested and fully compatible with Magento. Their built-in shared memory can also be used as Magento cache storage.
To reduce the memory usage and speed up PHP performance you can enable in php.ini only the minimum set of PHP extensions required to run your application. The necessary extensions are:
• SOAP (if the API is to be used)
Conclusion: Adding a PHP accelerator provides a performance boost from 42% on simple pages (homepage) to 500-600% when different PHP-files are used (URL’s list). APC accelerator
provides good results, but from our tests eAccelerator is 15-20% more efficient.
Caching On Magento
Magento is able to cache frequently-used data utilizing different cache backends.
When installing Magento the filesystem is set to be used as the cache backend by default. Using a cache backend will always improve the performance of Magento, and while the filesystem cache is the most reliable storage with unlimited size, it will not provide the best performance.
Magento can also work with the following cache backends that provide better performance than the filesystem cache backend:
• APC – a bytecode cache for PHP, andalso provides a shared memory storage for application data
• eAccelerator – a PHP accelerator that can also cache dynamic content
• memcached – a distributed, high-performance caching system
Please make sure that if you are using APC, eAccelerator or memcached, you configure them with enough memory to include all cache data, otherwise they may purge required cache hierarchy structures and break the cache.
Conclusion: It may be required to disable the built-in Magento cache (that is enabled after installation by default) during active development, but please make sure that
caching is enabled on production sites as disabled cache makes the store frontend 5-6 times slower and less responsive under load.
Conclusion: The APC cache backend improves the results, which are 2-3 times better than the default filesystem cache backend. The memcached cache backend shows 10-15% better results than APC. And from the tests the eAccelerator cache backend shows the best results which are 5-10% faster than memcached.
Magento uses PHP sessions to store customer session data.
The default method is to use filesystem storage, which works well if you are using a single webserver. Its performance can be improved by configuring a tmpfs in-memory partition to avoid extra hard drive I/O activity.
In a clustered environment with multiple web-servers, the first option for handling sessions is to use a load-balancer capable of associating client requests to specific web-nodes based on client IP or client cookies. If you are in a clustered environment and not using a load-balancer capable of the above, it is necessary to share the session data between all the web-servers. Magento supports two additional session storage types that can be used in this case.
Storing session data in the database (though it is fully supported) is not recommended as it puts an additional load on the main database, and therefore requires a separate DB server to handle multiple connections efficiently under load in most cases.
memcached session storage is free of these disadvantages. memcached service can be run on one of the cluster servers to provide fast session storage for all web-nodes of the cluster. memcached session storage doesn’t show any performance improvements when used in a single-server configuration though, because of extra overhead processing compared to raw filesystem session files.
Conclusion: The default filesystem storage shows the best results on a single-server setup. The memcached session storage shows slightly different results (1-2% worse) and it can be considered an option in a clustered environment with a simple load-balancer setup. The database session storage should be used only in a clustered environment and only if the memcached storage cannot be used for some reason.
Directory Structure Optimization
Optimizing directory structure can also help fine tune Magento Enterprise Edition performance. It is highly recommended to use the Zend Framework distribution bundled within Magento Enterprise Edition as it is tweaked to significantly reduce the number of system calls required to locate a file in the directory structure. This is accomplished by commenting out all extra require_once directives within the configuration file. The additional require_once calls are not required because Magento Enterprise Edition implements its own autoload function that handles all necessary file requests on demand. Recent Magento Enterprise Edition versions (since version 1.3.x) include the Magento Enterprise Edition Compilation Module (Mage_Compiler) which provides extra optimization by placing all the files in one directory and combines the most used classes in a few single files.
Conclusion: Enabling Magento Enterprise Edition Compilation Module provides a 10-15% additional performance boost.
Flat Frontend Catalog
Starting in Magento 1.3.x the Flat Frontend Catalog module was introduced. The Flat Frontend Catalog module maintains an additional set of database tables to store catalog data in a linear format, with extra indexes that facilitate executing database queries on the frontend catalog pages.
Flat Frontend Catalog structures were implemented for both category and product data. Flat Categories are recommended for any Magento installation for improved performance, whereas the Flat Products is designed and recommended for catalogs that have over 1,000 SKU’s.
To enable one or both, first go to your administrator panel and navigate to System -> Cache Management. Under Catalog click on the Rebuild button next to Rebuild Flat Catalog Category or Rebuild Flat Catalog Product. Note: If only Flat Catalog Categories are used there is no need to rebuild the Flat Catalog Product.
Navigate to System -> Configuration, click on Catalog and select the Frontend tab. Choose Yes next to the appropriate selection of either Use Flat Catalog Category or Use Flat Catalog Product.
Conclusion: Enabling the Flat Catalog module does not add much to the homepage test results as the default homepage does not contain product listing information and the categories menu is efficiently cached by default. However, when it comes to browsing the frontend catalog the performance gain is around 2-3% on smaller catalogs and reaches 8-10% on larger catalogs with complex category structures and large numbers of products.
Admin Panel Separation
Admin panel operations are in general more resource consuming than the frontend activities. Often they require increasing PHP memory limits or having extra extensions compiled into PHP. Therefore, having a dedicated admin server can help make admin panel operation faster while not impacting the frontend configuration and performance. The separation can be done by specifying different base URL’s on a global level, and for frontend websites and stores. Each separate domain name can then be served by a separate server.
Proper Search Type Selection
Magento supports 3 search types that can be selected in configuration – LIKE, FULLTEXT, and their combination. FULLTEXT search is known to be faster and puts less load on the database.
Number of HTTP Requests Per Page
<reference> <action method=”addJs”><script>custom_js/gallery.js</script></action> <action method=”addJs”><script>custom_js/intro.js</script></action> </reference>
The example layout file above will combine those 2 scripts in a single file that will be added to the page with one request to js/index.php?c=auto&f=,custom_js/gallery.js,custom_js/intro.js.
Using Parallel Connections
You can find the detailed list of website design best practices at the Yahoo Developer Network at http://developer.yahoo.com/performance/rules.html
Media and Static Content Delivery
Though Apache is a fast and reliable web-server, there are other web-server options that are known to serve static content and media files more efficiently, consuming less memory and CPU time.
Widely used are nginx, lighttpd and tinyhttpd. These are multiplexing web-servers, which don’t have built-in scripting languages support, but can handle thousands of simultaneous connections per server.
Additional Performance Gains
Static content delivery can be improved using a caching reverse proxy, such as Squid, or an HTTP accelerator like Varnish. A reverse proxy can locally cache content received from Apache to reduce the load on the Apache backends.
Another way to reduce your server load and to get smaller network latencies is using a content delivery networks (CDN). Most CDN’s support pushing media content through a simple API and can be integrated with the Magento backend quite easily.
Magento is designed to utilize benefits of running a multi-server setup in a clustered environment. Web-nodes are not limited to be of exactly the same type. There might be different nodes performing different tasks (frontend servers, static content and media servers and a separate admin panel server).
Scaling DB nodes
Magento works with a database in a manner that easily allows separating database connections for read and write activities. Each particular module can use its own connections if needed, which is fully customizable and can be easily set in app/etc/local.xml.
The following configuration snippet shows how to setup 2 separate connections to master and slave DB servers:
<dbname><![CDATA[Magento Enterprise Edition]]></dbname>
<dbname><![CDATA[Magento Enterprise Edition]]></dbname>
<initStatements>SET NAMES utf8</initStatements>
Magento can be scaled over any number of additional web-servers, which will allow for the handling of more concurrent requests by simply introducing new web-nodes when the number of page views and visitors grow. When the number of visitors grows, doubling the number of web-nodes can provide up to 60-70% performance increase.
Our results show better performance on physical dedicated servers than cloud environment.
Software Versions Used
CentOS release 5.3 (Final) Linux 2.6.24-23-xen SMP x86_64 GNU/Linux
mysqlVer 14.14 Distrib 5.1.36, for redhat-linux-gnu (x86_64) using readline 5.1 PHP 5.2.10 Apache/2.2.3
memcached 1.2.5 SIEGE 2.69 Magento Enterprise Edition 184.108.40.206
If you still have any questions you are always welcome to contact us or comment.