This post comes from my observation of a large number of emails on Barcamp and other mailing lists on ‘choosing the right host’ for start up web infrastructure. There is also a general impression that substantial investments are required to serve applications on production infrastructure. There is also an impression that start ups start out from having a ‘big single box’ architecture. All of which are wrong ways to go about getting your popular application serve a growing user base at a low cost.

I thought I will get a couple of experts to talk about building scalable and performing infrastructure for your start up that won’t cost a bomb. These are coming. But before they come, I thought I will post this to reflect a novice’s (me, of course) views on how to do it. Read the full post for a few case studies.

• Architecture : this is very important. Before you come up with an architecture that can deal with scale, just know how you want to perform (response times, resource usage, throughput) because the scale will need to handle performance and there is a trade off between scale and performance. Cache where you can, partition your data (using shards), think about the file system to store content, think about using clouds like Amazon Web Services, etc.
• Design : do this carefully. For schema design, selection of storage engines, choosing which data types to use, selecting indexes to use and handling concurrent database connections, you need to think well and do it well.
• Configuration : From selecting which web and database server you can use (eg, lighthttpd vs Apache, BerkeleyDb vs MySql) to RAM used, output buffers, compressing data to keep alive connections, this is probably more choice than skill.

Whatever you do, benchmark and profile your systems as you go.Startups to Scale