To understand the issue of writable (or not writable) storage, you must understand the history and development of web hosting technology. Let me begin with …
A short history of web hosting
Web hosting is as old as the (commercial) Internet itself. Once there was the infrastructure allowing you to offer your most important product / ideas / photos of your cat to the whole world, people discovered their overwhelming urge to do just that. However, not many people had the things needed to make their stuff available in the world wide web; so companies were founded to provide the metal and cables for them. Here we were: web hosting providers. On the one hand, gratis hosting services, such as good ol' tripod.com, Angelfire (both are now owned by former concurrent Lycos) and GeoCities stepped forward to give away ad sponsored websites for everyone for free. Most of the time with easy-to-use website builders but quite limited in their capabilities. On the other hand, people needed more capable (in terms of available technology and dedicated resources) and above all ad-free services. So paid, shared web hosting providers came to pass and after a very short time, this became a highly competitive market where competitors praised their products in ever more strident voices. Your typical hosting offer of those days read like this:
- CGI-bin & PHP incl.
- 1,000 GB Traffic
- 1,000 MB Storage
- unlimited MySQL Databases
- 18 ½ mailboxes + unlimited mail forwards
- Norton Anti Virus 30 day trial free shipped
With or without writable storage
And this is why companies like Heroku can afford to provide a totally different hosting environment which does not even include former corner stones such as writable storage. Now, there is a large number of web programmers which are up for it and do look at this not as an constraint but a possibility - or so it is praised. However, not everything new is good and what works for Ruby or Python developers not necessary work as well for PHP coders. In PHP, you have a huge legacy of established and widely used applications. Of course, if you, the PHP developer, start from scratch, writing a completely new app, you could do it probably just as well as the next Ruby guy.
Why without writable storage anyway?
If you are not a hosting provider, you might think: Why not just keep the writable storage? Why drop it in the first place? Well, here is the problem: Those new hosting providers with their fancy scalability need an infrastructure which is scalable to the same degree as their end-user-product! Those infrastructures, eg provided by Amazon AWS and Rackspace Cloud, come with their own quirks and problems. For example: you cannot predict the IP addresses of your next virtual machine. This one thing right here renders most, if not any, classic failover solutions impossible - or at least so inefficient or fragile that you cannot use them in any production environment. But without them, no hosting provider in their right mind would offer any kind of storage! So we and any other creator of such a modern hosting environment, have to come up with new solutions, which are neither widely documented in the Internet nor tested in millions of hours (hence new). If you cannot or want not expend the effort: no writable storage.
Contra writable storage
Having no writable storage available in the first place forces you to use other storage providers, eg Amazon S3. A direct result is, that any request to the "static" contents in the storage are not directed to the same machines your app runs on. This not only frees resources on the app server, but is also gives you a good starting situation for putting a CDN on top later on.