open source... infrastructure?

so, back on the topic of tools we'd all like to have but nobody'd like to write. that basically covers "tools" and generic glue to provide specific functions. but what about real world (soft and hard) infrastructure used to provide an enterprise-sized service and maintain it? i'm talking about soup-to-nuts planning for a network with thousands of machines, 100,000 users and millions of clients/customers/visitors.

first off, what hardware do you use? what network gear for the internet pipe, switches for the racks and storage devices? what is your expected backup capacity? do you even need hard drives in your servers? (cutting power cuts cooling cuts costs, and moving parts can be sinful to replace) how are you accounting for failover and redundancy?

what software do you use? is your web farm serving a litany of different interpreted languages and required versions and support libraries or is your one site purpose-built with one framework and language? do you use multiple http servers and proxies or one with an incredibly flexible configuration? what kind of operating system(s) are you going to support, and how do you maintain them? how much customization of open source apps are you going to need and how are you managing it? how many teams work with your platform and what methods of authentication and authorization will you use? how are you deploying changes, reverting them, auditing, etc?

how are you dealing with people? what are your coding best practices and do you enforce them? what is the one version control system you're going to use for the next 10 years and how have you set up your repositories? what's your ticketing and bug-reporting systems like? do you have anyone working between teams to make sure communication remains tight and everyone knows what everyone is doing (more or less)?

of course this is basically "how do i run an enterprise network?" plus glue code and hints/tips. but the point is to make most of the decisions and ensure common problems are avoided while reducing the duplication of work inherent in building out any enterprise network using open source tools. i think this is part of the point of open source: get your ideas down on paper, implement them, show your work and let people modify it and contribute their work back to let everyone benefit. this would apply to everything from the planning stages of building out a large site to the little glue scripts that rotate logs and build code changes.

what i'm talking about would not make a lot of people happy. it's probably a lot like car tuners, where the people who do the work of tuning cars are not happy when the people with tuned cars share their tuning maps. their source of income (tuning cars) is supposedly threatened by people just downloading a pre-made map file. but there's more than one way to map a car, and more than one way to build a network. but i'll be happier to not have to duplicate my work and just be able to get stuff done. managers might like not having to throw away more time and money on duplicate work. and it'd be a fun project to work on =)