I'm sure in 10 years people will eventually get just why it's a crappy idea to reinvent the wheel in Operations departments. Maybe somebody will finally standardize on a set of software configurations to deploy to manage an enterprise network. Could just be wishful thinking, but you never know.
Right now I work at a large non-profit as a small cog that's part of some big wheels. There's lots of people on my team that all basically do one or two specific jobs. In terms of how we go about accomplishing tasks it's incredibly inefficient. But that's the nature of big non-profits, I assume.
One example of how this "devops" idea fails came up recently. Somebody wanted a tool that rotates logs passed via standard-in. This is an old problem: you've got to keep processing logs but you can't afford the downtime of moving the log and restarting the service. So you open a pipe to a program which handles the juggling of logs for you. The easy choice is cronolog, a very old, stable open source tool.
When I asked if we could install it on all our RHEL boxes, I was told there was too much red tape involved with just getting a 'yum install' done, so I should just compile and install the software locally. (Yeah, i'm serious.) So after doing this I modify the script we need to use cronolog, test it, and it works great as expected.
Once i'm ready to push this out everywhere i'm told to hold off, that we need a better solution. Better than 'its done and working'? It turns out, if you use 3rd party software like this in our environment, three things happen:
1. Since we don't pay for the software there's no support contract. If there's no support, apparently it's too much trouble to find somebody who knows C in a building full of developers to support the software.
2. As a side-effect of #1, the security team can't get updates from a vendor and thus might not allow us to use the software as it might have unknown security holes.
3. We have to change our build standard to include the new software.
Now keep in mind, cronolog is shipped with RHEL. It just isn't installed on the machines. So getting it installed brings all sorts of red tape questions. What's their solution? Write something from scratch in-house. This of course is a lovely paradox, because:
1. We have to support the in-house solution now.
2. Nobody is going to audit the in-house solution for security holes.
Of course we don't have to change our build standard and it isn't a security conflict for one simple reason: Everyone ignores in-house software. That's right: the loophole to the regulation is to simply completely ignore auditing for internal tools. And there's an interesting point about devops/engops.
If you had actually paid for a product instead of developing it in-house or getting an open source solution, you would have assurance of both the attention to and patching for security holes, and you would just use it - there's no custom scripting or in-house wrapping required. Not only does it speed up your work to pay for it but it makes it more secure (in a fashion) and prevents you from having to spend development cycles.
The next time you decide to hire a 'devop' or 'deveng', consider how much money and time you'd save if you just spent a couple hundred or grand on completed, supported tools. (And as an aside, try not to allow gaping loopholes in logic like the ones pointed out above)
Edit: my mistake, cronolog is not shipped with RHEL (why I do not know). I ended up writing my own as it's much simpler than I originally thought. However, getting them to support 'logrotate' on solaris is looking like a bigger challenge...
No comments:
Post a Comment