I use facebook. It's handy. The interface is very clean and usually pretty responsive, and I like the way it presents a stream of online social interaction between people. But what I don't like is their "tactics" to deal with people who try to interface with the site.
Recently I wanted to try and scape my own facebook profile for my friends' information so I could aggregate it and view it on my own terms. If your facebook was suddenly gone, wouldn't you want links back to your previous friends or contact information to get ahold of them again? A link to one such perl script here shows the typical response of Facebook to anyone who writes such a script. It was even removed from the Wayback Machine at archive.org, even though the rest of the site from when the script existed is still there.
Another site here details how a developer had his profile disabled due to trying to export his friends' contact information for an app he develops. It's not unusual to see an application log into a service and extract e-mail addresses so you can find the same people on a different service. However, getting your account dropped because of it is annoying to say the least.
At least there's some content out there that hasn't been taken down. This blog post shows how to log in and update your status. Hopefully this will get me close to a workable script; i'll have to create a fake profile to avoid my normal one being "deactivated", though.
Wednesday, December 23, 2009
Monday, December 14, 2009
on job requirements and resource utilization
If i'm hired as a sysadmin, make me do sysadmin work. Don't make me do programming work.
If i'm hired as a programmer, make me do programmer work. Don't make me do sysadmin work.
Allow me to do either if necessary, but don't task me with something I wasn't hired for.
There's a very logical and simple reason for this. Would you hire a plumber to fix your shower drain and then instruct him to teach a high school English class? It may be possible for him to perform the basic duties of the teacher. But don't expect the kids to make it in to any nice colleges.
Likewise, if you hire a sysadmin and make him take on a programming project you're going to end up with a retarded program. If you really want him to take on a programming project, evaluate him as you would a software developer you were about to hire (because that's what you've just effectively changed his title to). Is this the person i'd want to hire full-time to develop software? Am I confident he is the best person qualified to take on this job? If not, you're barking up the wrong tree.
I'd trust a developer with my system about as far as I could throw them. I think most sysadmins would agree they would rather a developer have less access over a system than more. We've seen the mistakes they make when they don't fully understand the platform they're developing for. The same mistakes are made by sysadmins when developing with a language or framework they don't fully understand, or without the fundamental knowledge of how to develop and maintain software properly.
The job of a "sysadmin/coder" is effectively asking for half a sysadmin, half a programmer (unless you're paying me double the salary). And that's probably what you'd get in the end. We gain experience and strength in our fields by working on them for the majority of our time, not by dabbling here and there and becoming jack of all trades, master of none.
If i'm hired as a programmer, make me do programmer work. Don't make me do sysadmin work.
Allow me to do either if necessary, but don't task me with something I wasn't hired for.
There's a very logical and simple reason for this. Would you hire a plumber to fix your shower drain and then instruct him to teach a high school English class? It may be possible for him to perform the basic duties of the teacher. But don't expect the kids to make it in to any nice colleges.
Likewise, if you hire a sysadmin and make him take on a programming project you're going to end up with a retarded program. If you really want him to take on a programming project, evaluate him as you would a software developer you were about to hire (because that's what you've just effectively changed his title to). Is this the person i'd want to hire full-time to develop software? Am I confident he is the best person qualified to take on this job? If not, you're barking up the wrong tree.
I'd trust a developer with my system about as far as I could throw them. I think most sysadmins would agree they would rather a developer have less access over a system than more. We've seen the mistakes they make when they don't fully understand the platform they're developing for. The same mistakes are made by sysadmins when developing with a language or framework they don't fully understand, or without the fundamental knowledge of how to develop and maintain software properly.
The job of a "sysadmin/coder" is effectively asking for half a sysadmin, half a programmer (unless you're paying me double the salary). And that's probably what you'd get in the end. We gain experience and strength in our fields by working on them for the majority of our time, not by dabbling here and there and becoming jack of all trades, master of none.
building a robot
i'd like to build an R2D2-like robot.
the goal would be a 3-4ft robot which is sturdy enough to fall down stairs and survive, can push buttons, go over uneven terrain, be remote-controlled via cellular modem, be equipped with a webcam and screen and contain a locking storage compartment.
i had some fancy plans of how i'd make it work just like a real R2D2. it'd probably be very costly and take something like 3 years to build, based on my current knowledge. so i simplified.
Now all I need is about $3000 and a nerd with a mechanic shop and a big heart.
the goal would be a 3-4ft robot which is sturdy enough to fall down stairs and survive, can push buttons, go over uneven terrain, be remote-controlled via cellular modem, be equipped with a webcam and screen and contain a locking storage compartment.
i had some fancy plans of how i'd make it work just like a real R2D2. it'd probably be very costly and take something like 3 years to build, based on my current knowledge. so i simplified.
- do away with complexity and just strip a golf cart - make it 2 wheel drive with the 3rd wheel just a castor
- body could be two trash cans for more support
- a rectangular frame to simplify fabbing - though mandrel-bent isn't complicated or expensive these days
- borrow someone's mechanic shop, i know people with welding stuff
- all solid state electronics
- there'd need to be significant ballast on the caster wheel, and lowest center of gravity possible; probably put the battery by the caster wheel and cpu behind it
- driving would be firmware-controlled tied to the proximity sensors and the CPU would override
- buying a pre-existing arm would be best. otherwise, one pneumatic lifter up out of the can, and one that shoots out forward to push a button, with the webcam on the head
- firmware would probably take the longest to finish
- could put a hole in the middle of the can and turn this thing into a sex toy, rent it out to parties to subsidize the costs
- an LCD "face"
- to simplify the "CPU" could use an android phone or netbook. if there's a power failure you could still communicate with it (and locate via GPS)
- by using trash cans for the body and reinforcing the outside with a mandrel-bent tubular steel frame i could put a keg inside and make it a self-propelled kegerator
Now all I need is about $3000 and a nerd with a mechanic shop and a big heart.
Wednesday, December 9, 2009
open source... infrastructure?
so, back on the topic of tools we'd all like to have but nobody'd like to write. that basically covers "tools" and generic glue to provide specific functions. but what about real world (soft and hard) infrastructure used to provide an enterprise-sized service and maintain it? i'm talking about soup-to-nuts planning for a network with thousands of machines, 100,000 users and millions of clients/customers/visitors.
first off, what hardware do you use? what network gear for the internet pipe, switches for the racks and storage devices? what is your expected backup capacity? do you even need hard drives in your servers? (cutting power cuts cooling cuts costs, and moving parts can be sinful to replace) how are you accounting for failover and redundancy?
what software do you use? is your web farm serving a litany of different interpreted languages and required versions and support libraries or is your one site purpose-built with one framework and language? do you use multiple http servers and proxies or one with an incredibly flexible configuration? what kind of operating system(s) are you going to support, and how do you maintain them? how much customization of open source apps are you going to need and how are you managing it? how many teams work with your platform and what methods of authentication and authorization will you use? how are you deploying changes, reverting them, auditing, etc?
how are you dealing with people? what are your coding best practices and do you enforce them? what is the one version control system you're going to use for the next 10 years and how have you set up your repositories? what's your ticketing and bug-reporting systems like? do you have anyone working between teams to make sure communication remains tight and everyone knows what everyone is doing (more or less)?
of course this is basically "how do i run an enterprise network?" plus glue code and hints/tips. but the point is to make most of the decisions and ensure common problems are avoided while reducing the duplication of work inherent in building out any enterprise network using open source tools. i think this is part of the point of open source: get your ideas down on paper, implement them, show your work and let people modify it and contribute their work back to let everyone benefit. this would apply to everything from the planning stages of building out a large site to the little glue scripts that rotate logs and build code changes.
what i'm talking about would not make a lot of people happy. it's probably a lot like car tuners, where the people who do the work of tuning cars are not happy when the people with tuned cars share their tuning maps. their source of income (tuning cars) is supposedly threatened by people just downloading a pre-made map file. but there's more than one way to map a car, and more than one way to build a network. but i'll be happier to not have to duplicate my work and just be able to get stuff done. managers might like not having to throw away more time and money on duplicate work. and it'd be a fun project to work on =)
first off, what hardware do you use? what network gear for the internet pipe, switches for the racks and storage devices? what is your expected backup capacity? do you even need hard drives in your servers? (cutting power cuts cooling cuts costs, and moving parts can be sinful to replace) how are you accounting for failover and redundancy?
what software do you use? is your web farm serving a litany of different interpreted languages and required versions and support libraries or is your one site purpose-built with one framework and language? do you use multiple http servers and proxies or one with an incredibly flexible configuration? what kind of operating system(s) are you going to support, and how do you maintain them? how much customization of open source apps are you going to need and how are you managing it? how many teams work with your platform and what methods of authentication and authorization will you use? how are you deploying changes, reverting them, auditing, etc?
how are you dealing with people? what are your coding best practices and do you enforce them? what is the one version control system you're going to use for the next 10 years and how have you set up your repositories? what's your ticketing and bug-reporting systems like? do you have anyone working between teams to make sure communication remains tight and everyone knows what everyone is doing (more or less)?
of course this is basically "how do i run an enterprise network?" plus glue code and hints/tips. but the point is to make most of the decisions and ensure common problems are avoided while reducing the duplication of work inherent in building out any enterprise network using open source tools. i think this is part of the point of open source: get your ideas down on paper, implement them, show your work and let people modify it and contribute their work back to let everyone benefit. this would apply to everything from the planning stages of building out a large site to the little glue scripts that rotate logs and build code changes.
what i'm talking about would not make a lot of people happy. it's probably a lot like car tuners, where the people who do the work of tuning cars are not happy when the people with tuned cars share their tuning maps. their source of income (tuning cars) is supposedly threatened by people just downloading a pre-made map file. but there's more than one way to map a car, and more than one way to build a network. but i'll be happier to not have to duplicate my work and just be able to get stuff done. managers might like not having to throw away more time and money on duplicate work. and it'd be a fun project to work on =)
Tuesday, December 8, 2009
breaking up with BK & RESTful APIs with curl
This article sparked a mini-firestorm on BrightKite. As a couple of the comments at the end of the story mention, several users were apparently deleted from BrightKite due to their negative reactions to this story. There was also a number of users that were deleted because of some kind of AT&T-related "removal" notifications, but some users who were deleted did not use AT&T. In the end BrightKite fucked up yet again.
I am finally getting around to deleting my BrightKite account. For some reason all of my pixelpipe posts have been failing to Twitter but succeeding on BK, so people have actually been commenting on my BK account recently. I've also been getting some strange "love spam" through BK, which is yet another lesson in why this new BK 2.0 sucks: there is no "report" button for posts or users, so I can't flag something as spam. Wonderful new system, guys.
So anyway, this was a nice quick introduction to working with RESTful APIs with curl. I found one example of how to delete all brightkite posts and decided a bash one-liner would be simpler. This page describes how to specify the method and other REST-related options using curl. Here is the one-liner I ended up using:
Of course, being BrightKite it doesn't work the way it should. Only a small amount of posts get deleted at a time, I don't know why. Intermittently you'll receive a spam of HTML 404 pages, and every single request (authenticated or not) gives a '403 Forbidden' error. No matter; running the one-liner over and over gets them all eventually.
When i'm done i'll post a couple about how BK sucks, and not delete the account outright: I want my opinions of their suckitude to linger for others to discover. It's pretty sad, too: BK actually worked in the beginning and was pretty useful when the few other users down here used it frequently.
I am finally getting around to deleting my BrightKite account. For some reason all of my pixelpipe posts have been failing to Twitter but succeeding on BK, so people have actually been commenting on my BK account recently. I've also been getting some strange "love spam" through BK, which is yet another lesson in why this new BK 2.0 sucks: there is no "report" button for posts or users, so I can't flag something as spam. Wonderful new system, guys.
So anyway, this was a nice quick introduction to working with RESTful APIs with curl. I found one example of how to delete all brightkite posts and decided a bash one-liner would be simpler. This page describes how to specify the method and other REST-related options using curl. Here is the one-liner I ended up using:
curl -s -G -u user:pass http://brightkite.com/people/user/objects.xml | \
grep "<id>" | sed -e 's/^.*<id>\(.\+\)<\/.*$/\1/g' | \
xargs -n 1 -I {} curl -s -G -u user:pass -X DELETE http://brightkite.com/objects/{}.xml
Of course, being BrightKite it doesn't work the way it should. Only a small amount of posts get deleted at a time, I don't know why. Intermittently you'll receive a spam of HTML 404 pages, and every single request (authenticated or not) gives a '403 Forbidden' error. No matter; running the one-liner over and over gets them all eventually.
When i'm done i'll post a couple about how BK sucks, and not delete the account outright: I want my opinions of their suckitude to linger for others to discover. It's pretty sad, too: BK actually worked in the beginning and was pretty useful when the few other users down here used it frequently.
Monday, December 7, 2009
everybody wants to work, but not for free
It's fun to code something new or challenging. It's not fun to code a boring piece of glue just to simplify some mundane task like deployment, rollback, authorization or peer-review. That stuff doesn't have a real "payback" in terms of emotional satisfaction with completing a project. Design a new file transfer tool and protocol? That's sexy. Making a QA tool to handle code changes or deployments before they hit a live site? Not so sexy.
So we do this work for our companies as part of the job. We write our little wrappers and snippets of code, push it out and let the company retain this hack to keep things running smoothly. But you'll notice nobody really designs open-source enterprise code/web-app management systems. That's not sexy and not relevant to an open-source hacker's dreams of getting the tools they want completed and in the hands of developers and admins world-wide. The end result is reinventing the wheel.
How many times has a complicated deployment system been crafted at a large company, only ever to be used there by a handful of developers? What's the gain by keeping this internal tool locked up? What are you really gaining by spending these man-hours and seeing it go unmaintained and undeveloped? Shouldn't we perhaps be exploring every opportunity to expose our internal tools to the world so that they may be flushed out, improved upon and eventually re-used by ourselves and others?
But many of these tools aren't suitable for a general purpose. We don't want to write a deployment system that works in any site in the world, so we make it work for us. This'll work for svn but not necessarily git. That'll work for Apache and mod_perl but not with Tomcat and java. You may need to tar up your work and keep it in a repository while i'll use a quick interface to my filer's snapshots and source code to manage rollbacks. Don't even ask about change management.
When is the focus going to shift from the latest, coolest language and framework to the gritty glue that keeps it all flowing? To make a site that really works you need it to be well-oiled, not just in form but in function. I'd love to get paid just to make tools that work with any site, anywhere, any time, one single way. But that will never happen. I'll get paid to make it just work, and that's what i'll end up implementing. I find it a waste to spend your time polishing a grommet when you've got the operation of a whole machine to see to. Some day perhaps we can identify what we lack and knock it out bit by bit. In the meantime i'm just reinventing the wheel.
So we do this work for our companies as part of the job. We write our little wrappers and snippets of code, push it out and let the company retain this hack to keep things running smoothly. But you'll notice nobody really designs open-source enterprise code/web-app management systems. That's not sexy and not relevant to an open-source hacker's dreams of getting the tools they want completed and in the hands of developers and admins world-wide. The end result is reinventing the wheel.
How many times has a complicated deployment system been crafted at a large company, only ever to be used there by a handful of developers? What's the gain by keeping this internal tool locked up? What are you really gaining by spending these man-hours and seeing it go unmaintained and undeveloped? Shouldn't we perhaps be exploring every opportunity to expose our internal tools to the world so that they may be flushed out, improved upon and eventually re-used by ourselves and others?
But many of these tools aren't suitable for a general purpose. We don't want to write a deployment system that works in any site in the world, so we make it work for us. This'll work for svn but not necessarily git. That'll work for Apache and mod_perl but not with Tomcat and java. You may need to tar up your work and keep it in a repository while i'll use a quick interface to my filer's snapshots and source code to manage rollbacks. Don't even ask about change management.
When is the focus going to shift from the latest, coolest language and framework to the gritty glue that keeps it all flowing? To make a site that really works you need it to be well-oiled, not just in form but in function. I'd love to get paid just to make tools that work with any site, anywhere, any time, one single way. But that will never happen. I'll get paid to make it just work, and that's what i'll end up implementing. I find it a waste to spend your time polishing a grommet when you've got the operation of a whole machine to see to. Some day perhaps we can identify what we lack and knock it out bit by bit. In the meantime i'm just reinventing the wheel.
Subscribe to:
Posts (Atom)