how NOT to design software for ease of troubleshooting

psypete@pinhead:~/svn/bin/src/etherdump$ svn up
At revision 211.
Killed by signal 15.

strace gives no indication wtf is going on and there's no debugging mode to give me more information. Of course this is subversion, so instead of a simple man page to give me some help I have to read through the 'svn book' or run commands 20 times to even know there's no debugging flags (afaict).

What is the actual problem and fix, after 15 minutes of googling? Some version bump re-introduced a bug (I didn't know I even upgraded subversion so perhaps this is some other thing making the bug pop up) that causes svn to kill itself if ssh isn't playing nice. Effectively you have to use "-q" anywhere that svn calls ssh, which was my weird tunnel subversion config change.

The tool could have spit out something like "hey, ssh is giving me shit, so i'm bailing; check out ssh" and it would have greatly decreased the time it took me to resolve the issue. Instead it just committed hara-kiri and told me a cryptic signal number. This is not how to design a tool made for user interaction.

a brief introduction to bad internet paywall security

for some reason everybody seems to leave some hole in an internet paywall you can go through to get free internet access. there are some obvious methods, and some less obvious methods. at the end of the day, though, you should be aware of all of these when you deploy one.

ip over dns

this one is a given. if you have a caching dns server/forwarder, ip over dns like iodine or NSTX will usually provide you with a somewhat-unstable-but-probable internet access. the fix is of course to just tell dnsmasq to point all lookups by an unauthorized client to your http server and provide an http redirect to the paywall site. apparently this is ridiculously hard for admins to comprehend.

tunneling out through firewall pinholes

if admins set their firewalls up right, there should be no packets originating from an unauthorized wifi client which can hit a host on the internet. apparently it's much easier to just allow any wifi client to connect to udp port 53 on a remote host without even using a real dns service to pass along the query. openvpn listening on port 53 becomes highly useful here. a creative hacker could use something like a google voice-powered SMS-controlled app to report back any SYN packets in a 10-minute window and just try all 65k ports to find an open pinhole in a firewall.

ip over icmp

this one isn't nearly as likely to work as the last two, but when it does work it's much more stable a connection than ip over dns. examples are hans and ICMPTX. however it's usually rate limited to around 23kB/s in my experience (and it's probably much much much slower on IPv6, according to the spec only allowing something like 4 ICMP messages per second?), so if you can use a tunnel straight to a remote host without going through another protocol and its overhead, all the better.

overly permissive transparent squid proxy

so far i think i've only found one such proxy that successfully denies http requests to unauthorized users. people just don't seem to understand that even if your proxy doesn't have an IP address i can still use it. a very simple example is just doing

`echo -en "GET http://www.google.com/ HTTP/1.1\nHost: www.google.com\n\n" | nc www.google.com 80`

. if this succeeds, their proxy is allowing anyone to just go right through to the internets without authing. to use this in practice download ProxyTunnel and use Dag's SSH-over-HTTP method to open an ssh tunnel with SOCKS5 proxy, or hell, a ppp-over-ssh tunnel to get Hulu to work. you should try both port 80 and 443 with this method as sometimes they'll only allow one outbound through the proxy. also take note that though the default transparent proxy might be too restrictive, you should scan the default route and the rest of the network with nmap for more open proxy ports like 3128, 8080, etc (hint: AT&T's open proxy port is non-standard). for the most part some variation on this ssh config line will get you what you want:

ProxyCommand proxytunnel -p www.google.com:80 -r remotehost:public_http_port -d remotehost:internal_ssh_port -H "User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Win32)\n"

MAC address/IP address cloning

this is probably the easiest/most reliable method to get through a paywall. if someone else is already authed, just sniff the network, find their MAC and IP address, set it as your own, and start browsing. to be honest i don't ever use this method but it should work in theory. if they enforce WPA encryption it should make this method difficult to impossible, though i'm really not up to speed on all WPA attacks.

virgin mobile broadband2go in linux

Recently I wanted to get internet in my home for cheap. My friend recommended Virgin Mobile's mobile broadband since it is now only $40 a month, pre-paid (no contract) and unlimited 3g data. This is by far the best deal you can get for mobile internet access in this country. Every other service is both more expensive and has a tiny data cap, and usually requires a contract.

It's no wonder every wal-mart I visited in a 20 mile radius was sold out of the Mifi, a battery-powered wifi hotspot and 3g modem. At $150 it's not cheap at all, but the supposed ease of set-up and ability to share internet with up to 4 wifi devices makes the convenience worth the price. Since I couldn't find one I opted for the Novatell Wireless MC760 usb 3g modem. At $80 it's much more affordable, but much more annoying to get working.

Only windows and mac are supported by the mc760. Normally this just means finding some half-working Linux driver and getting no support, which is pretty standard in the Linux world. In this case it's much much worse: you have to use windows or mac drivers and software to perform some magical rituals in the firmware before it'll even connect to the service. So there's really no way to use it without at least setting it up in windows or mac.

I of course didn't want to go along with this, mostly because it would be annoying to pirate a copy of windows just to get some crappy modem working. I tried for a couple days to get something to budge without a real windows install. I even eventually installed a VM of windows xp to try to set it up the "normal way" in a VM inside linux, but that still didn't work. I did end up using a windows machine to activate it finally. I'm still not sure I couldn't do it all from Linux, though.

So this is what I found out about the device. You plug it in and it does this wonderful thing where it pretends to be a USB CDROM and auto-runs a windows driver installer. The only way to turn this off in Linux is to use usb_modeswitch to detect the USB device and perform some magic to switch it to a ttyUSB0 modem/serial device. As usual, Slackware did this for me automatically without me knowing it, so I actually didn't even have to set that up.

The next thing I found out was how to configure the device as a modem. After hours and hours of googling and testing I found the secret ppp configuration that allows the modem to be controlled in Linux:

/etc/ppp/peers/virgin

460800
user Internet
password Internet
debug
connect '/usr/sbin/chat -f /etc/ppp/peers/chat-virgin-3'
crtscts
noipdefault
lock
modem
/dev/ttyUSB0
usepeerdns
defaultroute
connect-delay 5000
novj

/etc/ppp/peers/chat-virgin-3

TIMEOUT 10
ECHO ON
ABORT '\nBUSY\r'
ABORT '\nERROR\r'
ABORT '\nNO ANSWER\r'
ABORT '\nNO CARRIER\r'
ABORT '\nNO DIALTONE\r'
ABORT '\nRINGING\r\n\r\nRINGING\r'
"" "ATZ"
OK "AT&F"
TIMEOUT 60
SAY "CALLING ..."
OK "ATD*99***1#"
CONNECT c

/etc/ppp/pap-secrets

Internet * Internet *

All you really need to dial up the modem is "ATZ\nATDT*99\n" or similar. Some people use 777, but 99 works for me. The PAP username and password is "Internet". Now, using just these settings with a completely pristine modem you can actually connect to Sprint PCS' network. You get a 10.0.0.0/8 address and two Sprint PCS dns servers (68.28.146.92, 68.28.154.92) and one P-t-P gateway: 68.28.145.69 (though that may just be one of several gateways). The very trivial auditing I did showed DNS worked but ICMP, TCP and UDP were almost nonexistent. Their firewalls seem to be non-shitty, however, a DNS tunnel would probably still work.

What's REALLY fucked up is virgin won't give you the URL to sign up or activate your card. You need to install the software and click on it to take you to a public URL they could have told you before. (Thanks a lot for wasting a ton of my time, virgin mobile) So you go to this URL and sign up with the device - NOT using the device, mind you. You need the internet (and a computer with administrator rights) to do that. Oh, and their website sucks - I had to call tech support for them to tell me to clear my cache and restart my browser about 4 times in between parts of the sign-up process because their shitty webapp couldn't understand the concept of expiring or reloading a cookie or session id. If you can just try to set the whole thing up with their customer service at 1-888-322-1122.

You register your address and credit card with the device's ESN and get a login/pin code for their website. Then you log in and pick a plan and fill it up with money. The login and an additional code for activating the card (the MSID) are both new phone numbers local to your zip code. With the software installed on a computer and an account set up, you can begin to activate the device. You connect once and the software redirects you to a very simple, easily guessable URL based on the phone numbers above. This then tells you new numbers (which IIRC were the phone numbers above) to insert in fields in the connection software to complete the activation process. When you go to plug them in you notice the default values are zeros along with the last 4 of the ESN. I saw some links during my googling which makes me think some specific AT commands would allow you to register the device without using Windows or Mac. Somebody please sniff the usb connection and verify this for me.

Does this activation process require your PIN code? No. Does this require anything but two phone numbers related to an account with money in it? No. Would it be possible to spoof more than one device on their network with the same settings at the same time? Perhaps, but I bet they have a way to find dupes. (Keep in mind, this MC760 also contains a GPS receiver which i'm still trying to figure out how to tap into)

Anyway. After finally disconnecting and connecting again, the internets is go. Unplug from the crappy windoze/mac you've been forced to use to activate this thing, plug it into your linux box, wait a minute and then run `pppd call virgin'. You should be connected, given a public IP and the internet should just do it's thing. The funny thing? All the settings once you're connected are the same as when we connected before the activation. Only the IP address is changed. HMMMMM. I wonder if we could just spoof an IP address and use the internets without activating? Again, this thing has GPS built in so don't think you wouldn't be tracked down.

The speeds i'm getting vary from 6Kbps to 1.2Mbps down and 1Kbps to 30Kbps up. This is not completely out of the range of current 4G connections, as embarrassing as that is for 4G users. So far in about half a day the connection has cut out twice for several minutes at a time and the card is extremely hot. I would recommend the Mifi if you have the cash.

edit when the usb card gets hot, it gets HOT. this makes performance suffer. example:

reason number 9039808204802 why i hate RPM

[root@dhcp9001 ~]# rpm -Uvh dhcp-3.1.9999-2.cbs.i386.rpm
########################################### [100%]
        package dhcp-3.0.5-7.el5 (which is newer than dhcp-3.1.9999-2.cbs) is already installed

NO, IT IS NOT NEWER. FUCK YOU. INSTALLLLLLL!!!!!

quality control in your network

(disclaimer: i've never worked in quality control, but this is my view of it as someone who has had to work with QC)

while i sit here at 3 in the morning waiting for a server daemon to dutifully seg fault leaving me to continue debugging, i reflect on how quality control is lacking from so many networks both large and small. a single oversight *can* mean the difference between your company losing money or going under, so you must be aware of any potential problems at all times.

first of all, what is quality control and why do you need it? well, chances are if you do your job halfway right, you're already doing it. quality control is basically double-checking that the methods you use to do your job are correct. it doesn't verify that the final product is good; it's more like, tripwire in procedure form. it's making sure that things work the way you expect them to. you are already doing it when you verify what development libraries are installed. when you run unit tests against your software. when your change management system verifies a user is allowed to commit that particular piece of code, or restart that service. it's checking the tapes to make sure the backup robot is functioning. it's verifying configs are written properly on the router and the updated ones are regularly saved in version control.

typically you don't need much quality control in the average network. some product development may require strict control and observation of policies and procedures, which is usually only reinforced due to the risk of random audits or inspections. depending on your environment you may be required to do very little or no quality control at all. but i'd like to tell you about the quality control you should be doing.

the quality control people aren't usually technical people. a lot of the time they'll work with a team member of whatever they're checking out, ask questions and make notes. the first big formal procedures don't include everything. usually details get hashed out while the QC engineer talks to someone (a dev in this case) about what they do and how to check that what they did worked.

the basic principles you should keep in mind when applying quality control to your network are as follows:
1. Keep It Simple, Stupid. it doesn't have to be verbose or complex. be flexible. be easy.
2. it should be possible for someone to check the work of the quality control engineer(s).
3. you don't want to define how everything works; only how to tell if it's working as expected.
4. your goal is to make sure there are no catastrophic failures. you don't have to account for every blip along the road as long as the road is open.
5. start with the big things and move down once all the big things are covered. close off those single points of failure and move on to the other pressing issues.

hopefully this post will help to give you an idea of how you can apply quality control to your network to get an improvement in the overall quality of service you provide. half of this is just making sure things works right, and then the other half is reviewing that there are records that it's been used properly in the past. here's some stuff you can do.

developers
double-check that your software is being created correctly. check that the libraries on your development boxes match up with what's going into QA or production. use unit tests on your code. make sure everything goes into version control *before* it ever hits QA or production, and make sure you know who made what change and why. make sure the method of deployment can be reversed at any time. make sure you follow change management procedures when necessary.

sysadmins
double-check that you've confirmed with everyone before you push a new piece of software to QA or production, and that you can roll it back when necessary. so check that your change management is working. it's good to have a list of the major and sub-major software that different development teams rely on (usually libraries) and get a change-management approval before ever pushing this stuff out. do it early so devs have time to test their shit with the new software. make sure your backups work correctly. you should be able to confirm logs and destination files to ensure the backups are going well regularly. if those or any other automated process fails it should generate an alert, and you should be able to verify those alerts are going out as expected (did /var fill up and is sendmail unable to work now?). make sure all security patches are applied in a timely manner. make sure all service-monitoring systems are working, and that failover of critial systems is in place and works as expected. make a list of all critical infrastructure and make sure all of it has hot-spare failover systems waiting in the wings. provide for methods to remote troubleshoot in the event of total internet or system collapse. make sure any network gear you depend on also has hot failovers that work.

there's more implementation-specific details you sometimes need to get into with QC. i want to get more into how to begin making procedures for these systems but it's way past my bedtime. will continue when i am not so sleepy.

you poor apple users

i really feel sorry for people who recently got the iphone 4. i sat at work, listening to my co-worker half-heartedly explain how his iphone sometimes gets really bad reception, but that it's ok. that he naturally holds his phone in exactly the way necessary to avoid the signal loss. how it's not alright that it has this problem, but it's also ok because it doesn't affect him for the most part.

i just had this twang of empathy. like, i see now... you really don't have a choice. what are you going to do, return the phone and get a non-iphone 4? it's not an option for him. in one sense because it's now ingrained into his life; the apps, the services, the way the phone feels, the way he uses it.... that's part of him now. he can't let it go. it's scary to think of not having an iphone. and that's the second sad thing. it's actually changed his manner of thinking and now he can't get away from the thing. it's like an addiction. he can't see himself without it, and now he's trapped by it.

this is a guy who less than a year ago had never owned an iphone. he's technically a "late adopter." yet in less than a year apple has not only converted him, they've made him their slave. in some ways i can relate... since getting a smart phone i feel like i need to have a more powerful one. i need to be able to stream video, or use irc or ssh remotely, or browse web pages fast and in full render. do i actually need these things? fuck no. i was perfectly fine with my old brick sony ericsson with the wonderful camera, sending pictures and using web pages just fine. now i sit in at&t stores trying to sell myself on buying the newest piece of shit android phone which still doesn't have as good a camera (and definitely no xenon flash) like my old phone.

so i get it now. i'll stop pointing fingers. i'll stop trying to convince you. because i know you can't escape it. you're trapped in a tar pit of technology and you can't get out even if you wanted to. i feel sorry that you're stuck with a (somewhat) shitty phone with a shitty app development model on a really shitty carrier. i wish we could all just have open-brand open-carrier open-market smartphones and share information and use the internet as freely on our mobile devices as we can on our computers.

but we can't. we probably won't ever, or not for a long time anyway. they figured out the way to trap us and steal more of our money than they ever did with the PC or laptop. it's overly-expensive unlocked phones and contracts and insurances and data plans and messaging plans. i pay $110 a month for a "standard" phone plan with "unlimited" data and "unlimited" messaging. that's $2,640 dollars every 2 years (without the extra amount i pay for a new phone every 1.5 years). i don't even buy a laptop or PC that often, or gaming machines or games. i just paid $520 for an almost-new laptop, the first big purchase in over 5 years.

this is kind of depressing me. i feel like i'm trapped too.

what's wrong with my breakfast?

a recent survey found that 85% of a selection of engineers don't use twitter. they cited not caring what people had for breakfast as a reason they don't use it. this is my response.

what's so wrong with my breakfast that you can't stomach the information? i realize it's useless. i realize you don't care. but also realize, i don't care that you don't care. i am one of the mindless drones of twitter and [formerly] brightkite and facebook and myspace that update our status with whatever mindless drivel we happen to think is important in the moment at that time.

there's not much logic involved. wanna say something? say it. people will listen or they won't. but i don't expect them to. it's more of a general smattering of my thoughts and a few choice life experiences that people can refer to if they wish in the future. it can be a way for a potential employer to see if i might be a fit for their organization. it might be a way for a single young lady (or lad?) to determine if i'm worth sending a poke/message/direct tweet/whatever. perhaps i just want people to know i know certain things, or have certain opinions. whatever it is it's certainly one part exhibitionism, one part honest sharing of experiences and thoughts.

it would be nice if it had more function. perhaps a symbol prefix to be flagged in a number of ways: exclamation mark for an urgent or important message, question mark to "crowdsource" (ugh), pound sign to advertise an event, dollar sign to advertise a neat deal or other ad. extra modifiers could be postfixed to give further detail about the post. then each user could set their preferences of what kind of information comes into their stream from their friends.

that doesn't exist, though. what does exist is "i'm at place XYZ and i'm having a blast!" or, "this is a picture of a fucking AWESOME fish taco" or, "who else thought iron man 2 was kind of lame?" my posts aren't intended for the general public. they're not always even meant to be useful. i send this shit out because i'm bored and i want to share with my friends. they don't even have to interact with me. in some small way i am enriching lives with my pointless drivel. i help kill boredom. sometimes i share articles which i find useful or informative. sometimes i share music that means something to me.

you probably won't find it useful at first. you'll probably be too bored with it to even start. but once you have a good chunk of friends added into your account, you'll notice you can interact with them. you can follow what some of them are doing. maybe even find out something you wouldn't have without the service. yes, we should all have real lives and not be so connected to a text interface all the time. but sometimes technology can be more than just a distraction.

granted: most of twitter is pointless, and the examples i give which contain somewhat useful information is by and large not close to the majority of the tweets out there. twitter kinda sucks. but facebook is a good example of what it could be. the old brightkite was a good example of what it could be. maybe in the future all these techie news aggregating sites can turn into an honest-to-goodness social network of just nerds so we can all collaborate interactively and enrich each other's work and personal lives on common ground, where we can hack on it and do what we want with our own little home on the internets. but that's probably just a pipe dream.

i don't care if you use it because i don't watch my twitter feed tbh. but you can get a PixelPipe account, add your friends that are on the various social networks, and share with them all at once. you don't even have to follow what they're saying. just say something. try to make it useful; who knows, maybe you'll start a trend.