"jump server". just the term itself conjures up an image of "getting around" security or the network. it's a HACK. unless there's a big problem with your network, you should be able to allow access directly to the server you need to get to. connecting to one host just to connect to another host is retarded.
the only thing that is a potential benefit is that you're essentially forcing any network communication through one protocol (which can subsequently be circumvented on the jump server, depending) and (again, depending on the jump server) authenticating twice.
the bad things? it's incredibly, incredibly slow to transfer files. functionality with different protocols becomes broken. and you're circumventing the firewalls and network security. once you tunnel to the jump box it becomes much more difficult to determine who is connecting to where (after the jump box). and attacks on the internal network get much more interesting, not to mention if you escalate privs on the jump box you can piggyback any connection any other user is making from the jump box. not to mention you're forcing a new layer of complication onto your users so doing their job becomes more of a hassle - which almost by definition inspires people to break good convention for the sake of convenience. not to mention it's a waste of resources.
systems guys, don't jerk your users around. if there's a way you can get something done quicker, do it. for example: resizing logical partitions in a VM guest.
if your user wants 10GB more added to their work partition, get a procedure in place so you can do it live. rebooting the server should not be necessary for most admin tasks on a unix host.
don't believe me? read this blog post explaining how to extend an LVM volume while the box is still up. hey, now i don't have to wait a day to keep doing my work on a server which was allocated way fewer resources than it should have had!
Monday, September 26, 2011
Monday, September 19, 2011
secure kickstarting of new linux servers
PXE is not secure. Not only does it rely on broadcast requests for a PXE server, it uses UDP and TFTP to serve files, thus removing any remaining security features. It also can't be used on WANs and typically requires admin infrastructure and VLANs set up wherever the boxes will be installed, so lots of admin overhead is required.
To get around these problems and provide a secure mechanism for remote install you should use a pre-built linux image on CD-ROM or floppy disk. Most servers today come with one or the other, and both provide enough space to include a kernel and tiny compressed initrd with barebones networking tools.
The kernel should obviously be the newest vanilla kernel possible, patched to include any relevant hardware support. The more vanilla the better as you can quickly pick up the newest released kernel and build it without needing to modify vendor-specific patches.
The initrd should probably be based on an LZMA-compressed mini filesystem or cpio archive. Usually something custom like squashfs works the best. You'll need to build busybox and the dropbear ssh client in a uClibc buildroot environment to make it as small as possible. Bundle an ssh key for the admin server along with DNS and IP information for the server (in case DNS resolution fails, it can try the last known good IP address(es)). It should try indefinitely to get an IP address, and once it gets one it should make an SSH connection. Once SSH connection is established it should download install scripts and execute them. All downloads happen through the secure SSH tunnel.
The initrd should also include proxytunnel and potentially openvpn or another UDP-based SSL tunnel so it can fall back to trying to connect through HTTP or udp port 53. You may need cntlm to work with NTLM proxies as proxytunnel's support does not seem to work for all versions of NTLM. Also, rsync should probably be included, or downloaded immediately once a network connection is established. Big apps can always be downloaded to a tmpfs partition later once the SSH connection is established. The initrd should also use a bootloader that can pass custom arguments to the initrd at boot time, for example to specify proxy or IP settings. It should probably support Web Proxy Autodiscovery Protocol for corporate environments.
If you have room, you may also want to bundle grub with the initrd. This can help you recover a system if it fails and it will allow you to install grub over whatever's currently on the hard drive. A good grub configuration to install would include options to boot from hard drive, CDROM or floppy, so as long as you have remote console you can reboot the box and select the install image from grub at boot time without needing to change BIOS settings.
Finally, each client image should be modified before burning/imaging to have custom ssh keys. You want each boot image to be able to be revoked from the admin server's login list, in case a boot image/disk is compromised or stolen. Granted, you're not giving this thing any more rights than access to your kickstart file tree, and that shouldn't have anything super confidential on it anyway. To customize the ssh keys per image you can have a script which generates new images, creates keys for each one and renames the image file to something specific to the machine. Match up the specific piece of hardware with this unique image file in your network's inventory database.
The kickstart server should have IP addresses dedicated only for the kickstart clients. An HTTP reverse proxy should be installed for ports 80 and 443 so that the client can use proxytunnel to connect to SSH through HTTP proxies. Optionally an openvpn daemon should be enabled on port 53 in case a client's firewall has an open outbound port 53. Each kickstart server's SSH daemon (listening only on the kickstart IPs) should use the same host keys so they can be copied to the initrd and you won't have to worry about the server host keys changing per box, messing up the initial connection from the clients. You could manage all the kickstart host keys independently but why complicate matters further?
In the end what you have is a client install CD or floppy which can boot up on a network, connect securely to a remote server, configure itself and follow setup instructions given by the remote server. It can even deliver detailed information about the host on boot-up so it can then download instructions specific to the machine type. When the machine boots up for the first time, select the CDROM or floppy and run it; as it boots it can install grub on the hard drive so you never even have to change the BIOS boot order. I recommend having the initrd grub config default to boot from hard disk; you can always manually select the remote install function and it's safer to default to booting from hard disk if the CD or floppy will stay in the machine.
Yes, this is a lot more maintenance than a simple PXE server. This is not intended for use in all environments. But if you need a truly secure, remote-accessible machine kickstarting solution, this one will do the job across all kinds of network types.
P.S. You can substitute ssh for a minimal HTTPS client and a copy of the server's certificate on the initrd so you don't have to rely on CAs. I personally don't trust 3rd party certificate authorities (as more and more evidence shows that states can snoop on SSL traffic without problems).
To get around these problems and provide a secure mechanism for remote install you should use a pre-built linux image on CD-ROM or floppy disk. Most servers today come with one or the other, and both provide enough space to include a kernel and tiny compressed initrd with barebones networking tools.
The kernel should obviously be the newest vanilla kernel possible, patched to include any relevant hardware support. The more vanilla the better as you can quickly pick up the newest released kernel and build it without needing to modify vendor-specific patches.
The initrd should probably be based on an LZMA-compressed mini filesystem or cpio archive. Usually something custom like squashfs works the best. You'll need to build busybox and the dropbear ssh client in a uClibc buildroot environment to make it as small as possible. Bundle an ssh key for the admin server along with DNS and IP information for the server (in case DNS resolution fails, it can try the last known good IP address(es)). It should try indefinitely to get an IP address, and once it gets one it should make an SSH connection. Once SSH connection is established it should download install scripts and execute them. All downloads happen through the secure SSH tunnel.
The initrd should also include proxytunnel and potentially openvpn or another UDP-based SSL tunnel so it can fall back to trying to connect through HTTP or udp port 53. You may need cntlm to work with NTLM proxies as proxytunnel's support does not seem to work for all versions of NTLM. Also, rsync should probably be included, or downloaded immediately once a network connection is established. Big apps can always be downloaded to a tmpfs partition later once the SSH connection is established. The initrd should also use a bootloader that can pass custom arguments to the initrd at boot time, for example to specify proxy or IP settings. It should probably support Web Proxy Autodiscovery Protocol for corporate environments.
If you have room, you may also want to bundle grub with the initrd. This can help you recover a system if it fails and it will allow you to install grub over whatever's currently on the hard drive. A good grub configuration to install would include options to boot from hard drive, CDROM or floppy, so as long as you have remote console you can reboot the box and select the install image from grub at boot time without needing to change BIOS settings.
Finally, each client image should be modified before burning/imaging to have custom ssh keys. You want each boot image to be able to be revoked from the admin server's login list, in case a boot image/disk is compromised or stolen. Granted, you're not giving this thing any more rights than access to your kickstart file tree, and that shouldn't have anything super confidential on it anyway. To customize the ssh keys per image you can have a script which generates new images, creates keys for each one and renames the image file to something specific to the machine. Match up the specific piece of hardware with this unique image file in your network's inventory database.
The kickstart server should have IP addresses dedicated only for the kickstart clients. An HTTP reverse proxy should be installed for ports 80 and 443 so that the client can use proxytunnel to connect to SSH through HTTP proxies. Optionally an openvpn daemon should be enabled on port 53 in case a client's firewall has an open outbound port 53. Each kickstart server's SSH daemon (listening only on the kickstart IPs) should use the same host keys so they can be copied to the initrd and you won't have to worry about the server host keys changing per box, messing up the initial connection from the clients. You could manage all the kickstart host keys independently but why complicate matters further?
In the end what you have is a client install CD or floppy which can boot up on a network, connect securely to a remote server, configure itself and follow setup instructions given by the remote server. It can even deliver detailed information about the host on boot-up so it can then download instructions specific to the machine type. When the machine boots up for the first time, select the CDROM or floppy and run it; as it boots it can install grub on the hard drive so you never even have to change the BIOS boot order. I recommend having the initrd grub config default to boot from hard disk; you can always manually select the remote install function and it's safer to default to booting from hard disk if the CD or floppy will stay in the machine.
Yes, this is a lot more maintenance than a simple PXE server. This is not intended for use in all environments. But if you need a truly secure, remote-accessible machine kickstarting solution, this one will do the job across all kinds of network types.
P.S. You can substitute ssh for a minimal HTTPS client and a copy of the server's certificate on the initrd so you don't have to rely on CAs. I personally don't trust 3rd party certificate authorities (as more and more evidence shows that states can snoop on SSL traffic without problems).
Subscribe to:
Posts (Atom)