Captive Portal Security part 1

CAPTIVE PORTAL SECURITY, pt. 1

by Peter Willis


INTRODUCTION

This brief paper will focus on the security policies of networks which require payment or web-based authentication before authorization. The topics of discussion will range from common set-up of captive portals, the methods available to circumvent authorization and ways to prevent attacks. A future paper will go over the captive portals themselves, their design and attack considerations.

TABLE OF CONTENTS

I. CAPTIVE PORTAL OVERVIEW
II. ATTACK METHODOLOGY
III. DEFENSE TACTICS
IV. SUMMARY
V. REFERENCES


I. CAPTIVE PORTAL OVERVIEW

What is a captive portal? A captive portal (or CP) is a generic term for any network which presents an interface, usually via a website, to authorize a user to access a network or the internet. Typically these will be in the fashion of a website where one accepts an agreement not to tamper with the network. It can also cover college campus log-in screens, corporate guest account access, for-pay wifi hotspots, etc.

In this paper I will mostly be discussing the "wifi hotspot" form of CP, where either after payment, user authorization or clicking on an "Accept" button will get you access to the internet. Many attacks to circumvent these networks rely on remote dedicated servers and thus won't be suitable to bypass all forms of CPs (as some networks are intended to grant access to internal-only networks).


II. ATTACK METHODOLOGY

There are many different attacks one can perform to circumvent a CP, ranging from the simplistic to complex. Each has its benefits and weaknesses, relating to performance, reliability and possibility of detection by IDS. I will briefly outline the methods here with more examples/detail given during the paper.


1. Tunnel over DNS

This method involves tunneling network and transport-level packets encoded over DNS records. The reason this attack works is due to the design of the DNS system. A domain or sub-domain can delegate where the answers for their DNS records come from using an NS record. The NS record points to your 3rd-party server. The useful part for bypassing CPs comes in here because most caching nameservers (like those on a CP) must forward on requests to your server if they want to be answered properly, which means we can use the CP as a sort of DNS proxy. Set up a custom nameserver that can pack and unpack packets and a client to transmit them and you have yourself a two-way connection with a server on the internet. Make an SSH connection through it and you have a secure tunnel.

One tool used to exploit this property of DNS is iodine[1]. This tool has enhanced security and methods to auto-detect the type of network it passes through and get the best connection possible. Unfortunately it doesn't work as well in the real world as it does on paper.

The original implementation, OzymanDNS by Daniel Kaminsky, is (once modified) actually the most reliably working solution, even if it works as a presentation layer instead of tunneling raw IP packets. This actually gives a couple benefits in that it can be run by an unpriviledged user (on the client side) and is modular/easy to hack on (written in perl). I have modified a version of this code[2] to optimize it and reduce CPU use, but it's not pretty: an average of about 7 kilobits per second is what you'll see. But if you really need brief access it works just about anywhere.


2. Tunnel over ICMP

Surprisingly difficult to exploit in the real world, this protocol offers an alternative tunnel through a firewall for those that don't block it. Networks use ICMP for all kinds of things, most notably passing errors and routing information when necessary as well as the ubiquitous 'ping' messages. If you can't ping there may still be alternative methods to tunnel data through this protocol, but in general the ping is the simplest.

The attack works like this: send an ICMP message (which being layer 3 is not always restricted the way IP sub-protocols are) with an IP or other packet encoded in its payload similar to tunneling over DNS. The benfit of this method is (when it works) it can provide a highly reliable connection without excessive lag (around 24 kilobits per second bandwidth on average). The tool icmptx[3] works well.

A bigger problem looms with IPv6 though, as there is a newer ICMP (icmpv6) which imposes restrictions on the number of icmp packets per second - around 6 per second. This seriously hampers the performance of any application trying to tunnel data through the protocol, but luckily nobody uses IPv6 yet. In general most CPs seem to block ICMP packets.


3. Firewall pinholes

Seldom probed by anyone but the most desperate CP hackers, most firewalls do allow one or more tcp or udp ports outbound to remote hosts (usually on the internet). These will allow you to tunnel an arbitrary protocol as long as it works over the transport protocol of the pinhole.

My favorite of these is tunneling OpenVPN tunnels over udp port 53 - the DNS port. Some CPs allow port 53 outbound to any host on the internet without checking to see if it's passing DNS requests, which is a big mistake. Unfortunately some CPs intercept and rewrite any traffic going over port 53 to provide their own custom DNS responses which breaks this method.

Sometimes strange high-numbered ports are open to the remote host. The simple way to tell this is to run a tcpdump or custom application on the remote host and scan every port from the CP's network to identify the one that's open. With an automated script this can be accomplished easily and once a hole is found the tunnel will provide full bandwidth to the attacker. For attacking CPs that allow access to internal networks you may be able to craft specific packets which when successfully passed through present a different response, thus enumerating open ports.


4. Transparent proxies

Often times the best holes come from an incorrectly set-up network proxy. Often the ACLs on HTTP proxies are set up only enough to block basic requests and don't handle the whole spectrum of possible HTTP requests. Sometimes you can use the proxy as simply as configuring the CP's gateway IP in your browser's proxy settings. However, sometimes a 3rd party server will help us with the extra mile of getting through their ACLs.

The simplest attack in this case is simply looking for an open proxy on some host on the network. Nmap[4] is bundled with scripts which will quickly detect open proxies. Sometimes one needs to abuse the HTTP specifications to find a hole in a particular implementation of an HTTP proxy. Some ACLs can be bypassed by merely changing CRLF to LF in your request or using a different HTTP method. Some authentication/authorization software even have rules in place to allow you to bypass authorization by adding a "?.jpg", "?.css", "?.xml" or other extention to your request. Sometimes header injection can be used to provide a quick jump through a backend proxy.

However, this may only give you HTTP or HTTPS access through the CP. To pass arbitrary data through the proxy it must allow you to use the CONNECT method to reach a remote host. Usually this is allowed because the SSL port (tcp port 443) needs to transmit encrypted packets which are only passed through a proxy with the CONNECT method. If a CONNECT is supported we can take advantage of a 3rd-party server on the internet to tunnel arbitrary packets over HTTP.

An HTTP proxy is set up on the internet which passes connections on to an internal SSH server on port 22. A client then uses proxytunnel[5] to connect to the CP's gateway and from there issues a CONNECT to the 3rd-party proxy, usually on port 443 (but port 80 is a good alternative to have set up). The final request is made to CONNECT to that machine's ssh port. If all goes well you should be able to use this new connection with your SSH client and forward arbitrary data over it. All the CP sees is HTTP or HTTPS traffic being forwarded from one proxy to another. The end result is a high-bandwidth tunnel through the proxy.

A tool[6] included with this paper will allow you to quickly probe ports found on a CP gateway for common problems in HTTP implementations and look for a way out.


5. MAC spoofing

The last method is the only one which is present in all wireless and some wired CPs by design. Once a host is authorized by the CP, its mac and IP address are allowed unrestricted access. All one needs to do is sniff traffic on the network, find a host that is authorized, and spoof its IP and mac address. Spoofing a mac is dependent on your network card and driver but most modern network devices today support it. The downside of course is you have to observe someone already authenticated, but in places such as a crowded airport lobby this may be less difficult than it seems.


6. Miscellaneous

Of course there are many more methods to circumvent CP access with varying degrees of success. Other methods include use of fuzzers, creative source routing/probing, the abuse of cisco protocols for proxy clustering, and the abuse of "convenience" features of some authentication servers.


III. DEFENSE TACTICS


I can't tell you how to perform these attacks without at least trying to tell you how to fix your own broken networks, so here's the white-hat portion of the paper.

First of all, block all unauthenticated traffic (of any layer) destined for the internet. There's no reason for any client to be able to pass any packets without being authed, so just put that rule at the top of your firewall. That'll stop firewall pinhole attacks and IP over ICMP.

To stop IP over DNS, simply have all DNS requests from unauthed sources return the same record every time... The IP of your CP gateway. Not only will this work fine with your transparent HTTP proxy (your proxy should be sending a redirect to your gateway's website anyway for authentication purposes), you can also run many service-specific proxies that will handle requests and inform the client to auth first via http. SSL won't work, but screw them for trying to be secure.

To stop HTTP transparent proxy abuse, pick a very secure and well-established proxy server that's stable and has all security patches and bug fixes up to date. Make sure you craft your ACLs to explicitly stop ALL requests coming from an unauthed host. It should only return the redirect to your authentication website. If for example the server differentiates between requests which are terminated by LF or CRLF, you should probaly write a lot more rules than less and cover all the possibilities.

For other methods of abuse, make sure your CP solution has no unneeded services or ports available on the network. These can be easy targets for attack (who here has checked if snmp is enabled on their gateway? ssh?) and the less attack vectors the less likely an attacker will get through. Audit all your systems to ensure they don't accidentally allow more access than is necessary.


IV. SUMMARY


Almost all CPs today can be bypassed in one way or another. Clearly there's a certain amount of risk vs reward where a company just doesn't care if the top 1% of users can gain access without authorization. That's good for us hackers, but can be bad for the CP provider.

Did you know that since its inception, Starbucks and McDonald's for-pay wifi had open squid proxies on every default gateway? All you needed to do was run an nmap scan to find it, set it in your browser's proxy list and surf for free. (Oh, and the payment network McDonald's uses to process credit cards uses the same network) Did you know there's a major mobile voice and data service provider which has not one but THREE holes in its service, allowing anyone to use their service for free anywhere in the USA with 3G coverage?

Even if Bradley Manning was not able to get a CDROM of US embassy cables out of a secured facility, he might have been able to find a way though network firewalls using these same techniques and tunnel the data out. It's not that far-fetched: as corporations increasingly lock the internet away from their employees, industrious hackers find new more challenging methods to circumvent their restrictions and get the access they want. With this in mind, those implementing the controls should be wary of the many methods of attack involved and the potential for abuse.

This paper has not covered the possibility of attacking the web applications which authenticate users trying to gain access to the network. I am not a web application pen tester so i'll let someone else review those holes.


V. REFERENCES

1. iodine - http://code.kryo.se/iodine/

2. dnstunnel - http://psydev.syw4e.info/new/dnstunnel/

3. icmptx - http://thomer.com/icmptx/

4. nmap - http://nmap.org/

5. proxytunnel - http://proxytunnel.sourceforge.net/

6. quickprobeportal.pl - http://psydev.syw4e.info/new/misc/quickprobeportal.pl

holy ipv6, batman

i just realized my server is passing ipv6 traffic through ssh for my clients. i enabled ipv6 on a windows laptop (netsh interface ipv6 install, not ipv6 install) and told putty to connect to an ipv4 address, tunneling a dynamic socks proxy on both ipv4 and ipv6 to my remote server (which has an average ipv4 and ipv6 network with 6to4 set up for whatever my isp's 6to4 gateway is). then set up my browser to use the dynamic forward port as its proxy, and hit 'ipv6.google.com'.

BOOM. page comes up. ipv6.google.com<->my6to4box<->putty<->windows<->browser. it just friggin works! http://test-ipv6.com/ says it is indeed the ipv6 addy of my6to4box that it's seeing, so ipv4 and ipv6 really are being tunneled automatically. this is pretty cool.

NOW WHY ISN'T EVERYONE DOING THIS YET?!

(what's funny is i only tested this today because xkcd kindly informed its readership that they finally fucking set up an AAAA record which i'd been complaining about for over a year)

how i browse the web

if i were really smart i'd set up SELinux and use sandbox -X to create a secure sandbox for my Firefox and other networked applications. i still want to get around to this one day, but it seems too complicated for me to learn how to write all the rules necessary in 30 minutes.

instead i use a combination of tunneling and secure filtering to lock down my browsing session. first of all, all my traffic goes through an ssh SOCKS tunnel to a VPS i pay for (these can be as little as $4 with a 100GB or more bandwidth cap, so more than enough for general browsing needs). this immediately solves the "starbucks sniffer" problem, and thus my only worry left is the traffic from my VPS to the websites i'm connecting to. this works everywhere i have internet access, using my server-side HTTP-to-SSH proxy (example here and here) and proxytunnel (i need to check out corkscrew though).

for those connections and the content delivered from them i have an assortment of Firefox plugins. my current list of Firefox plugins are: Adblock Plus, Cert Viewer Plus, Certificate Patrol, Expiry Canary, facebooksecurelogin, Force-TLS, HTTPS-Everywhere, NoScript, Safe, SSL Blacklist, and WOT.

the end result? i have a lot more insight as to what is going on behind the scenes as i browse the web. every time an old SSL certificate is replaced with a new one, i get a notification with a diff of the changes. when a site's certificate is about to expire i am notified, thus i'll have advance warning if a site could be potentially exploited or unavailable in the future. all connections to frequently-visited sites such as Wikipedia, Facebook, Google, PayPal, Twitter are forced to use SSL. if i connect to an HTTPS page, the border around my browser window is changed to green, certifying the whole page indeed is using SSL. if there is an element in the page which does not use HTTPS, the border is red. if i submit a form at any time and it doesn't go to an HTTPS url, i am warned before i can press submit. if any certificate uses MD5, i am warned. and when browsing google and other websites i am warned if the site has a low or bad rating, has been reported as a malware site, etc (and it's usually right on the money). of course with NoScript any site i don't explicitly trust can't load any potentially-malicious JavaScript, XSS attacks are prevented, and i can even force all cookies and javascript to use SSL to prevent interception or injection (ala Firesheep).

with all these protections i have much more visibility into whether a site i'm on could potentially have malicious content, and my interactions with these and other sites are inherently more secure. of course most of these plugins are only effective on the most popular sites by default, since complex rules often have to be written to allow specific requests to prevent complicated attacks. but at least we're starting to get more secure by default instead of less.

utf8 terminals

UTF-8 lovin' for my terminals:

(in bash)
LANG=en_US.UTF-8
LC_CTYPE=en_US.UTF-8
(in irssi)
/set recode_autodetect_utf8 ON
/set term_type utf-8
/set term_charset utf-8
(for your terminal)
uxterm -sb -bg black -fg green -g 100x25
(for screen)
screen -U
(for tmux)
tmux -u

my damn fonts keep having a problem with chinese and other languages if i don't use the default font and size. luckily it's barely usable, but still pretty large. more application-specific details here.

safe subversion backup with rsync

Let's say you have a subversion repository and you want to keep a backup on a remote host. Doing a "cp -R" is unsafe as is mentioned here, so your two safe methods of copying a subversion repo are 'svnadmin hotcopy' and 'svnadmin dump'. The former only makes a local copy, but the latter creates a single dumpfile on stdout, which is the most flexible method (though it does not grab repo configs).

The simple method to back up the repo from one host to another would be the following:

• ssh user@remote-host "svnadmin dump REPOSITORY" | svnadmin load REPOSITORY

That would create an identical copy of remote-host's repository on the local host. However, for big subversion repositories this could take a lot of time and bandwidth. Here is the form to use disk space to save time and bandwidth:

• ssh user@remote-host "svnadmin dump REPOSITORY > REPO.dump" && rsync -azP user@remote-host:REPO.dump REPO.dump && svnadmin load REPOSITORY < REPO.dump

This makes a dump on the remote host and rsync's the difference of the dump file to the local host. Note that the dump file is uncompressed and rather large, so if you have lots of spare cycles you can pipe the output of 'svnadmin dump' into 'lzma -c' and the opposite for 'svnadmin load'. (The rsync '-z' flag uses gzip compression, but lzma will save you much more space and thus possibly more time)

edit lol, i just realized compressing before rsync is probably pointless, other than reducing the size on local disk. also 'svnadmin hotcopy' is probably the exact same if not better than dumping to a local file vs piping from one host to another (and saves the config).

screw you, facebook

I use facebook. It's handy. The interface is very clean and usually pretty responsive, and I like the way it presents a stream of online social interaction between people. But what I don't like is their "tactics" to deal with people who try to interface with the site.

Recently I wanted to try and scape my own facebook profile for my friends' information so I could aggregate it and view it on my own terms. If your facebook was suddenly gone, wouldn't you want links back to your previous friends or contact information to get ahold of them again? A link to one such perl script here shows the typical response of Facebook to anyone who writes such a script. It was even removed from the Wayback Machine at archive.org, even though the rest of the site from when the script existed is still there.

Another site here details how a developer had his profile disabled due to trying to export his friends' contact information for an app he develops. It's not unusual to see an application log into a service and extract e-mail addresses so you can find the same people on a different service. However, getting your account dropped because of it is annoying to say the least.

At least there's some content out there that hasn't been taken down. This blog post shows how to log in and update your status. Hopefully this will get me close to a workable script; i'll have to create a fake profile to avoid my normal one being "deactivated", though.

breaking up with BK & RESTful APIs with curl

This article sparked a mini-firestorm on BrightKite. As a couple of the comments at the end of the story mention, several users were apparently deleted from BrightKite due to their negative reactions to this story. There was also a number of users that were deleted because of some kind of AT&T-related "removal" notifications, but some users who were deleted did not use AT&T. In the end BrightKite fucked up yet again.

I am finally getting around to deleting my BrightKite account. For some reason all of my pixelpipe posts have been failing to Twitter but succeeding on BK, so people have actually been commenting on my BK account recently. I've also been getting some strange "love spam" through BK, which is yet another lesson in why this new BK 2.0 sucks: there is no "report" button for posts or users, so I can't flag something as spam. Wonderful new system, guys.

So anyway, this was a nice quick introduction to working with RESTful APIs with curl. I found one example of how to delete all brightkite posts and decided a bash one-liner would be simpler. This page describes how to specify the method and other REST-related options using curl. Here is the one-liner I ended up using:

curl -s -G -u user:pass http://brightkite.com/people/user/objects.xml | \
 grep "<id>" | sed -e 's/^.*<id>\(.\+\)<\/.*$/\1/g' | \
 xargs -n 1 -I {} curl -s -G -u user:pass -X DELETE http://brightkite.com/objects/{}.xml

Of course, being BrightKite it doesn't work the way it should. Only a small amount of posts get deleted at a time, I don't know why. Intermittently you'll receive a spam of HTML 404 pages, and every single request (authenticated or not) gives a '403 Forbidden' error. No matter; running the one-liner over and over gets them all eventually.

When i'm done i'll post a couple about how BK sucks, and not delete the account outright: I want my opinions of their suckitude to linger for others to discover. It's pretty sad, too: BK actually worked in the beginning and was pretty useful when the few other users down here used it frequently.

http/socks proxies and ssh tunneling

It seems there exists no tool to simply convert one type of tunnel into another. SSH supports both tcp port forwarding and a built-in SOCKS proxy, both of which are incredibly useful. But it lacks a native HTTP proxy. The sad truth is, most applications today only seem to support HTTP proxies or are beginning to support SOCKS. Until such time that SOCKS proxies are universally accepted in networking apps, I need an HTTP proxy for my SSH client.

There are some LD_PRELOAD apps which will overload network operations with their own and send them through a proxy (like ProxyChains). This seems an unportable and more hacky solution than I would use (are you going to change all your desktop links for 'audacious' to be prefixed with 'proxychains'?). I am willing to write a HTTP-to-SOCKS proxy but don't have the time just yet. CPAN seems to have a pure-perl HTTP proxy server, and I can probably leverage IO::Socket::Socks to connect to the SOCKS5 server in ssh.

In any case, my immediate needs are fulfilled: I can use 'curl' to tunnel through ssh's SOCKS for most of my needs. This blog entry on tunneling svn was a useful quick hack to commit my changes through an ssh forwarded port, but much more of a hack than i'm willing to commit to; i'd rather just enable or disable an http proxy.

large scale ganglia

Ganglia is currently the end-all be-all of open source host metric aggregation and reporting. There are a couple other solutions which are slowly emerging to replace it, but nothing as well entrenched. Hate on RRDs as much as you want but they're basically the defacto standard for storing and reporting numeric metrics. It can be hairy trying to figure out how to configure it all though, so here's an overview of getting it set up on your network.

First install rrdtool and all of ganglia tools on every host. Keep the web interface ('web' in the ganglia tarball) off to the side for now. The first thing you should consider is configuring IGMP on your switch to take advantage of multicast groups. If you don't you run the risk of causing broadcast storms which could potentially wreak havoc on your network equipment, causing ignored packets and other anomalies depending on your traffic.

In order to properly manage a large-scale installation you'll need to juggle some configuration management software. You need to configure gmond on each cluster node so that the cluster multicasts on a unique port for that lan (in fact, keep all of your clusters on unique ports for simplicity's sake). If you have a monitoring or admin host on each lan you'll configure gmond to listen for multicast [or unicast] metrics from one or more clusters on that lan. Then you'll need to configure your gmetad's to collect stats from either each individual cluster node or the "collector" gmond nodes on the monitoring/admin hosts. Our network topology is as follows:

Monitor box ->
              DC1 ->
                    DC1.LAN1 ->
                               Cluster 1
                               Cluster 2
                    DC1.LAN2 ->
                               Cluster 3
                    DC1.LAN3 ->
                               Cluster 4
              DC2 ->
                    DC2.LAN1 ->
                               Cluster 1
                               Cluster 2
                               Cluster 3
                               Cluster 4
                    DC2.LAN2 ->
                               Cluster 1

The monitor box runs gmetad and only has two data_source entires: DC1 and DC2.
DC1 and DC2 run gmetad and each has a data_source for each LAN.
Each LAN has its own monitor host running gmond which collects metrics for all clusters on their respective LAN.
The clusters are simply multicasting gmond's configured with a specific cluster name and multicast port running on cluster nodes.
The main monitor box, the DC boxes and the LAN boxes all run apache2+php5 with the same docroot (the ganglia web interface). The configs are set to load gmetad data from localhost on one port.
Each gmetad has its "authority" pointed at its own web server URL.

(Tip: in theory you could run all of that off a single host by making sure all the gmetad's use unique ports and modifying the web interface code to load the config settings based on the requesting URL, to change the gmetad port as necessary)

In the end what you get is a main page which only shows the different DCs as grids. As you click one it loads a new page which shows that DC's LANs as grids. Clicking those will show summaries of that LAN's clusters. This allows you to lay out your clusters across your infrastructure in a well-balanced topology and gives you the benefit of some additional redundancy if one LAN or DC's WAN link goes down.

We used to use a single gmetad host and web interface for all clusters. This makes it extremely easy to query any given piece of data at once from a single gmetad instance and see all the clusters on one web page. The problem was we had too much data. Gmetad could not keep up with the load and the box was crushed by disk IO. We lessened this by moving to a tmpfs mount, archiving RRDs and pruning any older than 60 days. Spreading out to multiple hosts also lessened the need for additional RAM and lowered network use and latency across long-distance links.

If you think you won't care about your lost historical data, think again. Always archive your RRDs to keep fine-grained details for later planning. Also keep in mind that as your clusters change your RRD directory can fill up with clutter. Hosts which used to be in one cluster and are now in another are not cleaned up by ganglia; they will sit on the disk unused. Only take recently-modified files into account when determining the total size of your RRDs. Also, for big clusters make your poll time a little longer to prevent load from ramping up too often.

I'll add config examples later. There are many guides out there that show how to set up the fine details. As far as I can tell this is the simplest way to lay out multiple grids and lessen the load on an overtaxed gmetad host.

how much memory is my application really using?

In order to properly visualize how much memory is being used by an application, you must understand the basics of how memory is managed by the Virtual Memory Manager.

Memory is allocated by the kernel in units called 'pages'. Memory in userland applications is lumped into two generic categories: pages that have been allocated and pages that have been used. This distinction is based on the idea of 'overcommittal' of memory. Many applications attempt to allocate much more memory for themselves than they ever use. If they attempt to allocate themselves memory and the kernel refuses, the program usually dies (though that's up to the application to handle). In order to prevent this, the kernel allows programs to over-commit themselves to more memory than they could possibly use. The result is programs think they can use a lot of memory when really your system probably doesn't even have that much memory.

The Linux kernel supports tuning of several VM parameters including overcommit. There are 3 basic modes of operation: heuristic, always and never. Heuristic overcommit (the default) allows programs to overcommit memory and determines if the allocation is wildly more than the system contains, and if so won't allow allocation. Always overcommit will pretty much guarantee an allocation. Never overcommit will stricly only allow allocation for the amount of swap + a configurable percentage of real memory (RAM). The benefit with never overcommit is that applications won't simply be killed due to the system running out of free pages; instead it will receive a nice friendly error when trying to allocate memory and *can* decide for itself what to do.

Now, let's say the program has allocated all it needs. The userland app now uses some of the memory. But how much is it using? How much memory is available for other programs? The dirty truth is, this is hard to figure out mostly because the kernel won't give us any easy answers. Most older kernels simply don't have the facility to report what memory is being used so we have to guesstimate. What we want to know is, given one or more processes, how much ram is actually being used versus what is allocated?

With the exception of very new kernels, all you can really tell about the used memory of a program is how much memory is in the physical RAM. This is called the Resident Set Size, or RSS. Since the 2.6.13 kernel we can use the 'smaps' /proc/ file to get a better idea of the RSS use. The RSS is split into two basic groups: shared (pages used by 2 or more processes) and private (pages used by only one process). We know that the memory that a process physically takes up in RAM combines the shared and private RSS, but this is misleading: the shared memory is used by many programs, so if we were to count this more than once it would falsely inflate the amount of used memory. The solution is to take all the programs whose memory you want to measure and count all the private pages they use, and then (optionally) count the shared pages too. A catch here is that there may be many other programs using the same shared pages, so this can only be considered a 'good idea' about the total memory your application is using (it may be an over-estimation).

With recent kernels some extra stats have been added to the kernel which can aid in giving a good idea about the amount of memory used. Pss is a value that can be found in smaps with kernels greater than 2.6.24. It contains the private RSS as well as the amount of shared memory divided by the number of processes that are using it, which (when added with other processes also sharing the same memory) gives you a closer idea to the amount of memory really being used - but again this is not completely accurate due to the many programs which may all be using different shared memory. In very recent kernels the 'pagemap' file allows a program to examine all the pages allocated and get a very fine-tuned look at what is allocated by what. This is also useful for determining what is in swap, which otherwise would be impossible to find out.

Based on all this information, one thing should be clear: without a modern kernel, you will never really know how much memory your processes are using! However, guesstimation can be very accurate and so we will try our best with what we have. I have created a perl script which can give you a rough idea of memory usage based on smaps information. You can pass it a program name and it will try to summarize the memory of all processes based on that program name. Alternatively you can pass it pid numbers and it will give stats on each pid and a summary. For example, to check the memory used by all apache processes, run 'meminfo.pl httpd'. To check the memory of all processes on the system, run 'ps ax | awk '{print $1}' | xargs meminfo.pl'.

Some guidelines for looking at memory usage:

* Ignore the free memory, buffers and cache settings. 99% of the time this will not apply to you and it is misleading. The buffer and cache may be reclaimed at any time if the system is running out of resources, and free memory is completely pointless - this relates to pages of physical memory not yet allocated, and realistically your RAM should be mostly allocated most of the time ( for buffers, cache, etc as well as miscellaneous kernel and userland memory ).

* If you don't have Pss or Pagemap in your kernel, a rough guess of used memory can be had by either adding up all the Private RSS of every process or subtracting the Shared RSS from the RSS total for every process. This still doesn't account for kernel memory which is actually in use and other factors but it's a good start.

* Do not make the mistake of confusing swap use with 'running out of memory'. Swap is always used by the kernel even if you have tons of free memory. The kernel tries to intelligently move inactive memory to swap to keep a balance between responsiveness and speed of memory allocation/buffer+cache reclimation. Basically, it's better to have lots of memory you don't use in swap than in physical RAM. You can tune your VM to swap less or more but it depends on your application.

some tricks to stay hidden

[1] If you're in a shell and typing some commands which you don't want to be preserved in your .bash_history, just run "kill -9 $$" when you're done with your shell. The parent process (shell) will be killed and not write anything to the .bash_history file.

[2] You want to log on to a box and use it interactively but don't feel like being monitored in the wtmp, use "ssh -v -T user@host" to log in. The verbose will tell you once you've successfully logged in. Now run 'last | head -n 5'. Your current login should not show up in the list. (Your login will probably still be logged by sshd in a separate file, but it's less noticeable)

[3] Encrypt your shell scripts: http://blogs.koolwal.net/2009/01/20/howto-encrypting-a-shell-script-on-a-linux-or-unix-based-system/
[3.1] Also use my script: http://themes.freshmeat.net/projects/encsh

[4] Encrypt all the files in a directory with cryptdir(1).