OutOfMemoryError vs StackOverflow

Key points

• StackOverflowError means one thread ran out of stack. Almost always runaway recursion, occasionally a very deep call chain.
• OutOfMemoryError: Java heap space means the heap is full and even a full GC could not free enough room.
• Raising -Xmx only buys time. If the cause is retention, the bigger heap simply fails a bit later.
• Other flavours name their pot: Metaspace, Direct buffer memory, unable to create native thread.
• Both are an Error, not an Exception. Catching one to carry on leaves the process in a state you cannot trust.
• Always run with -XX:+HeapDumpOnOutOfMemoryError. Without the dump you are just guessing afterwards.

Example

public class Main {

    static int depth = 0;

    static void dig() { depth++; dig(); }

    public static void main(String[] args) {
        try {
            dig();
        } catch (StackOverflowError e) {
            System.out.println("StackOverflowError after " + depth + " frames");
            System.out.println("-> one thread stack filled with call frames, not objects");
            System.out.println("-> fix the recursion, or raise the stack with -Xss1m");
        }
        System.out.println();
        Runtime rt = Runtime.getRuntime();
        System.out.println("the heap is a separate pot: " + (rt.freeMemory() >> 20)
                + " MB free, " + (rt.maxMemory() >> 20) + " MB max");
        System.out.println("OutOfMemoryError means the heap is full and GC cannot free enough.");
        System.out.println("Both are Errors, not Exceptions. Do not plan to recover from them.");
    }
}

Stack overflow is a code bug. Heap exhaustion is usually a retention bug.