"Caveat concerning mutable keys: If a key in an equal?-based hash table is mutated (e.g., a key string is modified with string-set!), then the hash table’s behavior for insertion and lookup operations becomes unpredictable." - http://docs.racket-lang.org/reference/hashtables.html

If you're just interested in what's happening in the current implementation, I have a guess for your table-in-table example: When you initially put h2 in the table, its hash is calculated, and it's filed in a bin of entries that have that hash. Then when you look up h2 again, it calculates a different hash, so it searches the wrong bin and doesn't find the entry. When you look up (table), it finds the right bin, but the only key in that bin is h2. Since h2 is not currently equal? to (table), the search fails again.

I can't reproduce your second example, but what I get is consistent with this explanation:

  arc> h
  #hash(((3 2 . nil) . 3))
  arc> (h '(3 2))
  nil
  arc> (h '(1 2))
  nil

  arc> h
  #hash(((3 2) . 3))
  arc> (h '(3 2))
  nil
  arc> (h '(1 2))
  nil