If the registry key cannot be created in the Services tree (btw, you didn't write if you're able to create it manually from regedit or not) - and if the registry compacting didn't help (=shrink the hive size), then I really don't know what to suggest (except for reinstalling the OS).
You may try to export the registry (HKEY_LOCAL_MACHINE\System) to a .reg file and then try to find what uses so much space there... but if the hive is corrupted internally, it doesn't have to be included in the exported file. And, even if you find a big subtree and even if it could be deleted - there's no guarantee than the hive could be compacted later.
![Sad :(](https://forum.avast.com/Smileys/default/sad.gif)
Btw, I don't think this problem affects avast! only - I'd expect other programs couldn't be installed as well (at least those installing some services).