Addendum: I was able to resolve the main problem by cleaning up the OS users from the previous installation and repeating the setup on a system with deinstalled application server but residing MaxDB.
How to uninstall the MaxDB without knowledge of the dba and control passwords still remains a mystery for me. But because my main intention (installing a MiniSAP for learning purposes) is fulfilled, the answer isn't so vital any more. Obviously MaxDB can be overwritten and so a workaround is available.
Btw the typo described above seems to be a '!' character in the master password that was accepted by the setup but doesn't conform to the MaxDB guidelines. Only an assumption, but now on the second approach with only another password the logon works ...
Best regards
Rainer