The Marantz AVR has a floated ground (two prongs IEC), it doesn't matter to what socket is connected as it has no ground connection to the mains plug. The Atom has a ground switch on the back panel, it can be set either to floating or default (grounded). What position is yours set? No guarantee, but what you can try is to connect one end of a wire to the signal ground connection on the back panel of the AVR and the other end to the A6 chassis. Loosen a screw and connect to wire to it. It might be that the screws are painted, in this case sanding the screw thread should help. I had a similar problem with the connection of the subwoofer to the AVR. The wiring made the hum disappear.
Exactly, so I just connected the Signal Ground of the AVR to one of the screws of the A6. No effect unfortunately. The hum is gone as soon as I change the output on the A6 e.g. to HDMI, XLR, or after the "click". The Naim Atom interestingly is almost silent when I select the floating ground, and makes slightly more noise compared with the default setting. In any case, compared to the A6, the AVR is practically silent when connected with the AVR at the same volume levels.
I have my a6 connected to the AVR only with HDMI. If I have some time I will check what the RCA will do. Almost all AVR have a floating ground but it seems they are not wired the same way. Before the Denon I have now, I had an Anthem that it was absolutely silent. When I passed to Denon I started haveing hum problems with the subwoofer and my old T+A stereo amplifier to which I connected the front pre-out of the AVR. Wiring with the Denon signal ground helped in my case.
Yep that makes sense. For me though, the shortest way out seems to go for the XLR output and find (ideally a balun or) an XLR to RCA cable for connecting with the AVR.
I don't think the adapter will solve anything. It will route the hot to the RCA center pin, the cold to RCA ground and will either route the shield to the RCA ground too or float it. In fact you will have nothing of the XLR advantage.
I connected the RCA of my A6 to the the Denon AVR, which ground is floated too, no hum. You said that when floating the Atom ground, this is humming too, although to a lesser degree. Are you using the same power cord as for the A6?