Question about rsync://us.rsync.archive.ubuntu.com failures

[Seth Lohr]

Apologies, sent a reply to the wrong email subject: https://lists.ubuntu.com/archives/ubuntu-mirrors/2024-June/000596.html

But also yes, 2001:5a8:601:4003:157:131:224:201 is our mirrors server at Sonic.net

-----Original Message-----
From: Seth Lohr <seth.lohr at sonic.com> 
Sent: Wednesday, June 12, 2024 6:34 AM
To: Haw Loeung <haw.loeung at canonical.com>
Cc: ubuntu-mirrors at lists.ubuntu.com
Subject: RE: Question about rsync://us.rsync.archive.ubuntu.com failures

Hello Haw,

To your 2nd point, the last two syncs were successful, so it might've been the bug fix.

To your 1st point, I don't believe we have multiple connections open at any given time.  Running our rsync, I looked at netstat and tcpdump output while the sync was running.  Didn't see more than a couple connections open at a time - and only while we run our sync.

Will report back here if we see more issues, but thanks for the quick response and effort.  Hopefully that fixes it.

-----Original Message-----
From: Haw Loeung <haw.loeung at canonical.com>
Sent: Tuesday, June 11, 2024 4:46 PM
To: Seth Lohr <seth.lohr at sonic.com>
Cc: ubuntu-mirrors at lists.ubuntu.com
Subject: Re: Question about rsync://us.rsync.archive.ubuntu.com failures

On Wed, Jun 12, 2024 at 09:21:35AM +1000, Haw Loeung wrote:
> Hi Seth,
> 
> On Tue, Jun 11, 2024 at 04:03:12PM +0000, Seth Lohr wrote:
> > Hello everyone,
> > 
> > Our company is on the listed mirrors: 
> > https://launchpad.net/ubuntu/+archivemirrors
> > 
> > I looked through the announcement and discussion mailing lists for mirrors, and didn't see anything to my below post - but if I missed something, please let me know.
> > 
> > Starting around 10pm PDT on 2024-06-06, we started receiving errors like the following:
> > 
> > Ubuntu archive mirror operations...
> > Lock established for process 117784.
> > Trying host rsync://us.rsync.archive.ubuntu.com
> > Initiating Ubuntu archive pool sync...
> > rsync: safe_read failed to read 1 bytes [Receiver]: Connection reset 
> > by peer (104) rsync error: error in rsync protocol data stream (code
> > 12) at io.c(276) [Receiver=3.1.2] Archive pool sync failed.
> > 
> > This error has become much more prevalent since around 9am on
> > 2024-06-08
> > 
> > Since then, in the last 13 sync attempts, only one has worked.
> > 
> > I've reviewed our side (IP is 157.131.224.201 for our mirror, FYI), and don't see evidence of anything changing.
> > 
> > Just wondering if I missed an announcement somewhere, or if anyone else is seeing the same behavior.
> > 
> > I can provide a LOT more details about how we're rsycing if needed.
> 
> Would your IP address also be 2001:5a8:601:4003:157:131:224:201? Same 
> netblock owner and all.
> 
> If so, we're seeing this:
> 
> | Jun 10 04:48:02 prod-archive-servers-1 systemd[1]: rsyncd.socket: Too many incoming connections (6) from source 2001:5a8:601:4003:157:131:224:201, dropping connection.
> 
> We have connection limits to only 5 per source (systemd socket 
> MaxConnectionsPerSource=5).
> 

Actually, I think it's due to this bug - https://github.com/systemd/systemd/issues/20685

I've added `CollectMode=inactive-or-failed` so I think that should permanently fix what you were seeing.


Regards,

Haw