ubuntu-mirrors Digest, Vol 133, Issue 2

Seth Lohr seth.lohr at sonic.com
Wed Jun 12 13:33:07 UTC 2024 

Hello Haw,

To your 2nd point, the last two syncs were successful, so it might've been the bug fix.

To your 1st point, I don't believe we have multiple connections open at any given time.  Running our rsync, I looked at netstat and tcpdump output while the sync was running.  Didn't see more than a couple connections open at a time - and only while we run our sync.

Will report back here if we see more issues, but thanks for the quick response and effort.  Hopefully that fixes it.

-----Original Message-----
From: ubuntu-mirrors <ubuntu-mirrors-bounces at lists.ubuntu.com> On Behalf Of ubuntu-mirrors-request at lists.ubuntu.com
Sent: Wednesday, June 12, 2024 5:00 AM
To: ubuntu-mirrors at lists.ubuntu.com
Subject: ubuntu-mirrors Digest, Vol 133, Issue 2

Send ubuntu-mirrors mailing list submissions to
	ubuntu-mirrors at lists.ubuntu.com

To subscribe or unsubscribe via the World Wide Web, visit
	https://lists.ubuntu.com/mailman/listinfo/ubuntu-mirrors
or, via email, send a message with subject or body 'help' to
	ubuntu-mirrors-request at lists.ubuntu.com

You can reach the person managing the list at
	ubuntu-mirrors-owner at lists.ubuntu.com

When replying, please edit your Subject line so it is more specific than "Re: Contents of ubuntu-mirrors digest..."


Today's Topics:

   1. Question about rsync://us.rsync.archive.ubuntu.com failures
      (Seth Lohr)
   2. Re: Question about rsync://us.rsync.archive.ubuntu.com
      failures (Haw Loeung)
   3. Re: Question about rsync://us.rsync.archive.ubuntu.com
      failures (Haw Loeung)


----------------------------------------------------------------------

Message: 1
Date: Tue, 11 Jun 2024 16:03:12 +0000
From: Seth Lohr <seth.lohr at sonic.com>
To: "ubuntu-mirrors at lists.ubuntu.com"
	<ubuntu-mirrors at lists.ubuntu.com>
Subject: Question about rsync://us.rsync.archive.ubuntu.com failures
Message-ID:
	<DS0PR20MB6783533C9DB21D34904C84C6F0C72 at DS0PR20MB6783.namprd20.prod.outlook.com>
	
Content-Type: text/plain; charset="utf-8"

Hello everyone,

Our company is on the listed mirrors: https://launchpad.net/ubuntu/+archivemirrors

I looked through the announcement and discussion mailing lists for mirrors, and didn't see anything to my below post - but if I missed something, please let me know.

Starting around 10pm PDT on 2024-06-06, we started receiving errors like the following:

Ubuntu archive mirror operations...
Lock established for process 117784.
Trying host rsync://us.rsync.archive.ubuntu.com
Initiating Ubuntu archive pool sync...
rsync: safe_read failed to read 1 bytes [Receiver]: Connection reset by peer (104) rsync error: error in rsync protocol data stream (code 12) at io.c(276) [Receiver=3.1.2] Archive pool sync failed.

This error has become much more prevalent since around 9am on 2024-06-08

Since then, in the last 13 sync attempts, only one has worked.

I've reviewed our side (IP is 157.131.224.201 for our mirror, FYI), and don't see evidence of anything changing.

Just wondering if I missed an announcement somewhere, or if anyone else is seeing the same behavior.

I can provide a LOT more details about how we're rsycing if needed.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/ubuntu-mirrors/attachments/20240611/7c8eac1e/attachment-0001.html>

------------------------------

Message: 2
Date: Wed, 12 Jun 2024 09:21:35 +1000
From: Haw Loeung <haw.loeung at canonical.com>
To: Seth Lohr <seth.lohr at sonic.com>
Cc: "ubuntu-mirrors at lists.ubuntu.com"
	<ubuntu-mirrors at lists.ubuntu.com>
Subject: Re: Question about rsync://us.rsync.archive.ubuntu.com
	failures
Message-ID: <Zmjb_0HtcSTyCdbF at dharkan.local>
Content-Type: text/plain; charset="utf-8"

Hi Seth,

On Tue, Jun 11, 2024 at 04:03:12PM +0000, Seth Lohr wrote:
> Hello everyone,
> 
> Our company is on the listed mirrors: 
> https://launchpad.net/ubuntu/+archivemirrors
> 
> I looked through the announcement and discussion mailing lists for mirrors, and didn't see anything to my below post - but if I missed something, please let me know.
> 
> Starting around 10pm PDT on 2024-06-06, we started receiving errors like the following:
> 
> Ubuntu archive mirror operations...
> Lock established for process 117784.
> Trying host rsync://us.rsync.archive.ubuntu.com
> Initiating Ubuntu archive pool sync...
> rsync: safe_read failed to read 1 bytes [Receiver]: Connection reset 
> by peer (104) rsync error: error in rsync protocol data stream (code 
> 12) at io.c(276) [Receiver=3.1.2] Archive pool sync failed.
> 
> This error has become much more prevalent since around 9am on 
> 2024-06-08
> 
> Since then, in the last 13 sync attempts, only one has worked.
> 
> I've reviewed our side (IP is 157.131.224.201 for our mirror, FYI), and don't see evidence of anything changing.
> 
> Just wondering if I missed an announcement somewhere, or if anyone else is seeing the same behavior.
> 
> I can provide a LOT more details about how we're rsycing if needed.

Would your IP address also be 2001:5a8:601:4003:157:131:224:201? Same netblock owner and all.

If so, we're seeing this:

| Jun 10 04:48:02 prod-archive-servers-1 systemd[1]: rsyncd.socket: Too many incoming connections (6) from source 2001:5a8:601:4003:157:131:224:201, dropping connection.

We have connection limits to only 5 per source (systemd socket MaxConnectionsPerSource=5).



Regards,

Haw
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 228 bytes
Desc: not available
URL: <https://lists.ubuntu.com/archives/ubuntu-mirrors/attachments/20240612/10785104/attachment-0001.sig>

------------------------------

Message: 3
Date: Wed, 12 Jun 2024 09:45:59 +1000
From: Haw Loeung <haw.loeung at canonical.com>
To: Seth Lohr <seth.lohr at sonic.com>
Cc: "ubuntu-mirrors at lists.ubuntu.com"
	<ubuntu-mirrors at lists.ubuntu.com>
Subject: Re: Question about rsync://us.rsync.archive.ubuntu.com
	failures
Message-ID: <Zmjht9IbLy_E9iNx at dharkan.local>
Content-Type: text/plain; charset="utf-8"

On Wed, Jun 12, 2024 at 09:21:35AM +1000, Haw Loeung wrote:
> Hi Seth,
> 
> On Tue, Jun 11, 2024 at 04:03:12PM +0000, Seth Lohr wrote:
> > Hello everyone,
> > 
> > Our company is on the listed mirrors: 
> > https://launchpad.net/ubuntu/+archivemirrors
> > 
> > I looked through the announcement and discussion mailing lists for mirrors, and didn't see anything to my below post - but if I missed something, please let me know.
> > 
> > Starting around 10pm PDT on 2024-06-06, we started receiving errors like the following:
> > 
> > Ubuntu archive mirror operations...
> > Lock established for process 117784.
> > Trying host rsync://us.rsync.archive.ubuntu.com
> > Initiating Ubuntu archive pool sync...
> > rsync: safe_read failed to read 1 bytes [Receiver]: Connection reset 
> > by peer (104) rsync error: error in rsync protocol data stream (code 
> > 12) at io.c(276) [Receiver=3.1.2] Archive pool sync failed.
> > 
> > This error has become much more prevalent since around 9am on 
> > 2024-06-08
> > 
> > Since then, in the last 13 sync attempts, only one has worked.
> > 
> > I've reviewed our side (IP is 157.131.224.201 for our mirror, FYI), and don't see evidence of anything changing.
> > 
> > Just wondering if I missed an announcement somewhere, or if anyone else is seeing the same behavior.
> > 
> > I can provide a LOT more details about how we're rsycing if needed.
> 
> Would your IP address also be 2001:5a8:601:4003:157:131:224:201? Same 
> netblock owner and all.
> 
> If so, we're seeing this:
> 
> | Jun 10 04:48:02 prod-archive-servers-1 systemd[1]: rsyncd.socket: Too many incoming connections (6) from source 2001:5a8:601:4003:157:131:224:201, dropping connection.
> 
> We have connection limits to only 5 per source (systemd socket 
> MaxConnectionsPerSource=5).
> 

Actually, I think it's due to this bug - https://github.com/systemd/systemd/issues/20685

I've added `CollectMode=inactive-or-failed` so I think that should permanently fix what you were seeing.


Regards,

Haw
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 228 bytes
Desc: not available
URL: <https://lists.ubuntu.com/archives/ubuntu-mirrors/attachments/20240612/f13e9381/attachment-0001.sig>

------------------------------

Subject: Digest Footer

--
ubuntu-mirrors mailing list
ubuntu-mirrors at lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-mirrors


------------------------------

End of ubuntu-mirrors Digest, Vol 133, Issue 2
**********************************************

More information about the ubuntu-mirrors mailing list