We had a few clients complaining about corrupted downloads from our mirror when trying to run upcp CPanel update, so I checked it out.

It looks like the NGINX Proxy Cache had corruption, and it had a file “http://cpproxy.afrixx.com/cpanelsync/11.68.0.26/binaries/linux-c7-x86_64/bin/setsiteip.xz” that was invalid, and corrupted based on the expected checksum.

I’ve double checked the health of the cpproxy, and it seems okay, although it was rebooted recently, I am not sure if I was not the first one to look at the issue.
I can see from the logs, we served him a cached “HIT” to the clients box.

15x.0.165.4 – – [18/Jan/2018:00:52:27 +0200] “GET /cpanelsync/11.68.0.26/binaries/linux-c7-x86_64/bin/setsiteip.xz HTTP/1.1” 200 1026207 “-” “HTTP-Tiny/0.068” “-” “Cache-Status:HIT”
There were 10 attempts from his box, as well as some others from other boxes at the time.

We last “missed” an attempt to get this file at the following date:
access.log-20180116.gz:15x.0.160.155 – – [16/Jan/2018:01:32:08 +0200] “GET /cpanelsync/11.66.0.34/binaries/linux-c7-x86_64/bin/setsiteip.xz HTTP/1.1” 200 2122005 “-” “HTTP-Tiny/0.068” “-” “Cache-Status:MISS”

At this point we could assume it was cached on this MISS.
This means anyone wanting to do an UPCP though the proxy since this date would be sad.

I have taken the exact file the client complained about, and checked if the cache has expired and redownloaded the file.

I downloaded our cache’s file to “CACHE”:
root@alyx:~/cp# wget -O CACHE http://cpproxy.afrixx.com/cpanelsync/11.68.0.26/binaries/linux-c7-x86_64/bin/setsiteip.xz
–2018-01-18 22:42:32– http://cpproxy.afrixx.com/cpanelsync/11.68.0.26/binaries/linux-c7-x86_64/bin/setsiteip.xz
Resolving cpproxy.afrixx.com (cpproxy.afrixx.com)… 1xx.242.144.85
Connecting to cpproxy.afrixx.com (cpproxy.afrixx.com)|1xx.242.144.85|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: unspecified [application/x-xz]
Saving to: ‘CACHE’

CACHE [ <=> ] 1.32M –.-KB/s in 0.07s

2018-01-18 22:42:32 (19.1 MB/s) – ‘CACHE’ saved [1383264]

And I downloaded the CPANEL Mirror’s version to NON-CACHE:

root@alyx:~/cp# wget -O NON-CACHE http://httpupdate.cpanel.net/cpanelsync/11.68.0.26/binaries/linux-c7-x86_64/bin/setsiteip.xz
–2018-01-18 22:42:52– http://httpupdate.cpanel.net/cpanelsync/11.68.0.26/binaries/linux-c7-x86_64/bin/setsiteip.xz
Resolving httpupdate.cpanel.net (httpupdate.cpanel.net)… 67.159.2.2, 67.205.110.4, 208.109.109.239, …
Connecting to httpupdate.cpanel.net (httpupdate.cpanel.net)|67.159.2.2|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: unspecified [application/x-xz]
Saving to: ‘NON-CACHE’

NON-CACHE [ <=> ] 1.32M 871KB/s in 1.6s

2018-01-18 22:42:54 (871 KB/s) – ‘NON-CACHE’ saved [1383264]

When we compare the files md5 checksum, I can see the files are the same:

root@alyx:~/cp# md5sum NON-CACHE
401dbed2bd9075e2896e596738163437 NON-CACHE
root@alyx:~/cp# md5sum CACHE
401dbed2bd9075e2896e596738163437 CACHE

I’ve also run an upcp on a different box that is pointing to our cache, and it seems to be fine.

I’ll assume the cache expired and the file rotated, clearing the error condition.

Mew