[00:55:01] <wikibugs>	 10Traffic, 10Operations, 10Performance-Team: Determine cause of upload.wikimedia.org requests routed to text-lb (404 Not Found) - https://phabricator.wikimedia.org/T207340 (10Krinkle)
[07:00:23] <wikibugs>	 10Traffic, 10Operations, 10ops-eqsin: Degraded RAID on cp5010 - https://phabricator.wikimedia.org/T214274 (10Vgutierrez) initial failure at 01:39: ` vgutierrez@cp5010:~$ grep sdb /var/log/kern.log |grep -v "__ext4_get_inode_loc" |grep -v "IO failure" Jan 21 01:39:17 cp5010 kernel: [7472180.491194] blk_update...
[08:51:32] <vgutierrez>	 another one got renewed during the weekend as expected
[08:51:39] <vgutierrez>	 willikins:~ vgutierrez$ echo | openssl s_client -connect netbox.wikimedia.org:443 -servername netbox.wikimedia.org 2>/dev/null | openssl x509 -noout -dates
[08:51:39] <vgutierrez>	 notBefore=Jan 19 15:00:22 2019 GMT
[10:54:12] <elukey>	 vgutierrez: o/
[11:13:47] <elukey>	 I am re-enabling puppet fleetwide, after a test on cp5001 I got
[11:13:47] <elukey>	 Notice: /Stage[main]/Cacheproxy::Instance_pair/Varnish::Instance[upload-backend]/Systemd::Service[varnish-hospital]/Service[varnish-hospital]/ensure: ensure changed 'stopped' to 'running'
[11:14:02] <elukey>	 that seems ok but if anybody could triple check that would be good :)
[11:15:21] <vgutierrez>	 thx elukey 
[11:15:27] * vgutierrez checking
[11:16:31] <elukey>	 vgutierrez: if you have a minute to help me re-enabling puppet on the cp nodes (since they are very delicate) I'd be really glad
[11:16:35] <elukey>	 just to avoid surprises
[11:16:58] <vgutierrez>	 sure
[11:17:57] <vgutierrez>	 elukey: what do you want me to do? :)
[11:18:08] <vgutierrez>	 or what's the current status?
[11:18:57] <elukey>	 vgutierrez: I am slowly re-enabling puppet everywhere after a change to the admin module (no-op), the cp nodes are still disabled
[11:19:18] <vgutierrez>	 besides cp5001, right?
[11:19:24] <elukey>	 exacrlt
[11:19:34] <elukey>	 and cp1008
[11:19:39] <vgutierrez>	 ack
[11:19:48] <vgutierrez>	 our lovely pink unicorn is doing fine :)
[11:19:53] <elukey>	 :)
[11:19:55] <elukey>	 thanks a lot
[11:31:36] <vgutierrez>	 looking good on every site (1 host each)
[11:32:40] <elukey>	 vgutierrez: super
[11:45:32] <elukey>	 vgutierrez: going to re-enable puppet on all cp* then
[11:46:19] <vgutierrez>	 ack
[11:46:27] <vgutierrez>	 feel free to skip cp5007-5012
[11:46:42] <vgutierrez>	 I've a cumin batch running on them :)
[11:47:09] <elukey>	 :)
[16:15:47] <wikibugs>	 10Traffic, 10Wikimedia-Apache-configuration, 10Operations, 10Toolforge: Add new Tool Labs IPs to Varnish rate limit whitelist - https://phabricator.wikimedia.org/T214313 (10Nemo_bis)
[16:17:35] <wikibugs>	 10Traffic, 10Wikimedia-Apache-configuration, 10Operations, 10Toolforge: Add new Tool Labs IPs to Varnish rate limit whitelist - https://phabricator.wikimedia.org/T214313 (10Nemo_bis) I created this more specific task for Tools as requested, but there is a (more general?) Labs task at T213475
[16:21:22] <wikibugs>	 10Traffic, 10Cloud-VPS, 10Operations, 10serviceops: Difficulties to create offline version of Wikipedia because of HTTP 429 response - https://phabricator.wikimedia.org/T213475 (10akosiaris) >>! In T213475#4883423, @Kelson wrote: > I'm not sure to fully understand the technical explanation. Is the problem...
[16:26:35] <wikibugs>	 10Traffic, 10Wikimedia-Apache-configuration, 10Operations, 10Toolforge: Add new Tool Labs IPs to Varnish rate limit whitelist - https://phabricator.wikimedia.org/T214313 (10Nemo_bis) p:05Triage→03High
[16:45:55] <wikibugs>	 10Traffic, 10Wikimedia-Apache-configuration, 10Operations, 10Toolforge: Add new Tool Labs IPs to Varnish rate limit whitelist - https://phabricator.wikimedia.org/T214313 (10Krenair) Tools cannot be done separately, it does not have an IP space of it's own, tools instances are scattered around the same netw...
[20:07:38] <wikibugs>	 10Traffic, 10Wikimedia-Apache-configuration, 10Operations, 10Toolforge: Add new Tool Labs IPs to Varnish rate limit whitelist - https://phabricator.wikimedia.org/T214313 (10faidon) Per our earlier conversations (T208986, T174596, T209011), I think we should just use the WMCS public IP space to make these k...
[20:10:17] <wikibugs>	 10Traffic, 10Wikimedia-Apache-configuration, 10Operations, 10Toolforge: Add new Tool Labs IPs to Varnish rate limit whitelist - https://phabricator.wikimedia.org/T214313 (10Cyberpower678) >>! In T214313#4897303, @faidon wrote: > Per our earlier conversations (T208986, T174596, T209011), I think we should j...
[20:12:34] <wikibugs>	 10Traffic, 10Cloud-VPS, 10Operations, 10serviceops: Wikimedia varnish rules no longer exempt all Cloud VPS/Toolforge IPs from rate limits (HTTP 429 response) - https://phabricator.wikimedia.org/T213475 (10bd808)
[20:12:58] <wikibugs>	 10Traffic, 10Cloud-VPS, 10Operations, 10serviceops: Wikimedia varnish rules no longer exempt all Cloud VPS/Toolforge IPs from rate limits (HTTP 429 response) - https://phabricator.wikimedia.org/T213475 (10bd808)
[20:23:42] <wikibugs>	 10Traffic, 10Cloud-VPS, 10Operations, 10serviceops: Wikimedia varnish rules no longer exempt all Cloud VPS/Toolforge IPs from rate limits (HTTP 429 response) - https://phabricator.wikimedia.org/T213475 (10Cyberpower678) @bd808 just invited me here.  Ever since the Cloud VPS migration, Cyberbot has been hit...
[20:25:16] <wikibugs>	 10Traffic, 10Cloud-VPS, 10Operations, 10serviceops: Wikimedia varnish rules no longer exempt all Cloud VPS/Toolforge IPs from rate limits (HTTP 429 response) - https://phabricator.wikimedia.org/T213475 (10Cyberpower678) p:05Normal→03High I'm also boldly raising the priority as from what I gather I'm li...