[07:22:24] hi, there is 2 scheduled network maintenance periods tomorrow, will that need some datacenter depooling ? [07:23:01] hmmm I only see the zayo one in the calendar [07:23:15] there is zayo and telia [07:23:16] oh tomorrow... sorry E_COFEE [07:23:21] *E_COFFEE ¬¬ [07:24:02] one in whashington and another in california, buy maybe that could affect the redundancy [07:24:33] vgutierrez: can you take care of either checking, or asking some of your colleagues later in the day? [07:25:46] sure, I'll ping XioNoX as soon as he is online [08:31:23] elukey: varnishkafka is still working as expected on cache::misc nodes? [08:34:06] yep nothing on fire from https://grafana.wikimedia.org/dashboard/db/varnishkafka?orgId=1&var-instance=webrequest&var-host=All&from=now-2d&to=now [08:34:09] <3 [08:34:32] I guess I need to try harder to get my "I broke wikipedia" t-shirt [08:34:57] this one would be "I broke analytics" that is far less :D [08:36:18] elukey: you never know.. it could cause a resource starvation on the cp nodes [08:36:52] vk going awol and bringing down all the nodes [08:36:53] interesting [08:36:54] :P [08:37:22] well.. don't jinx it :P [09:35:38] elukey: 24 hours have passed.. let's continue with cache::upload? [09:36:38] vgutierrez: I am ok with that, but maybe after the Grafana maintenance? [09:37:24] elukey: sure :) [09:37:36] elukey: the 30 secs between nodes is ok for upload as well? [09:37:43] or you want something bigger here? [09:41:31] nah should be fine thanks! [09:41:37] great [12:43:52] elukey: so.. let's go with cache::upload? [12:44:43] +1 [12:47:34] nice [13:16:29] elukey: cache::upload nodes updated, looking good from my side [13:19:20] all good from metrics! [13:19:58] <3 [13:58:07] 10Traffic, 10Operations, 10Pybal, 10Patch-For-Review: pybal's "can-depool" logic only takes downServers into account - https://phabricator.wikimedia.org/T184715#4319365 (10Joe) 05Resolved>03Open [14:01:02] 10Traffic, 10Operations, 10Pybal, 10Patch-For-Review: pybal's "can-depool" logic only takes downServers into account - https://phabricator.wikimedia.org/T184715#4319368 (10Joe) Reopened as this is still not fixed, see https://wikitech.wikimedia.org/wiki/Incident_documentation/20180626-LoadBalancers [14:31:14] vgutierrez: we're still going to be good in term of redundancy with those links maintenances [14:39:32] er, need to update all network devices with new NTP servers, logs are flooded with "NTP Server Unreachable" [14:50:00] ah yes, easy to miss heh [14:53:26] tcpdump on a port to see if there is still traffic before shutting it down? :) [14:59:03] I added a line to the decommission checklist https://wikitech.wikimedia.org/wiki/Server_Lifecycle#Reclaim_to_Spares_OR_Decommission [15:34:58] out of context this is great "Arzhel needs a cold potato button" [15:43:37] 10netops, 10Operations, 10fundraising-tech-ops: new pfw policy for monitor server - https://phabricator.wikimedia.org/T198237#4319659 (10ayounsi) a:03cwdent That config adds the two policies `prometheus2_node_exporters` and `prometheus2_misc` after the global `deny_and_log`. They need to be moved before. [16:40:59] XioNoX: sorry about that :( [16:44:41] no worries :) nothing is broken and should be quick to fix with automation [16:45:20] * volans heard automation... in doubt if offering help or running away [16:45:48] volans: you're biased till automation tasks [16:45:56] * vgutierrez hides [17:01:46] volans: E_CANTPARSE :) [17:02:08] eheheh [17:25:07] 10Traffic, 10Commons, 10MediaWiki-File-management, 10Multimedia, and 11 others: Define an official thumb API - https://phabricator.wikimedia.org/T66214#4319891 (10Mholloway) [17:38:10] 10Wikimedia-Apache-configuration, 10Discovery, 10Zero, 10Mobile: m.wikipedia.org and zero.wikipedia.org should redirect how/where - https://phabricator.wikimedia.org/T69015#4319906 (10Mholloway) [18:15:50] 10Traffic, 10Operations, 10RESTBase, 10Patch-For-Review, 10Services (later): Split slash decoding from general percent normalization in Varnish VCL - https://phabricator.wikimedia.org/T127387#4320004 (10Mholloway) [18:19:46] XioNoX: BTW, we should add some checks to avoid this happening again [18:20:21] bblack, XioNoX: meeting topics for tomorrow @ https://etherpad.wikimedia.org/p/Traffic-2018-06-28 [18:23:24] 10netops, 10Operations, 10SRE-Access-Requests: Get Papaul access to network equipment - https://phabricator.wikimedia.org/T198344#4320010 (10faidon) p:05Triage>03Normal [19:53:56] 10netops, 10Operations, 10ops-eqiad: replace mr1-eqiad - https://phabricator.wikimedia.org/T185171#4320260 (10ayounsi) [20:06:52] 10netops, 10Operations, 10fundraising-tech-ops: new pfw policy for monitor server - https://phabricator.wikimedia.org/T198237#4320365 (10cwdent) 05Open>03Resolved works! [21:02:30] 10netops, 10Operations: Rack/cable/configure mr1-eqiad - https://phabricator.wikimedia.org/T187820#4320527 (10ayounsi) [21:02:32] 10netops, 10Operations: Rack/cable/configure mr1-eqiad - https://phabricator.wikimedia.org/T187820#3986943 (10ayounsi) [21:02:35] 10netops, 10Operations, 10ops-eqiad: replace mr1-eqiad - https://phabricator.wikimedia.org/T185171#4320530 (10ayounsi) [21:02:39] 10netops, 10Operations, 10ops-eqiad: replace mr1-eqiad - https://phabricator.wikimedia.org/T185171#3908273 (10ayounsi) [21:04:04] 10netops, 10Operations, 10ops-eqiad: replace mr1-eqiad - https://phabricator.wikimedia.org/T185171#3908273 (10ayounsi) a:05ayounsi>03Cmjohnson Assigning to Chris for the wipe/unrack/decom/etc. [21:06:55] 10netops, 10Operations, 10SRE-Access-Requests: Get Papaul access to network equipment - https://phabricator.wikimedia.org/T198344#4320539 (10ayounsi) a:03ayounsi Taking the task for the actual account creation. [21:12:37] 10netops, 10Operations, 10fundraising-tech-ops: new pfw policy for monitor server - https://phabricator.wikimedia.org/T198237#4320545 (10ayounsi) For the record, the issue was that I was doing a `load merge` instead of a `load replace` [21:50:39] 10Wikimedia-Apache-configuration, 10Discovery, 10Zero, 10Mobile: m.wikipedia.org and zero.wikipedia.org should redirect differently - https://phabricator.wikimedia.org/T69015#4320602 (10Jdforrester-WMF) [21:50:48] 10Wikimedia-Apache-configuration, 10Discovery, 10Zero, 10Mobile: m.wikipedia.org and zero.wikipedia.org should redirect differently - https://phabricator.wikimedia.org/T69015#950330 (10Jdforrester-WMF) AFAICT this looks done?