[05:11:05] 10Traffic, 10Operations, 10Phabricator, 10serviceops, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Phabricator downtime due to aphlict and websockets (aphlict current disabled) - https://phabricator.wikimedia.org/T238593 (10DannyS712) a:03mmodell [06:45:11] Krinkle: it is, let me know when to proceed [07:25:18] ema: ok, I'm at a conference today but will let you know when I have a break to monitor it, assuming you'd like me standing by [07:25:41] I'll prep a few test cases [07:38:06] Krinkle: ack, thanks [10:18:41] 10Traffic, 10Operations, 10Phabricator, 10serviceops, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Phabricator downtime due to aphlict and websockets (aphlict current disabled) - https://phabricator.wikimedia.org/T238593 (10Aklapper) [10:31:58] 10Traffic, 10Operations, 10Patch-For-Review: Upgrade cache cluster to debian buster - https://phabricator.wikimedia.org/T242093 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts: ` cp4026.ulsfo.wmnet ` The log can be found in `/var/log/wmf-auto-reima... [10:44:06] 10Traffic, 10Operations, 10Patch-For-Review: Upgrade cache cluster to debian buster - https://phabricator.wikimedia.org/T242093 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['cp4026.ulsfo.wmnet'] ` Of which those **FAILED**: ` ['cp4026.ulsfo.wmnet'] ` [10:44:24] 10Traffic, 10Operations, 10Patch-For-Review: Upgrade cache cluster to debian buster - https://phabricator.wikimedia.org/T242093 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts: ` cp4026.ulsfo.wmnet ` The log can be found in `/var/log/wmf-auto-reima... [10:44:27] 10Traffic, 10Operations, 10Patch-For-Review: Upgrade cache cluster to debian buster - https://phabricator.wikimedia.org/T242093 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['cp4026.ulsfo.wmnet'] ` Of which those **FAILED**: ` ['cp4026.ulsfo.wmnet'] ` [10:44:58] 10Traffic, 10Operations, 10Patch-For-Review: Upgrade cache cluster to debian buster - https://phabricator.wikimedia.org/T242093 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts: ` cp4026.ulsfo.wmnet ` The log can be found in `/var/log/wmf-auto-reima... [11:16:50] 10Traffic, 10Operations: Upgrade cache cluster to debian buster - https://phabricator.wikimedia.org/T242093 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['cp4026.ulsfo.wmnet'] ` Of which those **FAILED**: ` ['cp4026.ulsfo.wmnet'] ` [11:16:53] 10Traffic, 10Operations: Upgrade cache cluster to debian buster - https://phabricator.wikimedia.org/T242093 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts: ` cp4026.ulsfo.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/202001231116_vgutie... [11:45:11] 10Traffic, 10Operations: buster installation issues on cache nodes - https://phabricator.wikimedia.org/T243506 (10Vgutierrez) [11:46:00] 10Traffic, 10Operations: buster installation issues on cache nodes - https://phabricator.wikimedia.org/T243506 (10Vgutierrez) p:05Triage→03Normal [11:54:25] 10Traffic, 10Operations: Upgrade cache cluster to debian buster - https://phabricator.wikimedia.org/T242093 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['cp4026.ulsfo.wmnet'] ` Of which those **FAILED**: ` ['cp4026.ulsfo.wmnet'] ` [11:56:16] 10Traffic, 10Operations, 10Patch-For-Review: buster installation issues on cache nodes - https://phabricator.wikimedia.org/T243506 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts: ` cp4026.ulsfo.wmnet ` The log can be found in `/var/log/wmf-auto-re... [13:04:25] 10Traffic, 10Operations: buster installation issues on cache nodes - https://phabricator.wikimedia.org/T243506 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['cp4026.ulsfo.wmnet'] ` Of which those **FAILED**: ` ['cp4026.ulsfo.wmnet'] ` [13:06:32] XioNoX: FYI "It's the Network Partner Portal - see your peering agreement for a URI" (re FB NPP) [13:10:07] 10Traffic, 10Operations: buster installation issues on cache nodes - https://phabricator.wikimedia.org/T243506 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts: ` cp4026.ulsfo.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/202001231309_vgu... [13:19:12] 10Traffic, 10Operations: buster installation issues on cache nodes - https://phabricator.wikimedia.org/T243506 (10Vgutierrez) 05Open→03Resolved a:03Vgutierrez The final culprit were 3 syntax errors on netboot.cfg as part of https://gerrit.wikimedia.org/r/#/q/Id93d599c6ef0efc5caa2d8cccc83773644bd7ec6 as s... [13:19:15] 10Traffic, 10Operations: Upgrade cache cluster to debian buster - https://phabricator.wikimedia.org/T242093 (10Vgutierrez) [13:41:38] 10Traffic, 10Operations: buster installation issues on cache nodes - https://phabricator.wikimedia.org/T243506 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['cp4026.ulsfo.wmnet'] ` and were **ALL** successful. [13:50:20] 10Domains, 10Traffic, 10Design-Research, 10Operations: Register wikipersonas.org and redirect URL - https://phabricator.wikimedia.org/T241944 (10Aklapper) @Dendelele: Can you please reply to the last three comments? Thanks. [13:53:50] 10Traffic, 10Operations, 10Performance Issue: Current performance issues - https://phabricator.wikimedia.org/T242228 (10Aklapper) >>! In T242228#5786329, @Joe wrote: > An incident report will be published later on wikitech at https://wikitech.wikimedia.org/wiki/Incident_documentation This is https://wikitec... [14:47:20] _joe_: can we perform a rename from nginx -> ats-tls one host at a time? [14:47:57] https://github.com/wikimedia/puppet/blob/production/hieradata/common/service.yaml#L2396-L2397 [14:48:12] it looks like I cannot do that [14:48:37] <_joe_> vgutierrez: you can add the ats-tls service, add the same weights, then switch all the conftool pool [14:49:19] so I need to add both of them to all nodes and then switch the service [14:49:23] ack [14:49:25] <_joe_> yes [14:49:36] <_joe_> see what I did with kubernetes last week [16:25:21] 10Traffic, 10Operations: Move cache upload cluster from nginx to ats-tls - https://phabricator.wikimedia.org/T231433 (10Vgutierrez) [16:25:24] 10Traffic, 10Operations, 10Patch-For-Review: Get rid of nginx puppetization for cache upload - https://phabricator.wikimedia.org/T236120 (10Vgutierrez) 05Open→03Resolved [16:25:38] 10Traffic, 10Operations, 10Patch-For-Review: Remove nginx puppetization for cache text/text_ats - https://phabricator.wikimedia.org/T238625 (10Vgutierrez) 05Open→03Resolved a:03Vgutierrez [16:36:16] 10Traffic, 10Operations, 10Patch-For-Review: Upgrade cache cluster to debian buster - https://phabricator.wikimedia.org/T242093 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts: ` cp4032.ulsfo.wmnet ` The log can be found in `/var/log/wmf-auto-reima... [17:02:44] 10Traffic, 10Operations, 10Patch-For-Review: Upgrade cache cluster to debian buster - https://phabricator.wikimedia.org/T242093 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['cp4032.ulsfo.wmnet'] ` Of which those **FAILED**: ` ['cp4032.ulsfo.wmnet'] ` [17:28:41] 10Traffic, 10Operations: Upgrade cache cluster to debian buster - https://phabricator.wikimedia.org/T242093 (10Vgutierrez) on buster, systemd is not quite happy with trafficserver using /var/run: `Jan 23 17:15:40 cp4032 systemd[1]: /lib/systemd/system/trafficserver.service:8: PIDFile= references path below leg... [18:51:35] 10Traffic, 10Operations, 10ops-eqsin: rack/setup/install ps[12]-60[34]-eqsin - https://phabricator.wikimedia.org/T242250 (10RobH) I've not seen @bblack in IRC since posting the above comment, I suspect due to pre-all-hands-rush. We have SRE meeting time set aside during all hands, so I'll sync up with @bbla... [19:07:41] ema: it you're around still I'll be home in about 30min