[00:00:23] <paladox>	 Wow
[00:04:16] <shinken-wm_>	 RECOVERY - Host deployment-xhgui02 is UP: PING OK - Packet loss = 0%, RTA = 2.25 ms
[00:06:26] <wikibugs>	 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Release Pipeline (Blubber): Automate Deployment of Blubber - https://phabricator.wikimedia.org/T253264 (10jeena) 05Open→03Resolved
[00:14:13] <shinken-wm_>	 PROBLEM - Host deployment-xhgui02 is DOWN: CRITICAL - Host Unreachable (172.16.1.202)
[06:06:34] <wikibugs>	 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10MediaWiki-extensions-InterwikiSorting, 10Patch-For-Review, 10Technical-Debt: Undeploy the InterwikiSorting extension from Wikipedia production - https://phabricator.wikimedia.org/T253764 (10DannyS712)
[06:33:11] <wikibugs>	 10Phabricator, 10itWikiCon-2020: Create a countdown for itWikiCon 2020 call for papers - https://phabricator.wikimedia.org/T255668 (10valerio.bozzolan) Thank you @Aklapper. I also wonder if we could just [[ https://en.wikipedia.org/wiki/Wikipedia:Be_bold | be bold ]] and set the application visibility to //Pub...
[06:44:12] <shinken-wm_>	 RECOVERY - Host deployment-xhgui02 is UP: PING OK - Packet loss = 0%, RTA = 1.64 ms
[06:49:15] <shinken-wm_>	 PROBLEM - Host deployment-xhgui02 is DOWN: CRITICAL - Host Unreachable (172.16.1.202)
[07:15:16] <wikibugs>	 (03CR) 10Hashar: "From a quick online doc, it seems "go get" always retrieve the HEAD of the repository and does not have a way to retrieve a specific tag. " [integration/config] - 10https://gerrit.wikimedia.org/r/610292 (https://phabricator.wikimedia.org/T257456) (owner: 10Hashar)
[07:27:02] <wikibugs>	 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO, 10Operations, 10Release Pipeline, and 2 others: Migrate production services to kubernetes using the pipeline - https://phabricator.wikimedia.org/T198901 (10akosiaris)
[07:32:25] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] Add CI for mediawiki/extensions/FlexDiagrams [integration/config] - 10https://gerrit.wikimedia.org/r/610730 (owner: 10Yaron Koren)
[07:33:33] <wikibugs>	 (03Merged) 10jenkins-bot: Add CI for mediawiki/extensions/FlexDiagrams [integration/config] - 10https://gerrit.wikimedia.org/r/610730 (owner: 10Yaron Koren)
[07:38:16] <wikibugs>	 (03PS1) 10Hashar: dockerfiles: drop backports, add git proto v2 to ci-stretch [integration/config] - 10https://gerrit.wikimedia.org/r/611181 (https://phabricator.wikimedia.org/T256844)
[07:38:38] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] dockerfiles: drop backports, add git proto v2 to ci-stretch [integration/config] - 10https://gerrit.wikimedia.org/r/611181 (https://phabricator.wikimedia.org/T256844) (owner: 10Hashar)
[07:40:34] <wikibugs>	 (03CR) 10Hashar: "Part of the ongoing effort to sunset stretch-backports ( T256877 )." [integration/config] - 10https://gerrit.wikimedia.org/r/611181 (https://phabricator.wikimedia.org/T256844) (owner: 10Hashar)
[08:09:16] <shinken-wm_>	 RECOVERY - Host deployment-xhgui02 is UP: PING OK - Packet loss = 0%, RTA = 1.34 ms
[08:14:14] <shinken-wm_>	 PROBLEM - Host deployment-xhgui02 is DOWN: CRITICAL - Host Unreachable (172.16.1.202)
[08:49:20] <wikibugs>	 (03CR) 10Lokal Profil: [V: 03+2 C: 03+2] Restore access settings to default [services/wikispeech/wikispeech-server] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/610735 (https://phabricator.wikimedia.org/T253851) (owner: 10Lokal Profil)
[09:07:57] <wikibugs>	 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Patch-For-Review, 10Release, 10Train Deployments: 1.35.0-wmf.40 deployment blockers - https://phabricator.wikimedia.org/T256668 (10Jdforrester-WMF) 05Open→03Resolved Let's call this one done.
[09:09:51] <wikibugs>	 10Continuous-Integration-Config, 10Gerrit, 10Release-Engineering-Team-TODO, 10Developer Productivity, 10Patch-For-Review: Jenkins comments no longer formatted on Gerrit 3 - https://phabricator.wikimedia.org/T256575 (10hashar) 05Resolved→03Open I definitely love the addition of the table of results un...
[09:09:56] <wikibugs>	 10Gerrit, 10Patch-For-Review: Gerrit 3.2 upgrade - https://phabricator.wikimedia.org/T254158 (10hashar)
[09:41:55] <wikibugs>	 (03CR) 10Vgutierrez: "I'd say we should make acme-chief play well with the master/HEAD version of pebble.. cause those changes at some point will be deployed on" [integration/config] - 10https://gerrit.wikimedia.org/r/610292 (https://phabricator.wikimedia.org/T257456) (owner: 10Hashar)
[09:44:46] <wikibugs>	 (03CR) 10Hashar: "I propose to build this container then add a new Jenkins job that relies on it but make it non voting (it will not vote v-1 on failure). T" [integration/config] - 10https://gerrit.wikimedia.org/r/610292 (https://phabricator.wikimedia.org/T257456) (owner: 10Hashar)
[10:13:29] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] layout: [mediawiki/services/chromium-render/deploy] Archive [integration/config] - 10https://gerrit.wikimedia.org/r/607746 (https://phabricator.wikimedia.org/T217114) (owner: 10Jforrester)
[10:14:50] <wikibugs>	 (03Merged) 10jenkins-bot: layout: [mediawiki/services/chromium-render/deploy] Archive [integration/config] - 10https://gerrit.wikimedia.org/r/607746 (https://phabricator.wikimedia.org/T217114) (owner: 10Jforrester)
[10:15:42] <James_F>	 !log Zuul: [mediawiki/services/chromium-render/deploy] Archive T217114
[10:15:44] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[10:15:45] <stashbot>	 T217114: Migrate Proton to k8s and  nodejs 10 - https://phabricator.wikimedia.org/T217114
[10:15:58] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10JavaScript: Upgrade all CI jobs from node6/npm3 to node10/npm6 across all project... - https://phabricator.wikimedia.org/T211784
[10:16:49] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/607747 (https://phabricator.wikimedia.org/T217114) (owner: 10Jforrester)
[10:18:00] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: Drop chromium-render-deploy-npm-node-6-docker, unused [integration/config] - 10https://gerrit.wikimedia.org/r/607747 (https://phabricator.wikimedia.org/T217114) (owner: 10Jforrester)
[10:21:09] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Core Platform Team: Beta cluster changeprop job queue down since 2020-06-18 - https://phabricator.wikimedia.org/T257620 (10hnowlan) a:03hnowlan
[10:39:45] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Core Platform Team: Beta cluster changeprop job queue down since 2020-06-18 - https://phabricator.wikimedia.org/T257620 (10hnowlan) This is a side effect of the migration to Docker - I think the docker instance either didn't pick up the correct config or needed to be restarted...
[10:46:00] <shinken-wm_>	 PROBLEM - Free space - all mounts on deployment-snapshot01 is CRITICAL: CRITICAL: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found)deployment-prep.deployment-snapshot01.diskspace.root.byte_percentfree (<10.00%)
[10:46:32] <wikibugs>	 (03PS2) 10Jforrester: layout: [pywikibot/core] Drop fasttest-py27 [integration/config] - 10https://gerrit.wikimedia.org/r/610295 (https://phabricator.wikimedia.org/T213287) (owner: 10Xqt)
[10:46:33] <wikibugs>	 (03PS1) 10Jforrester: jjb: Add pywikibot-core-tox-fasttest-py37-docker [integration/config] - 10https://gerrit.wikimedia.org/r/611255 (https://phabricator.wikimedia.org/T213287)
[10:46:35] <wikibugs>	 (03PS1) 10Jforrester: layout: [pywikibot/core] Add fasttest-py37 as experimental [integration/config] - 10https://gerrit.wikimedia.org/r/611256 (https://phabricator.wikimedia.org/T213287)
[10:46:39] <wikibugs>	 (03PS1) 10Jforrester: layout: [pywikibot/core] Add fasttest-py37 as voting [integration/config] - 10https://gerrit.wikimedia.org/r/611257 (https://phabricator.wikimedia.org/T213287)
[10:46:41] <wikibugs>	 (03PS1) 10Jforrester: jjb: Drop pywikibot-core-tox-fasttest-py27-docker, no longer used [integration/config] - 10https://gerrit.wikimedia.org/r/611258 (https://phabricator.wikimedia.org/T213287)
[10:48:52] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/611255 (https://phabricator.wikimedia.org/T213287) (owner: 10Jforrester)
[10:48:58] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] layout: [pywikibot/core] Add fasttest-py37 as experimental [integration/config] - 10https://gerrit.wikimedia.org/r/611256 (https://phabricator.wikimedia.org/T213287) (owner: 10Jforrester)
[10:50:09] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: Add pywikibot-core-tox-fasttest-py37-docker [integration/config] - 10https://gerrit.wikimedia.org/r/611255 (https://phabricator.wikimedia.org/T213287) (owner: 10Jforrester)
[10:50:15] <wikibugs>	 (03Merged) 10jenkins-bot: layout: [pywikibot/core] Add fasttest-py37 as experimental [integration/config] - 10https://gerrit.wikimedia.org/r/611256 (https://phabricator.wikimedia.org/T213287) (owner: 10Jforrester)
[10:52:40] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Gerrit, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), and 2 others: Allow hiding of non-discussion comments in Gerrit - https://phabricator.wikimedia.org/T48148 (...
[10:56:01] <shinken-wm_>	 RECOVERY - Free space - all mounts on deployment-snapshot01 is OK: OK: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found)
[11:00:11] <wikibugs>	 (03PS1) 10Jforrester: parameter_functions: VisualEditor and TemplateData are now tarballed [integration/config] - 10https://gerrit.wikimedia.org/r/611271 (https://phabricator.wikimedia.org/T243258)
[11:00:58] <wikibugs>	 (03CR) 10Jforrester: "Not just yet. :-)" [integration/config] - 10https://gerrit.wikimedia.org/r/611271 (https://phabricator.wikimedia.org/T243258) (owner: 10Jforrester)
[11:03:05] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Gerrit, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), and 2 others: Allow hiding of non-discussion comments in Gerrit - https://phabricator.wikimedia.org/T48148 (...
[11:06:50] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Gerrit, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), and 2 others: Allow hiding of non-discussion comments in Gerrit - https://phabricator.wikimedia.org/T48148 (...
[11:08:40] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Gerrit, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), and 2 others: Allow hiding of non-discussion comments in Gerrit - https://phabricator.wikimedia.org/T48148 (...
[11:10:42] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Gerrit, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), and 2 others: Allow hiding of non-discussion comments in Gerrit - https://phabricator.wikimedia.org/T48148 (...
[11:41:15] <wikibugs>	 (03CR) 10Vgutierrez: "> Patch Set 2:" [integration/config] - 10https://gerrit.wikimedia.org/r/610292 (https://phabricator.wikimedia.org/T257456) (owner: 10Hashar)
[12:38:40] <wikibugs>	 (03PS3) 10Hashar: dockerfiles: use golang from Buster to build pebble [integration/config] - 10https://gerrit.wikimedia.org/r/610292 (https://phabricator.wikimedia.org/T257456)
[12:38:42] <wikibugs>	 (03PS1) 10Hashar: Add extra non voting job for acme-chief [integration/config] - 10https://gerrit.wikimedia.org/r/611303 (https://phabricator.wikimedia.org/T257456)
[12:40:32] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "Thanks :)  The Jenkins job is added in the child change https://gerrit.wikimedia.org/r/c/integration/config/+/611303/1" (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/610292 (https://phabricator.wikimedia.org/T257456) (owner: 10Hashar)
[12:41:34] <wikibugs>	 (03Merged) 10jenkins-bot: dockerfiles: use golang from Buster to build pebble [integration/config] - 10https://gerrit.wikimedia.org/r/610292 (https://phabricator.wikimedia.org/T257456) (owner: 10Hashar)
[13:00:33] <hashar>	 !log Successfully tagged docker-registry.discovery.wmnet/releng/tox-acme-chief:0.5.0 # T257456
[13:00:36] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[13:00:36] <stashbot>	 T257456: Change CI image for acme-chief from Stretch to Buster - https://phabricator.wikimedia.org/T257456
[13:02:00] <wikibugs>	 (03PS2) 10Hashar: Add extra non voting job for acme-chief [integration/config] - 10https://gerrit.wikimedia.org/r/611303 (https://phabricator.wikimedia.org/T257456)
[13:04:19] <wikibugs>	 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))): Build a Seakeeper MVP - https://phabricator.wikimedia.org/T248876 (10LarsWirzenius) 05Open→03Declined
[13:04:40] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))): Create a threat model for new CI - https://phabricator.wikimedia.org/T240679 (10LarsWirzenius) 05Open→03Declined
[13:04:42] <wikibugs>	 10Release-Engineering-Team (Kanban): Consider and evaluate possible new CI tooling - https://phabricator.wikimedia.org/T217325 (10LarsWirzenius)
[13:04:59] <wikibugs>	 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))): Ask Legal if we can sign Argo's CLA - https://phabricator.wikimedia.org/T250152 (10LarsWirzenius) 05Open→03Declined
[13:09:23] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] Add extra non voting job for acme-chief [integration/config] - 10https://gerrit.wikimedia.org/r/611303 (https://phabricator.wikimedia.org/T257456) (owner: 10Hashar)
[13:10:26] <wikibugs>	 (03Merged) 10jenkins-bot: Add extra non voting job for acme-chief [integration/config] - 10https://gerrit.wikimedia.org/r/611303 (https://phabricator.wikimedia.org/T257456) (owner: 10Hashar)
[13:23:48] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Acme-chief, 10Patch-For-Review: Change CI image for acme-chief from Stretch to Buster - https://phabricator.wikimedia.org/T257456 (10hashar) I have deployed...
[13:59:16] <shinken-wm_>	 RECOVERY - Host deployment-xhgui02 is UP: PING OK - Packet loss = 0%, RTA = 0.65 ms
[13:59:52] <wikibugs>	 10Phabricator: Issue uploading screenshots on phab tickets: "Exception: No configured storage engine can store this file." - https://phabricator.wikimedia.org/T257635 (10Aklapper)
[14:03:18] <wikibugs>	 10Phabricator (Upstream), 10Upstream: Feature request: New notification option for when somebody @mentions you in a Task - https://phabricator.wikimedia.org/T257633 (10Aklapper) 05Open→03Declined > Would it be possible for Maniphest Tasks to add a new option for "When somebody @mentions me"?  Unlikely, as...
[14:04:11] <shinken-wm_>	 PROBLEM - Host deployment-xhgui02 is DOWN: CRITICAL - Host Unreachable (172.16.1.202)
[14:14:55] <wikibugs>	 (03CR) 10Hashar: "The logic is now in the quibble repository script "utils/ci-fullrun.sh" and we reuse the existing containers.   The whole serie of patches" [integration/config] - 10https://gerrit.wikimedia.org/r/589438 (https://phabricator.wikimedia.org/T235118) (owner: 10Awight)
[14:14:59] <wikibugs>	 (03Abandoned) 10Hashar: [WIP] Quibble self-test image [integration/config] - 10https://gerrit.wikimedia.org/r/589438 (https://phabricator.wikimedia.org/T235118) (owner: 10Awight)
[14:16:25] <wikibugs>	 (03CR) 10Hashar: [C: 03+1] layout: [pywikibot/core] Drop fasttest-py27 [integration/config] - 10https://gerrit.wikimedia.org/r/610295 (https://phabricator.wikimedia.org/T213287) (owner: 10Xqt)
[14:16:29] <wikibugs>	 (03CR) 10Hashar: [C: 03+1] jjb: Drop pywikibot-core-tox-fasttest-py27-docker, no longer used [integration/config] - 10https://gerrit.wikimedia.org/r/611258 (https://phabricator.wikimedia.org/T213287) (owner: 10Jforrester)
[14:27:53] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] dockerfiles: drop backports, add git proto v2 to ci-stretch [integration/config] - 10https://gerrit.wikimedia.org/r/611181 (https://phabricator.wikimedia.org/T256844) (owner: 10Hashar)
[14:39:19] <wikibugs>	 10Release-Engineering-Team-TODO, 10Wikimedia-Site-requests, 10foundation.wikimedia.org: Enable SUL accounts on Governance wiki - https://phabricator.wikimedia.org/T205347 (10AlexisJazz) >>! In T205347#4612837, @Jdforrester-WMF wrote: > So, in thinking about this, I'm a little worried on how to get it done. W...
[14:47:33] <wikibugs>	 (03CR) 10Hashar: "Forgot, we first need the parent to drop stretch-backports which is dependent change https://gerrit.wikimedia.org/r/c/operations/puppet/+/" [integration/config] - 10https://gerrit.wikimedia.org/r/611181 (https://phabricator.wikimedia.org/T256844) (owner: 10Hashar)
[15:03:02] <shinken-wm_>	 PROBLEM - Free space - all mounts on deployment-snapshot01 is CRITICAL: CRITICAL: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found)deployment-prep.deployment-snapshot01.diskspace.root.byte_percentfree (<10.00%)
[15:05:05] <wikibugs>	 10Release-Engineering-Team (Onboarding), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10User-brennen: Onboard Ahmon Dancy - Software Engineer in RelEng - https://phabricator.wikimedia.org/T256074 (10thcipriani) 05Open→03Resolved a:03dancy The major su...
[15:06:36] <shinken-wm_>	 PROBLEM - Free space - all mounts on integration-agent-docker-1004 is CRITICAL: CRITICAL: integration.integration-agent-docker-1004.diskspace._srv.byte_percentfree (<20.00%)
[15:13:00] <shinken-wm_>	 RECOVERY - Free space - all mounts on deployment-snapshot01 is OK: OK: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found)
[15:16:38] <shinken-wm_>	 RECOVERY - Free space - all mounts on integration-agent-docker-1004 is OK: OK: All targets OK
[15:21:58] <shinken-wm_>	 PROBLEM - Parsoid on deployment-parsoid11 is CRITICAL: connect to address 172.16.1.115 and port 8000: Connection refused
[15:24:24] <wikibugs>	 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Patch-For-Review, 10Release, 10Train Deployments: 1.35.0-wmf.40 deployment blockers - https://phabricator.wikimedia.org/T256668 (10Lucas_Werkmeister_WMDE) FTR, {T257625} seems to be new with wmf.40, but I don...
[15:36:40] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Patch-For-Review, 10Technical-Debt: Clear  /srv/.git on contint1001; move integration.wikimedia.org docroot to new location - https://phabricator.wikimedia.org/T149924 (10hashar...
[15:39:04] <wikibugs>	 10Phabricator (Upstream), 10Upstream: Feature request: New notification option for when somebody @mentions you in a Task - https://phabricator.wikimedia.org/T257633 (10epriestley) In some cases, you can currently do something similar with "Mail Stamps". Mail which mentions you is stamped with `mention(@usernam...
[16:34:15] <shinken-wm_>	 RECOVERY - Host deployment-xhgui02 is UP: PING OK - Packet loss = 0%, RTA = 0.45 ms
[16:39:14] <shinken-wm_>	 PROBLEM - Host deployment-xhgui02 is DOWN: CRITICAL - Host Unreachable (172.16.1.202)
[16:40:57] <wikibugs>	 (03PS1) 10Hashar: Support DOCUMENT_ROOT being a symbolic link [integration/docroot] - 10https://gerrit.wikimedia.org/r/611377 (https://phabricator.wikimedia.org/T149924)
[16:41:43] <wikibugs>	 (03CR) 10Hashar: "Thank you Krinkle for your guidance about scap DocumentRoot being a symlink  AND for the existing test suite ! :]" [integration/docroot] - 10https://gerrit.wikimedia.org/r/611377 (https://phabricator.wikimedia.org/T149924) (owner: 10Hashar)
[16:56:31] <shinken-wm_>	 PROBLEM - Host deployment-cpjobqueue is DOWN: CRITICAL - Host Unreachable (172.16.4.124)
[16:56:31] <shinken-wm_>	 PROBLEM - Host deployment-changeprop is DOWN: CRITICAL - Host Unreachable (172.16.5.21)
[17:09:48] <Lucas_WMDE>	 I’m trying to push a mediawiki-config change to gerrit but it’s being extremely slow…
[17:10:36] <Lucas_WMDE>	 wait, it probably shouldn’t be trying to write 114952 objects?
[17:10:41] * Lucas_WMDE Ctrl+Cs for now
[17:11:34] <Lucas_WMDE>	 ok, with `git review` it worked 🤷
[17:40:37] <RhinosF1>	 andre__: their email is always set to joaquinito2052 at gmail dot com
[17:40:46] <RhinosF1>	 For ldap so that might help
[17:40:55] <andre__>	 RhinosF1: No Phab accounts
[17:41:08] <andre__>	 I don't do LDAP :)
[17:41:09] <greg-g>	 twentyafterfour: Hiya, RhinosF1 is having an mysql connection issue with search when searing for "-riggle-2" (I confirmed it myself, others haven't)
[17:41:19] <andre__>	 RhinosF1: oh, you mean the account is not linked via SUL? I see
[17:41:31] <RhinosF1>	 andre__: It could be / It might not be
[17:41:55] <mutante>	 so when i search for -riggle-2 i don' see an error BUT i can confirm it is taking forever 
[17:42:07] <RhinosF1>	 But he nearly always creates one
[17:42:11] <mutante>	  the - makes it try to remove the word from search?
[17:42:24] <RhinosF1>	 mutante: maybe that's why then
[17:42:36] <mutante>	 yea, i see the exception after a while of it loading
[17:42:57] <mutante>	 searching for just "riggle2" is normal and fast
[17:43:00] <RhinosF1>	 andre__: he'll either have connected ldap or wiki and he always uses joaquinito2052 at gmail dot com as an ldap email so maybe same for phab/wiki
[17:43:04] <mutante>	 i think it's the leading -
[17:43:23] <RhinosF1>	 Ah
[17:44:42] <mutante>	 there is nobody with an email starting with joaquinito* in LDAP
[17:44:47] <mutante>	 if that's what you are after
[17:45:09] <mutante>	 oh, there is :)
[17:45:22] <RhinosF1>	 There's numerous
[17:45:38] <mutante>	 RhinosF1: so you need any results from an LDAP search .. or ?
[17:45:53] <mutante>	 yea, i search email instead of mail. my bad. there are numerous indeed
[17:46:07] <RhinosF1>	 mutante: any that aren't disabled will need doing so and phab accounts disabling
[17:46:23] <andre__>	 RhinosF1: Ran an SQL query; I'll disable four Phab users with such email addresses. Thanks!
[17:47:02] <RhinosF1>	 andre__: Thanks! If they're linked to unlocked wiki accounts let me know and i'll get them locked
[17:47:28] <mutante>	 RhinosF1: one is called RhinosF1-2 
[17:47:50] <RhinosF1>	 mutante: is that the one that wanted a laptop in the cloud or new?
[17:48:13] <mutante>	 RhinosF1: i have no idea. never heard of that and can't tell from just LDAP user names
[17:48:35] <RhinosF1>	 mutante: yep, https://phabricator.wikimedia.org/p/RhinosF1-2/
[17:48:50] <RhinosF1>	 That's from december and got him wmf banned
[17:49:02] <andre__>	 RhinosF1: Nickelodeon22222 is already blocked globally; Nulledteam and Obiwan2208-2 and Fas3943 not yet
[17:49:14] <shinken-wm_>	 RECOVERY - Host deployment-xhgui02 is UP: PING OK - Packet loss = 0%, RTA = 0.71 ms
[17:49:17] <RhinosF1>	 andre__: Thanks!
[17:49:34] <andre__>	 Those four ^ all have a "joaquinito****@gmail.com" email scheme in Phab for their Phab accounts
[17:49:42] <mutante>	 you want me to paste the list of user names somewhere or you are good?
[17:49:51] <mutante>	 in PM to anyone?
[17:50:01] <RhinosF1>	 andre__: perfect!
[17:50:10] <RhinosF1>	 mutante: drop me a pm
[17:50:56] <mutante>	 pasted 17 CNs
[17:51:49] <mutante>	 8 unique email addresses among them. pasted as well
[17:52:04] <RhinosF1>	 andre__: try @gmx.es as well per ^
[17:52:18] <mutante>	 yes, one is gmx.es the rest is gmx.com
[17:52:39] <mutante>	 and one gmail
[17:54:52] <RhinosF1>	 ^
[17:55:07] <RhinosF1>	 Sent them to the office as well to go and look at them and lock
[17:59:14] <shinken-wm_>	 PROBLEM - Host deployment-xhgui02 is DOWN: CRITICAL - Host Unreachable (172.16.1.202)
[18:07:46] <ebernhardson>	 Minor phabricator thing i dont understand. I set the email address on mediawiki.org for Cindy-the-browser-test-bot to bernhardsonerik@gmail.com. I then logged into phabricator (first time), and it says it sent a confirmation email to cindythebrowsertestbot@wikimedia.org
[18:08:39] <ebernhardson>	 Prior to the new email address, it was pointed at smalyshev@wikimedia.org. Not sure where it got the @wikimedia.org address
[18:15:55] <mutante>	 ebernhardson: i don't know where it gets it from but i can tell you one thing. "undeliverable: Address cindythebrowsertestbot@wikimedia.org does not exist"
[18:19:07] <ebernhardson>	 mutante: yea, i'm certain that address doesn't exist as well :)
[18:27:15] <mutante>	 ebernhardson: that email address is in the phab mysql db
[18:27:20] <mutante>	 for user ID 6330
[18:30:11] <ebernhardson>	 hmm, i wonder how it got there. I suppose it's possible smalyshev set that some time ago (he's since left)
[18:33:36] <mutante>	 maybe it was first an LDAP user and later connected as MW user
[18:34:45] <mutante>	 ebernhardson: are you logged in as cindy on phab? go to Settings -> Email addresses. there can be multiple ones, primary and others
[18:34:59] <mutante>	 maybe it's listed there? then you can remove it
[18:35:31] <ebernhardson>	 mutante: the only page i can get from phab is "Must Verify Email", the two options are send another email and logout
[18:36:14] <ebernhardson>	 its not suport important, i was just going to use this to have an instance in cloud paste test run outputs to phab
[18:37:11] <mutante>	 hm, ok
[18:37:24] <ebernhardson>	 I didn't really want to put my real phab api key in it :)
[20:37:35] <wikibugs>	 10Gerrit: Horizontal scrollbars are missing when needed in Gerrit - https://phabricator.wikimedia.org/T257705 (10Huji)
[20:37:44] <wikibugs>	 10Gerrit: Horizontal scrollbars are missing when needed in Gerrit - https://phabricator.wikimedia.org/T257705 (10Huji)
[20:37:47] <wikibugs>	 10Gerrit, 10Patch-For-Review: Gerrit 3.2 upgrade - https://phabricator.wikimedia.org/T254158 (10Huji)
[21:19:02] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))): CI fetching tags should also create the local tag references - https://phabricator.wikimedia.org/T186496 (10hashar) The impleme...
[21:54:35] <dancy>	 !log Running `puppet agent --test` on deployment-deploy01.deployment-prep.eqiad.wmflab
[21:54:36] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[22:32:31] <wikibugs>	 10Continuous-Integration-Config: Test for the introduction of new global variables - https://phabricator.wikimedia.org/T257710 (10Huji)
[22:39:17] <wikibugs>	 10Continuous-Integration-Config: Test for the introduction of new global variables - https://phabricator.wikimedia.org/T257710 (10DannyS712) Not sure taint check would be the right place, but rather codesniffer. @Daimona thoughts? Uploading a patch momentarily to expand ExtendClassUsageSniff, which checks for us...
[22:39:48] <wikibugs>	 (03PS1) 10DannyS712: Expand ExtendClassUsageSniff with subclasses of ContextSource [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/611222 (https://phabricator.wikimedia.org/T257710)
[22:41:02] <wikibugs>	 10Continuous-Integration-Config, 10Patch-For-Review: Test for the introduction of new global variables - https://phabricator.wikimedia.org/T257710 (10Daimona) >>! In T257710#6298011, @DannyS712 wrote: > Not sure taint check would be the right place, but rather codesniffer.  Yeah, definitely not taint-check. I'...
[22:56:32] <wikibugs>	 (03PS2) 10DannyS712: Expand ExtendClassUsageSniff with subclasses of ContextSource [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/611222 (https://phabricator.wikimedia.org/T257710)
[22:57:47] <wikibugs>	 (03PS3) 10DannyS712: Expand ExtendClassUsageSniff with subclasses of ContextSource [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/611222 (https://phabricator.wikimedia.org/T257710)
[22:59:37] <wikibugs>	 (03PS4) 10DannyS712: Expand ExtendClassUsageSniff with subclasses of ContextSource [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/611222 (https://phabricator.wikimedia.org/T257710)
[23:00:02] <wikibugs>	 (03CR) 10DannyS712: "This change is ready for review." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/611222 (https://phabricator.wikimedia.org/T257710) (owner: 10DannyS712)
[23:01:56] <wikibugs>	 10Continuous-Integration-Config, 10MediaWiki-Codesniffer, 10Patch-For-Review: Test for the introduction of new global variables - https://phabricator.wikimedia.org/T257710 (10DannyS712)
[23:08:35] <wikibugs>	 (03CR) 10Huji: "I am not familiar with how to test pathces for codesniffer. Removing myself as a reviewer." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/611222 (https://phabricator.wikimedia.org/T257710) (owner: 10DannyS712)
[23:17:06] <wikibugs>	 10Gerrit: "Move changes endpoint is disabled" - https://phabricator.wikimedia.org/T257711 (10dpifke)
[23:18:48] <wikibugs>	 (03CR) 10Thcipriani: [C: 03+1] "Nicely done, good test coverage, and comments covering this functionality." [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/610922 (https://phabricator.wikimedia.org/T257526) (owner: 10Jeena Huneidi)
[23:22:05] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Scap, 10Patch-For-Review: beta-publish-deb failure do to "/srv/packages/db/LOG: permission denied" - https://phabricator.wikimedia.org/T250157 (10thcipriani) a:03dancy
[23:22:22] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Scap, 10Patch-For-Review: beta-publish-deb failure do to "/srv/packages/db/LOG: permission denied" - https://phabricator.wikimedia.org/T250157 (10thcipriani) p:05Triage→03Medium
[23:22:45] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Scap, 10Patch-For-Review: beta-publish-deb failure do to "/srv/packages/db/LOG: permission denied" - https://phabricator.wikimedia.org/T250157 (10thcipriani)
[23:23:48] <wikibugs>	 10Phabricator, 10Operations, 10Security-Team: Can't access phabricator from my server - https://phabricator.wikimedia.org/T257507 (10Legoktm) Can we exempt Isarra's IP from the blocklist in the meantime?
[23:24:15] <shinken-wm_>	 RECOVERY - Host deployment-xhgui02 is UP: PING OK - Packet loss = 0%, RTA = 2.13 ms
[23:34:13] <shinken-wm_>	 PROBLEM - Host deployment-xhgui02 is DOWN: CRITICAL - Host Unreachable (172.16.1.202)
[23:43:59] <dpifke>	 !log Cherry-picking https://gerrit.wikimedia.org/r/c/performance/arc-lamp/+/611465 into beta, should only affect deployment-webperf12.
[23:44:01] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[23:49:05] <wikibugs>	 10Phabricator, 10Operations, 10Security-Team: Can't access phabricator from my server - https://phabricator.wikimedia.org/T257507 (10Isarra) If the IP block exempt right onwiki also works (at least some of the time) per T254568, maybe I should just go request that? I never bothered because I so rarely actual...
[23:54:20] <shinken-wm_>	 PROBLEM - Host deployment-sentry01 is DOWN: CRITICAL - Host Unreachable (172.16.5.16)
[23:58:35] <wikibugs>	 10Phabricator, 10Operations, 10Security-Team: Can't access phabricator from my server - https://phabricator.wikimedia.org/T257507 (10Reedy) >>! In T257507#6298076, @Isarra wrote: > If the IP block exempt right onwiki also works (at least some of the time) per T254568, maybe I should just go request that? I n...