[00:14:49] 10Release-Engineering-Team (Kanban), 10Community-Tech, 10LDAP-Access-Requests: Phab/Wikitech/Gerrit accounts for Harumi Monroy - https://phabricator.wikimedia.org/T222110 (10Dzahn) [00:15:25] 10Release-Engineering-Team (Kanban), 10Community-Tech, 10LDAP-Access-Requests: Phab/Wikitech/Gerrit accounts for Harumi Monroy - https://phabricator.wikimedia.org/T222110 (10Dzahn) [00:23:11] 10Release-Engineering-Team (Kanban), 10Community-Tech, 10LDAP-Access-Requests: Phab/Wikitech/Gerrit accounts for Harumi Monroy - https://phabricator.wikimedia.org/T222110 (10Dzahn) [00:23:31] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Community-Tech, 10LDAP-Access-Requests: Phab/Wikitech/Gerrit accounts for Harumi Monroy - https://phabricator.wikimedia.org/T222110 (10Dzahn) [00:23:43] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Community-Tech, 10LDAP-Access-Requests, 10Toolforge: Phab/Wikitech/Gerrit accounts for Harumi Monroy - https://phabricator.wikimedia.org/T222110 (10Dzahn) [00:24:31] PROBLEM - Citoid on deployment-sca01 is CRITICAL: connect to address 172.16.5.13 and port 1970: Connection refused [00:24:31] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Community-Tech, 10LDAP-Access-Requests, 10Toolforge: Phab/Wikitech/Gerrit accounts for Harumi Monroy - https://phabricator.wikimedia.org/T222110 (10Dzahn) - added to 'wmf' LDAP group (per request for +2 on gerrit repos) [00:24:48] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Community-Tech, 10LDAP-Access-Requests, 10Toolforge: Phab/Wikitech/Gerrit accounts for Harumi Monroy - https://phabricator.wikimedia.org/T222110 (10Dzahn) p:05Triage→03High [00:32:25] who approves pending phabricator users? [00:32:36] i see eh.. a LOT that say "needs approval" [00:37:25] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Community-Tech, 10LDAP-Access-Requests, 10Toolforge: Phab/Wikitech/Gerrit accounts for Harumi Monroy - https://phabricator.wikimedia.org/T222110 (10Dzahn) [00:41:21] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Community-Tech, 10LDAP-Access-Requests, 10Toolforge: Phab/Wikitech/Gerrit accounts for Harumi Monroy - https://phabricator.wikimedia.org/T222110 (10Dzahn) - approved Phabricator user (fwiw, i could not find docs on this or that this new requirement ex... [00:41:34] 10Beta-Cluster-Infrastructure, 10Release Pipeline, 10serviceops, 10Core Platform Team Backlog (Next), and 2 others: Migrate Beta cluster services to use Kubernetes - https://phabricator.wikimedia.org/T220235 (10mobrovac) p:05Normal→03High Raising the prio and moving to //next// since we'll have to atta... [00:43:14] (03PS17) 10Kosta Harlan: Establish codehealth pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/502606 (https://phabricator.wikimedia.org/T218598) [00:44:32] RECOVERY - Citoid on deployment-sca01 is OK: HTTP OK: HTTP/1.1 200 OK - 921 bytes in 0.026 second response time [00:55:39] 10Beta-Cluster-Infrastructure: Migrate away from Debian Jessie to Debian Stretch - https://phabricator.wikimedia.org/T218729 (10Krenair) [00:57:23] 10Phabricator, 10Project-Admins, 10Release-Engineering-Team (Next), 10Operations, and 2 others: Document how to convert projects into subprojects/milestones etc - https://phabricator.wikimedia.org/T221112 (10Dzahn) [00:58:48] 10Phabricator, 10Project-Admins, 10Release-Engineering-Team (Next), 10Operations, and 2 others: Document how to convert projects into subprojects/milestones etc - https://phabricator.wikimedia.org/T221112 (10Dzahn) >>! In T221112#5121789, @mmodell wrote: @Dzahn can you help me figure out how to allow @akla... [01:02:56] 10Phabricator, 10Project-Admins, 10Release-Engineering-Team (Next), 10Operations, and 2 others: Document how to convert projects into subprojects/milestones etc (sudo privileges for phab admins to run move_project script) - https://phabricator.wikimedia.org/T221112 (10Dzahn) [01:11:26] 10MediaWiki-Codesniffer, 10good first bug: Add sniff to detect usage of deprecated $wgUser - https://phabricator.wikimedia.org/T50963 (10hashar) 05Open→03Declined I guess that will be achieved by actually deprecating $wgUser: T159299 [01:33:26] 10Continuous-Integration-Infrastructure: wmf-quibble-vendor-mysql-hhvm-docker job sometime take 40+ minutes to run - https://phabricator.wikimedia.org/T222023 (10hashar) On contint1001, I got the build time of all wmf-quibble jobs using: xml_grep --text_only 'duration|builtOn' /srv/jenkins/builds/wmf-quibbl... [01:35:39] !log Deleting integration-slave-docker-1037 (bigram) it is too slow for some reason # T222023 [01:35:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [01:35:41] T222023: wmf-quibble-vendor-mysql-hhvm-docker job sometime take 40+ minutes to run - https://phabricator.wikimedia.org/T222023 [01:38:22] 10Continuous-Integration-Infrastructure: wmf-quibble-vendor-mysql-hhvm-docker job sometime take 40+ minutes to run - https://phabricator.wikimedia.org/T222023 (10hashar) 05Open→03Resolved a:03hashar [01:44:11] PROBLEM - Host integration-slave-docker-1037 is DOWN: CRITICAL - Host Unreachable (172.16.3.87) [02:01:48] !log Polled in integration-slave-docker-1055 and ntegration-slave-docker-1056 [02:01:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [02:10:05] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [02:14:37] RECOVERY - Content Translation Server on deployment-sca02 is OK: HTTP OK: HTTP/1.1 200 OK - 904 bytes in 0.029 second response time [02:15:05] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.053 second response time [02:26:05] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [02:55:14] Hi, I found a problem at tests [02:55:25] Can anyone check why this happening: https://integration.wikimedia.org/ci/job/mediawiki-quibble-composertest-php70-docker/17924/console [02:55:36] 04:40:20 Unable to find image 'docker-registry.wikimedia.org/releng/castor:0.2.0' locally [02:55:44] 04:40:25 docker: open /var/lib/docker/tmp/GetImageBlob783615918: no such file or directory. [02:56:17] Patch where I found this: https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/507176/ [03:05:05] 10Continuous-Integration-Infrastructure: mediawiki-quibble-composertest-php70-docker failure: Unable to find image 'docker-registry.wikimedia.org/releng/castor:0.2.0' locally - https://phabricator.wikimedia.org/T222131 (10Zoranzoki21) [03:05:12] 10Continuous-Integration-Infrastructure: mediawiki-quibble-composertest-php70-docker failure: Unable to find image 'docker-registry.wikimedia.org/releng/castor:0.2.0' locally - https://phabricator.wikimedia.org/T222131 (10Zoranzoki21) p:05Triage→03High [03:06:07] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.426 second response time [03:22:05] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [03:39:10] PROBLEM - Citoid on deployment-sca02 is CRITICAL: connect to address 172.16.5.112 and port 1970: Connection refused [03:44:07] RECOVERY - Citoid on deployment-sca02 is OK: HTTP OK: HTTP/1.1 200 OK - 921 bytes in 0.027 second response time [03:50:07] PROBLEM - Citoid on deployment-sca02 is CRITICAL: connect to address 172.16.5.112 and port 1970: Connection refused [03:52:06] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.669 second response time [03:58:15] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:10:09] RECOVERY - Citoid on deployment-sca02 is OK: HTTP OK: HTTP/1.1 200 OK - 921 bytes in 0.025 second response time [04:23:03] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.046 second response time [04:39:08] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [05:23:52] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<20.00%) [05:49:37] PROBLEM - Content Translation Server on deployment-sca02 is CRITICAL: connect to address 172.16.5.112 and port 8080: Connection refused [05:54:41] RECOVERY - Content Translation Server on deployment-sca02 is OK: HTTP OK: HTTP/1.1 200 OK - 904 bytes in 0.027 second response time [05:57:07] 10Continuous-Integration-Infrastructure: mediawiki-quibble-composertest-php70-docker failure: Unable to find image 'docker-registry.wikimedia.org/releng/castor:0.2.0' locally - https://phabricator.wikimedia.org/T222131 (10Nikerabbit) Also happened in https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-... [05:59:57] 10Continuous-Integration-Infrastructure: mediawiki-quibble-composertest-php70-docker failure: Unable to find image 'docker-registry.wikimedia.org/releng/castor:0.2.0' locally - https://phabricator.wikimedia.org/T222131 (10Ladsgroup) p:05High→03Unbreak! Aand https://gerrit.wikimedia.org/r/c/mediawiki/core/+/5... [07:03:51] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:49:40] (03PS1) 10WMDE-leszek: Edit Project Config [extensions/WikibaseSchema] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/507252 [07:49:59] (03Abandoned) 10WMDE-leszek: Edit Project Config [extensions/WikibaseSchema] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/507252 (owner: 10WMDE-leszek) [08:18:29] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10Wikidata, and 5 others: Introduce wikidata termbox SSR to kubernetes - https://phabricator.wikimedia.org/T220402 (10Pablo-WMDE) @mobrovac During T221755 & T221754 we tended to [[ https://ssr-termbox.wmflabs.org/?spec | `/?spec` ]] and [[ https... [09:03:04] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.028 second response time [09:19:03] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [09:29:05] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.023 second response time [09:49:04] 10Continuous-Integration-Infrastructure, 10Wikimedia-production-error (Shared Build Failure): mediawiki-quibble-composertest-php70-docker failure: Unable to find image 'docker-registry.wikimedia.org/releng/castor:0.2.0' locally - https://phabricator.wikimedia.org/T222131 (10Lucas_Werkmeister_WMDE) [10:00:05] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [10:05:04] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.024 second response time [11:05:08] 10Continuous-Integration-Infrastructure, 10Wikimedia-production-error (Shared Build Failure): mediawiki-quibble-composertest-php70-docker failure: Unable to find image 'docker-registry.wikimedia.org/releng/castor:0.2.0' locally - https://phabricator.wikimedia.org/T222131 (10Lucas_Werkmeister_WMDE) It looks lik... [11:07:51] 10Continuous-Integration-Infrastructure, 10Wikimedia-production-error (Shared Build Failure): mediawiki-quibble-composertest-php70-docker failure: Unable to find image 'docker-registry.wikimedia.org/releng/castor:0.2.0' locally - https://phabricator.wikimedia.org/T222131 (10Lucas_Werkmeister_WMDE) On the other... [11:09:06] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): Fix partitions on CI slaves , some are missing /var/lib/docker - https://phabricator.wikimedia.org/T221040 (10hashar) I have added two Jenkins slaves with `role::ci::slave::labs::docker` with `$docker_lvm_volume = true`. One had a su... [11:24:34] 10Continuous-Integration-Infrastructure, 10Wikimedia-production-error (Shared Build Failure): mediawiki-quibble-composertest-php70-docker failure: Unable to find image 'docker-registry.wikimedia.org/releng/castor:0.2.0' locally - https://phabricator.wikimedia.org/T222131 (10Lucas_Werkmeister_WMDE) Okay, my con... [11:32:44] 10Gerrit, 10Repository-Admins, 10Shape Expressions, 10Wikidata, 10Wikidata-Campsite: rename repository for WikibaseSchema - https://phabricator.wikimedia.org/T221946 (10WMDE-leszek) a:05WMDE-leszek→03None Repository with the new name created: https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/... [11:35:21] 10Continuous-Integration-Infrastructure, 10Operations: Jessie rsyslog_8.1901.0-1~bpo8+wmf1_amd64.deb package fails to upgrade - https://phabricator.wikimedia.org/T222166 (10hashar) [11:45:05] (03PS1) 10Lucas Werkmeister (WMDE): Update for WikibaseSchema → EntitySchema rename [integration/config] - 10https://gerrit.wikimedia.org/r/507298 (https://phabricator.wikimedia.org/T221946) [11:45:53] 10Gerrit, 10Repository-Admins, 10Shape Expressions, 10Wikidata, and 2 others: rename repository for WikibaseSchema - https://phabricator.wikimedia.org/T221946 (10Lucas_Werkmeister_WMDE) [11:50:49] emails from Gerrit seem to be pretty delayed [11:51:02] I’m getting emails now for changes made during SWAT almost half an hour ago [11:55:18] herron ^^ (i think that's related to exim on gerrit) [11:55:50] we had this problem last night (only for the fact gerrit's thread's got stuck due to a bug in java which we upgraded to fix it). [11:59:09] 10Gerrit, 10Repository-Admins, 10Shape Expressions, 10Wikidata, and 2 others: rename repository for WikibaseSchema - https://phabricator.wikimedia.org/T221946 (10Lucas_Werkmeister_WMDE) [11:59:58] 10Continuous-Integration-Infrastructure, 10Wikimedia-production-error (Shared Build Failure): CI job fails docker: open /var/lib/docker/tmp/GetImageBlob990376239: no such file or directory - https://phabricator.wikimedia.org/T222131 (10hashar) [12:01:26] 10Gerrit, 10Repository-Admins, 10Shape Expressions, 10Wikidata, and 2 others: rename repository for WikibaseSchema - https://phabricator.wikimedia.org/T221946 (10Michael) [12:04:54] 10Continuous-Integration-Infrastructure, 10Wikimedia-production-error (Shared Build Failure): CI job fails docker: open /var/lib/docker/tmp/GetImageBlob990376239: no such file or directory - https://phabricator.wikimedia.org/T222131 (10hashar) 05Open→03Resolved a:03hashar The actually issue is Docker fai... [12:09:56] 10Continuous-Integration-Infrastructure, 10Wikimedia-production-error (Shared Build Failure): CI job fails docker: open /var/lib/docker/tmp/GetImageBlob990376239: no such file or directory - https://phabricator.wikimedia.org/T222131 (10hashar) And I have mailed wikitech-l to announce it to everyone. I apologi... [12:23:04] (03CR) 10Daniel Kinzler: [C: 03+1] "Seems useful to me, and the regex looks good too at a glance." [integration/config] - 10https://gerrit.wikimedia.org/r/487037 (owner: 10Lucas Werkmeister (WMDE)) [12:31:49] (03CR) 10Jbond: [C: 03+1] "LGTM and welcome improvement" [integration/config] - 10https://gerrit.wikimedia.org/r/487037 (owner: 10Lucas Werkmeister (WMDE)) [12:50:01] (03CR) 10Hashar: [C: 03+2] "That patch is absolutely lovely. I am wondering why none of us thought about it before!! :-]" [integration/config] - 10https://gerrit.wikimedia.org/r/487037 (owner: 10Lucas Werkmeister (WMDE)) [12:51:33] (03Merged) 10jenkins-bot: Trigger recheck on comments starting with recheck [integration/config] - 10https://gerrit.wikimedia.org/r/487037 (owner: 10Lucas Werkmeister (WMDE)) [12:52:56] (03CR) 10Hashar: [C: 03+2] "Deployed!" [integration/config] - 10https://gerrit.wikimedia.org/r/487037 (owner: 10Lucas Werkmeister (WMDE)) [12:53:38] (03CR) 10Lucas Werkmeister (WMDE): "Awesome, thanks! Now I almost look forward to the next CI breakage so I can try this out :D" [integration/config] - 10https://gerrit.wikimedia.org/r/487037 (owner: 10Lucas Werkmeister (WMDE)) [12:55:18] (03CR) 10Hashar: [C: 03+2] "Well I guess it should work ? At least the tests say so!" [integration/config] - 10https://gerrit.wikimedia.org/r/487037 (owner: 10Lucas Werkmeister (WMDE)) [13:04:44] Hmm looks like wikibugs is sending c+2 in the msg even if it was done prior to that msg [13:08:04] (03CR) 10Lucas Werkmeister (WMDE): "Thanks, tested at Iec61a51110 and announced to wikitech-l in https://lists.wikimedia.org/pipermail/wikitech-l/2019-April/092030.html (look" [integration/config] - 10https://gerrit.wikimedia.org/r/487037 (owner: 10Lucas Werkmeister (WMDE)) [13:14:41] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Request Sauce Labs access for niedzielski - https://phabricator.wikimedia.org/T206358 (10zeljkofilipin) Looks like the last time I've sent the invitation to your private e-mail. I've sent it now to your wikimedia e-mail. Let me know if you have trouble... [13:14:51] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Request Sauce Labs access for niedzielski - https://phabricator.wikimedia.org/T206358 (10zeljkofilipin) 05Open→03Resolved [13:18:31] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:40:17] PROBLEM - Host deployment-conf03 is DOWN: CRITICAL - Host Unreachable (172.16.5.30) [13:47:49] PROBLEM - Content Translation Server on deployment-sca02 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:51:07] 10Continuous-Integration-Infrastructure, 10Operations: Jessie rsyslog_8.1901.0-1~bpo8+wmf1_amd64.deb package fails to upgrade - https://phabricator.wikimedia.org/T222166 (10fgiunchedi) Indeed, we've ran into the same problem on {T219764}. tl;dr the solution is to repeat the install: `apt install rsyslog rsyslo... [14:08:31] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - string 'Wikipedia' not found on 'https://en.wikipedia.beta.wmflabs.org:443/wiki/Main_Page?debug=true' - 325 bytes in 0.007 second response time [14:09:11] oh noes [14:12:04] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - string 'Wikipedia' not found on 'https://en.m.wikipedia.beta.wmflabs.org:443/wiki/Main_Page?debug=true' - 325 bytes in 0.008 second response time [14:13:55] 10Beta-Cluster-Infrastructure: Beta Varnish text frontend won't (re)start on deployment-cache-text05 - https://phabricator.wikimedia.org/T222177 (10Gilles) [14:14:06] 10Beta-Cluster-Infrastructure: Beta Varnish text frontend won't (re)start on deployment-cache-text05 - https://phabricator.wikimedia.org/T222177 (10Gilles) p:05Triage→03Unbreak! [14:14:54] 10Beta-Cluster-Infrastructure: Beta Varnish text frontend won't (re)start on deployment-cache-text05 - https://phabricator.wikimedia.org/T222177 (10Gilles) [14:18:57] might be edac556d5777063538378c3e7b643077b747d95a ? [14:19:42] * Krenair tryin [14:25:07] well that didn't fix it [14:28:25] see -traffic [14:41:38] 10Beta-Cluster-Infrastructure: Beta Varnish text frontend won't (re)start on deployment-cache-text05 - https://phabricator.wikimedia.org/T222177 (10Gilles) 05Open→03Resolved a:03Gilles This required to add some new necessary hiera values on Beta, that were added recently to production. [14:42:05] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 36662 bytes in 0.761 second response time [14:43:32] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 48304 bytes in 0.675 second response time [15:00:55] 10Gerrit, 10Repository-Admins, 10Shape Expressions, 10Wikidata, and 2 others: rename repository for WikibaseSchema - https://phabricator.wikimedia.org/T221946 (10Lucas_Werkmeister_WMDE) [15:02:12] hello, I +2'ed 506953 [15:02:22] and think there is a problem [15:02:47] because verified +2 happened at :47 [15:04:19] <_joe_> no one's here [15:05:00] <_joe_> jijiki: I'll submit that patch manually [15:05:27] ok [15:05:32] <_joe_> merged, go on [15:08:40] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Request Sauce Labs access for niedzielski - https://phabricator.wikimedia.org/T206358 (10Niedzielski) 05Resolved→03Open Thanks for looking into this, @zeljkofilipin! I tried to use the invitation but I think we're out of licenses :[ The error I get... [15:11:21] 10Gerrit, 10LDAP: Gerrit login failure for user tk-999 - https://phabricator.wikimedia.org/T222186 (10TK-999) [15:18:33] (03CR) 10Zfilipin: "> Patch Set 3:" [integration/config] - 10https://gerrit.wikimedia.org/r/460527 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [15:26:57] 10Gerrit, 10Shape Expressions, 10Wikidata, 10Wikidata-Campsite (Wikidata-Campsite-Iteration-∞): Replace WikibaseSchema repository content with message pointing to EntitySchema - https://phabricator.wikimedia.org/T222192 (10Lucas_Werkmeister_WMDE) [15:33:05] 10Gerrit, 10LDAP: Gerrit login failure for user tk-999 - https://phabricator.wikimedia.org/T222186 (10TK-999) Note: I've tried other case variations of my Wikitech username as well (TK-999, Tk-999, tk-999), but to no avail. [15:39:43] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Request Sauce Labs access for niedzielski - https://phabricator.wikimedia.org/T206358 (10zeljkofilipin) @AndyRussG @dduvall @Mattflaschen-WMF @Niedzielski @SBisson I have deleted your sauce labs accounts. As far as I could see at the team management pa... [15:41:03] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Request Sauce Labs access for niedzielski - https://phabricator.wikimedia.org/T206358 (10SBisson) Haven't used it in years. Thanks for cleaning that up. [15:43:23] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Request Sauce Labs access for niedzielski - https://phabricator.wikimedia.org/T206358 (10zeljkofilipin) @Niedzielski try again. I've removed unused accounts. It should work now. Please resolve the ticked if it works. :) [15:50:44] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Request Sauce Labs access for niedzielski - https://phabricator.wikimedia.org/T206358 (10Niedzielski) 05Open→03Resolved Thanks @zeljkofilipin (and thank you for such prompt resolution)! It works! \o/ [15:59:32] 10Release-Engineering-Team (Kanban), 10MW-1.32-notes, 10MW-1.32-release: Release 1.32.1 as a maintenance release - https://phabricator.wikimedia.org/T213595 (10Reedy) [16:00:40] 10Release-Engineering-Team (Kanban), 10MW-1.32-notes, 10MW-1.32-release: Release 1.32.1 as a maintenance release - https://phabricator.wikimedia.org/T213595 (10Reedy) Are we going to announce it? :) We need to get T205041 done soon, so this has some affect on that too [16:03:38] RECOVERY - Content Translation Server on deployment-sca02 is OK: HTTP OK: HTTP/1.1 200 OK - 904 bytes in 0.027 second response time [16:08:34] (03PS2) 10Lucas Werkmeister (WMDE): Update for WikibaseSchema → EntitySchema rename [integration/config] - 10https://gerrit.wikimedia.org/r/507298 (https://phabricator.wikimedia.org/T222189) [16:08:36] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): TEC3:O1:O1.2 Goal – Formalize the collection of CI infrastructure and tooling metrics - https://phabricator.wikimedia.org/T205923 (10thcipriani) [16:12:06] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10serviceops: Migrate ORES to kubernetes - https://phabricator.wikimedia.org/T220400 (10thcipriani) [16:12:41] 10Gerrit, 10Operations, 10cloud-services-team, 10serviceops: Change /r/p/ to /r/ on all hosts (where https://gerrit.wikimedia.org/r/p/ exists) - https://phabricator.wikimedia.org/T222093 (10Paladox) [16:12:48] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10serviceops, and 2 others: TEC3:O3:O3.1:Q4 Goal - Move cpjobqueue, Wikidata Termbox SSR (new service), Kask (session storage service) and ORES (partially) through the production CD Pipeline - https://phabricator.wikimedia.org/T220398 (10thcipria... [16:14:19] 10Release-Engineering-Team, 10Release Pipeline, 10serviceops, 10Core Platform Team Backlog (Watching / External), 10Services (watching): TEC3:O3:O3.1:Q3 Goal - Move cxserver, citoid, changeprop, eventgate (new service) and ORES (partially) through the production ... - https://phabricator.wikimedia.org/T212801 [16:16:11] 10Release-Engineering-Team (Backlog), 10ORES, 10Operations, 10Release Pipeline, and 2 others: Execution of the deployment pipeline should be configurable via .pipeline/config.yaml - https://phabricator.wikimedia.org/T210267 (10thcipriani) [16:19:05] (03CR) 10WMDE-leszek: "Understood, makes sense now. Can we maybe have the job deployed again, to see if it does what it's supposed to do now?" [integration/config] - 10https://gerrit.wikimedia.org/r/460527 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [16:20:19] 10Release-Engineering-Team (Backlog), 10Release Pipeline: Post generated docs for pipelinelib - https://phabricator.wikimedia.org/T222199 (10thcipriani) [16:20:45] 10Release-Engineering-Team (Kanban), 10Release Pipeline: Post generated docs for pipelinelib - https://phabricator.wikimedia.org/T222199 (10thcipriani) p:05Triage→03Normal a:03dduvall [16:21:51] 10Continuous-Integration-Infrastructure, 10Wikidata, 10Wikidata-Campsite: Run browser tests as part of "npm test" of wikidata/query/gui - https://phabricator.wikimedia.org/T222200 (10WMDE-leszek) [16:22:00] 10Release-Engineering-Team, 10MobileFrontend, 10Browser-Tests, 10MW-1.34-notes (1.34.0-wmf.3; 2019-04-30), and 3 others: AssertionError: false === true at thereShouldBeALinkToCreateMyUserPage on wmf-quibble PHP jobs - https://phabricator.wikimedia.org/T221860 (10ovasileva) [16:23:03] 10Continuous-Integration-Infrastructure, 10Wikidata, 10Wikidata-Campsite: Run browser tests as part of "npm test" of wikidata/query/gui - https://phabricator.wikimedia.org/T222200 (10WMDE-leszek) @hashar @zeljkofilipin this seems to beyond our (WMDE) realm. We're happy to provide more context/information, bu... [16:31:45] 10Release-Engineering-Team (Kanban), 10MW-1.34-notes (1.34.0-wmf.3; 2019-04-30), 10Patch-For-Review, 10Readers-Web-Backlog (Readers-Web-Kanbanana-Board-2018-19-Q4), and 2 others: Enable a MobileFrontend daily Selenium build - https://phabricator.wikimedia.org/T220659 (10ovasileva) We were not able to estim... [16:38:22] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [16:41:22] 10Release-Engineering-Team (Kanban), 10Developer Productivity, 10local-charts: TEC12:O1:O1.1 Goal – Create a production tooling compliant Dockerfile that sets up a basic LAMP stack, installs MediaWiki, and can run unit tests - https://phabricator.wikimedia.org/T222205 (10Jdforrester-WMF) p:05Triage→03Norm... [16:43:49] 10Release-Engineering-Team (Kanban), 10Developer Productivity, 10local-charts: TEC12:O1:O1.1 Goal – Create a production tooling compliant Dockerfile that sets up a basic LAMP stack, installs MediaWiki, and can run unit tests - https://phabricator.wikimedia.org/T222205 (10Jdforrester-WMF) a:05brennen→03jee... [16:48:04] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Release Pipeline: Post generated docs for pipelinelib - https://phabricator.wikimedia.org/T222199 (10Jdforrester-WMF) [16:48:21] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [16:48:39] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Release Pipeline: Post generated docs for pipelinelib - https://phabricator.wikimedia.org/T222199 (10Jdforrester-WMF) Code is in Groovy, so this will need a bespoke docker image. [17:01:38] !log (beta): Update mobileapps to 142ba30 [17:01:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:08:11] (03PS1) 10MacFan4000: Archive EditAccount [integration/config] - 10https://gerrit.wikimedia.org/r/507377 (https://phabricator.wikimedia.org/T221494) [17:11:20] (03PS2) 10MacFan4000: Archive EditAccount [integration/config] - 10https://gerrit.wikimedia.org/r/507377 (https://phabricator.wikimedia.org/T221494) [17:14:58] 10Release-Engineering-Team (Kanban), 10MW-1.32-notes, 10MW-1.32-release: Release 1.32.1 as a maintenance release - https://phabricator.wikimedia.org/T213595 (10mmodell) Ok I will send the following to the appropriate mailing lists: ` Although the archives have been available for quite some time, this minor... [17:45:42] (03PS7) 10Thcipriani: post pipeline: fix ref filter [integration/config] - 10https://gerrit.wikimedia.org/r/494778 (https://phabricator.wikimedia.org/T96390) [17:50:10] 10Phabricator, 10Cleanup, 10Repository-Admins: Do not remove Wikidata items when archiving extensions and skins - https://phabricator.wikimedia.org/T221771 (10Aklapper) [17:50:21] 10Phabricator, 10Cleanup, 10Repository-Admins: Do not remove Wikidata items when archiving extensions and skins - https://phabricator.wikimedia.org/T221771 (10Aklapper) This seems to be about https://phabricator.wikimedia.org/maniphest/task/edit/form/33/ in the #Phabricator configuration. [17:53:33] (03CR) 10Thcipriani: [C: 03+2] post pipeline: fix ref filter [integration/config] - 10https://gerrit.wikimedia.org/r/494778 (https://phabricator.wikimedia.org/T96390) (owner: 10Thcipriani) [17:56:31] (03Merged) 10jenkins-bot: post pipeline: fix ref filter [integration/config] - 10https://gerrit.wikimedia.org/r/494778 (https://phabricator.wikimedia.org/T96390) (owner: 10Thcipriani) [18:00:38] !log reloading zuul to deploy https://gerrit.wikimedia.org/r/494778 [18:00:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:05:05] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [18:14:09] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [18:17:46] 10Phabricator, 10Cleanup, 10Repository-Admins: Do not remove Wikidata items when archiving extensions and skins - https://phabricator.wikimedia.org/T221771 (10Dinoguy1000) >>! In T221771#5148038, @Aklapper wrote: > This seems to be about https://phabricator.wikimedia.org/maniphest/task/edit/form/33/ in the #... [18:29:55] 10Release-Engineering-Team (Kanban), 10MW-1.32-notes, 10MW-1.32-release: Release 1.32.1 as a maintenance release - https://phabricator.wikimedia.org/T213595 (10Reedy) Needs the git tags pushing too :) [18:34:36] 10Gerrit, 10Repository-Admins, 10Shape Expressions, 10Wikidata, and 2 others: rename repository for WikibaseSchema - https://phabricator.wikimedia.org/T221946 (10Reedy) [19:06:52] 10Continuous-Integration-Infrastructure, 10Parsoid: phpunit-coverage-docker-publish fails because of incorrect version of PHP - https://phabricator.wikimedia.org/T222219 (10Zoranzoki21) [19:08:53] 10Continuous-Integration-Infrastructure, 10Parsoid: phpunit-coverage-docker-publish fails because of incorrect version of PHP - https://phabricator.wikimedia.org/T222219 (10ssastry) [19:20:41] (03CR) 10MusikAnimal: [C: 03+1] [DarkMode] Add new extension [integration/config] - 10https://gerrit.wikimedia.org/r/506934 (https://phabricator.wikimedia.org/T221877) (owner: 10Samwilson) [19:24:04] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [19:44:04] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.020 second response time [19:50:04] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [19:58:06] 10MediaWiki-Releasing, 10Release-Engineering-Team (Kanban), 10MW-1.33-release: Branch REL1_33 for MediaWiki and deployed extensions - https://phabricator.wikimedia.org/T220653 (10Jdforrester-WMF) [20:00:06] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.039 second response time [20:19:08] PROBLEM - Citoid on deployment-sca02 is CRITICAL: connect to address 172.16.5.112 and port 1970: Connection refused [20:29:10] RECOVERY - Citoid on deployment-sca02 is OK: HTTP OK: HTTP/1.1 200 OK - 921 bytes in 0.024 second response time [21:13:07] 10Release-Engineering-Team, 10Developer Productivity, 10Epic: Improve Developer Tooling - https://phabricator.wikimedia.org/T212449 (10brennen) [21:13:09] 10Release-Engineering-Team (Kanban), 10Developer Productivity, 10local-charts: Script SSHFS setup in local-charts - https://phabricator.wikimedia.org/T218364 (10brennen) [21:27:36] 10Release-Engineering-Team (Kanban), 10Developer Productivity, 10Release Pipeline, 10local-charts, 10Patch-For-Review: Define a base docker-pkg template and .pipeline/blubber.yaml for mediawiki/core - https://phabricator.wikimedia.org/T218360 (10brennen) [21:27:38] 10Release-Engineering-Team (Kanban), 10Developer Productivity, 10local-charts: TEC12:O1:O1.1 Goal – Create a production tooling compliant Dockerfile that sets up a basic LAMP stack, installs MediaWiki, and can run unit tests - https://phabricator.wikimedia.org/T222205 (10brennen) 05Open→03Resolved The cu... [21:36:56] 10Release-Engineering-Team (Backlog), 10Developer Productivity, 10local-charts, 10Epic: TEC12:O1:O1.3 Goal – Create a generic method for installing extensions and skins in the image - https://phabricator.wikimedia.org/T222232 (10Jdforrester-WMF) p:05Triage→03Normal [21:37:39] 10Continuous-Integration-Infrastructure, 10Operations: Jessie rsyslog_8.1901.0-1~bpo8+wmf1_amd64.deb package fails to upgrade - https://phabricator.wikimedia.org/T222166 (10Dzahn) p:05Triage→03High [21:37:51] 10Release-Engineering-Team (Backlog), 10Developer Productivity, 10local-charts, 10Epic: TEC12:O1:O1.5 Goal – A helm-based dependency chart for installing extensions and skins along with services they require - https://phabricator.wikimedia.org/T222233 (10Jdforrester-WMF) p:05Triage→03Normal [21:39:49] 10Release-Engineering-Team, 10Developer Productivity, 10Browser-Tests, 10Epic: TEC12:O1:O1.4 Goal – Support running Selenium browser tests in the docker local development environment - https://phabricator.wikimedia.org/T222234 (10Jdforrester-WMF) p:05Triage→03Normal [21:52:27] 10Continuous-Integration-Infrastructure, 10Composer: Deprecation warnings related to mediawiki/core - https://phabricator.wikimedia.org/T222235 (10Zoranzoki21) [22:14:23] 10Release-Engineering-Team, 10Developer Productivity: FY201819 TEC12:O1 – Outcome: Local development is unified with testing and production - https://phabricator.wikimedia.org/T222237 (10Jdforrester-WMF) [22:16:04] 10Release-Engineering-Team, 10Developer Productivity, 10Browser-Tests, 10Epic: TEC12:O1:O1.4 Goal – Support running Selenium browser tests in the docker local development environment - https://phabricator.wikimedia.org/T222234 (10Jdforrester-WMF) [22:16:06] 10Release-Engineering-Team (Backlog), 10Developer Productivity, 10local-charts, 10Epic: TEC12:O1:O1.5 Goal – A helm-based dependency chart for installing extensions and skins along with services they require - https://phabricator.wikimedia.org/T222233 (10Jdforrester-WMF) [22:16:08] 10Release-Engineering-Team (Backlog), 10Developer Productivity, 10local-charts, 10Epic: TEC12:O1:O1.3 Goal – Create a generic method for installing extensions and skins in the image - https://phabricator.wikimedia.org/T222232 (10Jdforrester-WMF) [22:16:10] 10Release-Engineering-Team, 10Developer Productivity: FY201819 TEC12:O1 – Outcome: Local development is unified with testing and production - https://phabricator.wikimedia.org/T222237 (10Jdforrester-WMF) [22:16:12] 10Release-Engineering-Team (Kanban), 10Developer Productivity, 10local-charts: TEC12:O1:O1.1 Goal – Create a production tooling compliant Dockerfile that sets up a basic LAMP stack, installs MediaWiki, and can run unit tests - https://phabricator.wikimedia.org/T222205 (10Jdforrester-WMF) [22:16:16] 10Release-Engineering-Team, 10Developer Productivity: FY201819 TEC12:O1 Outcome – Local development is unified with testing and production - https://phabricator.wikimedia.org/T222237 (10Jdforrester-WMF) [22:27:35] 10Release-Engineering-Team, 10Scap: scap leaving /srv/patches in a mess - https://phabricator.wikimedia.org/T222240 (10Reedy) [22:31:15] 10Release-Engineering-Team (Kanban), 10MW-1.32-release: Release 1.32.1 as a maintenance release - https://phabricator.wikimedia.org/T213595 (10mmodell) >>! In T213595#5148182, @Reedy wrote: > Needs the git tags pushing too :) {nav icon=code-fork > git push origin 1.32.1 > icon=check, name=Done} [22:31:30] 10Release-Engineering-Team (Kanban), 10MW-1.32-release: Release 1.32.1 as a maintenance release - https://phabricator.wikimedia.org/T213595 (10Reedy) 05Open→03Resolved Thanks! :) [22:39:53] (03PS1) 10Thcipriani: release-notes: TOOLS_RELEASE_DIR -> RELEASE_TOOLS_DIR [integration/config] - 10https://gerrit.wikimedia.org/r/507513 [22:39:55] (03PS1) 10Thcipriani: train-deploy-notes: update sha1; src dir; image [integration/config] - 10https://gerrit.wikimedia.org/r/507514 [22:42:54] (03CR) 10Jforrester: [C: 03+1] release-notes: TOOLS_RELEASE_DIR -> RELEASE_TOOLS_DIR [integration/config] - 10https://gerrit.wikimedia.org/r/507513 (owner: 10Thcipriani) [23:14:44] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10Wikidata, and 5 others: Introduce wikidata termbox SSR to kubernetes - https://phabricator.wikimedia.org/T220402 (10mobrovac) >>! In T220402#5146064, @Pablo-WMDE wrote: > @mobrovac During T221755 & T221754 we tended to [[ https://ssr-termbox.w... [23:59:47] 10Release-Engineering-Team, 10Developer Productivity, 10Epic: FY201819 TEC12 Program – Developer Productivity - https://phabricator.wikimedia.org/T212449 (10Jdforrester-WMF)