[04:28:50] RECOVERY - Free space - all mounts on deployment-deploy01 is OK: OK: All targets OK [05:14:52] (03PS1) 10DannyS712: Remove handling of expectedException annotations in two sniffs [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/588200 [05:15:48] (03PS2) 10DannyS712: Remove handling of expectedException annotations in two sniffs [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/588200 [05:21:11] (03CR) 10jerkins-bot: [V: 04-1] Remove handling of expectedException annotations in two sniffs [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/588200 (owner: 10DannyS712) [07:23:02] 10Project-Admins, 10User-DannyS712: Cleaning up #mediawiki-general - https://phabricator.wikimedia.org/T249999 (10Aklapper) >>! In T249999#6050865, @DannyS712 wrote: > @Krinkle #mediawiki-special-pages already exists - should it not? In an ideal world, it should not. > What about splitting blocking (#mediawi... [07:52:38] PROBLEM - Free space - all mounts on deployment-logstash2 is CRITICAL: CRITICAL: deployment-prep.deployment-logstash2.diskspace._mnt.byte_percentfree (No valid datapoints found) deployment-prep.deployment-logstash2.diskspace._var_lib_elasticsearch.byte_percentfree (No valid datapoints found)deployment-prep.deployment-logstash2.diskspace.root.byte_percentfree (<100.00%) [08:52:47] (03CR) 10Majavah: Skip empty sections in train changelogs (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/587971 (https://phabricator.wikimedia.org/T249553) (owner: 10Majavah) [09:37:28] (03PS2) 10Awight: [WIP] Composite commands can be named [integration/quibble] - 10https://gerrit.wikimedia.org/r/588142 [09:37:30] (03PS2) 10Awight: [WIP] Sequence builder [integration/quibble] - 10https://gerrit.wikimedia.org/r/588143 [09:37:32] (03PS1) 10Awight: [DNM] Experimental merge [integration/quibble] - 10https://gerrit.wikimedia.org/r/588374 [09:41:23] (03CR) 10Umherirrender: [C: 03+1] "It is hard to check for version, when the old way is deprecated." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/588167 (https://phabricator.wikimedia.org/T250040) (owner: 10DannyS712) [09:43:57] (03CR) 10Umherirrender: "I would leave in these annotation at the places." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/588200 (owner: 10DannyS712) [10:59:11] (03PS15) 10Awight: Parallelism as a command object; parallelize short PHPUnits [integration/quibble] - 10https://gerrit.wikimedia.org/r/587885 (https://phabricator.wikimedia.org/T235449) [11:00:10] (03CR) 10jerkins-bot: [V: 04-1] Parallelism as a command object; parallelize short PHPUnits [integration/quibble] - 10https://gerrit.wikimedia.org/r/587885 (https://phabricator.wikimedia.org/T235449) (owner: 10Awight) [11:00:21] (03PS16) 10Awight: Parallelism as a command object; parallelize short PHPUnits [integration/quibble] - 10https://gerrit.wikimedia.org/r/587885 (https://phabricator.wikimedia.org/T235449) [11:33:51] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [labs/tools/github-pr-closer] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/588389 [11:33:53] (03CR) 10QChris: [V: 03+2 C: 03+2] Allow “Gerrit Managers” to import history [labs/tools/github-pr-closer] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/588389 (owner: 10QChris) [11:34:01] (03PS1) 10QChris: Import done. Revoke import grants [labs/tools/github-pr-closer] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/588390 [11:34:04] (03CR) 10QChris: [V: 03+2 C: 03+2] Import done. Revoke import grants [labs/tools/github-pr-closer] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/588390 (owner: 10QChris) [11:48:16] 10Continuous-Integration-Config, 10Release-Engineering-Team (Unit & Int & System Tooling), 10MediaWiki-Core-Testing, 10Quibble, and 2 others: Quibble: Run PHPUnit databaseless and database stages in parallel - https://phabricator.wikimedia.org/T235449 (10awight) I've parallelized the three databaseless PHP... [11:51:50] (03PS3) 10Awight: [WIP] Sequence builder [integration/quibble] - 10https://gerrit.wikimedia.org/r/588143 [12:01:21] (03PS17) 10Awight: Parallelism as a command object; parallelize short PHPUnits [integration/quibble] - 10https://gerrit.wikimedia.org/r/587885 (https://phabricator.wikimedia.org/T235449) [12:03:04] (03PS1) 10QChris: layout: Enable tox for labs/tools/github-pr-closer [integration/config] - 10https://gerrit.wikimedia.org/r/588392 [12:03:59] (03CR) 10jerkins-bot: [V: 04-1] layout: Enable tox for labs/tools/github-pr-closer [integration/config] - 10https://gerrit.wikimedia.org/r/588392 (owner: 10QChris) [12:05:18] (03PS2) 10QChris: layout: Enable tox for labs/tools/github-pr-closer [integration/config] - 10https://gerrit.wikimedia.org/r/588392 [12:06:41] (03CR) 10Majavah: [C: 03+1] "Thank you!" [integration/config] - 10https://gerrit.wikimedia.org/r/588392 (owner: 10QChris) [12:06:57] (03CR) 10QChris: [C: 03+1] layout: Enable tox for labs/tools/github-pr-closer [integration/config] - 10https://gerrit.wikimedia.org/r/588392 (owner: 10QChris) [12:07:01] thanks qchris! [12:07:36] yw. Sorry for the slow response to the repo request :-( [12:07:53] Once the PR got approved, tox should be running (I hope) [12:08:11] thank you [12:23:03] PROBLEM - SSH on integration-agent-docker-1005 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:42:50] RECOVERY - SSH on integration-agent-docker-1005 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u7 (protocol 2.0) [12:59:15] PROBLEM - Free space - all mounts on deployment-deploy02 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy02.diskspace._srv.byte_percentfree (<11.11%) [13:26:44] 10Beta-Cluster-Infrastructure: Problems on deployment-hadoop-test-1 - https://phabricator.wikimedia.org/T250021 (10Ottomata) 05Open→03Resolved a:03Ottomata I don't think we need any of them. Deleted all 3. [13:28:18] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Quality-and-Test-Engineering-Team (QTE), 10User-zeljkofilipin: Cypress: add folders to jenkins artifacts - https://phabricator.wikimedia.org/T248373 (10Jpita) this is easily done by adding: `config.screenshotsFolder = pr... [13:28:40] 10Beta-Cluster-Infrastructure: Request admin and interface admin rights on zhwiki beta cluster - https://phabricator.wikimedia.org/T249819 (10gao_wenhai) @DannyS712 Admin request is no longer needed, but interface admin right is still needed. Here are several discussions about the interface left on Village pump... [13:29:07] 10Beta-Cluster-Infrastructure: Request interface admin rights on zhwiki beta cluster - https://phabricator.wikimedia.org/T249819 (10gao_wenhai) [13:29:15] PROBLEM - Host deployment-hadoop-test-2 is DOWN: CRITICAL - Host Unreachable (172.16.4.69) [13:30:07] PROBLEM - Host deployment-hadoop-test-3 is DOWN: CRITICAL - Host Unreachable (172.16.4.76) [13:32:04] PROBLEM - Host deployment-hadoop-test-1 is DOWN: CRITICAL - Host Unreachable (172.16.4.68) [13:35:23] 10Beta-Cluster-Infrastructure: Request temporary interface admin right on zhwiki beta cluster - https://phabricator.wikimedia.org/T249819 (10gao_wenhai) [13:37:15] 10Beta-Cluster-Infrastructure: Request temporary admin and interface admin right on zhwiki beta cluster - https://phabricator.wikimedia.org/T249819 (10gao_wenhai) [13:41:46] 10Beta-Cluster-Infrastructure: Request temporary admin and interface admin right on zhwiki beta cluster - https://phabricator.wikimedia.org/T249819 (10gao_wenhai) >>! In T249819#6051634, @gao_wenhai wrote: > @DannyS712 Here are several discussions about the interface left on Village pump on Chinese Wikipedia. I'... [14:39:17] 10Beta-Cluster-Infrastructure: Cannot edit on Beta Commons (user detection bug?) - https://phabricator.wikimedia.org/T250085 (10Mholloway) [14:40:28] 10Beta-Cluster-Infrastructure: Cannot edit on Beta Commons (user detection bug?) - https://phabricator.wikimedia.org/T250085 (10Mholloway) [15:16:26] PROBLEM - Host deployment-cache-upload05 is DOWN: CRITICAL - Host Unreachable (172.16.6.210) [15:19:03] PROBLEM - Host deployment-cache-text05 is DOWN: CRITICAL - Host Unreachable (172.16.4.21) [15:25:34] 10Project-Admins, 10User-DannyS712: Cleaning up #mediawiki-general - https://phabricator.wikimedia.org/T249999 (10Krinkle) >>! In T249999#6051058, @Aklapper wrote: >>>! In T249999#6050865, @DannyS712 wrote: >> @Krinkle #mediawiki-special-pages already exists - should it not? > > In an ideal world, it should n... [15:33:44] (03CR) 10Mainframe98: "> Patch Set 2:" (032 comments) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/588162 (https://phabricator.wikimedia.org/T249872) (owner: 10Mainframe98) [15:38:11] (03PS3) 10Mainframe98: Add a sniff to detect and fix whitespace in the array indexer brackets [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/588162 (https://phabricator.wikimedia.org/T249872) [15:50:16] RECOVERY - Free space - all mounts on deployment-deploy02 is OK: OK: All targets OK [16:02:42] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)): Consider what paging who should get for what in RelEng/EngProd? - https://phabricator.wikimedia.org/T247427 (10thcipriani) [16:21:39] 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Release, 10Train Deployments, 10User-brennen: 1.35.0-wmf.31 deployment blockers - https://phabricator.wikimedia.org/T249963 (10brennen) [16:29:21] 10Gerrit, 10Release-Engineering-Team, 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10User-brennen: delete-wmf-branches: Convert deleted wmf branches to tags - https://phabricator.wikimedia.org/T250095 (10brennen) [16:30:44] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10User-brennen: delete-wmf-branches: Convert deleted wmf branches to tags - https://phabricator.wikimedia.org/T250095 (10brennen) [16:50:50] 10Release-Engineering-Team, 10Repository-Admins, 10Product-Analytics (Kanban): Create a repository and user for Product Analytics Oozie jobs? - https://phabricator.wikimedia.org/T230743 (10mpopov) Update: Gerrit repo `analytics/wmf-product` has been created for us to use [16:57:06] (03CR) 10Jforrester: [C: 03+1] "This is rather neat." [integration/quibble] - 10https://gerrit.wikimedia.org/r/588092 (owner: 10Awight) [17:17:40] (03CR) 10Jforrester: [C: 03+2] layout: Enable tox for labs/tools/github-pr-closer [integration/config] - 10https://gerrit.wikimedia.org/r/588392 (owner: 10QChris) [17:18:51] (03Merged) 10jenkins-bot: layout: Enable tox for labs/tools/github-pr-closer [integration/config] - 10https://gerrit.wikimedia.org/r/588392 (owner: 10QChris) [17:28:38] 10Beta-Cluster-Infrastructure: Cannot edit on Beta Commons (user detection bug?) - https://phabricator.wikimedia.org/T250085 (10DannyS712) Can reproduce - same error shown, same IP detected for me as well [17:29:34] (03PS1) 10Dduvall: ui: Stick table row for easier scrolled reading [integration/reporting] - 10https://gerrit.wikimedia.org/r/588440 [17:29:53] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:30:12] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:30:57] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:32:06] 10Beta-Cluster-Infrastructure: Cannot edit on Beta Commons (user detection bug?) - https://phabricator.wikimedia.org/T250085 (10RhinosF1) 172.16.1.181 should never show as connecting to public internet. It’s part of the BOGON IPs. [17:35:23] James_F: ^ can we get the cluster back up? [17:35:30] RhinosF1: No idea. Can we? [17:36:12] James_F: Wasn’t it one of your changes that were linked to it. I think php-fpm reboot fixes it [17:36:30] Whether it’s editable based on that task, I’m not sure [17:36:32] "One of your changes"? [17:36:38] "It"? [17:36:48] Beta Cluster has hundreds of things wrong with it. [17:37:35] The mysterious every-now-and-then-Beta-Cluster-just-goes-away-then-comes-back issue (presuming that it's only one issue) is undiagnosed, AFAIAA. :-( [17:37:38] * RhinosF1 goes looking for the task [17:37:44] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Quibble, 10Regression: Jenkins no longer fails build if mw-error.log or mw-dberror.log are non-empty - https://phabricator.wikimedia.org/T246358 (10nnikkhoui) [17:37:50] (03CR) 10Kosta Harlan: [C: 03+1] Rewrite main pipeline as a generator [integration/quibble] - 10https://gerrit.wikimedia.org/r/588092 (owner: 10Awight) [17:38:07] deployment-mediawiki-07 is up and I can ssh into it… [17:38:28] And it's running Apache and php-fpm. [17:39:04] Project beta-scap-eqiad build #296104: 04FAILURE in 4 min 21 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/296104/ [17:39:06] !log Restarting deployment-mediawiki-07 to see if it comes back. [17:39:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:39:21] 10Beta-Cluster-Infrastructure: repeated deployment-mediawiki-07 socket timeouts - https://phabricator.wikimedia.org/T246577 (10RhinosF1) [17:39:23] 10Beta-Cluster-Infrastructure: App server process on deployment-mediawiki-07 keeps breaking - https://phabricator.wikimedia.org/T248246 (10RhinosF1) [17:39:52] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 93224 bytes in 8.904 second response time [17:39:55] James_F: https://phabricator.wikimedia.org/T246577#5933608 [17:40:07] (03CR) 10Kosta Harlan: "> Would it be useful to implement your concept incrementally, for example by only converting the webserver at first? This might dovetail " [integration/quibble] - 10https://gerrit.wikimedia.org/r/568931 (https://phabricator.wikimedia.org/T234902) (owner: 10Kosta Harlan) [17:40:13] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 52127 bytes in 8.689 second response time [17:40:34] 10Beta-Cluster-Infrastructure, 10Operations, 10Traffic, 10User-DannyS712: 502 error on beta commons - https://phabricator.wikimedia.org/T250103 (10DannyS712) [17:41:02] 10Beta-Cluster-Infrastructure, 10Operations, 10Traffic, 10User-DannyS712: 502 error on beta commons - https://phabricator.wikimedia.org/T250103 (10DannyS712) [17:41:14] 10Beta-Cluster-Infrastructure, 10Operations, 10Traffic, 10User-DannyS712: 502 error on beta commons - https://phabricator.wikimedia.org/T250103 (10RhinosF1) Caused by a reboot to fix T246577 (again) [17:41:43] James_F: ^ can be merged or closed right? [17:41:44] (03CR) 10Kosta Harlan: [C: 03+1] "No objections from me, thanks for the effort on this" [integration/quibble] - 10https://gerrit.wikimedia.org/r/546167 (owner: 10Awight) [17:42:05] RhinosF1: Oh. The CDB change of SRE's and K.rinkle's? Yeah, that's likely taking up much more memory. [17:42:20] James_F: It is [17:42:29] (03CR) 10Kosta Harlan: [C: 03+1] Extract execution decorator [integration/quibble] - 10https://gerrit.wikimedia.org/r/588006 (owner: 10Awight) [17:42:39] 10Beta-Cluster-Infrastructure, 10Operations, 10Traffic, 10User-DannyS712: 502 error on beta commons - https://phabricator.wikimedia.org/T250103 (10RhinosF1) 05Open→03Resolved [17:43:59] 10Beta-Cluster-Infrastructure, 10Operations, 10Traffic, 10User-DannyS712: 502 error on beta commons - https://phabricator.wikimedia.org/T250103 (10Jdforrester-WMF) >>! In T250103#6052721, @RhinosF1 wrote: > Caused by a reboot to fix T246577 (again) No? This task was *why* I restarted the instance. [17:44:41] 10Beta-Cluster-Infrastructure, 10Operations, 10Traffic, 10User-DannyS712: 502 error on beta commons - https://phabricator.wikimedia.org/T250103 (10RhinosF1) >>! In T250103#6052734, @Jdforrester-WMF wrote: >>>! In T250103#6052721, @RhinosF1 wrote: >> Caused by a reboot to fix T246577 (again) > > No? This t... [17:44:54] James_F: that’s why it went down though [17:44:58] 10Beta-Cluster-Infrastructure: Repeated deployment-mediawiki-07 socket timeouts ("Next Hop Connection Failed") - https://phabricator.wikimedia.org/T246577 (10Jdforrester-WMF) [17:45:28] When the box is down I think it gets a different error from Next Hop Connection Failed doesn't it? Connection Timeout or something. [17:45:50] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-07 is OK: HTTP OK: HTTP/1.1 200 OK - 92918 bytes in 0.991 second response time [17:45:53] James_F: it just never loaded for me before I got you to fix [17:47:31] Normally it fixes itself, but yes. [17:47:56] Writing the changes to scap to stop generating the CDB files (and thus massively reducing the footprint) is the next step. [17:48:02] It eventually fixes itself, I’ve seen it last upwards of an hour though before it recovers [17:48:12] But it's all stalled on working out if we're going ahead with the non-CDB world at all. :-( [17:48:22] Ah [17:48:56] Yippee, build fixed! [17:48:57] Project beta-scap-eqiad build #296105: 09FIXED in 4 min 20 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/296105/ [17:49:06] See T99740 [17:49:06] T99740: Use static php array files for l10n cache at WMF (instead of CDB) - https://phabricator.wikimedia.org/T99740 [17:49:07] * RhinosF1 now goes to wonder if only commons is affected by the BOGON block bug [17:49:17] Yeah I saw it a while back [17:49:59] Oh, oops. [17:50:08] !log Zuul: Add CI for labs/tools/github-pr-closer [17:50:09] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:50:11] * James_F coughs. [17:50:19] * RhinosF1 still can’t get on enwp beta [17:50:55] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:51:00] James_F: ^ [17:51:14] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:51:25] 10Beta-Cluster-Infrastructure: Cannot edit on Beta Commons (user detection bug?) - https://phabricator.wikimedia.org/T250085 (10bd808) >>! In T250085#6052659, @RhinosF1 wrote: > [172.16.1.181](https://ipinfo.io/172.16.1.181) should never show as connecting to public internet. It’s part of the [BOGON](https://en.... [17:51:28] Hmm. Puppet on deployment-mediawiki-09 has been disabled for 7 weeks?! [17:51:32] That seems bad. [17:51:45] And probably has caused random issues. [17:51:57] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:53:28] Krenair: Don't suppose you're around? :-) [17:55:05] bd808: Surely something recently changed, or we’d have had that issue with (auto)blocks before [17:55:45] Hi, I see exciting things happening. Is this only affecting deployment-prep, or should we delay the SWAT window? [17:56:29] awight: Just Beta Cluster being Beta Cluster. [17:56:43] +1 thanks! [17:57:15] RhinosF1: quite possibly. there is a lot of ongoing work in the production CDN layer and something could have been changed there via Puppet that flowed into deployment-prep without some needed configuration updates. [17:57:38] RhinosF1: this is pretty much the long standing challenge with deployment-prep [17:57:42] Yup. :-( [17:57:51] It just about scaled when SRE was five people. [17:58:05] At least it’s in beta cluster and not anything else [17:58:17] * bd808 lost 3 months of his work life to keep deployment-prep working during the php5.3->HHVM migration [18:00:46] (03PS1) 10Dduvall: Include basic README and license [integration/reporting] - 10https://gerrit.wikimedia.org/r/588443 [18:02:48] James_F: not getting a hop error on commons [18:02:55] Show Request from 86.160.232.142 via deployment-cache-text06 frontend, Varnish XID 2658779 [18:02:55] Error: 503, Backend fetch failed at Mon, 13 Apr 2020 18:01:48 GMT instead [18:03:03] Ah, interesting. [18:03:30] !log Re-enabling puppet agent on deployment-mediawiki-09; disabled for 7 weeks with no comment. [18:03:33] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:03:40] James_F: I’ve seen it just ERR_CONNECTION_FAILED as well before [18:04:03] (03PS1) 10Dduvall: zuul: noop test/gate-and-submit for new integration/reporting repo [integration/config] - 10https://gerrit.wikimedia.org/r/588444 [18:04:22] !log Restarting deployment-mediawiki-09 too. [18:04:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:04:24] Enwp is ERR_CONNECTION_TIMED_OUT [18:04:53] RhinosF1: Please *NEVER* say that if you mean Beta Cluster. [18:05:09] James_F: Enwp bêta cluster [18:05:19] Better. :-) [18:05:27] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-09 is CRITICAL: connect to address 172.16.4.106 and port 80: Connection refused [18:06:26] * James_F sighs. [18:07:07] Looks like deployment-mediawiki-09 didn't come back up. [18:07:36] :( [18:07:53] PROBLEM - Host deployment-mediawiki-09 is DOWN: CRITICAL - Host Unreachable (172.16.4.106) [18:08:00] Correct, shinken-wm. [18:08:30] * James_F wonders who he has to kill to get horizon to remember the "hosts per page" setting for more than an hour or two. [18:08:55] lol [18:09:08] I mean, remembering my login would be nice too. [18:09:19] But I have my 2FA devices close to me all the time. [18:09:48] (03CR) 10Dduvall: [C: 03+2] zuul: noop test/gate-and-submit for new integration/reporting repo [integration/config] - 10https://gerrit.wikimedia.org/r/588444 (owner: 10Dduvall) [18:09:48] I think there's known bugs for the sessions [18:09:51] Aha, deployment-mediawiki-09 is now up. [18:10:15] marxarelli: Hah. [18:10:39] (03Merged) 10jenkins-bot: zuul: noop test/gate-and-submit for new integration/reporting repo [integration/config] - 10https://gerrit.wikimedia.org/r/588444 (owner: 10Dduvall) [18:10:57] James_F: :) [18:11:31] * James_F keeps wanting that job to be `no-op`. [18:11:43] Or `nöop`. [18:12:32] Hi folks, somehow I don't have +2 to the Gerrit repo created today for my Toolforge tool, can someone fix this :/ [18:12:52] RECOVERY - Host deployment-mediawiki-09 is UP: PING OK - Packet loss = 0%, RTA = 1.84 ms [18:13:06] The repo in question being https://gerrit.wikimedia.org/r/#/admin/projects/labs/tools/github-pr-closer and my Gerrit username is Majavah [18:13:48] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/588444 [18:13:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:14:17] Majavah: Ask Christian? Looks like he's not finished that step yet? https://gerrit.wikimedia.org/r/admin/groups/808797d1bd5e9dc23155a32418f2268d513af369,members [18:14:35] Huh? Did I overlook something? /me reads backlog. [18:14:40] Not sure if that's intentional or … aha. [18:14:46] Hey qchris. ;-) [18:14:54] qchris: you forgot to give me access to my own Gerrit repo :p [18:15:00] +2 access I mean [18:15:08] Let me check. I will give you +2 of course. [18:15:38] Majavah: Can you try again? [18:15:55] qchris: works now, thank you :D [18:16:09] Sorry. I meant to grant you +2. [18:16:20] Glad it works now. [18:16:32] no worries [18:27:01] (03CR) 10Jforrester: [C: 03+2] "Not sure I'd call myself a 'pro', but sure." [integration/quibble] - 10https://gerrit.wikimedia.org/r/587896 (owner: 10C. Scott Ananian) [18:28:10] (03CR) 10Jforrester: [C: 03+2] Add problematic values 1 and 1.0 to PHPUnitAssertEqualsSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/576045 (https://phabricator.wikimedia.org/T246662) (owner: 10Thiemo Kreuz (WMDE)) [18:28:25] (03Merged) 10jenkins-bot: Map mediawiki/services/parsoid to /workspace/src/services/parsoid [integration/quibble] - 10https://gerrit.wikimedia.org/r/587896 (owner: 10C. Scott Ananian) [18:28:54] (03Merged) 10jenkins-bot: Add problematic values 1 and 1.0 to PHPUnitAssertEqualsSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/576045 (https://phabricator.wikimedia.org/T246662) (owner: 10Thiemo Kreuz (WMDE)) [18:30:53] (03CR) 10Jforrester: [C: 03+2] Extract execution decorator [integration/quibble] - 10https://gerrit.wikimedia.org/r/588006 (owner: 10Awight) [18:31:45] (03Merged) 10jenkins-bot: Extract execution decorator [integration/quibble] - 10https://gerrit.wikimedia.org/r/588006 (owner: 10Awight) [18:33:13] (03CR) 10Jforrester: [C: 03+1] "The other minor issue is that the documentation is no longer quite as in-line as before, so changes to arguments may not prompt changes to" [integration/quibble] - 10https://gerrit.wikimedia.org/r/546167 (owner: 10Awight) [18:36:49] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-07 is OK: HTTP OK: HTTP/1.1 200 OK - 92919 bytes in 1.417 second response time [18:40:42] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 93224 bytes in 1.064 second response time [18:41:04] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 52126 bytes in 1.212 second response time [18:57:04] (03PS1) 10Jforrester: layout: [mediawiki/extensions/DiscussionTools] Add JS documentation template [integration/config] - 10https://gerrit.wikimedia.org/r/588458 (https://phabricator.wikimedia.org/T249869) [19:01:03] (03PS1) 10Jforrester: layout: [mediawiki/extensions/DiscussionTools] Add JS code coverage [integration/config] - 10https://gerrit.wikimedia.org/r/588460 (https://phabricator.wikimedia.org/T249869) [19:11:10] (03CR) 10Awight: "> The other minor issue is that the documentation is no longer quite" [integration/quibble] - 10https://gerrit.wikimedia.org/r/546167 (owner: 10Awight) [19:26:21] (03PS1) 10Brennen Bearnes: (DNM) delete-wmf-branches: tag at branchpoints [tools/release] - 10https://gerrit.wikimedia.org/r/588467 (https://phabricator.wikimedia.org/T250095) [19:33:36] 10Gerrit, 10Code-Review-Workgroup, 10Discovery, 10User-DannyS712: Cindy-the-browser-test-bot should use verified, not code review - https://phabricator.wikimedia.org/T249981 (10EBernhardson) At some point it was doing V+1 but there was a problem, unfortunately it's been so long I can't find what it was. Wh... [19:37:32] (03PS3) 10Awight: PEP 8: underscore prefix for internal functions [integration/quibble] - 10https://gerrit.wikimedia.org/r/588177 [19:37:52] (03CR) 10Awight: "PS 3: manual rebase" [integration/quibble] - 10https://gerrit.wikimedia.org/r/588177 (owner: 10Awight) [19:51:23] (03PS2) 10Brennen Bearnes: (DNM) delete-wmf-branches: tag at branchpoints [tools/release] - 10https://gerrit.wikimedia.org/r/588467 (https://phabricator.wikimedia.org/T250095) [19:57:50] PROBLEM - Free space - all mounts on deployment-memc06 is CRITICAL: (Service Check Timed Out) [19:58:51] 10phan, 10MobileFrontend, 10Parsing-Team, 10Parsoid: Duplicate MobileContext phan stub caused problems - https://phabricator.wikimedia.org/T250111 (10Reedy) [20:01:22] PROBLEM - Free space - all mounts on integration-agent-docker-1014 is CRITICAL: (Service Check Timed Out) [20:01:28] PROBLEM - Free space - all mounts on deployment-aqs03 is CRITICAL: (Service Check Timed Out) [20:01:33] PROBLEM - Free space - all mounts on deployment-maps05 is CRITICAL: (Service Check Timed Out) [20:02:41] 10phan, 10MobileFrontend, 10Parsing-Team, 10Parsoid: Duplicate MobileContext phan stub caused problems - https://phabricator.wikimedia.org/T250111 (10Reedy) Subbu deleted this in https://gerrit.wikimedia.org/r/#/c/mediawiki/vendor/+/588471/ in vendor... Leaving the task open for this to be solved in the lo... [20:02:42] RECOVERY - Free space - all mounts on deployment-memc06 is OK: OK: All targets OK [20:05:20] PROBLEM - Free space - all mounts on deployment-elastic07 is CRITICAL: (Service Check Timed Out) [20:06:12] RECOVERY - Free space - all mounts on integration-agent-docker-1014 is OK: OK: All targets OK [20:06:20] RECOVERY - Free space - all mounts on deployment-aqs03 is OK: OK: All targets OK [20:06:21] RECOVERY - Free space - all mounts on deployment-maps05 is OK: OK: All targets OK [20:09:57] RECOVERY - Free space - all mounts on deployment-etcd-01 is OK: OK: All targets OK [20:10:09] RECOVERY - Free space - all mounts on deployment-elastic07 is OK: OK: deployment-prep.deployment-elastic07.diskspace._var_lib_elasticsearch.byte_percentfree (No valid datapoints found) deployment-prep.deployment-elastic07.diskspace._var_log.byte_percentfree (No valid datapoints found) [20:10:53] (03CR) 10Awight: [C: 04-1] "I discovered this isn't capturing when the child fails. Hopefully it's an easy fix..." [integration/quibble] - 10https://gerrit.wikimedia.org/r/587885 (https://phabricator.wikimedia.org/T235449) (owner: 10Awight) [20:15:23] PROBLEM - Free space - all mounts on integration-agent-docker-1007 is CRITICAL: (Service Check Timed Out) [20:17:45] PROBLEM - Free space - all mounts on deployment-poolcounter06 is CRITICAL: (Service Check Timed Out) [20:20:07] (03PS18) 10Awight: Parallelism as a command object; parallelize short PHPUnits [integration/quibble] - 10https://gerrit.wikimedia.org/r/587885 (https://phabricator.wikimedia.org/T235449) [20:20:10] RECOVERY - Free space - all mounts on integration-agent-docker-1007 is OK: OK: All targets OK [20:21:12] (03CR) 10jerkins-bot: [V: 04-1] Parallelism as a command object; parallelize short PHPUnits [integration/quibble] - 10https://gerrit.wikimedia.org/r/587885 (https://phabricator.wikimedia.org/T235449) (owner: 10Awight) [20:22:39] RECOVERY - Free space - all mounts on deployment-poolcounter06 is OK: OK: All targets OK [20:24:22] RECOVERY - Free space - all mounts on deployment-cumin02 is OK: OK: All targets OK [20:37:38] (03CR) 10Jforrester: [C: 03+2] Provide GitClean as a command [integration/quibble] - 10https://gerrit.wikimedia.org/r/588082 (owner: 10Awight) [20:38:28] (03CR) 10Hashar: Map mediawiki/services/parsoid to /workspace/src/services/parsoid (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/587896 (owner: 10C. Scott Ananian) [20:38:30] (03Merged) 10jenkins-bot: Provide GitClean as a command [integration/quibble] - 10https://gerrit.wikimedia.org/r/588082 (owner: 10Awight) [20:42:15] (03PS2) 10Jforrester: Logspam: Set Flow's default content format to wikitext [integration/quibble] - 10https://gerrit.wikimedia.org/r/553305 (owner: 10Kosta Harlan) [20:42:53] (03CR) 10Jforrester: "Fixed the double ',', but does Parsoid work in quibble now that it's a PHP service?" [integration/quibble] - 10https://gerrit.wikimedia.org/r/553305 (owner: 10Kosta Harlan) [20:46:03] (03PS2) 10Jforrester: Clone only the target project at first [integration/quibble] - 10https://gerrit.wikimedia.org/r/588146 (https://phabricator.wikimedia.org/T211702) (owner: 10Awight) [20:48:13] (03CR) 10Jforrester: [C: 03+2] Clone only the target project at first [integration/quibble] - 10https://gerrit.wikimedia.org/r/588146 (https://phabricator.wikimedia.org/T211702) (owner: 10Awight) [20:49:07] (03Merged) 10jenkins-bot: Clone only the target project at first [integration/quibble] - 10https://gerrit.wikimedia.org/r/588146 (https://phabricator.wikimedia.org/T211702) (owner: 10Awight) [20:49:54] (03CR) 10Jforrester: [C: 03+1] "> Patch Set 15:" [integration/quibble] - 10https://gerrit.wikimedia.org/r/546167 (owner: 10Awight) [20:53:15] (03PS1) 10Jforrester: layout: [mediawiki/extensions/CirrusSearch] Support standalone [integration/config] - 10https://gerrit.wikimedia.org/r/588488 (https://phabricator.wikimedia.org/T221434) [21:02:15] (03CR) 10Jforrester: [C: 03+2] layout: [mediawiki/extensions/CirrusSearch] Support standalone [integration/config] - 10https://gerrit.wikimedia.org/r/588488 (https://phabricator.wikimedia.org/T221434) (owner: 10Jforrester) [21:03:07] (03Merged) 10jenkins-bot: layout: [mediawiki/extensions/CirrusSearch] Support standalone [integration/config] - 10https://gerrit.wikimedia.org/r/588488 (https://phabricator.wikimedia.org/T221434) (owner: 10Jforrester) [21:03:18] !log Zuul: [mediawiki/extensions/CirrusSearch] Support standalone [21:03:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:06:19] (03PS19) 10Awight: Parallelism as a command object; parallelize short PHPUnits [integration/quibble] - 10https://gerrit.wikimedia.org/r/587885 (https://phabricator.wikimedia.org/T235449) [21:06:26] (03CR) 10jerkins-bot: [V: 04-1] Parallelism as a command object; parallelize short PHPUnits [integration/quibble] - 10https://gerrit.wikimedia.org/r/587885 (https://phabricator.wikimedia.org/T235449) (owner: 10Awight) [21:13:43] (03PS1) 10Zoranzoki21: Archive the MSSQLBackCompat extension [integration/config] - 10https://gerrit.wikimedia.org/r/588490 (https://phabricator.wikimedia.org/T248965) [21:15:31] (03PS2) 10Zoranzoki21: Archive the MSSQLBackCompat extension [integration/config] - 10https://gerrit.wikimedia.org/r/588490 (https://phabricator.wikimedia.org/T248965) [21:16:28] (03CR) 10Awight: "> (This now needs rebasing, sorry.)" [integration/quibble] - 10https://gerrit.wikimedia.org/r/546167 (owner: 10Awight) [21:17:33] (03CR) 10Jforrester: Parallelism as a command object; parallelize short PHPUnits (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/587885 (https://phabricator.wikimedia.org/T235449) (owner: 10Awight) [21:18:26] (03CR) 10Jforrester: [C: 03+2] Archive the MSSQLBackCompat extension [integration/config] - 10https://gerrit.wikimedia.org/r/588490 (https://phabricator.wikimedia.org/T248965) (owner: 10Zoranzoki21) [21:19:03] (03PS20) 10Awight: Parallelism as a command object; parallelize short PHPUnits [integration/quibble] - 10https://gerrit.wikimedia.org/r/587885 (https://phabricator.wikimedia.org/T235449) [21:19:50] (03Merged) 10jenkins-bot: Archive the MSSQLBackCompat extension [integration/config] - 10https://gerrit.wikimedia.org/r/588490 (https://phabricator.wikimedia.org/T248965) (owner: 10Zoranzoki21) [21:20:30] !log Zuul: Archive the MSSQLBackCompat extension T248965 [21:20:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:20:35] T248965: Archive the MSSQLBackCompat extension - https://phabricator.wikimedia.org/T248965 [21:20:53] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, 10Patch-For-Review: Ensure we're testing appropriately and not over-testing across Wikimedia-deployed code - https://phabricator.wikimedia.org/T221434 (10Jdforrester-WMF) I've added [[https:... [21:22:59] (03CR) 10Awight: Parallelism as a command object; parallelize short PHPUnits (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/587885 (https://phabricator.wikimedia.org/T235449) (owner: 10Awight) [21:26:55] (03CR) 10Awight: [C: 03+1] "Just to mention it, there's a TODO for an important unit test still, that we short-circuit all parallelized jobs when the first one fails." [integration/quibble] - 10https://gerrit.wikimedia.org/r/587885 (https://phabricator.wikimedia.org/T235449) (owner: 10Awight) [21:31:26] (03PS16) 10Awight: Replace argparse with docopt [integration/quibble] - 10https://gerrit.wikimedia.org/r/546167 [21:31:40] (03CR) 10C. Scott Ananian: "> Not sure I'd call myself a 'pro', but sure." [integration/quibble] - 10https://gerrit.wikimedia.org/r/587896 (owner: 10C. Scott Ananian) [21:34:57] (03CR) 10Awight: "PS 16: rebase" [integration/quibble] - 10https://gerrit.wikimedia.org/r/546167 (owner: 10Awight) [21:36:45] (03PS3) 10Awight: Rewrite main pipeline as a generator [integration/quibble] - 10https://gerrit.wikimedia.org/r/588092 [21:37:12] (03CR) 10Awight: "PS 3: rebase" [integration/quibble] - 10https://gerrit.wikimedia.org/r/588092 (owner: 10Awight) [21:38:21] (03PS1) 10Zoranzoki21: Archive the OracleTextSearch extension [integration/config] - 10https://gerrit.wikimedia.org/r/588496 (https://phabricator.wikimedia.org/T249656) [21:38:57] (03PS2) 10Zoranzoki21: Archive the OracleTextSearch extension [integration/config] - 10https://gerrit.wikimedia.org/r/588496 (https://phabricator.wikimedia.org/T249656) [21:40:40] (03PS2) 10Awight: Simplify shell call syntax [integration/quibble] - 10https://gerrit.wikimedia.org/r/588113 [21:42:25] (03PS4) 10Awight: PEP 8: underscore prefix for internal functions [integration/quibble] - 10https://gerrit.wikimedia.org/r/588177 [21:43:43] (03CR) 10Jforrester: "> Patch Set 3:" [integration/quibble] - 10https://gerrit.wikimedia.org/r/587896 (owner: 10C. Scott Ananian) [21:44:22] (03PS3) 10Awight: Simplify shell call syntax [integration/quibble] - 10https://gerrit.wikimedia.org/r/588113 [21:45:26] (03CR) 10Dduvall: [C: 03+2] Include basic README and license [integration/reporting] - 10https://gerrit.wikimedia.org/r/588443 (owner: 10Dduvall) [21:45:32] (03Merged) 10jenkins-bot: Include basic README and license [integration/reporting] - 10https://gerrit.wikimedia.org/r/588443 (owner: 10Dduvall) [22:03:47] (03PS8) 10Awight: Split extension and skin npm and composer tests [integration/quibble] - 10https://gerrit.wikimedia.org/r/587888 [22:03:49] (03PS2) 10Awight: Split core npm and composer tests [integration/quibble] - 10https://gerrit.wikimedia.org/r/588087 [22:03:52] (03PS3) 10Awight: Sequence of commands as a command [integration/quibble] - 10https://gerrit.wikimedia.org/r/588083 [22:03:54] (03PS8) 10Awight: Clean-up: remove old parallel_run [integration/quibble] - 10https://gerrit.wikimedia.org/r/587890 [22:03:56] (03PS3) 10Awight: Wrap main pipeline in a Sequence [integration/quibble] - 10https://gerrit.wikimedia.org/r/588093 (https://phabricator.wikimedia.org/T249775) [22:04:23] Apologies for the spam [22:05:01] PROBLEM - Free space - all mounts on deployment-snapshot01 is CRITICAL: CRITICAL: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found)deployment-prep.deployment-snapshot01.diskspace.root.byte_percentfree (<10.00%) [22:05:10] (03CR) 10jerkins-bot: [V: 04-1] Clean-up: remove old parallel_run [integration/quibble] - 10https://gerrit.wikimedia.org/r/587890 (owner: 10Awight) [22:07:51] (03CR) 10Awight: "recheck" [integration/quibble] - 10https://gerrit.wikimedia.org/r/587890 (owner: 10Awight) [22:15:01] RECOVERY - Free space - all mounts on deployment-snapshot01 is OK: OK: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found) [22:32:30] (03CR) 10Awight: "I don't know what the test failure is about, but slightly concerned that it shows flapping related to the "Parallel" command, meaning ther" [integration/quibble] - 10https://gerrit.wikimedia.org/r/587890 (owner: 10Awight) [22:39:03] (03PS2) 10Awight: [DNM] Experimental merge [integration/quibble] - 10https://gerrit.wikimedia.org/r/588374 [22:39:18] (03CR) 10jerkins-bot: [V: 04-1] [DNM] Experimental merge [integration/quibble] - 10https://gerrit.wikimedia.org/r/588374 (owner: 10Awight) [22:42:02] (03CR) 10Awight: "recheck" [integration/quibble] - 10https://gerrit.wikimedia.org/r/588374 (owner: 10Awight) [22:43:40] 10Continuous-Integration-Config, 10Code-Health, 10Readers-Web-Backlog (Kanbanana-2019-20-Q4), 10Vue.js: Configure ESLint for Vue.js search development ahead of it being done for all repos - https://phabricator.wikimedia.org/T249304 (10Jdforrester-WMF) [22:44:04] 10Continuous-Integration-Config, 10Code-Health, 10Readers-Web-Backlog (Kanbanana-2019-20-Q4), 10Vue.js: Configure ESLint for Vue.js search development ahead of it being done for all repos - https://phabricator.wikimedia.org/T249304 (10Jdforrester-WMF) `eslint-plugin-no-jquery` is a fork of `eslint-plugin-j... [22:52:09] Gerrit file uploading in the UI has been merged - https://gerrit-review.googlesource.com/c/gerrit/+/246332 [23:41:57] 10phan, 10phan-taint-check-plugin, 10Patch-For-Review: Analyze new phan issues brought by taint-check - https://phabricator.wikimedia.org/T249934 (10Daimona)