[00:10:12] maintenance-disconnect-full-disks build 62050 integration-slave-docker-1034: OFFLINE due to disk space [00:23:37] (03PS10) 10Dduvall: WIP Provide configurable pipeline execution [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/499918 [00:24:16] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/499918 (owner: 10Dduvall) [00:27:56] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10MediaWiki-extensions-MultimediaViewer, 10MobileFrontend, and 9 others: Audit tests/selenium/LocalSettings.php file aiming at possibly deprecating the feature - https://phabricator.wikimedia.org/T199939 (10Krinkle) >>! In T199939#5095838, @Jdlrobs... [00:29:42] https://phabricator.wikimedia.org/p/GMoney0305/ [00:29:44] a user without SUL or LDAP login? [00:33:24] [Access Denied: Restricted Application] (Can View) You do not have permission to view this object. // Logged in users can take this action. [00:33:26] Huh [00:35:12] maintenance-disconnect-full-disks build 62055 integration-slave-docker-1034: OFFLINE due to disk space [00:38:41] 10Release-Engineering-Team, 10Analytics, 10ExternalGuidance, 10MW-1.33-notes (1.33.0-wmf.24; 2019-04-02), and 3 others: Unknown dependency: mw.externalguidance.init - https://phabricator.wikimedia.org/T219841 (10Jdlrobson) cc analytics and RelEng as there's potential for a lot of traffic to EventLogging (I... [00:42:56] Where's that from paladox? [00:43:07] oh, you tried to view a profile while logged out? [00:43:16] Krenair: from the link you pasted [00:43:18] And yup [00:43:20] oh I see [00:43:27] While logged in I see 'Template "" is not a valid template.' [00:43:31] while logged out I get what you posted [00:43:49] comment content can change depending on the user's logged in status... of course it can. [00:43:58] viewer's* [00:44:49] Did they set a view policy on what ever they pasted? [01:00:13] maintenance-disconnect-full-disks build 62060 integration-slave-docker-1034: OFFLINE due to disk space [01:25:11] maintenance-disconnect-full-disks build 62065 integration-slave-docker-1034: OFFLINE due to disk space [01:50:12] maintenance-disconnect-full-disks build 62070 integration-slave-docker-1034: OFFLINE due to disk space [02:15:11] maintenance-disconnect-full-disks build 62075 integration-slave-docker-1034: OFFLINE due to disk space [02:18:32] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10MediaWiki-Documentation, 10Regression, 10Upstream: Doxygen search.php no longer works on doc.wikimedia.org - https://phabricator.wikimedia.org/T218233 (10Krinkle) [02:40:11] maintenance-disconnect-full-disks build 62080 integration-slave-docker-1034: OFFLINE due to disk space [02:41:51] 10Release-Engineering-Team, 10Analytics, 10ExternalGuidance, 10MW-1.33-notes (1.33.0-wmf.25; 2019-04-09), and 3 others: Unknown dependency: mw.externalguidance.init - https://phabricator.wikimedia.org/T219841 (10Krinkle) p:05Unbreak!→03High [03:05:12] maintenance-disconnect-full-disks build 62085 integration-slave-docker-1034: OFFLINE due to disk space [03:23:57] (03PS2) 10Krinkle: ContentTranslation: Remove Wikibase and GuidedTour CI dependency [integration/config] - 10https://gerrit.wikimedia.org/r/500810 (https://phabricator.wikimedia.org/T219713) (owner: 10Santhosh) [03:24:00] (03CR) 10Krinkle: [C: 03+2] ContentTranslation: Remove Wikibase and GuidedTour CI dependency [integration/config] - 10https://gerrit.wikimedia.org/r/500810 (https://phabricator.wikimedia.org/T219713) (owner: 10Santhosh) [03:25:38] (03Merged) 10jenkins-bot: ContentTranslation: Remove Wikibase and GuidedTour CI dependency [integration/config] - 10https://gerrit.wikimedia.org/r/500810 (https://phabricator.wikimedia.org/T219713) (owner: 10Santhosh) [03:30:11] maintenance-disconnect-full-disks build 62090 integration-slave-docker-1034: OFFLINE due to disk space [03:55:11] maintenance-disconnect-full-disks build 62095 integration-slave-docker-1034: OFFLINE due to disk space [04:11:00] 10Release-Engineering-Team, 10Analytics, 10ExternalGuidance, 10MW-1.33-notes (1.33.0-wmf.24; 2019-04-02), and 3 others: Unknown dependency: mw.externalguidance.init - https://phabricator.wikimedia.org/T219841 (10Jdlrobson) Thank you @krinkle for taking care of this. [04:20:10] maintenance-disconnect-full-disks build 62100 integration-slave-docker-1034: OFFLINE due to disk space [04:39:52] 10Release-Engineering-Team, 10Analytics, 10ExternalGuidance, 10MW-1.33-notes (1.33.0-wmf.24; 2019-04-02), and 3 others: Unknown dependency: mw.externalguidance.init - https://phabricator.wikimedia.org/T219841 (10santhosh) Thanks @Jdlrobson and @Krinkle. [04:45:11] maintenance-disconnect-full-disks build 62105 integration-slave-docker-1034: OFFLINE due to disk space [05:10:12] maintenance-disconnect-full-disks build 62110 integration-slave-docker-1034: OFFLINE due to disk space [05:35:11] maintenance-disconnect-full-disks build 62115 integration-slave-docker-1034: OFFLINE due to disk space [06:00:10] maintenance-disconnect-full-disks build 62120 integration-slave-docker-1034: OFFLINE due to disk space [06:25:15] maintenance-disconnect-full-disks build 62125 integration-slave-docker-1034: OFFLINE due to disk space [06:50:10] maintenance-disconnect-full-disks build 62130 integration-slave-docker-1034: OFFLINE due to disk space [07:15:11] maintenance-disconnect-full-disks build 62135 integration-slave-docker-1034: OFFLINE due to disk space [07:40:12] maintenance-disconnect-full-disks build 62140 integration-slave-docker-1034: OFFLINE due to disk space [08:08:11] maintenance-disconnect-full-disks build 62145 integration-slave-docker-1034: OFFLINE due to disk space [08:30:15] maintenance-disconnect-full-disks build 62150 integration-slave-docker-1034: OFFLINE due to disk space [08:34:14] 10Release-Engineering-Team (Kanban), 10LDAP-Access-Requests, 10User-greg: Add legoktm to gerritadmin LDAP group (restoring previously held access) - https://phabricator.wikimedia.org/T219086 (10fgiunchedi) p:05Triage→03Normal [08:37:45] 10Continuous-Integration-Infrastructure, 10Operations: Upload Zuul 2.5.1-wmf6 package to apt.wikimedia.org - https://phabricator.wikimedia.org/T220380 (10fgiunchedi) p:05Triage→03Normal [08:38:06] 10Release-Engineering-Team (Backlog), 10Operations: mwdebug2001 and mwdebug2002 "/" almost full - https://phabricator.wikimedia.org/T219989 (10fgiunchedi) p:05Triage→03Normal [08:38:25] 10Phabricator, 10Release-Engineering-Team, 10Operations, 10Wikimedia-Incident: Analyze and amend (if necessary) workflow of user reporting and detecting large regressions/outages - https://phabricator.wikimedia.org/T219589 (10fgiunchedi) p:05Triage→03Normal [08:55:11] maintenance-disconnect-full-disks build 62155 integration-slave-docker-1034: OFFLINE due to disk space [09:34:36] 10Release-Engineering-Team, 10MediaWiki-extensions-LocalisationUpdate, 10Security: Reenable l10update in production - https://phabricator.wikimedia.org/T220481 (10Lucas_Werkmeister_WMDE) [09:36:11] 10Release-Engineering-Team, 10MediaWiki-extensions-LocalisationUpdate, 10Security: Reenable l10update in production - https://phabricator.wikimedia.org/T220481 (10Lucas_Werkmeister_WMDE) Side note: [LocalisationUpdate SAL entries](https://tools.wmflabs.org/sal/production?q=LocalisationUpdate) have apparently... [09:45:49] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 64.29% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [10:08:13] 10Continuous-Integration-Config, 10Growth-Team, 10StructuredDiscussions, 10Patch-For-Review, 10User-kostajh: Fix Flow random test failures - https://phabricator.wikimedia.org/T208988 (10hashar) >>! In T208988#5034498, @Tgr wrote: > Same issue in https://integration.wikimedia.org/ci/job/wmf-quibble-core-v... [10:14:49] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog), 10Patch-For-Review: Remove the "check" pipeline and Zuul's user-filter - https://phabricator.wikimedia.org/T192217 (10hashar) @EddieGP I wanted to remove the check pipeline, notably because most of those jobs were legacy anyway an... [10:33:09] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [10:34:53] 10Gerrit, 10Phabricator, 10Release-Engineering-Team (Backlog): Stop using Differential for code review - https://phabricator.wikimedia.org/T191182 (10hashar) We could do something similar with Gerrit. Roughly: * create the git repository under a Gerrit namespace such as `labs/tools/` or `wmcs/toolforge` or... [11:18:04] 10Release-Engineering-Team, 10MediaWiki-extensions-LocalisationUpdate, 10Security: Reenable l10update in production - https://phabricator.wikimedia.org/T220481 (10Peachey88) There has been runs since 2015: https://tools.wmflabs.org/sal/production?p=0&q=l10nupdate&d= Reedy manually ran it on 2017-01-23, i su... [11:35:46] (03PS1) 10Zfilipin: WIP Create selenium-daily-beta-MediaWiki targeting commons.wikimedia.beta.wmflabs.org [integration/config] - 10https://gerrit.wikimedia.org/r/502470 (https://phabricator.wikimedia.org/T219815) [12:47:17] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 64.29% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [13:40:37] PROBLEM - zuul_merger_service_running on contint1001 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/share/python/zuul/bin/python /usr/bin/zuul-merger https://www.mediawiki.org/wiki/Continuous_integration/Zuul [13:41:55] RECOVERY - zuul_merger_service_running on contint1001 is OK: PROCS OK: 1 process with regex args ^/usr/share/python/zuul/bin/python /usr/bin/zuul-merger https://www.mediawiki.org/wiki/Continuous_integration/Zuul [14:06:57] thcipriani: o/ i don't seem to have theh create new project option in gerrit anymore [14:06:59] should I not? [14:12:48] (03PS1) 10Cparle: Added missing dependency for WikibaseMediaInfo [integration/config] - 10https://gerrit.wikimedia.org/r/502497 [14:18:41] (03CR) 10Hashar: [C: 03+2] Added missing dependency for WikibaseMediaInfo [integration/config] - 10https://gerrit.wikimedia.org/r/502497 (owner: 10Cparle) [14:20:51] (03CR) 10jerkins-bot: [V: 04-1] Added missing dependency for WikibaseMediaInfo [integration/config] - 10https://gerrit.wikimedia.org/r/502497 (owner: 10Cparle) [14:21:47] (03CR) 10jerkins-bot: [V: 04-1] Added missing dependency for WikibaseMediaInfo [integration/config] - 10https://gerrit.wikimedia.org/r/502497 (owner: 10Cparle) [14:22:22] (03CR) 10Hashar: "I have deployed the config change" [integration/config] - 10https://gerrit.wikimedia.org/r/500810 (https://phabricator.wikimedia.org/T219713) (owner: 10Santhosh) [14:26:29] (03PS2) 10Hashar: Added missing dependency for WikibaseMediaInfo [integration/config] - 10https://gerrit.wikimedia.org/r/502497 (owner: 10Cparle) [14:26:47] (03CR) 10Hashar: [C: 03+2] Added missing dependency for WikibaseMediaInfo [integration/config] - 10https://gerrit.wikimedia.org/r/502497 (owner: 10Cparle) [14:28:43] (03Merged) 10jenkins-bot: Added missing dependency for WikibaseMediaInfo [integration/config] - 10https://gerrit.wikimedia.org/r/502497 (owner: 10Cparle) [14:32:53] who might I ask about getting two commits backported to .wm24? one of them was just merged a half hur ago so I couldn't have asked during the morning swat [14:43:05] (03PS2) 10Zfilipin: WIP Create selenium-daily-beta-MediaWiki targeting commons.wikimedia.beta.wmflabs.org [integration/config] - 10https://gerrit.wikimedia.org/r/502470 (https://phabricator.wikimedia.org/T219815) [14:44:10] (03CR) 10Zfilipin: "PS2 changes target from enwiki to commons." [integration/config] - 10https://gerrit.wikimedia.org/r/502470 (https://phabricator.wikimedia.org/T219815) (owner: 10Zfilipin) [14:48:31] 10Continuous-Integration-Infrastructure, 10MediaWiki-Installer: MediaWiki installer ignore extension when a requires is missing instead of failling - https://phabricator.wikimedia.org/T220514 (10hashar) [14:54:26] (03CR) 10Kosta Harlan: Better arg handling (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/496125 (https://phabricator.wikimedia.org/T218357) (owner: 10Hashar) [14:55:11] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [15:35:13] maintenance-disconnect-full-disks build 62235 integration-slave-docker-1043 (/srv: 100%): OFFLINE due to disk space [15:48:59] RECOVERY - Free space - all mounts on integration-slave-docker-1040 is OK: OK: All targets OK [15:48:59] PROBLEM - Free space - all mounts on integration-slave-docker-1043 is CRITICAL: CRITICAL: integration.integration-slave-docker-1043.diskspace._srv.byte_percentfree (<20.00%) [15:53:56] RECOVERY - Free space - all mounts on integration-slave-docker-1043 is OK: OK: All targets OK [15:56:05] (03Abandoned) 10Volans: operations-dnslint: force colors for tox [integration/config] - 10https://gerrit.wikimedia.org/r/491793 (owner: 10Volans) [16:00:12] maintenance-disconnect-full-disks build 62240 integration-slave-docker-1043: OFFLINE due to disk space [16:10:31] PROBLEM - Host deployment-db03 is DOWN: CRITICAL - Host Unreachable (172.16.5.23) [16:21:14] !log (beta): Update mobileapps to 3edfcad (T220045 T219411 T219667) - 3rd time is the charm [16:21:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:21:20] T219411: External links no longer have arrow icon in Light theme. - https://phabricator.wikimedia.org/T219411 [16:21:20] T219667: Add wikibase entity id for image files to media endpoint - https://phabricator.wikimedia.org/T219667 [16:21:20] T220045: Stop getting base CSS from live ResourceLoader requests - https://phabricator.wikimedia.org/T220045 [16:25:14] maintenance-disconnect-full-disks build 62245 integration-slave-docker-1043: OFFLINE due to disk space [16:28:13] who might I ask about getting two commits backported to .wm24? one of them was only merged after the eu swat so I couldn't do it then [16:29:39] apergos, you can set up the backport commits yourself then ask a deployer [16:29:51] to put it in the next window [16:29:57] or talk to greg-g to get an extra window for it if need be [16:30:21] I'd like to get them in before the train; the reason is that abstract dumps (well, and stubs, but I've run those manually) are broken for some big wikis without these [16:30:45] and I've live patched the code on those hosts so they jobs can continue but the next deploy will overwrite that [16:31:19] I\ll see what greg-g has to say, thanks [16:32:53] apergos: get the backported and ping twentyafterfour, he's on train today [16:33:11] here [16:33:25] thank you; what's the prcedure for backporting them? [16:33:28] the commits are: [16:33:46] https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/502150/ and https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/501869/ [16:33:58] use the cherry-pick button [16:34:03] ^ [16:34:06] put in the branch you want [16:34:16] it should generate a new gerrit change for you [16:34:40] - unless there are conflicts or something, then you may have to do it the old fashioned way on the CLI - [16:34:40] send the deployer the link to the change [16:34:45] I can +2 and deploy them [16:34:45] I've not ever used that feature of gerrit [16:34:52] let's see here :-) [16:37:31] wow that's nice [16:37:47] I almost never use the ui, even for commit message updates [16:38:00] !log deleting deployment-db03, unused since T216635 and shut down since 24th March. T219087 [16:38:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:38:08] T219087: Get rid of deployment-db0[34] - https://phabricator.wikimedia.org/T219087 [16:38:09] T216635: MySQL database on deployment-db03 does not start due to InnoDB issue - https://phabricator.wikimedia.org/T216635 [16:38:25] https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/502537/ https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/502538/ these look to be the new commits [16:38:50] branch 25 wasn't cut yet was it? otherwise I need to backport to that too [16:39:50] twentyafterfour: ? [16:40:53] apergos: if it's not created then you won't if it's in master :) [16:41:16] yeah, I was asking if it's been cut (and hoping it's not yet) [16:41:35] don't think so... [16:41:40] awesome [16:42:22] no I haven't cut the branch yet [16:42:50] that is good for me! [16:43:12] * twentyafterfour is annoyed that glowingbear doesn't alert me when a highlight happens in the current window, if the window is in the background it doesn't seem to notice [16:43:23] d'oh [16:43:48] * twentyafterfour resists the urge to fix it right now [16:43:55] ahahahahaha [16:44:27] twentyafterfour: yeah, ditto [16:44:29] :) [16:44:55] apergos: I gave them both a +2 [16:45:03] now we wait for jenkins :-D [16:45:27] Anything to be aware of deploying them or should I just pull the changes and deploy once jenkins is done? [16:45:29] in any case that means the train will not overwrite them and that's good enough for me [16:45:46] nope, there's nothing fancy [16:45:55] cool [16:46:27] thanks a million, now I can relax tonight, and look again at the state of things tomorrow [16:46:34] :) [16:46:38] you're welcome [16:50:12] maintenance-disconnect-full-disks build 62250 integration-slave-docker-1043: OFFLINE due to disk space [17:06:53] (03CR) 10Krinkle: [C: 03+2] "Thanks, sorry about that. Forgot yesterday!" [integration/config] - 10https://gerrit.wikimedia.org/r/500810 (https://phabricator.wikimedia.org/T219713) (owner: 10Santhosh) [17:15:15] maintenance-disconnect-full-disks build 62255 integration-slave-docker-1043: OFFLINE due to disk space [17:27:46] 10Continuous-Integration-Config, 10Wikipedia-Android-App-Backlog, 10Technical-Debt: Jenkins should build release Android APKs - https://phabricator.wikimedia.org/T104207 (10Charlotte) 05Open→03Invalid [17:40:13] maintenance-disconnect-full-disks build 62260 integration-slave-docker-1043: OFFLINE due to disk space [17:41:08] (03CR) 10KartikMistry: [C: 03+1] "It seems wikibase is needed: https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/44087/console :/" [integration/config] - 10https://gerrit.wikimedia.org/r/500810 (https://phabricator.wikimedia.org/T219713) (owner: 10Santhosh) [17:52:23] PROBLEM - Free space - all mounts on integration-slave-docker-1052 is CRITICAL: CRITICAL: integration.integration-slave-docker-1052.diskspace._srv.byte_percentfree (<22.22%) [17:57:05] (03PS1) 10Krinkle: ContentTranslation: Restore Wikibase CI dependency [integration/config] - 10https://gerrit.wikimedia.org/r/502560 (https://phabricator.wikimedia.org/T219713) [17:57:28] (03PS2) 10Krinkle: ContentTranslation: Restore Wikibase CI dependency [integration/config] - 10https://gerrit.wikimedia.org/r/502560 (https://phabricator.wikimedia.org/T219713) [17:57:44] (03CR) 10Krinkle: [C: 03+2] ContentTranslation: Restore Wikibase CI dependency [integration/config] - 10https://gerrit.wikimedia.org/r/502560 (https://phabricator.wikimedia.org/T219713) (owner: 10Krinkle) [17:58:53] 10Continuous-Integration-Infrastructure, 10Fresnel, 10Performance-Team, 10Quibble: Enable compression for MW web responses in Jenkins Jenkins (e.g. Quibble, Fresnel) - https://phabricator.wikimedia.org/T219694 (10hashar) Quibble indeed uses `php -S` which saved me to have to deal with a frontend web server... [17:59:18] (03Merged) 10jenkins-bot: ContentTranslation: Restore Wikibase CI dependency [integration/config] - 10https://gerrit.wikimedia.org/r/502560 (https://phabricator.wikimedia.org/T219713) (owner: 10Krinkle) [17:59:57] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/502560 [17:59:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:00:48] 10Continuous-Integration-Infrastructure, 10Fresnel, 10Performance-Team, 10Quibble: Enable compression for MW web responses in Jenkins Jenkins (e.g. Quibble, Fresnel) - https://phabricator.wikimedia.org/T219694 (10Krinkle) @hashar Maybe, yeah. I am wondering where that INI file is coming from, because afaik... [18:02:22] RECOVERY - Free space - all mounts on integration-slave-docker-1052 is OK: OK: All targets OK [18:02:59] 10Continuous-Integration-Infrastructure, 10Fresnel, 10Performance-Team, 10Quibble: Enable compression for MW web responses in Jenkins Jenkins (e.g. Quibble, Fresnel) - https://phabricator.wikimedia.org/T219694 (10Krinkle) p:05Triage→03Normal a:03Krinkle I'll patch Quibble to try this out. [18:05:14] maintenance-disconnect-full-disks build 62265 integration-slave-docker-1043: OFFLINE due to disk space [18:15:52] legoktm: he, so you know how Gerrit 2.15 (or maybe earlier) started re-transmitting existing scores with every subsequent comment? It also means that when test pipeline fails before coverage/fresnel, you'll get 3x CR -1 seemingly in IRC. for the same patch. [18:17:41] 10Continuous-Integration-Infrastructure, 10Fresnel, 10Performance-Team, 10Quibble: Enable compression for MW web responses in Jenkins Jenkins (e.g. Quibble, Fresnel) - https://phabricator.wikimedia.org/T219694 (10hashar) The php package should be https://packages.debian.org/source/stretch/php7.0 , with the... [18:30:12] maintenance-disconnect-full-disks build 62270 integration-slave-docker-1043: OFFLINE due to disk space [18:37:11] (03PS1) 10Umherirrender: [CodeMirror] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/502572 [18:40:29] (03PS1) 10Umherirrender: [CongressLookup] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/502576 [18:43:27] (03PS1) 10Umherirrender: [ContactPage] Add ConfirmEdit as dependency [integration/config] - 10https://gerrit.wikimedia.org/r/502577 [18:48:16] (03PS1) 10Umherirrender: [Dashiki] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/502580 [18:52:50] PROBLEM - Puppet errors on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 4.44% of data above the critical threshold [3.0] [18:53:11] PROBLEM - Host deployment-db04 is DOWN: CRITICAL - Host Unreachable (172.16.5.5) [18:55:17] maintenance-disconnect-full-disks build 62275 integration-slave-docker-1043: OFFLINE due to disk space [19:18:03] 10Release-Engineering-Team (Kanban), 10Code-Stewardship-Reviews, 10Graphoid, 10Operations, and 2 others: graphoid: Code stewardship request - https://phabricator.wikimedia.org/T211881 (10dr0ptp4kt) Hi all - we had a meeting on this and a question emerged. Is it possible to run Graph without Graphoid if the... [19:19:36] Krinkle: Do you think we're ready to bump MW core to node10? It's holding things back (selenium, etc.) [19:20:13] maintenance-disconnect-full-disks build 62280 integration-slave-docker-1043: OFFLINE due to disk space [19:24:44] 10Release-Engineering-Team, 10Security-Team, 10phan-taint-check-plugin: Skip php70 tests on new SecurityCheckPlugin development branch - https://phabricator.wikimedia.org/T220540 (10sbassett) [19:24:58] 10Release-Engineering-Team, 10Security-Team, 10phan-taint-check-plugin: Skip php70 tests on new SecurityCheckPlugin development branch - https://phabricator.wikimedia.org/T220540 (10sbassett) p:05Triage→03Normal [19:26:07] 10Release-Engineering-Team (Kanban), 10Code-Stewardship-Reviews, 10Graphoid, 10Operations, and 2 others: graphoid: Code stewardship request - https://phabricator.wikimedia.org/T211881 (10mobrovac) >>! In T211881#5098847, @dr0ptp4kt wrote: > Hi all - we had a meeting on this and a question emerged. Is it po... [19:28:50] James_F: I'm letting RelEng/Antoine take the lead on that. [19:28:59] I'm not prepared to deal with the fall out / follow-up [19:29:33] its basically one change for all mw repos for both npm-test/lint and selenium all at once. which is quite a lot. [19:29:38] (03PS1) 10SBassett: Skip php70 tests on new SecurityCheckPlugin development branch [integration/config] - 10https://gerrit.wikimedia.org/r/502590 (https://phabricator.wikimedia.org/T220540) [19:29:38] might need some prep work to narrow it down. [19:39:50] 10Release-Engineering-Team, 10Analytics, 10ExternalGuidance, 10MW-1.33-notes (1.33.0-wmf.24; 2019-04-02), and 3 others: Unknown dependency: mw.externalguidance.init - https://phabricator.wikimedia.org/T219841 (10Jdlrobson) 05Open→03Resolved a:03Jdlrobson Errors are back to their normal level. Thanks... [19:40:17] James_F: node6 -> node10 seems to be a breaking change for mediawiki :/ [19:41:00] zeljkof found one with fibers node module which is a dependency of webdriver.io [19:41:14] fibers@2.x works with node6 but not with node10 (ABI breakage) [19:41:25] fibers@3.x works with node10 but not with node6 :D [19:41:41] I also noticed another issue [19:42:01] which is that the REL1_27 branch would certainly need a different image than the one used for more recent branches [19:42:15] a recent example was the switch from jessie to stretch which came with the wrong libtidy version [19:42:21] and thus REL1_27 tests are now failling [19:42:22] :/ [19:42:25] anyway [19:42:30] lot of moving grounds here and there [19:42:32] 10Release-Engineering-Team (Kanban), 10Code-Stewardship-Reviews, 10Graphoid, 10Operations, and 2 others: graphoid: Code stewardship request - https://phabricator.wikimedia.org/T211881 (10Yurik) @mobrovac @dr0ptp4kt there is a much larger issue -- performance, and this is identical to maps: when a page load... [19:43:11] thcipriani: are you around? have a gerrit question [19:44:04] hashar: It is. But fuck the selenium tests. If we have to disable them for a month, so be it. [19:44:39] hashar: Everything in this area is a breaking change, including not doing anything. We have three weeks until EOL and half our tools have already decided to stop supporting it. [19:45:13] maintenance-disconnect-full-disks build 62285 integration-slave-docker-1043: OFFLINE due to disk space [19:48:03] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10MediaWiki-extensions-MultimediaViewer, 10MobileFrontend, and 9 others: Audit tests/selenium/LocalSettings.php file aiming at possibly deprecating the feature - https://phabricator.wikimedia.org/T199939 (10Jdlrobson) Cool! This helps this particul... [19:56:45] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10MediaWiki-extensions-MultimediaViewer, 10MobileFrontend, and 9 others: Audit tests/selenium/LocalSettings.php file aiming at possibly deprecating the feature - https://phabricator.wikimedia.org/T199939 (10kostajh) What I ended up doing [here](htt... [20:10:11] maintenance-disconnect-full-disks build 62290 integration-slave-docker-1043: OFFLINE due to disk space [20:11:12] 10Release-Engineering-Team (Kanban), 10Code-Stewardship-Reviews, 10Graphoid, 10Operations, and 2 others: graphoid: Code stewardship request - https://phabricator.wikimedia.org/T211881 (10dr0ptp4kt) Thanks @mobrovac @Yurik. Totally understood on the user observed performance (at least when it's on the crit... [20:16:10] James_F: yeah sounds good :]]] [20:16:24] James_F: and we don't run wdio tests on REL1_27 anyway, that was introduced later [20:16:41] 10Beta-Cluster-Infrastructure: Migrate away from Debian Jessie to Debian Stretch - https://phabricator.wikimedia.org/T218729 (10Krenair) [20:16:48] (03CR) 10Umherirrender: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/502580 (owner: 10Umherirrender) [20:16:57] James_F: I have a patch to add node10 npm6 to the Quibble images, gotta rebase/rebuild and try them out [20:20:11] * hashar hates php :/ [20:20:28] 10Release-Engineering-Team (Kanban), 10Code-Stewardship-Reviews, 10Graphoid, 10Operations, and 2 others: graphoid: Code stewardship request - https://phabricator.wikimedia.org/T211881 (10Milimetric) Maybe as a quick fix, graphs could initially render as a generic "graph loading" image. Then vega and all d... [20:20:35] how one can mock a php builtin function anyway? :/ [20:20:55] Oh, is this the PHP 7.0 vs. 7.2 Unicode fix? [20:21:15] this, [20:21:16] ? [20:21:30] na I found some oddity in the mediawiki installer :/ [20:21:39] and the way it uses extension registration [20:22:32] Oh, right. Bad news, we're doing more of that. [20:22:43] yeah so [20:22:56] if one clone an extension named hmm WikibaseMediaInfo [20:23:10] Oh, is this my fault? [20:23:33] (03PS1) 10Umherirrender: [DoubleWiki] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/502602 [20:23:39] given that WikibaseMediaInfo has a requires: extension: Wikibase [20:23:57] then the installer would fail to install WikibaseMediaInfo because the dependency is not around [20:24:03] but will happilly continue [20:24:10] anyway it is a feature somewhere ;] [20:24:24] as for php 7.0 vs 7.2 really I dont know what is happening on that front [20:24:33] "It's a mess." [20:25:09] then maybe we should do 7.0 first, phaseout hhvm and then do 7.2 [20:25:19] but yeah hard [20:26:32] hashar: We tried that. [20:27:09] But unfortunately we didn't have the resources to get it done in time (by last October, before the Big English fundraiser). [20:27:58] (03PS1) 10Umherirrender: [DynamicSidebar] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/502604 [20:30:40] screw that [20:30:54] mediawiki code is full of copy pasted/duplicated code all other the place :( [20:31:01] s/other/over/ [20:35:12] maintenance-disconnect-full-disks build 62295 integration-slave-docker-1043: OFFLINE due to disk space [20:42:00] (03PS1) 10Kosta Harlan: Establish codehealth pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/502606 (https://phabricator.wikimedia.org/T218598) [20:42:22] (03Abandoned) 10Kosta Harlan: Invoke wmf-sonar-scanner after PHP code coverage is generated [integration/config] - 10https://gerrit.wikimedia.org/r/501685 (https://phabricator.wikimedia.org/T218598) (owner: 10Kosta Harlan) [20:42:52] (03CR) 10jerkins-bot: [V: 04-1] Establish codehealth pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/502606 (https://phabricator.wikimedia.org/T218598) (owner: 10Kosta Harlan) [20:44:18] (03PS2) 10Kosta Harlan: Establish codehealth pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/502606 (https://phabricator.wikimedia.org/T218598) [20:45:47] (03CR) 10jerkins-bot: [V: 04-1] Establish codehealth pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/502606 (https://phabricator.wikimedia.org/T218598) (owner: 10Kosta Harlan) [20:47:52] RECOVERY - Puppet errors on integration-slave-jessie-1001 is OK: OK: Less than 1.00% above the threshold [2.0] [20:48:47] Can anyone help me with getting https://gerrit.wikimedia.org/r/502355 merged in integration/config? [20:48:52] hashar: are you still around? [20:50:39] (03PS3) 10Kosta Harlan: Establish codehealth pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/502606 (https://phabricator.wikimedia.org/T218598) [20:51:52] (03CR) 10jerkins-bot: [V: 04-1] Establish codehealth pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/502606 (https://phabricator.wikimedia.org/T218598) (owner: 10Kosta Harlan) [20:52:09] jdlrobson: yeah [20:52:15] seems straightforward :) [20:52:27] (03CR) 10Kosta Harlan: "I'm not sure I've done everything right here (in fact I know I didn't, since the tests fail) but help with moving this further along would" [integration/config] - 10https://gerrit.wikimedia.org/r/502606 (https://phabricator.wikimedia.org/T218598) (owner: 10Kosta Harlan) [20:53:32] hashar: there's also the follow up https://gerrit.wikimedia.org/r/#/c/integration/config/+/502356/1 [20:53:39] jdlrobson: https://integration.wikimedia.org/ci/job/selenium-daily-beta-Minerva/1/console [20:54:14] (watches with anticipation) [20:54:20] (03CR) 10Hashar: [C: 03+2] "Deployed and something might be running at https://integration.wikimedia.org/ci/job/selenium-daily-beta-Minerva/1/console" [integration/config] - 10https://gerrit.wikimedia.org/r/502355 (https://phabricator.wikimedia.org/T174018) (owner: 10Jdlrobson) [20:54:49] well [20:55:03] the MobileFrontend one isn't setup yet, but it would be good to have that merged (and failing) until I've done that so that RelEng is not a blocker for us to finish that one [20:55:13] I'm hoping to write the patch today [20:55:31] those jobs are screwed up really [20:55:37] they are named beta [20:55:45] but points to en.wikipedia [20:55:46] oh [20:55:47] no [20:55:52] that is en.wikipedia.beta ... [20:56:13] (03Merged) 10jenkins-bot: Daily Minerva Selenium Node.js job [integration/config] - 10https://gerrit.wikimedia.org/r/502355 (https://phabricator.wikimedia.org/T174018) (owner: 10Jdlrobson) [20:58:20] thanks hashar :) [20:59:20] (03PS4) 10Kosta Harlan: Establish codehealth pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/502606 (https://phabricator.wikimedia.org/T218598) [21:00:16] maintenance-disconnect-full-disks build 62300 integration-slave-docker-1043: OFFLINE due to disk space [21:00:41] (03CR) 10jerkins-bot: [V: 04-1] Establish codehealth pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/502606 (https://phabricator.wikimedia.org/T218598) (owner: 10Kosta Harlan) [21:03:55] 10Continuous-Integration-Infrastructure, 10MediaWiki-Installer: MediaWiki web installer do not show extension when their dependency is missing - https://phabricator.wikimedia.org/T220514 (10hashar) [21:06:09] (03PS1) 10QChris: Add tox setup for labs/tools/Isa [integration/config] - 10https://gerrit.wikimedia.org/r/502618 [21:08:38] 10Continuous-Integration-Infrastructure, 10MediaWiki-Installer: MediaWiki web installer do not show extension when their dependency is missing - https://phabricator.wikimedia.org/T220514 (10hashar) [21:09:00] PROBLEM - Free space - all mounts on integration-slave-docker-1037 is CRITICAL: CRITICAL: integration.integration-slave-docker-1037.diskspace._srv.byte_percentfree (<20.00%) [21:14:58] 10Release-Engineering-Team (Kanban), 10Code-Stewardship-Reviews, 10Graphoid, 10Operations, and 2 others: graphoid: Code stewardship request - https://phabricator.wikimedia.org/T211881 (10kaldari) Note that current versions of Vega (since 4.3.0) need ES6 support, so we would either need to transpile Vega wi... [21:19:00] RECOVERY - Free space - all mounts on integration-slave-docker-1037 is OK: OK: All targets OK [21:22:14] 10Continuous-Integration-Infrastructure, 10MediaWiki-Installer, 10MW-1.32-release: MediaWiki web installer do not show extension when their dependency is missing - https://phabricator.wikimedia.org/T220514 (10hashar) The extensions being hidden/skipped is due to https://gerrit.wikimedia.org/r/#/c/mediawiki/c... [21:25:12] maintenance-disconnect-full-disks build 62305 integration-slave-docker-1043: OFFLINE due to disk space [21:25:51] (03PS2) 10Jdlrobson: Run Node.js browser tests in MobileFrontend [integration/config] - 10https://gerrit.wikimedia.org/r/502356 (https://phabricator.wikimedia.org/T174018) [21:29:09] 10Release-Engineering-Team (Kanban), 10Code-Stewardship-Reviews, 10Graphoid, 10Operations, and 2 others: graphoid: Code stewardship request - https://phabricator.wikimedia.org/T211881 (10Yurik) Another, not yet mentioned consideration: There was a significant syntax change between Vega 1.5, Vega 2, and 3+... [21:32:41] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.33.0-wmf.25 deployment blockers - https://phabricator.wikimedia.org/T206679 (10mmodell) [21:34:27] 10Scap, 10Scoring-platform-team (Research): Investigate deployment concurrency limitations for ORES - https://phabricator.wikimedia.org/T188281 (10Harej) @akosiaris Should we still pursue this? [21:34:39] 10Scap, 10Scoring-platform-team (Research): Investigate deployment concurrency limitations for ORES - https://phabricator.wikimedia.org/T188281 (10Harej) p:05Triage→03Low [21:45:39] (03PS5) 10Kosta Harlan: Establish codehealth pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/502606 (https://phabricator.wikimedia.org/T218598) [21:46:47] (03PS6) 10Kosta Harlan: Establish codehealth pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/502606 (https://phabricator.wikimedia.org/T218598) [21:48:52] (03CR) 10jerkins-bot: [V: 04-1] Establish codehealth pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/502606 (https://phabricator.wikimedia.org/T218598) (owner: 10Kosta Harlan) [21:50:14] maintenance-disconnect-full-disks build 62310 integration-slave-docker-1043: OFFLINE due to disk space [21:56:24] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.25 deployment blockers - https://phabricator.wikimedia.org/T206679 (10mmodell) [22:01:26] (03PS2) 10SBassett: Skip php70 tests on new SecurityCheckPlugin development branch [integration/config] - 10https://gerrit.wikimedia.org/r/502590 (https://phabricator.wikimedia.org/T220540) [22:15:12] maintenance-disconnect-full-disks build 62315 integration-slave-docker-1043: OFFLINE due to disk space [22:40:15] maintenance-disconnect-full-disks build 62320 integration-slave-docker-1043: OFFLINE due to disk space [22:45:31] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.25 deployment blockers - https://phabricator.wikimedia.org/T206679 (10mmodell) [22:46:23] 10Continuous-Integration-Config, 10Fresnel, 10Performance-Team: The Fresnel job output (linked in Gerrit) should not start with unrelated setup steps - https://phabricator.wikimedia.org/T220561 (10Krinkle) [22:46:52] (03PS1) 10Krinkle: fresnel: Link directly to "Full Output" of Jobs [integration/config] - 10https://gerrit.wikimedia.org/r/502637 (https://phabricator.wikimedia.org/T220561) [22:49:51] (03CR) 10Krinkle: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/502637 (https://phabricator.wikimedia.org/T220561) (owner: 10Krinkle) [22:50:00] (03PS2) 10Krinkle: fresnel: Link directly to "Full Output" of Jobs [integration/config] - 10https://gerrit.wikimedia.org/r/502637 (https://phabricator.wikimedia.org/T220561) [22:52:57] 10Continuous-Integration-Infrastructure, 10ContentTranslation, 10Lexicographical data, 10Wikidata, 10Wikidata-Campsite: Merge Blocker: Test failures in WikibaseLexeme: Special:RecentChanges shows lemmas in title links to lexemes - https://phabricator.wikimedia.org/T220229 (10greg) Is there reason to beli... [22:56:20] (03PS3) 10Krinkle: fresnel: Link directly to "Full Output" of Jobs [integration/config] - 10https://gerrit.wikimedia.org/r/502637 (https://phabricator.wikimedia.org/T220561) [22:57:04] (03CR) 10Krinkle: [C: 03+2] fresnel: Link directly to "Full Output" of Jobs [integration/config] - 10https://gerrit.wikimedia.org/r/502637 (https://phabricator.wikimedia.org/T220561) (owner: 10Krinkle) [22:57:52] 10Continuous-Integration-Infrastructure, 10ContentTranslation, 10Lexicographical data, 10Wikidata, 10Wikidata-Campsite: Merge Blocker: Test failures in WikibaseLexeme: Special:RecentChanges shows lemmas in title links to lexemes - https://phabricator.wikimedia.org/T220229 (10Krinkle) [22:58:32] (03Merged) 10jenkins-bot: fresnel: Link directly to "Full Output" of Jobs [integration/config] - 10https://gerrit.wikimedia.org/r/502637 (https://phabricator.wikimedia.org/T220561) (owner: 10Krinkle) [23:05:11] maintenance-disconnect-full-disks build 62325 integration-slave-docker-1043: OFFLINE due to disk space [23:08:45] !log Reloading Zuul to deploy https://phabricator.wikimedia.org/T220561 [23:08:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:14:32] (03PS7) 10Kosta Harlan: Establish codehealth pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/502606 (https://phabricator.wikimedia.org/T218598) [23:17:50] (03CR) 10jerkins-bot: [V: 04-1] Establish codehealth pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/502606 (https://phabricator.wikimedia.org/T218598) (owner: 10Kosta Harlan) [23:21:23] (03PS8) 10Kosta Harlan: Establish codehealth pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/502606 (https://phabricator.wikimedia.org/T218598) [23:21:47] (03PS1) 10Krinkle: fresnel: Add section anchor to Jenkins links [integration/config] - 10https://gerrit.wikimedia.org/r/502644 (https://phabricator.wikimedia.org/T220561) [23:21:55] (03CR) 10Krinkle: [C: 03+2] fresnel: Add section anchor to Jenkins links [integration/config] - 10https://gerrit.wikimedia.org/r/502644 (https://phabricator.wikimedia.org/T220561) (owner: 10Krinkle) [23:22:13] (03CR) 10jerkins-bot: [V: 04-1] Establish codehealth pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/502606 (https://phabricator.wikimedia.org/T218598) (owner: 10Kosta Harlan) [23:22:52] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.25 deployment blockers - https://phabricator.wikimedia.org/T206679 (10Krinkle) [23:23:25] (03Merged) 10jenkins-bot: fresnel: Add section anchor to Jenkins links [integration/config] - 10https://gerrit.wikimedia.org/r/502644 (https://phabricator.wikimedia.org/T220561) (owner: 10Krinkle) [23:30:11] maintenance-disconnect-full-disks build 62330 integration-slave-docker-1043: OFFLINE due to disk space [23:35:50] (03PS1) 10Krinkle: Revert "fresnel: Add section anchor to Jenkins links" [integration/config] - 10https://gerrit.wikimedia.org/r/502648 [23:35:54] (03CR) 10Krinkle: [C: 03+2] Revert "fresnel: Add section anchor to Jenkins links" [integration/config] - 10https://gerrit.wikimedia.org/r/502648 (owner: 10Krinkle) [23:37:25] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.25 deployment blockers - https://phabricator.wikimedia.org/T206679 (10Krinkle) [23:37:43] (03Merged) 10jenkins-bot: Revert "fresnel: Add section anchor to Jenkins links" [integration/config] - 10https://gerrit.wikimedia.org/r/502648 (owner: 10Krinkle) [23:55:15] maintenance-disconnect-full-disks build 62335 integration-slave-docker-1043: OFFLINE due to disk space [23:56:30] 10Project-Admins: A way to track when a ticket was created with a pre-filled template - https://phabricator.wikimedia.org/T220568 (10MBinder_WMF)