[01:40:14] 10Deployment-Systems, 06Release-Engineering-Team (Kanban): Bot to handle recurring tasks to manage wikitech:Deployments - https://phabricator.wikimedia.org/T114488#3280822 (10mmodell) a:03mmodell [02:04:33] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<55.56%) [05:50:26] 06Release-Engineering-Team (Kanban): Follow-ups from project refactor - https://phabricator.wikimedia.org/T165596#3280932 (10mmodell) The deployments dashboard is set to be editable by members of #acl*releng ... [05:59:11] 06Release-Engineering-Team (Kanban): Follow-ups from project refactor - https://phabricator.wikimedia.org/T165596#3280933 (10mmodell) nevermind, I see the panel had the wrong permissions. Fixed that and changed the query. [05:59:33] 06Release-Engineering-Team (Kanban): Follow-ups from project refactor - https://phabricator.wikimedia.org/T165596#3280934 (10mmodell) [07:14:34] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:50:24] 10Browser-Tests-Infrastructure, 06Release-Engineering-Team (Kanban), 10MediaWiki-General-or-Unknown, 15User-zeljkofilipin: Run browser tests in core on every commit - https://phabricator.wikimedia.org/T111467#3280998 (10zeljkofilipin) 05Open>03Resolved a:03zeljkofilipin Resolved in https://gerrit.wik... [07:50:26] 10Browser-Tests-Infrastructure, 06Release-Engineering-Team, 07Epic, 13Patch-For-Review, 07Tracking: [EPIC] trigger browser tests from Gerrit (tracking) - https://phabricator.wikimedia.org/T55697#3281001 (10zeljkofilipin) [07:52:44] 10Browser-Tests-Infrastructure, 06Release-Engineering-Team (Backlog), 07Documentation, 07Easy, 13Patch-For-Review: mediawiki_selenium should document SauceLabs usage - https://phabricator.wikimedia.org/T98331#3281005 (10zeljkofilipin) 05Open>03Resolved [08:11:25] 06Release-Engineering-Team, 06Reading-Web-Backlog: Run subset of MobileFrontend browser tests on merges in core - https://phabricator.wikimedia.org/T165940#3281029 (10Jdlrobson) [08:13:05] (03PS2) 10Florianschmidtwelzow: Clone MobileFrontend when RevisionSlider is tested [integration/config] - 10https://gerrit.wikimedia.org/r/354667 [08:17:51] Yippee, build fixed! [08:17:51] Project beta-code-update-eqiad build #156497: 09FIXED in 1 min 51 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/156497/ [09:01:00] Project beta-update-databases-eqiad build #17228: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17228/ [09:01:47] 45 mins on a flow job [09:02:03] !log brought deployment-tin back online a while ago [09:02:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:05:42] 10Browser-Tests-Infrastructure, 06Release-Engineering-Team, 06Reading-Web-Backlog, 15User-zeljkofilipin: Run subset of MobileFrontend browser tests on merges in core - https://phabricator.wikimedia.org/T165940#3281270 (10zeljkofilipin) [09:07:43] (03PS1) 10Jdlrobson: Make mwext-MobileFrontend-npm-run-lint-modules job stable/non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/354911 (https://phabricator.wikimedia.org/T163882) [09:09:19] (03CR) 10jerkins-bot: [V: 04-1] Make mwext-MobileFrontend-npm-run-lint-modules job stable/non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/354911 (https://phabricator.wikimedia.org/T163882) (owner: 10Jdlrobson) [09:09:32] hmm, that doesn't look good: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/ [09:09:51] aborted or failed since the 19th [09:10:15] Wonder if it's tin or the db [09:10:24] !log beta-update-database-eqiad has been hitting the timelimit since May 19th [09:10:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:12:47] looks like it's going to fail around the same point [09:12:51] abort* [09:14:02] Looks different [09:14:03] 09:03:18 ...Update 'RemoveOrphanedEvents' already logged as completed. [09:14:40] yeah, it's right before the 8 flow ones [09:14:52] * greg-g shrugs [09:15:47] 09:03:18 #!/usr/bin/env php [09:15:47] 09:03:18 MediaWiki 1.30.0-alpha Updater [09:15:47] 09:03:18 [09:15:47] 09:03:18 Your composer.lock file is up to date with current dependencies! [09:15:47] 09:03:18 Going to run database updates for enwikiquote [09:15:53] Why doesn't it tell me which db? :( [09:16:05] 45 min is a very long time [09:16:21] * Reedy looks for /usr/local/bin/wmf-beta-update-databases.py [09:16:47] * Reedy makes a patch [09:19:24] [10:19:08] (PS1) Reedy: Print dbname before running update.php [puppet] - https://gerrit.wikimedia.org/r/354919 [09:19:53] If releng want to CR+1 that I'll try and get an opsen to merge [09:20:07] done [09:32:59] (03PS2) 10Jdlrobson: Make mwext-MobileFrontend-npm-run-lint-modules job stable/voting [integration/config] - 10https://gerrit.wikimedia.org/r/354911 (https://phabricator.wikimedia.org/T163882) [09:33:39] Project beta-update-databases-eqiad build #17229: 15ABORTED in 32 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17229/ [09:34:03] ^ that was me [09:34:36] Project beta-update-databases-eqiad build #17230: 15ABORTED in 56 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17230/ [09:35:04] (03PS3) 10Hashar: Make mwext-MobileFrontend-npm-run-lint-modules job stable/voting [integration/config] - 10https://gerrit.wikimedia.org/r/354911 (https://phabricator.wikimedia.org/T163882) (owner: 10Jdlrobson) [09:35:19] (03CR) 10Hashar: [C: 032] "!!!!" [integration/config] - 10https://gerrit.wikimedia.org/r/354911 (https://phabricator.wikimedia.org/T163882) (owner: 10Jdlrobson) [09:36:45] 10Continuous-Integration-Config, 10MobileFrontend, 06Reading-Web-Backlog, 13Patch-For-Review: Migrate experimental CI job mwext-MobileFrontend-npm-run-lint-modules to stable - https://phabricator.wikimedia.org/T163882#3281410 (10Jdlrobson) [09:36:56] 10Continuous-Integration-Config, 10MobileFrontend, 06Reading-Web-Backlog, 13Patch-For-Review: Migrate experimental CI job mwext-MobileFrontend-npm-run-lint-modules to stable - https://phabricator.wikimedia.org/T163882#3213309 (10Jdlrobson) [09:37:03] 10Continuous-Integration-Config, 10MobileFrontend, 06Reading-Web-Backlog, 03Reading-Web-Kanban-Board, 07Unplanned-Sprint-Work: Migrate experimental CI job mwext-MobileFrontend-npm-run-lint-modules to stable - https://phabricator.wikimedia.org/T163882#3213309 (10Jdlrobson) [09:37:42] (03Merged) 10jenkins-bot: Make mwext-MobileFrontend-npm-run-lint-modules job stable/voting [integration/config] - 10https://gerrit.wikimedia.org/r/354911 (https://phabricator.wikimedia.org/T163882) (owner: 10Jdlrobson) [09:41:08] 06Release-Engineering-Team (Kanban), 05Deployment Blockers, 05Release: MW-1.30.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T165957#3281421 (10greg) [09:41:25] 06Release-Engineering-Team (Kanban), 05Deployment Blockers, 05Release: MW-1.30.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T163512#3199999 (10greg) [09:50:02] 09:49:43 aawiki [09:50:02] 09:49:43 arwiki [09:50:02] 09:49:43 cawiki [09:50:02] 09:49:43 commonswiki [09:50:02] 09:49:43 #!/usr/bin/env php [09:50:03] 09:49:43 MediaWiki 1.30.0-alpha Updater [09:53:35] Does it multithread them? [09:53:54] It does... Doesn't it [09:53:55] gah [09:54:37] Does that mean... [09:54:37] cmd = "/usr/local/bin/mwscript update.php --wiki=%s --quick" % db [09:54:53] would be better as... an echo %s; /usr/local... [10:18:49] Project beta-update-databases-eqiad build #17231: 15ABORTED in 29 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17231/ [10:24:58] enwikisource [10:25:00] maybe [10:30:25] Yup, seems to be [10:34:05] greg-g: Found it [10:41:03] !log disabled jerkins on deployment-tin again [10:41:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:14:57] 06Release-Engineering-Team (Kanban), 05Deployment Blockers, 05Release: MW-1.30.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T163512#3281710 (10Jdforrester-WMF) [11:37:04] Project beta-code-update-eqiad build #156508: 04FAILURE in 1 min 1 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/156508/ [11:38:52] Yippee, build fixed! [11:38:52] Project beta-code-update-eqiad build #156509: 09FIXED in 51 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/156509/ [12:01:08] (03CR) 10Hashar: "\O/ lets get to that when I have finished lunch." [integration/config] - 10https://gerrit.wikimedia.org/r/354725 (https://phabricator.wikimedia.org/T160989) (owner: 10WMDE-leszek) [12:01:29] (03CR) 10Hashar: [C: 031] [DNM] Reduce the number of Wikibase jobs [integration/config] - 10https://gerrit.wikimedia.org/r/354725 (https://phabricator.wikimedia.org/T160989) (owner: 10WMDE-leszek) [12:01:47] (03PS3) 10Hashar: Reduce the number of Wikibase jobs [integration/config] - 10https://gerrit.wikimedia.org/r/354725 (https://phabricator.wikimedia.org/T160989) (owner: 10WMDE-leszek) [12:01:52] (03PS4) 10Hashar: Reduce the number of Wikibase jobs [integration/config] - 10https://gerrit.wikimedia.org/r/354725 (https://phabricator.wikimedia.org/T160989) (owner: 10WMDE-leszek) [12:02:06] (03CR) 10Hashar: [C: 032] Reduce the number of Wikibase jobs [integration/config] - 10https://gerrit.wikimedia.org/r/354725 (https://phabricator.wikimedia.org/T160989) (owner: 10WMDE-leszek) [12:03:47] (03Merged) 10jenkins-bot: Reduce the number of Wikibase jobs [integration/config] - 10https://gerrit.wikimedia.org/r/354725 (https://phabricator.wikimedia.org/T160989) (owner: 10WMDE-leszek) [12:04:54] (03CR) 10Hashar: "Deployed!" [integration/config] - 10https://gerrit.wikimedia.org/r/354725 (https://phabricator.wikimedia.org/T160989) (owner: 10WMDE-leszek) [12:05:30] Yippee, build fixed! [12:05:30] Project beta-update-databases-eqiad build #17232: 09FIXED in 5 min 38 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17232/ [12:05:44] i r fixeded it [12:05:59] !log deployment-tin is back online [12:06:03] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:17:08] Reedy: rock! [12:31:51] 06Release-Engineering-Team (Kanban), 10Wikidata, 07Story: [Story] Use composer-merge-plugin to include Wikidata components in mediawiki-vendor - https://phabricator.wikimedia.org/T95663#3282192 (10hoo) [12:32:14] 06Release-Engineering-Team (Kanban), 10Wikidata, 07Story: [Story] Use composer-merge-plugin to include Wikidata components in mediawiki-vendor - https://phabricator.wikimedia.org/T95663#1197083 (10hoo) [12:34:06] 10Gerrit, 06Release-Engineering-Team (Kanban), 10Wikidata, 13Patch-For-Review, and 2 others: [Task] Move PropertySuggester extension to gerrit - https://phabricator.wikimedia.org/T104309#3282206 (10Ladsgroup) a:05demon>03Ladsgroup [12:42:25] 10Continuous-Integration-Infrastructure (Little Steps Sprint), 10Wikidata: Revisit Jenkins jobs being triggered for Wikibase - https://phabricator.wikimedia.org/T160989#3282309 (10WMDE-leszek) Thanks @hashar! In the next days I'll try to make the change to the sqlite job so it is smaller and faster (we will st... [13:07:00] 06Release-Engineering-Team (Kanban), 10Wikidata, 07Story: [Story] Use composer-merge-plugin to include Wikidata components in mediawiki-vendor - https://phabricator.wikimedia.org/T95663#1197083 (10thcipriani) The initial step is composer dependencies, so it sounds like all wikidata extensions's composer depe... [13:18:03] 06Release-Engineering-Team (Kanban), 05Deployment Blockers, 05Release: MW-1.30.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T163512#3282568 (10Jdforrester-WMF) [14:11:55] 06Release-Engineering-Team (Kanban), 05Deployment Blockers, 05Release: MW-1.30.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T163512#3282705 (10Jdforrester-WMF) [14:12:57] 06Release-Engineering-Team (Kanban), 05Deployment Blockers, 05Release: MW-1.30.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T163512#3199999 (10Jdforrester-WMF) [14:29:35] PROBLEM - Host deployment-phab02 is DOWN: CRITICAL - Host Unreachable (10.68.19.232) [15:06:14] (03PS1) 10Hashar: HUGE WIP [integration/quibble] - 10https://gerrit.wikimedia.org/r/354997 [15:11:33] (03CR) 10jerkins-bot: [V: 04-1] HUGE WIP [integration/quibble] - 10https://gerrit.wikimedia.org/r/354997 (owner: 10Hashar) [15:12:39] why is puppet failing on deployment-imagescaler01 ? [15:13:47] Also why does alerts-beta-wm that keep sending notices instead of messages? [15:15:27] no mwdeploy on deployment-imagescaler01 [15:16:13] 10Beta-Cluster-Infrastructure: deployment-imagescaler01 has no mwdeploy user - https://phabricator.wikimedia.org/T166013#3282853 (10Reedy) [15:16:42] Reedy which puppet classes does that instance use? [15:16:57] No idea [15:17:01] Look on wikitech? [15:17:02] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team: deployment-imagescaler01 has no mwdeploy user - https://phabricator.wikimedia.org/T166013#3282865 (10Reedy) [15:17:24] I will try to look on the new openstack browser thingy that replaces looking on wikitech [15:18:11] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team: deployment-imagescaler01 has no mwdeploy user - https://phabricator.wikimedia.org/T166013#3282853 (10Reedy) [15:19:19] https://tools.wmflabs.org/openstack-browser/server/deployment-imagescaler01.deployment-prep.eqiad.wmflabs [15:26:44] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team: deployment-imagescaler01 has no mwdeploy user - https://phabricator.wikimedia.org/T166013#3282895 (10Reedy) Missing mediawiki::user in some way? [15:30:15] Reedy looking at https://github.com/wikimedia/puppet/blob/65ebfd4f8cf0e3692ac84689eda60ca5cb86bcc7/modules/role/manifests/beta/deploymentserver.pp#L4 shoulden't it include the class you just suggested above? [15:30:33] No idea, I don't know what the inclusion tree is [15:30:42] ok [15:37:28] (03PS2) 10Hashar: HUGE WIP [integration/quibble] - 10https://gerrit.wikimedia.org/r/354997 [15:41:00] (03CR) 10jerkins-bot: [V: 04-1] HUGE WIP [integration/quibble] - 10https://gerrit.wikimedia.org/r/354997 (owner: 10Hashar) [17:45:44] It's now purple ^^ [21:58:52] Yippee, build fixed! [21:58:52] Project selenium-PageTriage ยป chrome,beta,Linux,BrowserTests build #398: 09FIXED in 51 sec: https://integration.wikimedia.org/ci/job/selenium-PageTriage/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/398/ [23:00:32] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<11.11%)