[00:12:20] 10Phabricator, 10Wikimedia-Site-requests: Create a form for site requests - https://phabricator.wikimedia.org/T139844#3987822 (10Dereckson) No, but it would be nice to rephrase it and to ensure we've used this link at needed places. [00:24:52] 10Project-Admins: Create Project: Wikimedia-Hackathon-2018 - https://phabricator.wikimedia.org/T187843#3987840 (10Rfarrand) [00:25:01] 10Project-Admins: Create Project: Wikimedia-Hackathon-2018 - https://phabricator.wikimedia.org/T187843#3987853 (10Rfarrand) p:05Triage>03High [00:26:26] 10Project-Admins, 10User-MarcoAurelio: Create Project: Wikimedia-Hackathon-2018 - https://phabricator.wikimedia.org/T187843#3987857 (10MarcoAurelio) a:03MarcoAurelio [00:31:17] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [00:42:26] 10Project-Admins, 10User-MarcoAurelio: Create Project: Wikimedia-Hackathon-2018 - https://phabricator.wikimedia.org/T187843#3987903 (10MarcoAurelio) @Rfarrand @Aklapper I've created #wmhack18. Please let me know if you think it's okay or needs or lacks something. You should be able to edit most (if not all) th... [00:50:28] 10Project-Admins, 10Wikimedia-Hackathon-2018, 10User-MarcoAurelio: Create Project: Wikimedia-Hackathon-2018 - https://phabricator.wikimedia.org/T187843#3987921 (10MarcoAurelio) [01:06:35] 10Project-Admins, 10Wikimedia-Hackathon-2018, 10User-MarcoAurelio: Create Project: Wikimedia-Hackathon-2018 - https://phabricator.wikimedia.org/T187843#3987960 (10Aklapper) 05Open>03Resolved Tweaked the wording a bit. Thanks! [01:15:21] Why on *earth* would My -> Changes (gwtui) have target=_blank? [01:25:20] no_justification: heh [01:26:44] no_justification: https://gerrit-review.googlesource.com/#/c/gerrit/+/159532/ [01:26:54] (In polygerrit they remove it I think) [01:32:33] Gotcha [01:35:51] PROBLEM - Puppet errors on deployment-cpjobqueue is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [01:35:55] PROBLEM - Puppet errors on deployment-cassandra3-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [01:37:39] PROBLEM - Puppet errors on integration-slave-docker-1005 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [01:37:41] PROBLEM - Puppet errors on deployment-ms-be04 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [01:37:56] PROBLEM - Puppet errors on deployment-aqs01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [01:38:24] PROBLEM - Puppet errors on deployment-kafka05 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [01:38:44] PROBLEM - Puppet errors on deployment-changeprop is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [01:39:03] PROBLEM - Puppet errors on deployment-cumin is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [01:39:23] PROBLEM - Puppet errors on saucelabs-03 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [01:39:59] puppet errors seem to have started ^^ [01:46:16] Who (c|sh)ould I annoy (ie: CC/ping on the task) to have T187845 (regression in Special:Statistics which is already fixed, but needs a maintenance script run to have all problems vanished) done? [01:46:16] T187845: Run initSiteStats.php for all wikis - https://phabricator.wikimedia.org/T187845 [01:47:42] no_justification: (though that fixe was backportrd into 2.14, we still have to apply that :)) [01:47:49] Yeah [01:48:07] I'm about 99% sure it's my merge to mw-config is causing those failures. [01:48:10] Which *sucks* [01:48:49] Oh [01:50:17] It happened same time that the labsdb* machines started complaining. [01:50:34] The old WikipediaFirefoxMobileOS submodule complained on silver, which I was able to fix. [01:50:39] (deleting submodules is *wonky*) [01:51:01] Yep [01:53:12] no_justification: I would like to at some point experiment with reply to emails in gerrit (on the test instance) :) [01:53:22] Possibly using the same email setup as phab [01:53:35] 10Release-Engineering-Team (Kanban), 10Wikimedia-Apache-configuration: Cleanup remaining WikipediaMobileFirefoxOS references - https://phabricator.wikimedia.org/T187850#3988026 (10demon) p:05Triage>03High [01:53:52] Low priority for me :) [01:53:56] I think phab sends emails out using wmf email server and receives emails using exim [01:53:57] Yep [01:56:08] Yeah, exim is our standard mailserver @ wmf. [01:56:14] We use it for outgoing, would have to setup incoming [01:56:20] And the Phab model is likely good to follow [02:01:16] eddiegp: Running it for all wikis in small.dblist [02:02:34] no_justification: Thanks! For medium/big it'll probably take quite a long time I guess? [02:02:57] medium probably not. It's a pretty fast script. large....might wanna do 1-by-1 [02:03:18] It basically does a bunch of counts (against slaves, I'm not using --master flag) and then updates 1 row in 1 table [02:03:26] The counts on large wikis will be the slow bit [02:03:50] small.dblist almost done, already at letter T [02:03:51] Yeah I know, count(*) isn't exactly something the db likes [02:03:56] :) [02:04:41] All done! [02:05:20] Should I ask greg to get a window for medium/large then? Especially large 1-by-1 clearly isn't something you'd want to do in a drive-by manner. [02:05:56] no_justification: Nice, that was indeed fast. Thanks again! :) [02:12:51] 10Release-Engineering-Team: Write some version of foreachwiki(indblist) that respects replag and/or has some --delay parameter between wikis - https://phabricator.wikimedia.org/T187852#3988078 (10demon) [02:13:46] RECOVERY - Puppet errors on deployment-changeprop is OK: OK: Less than 1.00% above the threshold [0.0] [02:14:05] RECOVERY - Puppet errors on deployment-cumin is OK: OK: Less than 1.00% above the threshold [0.0] [02:14:06] eddiegp: It's the "Counting number of articles..." step that takes forever. [02:14:23] RECOVERY - Puppet errors on saucelabs-03 is OK: OK: Less than 1.00% above the threshold [0.0] [02:15:52] RECOVERY - Puppet errors on deployment-cpjobqueue is OK: OK: Less than 1.00% above the threshold [0.0] [02:15:54] RECOVERY - Puppet errors on deployment-cassandra3-01 is OK: OK: Less than 1.00% above the threshold [0.0] [02:15:57] no_justification: Really just articles (page count is fast)? In that case I might want to look into how to skip that step. There's a seperate script for that, which we run monthly by cron. [02:16:17] page count is easy, it's just count(*). But articles is a subset of pages and requires thought :) [02:16:39] There's no way to exclude it from initSiteStats [02:17:03] On the "E"s now for medium.dblit [02:17:35] Yeah, what I meant is that it might be worth to built a way ;) [02:17:41] RECOVERY - Puppet errors on integration-slave-docker-1005 is OK: OK: Less than 1.00% above the threshold [0.0] [02:17:41] RECOVERY - Puppet errors on deployment-ms-be04 is OK: OK: Less than 1.00% above the threshold [0.0] [02:17:55] RECOVERY - Puppet errors on deployment-aqs01 is OK: OK: Less than 1.00% above the threshold [0.0] [02:18:24] RECOVERY - Puppet errors on deployment-kafka05 is OK: OK: Less than 1.00% above the threshold [0.0] [02:24:00] eddiegp: In the "I"s [02:24:17] :) [02:34:32] eddiegp: If you're feeling bold, what would be cool is to abstract out what reports you update. [02:34:45] So you could do something like --update=edits,users and skip articles/pages [02:34:56] So you could do 1. Or all [02:34:57] :) [02:36:06] I'll have to look whether we already have a abstraction for --update=list,of,things for maintenance scripts. [02:36:06] So in the future, when someone busts the "images" count, we can easily fix it :D [02:36:15] Maintenance scripts are a mess [02:36:18] It's mostly my fault :( [02:36:29] This is relatively nice, just calling the class for each step. [02:37:07] There's a task somewhere about hiding all maintenance scripts in classes and only having one entry point (instead of an entry point for each one) [02:37:17] So like you'd do `mwmaint.php updatesitestats` [02:37:30] Or `mwmaint.php namespacedupes` [02:37:31] etc etc [02:37:41] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<22.22%) [02:38:00] I could amend it with --user --edits --article (one switch per step) options in about a minute, but that doesnt look as nice as --update=user,edits,article ;) [02:39:06] Well the idea is that we don't have to add a new param for each new report we created :) [02:39:12] (And could potentially let extensions plug in) [02:39:13] :) [02:39:27] Exactly. [02:39:29] !log beta: ran initSiteStats.php --update on all wikis [02:39:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [02:57:10] eddiegp: https://phabricator.wikimedia.org/T187845#3988089 [02:57:12] I'm done for tonight [02:57:54] no_justification: Alright, thanks (again)! I'm looking into the maintenance script right now :) [02:59:33] Added all of large.dblist as a checklist to the summary [02:59:46] So A) we can keep track and B) protects us in case someone moves something from medium->large before we're done :p [02:59:54] (or from large -> medium) [03:00:48] These are gonna be *slow* [03:05:40] I hope to make it slightly better by having that switch in the maintenance script ;) [05:07:02] 10Release-Engineering-Team (Kanban), 10Wikimedia-Apache-configuration: Cleanup remaining WikipediaMobileFirefoxOS references - https://phabricator.wikimedia.org/T187850#3988229 (10bd808) ``` $ ssh labsdb1009.eqiad.wmnet $ ls -ld /usr/local/lib/mediawiki-config/docroot/wikimedia.org/WikipediaMobileFirefoxOS drw... [06:43:52] 10Project-Admins, 10Wikimedia-Hackathon-2018, 10User-MarcoAurelio: Create Project: Wikimedia-Hackathon-2018 - https://phabricator.wikimedia.org/T187843#3988262 (10Rfarrand) Thank you!! [06:52:40] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:45:59] (03CR) 10Hashar: [C: 031] "Looks good :-}" [integration/config] - 10https://gerrit.wikimedia.org/r/412931 (https://phabricator.wikimedia.org/T185315) (owner: 10Zfilipin) [07:49:48] (03CR) 10Hashar: "fetch-composer-dev is no more used, it is sole use was for the operations-mw-config-composer-hhvm-jessie job. I have switched that job to " [integration/config] - 10https://gerrit.wikimedia.org/r/395100 (https://phabricator.wikimedia.org/T181938) (owner: 10Hashar) [07:55:30] (03PS5) 10Hashar: Drop unused jjb macro based on composer-dev-args.js [integration/config] - 10https://gerrit.wikimedia.org/r/395100 [07:58:44] (03CR) 10Hashar: [C: 032] "Noop in JJB as expected since it is no more used :-}" [integration/config] - 10https://gerrit.wikimedia.org/r/395100 (owner: 10Hashar) [07:59:59] (03Merged) 10jenkins-bot: Drop unused jjb macro based on composer-dev-args.js [integration/config] - 10https://gerrit.wikimedia.org/r/395100 (owner: 10Hashar) [08:04:50] 10Continuous-Integration-Infrastructure: mw-fetch-composer-dev.sh/composer-dev-args.js is too naive with value handling - https://phabricator.wikimedia.org/T181937#3807127 (10hashar) Thank you @Reedy ! [08:09:36] PROBLEM - Puppet errors on saucelabs-01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [08:10:56] 10Release-Engineering-Team, 10Utilities-code-utils: Write some version of foreachwiki(indblist) that respects replag and/or has some --delay parameter between wikis - https://phabricator.wikimedia.org/T187852#3988321 (10mmodell) > (I have no idea what to tag this with) ftfy [08:45:53] 10Beta-Cluster-Infrastructure, 10Analytics-EventLogging, 10Analytics-Kanban, 10User-Elukey: EventLogging broken in beta - https://phabricator.wikimedia.org/T185952#3988342 (10Krinkle) [08:49:33] RECOVERY - Puppet errors on saucelabs-01 is OK: OK: Less than 1.00% above the threshold [0.0] [09:01:24] 10Release-Engineering-Team, 10Utilities-code-utils: Write some version of foreachwiki(indblist) that respects replag and/or has some --delay parameter between wikis - https://phabricator.wikimedia.org/T187852#3988357 (10Krinkle) @demon As far as I know, individual maintenance scripts (even if run via foreachwi... [09:01:33] 10Release-Engineering-Team, 10MediaWiki-Maintenance-scripts, 10Utilities-code-utils: Write some version of foreachwiki(indblist) that respects replag and/or has some --delay parameter between wikis - https://phabricator.wikimedia.org/T187852#3988358 (10Krinkle) [09:50:14] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10MediaWiki-Vagrant, 10User-zeljkofilipin: Continuous integration for mediawiki-vagrant - https://phabricator.wikimedia.org/T183456#3988390 (10zeljkofilipin) 05Open>03declined Looks like this would not be useful even if we go... [09:58:25] 10Release-Engineering-Team (Kanban), 10Vector, 10User-zeljkofilipin: Selenium tests for Vector skin - https://phabricator.wikimedia.org/T187859#3988395 (10zeljkofilipin) [09:59:16] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Collaboration-Team-Triage, 10Notifications, and 4 others: New usermessage browser test is blocking merges in Minerva skin and Echo extension - https://phabricator.wikimedia.org/T185928#3988407 (10zeljkofilipin) >>! In T185928#3934433, @... [10:05:38] 10Release-Engineering-Team (Kanban), 10Vector, 10User-zeljkofilipin: Selenium tests for Vector skin - https://phabricator.wikimedia.org/T187859#3988395 (10zeljkofilipin) p:05Triage>03Low [10:11:24] 10Release-Engineering-Team (Kanban), 10MW-1.31-release-notes (WMF-deploy-2018-02-06 (1.31.0-wmf.20)), 10Patch-For-Review, 10User-zeljkofilipin: Q3 Selenium framework improvements - https://phabricator.wikimedia.org/T182421#3988440 (10zeljkofilipin) [10:11:29] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Collaboration-Team-Triage, 10Notifications, and 4 others: New usermessage browser test is blocking merges in Minerva skin and Echo extension - https://phabricator.wikimedia.org/T185928#3988439 (10zeljkofilipin) [10:11:49] 10Release-Engineering-Team (Kanban), 10MW-1.31-release-notes (WMF-deploy-2018-02-06 (1.31.0-wmf.20)), 10Patch-For-Review, 10User-zeljkofilipin: Q3 Selenium framework improvements - https://phabricator.wikimedia.org/T182421#3822892 (10zeljkofilipin) [10:11:49] PROBLEM - Free space - all mounts on deployment-eventlog02 is CRITICAL: CRITICAL: deployment-prep.deployment-eventlog02.diskspace.root.byte_percentfree (<100.00%) [10:11:53] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Collaboration-Team-Triage, 10Notifications, and 4 others: New usermessage browser test is blocking merges in Minerva skin and Echo extension - https://phabricator.wikimedia.org/T185928#3928891 (10zeljkofilipin) [10:12:11] 10Release-Engineering-Team (Kanban), 10Vector, 10User-zeljkofilipin: Selenium tests for Vector skin - https://phabricator.wikimedia.org/T187859#3988443 (10zeljkofilipin) [10:12:13] 10Release-Engineering-Team (Kanban), 10MW-1.31-release-notes (WMF-deploy-2018-02-06 (1.31.0-wmf.20)), 10Patch-For-Review, 10User-zeljkofilipin: Q3 Selenium framework improvements - https://phabricator.wikimedia.org/T182421#3822892 (10zeljkofilipin) [10:13:14] 10Release-Engineering-Team (Kanban), 10Vector, 10User-zeljkofilipin: Move one Selenium tests from mediawiki/core to mediawiki/skins/Vector - https://phabricator.wikimedia.org/T187859#3988395 (10zeljkofilipin) [10:13:48] 10Release-Engineering-Team (Kanban), 10Vector, 10User-zeljkofilipin: Move one Selenium tests from mediawiki/core to mediawiki/skins/Vector - https://phabricator.wikimedia.org/T187859#3988395 (10zeljkofilipin) [10:19:21] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Update README file for Selenium tests - https://phabricator.wikimedia.org/T187862#3988462 (10zeljkofilipin) [10:19:34] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Update README file for Selenium tests - https://phabricator.wikimedia.org/T187862#3988473 (10zeljkofilipin) p:05Triage>03Low [10:20:51] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Update README file for Selenium tests - https://phabricator.wikimedia.org/T187862#3988462 (10zeljkofilipin) [10:20:53] 10Release-Engineering-Team (Kanban), 10MW-1.31-release-notes (WMF-deploy-2018-02-06 (1.31.0-wmf.20)), 10Patch-For-Review, 10User-zeljkofilipin: Q3 Selenium framework improvements - https://phabricator.wikimedia.org/T182421#3988475 (10zeljkofilipin) [10:54:07] PROBLEM - Host deployment-videoscaler01 is DOWN: CRITICAL - Host Unreachable (10.68.19.130) [10:54:53] PROBLEM - Host deployment-tmh01 is DOWN: CRITICAL - Host Unreachable (10.68.16.211) [11:26:23] !log Refreshed TOR exit nodes & cleaned old abusefilter log data for all Beta Cluster wikis. [11:26:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:12:58] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Update README file for Selenium tests - https://phabricator.wikimedia.org/T187862#3988799 (10zeljkofilipin) [12:30:54] PROBLEM - Puppet errors on deployment-ores01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:33:25] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Collaboration-Team-Triage, 10Notifications, and 4 others: New usermessage browser test is blocking merges in Minerva skin and Echo extension - https://phabricator.wikimedia.org/T185928#3988896 (10zeljkofilipin) >>! In T185928#3934433, @... [13:17:05] 10Release-Engineering-Team (Kanban), 10Vector, 10Patch-For-Review, 10User-zeljkofilipin: Move one Selenium tests from mediawiki/core to mediawiki/skins/Vector - https://phabricator.wikimedia.org/T187859#3989006 (10zeljkofilipin) [13:19:52] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Collaboration-Team-Triage, 10Notifications, and 4 others: New usermessage browser test is blocking merges in Minerva skin and Echo extension - https://phabricator.wikimedia.org/T185928#3989010 (10zeljkofilipin) a:03zeljkofilip... [13:21:30] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Collaboration-Team-Triage, 10Notifications, and 4 others: New usermessage browser test is blocking merges in Minerva skin and Echo extension - https://phabricator.wikimedia.org/T185928#3928891 (10zeljkofilipin) 05Open>03Reso... [13:22:09] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-DonationInterface, 10Browser-Tests, 10User-zeljkofilipin: Write browser tests for DonationInterface - https://phabricator.wikimedia.org/T99955#3989020 (10zeljkofilipin) a:03zeljkofilipin [13:59:51] (03PS1) 10Thiemo Kreuz (WMDE): Fix SpaceBeforeControlStructureBraceSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/413161 [14:34:02] 10Release-Engineering-Team (Next): When "scap pull" does a (slow) CDB rebuild, it should tell me that that's what it's doing - https://phabricator.wikimedia.org/T162207#3155641 (10zeljkofilipin) This is annoying to experienced deployers and at least surprising for new deployers, if not scary. > #wikimedia-opera... [14:42:39] 10Release-Engineering-Team, 10Scap, 10scap2: Cleanup old cache dirs if still around - https://phabricator.wikimedia.org/T157743#3989175 (10zeljkofilipin) 05duplicate>03Open This is happening again. ``` zfilipin@mwdebug1002:~$ scap pull 14:18:30 Copying to mwdebug1002.eqiad.wmnet from tin.eqiad.wmnet 14:... [14:43:26] 10Deployments, 10Release-Engineering-Team (Kanban), 10Operations, 10User-Addshore: cannot delete non-empty directory: php-1.29.0-wmf.3 messages on 'scap sync' on mwdebug1002 - https://phabricator.wikimedia.org/T157030#3989178 (10zeljkofilipin) 05Resolved>03Open This is happening again. ``` zfilipin@mw... [14:43:41] 10Release-Engineering-Team, 10Scap, 10scap2: Cleanup old cache dirs if still around - https://phabricator.wikimedia.org/T157743#3989181 (10zeljkofilipin) 05Open>03Invalid [14:45:11] 10Deployments, 10Release-Engineering-Team (Kanban), 10Operations, 10User-Addshore: cannot delete non-empty directory: php-1.29.0-wmf.3 messages on 'scap sync' on mwdebug1002 - https://phabricator.wikimedia.org/T157030#3989184 (10zeljkofilipin) 05Open>03Resolved [14:48:35] 10Deployments, 10Release-Engineering-Team (Kanban), 10Operations, 10User-Addshore: cannot delete non-empty directory: php-1.29.0-wmf.3 messages on 'scap sync' on mwdebug1002 - https://phabricator.wikimedia.org/T157030#3989222 (10zeljkofilipin) 05Resolved>03Open a:05thcipriani>03demon @thcipriani sa... [14:53:15] PROBLEM - Host deployment-puppetdb01 is DOWN: CRITICAL - Host Unreachable (10.68.23.76) [15:09:34] 10Beta-Cluster-Infrastructure, 10Operations, 10Readers-Web-Backlog: Beta cluster api.php never responds - https://phabricator.wikimedia.org/T187891#3989296 (10Niedzielski) [15:10:04] 10Beta-Cluster-Infrastructure, 10Operations, 10Readers-Web-Backlog: Beta cluster api.php never responds - https://phabricator.wikimedia.org/T187891#3989312 (10Niedzielski) [15:11:29] 10Beta-Cluster-Infrastructure, 10Operations, 10Readers-Web-Backlog: Beta cluster api.php never responds - https://phabricator.wikimedia.org/T187891#3989296 (10Niedzielski) [15:14:01] 10Beta-Cluster-Infrastructure, 10Operations, 10Readers-Web-Backlog (Tracking): Beta cluster api.php never responds - https://phabricator.wikimedia.org/T187891#3989347 (10Niedzielski) [15:24:41] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-DonationInterface, 10Browser-Tests, 10User-zeljkofilipin: Write browser tests for DonationInterface - https://phabricator.wikimedia.org/T99955#3989408 (10zeljkofilipin) I have tried setting this up on my machine, but looks like there is no Donatio... [15:26:37] 10Deployments, 10Release-Engineering-Team (Kanban), 10Operations, 10User-Addshore: cannot delete non-empty directory: php-1.29.0-wmf.3 messages on 'scap sync' on mwdebug1002 - https://phabricator.wikimedia.org/T157030#3989410 (10demon) Has nothing to do with scap clean. We've been fighting this same error... [15:32:15] 10Deployments, 10Release-Engineering-Team (Kanban), 10Operations, 10User-Addshore: cannot delete non-empty directory: php-1.29.0-wmf.3 messages on 'scap sync' on mwdebug1002 - https://phabricator.wikimedia.org/T157030#3989437 (10thcipriani) >>! In T157030#3989410, @demon wrote: > Has nothing to do with sca... [15:35:44] 10Deployments, 10Release-Engineering-Team (Kanban), 10Operations, 10User-Addshore: cannot delete non-empty directory: php-1.29.0-wmf.3 messages on 'scap sync' on mwdebug1002 - https://phabricator.wikimedia.org/T157030#3989447 (10demon) Sorta. From what I can tell, `rsync` won't delete a destination directo... [15:37:12] 10Deployments, 10Release-Engineering-Team (Kanban), 10Operations, 10User-Addshore: cannot delete non-empty directory: php-1.29.0-wmf.3 messages on 'scap sync' on mwdebug1002 - https://phabricator.wikimedia.org/T157030#3989462 (10demon) Fwiw, this works just fine: SSH_AUTH_SOCK=/run/keyholder/proxy.sock... [15:38:04] 10Deployments, 10Release-Engineering-Team (Kanban), 10Operations, 10User-Addshore: cannot delete non-empty directory: php-1.29.0-wmf.3 messages on 'scap sync' on mwdebug1002 - https://phabricator.wikimedia.org/T157030#3989467 (10demon) In the old method, we just did ^^^ and never had a "partial" cleanup li... [15:42:38] 10Deployments, 10Release-Engineering-Team (Kanban), 10Operations, 10User-Addshore: cannot delete non-empty directory: php-1.29.0-wmf.3 messages on 'scap sync' on mwdebug1002 - https://phabricator.wikimedia.org/T157030#3989479 (10demon) I think there's two actionables here! # Make sure we delete these direc... [15:44:16] 10Continuous-Integration-Config: fatal: remote error: access denied or repository not exported: /mediawiki/extensions/ReadingLists - https://phabricator.wikimedia.org/T187897#3989482 (10Mholloway) [15:46:31] 10Deployments, 10Release-Engineering-Team (Kanban), 10Operations, 10User-Addshore: cannot delete non-empty directory: php-1.29.0-wmf.3 messages on 'scap sync' on mwdebug1002 - https://phabricator.wikimedia.org/T157030#3989505 (10demon) I'm curious if `scap clean` is the wrong approach. A daily (or heck, we... [15:55:56] 10Release-Engineering-Team, 10MediaWiki-Maintenance-scripts, 10Utilities-code-utils: Write some version of foreachwiki(indblist) that respects replag and/or has some --delay parameter between wikis - https://phabricator.wikimedia.org/T187852#3989531 (10demon) If a script has wait-for-slaves support, that wou... [15:59:16] 10Deployments, 10Release-Engineering-Team (Kanban), 10Operations, 10User-Addshore: cannot delete non-empty directory: php-1.29.0-wmf.3 messages on 'scap sync' on mwdebug1002 - https://phabricator.wikimedia.org/T157030#3989547 (10thcipriani) Just to create a small test case that demos what's going wrong: `... [16:04:17] 10Deployments, 10Release-Engineering-Team (Kanban), 10Operations, 10User-Addshore: cannot delete non-empty directory: php-1.29.0-wmf.3 messages on 'scap sync' on mwdebug1002 - https://phabricator.wikimedia.org/T157030#3989575 (10thcipriani) >>! In T157030#3989447, @demon wrote: > Sorta. From what I can tel... [16:08:55] (03PS1) 10Zfilipin: Running qunit-selenium job for DonationInterface [integration/config] - 10https://gerrit.wikimedia.org/r/413187 (https://phabricator.wikimedia.org/T99955) [16:11:50] (03CR) 10Zfilipin: [C: 032] Running qunit-selenium job for DonationInterface [integration/config] - 10https://gerrit.wikimedia.org/r/413187 (https://phabricator.wikimedia.org/T99955) (owner: 10Zfilipin) [16:13:15] (03Merged) 10jenkins-bot: Running qunit-selenium job for DonationInterface [integration/config] - 10https://gerrit.wikimedia.org/r/413187 (https://phabricator.wikimedia.org/T99955) (owner: 10Zfilipin) [16:13:51] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-DonationInterface, 10Browser-Tests, 10Patch-For-Review, 10User-zeljkofilipin: Write browser tests for DonationInterface - https://phabricator.wikimedia.org/T99955#3989612 (10Ejegg) Oh wow, thanks @zeljkofilipin! Sorry I didn't see your comments... [16:14:38] 10Release-Engineering-Team (Kanban), 10Fundraising-Backlog, 10MediaWiki-extensions-DonationInterface, 10Browser-Tests, and 2 others: Write browser tests for DonationInterface - https://phabricator.wikimedia.org/T99955#3989614 (10Ejegg) [16:15:35] !log Reloading Zuul to deploy d90d617e0794725280f9f2660effabea74533cbe [16:15:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:24:16] 10Release-Engineering-Team (Kanban), 10Fundraising-Backlog, 10MediaWiki-extensions-DonationInterface, 10Browser-Tests, and 2 others: Write browser tests for DonationInterface - https://phabricator.wikimedia.org/T99955#3989655 (10zeljkofilipin) I have set up Zuul so mediawiki-core-qunit-selenium-jessie job... [16:26:09] 10Release-Engineering-Team (Kanban), 10Fundraising-Backlog, 10MediaWiki-extensions-DonationInterface, 10Browser-Tests, and 2 others: Write browser tests for DonationInterface - https://phabricator.wikimedia.org/T99955#3989662 (10zeljkofilipin) >>! In T99955#3989612, @Ejegg wrote: > If you haven't found it... [16:33:18] PROBLEM - Puppet errors on deployment-logstash2 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:35:34] 10Release-Engineering-Team (Kanban), 10Fundraising-Backlog, 10MediaWiki-extensions-DonationInterface, 10Browser-Tests, and 2 others: Write browser tests for DonationInterface - https://phabricator.wikimedia.org/T99955#3989718 (10Ejegg) @zeljkofilipin the vagrant role should install a Main Page with a bunch... [16:36:25] 10Beta-Cluster-Infrastructure, 10Operations, 10Readers-Web-Backlog (Tracking): Beta cluster api.php never responds - https://phabricator.wikimedia.org/T187891#3989296 (10thcipriani) Looks like the api is able to render a trivial test or two for me currently: https://en.wikipedia.beta.wmflabs.org/w/api.php?ac... [16:44:14] no_justification apparently they improved performance in reindexing changes https://gerrit-review.googlesource.com/c/homepage/+/160970 heh (see diff) [16:56:57] eddiegp_: https://phabricator.wikimedia.org/T187845#3989827 [16:56:59] paladox: cool [16:58:31] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [16:59:28] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin, 10Wikimedia-Incident: Create selenium-core-jessie daily Jenkins job - https://phabricator.wikimedia.org/T185011#3989833 (10greg) p:05Triage>03Normal [17:07:57] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikimedia-Incident, 10Zuul: CI is running against parent patches, not the patches themselves for chained patches - https://phabricator.wikimedia.org/T187567#3989873 (10greg) [17:08:29] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikimedia-Incident, 10Zuul: CI is running against parent patches, not the patches themselves for chained patches - https://phabricator.wikimedia.org/T187567#3979150 (10greg) Follow-up is: * Incident report * Any other follow-up... [17:08:31] no_justification: Cool, thanks :) Should or shouldn't I assume that you'll continue to go through that list whenever you're bored enough? I'd revisit my plan to get a window for it in that case (which I'd only do to avoid the task rusting anyways, there's no harm if it isn't done all at once/slowly). [17:08:43] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikimedia-Incident, 10Zuul: CI is running against parent patches, not the patches themselves for chained patches - https://phabricator.wikimedia.org/T187567#3989883 (10hashar) **status update** Will write an incident report fo... [17:09:08] eddiegp: Well, a window isn't the best idea cuz we want some lag between them [17:09:32] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikimedia-Incident, 10Zuul: CI is running against parent patches, not the patches themselves for chained patches - https://phabricator.wikimedia.org/T187567#3989888 (10greg) a:03hashar [17:17:42] 10Beta-Cluster-Infrastructure, 10Operations, 10Readers-Web-Backlog (Tracking): Beta cluster api.php never responds - https://phabricator.wikimedia.org/T187891#3989938 (10Niedzielski) 05Open>03Resolved a:03thcipriani Fixed! Thank you @thcipriani! [17:19:03] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations, 10Readers-Web-Backlog (Tracking): Beta cluster api.php never responds - https://phabricator.wikimedia.org/T187891#3989962 (10thcipriani) >>! In T187891#3989938, @Niedzielski wrote: > Fixed! Thank you @thcipriani! glad to hea... [17:21:04] no_justification: Right, my impression just is that usually most deployers don't deploy things/run scripts and thus such tasks might stall unless one gets a window for it/pings someone specifically to do it. ;) That's why I thought I'd need one. But if I can assume that you'll do it from time to time (which was the question), I'll just wait patiently :) [17:24:00] PROBLEM - Puppet errors on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:27:25] eddiegp: I'm not "most" ;-) [17:27:39] Heh, I already noticed that ;) [17:33:48] 10Gerrit, 10Commons, 10MediaWiki-Special-pages, 10Multimedia, and 2 others: Special:Upload prepending "== {{int:filedesc}} ==" twice (Commons) - https://phabricator.wikimedia.org/T187302#3990053 (10Cparle) a:05matthiasmullie>03Cparle [17:35:09] no_justification: btw, when digging in puppet around some maintenance scripts run by crons lately, I wondered what the difference between "mwscriptwikiset" and "foreachwikiindblist" is. I found those scripts in puppet, but they seem to do the same thing to me (accept a dblist plus a maintenance script command, run the maintenance script with mwscript.php for each wiki in the dblist)? [17:35:53] 10Gerrit, 10Commons, 10MediaWiki-Special-pages, 10Multimedia, and 2 others: Special:Upload prepending "== {{int:filedesc}} ==" twice (Commons) - https://phabricator.wikimedia.org/T187302#3990057 (10Cparle) a:05Cparle>03matthiasmullie (one second thoughts, I'll just add myself as a reviewer in gerrit) [17:38:02] 10Release-Engineering-Team, 10MediaWiki-Maintenance-scripts, 10Utilities-code-utils: Write some version of foreachwiki(indblist) that respects replag and/or has some --delay parameter between wikis - https://phabricator.wikimedia.org/T187852#3988067 (10EddieGP) >>! In T187852#3989531, @demon wrote: > If a sc... [17:41:56] eddiegp: I barely even remember that mwscriptwikiset existed. [17:43:42] Heh, okay. [17:44:40] One of my ideas was that it does the same thing, and the second one was just built because noone remembered the first, so ... :D [17:45:10] Not an unlikely explanation around here :p [17:46:44] eddiegp: jfdi works well for no_justification and I [17:47:42] * eddiegp goes to cc Reedy and no_justification on everything ;) [17:47:57] You need to start shipping us beers then [17:48:44] Might be worth it sometimes :D [17:49:27] Warning about cc'ing me indiscriminately....I also unsubscribe from things I don't care about enough :p [17:50:00] Actually I like that, then I at least know you don't care. [17:50:09] I used to be subscribed to like 40k tasks. [17:50:10] :p [17:50:51] I don't want to imagine that bugmail. [18:23:18] 10Release-Engineering-Team (Kanban), 10Wikimedia-Apache-configuration, 10Patch-For-Review: Cleanup remaining WikipediaMobileFirefoxOS references - https://phabricator.wikimedia.org/T187850#3990229 (10bd808) [18:39:17] 10Release-Engineering-Team (Kanban), 10Wikimedia-Apache-configuration, 10Patch-For-Review: Cleanup remaining WikipediaMobileFirefoxOS references - https://phabricator.wikimedia.org/T187850#3990340 (10bd808) @chasemp wrote on https://gerrit.wikimedia.org/r/413095: ``` This worked seemingly fine for all 3 labs... [18:39:40] 10Release-Engineering-Team (Kanban), 10Wikimedia-Apache-configuration, 10Patch-For-Review: Cleanup remaining WikipediaMobileFirefoxOS references - https://phabricator.wikimedia.org/T187850#3990342 (10bd808) [18:54:27] PROBLEM - Puppet errors on deployment-secureredirexperiment is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:06:57] (03PS1) 10Thiemo Kreuz (WMDE): Grade errors about duplicate spaces in PHPDoc tags down to warnings [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/413218 [19:17:14] (03PS1) 10Thiemo Kreuz (WMDE): Remove unused code and function arguments from three sniffs [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/413221 [19:35:44] (03PS1) 10Thiemo Kreuz (WMDE): Streamline SpaceBeforeControlStructureBraceSniff implementation [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/413225 [19:38:04] (03PS1) 10Jforrester: Add npm-docker template for 3d2png service [integration/config] - 10https://gerrit.wikimedia.org/r/413227 [19:53:31] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 80.00% of data above the critical threshold [10.0] [20:00:32] (03PS1) 10Legoktm: Check against random pages and action=edit,action=history [integration/audit-resources] - 10https://gerrit.wikimedia.org/r/413234 [20:00:34] (03PS1) 10Legoktm: Rely on subprocess.Popen to be async instead of threading [integration/audit-resources] - 10https://gerrit.wikimedia.org/r/413235 [20:01:24] (03CR) 10Legoktm: [V: 032 C: 032] Check against random pages and action=edit,action=history [integration/audit-resources] - 10https://gerrit.wikimedia.org/r/413234 (owner: 10Legoktm) [20:01:26] (03CR) 10Legoktm: [V: 032 C: 032] Rely on subprocess.Popen to be async instead of threading [integration/audit-resources] - 10https://gerrit.wikimedia.org/r/413235 (owner: 10Legoktm) [20:09:58] legoktm: Thoughts on https://gerrit.wikimedia.org/r/#/c/413060/ ? [20:12:38] (03CR) 10Legoktm: [C: 04-1] "The `skip-if` portion needs to stay, you should only need to remove the failure- part." [integration/config] - 10https://gerrit.wikimedia.org/r/413060 (https://phabricator.wikimedia.org/T180630) (owner: 10Krinkle) [20:19:00] no_justification lol looks like upstream have just released 2.15rc3 so im adding download links to it https://gerrit-review.googlesource.com/c/homepage/+/161090 [20:19:22] no_justification they released v2.15-rc2 in novemeber 2 [20:19:34] that's a long time for another rc heh [20:20:14] And people say I take too long to release MW :p [20:20:32] lol [20:24:31] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T183961#3990749 (10mmodell) [20:25:05] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T183961#3868531 (10mmodell) [20:36:15] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, and 2 others: Phabricator: Clean up deadlocked apache processes - https://phabricator.wikimedia.org/T187790#3990786 (10Dzahn) restart cron has been installed on both servers [20:43:48] (03PS2) 10Krinkle: Disable custom 'checkstyleResult' failure link for phan jobs [integration/config] - 10https://gerrit.wikimedia.org/r/413060 (https://phabricator.wikimedia.org/T180630) [20:46:39] (03CR) 10Legoktm: [C: 031] Disable custom 'checkstyleResult' failure link for phan jobs [integration/config] - 10https://gerrit.wikimedia.org/r/413060 (https://phabricator.wikimedia.org/T180630) (owner: 10Krinkle) [20:48:18] legoktm: Thx, will roll out later [20:48:24] :) [20:48:29] legoktm: btw, any obvious reason why phpcs didn't catch this? https://gerrit.wikimedia.org/r/#/c/411286/4/tests/phpunit/includes/config/EtcdConfigTest.php [20:48:48] didn't catch what? [20:49:16] ah [20:49:18] the missing spaces around = [20:49:18] spacing issue [20:49:42] I'm not sure [20:49:45] (03PS3) 10Krinkle: Disable custom 'checkstyleResult' failure link for phan jobs [integration/config] - 10https://gerrit.wikimedia.org/r/413060 (https://phabricator.wikimedia.org/T180630) [20:49:47] (03CR) 10Krinkle: [C: 032] Disable custom 'checkstyleResult' failure link for phan jobs [integration/config] - 10https://gerrit.wikimedia.org/r/413060 (https://phabricator.wikimedia.org/T180630) (owner: 10Krinkle) [20:51:10] (03Merged) 10jenkins-bot: Disable custom 'checkstyleResult' failure link for phan jobs [integration/config] - 10https://gerrit.wikimedia.org/r/413060 (https://phabricator.wikimedia.org/T180630) (owner: 10Krinkle) [20:51:36] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/413060 [20:51:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:52:00] 10Continuous-Integration-Config, 10Performance-Team: Default link by jenkins-bot on Gerrit for mediawiki-core-php70-phan-docker does not show failures - https://phabricator.wikimedia.org/T180630#3990848 (10Krinkle) 05Open>03Resolved a:03Krinkle [20:52:26] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T183961#3990853 (10mmodell) [20:59:25] no_justification heh they special case the js for gitiles. All that js just to make things confusing heh [20:59:26] https://github.com/GerritCodeReview/gerrit/commit/a28dee069ff09ddf43c7151a938245af7eba3f89 [21:00:10] no love to gitiles for now [21:00:30] Hauskatze huh? [21:00:49] don't like gitiles, I prefer difussion or github [21:00:52] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T183961#3990872 (10mmodell) [21:01:37] 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible, 10User-Ryasmeen: Getting 400 error while trying to open a page with VE in Beta cluster - https://phabricator.wikimedia.org/T187837#3990880 (10Ryasmeen) 05Open>03Resolved [21:08:22] PROBLEM - Puppet errors on deployment-mx02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:15:53] no_justification i got the logo deployed to https://gerrit-new.wmflabs.org/r/?polygerrit=1 :) [21:21:50] (03CR) 10Hashar: [C: 04-1] "We gotta verify it is actually passing. Last time I tried the test suite was failing in multiple places." [integration/config] - 10https://gerrit.wikimedia.org/r/413227 (owner: 10Jforrester) [21:31:36] Can someone in releng respond to https://phabricator.wikimedia.org/T184805 ? I'm not sure I understand the question there but the DBAs are saying that they're blocked (on a thing I need) for want of a response from releng. [21:36:34] no_justification /me has finally found time to backport https://gerrit-review.googlesource.com/c/gerrit/+/161150 to 2.15 :) [21:36:38] (03CR) 10Brion VIBBER: "I have three patches to 3d2png which resolve the tests for me locally, linked from https://phabricator.wikimedia.org/T187904" [integration/config] - 10https://gerrit.wikimedia.org/r/413227 (owner: 10Jforrester) [21:41:57] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T183961#3991049 (10mmodell) [21:42:08] andrewbogott: I hadn't seen that. I'll respond [21:42:14] thanks [21:42:15] Easy question [21:46:58] 10Release-Engineering-Team, 10DBA, 10Operations, 10cloud-services-team, 10wikitech.wikimedia.org: Move some wikis to s5 - https://phabricator.wikimedia.org/T184805#3896851 (10demon) >>! In T184805#3896861, @jcrespo wrote: > Only adding #releng and #wmcs in case they can think of a reason not to move them... [21:49:39] (03CR) 10Hashar: [C: 04-1] "The CI containers definitions are hold in the repository integration/config.git under /dockerfiles/ . The definitions are consumed by a " [integration/config] - 10https://gerrit.wikimedia.org/r/413227 (owner: 10Jforrester) [21:49:50] andrewbogott: Ok ^. Sorry for not getting to that sooner....the tldr is "oh no problem go ahead" [21:50:20] no_justification: that's what I figured, thanks :) [21:56:47] (03CR) 10Brion VIBBER: "Awesome, thanks. :)" [integration/config] - 10https://gerrit.wikimedia.org/r/413227 (owner: 10Jforrester) [21:58:31] 10Release-Engineering-Team (Watching / External), 10DBA, 10Operations, 10cloud-services-team, 10wikitech.wikimedia.org: Move some wikis to s5 - https://phabricator.wikimedia.org/T184805#3991103 (10greg) [21:59:03] brion: thank you a ton for 3d2png, I gave it a try last summer but I have failed to figure out what was going on :] [21:59:09] brion: I will take care of the CI madness [21:59:18] :D [22:00:05] (03PS1) 10Hashar: Container and experimental jobs for 3d2png [integration/config] - 10https://gerrit.wikimedia.org/r/413261 [22:00:42] (03Abandoned) 10Hashar: Container and experimental jobs for 3d2png [integration/config] - 10https://gerrit.wikimedia.org/r/413261 (owner: 10Hashar) [22:04:41] samples/DavidStatue.stl 40M bah [22:06:00] yeah some decimation of meshes would go a long way on some of these models [22:06:55] James_F: thank you again for tricking me into adding s [22:07:06] James_F: thank you again for tricking me into adding CI support to yet another repo (3d2png) :-]]] [22:07:45] brion: it is not going to be a concern for CI. Network is fast enough and it is not like that repo will have thousands of builds per hour [22:07:58] yay [22:09:44] hashar: ;-) [22:12:53] 10Phabricator: Support token or cert in conduit.connect - https://phabricator.wikimedia.org/T186414#3991141 (10mmodell) As far as I understand it, token was introduced to replace the need for establishing a session with conduit.connect. So, a conduit token is equivalent to a sessionKey, you just need to include... [22:13:25] yay, cc no_justification ^^ [22:13:47] think we may have a peminent fix for T186370 now [22:13:47] T186370: its-phabricator seems to be broken on gerrit 2.14 - https://phabricator.wikimedia.org/T186370 [22:14:59] CXX(target) Release/obj.target/libANGLE/angle/src/common/event_tracer.o [22:14:59] gotta love npm compiling random stuff :] [22:16:50] that's part of the GL layer :D [22:16:59] 10Phabricator: Support token or cert in conduit.connect - https://phabricator.wikimedia.org/T186414#3991161 (10Paladox) Thanks. Will create a wip upstream for this :). [22:17:19] it uses a node+native implementation of WebGL backed by mesa, kinda wacky [22:18:34] (03PS2) 10Hashar: Container and experimental jobs for 3d2png [integration/config] - 10https://gerrit.wikimedia.org/r/413227 (owner: 10Jforrester) [22:21:01] Project mwext-phpunit-coverage-publish build #1278: 04FAILURE in 45 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/1278/ [22:24:18] (03CR) 10Hashar: [C: 032] "That seems to fail "properly" on my local machine :]" [integration/config] - 10https://gerrit.wikimedia.org/r/413227 (owner: 10Jforrester) [22:26:32] (03Merged) 10jenkins-bot: Container and experimental jobs for 3d2png [integration/config] - 10https://gerrit.wikimedia.org/r/413227 (owner: 10Jforrester) [22:29:51] what is the new assignee field in gerrit reviews? [22:30:45] subbu hi, it's used to get some one to review your change [22:30:57] ie for example i assign example 1 to review example change [22:31:36] !log Building docker image releng/npm-test-3d2png:0.1.0 and reloading Zuul | https://gerrit.wikimedia.org/r/413227 [22:31:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:32:01] i see .. so, it is to pick one among the list of reviewers who is responsible for finishing the review? [22:35:31] (03PS1) 10Hashar: Fix 3d2png repo name in Zuul configuration [integration/config] - 10https://gerrit.wikimedia.org/r/413269 [22:35:48] subbu yeh [22:36:04] k [22:37:39] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T183959#3991207 (10demon) 05Open>03Resolved [22:37:50] (03CR) 10Hashar: [C: 032] "And follow up: https://gerrit.wikimedia.org/r/413269 Fix 3d2png repo name in Zuul configuration" [integration/config] - 10https://gerrit.wikimedia.org/r/413227 (owner: 10Jforrester) [22:38:05] brion: so you can now try commenting in Gerrit "check experimental" and that will trigger the job [22:38:24] an example running from a dummy change is https://integration.wikimedia.org/ci/job/3d2png-npm-node-6-docker/1/console [22:38:33] it takes a while compiling dependencies for the gl module [22:38:45] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, and 2 others: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#3991212 (10mmodell) [22:38:49] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, and 2 others: Phabricator: Clean up deadlocked apache processes - https://phabricator.wikimedia.org/T187790#3991211 (10mmodell) 05Open>03Resolved [22:40:30] Project mwext-phpunit-coverage-publish build #1279: 04STILL FAILING in 42 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/1279/ [22:42:47] hashar: Yeah, I didn't make a typo in the repo name. ;-P [22:44:24] ;d [22:44:42] i did a check experimental on https://gerrit.wikimedia.org/r/#/c/413183/ [22:44:50] other patches would need the same [22:44:57] and when ever that is passing we can promote the job ;- [22:58:57] no_justification wip fix https://gerrit-review.googlesource.com/c/plugins/its-phabricator/+/161170 :) [22:59:07] brion: James_F: so there is some basic job running for 3d2png. No clue whether the failure is due to something missing in the container or a legit test failure :] [22:59:12] (i will need to run the tests on that to see if they fail) [22:59:32] I am off for today though :] [23:06:03] thx! [23:06:05] ah yer gone :D [23:14:15] * paladox tests the change out :) [23:15:32] woo that works [23:16:46] oh joys, lots of tests failures. [23:16:50] * paladox fixes :) [23:31:19] no_justification oh wow, this is a large list of fixes in the rc3 heh https://groups.google.com/forum/#!topic/repo-discuss/O7vHCqhM8a0 [23:31:29] (expecially polygerrit :)) [23:41:10] "http://crbug.com/gerrit/7668 NoteDb migration corrupts the site " [23:41:12] Hah! [23:55:09] no_justification yeh, they fixed that in jgit. [23:55:25] looks to all be safe now. I migrated gerrit-new, before that bug started so that was safe. [23:55:46] i then migrated gerrit-test after they fixed the problem [23:56:43] test fails with https://phabricator.wikimedia.org/P6729 [23:56:58] some how i doint see how that is failling, i updated the tests correctly i think