[00:04:24] 10Release-Engineering-Team (Kanban), 10Scap (Tech Debt Sprint FY201718-Q2), 10scap2, 10Patch-For-Review: Eliminate symlinks in mediawiki-config (as much as possible) - https://phabricator.wikimedia.org/T126306#3932982 (10demon) a:05demon>03None Won't be able to wrap this up soon, kinda busy. [00:05:34] 10Release-Engineering-Team (Kanban), 10Wikimedia-Incident: Address proximity of service deployments to train deployments problem - https://phabricator.wikimedia.org/T182733#3932990 (10demon) 05Open>03Resolved Good enough for now. [00:12:12] 10Gerrit, 10Gerrit-Migration: Provide static dump of Gerrit - https://phabricator.wikimedia.org/T617#3933023 (10demon) 05Open>03Resolved a:03demon >>! In T617#1843228, @greg wrote: >>>! In T617#1251499, @Ricordisamoa wrote: >>>>! In T617#1251479, @mmodell wrote: >>> Well bugzilla got imported into phabri... [00:12:34] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Gerrit-Migration: Provide static dump of Gerrit - https://phabricator.wikimedia.org/T617#3933026 (10demon) 05Resolved>03stalled [00:13:54] 10Gerrit: Support pulling in Picture of the day for the gerrit login page - https://phabricator.wikimedia.org/T185348#3933050 (10demon) 05Open>03declined In retrospect, this isn't nearly cool enough for the effort involved [00:17:44] 10Gerrit, 10Release-Engineering-Team (Someday), 10Operations: Reimage cobalt as stretch - https://phabricator.wikimedia.org/T176774#3933076 (10demon) p:05Low>03Lowest [00:20:47] 10Gerrit, 10Release-Engineering-Team (Someday), 10Operations: Reimage cobalt as stretch - https://phabricator.wikimedia.org/T176774#3933088 (10Dzahn) I wouldn't call this Lowest. Stretch is stable and we should have both gerrit servers on same distro version. I wonder what our blockers were. [00:21:16] 10Gerrit, 10Release-Engineering-Team (Someday), 10Operations: Reimage cobalt as stretch - https://phabricator.wikimedia.org/T176774#3933090 (10demon) No blockers, other than the time I could ever possibly find to do it. [00:53:13] no_justification: Hm.. so following yesterday's outage, seems like https://phabricator.wikimedia.org/T121597 would've prevented that. Assuming we find that that the canary logstash check was working yesterday, it might make sense to re-prioritise. The pre-promote local http check is much simpler than than what we ended up doing with logstash. In theory, the logstash check should catch a supetset of issues that a quick local check [00:53:13] would, but in practice, I think we've seen over the past year that it doesn't, not yet anyway. [00:54:03] Even a single http req for enwiki/MainPage or maintenance script eval would've caught most if not all fatal outages we've had :) [01:04:40] 10Scap (Scap3-MediaWiki-MVP), 10scap2, 10Wikimedia-Incident: Implement MediaWiki pre-promote checks - https://phabricator.wikimedia.org/T121597#3933273 (10Krinkle) [01:37:15] 10Scap (Scap3-MediaWiki-MVP), 10scap2, 10Wikimedia-Incident: Implement MediaWiki pre-promote checks - https://phabricator.wikimedia.org/T121597#3933336 (10Krinkle) [01:48:41] 10Release-Engineering-Team (Kanban), 10Scap, 10Patch-For-Review: Scap canary has a shifting baseline - https://phabricator.wikimedia.org/T183999#3933343 (10Krinkle) @thcipriani @greg In light of yesterday's incident, gentle reminder for T121597. The canary/logstash check can catch a wide range of errors fro... [02:10:35] 10Scap (Scap3-MediaWiki-MVP), 10scap2, 10Wikimedia-Incident: Implement MediaWiki pre-promote checks - https://phabricator.wikimedia.org/T121597#3933352 (10Krinkle) [02:11:49] Krinkle: Ack. I know thcipriani and I were planning to knock out T136839 tomorrow [02:11:50] T136839: Create a script to run test requests for the MediaWiki service - https://phabricator.wikimedia.org/T136839 [02:12:01] (I think that's the task....basically last blocker was getting the swagger spec into the docroot) [02:13:01] no_justification: Nice. I was previously hoping it wouldn't be blocked on that (in favour of checking a few urls directly on tin). But if we already have this (almost) in place, that'd be pretty cool! [02:13:21] I'll sync up with you tomorrow so we're on the same page [02:13:39] no_justification: I do want to make sure that we still consider stderr from php as well, given an unconditional php notice in wmf-config wouldn't likely lead to an http response that swagger can invalidate. [02:14:08] re: echo 1 | mwscript eval.php, or some other mwscript, or other way of checking php errors locally before canaries. [02:14:54] Can we pick this up tomorrow? My brain is pretty fried today, and I'm in desperate need of food and adult beverages [02:15:41] Sure, no rush :) Thanks for responding. [02:20:31] Yeah no worries, have a good evening! [02:50:35] 10Beta-Cluster-Infrastructure, 10Performance-Team: Make MediaWiki profiler in Beta match production - https://phabricator.wikimedia.org/T180766#3933367 (10Krinkle) [02:51:21] 10Beta-Cluster-Infrastructure, 10Performance-Team: Make MediaWiki profiler in Beta match production - https://phabricator.wikimedia.org/T180766#3768748 (10Krinkle) [02:52:26] 10Beta-Cluster-Infrastructure, 10Performance-Team: Make MediaWiki profiler in Beta match production - https://phabricator.wikimedia.org/T180766#3768748 (10Krinkle) >>! In T180766#3933248, @gerritbot wrote: > Change 403974 **merged** by jenkins-bot: > [operations/mediawiki-config@master] Initial profiler for Be... [02:59:11] 10Beta-Cluster-Infrastructure, 10Performance-Team: Set up XHGui for Beta Cluster - https://phabricator.wikimedia.org/T180761#3933371 (10Krinkle) [02:59:17] 10Beta-Cluster-Infrastructure, 10Performance-Team: Set up XHGui for Beta Cluster - https://phabricator.wikimedia.org/T180761#3768665 (10Krinkle) [03:00:08] 10Beta-Cluster-Infrastructure, 10Performance-Team: Make MediaWiki profiler in Beta match production - https://phabricator.wikimedia.org/T180766#3933373 (10Krinkle) 05Open>03Resolved Closing as this unblocks the parent task. The remaining work is separated into {T180761}. [03:00:12] Project mediawiki-core-code-coverage-php7 build #56: 04STILL FAILING in 12 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-php7/56/ [03:00:18] Project mediawiki-core-code-coverage build #3296: 04STILL FAILING in 17 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/3296/ [03:06:43] Project mediawiki-core-code-coverage-php7 build #57: 04STILL FAILING in 5.8 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-php7/57/ [03:40:39] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<55.56%) [04:01:22] Yippee, build fixed! [04:01:23] Project mediawiki-core-code-coverage-php7 build #58: 09FIXED in 47 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-php7/58/ [05:22:51] (03CR) 1001tonythomas: [C: 031] Newsletter: Add Echo as dependency [integration/config] - 10https://gerrit.wikimedia.org/r/401407 (owner: 10Florianschmidtwelzow) [05:25:57] (03PS2) 10Legoktm: Newsletter: Add Echo as dependency [integration/config] - 10https://gerrit.wikimedia.org/r/401407 (owner: 10Florianschmidtwelzow) [05:25:59] (03PS2) 10Legoktm: Add DeleteUserPages extension [integration/config] - 10https://gerrit.wikimedia.org/r/406852 (owner: 10Skizzerz) [05:26:01] (03PS7) 10Legoktm: Whitelist few users in CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/406242 (owner: 10Zoranzoki21) [05:26:03] (03PS3) 10Legoktm: Archive extension-CommunityHiring [integration/config] - 10https://gerrit.wikimedia.org/r/406511 (https://phabricator.wikimedia.org/T185845) (owner: 10MarcoAurelio) [05:26:05] (03PS4) 10Legoktm: Archive extension-CommunityApplications [integration/config] - 10https://gerrit.wikimedia.org/r/406501 (https://phabricator.wikimedia.org/T185844) (owner: 10MarcoAurelio) [05:26:07] (03PS2) 10Legoktm: Enable jenkins on wikibase/property-suggester-scripts [integration/config] - 10https://gerrit.wikimedia.org/r/406170 (https://phabricator.wikimedia.org/T185196) (owner: 10Ladsgroup) [05:26:09] (03PS4) 10Legoktm: Use extension-unittests-composer-non-voting for SendGrid extension testing [integration/config] - 10https://gerrit.wikimedia.org/r/404748 (https://phabricator.wikimedia.org/T185115) (owner: 10Phantom42) [05:26:11] (03PS3) 10Legoktm: Add mediawiki/libs/ObjectFactory [integration/config] - 10https://gerrit.wikimedia.org/r/406795 (https://phabricator.wikimedia.org/T147167) (owner: 10BryanDavis) [05:26:20] (03CR) 10Legoktm: [C: 032] Newsletter: Add Echo as dependency [integration/config] - 10https://gerrit.wikimedia.org/r/401407 (owner: 10Florianschmidtwelzow) [05:26:26] (03CR) 10Legoktm: [C: 032] Add DeleteUserPages extension [integration/config] - 10https://gerrit.wikimedia.org/r/406852 (owner: 10Skizzerz) [05:26:30] (03CR) 10Legoktm: [C: 032] Whitelist few users in CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/406242 (owner: 10Zoranzoki21) [05:26:35] (03CR) 10Legoktm: [C: 032] Archive extension-CommunityHiring [integration/config] - 10https://gerrit.wikimedia.org/r/406511 (https://phabricator.wikimedia.org/T185845) (owner: 10MarcoAurelio) [05:26:40] (03CR) 10Legoktm: [C: 032] Archive extension-CommunityApplications [integration/config] - 10https://gerrit.wikimedia.org/r/406501 (https://phabricator.wikimedia.org/T185844) (owner: 10MarcoAurelio) [05:26:44] (03CR) 10Legoktm: [C: 032] Enable jenkins on wikibase/property-suggester-scripts [integration/config] - 10https://gerrit.wikimedia.org/r/406170 (https://phabricator.wikimedia.org/T185196) (owner: 10Ladsgroup) [05:26:49] (03CR) 10Legoktm: [C: 032] Use extension-unittests-composer-non-voting for SendGrid extension testing [integration/config] - 10https://gerrit.wikimedia.org/r/404748 (https://phabricator.wikimedia.org/T185115) (owner: 10Phantom42) [05:26:55] (03CR) 10Legoktm: [C: 032] Add mediawiki/libs/ObjectFactory [integration/config] - 10https://gerrit.wikimedia.org/r/406795 (https://phabricator.wikimedia.org/T147167) (owner: 10BryanDavis) [05:27:45] (03Merged) 10jenkins-bot: Newsletter: Add Echo as dependency [integration/config] - 10https://gerrit.wikimedia.org/r/401407 (owner: 10Florianschmidtwelzow) [05:28:07] (03Merged) 10jenkins-bot: Add DeleteUserPages extension [integration/config] - 10https://gerrit.wikimedia.org/r/406852 (owner: 10Skizzerz) [05:28:09] (03Merged) 10jenkins-bot: Whitelist few users in CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/406242 (owner: 10Zoranzoki21) [05:28:11] (03Merged) 10jenkins-bot: Archive extension-CommunityHiring [integration/config] - 10https://gerrit.wikimedia.org/r/406511 (https://phabricator.wikimedia.org/T185845) (owner: 10MarcoAurelio) [05:28:13] (03Merged) 10jenkins-bot: Archive extension-CommunityApplications [integration/config] - 10https://gerrit.wikimedia.org/r/406501 (https://phabricator.wikimedia.org/T185844) (owner: 10MarcoAurelio) [05:28:42] !log legoktm@integration-slave-jessie-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf * [05:28:49] (03Merged) 10jenkins-bot: Enable jenkins on wikibase/property-suggester-scripts [integration/config] - 10https://gerrit.wikimedia.org/r/406170 (https://phabricator.wikimedia.org/T185196) (owner: 10Ladsgroup) [05:28:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [05:29:06] !log brought integration-slave-jessie-1003 back online after clearing disk space [05:29:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [05:29:22] (03Merged) 10jenkins-bot: Use extension-unittests-composer-non-voting for SendGrid extension testing [integration/config] - 10https://gerrit.wikimedia.org/r/404748 (https://phabricator.wikimedia.org/T185115) (owner: 10Phantom42) [05:29:25] (03Merged) 10jenkins-bot: Add mediawiki/libs/ObjectFactory [integration/config] - 10https://gerrit.wikimedia.org/r/406795 (https://phabricator.wikimedia.org/T147167) (owner: 10BryanDavis) [05:39:41] RECOVERY - Free space - all mounts on integration-slave-jessie-1003 is OK: OK: All targets OK [06:18:52] 10Continuous-Integration-Config, 10MediaWiki-extensions-SendGrid, 10Patch-For-Review: PHPUnit runner does not load required dependencies from composer.json (SendGrid extension) - https://phabricator.wikimedia.org/T185115#3933514 (10D3r1ck01) 05Open>03Resolved [06:43:40] I don't know what happened to gerrit but it's like *super* fast, so thank you [06:55:40] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [09:24:57] 10Release-Engineering-Team, 10Scap, 10Wikimedia-Incident: Scap sync-file: report the file on IRC/SAL on canary error rate failure - https://phabricator.wikimedia.org/T186064#3933621 (10Peachey88) [09:25:37] 10Release-Engineering-Team, 10Scap, 10Wikimedia-Incident: Scap: on canary failure, report the list of failed hosts - https://phabricator.wikimedia.org/T186065#3933623 (10Peachey88) [09:26:14] 10Release-Engineering-Team, 10Scap, 10Wikimedia-Incident: Scap sync-file: allow to sync multiple files in different directories - https://phabricator.wikimedia.org/T186067#3933625 (10Peachey88) [09:47:31] 10Continuous-Integration-Config, 10MediaWiki-extensions-SendGrid, 10Patch-For-Review: PHPUnit runner does not load required dependencies from composer.json (SendGrid extension) - https://phabricator.wikimedia.org/T185115#3933656 (10Phantom42) This fixed the issue! Thanks everyone for your help! Can we use `e... [11:56:21] (03PS1) 10Ladsgroup: Make ORES extension selenium tests mandatory [integration/config] - 10https://gerrit.wikimedia.org/r/406989 (https://phabricator.wikimedia.org/T184451) [12:23:15] (03PS1) 10MarcoAurelio: Register extreg-wos for tox-docker tests [integration/config] - 10https://gerrit.wikimedia.org/r/406999 [12:37:41] 10Project-Admins: Create a project for Hashtags-tool - https://phabricator.wikimedia.org/T186103#3933905 (10Samwalton9) [12:39:13] 10Project-Admins: Create a project for Tool-Hashtags - https://phabricator.wikimedia.org/T186103#3933918 (10Samwalton9) [14:57:24] 10Scap: Replace scap.args with docopt - https://phabricator.wikimedia.org/T186110#3934090 (10awight) [15:01:00] 10Scap: Replace scap.args with docopt - https://phabricator.wikimedia.org/T186110#3934101 (10awight) I see now that we're extending argparse, maybe I spoke too soon. But it still seems like a lot of untested custom logic that we might not need? [15:27:44] PROBLEM - Puppet errors on deployment-redis01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:33:43] PROBLEM - Puppet errors on deployment-redis02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:41:57] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Proton, 10Readers-Web-Backlog, and 2 others: Set up Jenkins for chromium-render and chromium-render-deploy repositories - https://phabricator.wikimedia.org/T179552#3934219 (10pmiazga) a:03pmiazga [15:44:11] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Wikimedia Portals needs libpng-dev for npm-browser-node-6 tests - https://phabricator.wikimedia.org/T186117#3934233 (10Jdrewniak) [16:13:17] 10Phabricator, 10Phabricator (Upstream), 10Epic, 10Upstream: [EPIC] Gather requirements from teams for Phab project management feature requests - https://phabricator.wikimedia.org/T105404#3934355 (10Aklapper) [16:13:20] 10Phabricator (Upstream), 10Upstream: allow users to export maniphest advanced search to csv - https://phabricator.wikimedia.org/T103009#3934352 (10Aklapper) 05stalled>03Open p:05Lowest>03Normal https://secure.phabricator.com/T13049 fixed this: In upstream under "Use Results > Export Data", a CSV item... [16:20:05] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Collaboration-Team-Triage, 10Notifications, and 5 others: New usermessage browser test is blocking merges in Minerva skin and Echo extension - https://phabricator.wikimedia.org/T185928#3934400 (10Addshore) > It's also not clear why this... [16:20:20] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Collaboration-Team-Triage, 10Notifications, and 5 others: New usermessage browser test is blocking merges in Minerva skin and Echo extension - https://phabricator.wikimedia.org/T185928#3934401 (10Addshore) Can this be closed as resolved... [16:25:58] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Collaboration-Team-Triage, 10Notifications, and 5 others: New usermessage browser test is blocking merges in Minerva skin and Echo extension - https://phabricator.wikimedia.org/T185928#3934416 (10Jdlrobson) The test could be restored in... [16:26:06] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Collaboration-Team-Triage, 10Notifications, and 5 others: New usermessage browser test is blocking merges in Minerva skin and Echo extension - https://phabricator.wikimedia.org/T185928#3934417 (10Jdlrobson) p:05Unbreak!>03Normal [16:27:48] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Collaboration-Team-Triage, 10Notifications, and 5 others: New usermessage browser test is blocking merges in Minerva skin and Echo extension - https://phabricator.wikimedia.org/T185928#3934433 (10Addshore) It could indeed be moved to Ve... [16:43:01] Yippee, build fixed! [16:43:02] Project mediawiki-core-code-coverage build #3297: 09FIXED in 1 hr 43 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/3297/ [17:26:37] 10Phabricator (Upstream), 10Upstream: allow users to export maniphest advanced search to csv - https://phabricator.wikimedia.org/T103009#3934563 (10DStrine) Yay! I'm looking forward to trying this out. [17:28:48] 10Release-Engineering-Team (Kanban), 10Wiki-Setup (Close): Close chairwiki - https://phabricator.wikimedia.org/T184961#3934572 (10demon) Account created (your usual username "Schiste"), temporary password sent. Sorry for the lag--all hands + team offsite :) [17:41:53] PROBLEM - Puppet errors on deployment-aqs01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [17:53:53] 10Beta-Cluster-Infrastructure, 10Analytics, 10Analytics-EventLogging, 10User-Elukey: EventLogging broken in beta - https://phabricator.wikimedia.org/T185952#3929484 (10Tbayer) >>! In T185952#3930959, @Tbayer wrote: > Thanks @elukey - yes, this was just a guess, based on the fact that the Hive tables stoppe... [18:00:11] 10Beta-Cluster-Infrastructure, 10Analytics, 10Analytics-EventLogging, 10User-Elukey: EventLogging broken in beta - https://phabricator.wikimedia.org/T185952#3929484 (10Nuria) @Tbayer this ticket is about our testing environment, there is no hadoop there. The kafka work @elukey is doimg is on labs. [18:02:08] (03PS2) 10Krinkle: Run tests on a Linux agent for WebPageTest [integration/config] - 10https://gerrit.wikimedia.org/r/406283 (https://phabricator.wikimedia.org/T165626) (owner: 10Phedenskog) [18:03:57] 10Beta-Cluster-Infrastructure, 10Analytics, 10Analytics-EventLogging, 10User-Elukey: EventLogging broken in beta - https://phabricator.wikimedia.org/T185952#3934702 (10elukey) Will keep this open for a couple of days to check the disk consumption after the last change. [18:14:42] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Collaboration-Team-Triage, 10Notifications, and 3 others: New usermessage browser test is blocking merges in Minerva skin and Echo extension - https://phabricator.wikimedia.org/T185928#3928891 (10Jdlrobson) [18:16:56] RECOVERY - Puppet errors on deployment-aqs01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:31:18] (03CR) 10Krinkle: [C: 032] "Deployed the new job." [integration/config] - 10https://gerrit.wikimedia.org/r/406283 (https://phabricator.wikimedia.org/T165626) (owner: 10Phedenskog) [18:32:45] (03Merged) 10jenkins-bot: Run tests on a Linux agent for WebPageTest [integration/config] - 10https://gerrit.wikimedia.org/r/406283 (https://phabricator.wikimedia.org/T165626) (owner: 10Phedenskog) [18:39:22] PROBLEM - Puppet errors on deployment-mx is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:40:25] 10Beta-Cluster-Infrastructure: Login session bug on Beta Commons - https://phabricator.wikimedia.org/T186133#3934774 (10Ramsey-WMF) [18:43:43] 10Gerrit, 10Release-Engineering-Team: Upgrade gerrit from 2.14.6 to 2.14.7 - https://phabricator.wikimedia.org/T186135#3934807 (10Paladox) [18:43:56] 10Gerrit, 10Release-Engineering-Team (Someday), 10Patch-For-Review: Update gerrit to 2.14.6 - https://phabricator.wikimedia.org/T156120#3310187 (10Paladox) [18:49:10] no_justification: So, will you run cleanupPreferences in --dryrun on some small wiki soon? :-) [18:49:17] Absolutely not [18:49:20] I have zero spare time [18:49:31] We have a rather large list of other deployers. [18:49:35] Understood. :-( [18:50:03] Yes, but if they run it and it screws stuff up it'll be you fixing the mess, hence I imagined you'd want to get ahead of the plan. [19:06:55] dry-run can hardly do any mess :-) [19:29:46] 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: Login session bug on Beta Commons - https://phabricator.wikimedia.org/T186133#3934954 (10Tgr) Session unreliability is a long-standing issue on beta (see e.g. {T172560}; there was another task that I can't find now where @Krenair tracked it down to re... [19:31:15] 10Phabricator: Herald rule to tag Maps from Collaboration-Maps - https://phabricator.wikimedia.org/T186143#3934964 (10Mattflaschen-WMF) [19:33:34] 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: Login session bug on Beta Commons - https://phabricator.wikimedia.org/T186133#3934970 (10Krenair) >>! In T186133#3934954, @Tgr wrote: > Session unreliability is a long-standing issue on beta (see e.g. {T172560}; there was another task that I can't fin... [19:37:21] 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: Login session bug on Beta Commons - https://phabricator.wikimedia.org/T186133#3934977 (10Tgr) That's the one, thanks. [19:54:02] 10Continuous-Integration-Infrastructure (shipyard): Migrate operations-mw-config-composer-hhvm-jessie to Docker - https://phabricator.wikimedia.org/T186145#3935015 (10hashar) [19:54:55] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban): Migrate operations-mw-config-composer-hhvm-jessie to Docker - https://phabricator.wikimedia.org/T186145#3935026 (10hashar) [19:57:54] James_F: Do you remember why the noc.wm.o/conf/ files are all foo.php.txt? [19:58:12] As long as we tell apache to serve it as text....why should the extension matter? [19:58:25] no_justification: For caching? [19:58:34] (Others', not ours.) [19:58:39] Why....do we care? [19:59:03] i think the extension for that is .phps [19:59:10] to view php source [20:00:03] Yes. We should do that [20:00:06] it makes it really hard to fuck up and accidentally execute those files? [20:00:16] *.phps would Just Work? [20:00:46] Maybe. [20:01:16] We can test this theory :p [20:01:26] Also: I hate symlinks [20:01:39] We should do some analytics on the traffic to noc.wm.o and see how many people actually care ;-) [20:02:49] Also: we still get a non-zero amount of traffic to WikipediaFirefoxMobileOS :( [20:02:53] I hate that submodule [20:05:53] no_justification: I'd guess that the number of people who care about noc.* who aren't in this channel is lower than ten. :-) [20:06:50] That would be my assumption. The dblists are probably the most widely cared about....I could see third party tools (in cloud services & otherwise) using it to generate lists of wikis to hit. [20:06:54] usually then it turns out some external project relies on noc. existing and copies all the configs or breaks completely :) [20:07:02] but yea [20:07:31] 10Gerrit, 10Release-Engineering-Team (Someday), 10Patch-For-Review: Update gerrit to 2.14.6 - https://phabricator.wikimedia.org/T156120#3935051 (10Paladox) This update will be done on friday https://wikitech.wikimedia.org/wiki/Deployments#Friday,_February_02 [20:07:43] hates submodules too [20:07:53] It's not even that its a submodule [20:08:52] It's that it's a completely dead/abandoned project that nobody supports (and contains a fork of MobileFrontend) [20:08:55] And confuses my grepping :p [20:09:27] no_justification: Just kill it? I know, I know, TimBL's CoolURIsDontChange, but… [20:09:44] I don't care about that. [20:09:49] something about that "sunsetting services" project? [20:10:00] Last time I tried, someone (I forget whom) said "maintenance cost is zero, who cares?" [20:10:05] And I didn't wanna fight it [20:10:19] It's clearly not zero, it's getting in your way. [20:10:28] Fuck it, let's jfdi now [20:10:30] #yolo [20:11:35] 10Phabricator, 10Release-Engineering-Team (Kanban), 10monitoring, 10Browser-Tests, 10User-zeljkofilipin: Develop tests for phabricator search to detect regressions / search quality issues - https://phabricator.wikimedia.org/T182160#3935059 (10mmodell) The first part of this is now finished. We have the s... [20:11:40] Apparently saying #yolo triggers greg-g's spidey sense. [20:12:37] 10Phabricator, 10Wikimedia-Logstash: Improve error reporting / integration between Kibana and Phabricator - https://phabricator.wikimedia.org/T185155#3935072 (10mmodell) p:05Normal>03Low a:05mmodell>03None no longer a priority [20:13:45] 10Continuous-Integration-Config, 10Patch-For-Review: Phase out jobs "pplint-HEAD" and "erblint-HEAD" - https://phabricator.wikimedia.org/T154894#3935085 (10hashar) [20:13:48] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10Puppet: Get rid of "import realm.pp" in manifests/site.pp - https://phabricator.wikimedia.org/T154915#3935082 (10hashar) 05Open>03stalled a:05hashar>03None Pending https://gerrit.wikimedia.or... [20:14:01] 10Phabricator: Add support for task types - https://phabricator.wikimedia.org/T93499#3935086 (10mmodell) a:05mmodell>03None [20:14:51] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Jenkins, 10Upstream: Jenkins Gearman plugin has deadlock on executor threads (was: Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - https://phabricator.wikimedia.org/T72597#3935101 (1... [20:15:30] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Patch-For-Review: When jenkins kills a build due to max execution time the docker containers stay running - https://phabricator.wikimedia.org/T176747#3935112 (10hashar) 05Open>03Resolved The bulk of it has been don... [20:15:40] * James_F grins at no_justification. [20:16:03] Bam: https://gerrit.wikimedia.org/r/407051 [20:16:03] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Jenkins: Move the beta cluster jobs to a dedicated/standalone Jenkins instance - https://phabricator.wikimedia.org/T183164#3845630 (10hashar) a:05hashar>03None I have no bandwidth available to do that. [20:17:16] 10Release-Engineering-Team (Kanban), 10MediaWiki-Vagrant, 10Patch-For-Review, 10User-zeljkofilipin: Create a test suite that compiles mediawiki-vagrant puppet manifests - https://phabricator.wikimedia.org/T183570#3935120 (10hashar) a:05hashar>03None **status update** We need a Jenkins job that compile... [20:17:19] 10Gerrit, 10Gerrit-Migration: Provide static dump of Gerrit - https://phabricator.wikimedia.org/T617#3935122 (10demon) a:05demon>03None [20:18:32] no_justification: Ha, I made https://gerrit.wikimedia.org/r/407052 ;-) [20:18:35] * James_F abandons his. [20:20:11] no_justification or thcipriani, could one of you help me troubleshoot a scap problem? I'm definitely doing something obviously wrong and silly but Bryan and I have burned several hours on this already. [20:20:32] Gimme a few minutes. [20:20:38] I'm in the middle of deleting code! [20:20:42] (my favorite hobby around here) [20:20:54] sure, just ping me when you have time to switch context [20:21:30] deleting code is fun [20:33:33] andrewbogott: I was about to go to lunch, but gimme the short version so I can think about it and help ya when I come back [20:36:11] no_justification: trying to set up a scap server and client on VMs: [20:36:18] abogott-scapserver.testlabs.eqiad.wmflabs [20:36:28] and abogott-horizonsourcedeploy.testlabs.eqiad.wmflabs [20:36:43] scap tries to call out to the target (horizonsourcedeploy) and can't because the key is rejected [20:36:45] that's it :) [20:36:59] I will make sure you have logins on those boxes so you can see for yourself. [20:37:14] This is unpleasant: pytest & pytest-cov crash when I test code that uses os.chdir, even with a context manager to reset to the old cwd. [20:37:31] andrewbogott: you'll need "keyholder arm" to load the key to be used [20:37:34] The particular thing I'm trying to deploy is in /srv/deployment/horizon/deploy/ but I don't think that's relevant for this problem, just a simple ssh also fails [20:37:37] mutante: yeah, done [20:38:00] andrewbogott is the ssh key located where ever scap stores it? [20:39:01] PROBLEM - Puppet errors on deployment-videoscaler01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [20:39:01] PROBLEM - Puppet errors on deployment-mediawiki07 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [20:39:47] is it a new deployment key made for this project? [20:40:04] awight: that pytest issue might actually be what twentyafterfour was running into. Plz file a task! [20:40:06] like f.e. ./modules/secret/secrets/keyholder/deploy_librenms.pub [20:40:29] no_justification: iinteresting… okay I have a unit test that demonstrates it, so I’ll upload with [DNM] [20:40:30] no_justification: what is your username on wikitech? [20:40:50] 10Beta-Cluster-Infrastructure, 10Analytics, 10Analytics-EventLogging, 10User-Elukey: EventLogging broken in beta - https://phabricator.wikimedia.org/T185952#3935208 (10Tgr) EventLogging records now end up in `/srv/log/eventlogging/all-events.log` but the DB tables seem broken again. [20:40:52] mutante: it's just whatever random fake-ish key comes from labs/private [20:40:59] but that key works in another pre-existing project [20:41:57] while all the keys have the same passphrase now, afaict each project still has its own deployment key [20:42:09] well, besides "deploy_service" [20:42:38] but if one is shown as loaded with keyholder.. then [20:42:52] i'd check /var/log/auth.log on the remote side [20:45:35] PROBLEM - Puppet errors on deployment-tmh01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [20:45:50] mutante: as far as I know it is trying to use deploy_service already [20:46:10] but maybe not, I will try to dig deeper [20:46:40] here is what I have currently: [20:46:42] https://www.irccloud.com/pastebin/6js1Kmk0/ [20:50:00] andrewbogott: "Chad" [20:50:09] 'k [20:50:30] Sorry for lag, I'm mobile at the moment. I'm reading but responding is slower haha [20:50:51] np — in theory you should now have login and sudo on those boxes if/when you want to poke around [20:51:12] twentyafterfour: I heard you might be interested: https://phabricator.wikimedia.org/D958 [21:09:35] (03PS1) 10Phedenskog: Remove testing IE test from the WebPageTest Linux instance. [integration/config] - 10https://gerrit.wikimedia.org/r/407067 (https://phabricator.wikimedia.org/T165626) [21:16:54] no_justification: so, I rebooted the deploy server and now it works. Just something messed up with the ssh agent state I guess :( [21:20:20] no_justification: twentyafterfour: FWIW, I have a patch to `coverage` which fixes the issue. https://github.com/nedbat/coveragepy/pull/37 [21:20:32] I’ve emailed the author, too… [21:31:29] (03CR) 10Krinkle: [C: 032] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/407067 (https://phabricator.wikimedia.org/T165626) (owner: 10Phedenskog) [21:31:34] Project mwext-phpunit-coverage-publish build #427: 04FAILURE in 2 min 35 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/427/ [21:32:00] Yippee, build fixed! [21:32:00] Project mwext-phpunit-coverage-publish build #428: 09FIXED in 25 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/428/ [21:32:56] (03Merged) 10jenkins-bot: Remove testing IE test from the WebPageTest Linux instance. [integration/config] - 10https://gerrit.wikimedia.org/r/407067 (https://phabricator.wikimedia.org/T165626) (owner: 10Phedenskog) [21:50:57] 10Beta-Cluster-Infrastructure, 10Analytics, 10Analytics-EventLogging, 10User-Elukey: EventLogging broken in beta - https://phabricator.wikimedia.org/T185952#3935453 (10Nuria) I reawoke mysql in beta, stop/start really. Must have been left in a bad state. resolving but reopen if you find other issues. I did... [21:51:38] 10Beta-Cluster-Infrastructure, 10Analytics, 10Analytics-EventLogging, 10User-Elukey: EventLogging broken in beta - https://phabricator.wikimedia.org/T185952#3935465 (10Nuria) 05stalled>03Resolved [21:51:49] 10Beta-Cluster-Infrastructure, 10Analytics, 10Analytics-EventLogging, 10User-Elukey: EventLogging broken in beta - https://phabricator.wikimedia.org/T185952#3929484 (10Nuria) 05Resolved>03stalled [21:52:07] 10Beta-Cluster-Infrastructure, 10Analytics, 10Analytics-EventLogging, 10User-Elukey: EventLogging broken in beta - https://phabricator.wikimedia.org/T185952#3929484 (10Nuria) Sorry, keeping open per @elukey's comment [21:55:08] 10Phabricator: Herald rule to tag Maps from Collaboration-Maps - https://phabricator.wikimedia.org/T186143#3934944 (10Catrope) I've created H276, lemme know if that looks good. Also, we should really just get you the rights to create/edit Herald rules. [21:55:14] 10Phabricator: Herald rule to tag Maps from Collaboration-Maps - https://phabricator.wikimedia.org/T186143#3935487 (10Catrope) 05Open>03Resolved a:03Catrope [22:01:02] !log updated mobileapps to 3d717fa on beta cluster [22:01:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:03:52] Project mwext-phpunit-coverage-publish build #437: 04FAILURE in 1 min 23 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/437/ [22:17:57] 10Phabricator (Upstream), 10Upstream: Phabricator profiles should be localizable - https://phabricator.wikimedia.org/T103466#3935529 (10mmodell) [22:18:09] 10Phabricator (Upstream), 10Developer-Relations, 10Legalpad, 10translatewiki.net, and 5 others: Use translatewiki.net to localize Phabricator - https://phabricator.wikimedia.org/T225#3935528 (10mmodell) 05Open>03Resolved [22:32:23] hmm https://phabricator.wikimedia.org/D921?vs=2488&id=2512 dosen't how purple for closed any more [22:41:07] 10Phabricator, 10Operations, 10Patch-For-Review: Switch phabricator from using apache to nginx - https://phabricator.wikimedia.org/T185644#3935575 (10mmodell) p:05High>03Low [22:42:59] 10Phabricator, 10VPS-project-codesearch: Consider adding a way to query https://codesearch.wmflabs.org/search/ from phabricator. - https://phabricator.wikimedia.org/T183608#3858510 (10mmodell) p:05Triage>03Low [22:44:24] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#3935586 (10mmodell) a:03mmodell As discussed with @dzahn at all hands, we should probably ju... [22:49:20] 10Phabricator: Phabricator should recognize "Bug: Tnnn" in commit messages same as "refs Tnnn" and connect the commit to the task - https://phabricator.wikimedia.org/T117434#3935618 (10mmodell) p:05Triage>03Low This is no longer an issue AFAIK, phabricator now has "Related Objects" which includes tasks with... [22:51:11] 10Phabricator: Herald rule to tag Maps from Collaboration-Maps - https://phabricator.wikimedia.org/T186143#3935622 (10Mattflaschen-WMF) @Catrope Looks good, thanks. [22:52:41] 10Phabricator: Error 503 from Differential when trying to upload a patch via the web interface - https://phabricator.wikimedia.org/T118664#3935628 (10mmodell) 05Open>03Resolved a:03mmodell I think this is no longer an issue. Please reopen if it happens again. [22:53:22] 10Phabricator, 10Access-Policy, 10Legalpad: Setup localisation for Legalpad to allow for interface to appear in native language - https://phabricator.wikimedia.org/T112184#3935632 (10mmodell) 05Open>03Resolved a:03mmodell This is now possible via translatewiki.net [22:54:25] 10Phabricator, 10Access-Policy, 10Legalpad: Setup localisation for Legalpad to allow for interface to appear in native language - https://phabricator.wikimedia.org/T112184#3935646 (10mmodell) see {T225} [22:55:27] 10Beta-Cluster-Infrastructure, 10Analytics, 10Analytics-EventLogging, 10User-Elukey: EventLogging broken in beta - https://phabricator.wikimedia.org/T185952#3935649 (10Tgr) MySQL seems to be running (I can connect and run queries) but new events still do not appear in the tables. (I'm testing with `InputDe... [22:55:31] 10Diffusion, 10Phabricator: Browsing submodules from within parent project changes fail - https://phabricator.wikimedia.org/T149094#3935651 (10mmodell) 05Open>03Resolved a:03mmodell is this still happening? Please reopen if you can still reproduce this. [22:57:56] 10Phabricator: Custom Policy for a file doesn't work as expected - https://phabricator.wikimedia.org/T170080#3935658 (10mmodell) 05Open>03declined yes this is the expected behavior for files - they don't have their own custom security when attached to other objects. [23:06:55] 10Phabricator, 10Release-Engineering-Team (Kanban): Make sure elasticsearch 6 is supported in phabricator - https://phabricator.wikimedia.org/T181393#3935699 (10mmodell) a:03mmodell [23:09:02] 10Continuous-Integration-Config, 10PollNY, 10Social-Tools: Patch for PollNY declaring dependency on SocialProfile inside extension.json will fail even though dependency is declared in integration/config/zuul/parameter_functions.py - https://phabricator.wikimedia.org/T185869#3935704 (10SamanthaNguyen) 05Open... [23:12:44] no_justification: ok, a new topic: I'm trying to push an upstream repo to gerrit but gerrit is rejecting it because it contains committers (specifically 'review@openstack.org') unknown to gerrit. [23:13:00] I'm told there's a way to do this and I'm not crazy for wanting to duplicate an upstream repo in gerrit... [23:13:06] Have to add "Forge Committer" [23:13:12] To ACL [23:13:15] Or is it Forge Author [23:13:26] One of them we grant globally so you can amend other people's changes and make a new patchset. [23:13:36] I think it's the latter you have to set individually [23:13:55] (we generally don't grant it everywhere since it allows you to sorta impersonate someone which is Evil) [23:14:11] no_justification forge author and forge committer [23:14:20] ok, looking... [23:14:53] 10MediaWiki-Releasing, 10RelEng-Archive-FY201718-Q1, 10Release-Engineering-Team (Kanban), 10MW-1.27-release, 10MW-1.28-release: Patch for 1.27.3/1.28.2 missing - https://phabricator.wikimedia.org/T164470#3935717 (10demon) 05Open>03Resolved Verified, gzip'd, signed and uploaded the patch, thanks for s... [23:15:56] 10Phabricator: Tag URL for milestone without board causes weird 404 rediret - https://phabricator.wikimedia.org/T186173#3935733 (10Mattflaschen-WMF) [23:16:15] 10Phabricator: Tag URL for milestone without board causes weird 404 redirect - https://phabricator.wikimedia.org/T186173#3935744 (10Mattflaschen-WMF) [23:24:55] no_justification: worked great, thank you! [23:27:18] 10Release-Engineering-Team, 10MediaWiki-Core-Tests, 10MediaWiki-extensions-ORES, 10MW-1.31-release-notes (WMF-deploy-2018-02-06 (1.31.0-wmf.20)), and 2 others: How do I test my extension's maintenance scripts? - https://phabricator.wikimedia.org/T184775#3935774 (10awight) Some docs added: https://www.medi... [23:37:32] Yippee, build fixed! [23:37:32] Project mwext-phpunit-coverage-publish build #438: 09FIXED in 32 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/438/ [23:53:54] 10Phabricator, 10Operations, 10Patch-For-Review: Switch phabricator from using apache to nginx - https://phabricator.wikimedia.org/T185644#3935820 (10Dzahn) @Joe and others: also see T182832 now