[00:00:24] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10fullscreenorgId=1 [00:03:03] (03PS1) 10Phantom42: Add BlueSpiceFoundation for BlueSpice extensions [integration/config] - 10https://gerrit.wikimedia.org/r/403325 (https://phabricator.wikimedia.org/T175794) [00:05:54] (03PS2) 10Phantom42: Add BlueSpiceFoundation dependency for BlueSpice extensions [integration/config] - 10https://gerrit.wikimedia.org/r/403325 (https://phabricator.wikimedia.org/T175794) [00:11:14] legoktm: what happened on Dec 27th? https://tools.wmflabs.org/coverage/mediawiki/chart.png [00:11:41] legoktm: I guess it's not an huge spike, in absolute numbers, just the scale/zoom makes it look huge [00:18:17] bd808: The amount of copy+paste in BS makes me sad :( [00:18:25] (and makes grepping annoying) [00:31:17] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [01:48:47] 10Gerrit, 10Release-Engineering-Team, 10releng-201516-q3, 10Developer-Relations, and 3 others: [RfC]: Migrate code review / management from Gerrit to Phabricator - https://phabricator.wikimedia.org/T119908#3888733 (10HappyDog) >>! In T119908#3887916, @EddieGP wrote: >>>! In T119908#3887786, @daniel wrote:... [04:48:07] 10Gerrit, 10Release-Engineering-Team, 10releng-201516-q3, 10Developer-Relations, and 3 others: [RfC]: Migrate code review / management from Gerrit to Phabricator - https://phabricator.wikimedia.org/T119908#3888823 (10labster) It's not as big of a a barrier to contribution as you think. Generally, when I w... [05:14:07] Yippee, build fixed! [05:14:08] Project mwext-phpunit-coverage-publish build #78: 09FIXED in 3 min 19 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/78/ [05:27:08] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<22.22%) [05:32:51] 10Continuous-Integration-Infrastructure, 10MW-1.31-release-notes (WMF-deploy-2018-01-09 (1.31.0-wmf.16)), 10MediaWiki-Platform-Team (MWPT-Q3-Jan-Mar-2018), 10Patch-For-Review: Generate PHP code coverage reports for extensions - https://phabricator.wikimedia.org/T71685#3888885 (10Legoktm) Just PHPUnit. {T11... [05:51:38] greg-g: most of https://gerrit.wikimedia.org/r/#/q/topic:missing-covers was merged that day [05:53:31] greg-g: tbh I was pretty disappointed that it only went up 2.5% [05:54:35] Based on that my estimate is that our real code coverage is probably 5% higher than what we report right now [05:54:45] due to missing or impossible to add @covers tags [06:03:53] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Cleanup, 10Repository-Admins, 10Scoring-platform-team: Investigate and clean up deprecated ORES repos - https://phabricator.wikimedia.org/T179610#3888923 (10Ladsgroup) So the plan was to move mediawiki services/ores/deploy to research/ores/deploy give it p... [06:53:01] 10Continuous-Integration-Config, 10BlueSpice, 10Patch-For-Review: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811#3888976 (10Osnard) Okay, I'll have a look at it as soon as https://gerrit.wikimedia.org/r/#/c/402826/ is merged [07:17:07] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:25:02] 10Gerrit, 10Release-Engineering-Team, 10releng-201516-q3, 10Developer-Relations, and 3 others: [RfC]: Migrate code review / management from Gerrit to Phabricator - https://phabricator.wikimedia.org/T119908#3888991 (10Paladox) >>! In T119908#3888733, @HappyDog wrote: >>>! In T119908#3887916, @EddieGP wrote:... [07:45:41] (03PS3) 10Legoktm: Add dependencies to BlueSpice* extensions [integration/config] - 10https://gerrit.wikimedia.org/r/402826 (owner: 10Robert Vogel) [07:48:08] (03CR) 10Legoktm: [C: 032] Add dependencies to BlueSpice* extensions [integration/config] - 10https://gerrit.wikimedia.org/r/402826 (owner: 10Robert Vogel) [07:49:16] (03Merged) 10jenkins-bot: Add dependencies to BlueSpice* extensions [integration/config] - 10https://gerrit.wikimedia.org/r/402826 (owner: 10Robert Vogel) [07:50:14] !log deployed https://gerrit.wikimedia.org/r/402826 [07:50:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:34:43] 10Continuous-Integration-Config, 10BlueSpice, 10Patch-For-Review: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811#2221854 (10Legoktm) I filed 3 subtasks for the different test failures I ran into. [08:39:06] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T180749#3768321 (10hashar) On tin /srv/mediawiki-staging/wikiversions.json had been left uncommitted. I have added it to git via https://gerrit.wikimedia.org/r/#/c/403360/ [08:40:52] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Zuul: Update zuul to upstream master - https://phabricator.wikimedia.org/T158243#3889124 (10hashar) The repackaged Zuul seems to work fine. I would need the package on https://people.wikimedia.org/~hashar/debs/zuul_2.5.0-8-gcbc7f62-wmf6/... [08:51:04] (03PS1) 10Hashar: debian-glkue for operations/debs/node-tunnel-agent [integration/config] - 10https://gerrit.wikimedia.org/r/403362 (https://phabricator.wikimedia.org/T183569) [08:51:15] (03CR) 10Hashar: [C: 032] debian-glkue for operations/debs/node-tunnel-agent [integration/config] - 10https://gerrit.wikimedia.org/r/403362 (https://phabricator.wikimedia.org/T183569) (owner: 10Hashar) [08:53:11] (03Merged) 10jenkins-bot: debian-glkue for operations/debs/node-tunnel-agent [integration/config] - 10https://gerrit.wikimedia.org/r/403362 (https://phabricator.wikimedia.org/T183569) (owner: 10Hashar) [09:04:42] (03PS1) 10Hashar: operations/debs/node-tunnel-agent -> voting debian-glue [integration/config] - 10https://gerrit.wikimedia.org/r/403365 (https://phabricator.wikimedia.org/T183569) [09:04:54] (03CR) 10Hashar: [C: 032] operations/debs/node-tunnel-agent -> voting debian-glue [integration/config] - 10https://gerrit.wikimedia.org/r/403365 (https://phabricator.wikimedia.org/T183569) (owner: 10Hashar) [09:05:29] 10Continuous-Integration-Config, 10MinusX, 10Google-Code-in-2017, 10MW-1.31-release-notes (WMF-deploy-2018-01-02 (1.31.0-wmf.15)), 10Patch-For-Review: Add MinusX to MediaWiki extensions and PHP library repos - https://phabricator.wikimedia.org/T175794#3889193 (10divadsn) [09:06:54] (03Merged) 10jenkins-bot: operations/debs/node-tunnel-agent -> voting debian-glue [integration/config] - 10https://gerrit.wikimedia.org/r/403365 (https://phabricator.wikimedia.org/T183569) (owner: 10Hashar) [09:22:24] 10Gerrit, 10Release-Engineering-Team, 10releng-201516-q3, 10Developer-Relations, and 3 others: [RfC]: Migrate code review / management from Gerrit to Phabricator - https://phabricator.wikimedia.org/T119908#3889210 (10HappyDog) >>! In T119908#3888823, @labster wrote: > It's not as big of a a barrier to cont... [09:37:31] (03CR) 10Hashar: [C: 032] "Merging first without any job so we can get the image published on the docker registry." [integration/config] - 10https://gerrit.wikimedia.org/r/403311 (owner: 10Hashar) [09:38:42] (03Merged) 10jenkins-bot: docker: stretch image to run rake [integration/config] - 10https://gerrit.wikimedia.org/r/403311 (owner: 10Hashar) [09:40:47] !log update docker-pkg images for releng/rake https://gerrit.wikimedia.org/r/#/c/403311/ [09:40:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:00:30] (03PS1) 10Hashar: Add experimental Docker rake jobs [integration/config] - 10https://gerrit.wikimedia.org/r/403367 [10:00:45] (03CR) 10Hashar: [C: 032] Add experimental Docker rake jobs [integration/config] - 10https://gerrit.wikimedia.org/r/403367 (owner: 10Hashar) [10:02:12] 10Release-Engineering-Team (Watching / External), 10MediaWiki-Platform-Team (MWPT-Q2-Oct-Dec-2017), 10Patch-For-Review, 10Performance-Team (Radar): Support multi-instance database hosts on mediawiki-config - https://phabricator.wikimedia.org/T178553#3889298 (10jcrespo) [10:02:48] (03Merged) 10jenkins-bot: Add experimental Docker rake jobs [integration/config] - 10https://gerrit.wikimedia.org/r/403367 (owner: 10Hashar) [10:08:46] (03PS1) 10Hashar: Allow empty /log for rake jobs [integration/config] - 10https://gerrit.wikimedia.org/r/403368 [10:10:17] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T180749#3889326 (10Addshore) [10:11:34] (03CR) 10Hashar: [C: 032] Allow empty /log for rake jobs [integration/config] - 10https://gerrit.wikimedia.org/r/403368 (owner: 10Hashar) [10:11:36] (03PS1) 10Hashar: Migrate VisualEditor rake job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/403371 [10:11:43] (03CR) 10Hashar: [C: 032] Migrate VisualEditor rake job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/403371 (owner: 10Hashar) [10:13:22] (03Merged) 10jenkins-bot: Allow empty /log for rake jobs [integration/config] - 10https://gerrit.wikimedia.org/r/403368 (owner: 10Hashar) [10:13:26] (03Merged) 10jenkins-bot: Migrate VisualEditor rake job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/403371 (owner: 10Hashar) [10:18:46] (03PS1) 10Hashar: puppet-module need a Docker rake experimental job [integration/config] - 10https://gerrit.wikimedia.org/r/403374 [10:23:42] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T180749#3889362 (10Addshore) [10:38:37] (03PS1) 10Hashar: Migrate puppet modules rake jobs to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/403377 [10:38:59] (03CR) 10Hashar: [C: 032] "I have tested them all :)" [integration/config] - 10https://gerrit.wikimedia.org/r/403377 (owner: 10Hashar) [10:39:19] 10Gerrit, 10Release-Engineering-Team, 10releng-201516-q3, 10Developer-Relations, and 3 others: [RfC]: Migrate code review / management from Gerrit to Phabricator - https://phabricator.wikimedia.org/T119908#3889378 (10Aklapper) **Please move "me too" / "UI bad" / "UI good" / "I don't like XYZ" / "+1!"s on G... [10:40:21] (03PS1) 10Hashar: Migrate wmf_styleguide-check rake job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/403379 [10:40:41] (03CR) 10Hashar: [C: 032] Migrate wmf_styleguide-check rake job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/403379 (owner: 10Hashar) [10:40:56] (03Merged) 10jenkins-bot: Migrate puppet modules rake jobs to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/403377 (owner: 10Hashar) [10:41:58] (03CR) 10Hashar: "check experimental" [ruby/api] - 10https://gerrit.wikimedia.org/r/118455 (owner: 10Hashar) [10:42:07] (03Merged) 10jenkins-bot: Migrate wmf_styleguide-check rake job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/403379 (owner: 10Hashar) [10:42:29] (03CR) 10jenkins-bot: Jenkins job validation (DO NOT SUBMIT) [ruby/api] - 10https://gerrit.wikimedia.org/r/118455 (owner: 10Hashar) [10:42:33] (03CR) 10Hashar: "check experimental" [selenium] - 10https://gerrit.wikimedia.org/r/330671 (owner: 10Hashar) [10:44:13] (03CR) 10jenkins-bot: Jenkins job validation (DO NOT SUBMIT) [selenium] - 10https://gerrit.wikimedia.org/r/330671 (owner: 10Hashar) [10:44:30] hashar: what are you doing? :) [10:48:38] (03PS1) 10Hashar: Migrate oojs/ui rake job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/403382 [10:48:54] (03CR) 10Hashar: [C: 032] Migrate oojs/ui rake job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/403382 (owner: 10Hashar) [10:50:05] (03PS1) 10Hashar: Migrate translatewiki rake job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/403383 [10:50:21] (03Merged) 10jenkins-bot: Migrate oojs/ui rake job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/403382 (owner: 10Hashar) [10:50:32] (03CR) 10Hashar: [C: 032] Migrate translatewiki rake job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/403383 (owner: 10Hashar) [10:51:58] (03Merged) 10jenkins-bot: Migrate translatewiki rake job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/403383 (owner: 10Hashar) [12:41:25] 10Beta-Cluster-Infrastructure, 10Wikimedia-Logstash: logstash-beta.wmflabs.org default dashboard missing - https://phabricator.wikimedia.org/T184602#3889603 (10Addshore) [12:52:04] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Zuul: Update zuul to upstream master - https://phabricator.wikimedia.org/T158243#3889629 (10Paladox) @hashar thanks :). Should we cc moritzm for uploading the package? [12:53:21] 10Beta-Cluster-Infrastructure, 10Wikimedia-Logstash: logstash-beta.wmflabs.org default dashboard missing - https://phabricator.wikimedia.org/T184602#3889603 (10Paladox) Does this use mediawiki vagrant for logstash installation? [13:06:58] 10Beta-Cluster-Infrastructure: deployment-fluorine puppet failure due to '/usr/sbin/usermod -u 10003 datasets' returned 4: usermod: UID '10003' already exists - https://phabricator.wikimedia.org/T117028#3889682 (10ArielGlenn) deployment-fluorine does not need this uid, the user is dead and gone and removed from... [13:34:49] (03PS1) 10Hashar: Migrate ruby/api and selenium rake job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/403398 [13:35:01] (03CR) 10Hashar: [C: 032] Migrate ruby/api and selenium rake job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/403398 (owner: 10Hashar) [13:37:00] (03CR) 10jerkins-bot: [V: 04-1] Migrate ruby/api and selenium rake job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/403398 (owner: 10Hashar) [13:37:02] (03CR) 10jerkins-bot: [V: 04-1] Migrate ruby/api and selenium rake job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/403398 (owner: 10Hashar) [13:47:38] (03PS2) 10Hashar: Migrate ruby/api and selenium rake job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/403398 [13:53:54] (03CR) 10Hashar: [C: 032] Migrate ruby/api and selenium rake job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/403398 (owner: 10Hashar) [13:55:07] (03Merged) 10jenkins-bot: Migrate ruby/api and selenium rake job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/403398 (owner: 10Hashar) [14:38:05] ah for once I'm already in the right channel: I can't re-add some of my panels to my dashboard in phab because the list of panels doesn't include them, it seems to re-get the same list of items every time I click 'load more results', an [14:38:07] d [14:38:37] autocomplete does not complete the name, though I can view the panel in the https://phabricator.wikimedia.org/dashboard/panel/ list [14:39:00] this is true for a bunch of my old panels, basically anything with less than W1000 is not showing up I think [14:39:22] and sure I could add to the problem by creating new panels identical to the old ones, but I'd rather not :-P [14:43:35] https://phabricator.wikimedia.org/typeahead/browse/PhabricatorDashboardPanelDatasource/?offset=300&format=html put 300 100 500, doesn't matter, same results [14:48:20] <_joe_> !log shutting down deployment-puppetdb01.deployment-prep.eqiad.wmflabs, unused [14:48:24] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:53:11] <_joe_> I will need someone to shut down the puppetdb instance in the phabricator project [14:53:16] PROBLEM - Host deployment-puppetdb01 is DOWN: CRITICAL - Host Unreachable (10.68.23.76) [14:55:27] <_joe_> reason: I won't convert those instances according to my current refactor of role::puppetmaster::puppetdb [16:26:45] FYI, I'll be rebooting tin to a new kernel tomorrow morning (Euro time), if you have any screen/tmux/SSH sessions that need to persist, please ping me [16:36:22] \o/ [16:54:52] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Scap: scap package installed by CI breaks apt and thus puppet - https://phabricator.wikimedia.org/T184118#3890315 (10thcipriani) [16:56:35] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Scap: scap package installed by CI breaks apt and thus puppet - https://phabricator.wikimedia.org/T184118#3872850 (10thcipriani) 05Open>03Resolved a:03thcipriani FWIW, I pushed https://phabricator.wikimedia.org/rMSCA58f5eca67156879a0... [17:12:00] 10Release-Engineering-Team (Kanban), 10Scap: Rethink beta scap deployment - https://phabricator.wikimedia.org/T184628#3890457 (10thcipriani) [17:18:33] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Zuul: Update zuul to upstream master - https://phabricator.wikimedia.org/T158243#3030929 (10greg) @hashar emailed the Ops list for assistance this EU morning. [17:28:48] PROBLEM - Puppet errors on deployment-snapshot01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:30:32] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Patch-For-Review: When jenkins kills a build due to max execution time the docker containers stay running - https://phabricator.wikimedia.org/T176747#3890597 (10hashar) p:05High>03Low **status update** There is al... [17:34:29] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Jenkins: Move the beta cluster jobs to a dedicated/standalone Jenkins instance - https://phabricator.wikimedia.org/T183164#3890604 (10hashar) p:05Normal>03Low E_TOO_MANY_TASKS for January. We can look at it after the offsite. [17:40:34] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Someday), 10Wikimedia-General-or-Unknown: Allow to test a mediawiki-config change to the beta cluster - https://phabricator.wikimedia.org/T136828#2349165 (10ArielGlenn) Bringing this back, I'd like to see the possibility of cherry-pick on one target... [17:54:03] greg-g: can I get a deployment window tomorrow 13:00 UTC? For one hour, because we need the patch stay online in the mwdebug1002 for around one hour [18:04:20] Amir1: what's up? [18:05:03] the timeslot is good, right before swat, so preemptive yes, just curious what's going on :) [18:33:27] what is that shinken warning, do I care? for the record, snapshot01's only problem is the same old problem, php-wikidiff2 for stretch [18:36:24] shinken-wm: would need to look at the puppet log to see if you care :) but we hope that people who maintain the things in prod look after them in beta as well :) [18:36:31] gah, apergos ^ [18:36:44] I know what the puppet log says, that's what I was reporting [18:37:02] i's only that issue and it's because there is no mw on php + stretch anywhere but there yet :-) [18:37:36] but I was actually asking: does shinken report on any puppet errors of any kind? since I'm not usually in here [18:37:36] oh [18:37:47] yeah [18:37:50] ah ok [18:38:04] so as long as I know what's going on, no need for additional panic... thanks! [18:38:18] word [18:38:35] who can I whine to about phab? (it's in the scrollback) [18:39:38] 10Continuous-Integration-Config, 10Proton, 10Readers-Web-Backlog, 10Patch-For-Review: Set up Jenkins for chromium-render and chromium-render-deploy repositories - https://phabricator.wikimedia.org/T179552#3890806 (10Jdlrobson) [18:40:01] apergos: mukunda, that's a weird issue that I haven't experienced (not a heavy dashboard user) [18:40:09] 10Continuous-Integration-Config, 10Proton, 10Readers-Web-Backlog, 10Patch-For-Review: Set up Jenkins for chromium-render and chromium-render-deploy repositories - https://phabricator.wikimedia.org/T179552#3728431 (10Jdlrobson) [18:40:18] ok [18:40:22] or in -devtools (I know I know, E_TOOMANYCHANNELS) [18:41:04] yeah if you click the 'fl' or whatever it is onthe right, it just gives some results, never far enough back to get my W5xx panels, I can type them in and it doesn't find them, either with a word in the name or the Wxxx number [18:41:31] I'm already moonlighting in this channel and over my limit... [18:41:39] apergos: yeah that's annoying. I've tried to fix it a couple of times and haven't gotten to far [18:41:48] ah so it's on your radar [18:41:59] yeah I think I reported it upstream as well [18:42:08] the whole dashboard feature is somewhat broken [18:42:12] it would have been ok if I didn't try to rearrange the order of panels in a tab panel, and it just dropped a bunch, because [18:42:22] they weren't separately on the dashboard (another bug i saw was reported) [18:42:25] 10Continuous-Integration-Config, 10Proton, 10Readers-Web-Backlog, 10Patch-For-Review: Set up Jenkins for chromium-render and chromium-render-deploy repositories - https://phabricator.wikimedia.org/T179552#3728431 (10Jdlrobson) [18:42:26] then I couldn't add them back [18:42:30] lame [18:42:34] so ... I made a couple new ones :-/ [18:42:35] sorry [18:42:58] 10Continuous-Integration-Config, 10Proton, 10Readers-Web-Backlog, 10Patch-For-Review, 10Readers-Web-Kanban-Board: Set up Jenkins for chromium-render and chromium-render-deploy repositories - https://phabricator.wikimedia.org/T179552#3890825 (10ovasileva) [18:43:11] it's ok that you made new ones. It's annoying that the search is broken and I will look into it again when I have a few minutes. It's not entirely straightforward or I would have had it fixed already [18:43:19] if there's a bug I can watch for the panel results being limited... [18:43:26] yeah clearly [18:43:53] either upstream or local, whichever [18:44:43] apergos: here's one apparently related to the same bug: https://phabricator.wikimedia.org/T171889 [18:44:48] I think there may be another [18:47:29] commented on that, thanks [18:48:17] it should at thelast let me type the thing in the field, and refuse to execute if it doesn't find the panel by direct lookup [18:48:25] instead of refusing to take the input at all [18:48:30] *at the least [18:48:48] hmm strange, I guess I _did_ fix this, but it's still not really working well [18:48:49] https://phabricator.wikimedia.org/T166236 [18:51:04] It only works for the results it finds, which right now is W13xx and later [18:51:15] the rest are lost in the mists of ancient history or broken code [18:51:46] it must order by desc and limit, and then not process the offset stuff right somehow [18:52:16] yeah, I'll look at it a bit more [18:52:23] because it's super annoying [18:52:27] obviously it's not urgent, there is a workaround [18:52:35] it just means needless proliferation of panels :-D [18:52:44] yeah [18:53:04] greg-g: sorry, was afk coming back to the home [18:54:02] It's about changing cache in constraint reports of wikibase [18:54:20] and the time it takes to invalidate the cache [18:54:29] PROBLEM - Puppet errors on deployment-secureredirexperiment is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:56:14] Amir1: go for it (add it) :) [18:56:25] Thank you [18:57:42] in other news, the /usr/local/bin/git-sync-upstream script is broken for /var/lib/git/labs/private on deployment-puppetmaster0 [18:57:43] 2 [18:58:12] it's been lke that since sometime in November at least, that's the date of the last thing in the log (a merge btw) before I forced a merge a few days ago [18:58:17] the specific error is: [18:58:25] 2018-01-10T09:40:02Z ERROR sync-upstream: Local diffs detected. Commit your changes! [18:58:39] weird [18:58:58] so I guess we need to soft rollback the repo til before any merges [18:59:13] get any local changes and put em on top [18:59:20] commit whatever is locally sitting around [18:59:22] ? [18:59:29] and then see if it runs [18:59:50] I'm sorry about the merge I did, this was before having a clue about how it's maintained [18:59:59] yeah [19:00:30] heh, merges might make more sense than perpetual cherry-picks [19:01:14] 1d0366392589c5e5084aabe6288b146ff90981a1 this one I think was what was there before I touched it [19:01:28] well if we have cherry picks that just get rebased on top [19:01:42] then the sync script will work [19:02:14] this is how I got bit, from a change in the actual repo that never showed up [19:03:13] git status shows it's these [19:03:15] modified: modules/secret/secrets/keyholder/deploy_librenms [19:03:15] modified: modules/secret/secrets/keyholder/deploy_librenms.pub [19:03:39] that are the problem, any idea who to poke? [19:28:11] (03PS1) 10Hashar: docker: specific image for mediawiki/vagrant [integration/config] - 10https://gerrit.wikimedia.org/r/403462 [19:35:35] (03CR) 10Hashar: [C: 032] "tested locally and works all fine. the job is just in experimental for now" [integration/config] - 10https://gerrit.wikimedia.org/r/403462 (owner: 10Hashar) [19:36:51] (03Merged) 10jenkins-bot: docker: specific image for mediawiki/vagrant [integration/config] - 10https://gerrit.wikimedia.org/r/403462 (owner: 10Hashar) [19:38:47] !log updating docker-pkg images for rake-vagrant https://gerrit.wikimedia.org/r/#/c/403462/ [19:38:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:41:51] apergos: I don't know about those files [19:42:00] git stash? [19:43:11] or just commit them if that is in the beta cluster private puppet repo [19:43:17] most of the things there are local commits [19:48:31] (03PS1) 10Hashar: Bump mediawiki/vagrant build timeout [integration/config] - 10https://gerrit.wikimedia.org/r/403467 [19:50:11] I'm working on cleaning it all up [19:50:22] I"m in a different directory and trying to cherry pick the local stuff on top [19:50:29] then I'll ad those files last in a new commit [19:50:54] and once I verify that the checked out contents are identical to the stuff in the live dir, I'll move the new repo into place [19:50:55] ugh [19:51:01] yeah it's the beta cluster one [19:53:10] (03CR) 10Hashar: [C: 032] Bump mediawiki/vagrant build timeout [integration/config] - 10https://gerrit.wikimedia.org/r/403467 (owner: 10Hashar) [19:55:54] (03Merged) 10jenkins-bot: Bump mediawiki/vagrant build timeout [integration/config] - 10https://gerrit.wikimedia.org/r/403467 (owner: 10Hashar) [20:20:18] (03PS1) 10Hashar: Cache bundler installed material between runs [integration/config] - 10https://gerrit.wikimedia.org/r/403471 [20:38:02] heads up, I have what sems to be a clean copy of private and the contents all match except of course .git [20:38:26] so I'm gonna move the old one away and the clean one into place on beta on deployment-puppetmaster02 [20:39:09] this is now done [20:39:28] * apergos bites nails [20:40:56] 2018-01-10T20:40:07Z INFO sync-upstream: Up-to-date: /var/lib/git/labs/private [20:41:01] successful sync! [20:41:18] the old copy is in /var/lib/git/labs/private_icky [20:41:27] in case I screwed something up [20:41:37] https://google.qualtrics.com/jfe/form/SV_0JKI9MBgY4eY4x7 [20:43:51] !log replaced /var/lib/git/labs/private on deployment-puppetmaster0 with a clean copy of the repo with all uncommitted changes checked in, all extra files checked in, all local commits rebased on top; old repo is in /var/lib/git/labs/private_icky [20:43:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:44:58] (03CR) 10Hashar: [C: 032] Cache bundler installed material between runs [integration/config] - 10https://gerrit.wikimedia.org/r/403471 (owner: 10Hashar) [20:45:24] !log that should be of course deployment-puppetmaster02 [20:45:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:45:33] PROBLEM - Puppet errors on deployment-imagescaler02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [20:51:20] (03Merged) 10jenkins-bot: Cache bundler installed material between runs [integration/config] - 10https://gerrit.wikimedia.org/r/403471 (owner: 10Hashar) [20:51:22] the imagescaler whine was because it couldn't find a secret key there during the split second of the move, just ran puppet again and it's fine [20:54:29] !log Updating rake / rake-vagrant docker containers | https://gerrit.wikimedia.org/r/#/c/403471/ [20:54:34] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:55:01] !log Updating '*rake-docker' jobs https://gerrit.wikimedia.org/r/#/c/403471/ [20:55:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:55:34] 10MediaWiki-Codesniffer: Remove FunctionComment.MissingParamComment from default rules - https://phabricator.wikimedia.org/T184650#3891274 (10Jdlrobson) [21:00:34] RECOVERY - Puppet errors on deployment-imagescaler02 is OK: OK: Less than 1.00% above the threshold [0.0] [21:02:18] 10MediaWiki-Codesniffer: Remove FunctionComment.MissingParamComment from default rules - https://phabricator.wikimedia.org/T184650#3891274 (10Reedy) https://github.com/wikimedia/mediawiki-tools-codesniffer/blob/efdf4bb1f362ca0ac6413c53c40011b84bca4fef/MediaWiki/Sniffs/Commenting/FunctionCommentSniff.php#L524 Re... [21:05:25] (03CR) 10Hashar: "Tested with https://integration.wikimedia.org/ci/job/mediawiki-vagrant-rake-docker/ and the cache works :)" [integration/config] - 10https://gerrit.wikimedia.org/r/403471 (owner: 10Hashar) [21:08:22] PROBLEM - Puppet errors on deployment-mx02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:09:59] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T180749#3891342 (10mmodell) [21:13:48] (03PS1) 10Reedy: Downgrade "Missing parameter comment" to warning [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/403483 (https://phabricator.wikimedia.org/T184650) [21:19:59] (03CR) 10jerkins-bot: [V: 04-1] Downgrade "Missing parameter comment" to warning [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/403483 (https://phabricator.wikimedia.org/T184650) (owner: 10Reedy) [21:23:56] (03PS2) 10Reedy: Downgrade "Missing parameter comment" to warning [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/403483 (https://phabricator.wikimedia.org/T184650) [21:26:23] (03CR) 10Reedy: [C: 04-1] "Well, that's not gonna work is it, reedy" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/403483 (https://phabricator.wikimedia.org/T184650) (owner: 10Reedy) [21:26:30] (03PS3) 10Reedy: Downgrade "Missing parameter comment" to warning [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/403483 (https://phabricator.wikimedia.org/T184650) [21:28:25] (03PS4) 10Reedy: Downgrade "Missing parameter comment" to warning [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/403483 (https://phabricator.wikimedia.org/T184650) [22:01:45] (03PS2) 10Hashar: Switch database backend to sqlite [integration/quibble] - 10https://gerrit.wikimedia.org/r/383161 [22:01:47] (03PS3) 10Hashar: docker: run as user nobody [integration/quibble] - 10https://gerrit.wikimedia.org/r/383160 [22:01:49] (03PS1) 10Hashar: Update ci-jessie docker image [integration/quibble] - 10https://gerrit.wikimedia.org/r/403539 [22:01:51] (03PS1) 10Hashar: The quibble script is the ENTRYPOINT [integration/quibble] - 10https://gerrit.wikimedia.org/r/403540 [22:03:22] (03CR) 10Hashar: [C: 032] Switch database backend to sqlite [integration/quibble] - 10https://gerrit.wikimedia.org/r/383161 (owner: 10Hashar) [22:03:59] (03Merged) 10jenkins-bot: Switch database backend to sqlite [integration/quibble] - 10https://gerrit.wikimedia.org/r/383161 (owner: 10Hashar) [22:04:05] (03CR) 10Hashar: [C: 032] docker: run as user nobody [integration/quibble] - 10https://gerrit.wikimedia.org/r/383160 (owner: 10Hashar) [22:04:27] (03Merged) 10jenkins-bot: docker: run as user nobody [integration/quibble] - 10https://gerrit.wikimedia.org/r/383160 (owner: 10Hashar) [22:04:31] (03CR) 10Hashar: [C: 032] Update ci-jessie docker image [integration/quibble] - 10https://gerrit.wikimedia.org/r/403539 (owner: 10Hashar) [22:04:52] (03Merged) 10jenkins-bot: Update ci-jessie docker image [integration/quibble] - 10https://gerrit.wikimedia.org/r/403539 (owner: 10Hashar) [22:06:22] (03PS2) 10Hashar: The quibble script is the ENTRYPOINT [integration/quibble] - 10https://gerrit.wikimedia.org/r/403540 [22:08:34] 10Continuous-Integration-Infrastructure, 10JavaScript: Generate JS code coverage reports for extensions - https://phabricator.wikimedia.org/T184657#3891521 (10Jdforrester-WMF) p:05Triage>03Normal [22:08:54] 10Continuous-Integration-Infrastructure, 10JavaScript: Generate JS code coverage reports for extensions - https://phabricator.wikimedia.org/T184657#3891521 (10Jdforrester-WMF) [22:08:57] 10Continuous-Integration-Config: Provide a CI job to generate JS code coverage reports for extensions by using karma-coverage instead of just karma - https://phabricator.wikimedia.org/T116808#3891534 (10Jdforrester-WMF) [22:08:59] 10Continuous-Integration-Infrastructure, 10MW-1.31-release-notes (WMF-deploy-2018-01-09 (1.31.0-wmf.16)), 10MediaWiki-Platform-Team (MWPT-Q3-Jan-Mar-2018), 10Patch-For-Review: Generate PHP code coverage reports for extensions - https://phabricator.wikimedia.org/T71685#738524 (10Jdforrester-WMF) [22:47:30] PROBLEM - SSH on integration-slave-docker-1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:52:22] RECOVERY - SSH on integration-slave-docker-1002 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u4 (protocol 2.0) [22:53:25] (03PS1) 10Awight: Provide Java 8 for mwdumper testing [integration/config] - 10https://gerrit.wikimedia.org/r/403552 [22:57:06] (03CR) 10Paladox: [C: 031] "Hmm i guess we missed this one, as ci requires java 8 due to jenkins java 8 bump when we did a upgrade." [integration/config] - 10https://gerrit.wikimedia.org/r/403552 (owner: 10Awight) [22:57:46] (03CR) 10Paladox: [C: 031] Provide Java 8 for mwdumper testing (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/403552 (owner: 10Awight) [23:00:04] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (Little Steps Sprint), 10Release-Engineering-Team (Someday): Get rid of Zend 5.5 tests for wmf branches - https://phabricator.wikimedia.org/T94149#3891641 (10Krinkle) [23:04:45] paladox: Thanks for noticing that—Looking through the “node” constraints, it looks like CI jobs are all still running on Jessie? [23:04:57] yep. [23:05:03] java 8 is on jessie too [23:05:20] only in jessie-backports, is that source installed? [23:05:21] (has to be too otherwise we wouldent be able to use ci as jenkins requires java8+ now. [23:05:26] aha [23:05:36] okay that’s convincing, thanks! [23:05:43] your welcome :). [23:06:12] (03PS2) 10Awight: Provide Java 8 for mwdumper testing [integration/config] - 10https://gerrit.wikimedia.org/r/403552 [23:07:54] (03CR) 10Paladox: [C: 031] "Thanks." [integration/config] - 10https://gerrit.wikimedia.org/r/403552 (owner: 10Awight) [23:09:41] (03CR) 10Brion VIBBER: [C: 031] "I have no way to test the config but I wants it, so here's a +1" [integration/config] - 10https://gerrit.wikimedia.org/r/403552 (owner: 10Awight) [23:10:03] 10Beta-Cluster-Infrastructure, 10Patch-For-Review, 10Puppet: Puppet broken on deployment-mx due to systemd on trusty - https://phabricator.wikimedia.org/T184244#3891691 (10Krenair) Created a new system, ran into the problem that https://gerrit.wikimedia.org/r/#/c/403326/ fixes [23:53:08] (03PS1) 10Chad: release: Remove upload tar, dubiously useful [tools/release] - 10https://gerrit.wikimedia.org/r/403561 [23:53:10] (03PS1) 10Chad: release: patchfile refactoring [tools/release] - 10https://gerrit.wikimedia.org/r/403562 [23:53:12] (03PS1) 10Chad: releases: make patch_dir configurable (and only configurable) [tools/release] - 10https://gerrit.wikimedia.org/r/403563 [23:53:14] (03PS1) 10Chad: releases: Unify root_dir/build_dir [tools/release] - 10https://gerrit.wikimedia.org/r/403564 [23:53:17] (03PS1) 10Chad: releases: use os.path.join() in a few places [tools/release] - 10https://gerrit.wikimedia.org/r/403565 [23:53:19] (03PS1) 10Chad: release: move get_git() to function, fix up export_dir [tools/release] - 10https://gerrit.wikimedia.org/r/403566 [23:53:43] (03CR) 10jerkins-bot: [V: 04-1] release: Remove upload tar, dubiously useful [tools/release] - 10https://gerrit.wikimedia.org/r/403561 (owner: 10Chad) [23:54:00] Here comes the cascade.... [23:59:02] (03CR) 10Chad: "recheck" [tools/release] - 10https://gerrit.wikimedia.org/r/403561 (owner: 10Chad)