[00:04:23] 10Phabricator: Notify Community-tech@lists.wikimedia.org when a task is UBN and tagged with #commtech - https://phabricator.wikimedia.org/T229294 (10MBinder_WMF) [01:06:35] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Security-Team: Ensure submodule updates (for security patches) are committed in the MW directory under /srv/mediawiki-staging - https://phabricator.wikimedia.org/T229285 (10thcipriani) 05Open→03Declined > It seems that o... [01:06:41] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201908), 10Release, 10Train Deployments: 1.34.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T220741 (10thcipriani) [02:20:37] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<55.56%) [02:26:25] (03PS4) 10Legoktm: Initial commit [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/525700 [02:30:06] (03CR) 10Legoktm: [C: 03+2] "PS4: Filled in more fields for setup.py." [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/525700 (owner: 10Legoktm) [02:33:25] (03Merged) 10jenkins-bot: Initial commit [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/525700 (owner: 10Legoktm) [02:35:49] (03PS1) 10Legoktm: Fix typo in classifier [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/526295 [02:39:58] (03CR) 10Legoktm: [C: 03+2] Fix typo in classifier [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/526295 (owner: 10Legoktm) [02:42:53] (03Merged) 10jenkins-bot: Fix typo in classifier [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/526295 (owner: 10Legoktm) [04:16:04] 10Gerrit, 10Gerrit-Privilege-Requests, 10Wikimedia-IEG-grant-review, 10User-MarcoAurelio: Give access to l10n-bot to wikimedia/iegreview repository - https://phabricator.wikimedia.org/T228490 (10abi_) 05Resolved→03Open @MarcoAurelio - Reopening this as this is not fixed Here's a patch in another repo... [06:24:38] (03CR) 10Giuseppe Lavagetto: [C: 03+1] "a couple nitpick you can safely ignore for now, but otherwise LGTM." (033 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/525117 (https://phabricator.wikimedia.org/T224857) (owner: 10Thcipriani) [06:51:11] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10ValueView, 10Wikidata, and 2 others: Fix the data-values/value-view repo to work on node10 - https://phabricator.wikimedia.org/T229276 (10Lydia_Pintscher) [07:10:36] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [08:07:28] (03PS4) 10Kosta Harlan: [quibble-coverage] Fix code coverage generation for core codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/521190 [08:17:02] (03PS2) 10Kosta Harlan: [jjb] Use quibble-coverage:0.0.34-1 for mwcore codehealth job [integration/config] - 10https://gerrit.wikimedia.org/r/521192 [10:55:21] 10Phabricator: Edit Subtasks/Parent Tasks window closes when selecting task by clicking on its title - https://phabricator.wikimedia.org/T229321 (10Alicia_Fagerving_WMSE) [13:53:25] branch cut hasn't happened yet, right? [13:53:32] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Security-Team: Ensure submodule updates (for security patches) are committed in the MW directory under /srv/mediawiki-staging - https://phabricator.wikimedia.org/T229285 (10Krinkle) I'm familiar with (under default Git setti... [13:53:57] (just making sure nothing will need backporting for T229244) [13:53:57] T229244: Edit link is missing from structured caption panel on Beta - https://phabricator.wikimedia.org/T229244 [14:03:49] James_F: want to give https://gerrit.wikimedia.org/r/#/c/521190/ a go? [14:05:39] if whomever's in charge wouldn't mind postponing the branch cut until https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/WikibaseMediaInfo/+/526399/ lands, that's be super [14:06:26] (we're gonna try a `recheck` before force-merging over spurious test failures since it's UBN) [14:12:28] kostajh: btw, this might be of interest if you haven't seen it already. https://doc.wikimedia.org/mediawiki-core/master/phpmetrics/ [14:12:43] I don't look at it very often, but it came about a few months ago [14:13:03] Nice. I remember seeing it but hadn't looked at it in a while. [14:15:13] This one is somewhat "fun" - https://doc.wikimedia.org/mediawiki-core/master/phpmetrics/relations.html [14:22:40] yay [14:22:44] https://doc.wikimedia.org/mediawiki-core/master/phpmetrics/coupling.html too :) [14:36:23] mdholloway: wmf.16 hasn't been cut yet, no. Ping brennen who's doing the train today (for their first time) and thcipriani . [14:36:43] Krinkle: Good grief. [14:37:33] (03CR) 10Jforrester: [C: 03+2] [quibble-coverage] Fix code coverage generation for core codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/521190 (owner: 10Kosta Harlan) [14:37:39] will do, thanks. [14:38:11] (well, it's been done, so ACK :) ) [14:43:42] (03Merged) 10jenkins-bot: [quibble-coverage] Fix code coverage generation for core codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/521190 (owner: 10Kosta Harlan) [14:44:11] mdholloway: So should https://phabricator.wikimedia.org/T229279 be marked as a train blocker? [14:44:36] As well as T229244 [14:44:37] T229244: Edit link is missing from structured caption panel on Beta - https://phabricator.wikimedia.org/T229244 [14:44:55] !log Docker: Pushing quibble-coverage:0.0.34-1 [14:44:55] mdholloway: you've got a bit, but ^ might not be a bad idea so we don't miss that and have to backport [14:44:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:45:24] I think we're considering https://phabricator.wikimedia.org/T229244 to encompass both sub-cases of edit link breakage, and T229279 to be for longer-term discussion about how the hook should behave [14:45:25] T229279: wikibase.entityPage.entityLoaded does not fire for entities with no data yet entered - https://phabricator.wikimedia.org/T229279 [14:46:40] Ah, good point about the admin angle, will mark UBN and block train for now [14:46:57] mdholloway: Because the patch you're waiting to merge is tagged against T229279 but not T229244 hence my confusion. :-) [14:47:00] Yeah. [14:48:45] two days in a row I haven't had any backports for you, how 'bout them apples :-P [14:49:14] apergos: It won't last. ;-) [14:49:32] way to bring me down, d00d [14:51:42] (03PS3) 10Jforrester: jjb: [mwcore-codehealth*] Use quibble-coverage:0.0.34-1 [integration/config] - 10https://gerrit.wikimedia.org/r/521192 (owner: 10Kosta Harlan) [14:51:53] kostajh: It's live. [14:52:11] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/521192 (owner: 10Kosta Harlan) [14:52:25] James_F: thanks! I'll keep an eye out to see if it works [14:52:30] mdholloway: It's finally landed. [14:54:38] (03Merged) 10jenkins-bot: jjb: [mwcore-codehealth*] Use quibble-coverage:0.0.34-1 [integration/config] - 10https://gerrit.wikimedia.org/r/521192 (owner: 10Kosta Harlan) [14:54:58] kostajh: First run is https://integration.wikimedia.org/ci/job/mwcore-codehealth-patch/1874/console [14:55:51] Looks like the beta-scap-eqiad job has begun failing [14:56:01] * James_F looks. [14:56:30] `Call to mwscript eval.php returned: None`. Fun. [14:56:32] James_F: hmm, `The "--coverage-clover" option does not exist.`. I'll look at it [14:56:41] when did it start failing? [14:56:52] cdanis: 30-Jul-2019 14:34 [14:56:59] (UTC) [14:57:02] it is possible I am responsible [14:57:02] hm that's a bit ago [14:57:12] It passed at 30-Jul-2019 14:24 UTC. [14:57:14] https://integration.wikimedia.org/ci/view/Beta/job/beta-scap-eqiad/ [14:57:36] (Why isn't the bot in here complaining that the job has broken?) [14:58:57] James_F: argh. Double dash. Sorry :( I'll submit new patches... [14:59:09] kostajh: Ha. [14:59:18] composer scripts... [15:00:21] cdanis: You're thinking the dbctl changes are related? [15:00:33] James_F: for a minute I was. but actually it looks like etcd.php gets sourced on beta cluster, so the mechanism for the breakage I just imagined shouldn't be possible [15:01:06] Right. [15:01:37] Did the puppet side stuff for dbctl make it into Beta? [15:02:03] there's no puppet-side stuff needed on appservers [15:02:31] there's no new code in Mediawiki proper, even; on the appserver side it's just a config change (with a modest amount of code there) [15:02:39] Right. [15:03:08] the timeline does match up, though :\ [15:03:41] (03PS1) 10Kosta Harlan: [quibble-coverage] Fix coverage generation script from 0.0.34-1 [integration/config] - 10https://gerrit.wikimedia.org/r/526445 [15:03:58] It'd be nice to know what it's trying to eval. [15:04:51] (03PS1) 10Kosta Harlan: jjb: [mwcore-codehealth*] Use quibble-coverage:0.0.34-2 [integration/config] - 10https://gerrit.wikimedia.org/r/526446 [15:07:06] https://phabricator.wikimedia.org/P8827 [15:07:17] this looks fine to me, and I'm pretty sure I'm on one of the hosts in question? [15:13:19] ahhh, this is an mwscript on the beta deploy host [15:13:58] YEP! my fault. [15:14:16] Fatal error: Uncaught exception 'ConfigException' with message 'No entry found for 'eqiad/dbconfig'.' in /srv/mediawiki-staging/php-master/includes/config/EtcdConfig.php:116 [15:16:05] _joe_: is there an equivalent conftool-data tree for beta cluster? how was any etcd data that is present there created? [15:16:33] <_joe_> no there is none, and IIRC we don't use etcd data in mediawiki there [15:16:50] <_joe_> we definitely shouldn't [15:17:07] <_joe_> and I see why your code is a problem :) [15:17:24] I didn't change anything about etcd.php getting loaded in labs, though [15:18:24] and in labs there is some data in etcd. [15:18:49] cdanis@deployment-etcd-01:~$ etcdctl -C http://127.0.0.1:2378 get /conftool/v1/mediawiki-config/eqiad/ReadOnly [15:18:52] {"val": false} [15:18:57] I suspect created by hand [15:19:13] <_joe_> I suspect so as well [15:19:27] <_joe_> because something got changed at some point, meh [15:19:37] <_joe_> sigh [15:20:27] <_joe_> so this is an interesting piece of tech debt no one thought about :P [15:20:57] I can skip it if the realm == labs [15:21:00] that would work around this for now [15:21:03] but .... [15:21:17] <_joe_> I don't think it's an issue, per se [15:21:23] <_joe_> if we skip it [15:21:35] <_joe_> we just need to also set the readonly variable and the dc one I guess [15:21:53] <_joe_> do you need some help with doing that? [15:24:11] I am not sure of the exact best approach here [15:24:59] you are thinking, in etcd.php, we look at wmfRealm, and in the case of $wmfRealm === 'labs', we set some variables by hand instead of reading from etcd? [15:26:38] unrelatedly, for whoever set this stuff by hand earlier, I'm honestly not yet sure *how* they did. deployment-deploy01 has /root/.etcdrc, but no etcdctl. deployment-etcd-01 has etcdctl, but no /root/.etcdrc [15:26:49] <_joe_> cdanis: exactly [15:27:16] <_joe_> cdanis: I think I probably did it with curl :) [15:30:36] <_joe_> cdanis: ok I see, we used the beta cluster to test etcdconfig [15:30:42] yeah it looks like it [15:30:45] https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/351767 [15:30:48] https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/375108 [15:30:58] <_joe_> ok so [15:31:09] I'm fine just manually setting a null dbconfig object in beta cluster, fwiw [15:31:10] <_joe_> we shall add the dbctl stuff there too? [15:31:40] there's no conftool in beta cluster, and that seems like a fair bit of work, so I don't think we should be using dbctl to manage the configuration there [15:31:49] but a dummy dbconfig key in etcd seems fine [15:31:50] <_joe_> yeah an empty array would do it probably, if you protect overriding wgLBFactoryConf [15:31:57] yeah [15:32:05] we can do that for the full rollout [15:33:34] <_joe_> sure [15:33:39] <_joe_> add a stupid key for now [15:33:46] <_joe_> do you know how? [15:33:54] <_joe_> it must be valid json though :) [15:34:01] done [15:34:02] <_joe_> so a value of "[]" [15:34:37] <_joe_> it works again [15:34:46] https://phabricator.wikimedia.org/P8828 [15:34:49] yeah [15:34:51] seems fine [15:36:14] okay, before I do the full deploy I will condition some stuff on realm != labs [15:37:11] dumb question, how does one append to the beta SAL? [15:38:13] also, does beta only exist in eqiad? [15:38:43] cdanis: Just do !log in here. [15:38:45] a !log in this channel ends up on https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:38:57] And yes, only eqiad for Beta Cluster. [15:39:03] alternatively, !log deployment-prep in #wikimedia-cloud ends up on https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [15:39:19] !log manually set dbctl etcd key to dummy value: cdanis@deployment-etcd-01:~$ etcdctl -u root -C http://127.0.0.1:2378 set /conftool/v1/mediawiki-config/eqiad/dbconfig [15:39:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:48:23] mdholloway: OK, Beta Cluster is now updating. Is T229279 confirmed as fixed so the train can roll? [15:48:23] T229279: wikibase.entityPage.entityLoaded does not fire for entities with no data yet entered - https://phabricator.wikimedia.org/T229279 [15:48:43] (And T229244 for that matter.) [15:48:43] T229244: Edit link is missing from structured caption panel on Beta - https://phabricator.wikimedia.org/T229244 [15:50:16] i just switched to this buffer to ask the same question. [15:50:36] I'll do a quick check after the code lands and then confirm [15:50:50] mdholloway: thanks [15:53:36] (03CR) 10Jforrester: [C: 03+2] [quibble-coverage] Fix coverage generation script from 0.0.34-1 [integration/config] - 10https://gerrit.wikimedia.org/r/526445 (owner: 10Kosta Harlan) [15:55:06] (03Merged) 10jenkins-bot: [quibble-coverage] Fix coverage generation script from 0.0.34-1 [integration/config] - 10https://gerrit.wikimedia.org/r/526445 (owner: 10Kosta Harlan) [15:57:49] James_F: brennen: LGTM. I'll update tickets. [15:58:18] cool [15:58:59] i'm planning to start cutting the branch once that's clear. [15:59:11] ^ cc: thcipriani [15:59:21] ack [16:05:33] (03PS1) 10Giuseppe Lavagetto: operations-puppet: allow to use to run tests locally [integration/config] - 10https://gerrit.wikimedia.org/r/526467 [16:11:13] awright, no blockers are currently listed, moving forward. [16:17:10] !log Docker: Pushing quibble-coverage:0.0.34-2 [16:17:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:18:28] (03CR) 10Jforrester: [C: 03+2] jjb: [mwcore-codehealth*] Use quibble-coverage:0.0.34-2 [integration/config] - 10https://gerrit.wikimedia.org/r/526446 (owner: 10Kosta Harlan) [16:21:47] (03Merged) 10jenkins-bot: jjb: [mwcore-codehealth*] Use quibble-coverage:0.0.34-2 [integration/config] - 10https://gerrit.wikimedia.org/r/526446 (owner: 10Kosta Harlan) [16:40:07] James_F: sigh. The magic invocation is: `composer phpunit:coverage -- {args}`, where composer.json's phpunit:coverage script has `php -d zend_extension=xdebug.so vendor/bin/phpunit --testsuite=core:unit --exclude-group Dump,Broken"` [16:46:33] kostajh: Ha. [16:52:59] (03Abandoned) 10Kosta Harlan: Stop PHPUnit execution on failure [integration/quibble] - 10https://gerrit.wikimedia.org/r/517188 (https://phabricator.wikimedia.org/T225871) (owner: 10Kosta Harlan) [16:59:43] PROBLEM - SSH access on cobalt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Gerrit [17:00:06] huh [17:01:09] RECOVERY - SSH access on cobalt is OK: SSH OK - GerritCodeReview_2.15.14-16-g855b179b5f (SSHD-CORE-1.6.0) (protocol 2.0) https://wikitech.wikimedia.org/wiki/Gerrit [17:47:43] (03CR) 10Jeena Huneidi: "> Patch Set 2: Code-Review-1" (031 comment) [releng/local-charts] - 10https://gerrit.wikimedia.org/r/525888 (https://phabricator.wikimedia.org/T228913) (owner: 10Jeena Huneidi) [18:06:01] (03CR) 1020after4: [C: 03+1] Use mediawiki-dev chart from releases.wikimedia.org/charts/ (031 comment) [releng/local-charts] - 10https://gerrit.wikimedia.org/r/525888 (https://phabricator.wikimedia.org/T228913) (owner: 10Jeena Huneidi) [19:49:47] PROBLEM - Free space - all mounts on deployment-mediawiki-07 is CRITICAL: CRITICAL: deployment-prep.deployment-mediawiki-07.diskspace.root.byte_percentfree (<40.00%) [19:57:53] (03PS4) 10markahershberger: Tarball creation [tools/release] - 10https://gerrit.wikimedia.org/r/521559 (https://phabricator.wikimedia.org/T217960) [19:58:36] (03CR) 10jerkins-bot: [V: 04-1] Tarball creation [tools/release] - 10https://gerrit.wikimedia.org/r/521559 (https://phabricator.wikimedia.org/T217960) (owner: 10markahershberger) [20:42:58] (03PS1) 10markahershberger: Disable hhvm/php5.x (composer-hhvm-docker) tests for release-tools [integration/config] - 10https://gerrit.wikimedia.org/r/526524 (https://phabricator.wikimedia.org/T229370) [20:44:06] (03PS1) 10Mholloway: Start branching mediawiki/extensions/MachineVision [tools/release] - 10https://gerrit.wikimedia.org/r/526526 (https://phabricator.wikimedia.org/T227348) [20:44:24] (03CR) 10jerkins-bot: [V: 04-1] Disable hhvm/php5.x (composer-hhvm-docker) tests for release-tools [integration/config] - 10https://gerrit.wikimedia.org/r/526524 (https://phabricator.wikimedia.org/T229370) (owner: 10markahershberger) [20:44:45] (03CR) 10markahershberger: "Addressed the other test failures and opened" (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/521559 (https://phabricator.wikimedia.org/T217960) (owner: 10markahershberger) [20:50:25] (03PS3) 10Jeena Huneidi: Use mediawiki-dev chart from releases.wikimedia.org/charts/ [releng/local-charts] - 10https://gerrit.wikimedia.org/r/525888 (https://phabricator.wikimedia.org/T228913) [21:07:04] (03CR) 10Jforrester: [C: 03+2] operations-puppet: allow to use to run tests locally [integration/config] - 10https://gerrit.wikimedia.org/r/526467 (owner: 10Giuseppe Lavagetto) [21:08:37] (03Merged) 10jenkins-bot: operations-puppet: allow to use to run tests locally [integration/config] - 10https://gerrit.wikimedia.org/r/526467 (owner: 10Giuseppe Lavagetto) [21:10:35] * paladox is surprised by this bug https://gerrit-review.googlesource.com/c/gerrit/+/232732 [21:17:51] !log Docker: Push operations-puppet:0.5.3 [21:17:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:36:34] (03PS1) 10Jforrester: jjb: [operations-puppet-tests-stretch-docker] Bump to latest image for consistency [integration/config] - 10https://gerrit.wikimedia.org/r/526537 [21:39:13] legoktm: Should I expect an amazing particular new thing from LibUp given your spate of patches, or are you just being awesome and making things work better? [21:40:51] the latter mostly. I have the log running in a background terminal, and am just fixing stuff I see scroll by [21:41:35] my (self imposed) deadline to get 2.0 up and running was the end of july, so tomorrow I plan on putting it into fully automated mode, and it'll start pushing patches again [21:41:43] Ha. :-) [21:42:49] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/526537 (owner: 10Jforrester) [21:45:03] (03Merged) 10jenkins-bot: jjb: [operations-puppet-tests-stretch-docker] Bump to latest image for consistency [integration/config] - 10https://gerrit.wikimedia.org/r/526537 (owner: 10Jforrester) [21:56:43] (03CR) 10Jforrester: "Has the extension had a base-level security review?" [tools/release] - 10https://gerrit.wikimedia.org/r/526526 (https://phabricator.wikimedia.org/T227348) (owner: 10Mholloway) [22:01:37] (03CR) 10Mholloway: "There's a security concept review in progress (T227591). My understanding (via Adam) is that the Security team would appreciate a working" [tools/release] - 10https://gerrit.wikimedia.org/r/526526 (https://phabricator.wikimedia.org/T227348) (owner: 10Mholloway) [22:04:21] (03CR) 10Jforrester: "> Patch Set 1:" [tools/release] - 10https://gerrit.wikimedia.org/r/526526 (https://phabricator.wikimedia.org/T227348) (owner: 10Mholloway) [22:09:26] (03CR) 10Mholloway: "Yes, born Wikimedia. We could probably get by strictly on Beta for a couple of weeks, but we're going to want to start branching for produ" [tools/release] - 10https://gerrit.wikimedia.org/r/526526 (https://phabricator.wikimedia.org/T227348) (owner: 10Mholloway) [22:09:45] (03CR) 10Jforrester: [C: 03+2] Start branching mediawiki/extensions/MachineVision [tools/release] - 10https://gerrit.wikimedia.org/r/526526 (https://phabricator.wikimedia.org/T227348) (owner: 10Mholloway) [22:10:22] (03Merged) 10jenkins-bot: Start branching mediawiki/extensions/MachineVision [tools/release] - 10https://gerrit.wikimedia.org/r/526526 (https://phabricator.wikimedia.org/T227348) (owner: 10Mholloway) [22:33:48] (03CR) 10Jforrester: "Do we ever run release-tools on HHVM machines in production? Certainly `php` points to HHVM on `deploy1001`. If so, presumably we can't me" [integration/config] - 10https://gerrit.wikimedia.org/r/526524 (https://phabricator.wikimedia.org/T229370) (owner: 10markahershberger)