[00:01:42] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-Ryasmeen: Recover from corrupted beta MySQL slave (deployment-db04) - https://phabricator.wikimedia.org/T216067 (10mmodell) >>! In T216067#4986616, @ArielGlenn wrote: > The page title looks wrong: 'Nove Resource:... [00:04:48] Yippee, build fixed! [00:04:49] Project beta-scap-eqiad build #239402: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/239402/ [00:09:52] 10Phabricator, 10Proton, 10Reading-Infrastructure-Team-Backlog: Update Herald (H228) to include project 2960 - https://phabricator.wikimedia.org/T217078 (10MBinder_WMF) [00:11:20] (03CR) 10Jeena Huneidi: [C: 03+1] Pipeline: use wmf base image [blubber] - 10https://gerrit.wikimedia.org/r/493090 (owner: 10Thcipriani) [00:46:16] (03PS1) 10BryanDavis: Add Srishti Sethi to the CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/493139 [00:49:07] (03PS2) 10BryanDavis: Add Srishti Sethi to the CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/493139 [06:26:19] (03CR) 10Smalyshev: [C: 03+1] Enable phan for WikibaseLexeme [integration/config] - 10https://gerrit.wikimedia.org/r/493061 (https://phabricator.wikimedia.org/T215556) (owner: 10Ladsgroup) [06:32:01] 10Continuous-Integration-Config, 10Wikidata, 10Wikidata-Query-Service, 10Discovery-Wikidata-Query-Service-Sprint, 10User-Smalyshev: Set up Blazegraph test suite on CI - https://phabricator.wikimedia.org/T216855 (10Smalyshev) [07:02:13] 10Continuous-Integration-Infrastructure: Add CI entry point to run MinusX on mediawiki/core - https://phabricator.wikimedia.org/T188022 (10hashar) 05Resolved→03Open https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/402166/ got reverted. [07:03:09] 10Continuous-Integration-Config, 10MinusX, 10MW-1.33-notes (1.33.0-wmf.12; 2019-01-08): minus-x broke MediaWiki core CI: Too many arguments, expected arguments "command" "path" - https://phabricator.wikimedia.org/T212766 (10hashar) 05Open→03Resolved a:03hashar That got fixed by reverting the addition o... [08:23:09] (03CR) 10Giuseppe Lavagetto: Fix OpcacheManager.invalidate_all() (032 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/492738 (owner: 10Giuseppe Lavagetto) [08:30:56] Project beta-scap-eqiad build #239447: 04FAILURE in 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/239447/ [08:44:38] Yippee, build fixed! [08:44:39] Project beta-scap-eqiad build #239448: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/239448/ [08:46:03] Project beta-scap-eqiad build #239449: 04FAILURE in 4 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/239449/ [09:04:37] Yippee, build fixed! [09:04:37] Project beta-scap-eqiad build #239450: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/239450/ [09:16:00] 10Beta-Cluster-Infrastructure, 10Analytics, 10User-Elukey: TCP connections between analytics and deployment-prep - https://phabricator.wikimedia.org/T208870 (10elukey) 05Open→03Resolved [09:22:04] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-Ryasmeen: Recover from corrupted beta MySQL slave (deployment-db04) - https://phabricator.wikimedia.org/T216067 (10hashar) Thank you so much! wikilove [09:24:27] (03CR) 10Hashar: "Pending support in the repository: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/WikibaseLexeme/+/493024/" [integration/config] - 10https://gerrit.wikimedia.org/r/493061 (https://phabricator.wikimedia.org/T215556) (owner: 10Ladsgroup) [09:25:10] (03CR) 10Hashar: [C: 03+2] Archive WikibaseQualityExternalValidation extension [integration/config] - 10https://gerrit.wikimedia.org/r/492457 (https://phabricator.wikimedia.org/T204490) (owner: 10Zoranzoki21) [09:26:33] (03Merged) 10jenkins-bot: Archive WikibaseQualityExternalValidation extension [integration/config] - 10https://gerrit.wikimedia.org/r/492457 (https://phabricator.wikimedia.org/T204490) (owner: 10Zoranzoki21) [09:36:19] (03PS1) 10DCausse: Add phan to Elastica [integration/config] - 10https://gerrit.wikimedia.org/r/493174 [09:46:18] 10Continuous-Integration-Infrastructure, 10Upstream: chromium 72 crash when used with --remote-debugging-port - https://phabricator.wikimedia.org/T216702 (10hashar) [09:46:58] 10Continuous-Integration-Infrastructure, 10Upstream: chromium 72 crash when used with --remote-debugging-port - https://phabricator.wikimedia.org/T216702 (10hashar) p:05Triage→03Normal [09:48:06] (03CR) 10DCausse: "test patch that should fail on phan: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Elastica/+/493165" [integration/config] - 10https://gerrit.wikimedia.org/r/493174 (owner: 10DCausse) [09:49:15] dcausse: on peut tenter ;) [09:49:31] (03CR) 10Hashar: [C: 03+2] Add phan to Elastica [integration/config] - 10https://gerrit.wikimedia.org/r/493174 (owner: 10DCausse) [09:51:02] (03Merged) 10jenkins-bot: Add phan to Elastica [integration/config] - 10https://gerrit.wikimedia.org/r/493174 (owner: 10DCausse) [09:52:45] dcausse: I did a recheck https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Elastica/+/493165/ [09:54:39] hashar: thanks! [09:55:13] dcausse: phan fails [09:55:17] phan sec check passes :) [09:55:21] good :) [10:25:52] (03PS1) 10Hashar: utils: zuul layout generator [integration/config] - 10https://gerrit.wikimedia.org/r/493180 [10:51:31] (03PS1) 10Hashar: zuul: remove "check" pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/493188 (https://phabricator.wikimedia.org/T192217) [10:52:59] (03CR) 10jerkins-bot: [V: 04-1] zuul: remove "check" pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/493188 (https://phabricator.wikimedia.org/T192217) (owner: 10Hashar) [10:53:50] (03CR) 10Hashar: "The check pipeline triggered 142 builds over the last 30 days, most users being whitelisted anyway." [integration/config] - 10https://gerrit.wikimedia.org/r/493188 (https://phabricator.wikimedia.org/T192217) (owner: 10Hashar) [11:31:41] 10Continuous-Integration-Config, 10Lexicographical data, 10Wikidata, 10Patch-For-Review, and 3 others: Enable phan checks for WikibaseLexeme extension - https://phabricator.wikimedia.org/T215556 (10Michael) [12:06:10] (03PS1) 10Michael Große: Enable phan for WikibaseSchema [integration/config] - 10https://gerrit.wikimedia.org/r/493213 (https://phabricator.wikimedia.org/T217180) [12:07:53] (03PS2) 10Hashar: zuul: remove "check" pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/493188 (https://phabricator.wikimedia.org/T192217) [13:33:46] (03CR) 10Michael Große: "This change is ready for review." [integration/config] - 10https://gerrit.wikimedia.org/r/493213 (https://phabricator.wikimedia.org/T217180) (owner: 10Michael Große) [13:35:53] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] Enable phan for WikibaseSchema [integration/config] - 10https://gerrit.wikimedia.org/r/493213 (https://phabricator.wikimedia.org/T217180) (owner: 10Michael Große) [13:38:47] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] Enable phan for WikibaseLexeme [integration/config] - 10https://gerrit.wikimedia.org/r/493061 (https://phabricator.wikimedia.org/T215556) (owner: 10Ladsgroup) [13:55:27] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T206673 (10Addshore) [14:19:12] 10Continuous-Integration-Config: npm 6 consistently fails with "Z_DATA_ERROR: invalid distance too far back" on some repos - https://phabricator.wikimedia.org/T215562 (10Krinkle) [14:36:29] 10Release-Engineering-Team, 10CirrusSearch, 10Discovery-Search (Current work), 10MW-1.32-release: Build failures for Elastica and CirrusSearch on REL1_32 - https://phabricator.wikimedia.org/T216612 (10Addshore) [15:04:03] 10Continuous-Integration-Infrastructure, 10Upstream: chromium 72 crash when used with --remote-debugging-port - https://phabricator.wikimedia.org/T216702 (10MoritzMuehlenhoff) Should be fixed now: https://lists.debian.org/debian-security-announce/2019/msg00039.html [16:02:47] (03PS1) 10Sbisson: Make GrowthExperiments depend on PageImages and PageViewInfo [integration/config] - 10https://gerrit.wikimedia.org/r/493261 (https://phabricator.wikimedia.org/T216217) [16:10:42] (03CR) 10Kosta Harlan: [C: 03+1] Make GrowthExperiments depend on PageImages and PageViewInfo [integration/config] - 10https://gerrit.wikimedia.org/r/493261 (https://phabricator.wikimedia.org/T216217) (owner: 10Sbisson) [16:11:10] https://groups.google.com/forum/#!topic/repo-discuss/YaGwCQZTDCU - gerrit hackathon [16:19:00] thcipriani what error did you get for LocalUsernamesToLowerCase ? [16:19:25] paladox: all sorts of injector errors [16:19:29] * thcipriani finds scrollback [16:19:33] ok :) [16:21:36] paladox: https://phabricator.wikimedia.org/P8139 (although my test instance is configured differently than prod in a few ways) [16:21:58] ah [16:22:14] thcipriani im pretty sure that error has nothing to do with the way it's configured :) [16:22:41] * paladox will ask david p when he's around :) [16:22:48] I agree, seems like it should work regardless of configuration [16:22:54] thanks paladox :) [16:23:25] thcipriani looks like we have reports about our gerrit install reporting wrong content header [16:23:26] https://github.com/dpursehouse/pygerrit2/issues/54 [16:24:11] (upstream brought that to my attention) he tested with our fork and coulden't reproduce, but he's going to test against https://gerrit.git.wmflabs.org/r/ to see if the problem happens for that site too [16:24:53] I would bet it's something to do with the apache config, but not sure/haven't looked into it [16:25:19] yep, that's what he says too. [16:26:17] thcipriani it's fixed with 2.16! [16:26:28] (at least it runs for me on gerrit.git.wmflabs.org) [16:26:32] what's fixed? [16:26:35] LocalUsernamesToLowerCase [16:27:51] Krinkle: howdy ;) You reported some issues with mediawiki jobs erroneously triggered on some branches. I looked a bit at it and yeah zuul branches/filters are a mess ;((( [16:28:29] hashar: Just wanted to confirm that it's okay to change so that mediawiki-* doesn't run when wmf-* already runs, for swat/wmf branche. [16:28:32] Krinkle: I could not find in the backlog the exact issue you reported here, if you have one please do mail it / paste it / task it. I will use it as a test case for a fixup ;/ [16:28:36] ahh [16:28:52] I might have time for it at some point, just wanted to check first if it's intentional. [16:29:23] I can't quite remember what the mediawiki-* jobs are for though :-/ [16:29:37] surely, I should have added a nice description for each of those variants [16:29:46] then [16:30:04] there is an issue when applying filters/branches. I think only the first match is applied [16:30:22] or to say it otherwise, the branches / skip-if filters are not merged when a job is matched by several entries in - jobs: [16:30:23] ;( [16:31:07] (also unrelated, Chromium 0.72 is broken on Debian Stretch. It segfaults :/) [16:59:09] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T206673 (10hashar) [16:59:54] hashar: FYI not sure if expected but Work requests waiting in Zuul Gearman server has been unknown for ~24h now in icinga [17:01:16] godog: then maybe the icinga check is broken somehow [17:01:23] at least the metric in statsd looks good based on https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1 [17:04:11] hashar: ack, thanks, might take a closer look tomorrow [17:09:53] regarding the beta cluster, anyone any idea why this might be giving me a 404? https://wikidata.beta.wmflabs.org/w/api.php?action=wbsearchentities&search=P245962&format=json&language=en&uselang=en&type=property [17:10:58] or just https://wikidata.beta.wmflabs.org/w/api.php [17:12:10] 10Beta-Cluster-Infrastructure: https://wikidata.beta.wmflabs.org/w/api.php gets 404 - https://phabricator.wikimedia.org/T217261 (10Addshore) [17:12:19] 10Beta-Cluster-Infrastructure: https://wikidata.beta.wmflabs.org/w/api.php gets 404 - https://phabricator.wikimedia.org/T217261 (10Addshore) [17:13:08] 10Beta-Cluster-Infrastructure: https://wikidata.beta.wmflabs.org/w/api.php gets 404 - https://phabricator.wikimedia.org/T217261 (10Addshore) p:05Triage→03High [17:13:25] 10Beta-Cluster-Infrastructure, 10Wikidata, 10Wikidata-Campsite: https://wikidata.beta.wmflabs.org/w/api.php gets 404 - https://phabricator.wikimedia.org/T217261 (10Addshore) [17:13:31] 10Beta-Cluster-Infrastructure, 10Wikidata: https://wikidata.beta.wmflabs.org/w/api.php gets 404 - https://phabricator.wikimedia.org/T217261 (10Addshore) [17:17:43] 10Beta-Cluster-Infrastructure, 10Wikidata: Beta site api.phps gets 404 - https://phabricator.wikimedia.org/T217261 (10Addshore) [17:18:14] 10Beta-Cluster-Infrastructure, 10Wikidata: Beta site api.phps gets 404 - https://phabricator.wikimedia.org/T217261 (10Addshore) [17:18:17] Reedy: ^^ any guesses? :D [17:19:06] someone broke the symlinks/docroots? [17:19:18] I'm in an interview, so can't look for a bit [17:19:42] ack [17:19:44] *digs* [17:21:07] (03PS4) 10Thcipriani: Train notes: automatically upload changelog [integration/config] - 10https://gerrit.wikimedia.org/r/492758 [17:21:09] I'm guessing it's one of those [17:21:55] (03CR) 1020after4: [C: 03+2] "lets do it" [integration/config] - 10https://gerrit.wikimedia.org/r/492758 (owner: 10Thcipriani) [17:21:57] im not even sure where they are for beta [17:22:38] Same place [17:22:47] Look at the apache config on one of the mw servers to see which they're using? [17:22:53] ack [17:23:59] (03Merged) 10jenkins-bot: Train notes: automatically upload changelog [integration/config] - 10https://gerrit.wikimedia.org/r/492758 (owner: 10Thcipriani) [17:26:28] oh, i forgot, i have an issue with sshing to beta machines right now ... [17:26:37] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T206673 (10alaa_wmde) [17:28:05] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T206673 (10Addshore) [17:46:56] 10Beta-Cluster-Infrastructure: Beta cluster is showing 404 while trying to open VE - https://phabricator.wikimedia.org/T217266 (10Ryasmeen) [17:48:22] 10Beta-Cluster-Infrastructure: Beta cluster is showing 404 while trying to open VE - https://phabricator.wikimedia.org/T217266 (10Ryasmeen) [17:48:29] 10Beta-Cluster-Infrastructure, 10Wikidata: Beta site api.phps gets 404 - https://phabricator.wikimedia.org/T217261 (10Ryasmeen) [18:11:58] addshore, need help with SSH? [18:12:11] interesting that it's nginx showing that 404 [18:12:26] 10Release-Engineering-Team (Kanban), 10Scap: Automate updating deployment notes - https://phabricator.wikimedia.org/T196516 (10thcipriani) a:03thcipriani [18:13:02] 10Release-Engineering-Team (Kanban), 10Scap: Automate weekly branch cut - https://phabricator.wikimedia.org/T196517 (10mmodell) @thcipriani and I went over all of the issues that were previously blockers to getting this done and we've decided that this is now officially unblocked. We should be able to do this... [18:13:25] krenair@deployment-mediawiki-07:~$ curl -H "Host: en.wikipedia.beta.wmflabs.org" http://deployment-mediawiki-07/w/api.php [18:13:35] this gets me the MW API Help page [18:13:56] interestingly, not on -09 [18:14:07] The last Puppet run was at Wed Feb 27 16:22:01 UTC 2019 (111 minutes ago). Puppet is disabled. jmm testing PHP packages [18:14:24] looks like index.php is a 404 there too [18:14:42] jmm would be moritzm [18:15:07] 10Beta-Cluster-Infrastructure, 10Performance-Team: Move XHGui from tungsten to webperf-002 - https://phabricator.wikimedia.org/T180761 (10Krinkle) [18:16:03] Krenair: if your around to help me poke that would be great [18:16:19] let me try and ssh to deployment-mediawiki-07 [18:16:34] 10Beta-Cluster-Infrastructure, 10Wikidata: Beta site api.phps gets 404 - https://phabricator.wikimedia.org/T217261 (10Krenair) looks fine on deployment-mediawiki-07, but -09: `krenair@deployment-mediawiki-09:~$ curl -H "Host: en.wikipedia.beta.wmflabs.org" http://deployment-mediawiki-09/w/api.php ... [18:20:43] addshore, any luck? [18:22:17] 10Release-Engineering-Team (Kanban), 10Scap: Automate weekly branch cut - https://phabricator.wikimedia.org/T196517 (10mmodell) We can do almost the entire branch cut via API: either by calling https://gerrit-review.googlesource.com/Documentation/cmd-create-branch.html or the equivalent https rest api in gerrit. [18:22:41] 10Release-Engineering-Team (Kanban), 10Scap: Automate weekly branch cut - https://phabricator.wikimedia.org/T196517 (10mmodell) itshappening [18:27:38] !log updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/c/integration/config/+/492758 [18:27:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:28:11] Krenair: nope [18:28:45] im trying to get there through the secondary.bastion.wmflabs.org [18:28:46] 10Release-Engineering-Team: Create production code deployment management process - https://phabricator.wikimedia.org/T203703 (10Jrbranaa) Another thoughts to capture... This process should probably result in a workflow that is as automated as possible. For example - initiated by the creation of a phab task tha... [18:28:50] i can ssh to secondary.bastion.wmflabs.org itself, and that works [18:29:01] then trying to proxy to deployment-mediawiki-07.eqiad.wmflabs:22 through it just kind of hangs [18:29:14] addshore, can you run it with -vvv and paste the output? [18:29:30] yeh, im just waiting for it to finish so i get the full output [18:29:41] if it's already hung that's probably enough [18:30:21] https://www.irccloud.com/pastebin/ZidQTVyt/ [18:31:10] The last bit is just [18:31:14] https://www.irccloud.com/pastebin/jSbsUx6J/ [18:32:10] I can ssh to other instances in other projects [18:32:15] via the same bastion [18:32:25] just seems to be some issue with deployment-prep for me O_o [18:33:26] 10Release-Engineering-Team (Kanban), 10Scap: Automate weekly branch cut - https://phabricator.wikimedia.org/T196517 (10mmodell) The rest api is `PUT /projects/{project-name}/branches/{branch-id}` ( [[ https://gerrit-review.googlesource.com/Documentation/rest-api-projects.html#create-branch | api docs ]] ) The... [18:35:41] I looked at the deployment groups and I don't see an entry for something like "test.wikipedia.org" but for wikicommons and/or wikisource. Does something like that exist? [18:35:44] 10Release-Engineering-Team (Kanban), 10MediaWiki-Release-Tools: Automate weekly branch cut - https://phabricator.wikimedia.org/T196517 (10mmodell) [18:36:38] 10Release-Engineering-Team: Create production code deployment management process - https://phabricator.wikimedia.org/T203703 (10greg) p:05Triage→03Normal [18:37:08] aezell, no [18:37:18] Krenair: Thanks. [18:37:23] 10Release-Engineering-Team (Kanban), 10MediaWiki-Release-Tools: Automate weekly branch cut - https://phabricator.wikimedia.org/T196517 (10mmodell) [18:37:51] addshore, well you are in the group [18:37:59] addshore, can you paste your ~/.ssh/config? [18:38:08] aezell: we have a test commons wiki, if that's what you are asking [18:38:16] we do? [18:38:29] it's a temporary thing for structured data, so not sure how well it matches real commons [18:38:38] so we do. [18:38:46] ooh that is new [18:38:51] created this year [18:38:54] Interesting. That might work for my purposes. [18:39:06] What's the URL? [18:39:11] sorry aezell, didn't know about that [18:39:14] testcommons.wikimedia.org ? [18:39:21] https://www.irccloud.com/pastebin/wfEg7O08/ [18:39:22] Krenair: ^^ [18:39:24] hmm [18:39:40] test-commons [18:39:44] aezell, https://test-commons.wikimedia.org/wiki/Main_Page [18:40:03] Krenair and tgr - Thanks! [18:40:26] addshore, that looks okay... [18:40:31] Krenair: yup [18:40:52] https://www.irccloud.com/pastebin/kRkcpIAN/ [18:40:56] Krenair: there is also that bit ^^ [18:41:29] Krenair:so im trying ot go to deployment-mediawiki-07.eqiad.wmflabs, is there anything in the logs of that host, or of the secondary.bastion.wmflabs.org im going through? [18:41:44] i guess I could check the secondary.bastion.wmflabs.org myself perhaps, oh, but i might not be able to see the logs there [18:41:48] addshore, try it with the proper FQDN? [18:41:53] deployment-mediawiki-07.deployment-prep.eqiad.wmflabs [18:42:25] trying [18:42:29] but hanging in the same bit [18:43:40] addshore, okay can you try it with -vvv and dump the whole thing, including command and all output? [18:43:59] yup, 2 secs, shall we take this to Pm instead of spamming this channel? :D [18:44:58] you can [18:45:06] nothing else is going on in the channel though [18:50:09] we've figured it out, it's not addshore's setup at all, but a mistake in puppet [18:50:13] I'm making a patch [18:50:17] =] [18:50:36] (03PS1) 10Thcipriani: release-notes: Create leading Dockerfile dirs [integration/config] - 10https://gerrit.wikimedia.org/r/493288 [18:52:28] addshore, on the instance you were able to get into, can you check if the ferm service is running? [18:52:55] on mw-07? [18:52:59] no [18:53:02] the one you were able to get into [18:53:09] under wikibase-registry [18:53:23] https://gerrit.wikimedia.org/r/493289 [18:53:24] oh, but via the secondary bastion? [18:53:49] (03PS2) 10Thcipriani: release-notes: Create leading Dockerfile dirs [integration/config] - 10https://gerrit.wikimedia.org/r/493288 [18:54:34] Krenair: is "ferm" the exact service name? if so, it is not running [18:55:03] addshore, does `sudo iptables -L` have anything about ssh? [18:55:43] not that i can see [18:55:45] https://www.irccloud.com/pastebin/grWu4I1G/ [18:56:55] Krenair: shell we move to -cloud? or? [18:57:32] * Krenair shrugs [18:57:53] addshore, that explains why you were able to get to that host but not deployment-mediawiki-07 [18:58:19] deployment-mediawiki-07 has ferm setup and restricting SSH to only the listed bastions, but the list was wrong [18:58:28] your host did not have SSH restricted so you could get there from anywhere within the network [18:58:55] gotcha [18:59:48] right, looks like the patch is merged, so in a bit i should be able to switch back to secondary [19:00:02] so I think all weird SSH behaviour is now explained :) [19:00:07] yup [19:00:49] puppet on mediawiki-07 is not happy [19:01:26] (03PS3) 10Thcipriani: release-notes: Create leading Dockerfile dirs [integration/config] - 10https://gerrit.wikimedia.org/r/493288 [19:02:59] why is puppet not fixing ferm [19:03:42] it's very upset about nginx but still [19:05:43] (03CR) 10Thcipriani: [C: 03+2] release-notes: Create leading Dockerfile dirs [integration/config] - 10https://gerrit.wikimedia.org/r/493288 (owner: 10Thcipriani) [19:06:10] ugh I had forgotten what a mess ferm is here [19:07:05] (03CR) 10Smalyshev: [C: 03+1] Enable phan for WikibaseSchema [integration/config] - 10https://gerrit.wikimedia.org/r/493213 (https://phabricator.wikimedia.org/T217180) (owner: 10Michael Große) [19:07:09] (03Merged) 10jenkins-bot: release-notes: Create leading Dockerfile dirs [integration/config] - 10https://gerrit.wikimedia.org/r/493288 (owner: 10Thcipriani) [19:08:39] !log updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/c/integration/config/+/492758 (take II) [19:08:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:09:18] it looks like the puppet repo updater has not been running [19:09:31] just ran a fetch and found it drops to u+17-46 from u+17 [19:09:58] that explains why ferm changes were not taking effect [19:10:45] right that fixed the config [19:11:12] of course it didn't actually fix iptables because the ferm config is still considered broken by the broken version of ferm we're using because... why are we still using this [19:12:26] oh right, the roll out of the fix was cancelled in favour of faidon's one: https://phabricator.wikimedia.org/T153468#4739672 [19:13:48] 10Beta-Cluster-Infrastructure, 10DNS, 10Operations, 10Traffic, and 4 others: Ferm's upstream Net::DNS Perl library questionable handling of NOERROR responses without records causing puppet errors when we try to @resolve AAAA in labs - https://phabricator.wikimedia.org/T153468 (10Krenair) This is still caus... [19:17:30] okay I'm going to live hack the prometheus ferm config on our puppetmaster to stop this [19:17:52] we can't have other ferm updates being broken like this [19:18:08] 10Beta-Cluster-Infrastructure, 10Wikidata, 10User-Ryasmeen: Beta site api.phps gets 404 - https://phabricator.wikimedia.org/T217261 (10MoritzMuehlenhoff) 05Open→03Resolved a:03MoritzMuehlenhoff Fixed, I reverted the packages for now, I'll need to revisit this in mw-vagrant or so. [19:20:03] !log reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/492758 [19:20:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:21:48] 10Beta-Cluster-Infrastructure, 10Wikidata, 10User-Ryasmeen: Beta site api.phps gets 404 - https://phabricator.wikimedia.org/T217261 (10Krenair) by all means test it on beta, just maybe remove it if stuff happens like the entire wiki going 404 on that host [19:22:45] hm, hey yall, anybody had problems creating admin/vagrant user account in mw-vagrant before? [19:22:47] i'm getting [19:22:53] Uncaught RuntimeException: RedisConnectionPool requires a Redis client library. See https://www.mediawiki.org/wiki/Redis#Setup [19:23:05] i'v [19:23:07] i've done apt-get install redis-server php-redis [19:23:21] 10Beta-Cluster-Infrastructure, 10DNS, 10Operations, 10Traffic, and 4 others: Ferm's upstream Net::DNS Perl library questionable handling of NOERROR responses without records causing puppet errors when we try to @resolve AAAA in labs - https://phabricator.wikimedia.org/T153468 (10Krenair) Added a new deploy... [19:29:45] cumin 66/67.... [19:29:49] * Krenair twiddles thumbs [19:47:32] 10Project-Admins, 10Community-Tech: Create a tag for new Community Tech project 'E-Book Export' - https://phabricator.wikimedia.org/T217277 (10jmatazzoni) [19:47:46] woah, deployment-logstash2 key changed? [19:48:49] noticed some things in the sal from yesterday that are possibly related to that: https://tools.wmflabs.org/sal/releng?d=2019-02-26 [19:50:08] restarting a service and deleting some logs isn't going to result in an ssh host key change [19:50:50] true enough [19:51:01] (03CR) 10MaxSem: [C: 03+2] Prohibit aliases is_long, is_double and is_real [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/492663 (owner: 10Mainframe98) [19:51:25] doubt someone has decided to replace it and not increment the number as it's got a 2015 deprecated image [19:51:50] maybe it was one of those that randomly changed host key during the eqiad1 migration. never did get an explanation on that one [19:51:56] (03Merged) 10jenkins-bot: Prohibit aliases is_long, is_double and is_real [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/492663 (owner: 10Mainframe98) [19:52:23] (03CR) 10jenkins-bot: Prohibit aliases is_long, is_double and is_real [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/492663 (owner: 10Mainframe98) [19:53:09] weird ferm thing on -imagescaler02 and -mediawiki-09 [19:53:25] gonna file a new ticket about this one [19:54:45] 10Beta-Cluster-Infrastructure: Ferm failing to start in a new way on deployment-(imagescaler02|mediawiki-09) - https://phabricator.wikimedia.org/T217279 (10Krenair) [19:56:54] I don't have time to chase every single problem right now but I notice a few apache hosts (e.g. -mediawiki-07) where nginx is trying to nick apache's port 80 and puppet is very upset over it. someone should look at that [20:13:24] btw the answer to my question way ^ is : https://phabricator.wikimedia.org/T213016 [20:14:16] (03CR) 10Catrope: [C: 03+1] Make GrowthExperiments depend on PageImages and PageViewInfo [integration/config] - 10https://gerrit.wikimedia.org/r/493261 (https://phabricator.wikimedia.org/T216217) (owner: 10Sbisson) [20:18:41] thcipriani https://gerrit.wikimedia.org/r/#/c/operations/software/gerrit/+/493311/-1..1 [20:19:07] (upgrades to 2.16.6) and also now i can install image-diff as discussed! (as that release includes the dep that plugin needs) [20:22:08] (03CR) 10Thcipriani: [C: 03+1] "there's also a conftool_config default config parameter to remove in scap/config.py" [tools/scap] - 10https://gerrit.wikimedia.org/r/491412 (owner: 10Giuseppe Lavagetto) [20:28:40] paladox: nice :) [20:31:10] 10Release-Engineering-Team, 10Scap, 10serviceops: Deploy scap 3.9.1-1 - https://phabricator.wikimedia.org/T217287 (10thcipriani) [20:31:48] 10Release-Engineering-Team (Watching / External), 10Scap, 10serviceops: Deploy scap 3.9.1-1 - https://phabricator.wikimedia.org/T217287 (10thcipriani) [20:38:38] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.33.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T206673 (10hashar) commonswiki is emitting exceptions so I have rollbacked group1. Filled as T217285 [20:52:54] Project beta-scap-eqiad build #239509: 04FAILURE in 1.2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/239509/ [21:03:57] thcipriani im going to merge my prettify ci change (it dosen't work if the comments are hidden) [21:04:04] but i have a change to fix that upstream [21:04:15] using https://gerrit-review.googlesource.com/c/gerrit/+/215792 [21:04:42] Yippee, build fixed! [21:04:43] Project beta-scap-eqiad build #239510: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/239510/ [21:22:00] 10Phabricator, 10Proton, 10Reading-Infrastructure-Team-Backlog: Update Herald (H228) to include project 2960 - https://phabricator.wikimedia.org/T217078 (10mmodell) [21:22:03] 10Phabricator, 10Release-Engineering-Team (Kanban): Unable to edit Herald rules:: "Argument 1 passed to HeraldTokenizerFieldValue::setValueMap() must be of the type array, object given" - https://phabricator.wikimedia.org/T217082 (10mmodell) 05Open→03Resolved [21:44:29] thcipriani, marxarelli : After looking at some of the helm chart info I think Kask we’ll need the `service-pipeline-test-and-publish` template after alI. https://gerrit.wikimedia.org/r/#/c/integration/config/+/490678/ [21:44:49] Want to make sure before I update the PR. [21:51:29] clarakosi: sure, service-pipeline-test-and-publish is a post-merge job that will run helm test and then build and publish your blubber production variant [21:52:01] service-pipeline-test will run for new patchsets and when someone +2's before pre-merge [21:52:25] er just per-merge (not "before pre-merge") [21:54:03] (03PS4) 10Clarakosi: Setup Basic CI for mediawiki/services/kask [integration/config] - 10https://gerrit.wikimedia.org/r/490678 (https://phabricator.wikimedia.org/T209106) [21:55:06] 10Project-Admins, 10Community-Tech: Create a tag for new Community Tech project 'E-Book Export' - https://phabricator.wikimedia.org/T217277 (10Aklapper) 05Open→03Resolved Requested public project #E-Book-Export has been created: https://phabricator.wikimedia.org/project/view/3917/ Please encourage interes... [21:56:08] thcipriani: ok. thanks! And done btw: https://gerrit.wikimedia.org/r/#/c/integration/config/+/490678/ [21:58:34] (03PS5) 10Thcipriani: Setup Basic CI for mediawiki/services/kask [integration/config] - 10https://gerrit.wikimedia.org/r/490678 (https://phabricator.wikimedia.org/T209106) (owner: 10Clarakosi) [21:58:55] (03PS6) 10Thcipriani: Setup Basic CI for mediawiki/services/kask [integration/config] - 10https://gerrit.wikimedia.org/r/490678 (https://phabricator.wikimedia.org/T209106) (owner: 10Clarakosi) [21:59:14] (03CR) 10Thcipriani: [C: 03+2] Setup Basic CI for mediawiki/services/kask [integration/config] - 10https://gerrit.wikimedia.org/r/490678 (https://phabricator.wikimedia.org/T209106) (owner: 10Clarakosi) [22:00:51] Project beta-scap-eqiad build #239515: 04FAILURE in 9 min 25 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/239515/ [22:01:02] (03PS1) 10Paladox: Gerrit: Add CI for operations/software/gerrit (includes new docker image) [integration/config] - 10https://gerrit.wikimedia.org/r/493328 [22:01:09] (03Merged) 10jenkins-bot: Setup Basic CI for mediawiki/services/kask [integration/config] - 10https://gerrit.wikimedia.org/r/490678 (https://phabricator.wikimedia.org/T209106) (owner: 10Clarakosi) [22:02:28] !log reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/490678/ [22:02:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:03:34] ^ clarakosi deployed! next time you make a patchset or merge a patchset or push a tag: pipeline will be run [22:04:08] thcipriani do you know if it's possible to use 2 docker images in one? [22:04:17] Ie from https://github.com/wikimedia/integration-config/tree/master/dockerfiles we would need java8 and nodejs [22:04:38] thanks thcipriani! Gonna go test it now :) [22:04:43] paladox: sounds like you'll just need a new docker image [22:04:49] ah ok [22:05:03] clarakosi: awesome :) [22:11:43] clarakosi: looks like you need "make" https://integration.wikimedia.org/ci/job/service-pipeline-test/476/console I had to add a few things that the golang image had that the wmf base golang image doesn't cf: https://gerrit.wikimedia.org/r/#/c/blubber/+/493090/1/.pipeline/blubber.yaml [22:12:27] Yippee, build fixed! [22:12:27] Project beta-scap-eqiad build #239516: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/239516/ [22:12:37] ahh ok. Will try thcipriani [22:13:39] 10Phabricator, 10Release-Engineering-Team, 10User-MModell: Make sure elasticsearch 6 is supported in phabricator - https://phabricator.wikimedia.org/T181393 (10EBernhardson) This upgrade is scheduled for next week, assuming this weeks train rolls forward as expected. [22:36:54] thcipriani: is there away to override the environment in a blubber.yaml? [22:37:07] thcipriani: for test and prep? [22:37:57] like, for the test entrypoint and the build command? [22:38:35] you need an envvar you mean? "runs: {environment: {HELLO_WORLD: '1'}}" [22:38:57] should work [22:39:24] roger [22:39:49] thcipriani: for builder too? [22:40:54] urandom: that'll just export an envvar inside a specific variant [22:41:14] so if you need it for the builder, you could use it in the variant that has the builder [22:41:36] gotcha [22:44:31] I want to import my PEG.js fork into Gerrit [22:45:25] TimStarling you should be able to use git --mirror [22:45:36] *git push --mirror [22:46:02] I don't know if it's necessary to have all the branches from the original, if that's what that does [22:46:28] the idea is to rename it to wikipeg and to develop it further away from the mainline [22:47:10] do you think it should be a top-level project in Gerrit? [22:47:35] depends, do you have any other plans to add repo's with similar names? [22:48:16] it's a support project for Parsoid at present, so I guess one other option would be mediawiki/services/parsoid/wikipeg [22:48:39] although I don't like to use non-empty projects as parents, messes up the tree [22:49:15] but this is as part of the project to port Parsoid to PHP, so that name will be obsolete within a year [22:49:32] TimStarling you can use any parents (dosen't have to be mediawiki/services/parsoid) you could have mediawiki/services/parsoid/xxx (without using that as a parent) [22:50:12] though if the plan is to deprecate mediawiki/services/parsoid (then go with a top level name) :) [22:52:05] ok, I'm putting it at the top level [22:53:40] +1 seems like a good place for it. [23:01:34] (03PS1) 10Hashar: doc: add docker-pkg [integration/docroot] - 10https://gerrit.wikimedia.org/r/493335 [23:04:57] remote: ERROR: In commit cb640cd0b2835dbc74abb627c5dff5d1d3ee672a [23:04:57] remote: ERROR: committer email address david@majda.cz [23:04:57] remote: ERROR: does not match your user account and you have no 'forge committer' permission. [23:05:11] I guess I should get the forge committer permission and try again? [23:06:48] yeah, you'll need to temporarily enable that permission for the initial mirror [23:07:32] TimStarling: i think you can do that from Projects->Access in phab, if you don't those rights i'm pretty sure i do [23:07:43] s/phab/gerrit [23:08:29] thanks, I've figured it out I think [23:08:45] I'm a Gerrit administrator so in theory I can do most things [23:09:01] s/most/all [23:09:05] Is the backports repo enabled on the wmf's stretch image? [23:09:05] yup, should be same [23:09:11] (03PS2) 10Paladox: Gerrit: Add CI for operations/software/gerrit (includes new docker image) [integration/config] - 10https://gerrit.wikimedia.org/r/493328 [23:09:32] or put another way... where is the source for that? [23:10:17] (03PS3) 10Paladox: Gerrit: Add CI for operations/software/gerrit (includes new docker image) [integration/config] - 10https://gerrit.wikimedia.org/r/493328 [23:11:01] urandom: looks like it is: https://phabricator.wikimedia.org/P8141 [23:11:09] TimStarling you need to grant your self the "Forge Committer Identity" right [23:11:30] and possibly Forge Author Identity [23:11:31] yeah, it's done, the push worked [23:11:35] k [23:11:55] thcipriani: what would be the way to install a backport via your blubber.yaml? [23:12:14] thcipriani: how would one do the equiv of apt -t stretch-backports install foo? [23:14:31] urandom: we don't support specifying repo, but version should still work, i.e. {apt: packages: [package=]} [23:14:45] thcipriani: cool, let's try! [23:16:50] (03PS4) 10Paladox: Gerrit: Add CI for operations/software/gerrit (includes new docker image) [integration/config] - 10https://gerrit.wikimedia.org/r/493328 [23:17:41] (03PS5) 10Paladox: Gerrit: Add CI for operations/software/gerrit (includes new docker image) [integration/config] - 10https://gerrit.wikimedia.org/r/493328 [23:21:18] (03PS6) 10Paladox: Gerrit: Add CI for operations/software/gerrit (includes new docker image) [integration/config] - 10https://gerrit.wikimedia.org/r/493328 [23:22:09] thcipriani: that's probably not going to be an equivalent [23:22:39] because of dependencies also package in backports [23:22:52] it's going to be an awful game of wackamole at best [23:24:11] ugh, so package=version failed? dependency problems? [23:24:14] does anyone know how i can build just the docker images in https://gerrit.wikimedia.org/r/493328 ? [23:25:12] urandom: currently blubber doesn't support specifying a target release, unfortunately. [23:25:49] k [23:26:19] thcipriani: package=version succeeds, but it won't pull in transitive dependencies also in backports [23:26:41] ah [23:28:10] it doesn't seem like it'd be hard to add support for a target release into blubber, if that's what's needed (sounds like a it is) [23:28:44] this might not be great idea anyway... the particular backport I was considering [23:36:03] thcipriani upstream are now aware of the ldap command issue with 2.15! (he at least said he would look at it later!) [23:36:06] https://bugs.chromium.org/p/gerrit/issues/detail?id=10531 [23:36:53] ah, interesting, good that it's a known, I guess [23:37:14] that's after me telling them :) [23:38:13] heh, oh :) [23:40:16] 10Project-Admins, 10Community-Tech: Create a tag for new Community Tech project 'E-Book Export' - https://phabricator.wikimedia.org/T217277 (10jmatazzoni) Hi Andre, Thanks for the e-book-export tag . I just realized I didn’t specify any Hearald rules for th... [23:42:44] thcipriani do you know how i build the docker images in https://gerrit.wikimedia.org/r/493328 [23:42:46] to test locally? [23:42:57] the readme dosen't explain how to build :) [23:43:20] info is a little buried: https://www.mediawiki.org/wiki/Continuous_integration/Docker#Build_images_locally [23:43:34] (03PS1) 10Smalyshev: CI configuration for Blazegraph [integration/config] - 10https://gerrit.wikimedia.org/r/493340 (https://phabricator.wikimedia.org/T216855) [23:43:38] ah [23:43:40] thanks! [23:44:00] paladox: it might be best to copy just the images you need to a directory on their own, otherwise docker-pkg is going to build all the images in dockerfiles [23:44:12] ah ok [23:44:14] which takes up a lot of space/takes some time [23:44:24] (03CR) 10jerkins-bot: [V: 04-1] CI configuration for Blazegraph [integration/config] - 10https://gerrit.wikimedia.org/r/493340 (https://phabricator.wikimedia.org/T216855) (owner: 10Smalyshev) [23:53:45] 10Phabricator: JohnLewis getting added to tasks when replying by email - https://phabricator.wikimedia.org/T217302 (10DStrine)