[00:03:06] James_F: ok will do once it finishes the test run [00:03:35] I wonder though why all that tower needs to be together? [00:03:51] SMalyshev: Because MediaWiki is a catastrophic monolith. [00:04:14] SMalyshev: Any extension can do anything it likes to break any global at any time. [00:04:31] * James_F grumps off home. :-) [00:08:12] James_F: https://integration.wikimedia.org/ci/job/mwselenium-quibble-docker/9972/ :( [00:08:29] that means 497015 cannot be merged I presume [00:15:55] Project beta-scap-eqiad build #241559: 04FAILURE in 9 min 46 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/241559/ [00:16:16] 10Continuous-Integration-Infrastructure: Jenkins jobs regularly being queued while resources appear to be readily available - https://phabricator.wikimedia.org/T218458 (10Krinkle) [00:20:09] SMalyshev: When patching core or a wmf-deployed extension, we run a Jenkins job that (aside from the project you're changing) also tests the others. In theory, if your tests pass on master of everything else, but one of those has a +2 merge pending, it might be that your change passes now, but would fail after it merges. In order to prevent the master branch of any project failing whilst the last commit passed, we re-run the tests on +2, and [00:20:10] in a serial pipeline. To optimise this, however, Zuul speculatively starts testing patch 1 to mwext X with core patch 2 (ahead in the queue). The only thing that slows it down is if 1) the patch fails (yay, we prevented a problem) or 2) if the thing ahead in the queue has jobs slower than yours, so while it starts testing, it won't merge until it has the clear green. [00:22:25] In case of 1, we restart the parallel stack without the broken patch. This is pretty rare since this would only happen if that patch genuinely only fails with the other pending patch. Case 2 is much more common. E.g. if a Wikibase patch is in flight before a core patch, the core patch takes twice as long due to all the variants Wikibase covers. This has gotten better but it's still longer. [00:22:44] WikibaseCirrusSearch doesn't have this issue though. It's actually faster than core. [00:23:16] the reason for the stack not making progress right now, however, is unrelated to any of that. [00:23:23] It's https://phabricator.wikimedia.org/T218458 [00:26:41] Yippee, build fixed! [00:26:41] Project beta-scap-eqiad build #241560: 09FIXED in 9 min 26 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/241560/ [02:40:28] Project beta-scap-eqiad build #241572: 04FAILURE in 13 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/241572/ [02:53:32] Yippee, build fixed! [02:53:33] Project beta-scap-eqiad build #241573: 09FIXED in 9 min 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/241573/ [02:55:08] PROBLEM - Puppet staleness on deployment-restbase02 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [43200.0] [03:02:20] PROBLEM - Puppet staleness on deployment-mediawiki-09 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [43200.0] [03:25:24] Project beta-scap-eqiad build #241576: 04FAILURE in 8 min 24 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/241576/ [03:36:33] Yippee, build fixed! [03:36:33] Project beta-scap-eqiad build #241577: 09FIXED in 9 min 48 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/241577/ [04:08:45] Project beta-scap-eqiad build #241580: 04FAILURE in 9 min 20 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/241580/ [04:19:40] Yippee, build fixed! [04:19:40] Project beta-scap-eqiad build #241581: 09FIXED in 9 min 34 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/241581/ [05:46:14] Project beta-scap-eqiad build #241589: 04FAILURE in 8 min 52 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/241589/ [05:57:11] Yippee, build fixed! [05:57:11] Project beta-scap-eqiad build #241590: 09FIXED in 9 min 37 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/241590/ [07:02:33] Project beta-scap-eqiad build #241596: 04FAILURE in 9 min 28 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/241596/ [07:13:34] Yippee, build fixed! [07:13:35] Project beta-scap-eqiad build #241597: 09FIXED in 9 min 41 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/241597/ [08:08:05] 10Release-Engineering-Team (Watching / External), 10Scap, 10serviceops, 10User-jijiki: Allow scap sync to deploy gradually - https://phabricator.wikimedia.org/T212147 (10jijiki) [08:15:25] 10Scap, 10serviceops: Define a mediawiki "version" - https://phabricator.wikimedia.org/T218412 (10jijiki) [08:16:14] 10Scap, 10Operations, 10serviceops, 10User-jijiki: Introduce state to Scap - https://phabricator.wikimedia.org/T209881 (10jijiki) [08:16:16] 10Scap, 10serviceops: Define a mediawiki "version" - https://phabricator.wikimedia.org/T218412 (10jijiki) [08:16:40] 10Scap, 10Operations, 10serviceops, 10Goal, 10User-jijiki: SRE FY2019 Q3:TEC6: First steps towards Canary Deployments - https://phabricator.wikimedia.org/T213156 (10jijiki) [10:04:03] Project mediawiki-core-doxygen-docker build #5456: 04FAILURE in 0.16 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/5456/ [10:08:21] PROBLEM - puppet last run on contint1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_jenkins CI slave scripts] [10:09:50] Project beta-code-update-eqiad build #238969: 04FAILURE in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/238969/ [10:13:56] 10Gerrit, 10Release-Engineering-Team: gerrit.wikimedia.org is down - https://phabricator.wikimedia.org/T218472 (10Daimona) [10:14:09] 10Gerrit, 10Release-Engineering-Team: gerrit.wikimedia.org is down - https://phabricator.wikimedia.org/T218472 (10Daimona) p:05Triage→03Unbreak! [10:14:17] Project beta-code-update-eqiad build #238970: 04STILL FAILING in 1 min 16 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/238970/ [10:24:18] Project beta-code-update-eqiad build #238971: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/238971/ [10:34:18] Project beta-code-update-eqiad build #238972: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/238972/ [10:34:21] PROBLEM - puppet last run on contint2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_jenkins CI slave scripts] [10:39:03] 10Gerrit, 10Release-Engineering-Team: gerrit.wikimedia.org is down - https://phabricator.wikimedia.org/T218472 (10Peachey88) `lang=irc !log stop apache on cobalt for maintenance ` [10:44:18] Project beta-code-update-eqiad build #238973: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/238973/ [10:54:18] Project beta-code-update-eqiad build #238974: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/238974/ [11:04:03] Project mediawiki-core-doxygen-docker build #5457: 04STILL FAILING in 0.11 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/5457/ [11:04:18] Project beta-code-update-eqiad build #238975: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/238975/ [11:14:18] Project beta-code-update-eqiad build #238976: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/238976/ [11:24:18] Project beta-code-update-eqiad build #238977: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/238977/ [11:34:17] Project beta-code-update-eqiad build #238978: 04STILL FAILING in 1 min 16 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/238978/ [11:44:18] Project beta-code-update-eqiad build #238979: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/238979/ [11:54:17] Project beta-code-update-eqiad build #238980: 04STILL FAILING in 1 min 16 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/238980/ [12:04:03] Project mediawiki-core-doxygen-docker build #5458: 04STILL FAILING in 0.12 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/5458/ [12:04:18] Project beta-code-update-eqiad build #238981: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/238981/ [12:14:18] Project beta-code-update-eqiad build #238982: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/238982/ [12:24:18] Project beta-code-update-eqiad build #238983: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/238983/ [12:34:18] Project beta-code-update-eqiad build #238984: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/238984/ [12:41:21] 10Gerrit, 10Release-Engineering-Team: gerrit.wikimedia.org is down - https://phabricator.wikimedia.org/T218472 (10MarcoAurelio) Well if gerrit is on maintenance and apache has been disconnected then this downtime is to be expected :-) Given that this is not an involuntary outage, shall we degrade from UBN? [12:44:18] Project beta-code-update-eqiad build #238985: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/238985/ [12:46:27] 10Gerrit, 10Release-Engineering-Team: gerrit.wikimedia.org is down - https://phabricator.wikimedia.org/T218472 (10Daimona) @MarcoAurelio Well, "expected" to an extent, as I couldn't find any information about it before the actual shutdown. And BTW, is there a related task/discussion/you-name-it? As for the pr... [12:47:50] 10Gerrit, 10Release-Engineering-Team, 10Operations: gerrit.wikimedia.org is down - https://phabricator.wikimedia.org/T218472 (10Paladox) [12:50:42] Daimona: well, I'm not aware of any task or wikitech-l discussion, no. Maybe the downtime was not scheduled but had to be done? paladox surely knows :) [12:50:56] hauskatze the downtime is no scheduled. [12:51:03] and from what i hear they are trying to fall over [12:51:23] what broke? [12:51:40] Im not sure [12:51:57] hauskatze: indeed, I couldn't find anything aside from little info from #wikimedia-operations [12:52:02] but gerrit2001 cannot be used due to db problems (no db in codfw) :( [12:53:21] the task being https://phabricator.wikimedia.org/T176532 [12:54:18] Project beta-code-update-eqiad build #238986: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/238986/ [12:55:36] Hm [12:56:06] Well it must be something serious [12:56:14] yes it is [12:56:58] Indeed [12:57:22] That's why I was expecting some sort of discussion somewhere, e.g. a security task or something like that [12:57:46] security task would only be if there's a security leak etc. But this seems like hardware. [12:58:09] Speculation isn't helpful [12:59:35] Speculation? [12:59:54] Well, I'm sure releng will let us know in due time [13:00:54] The "security task" was just an example but yes, I'm with hauskatze above [13:01:52] This means I have some time to expand a draft I am making about the quaestio perpetua de repetundis [13:04:04] Project mediawiki-core-doxygen-docker build #5459: 04STILL FAILING in 0.19 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/5459/ [13:04:18] Project beta-code-update-eqiad build #238987: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/238987/ [13:14:18] Project beta-code-update-eqiad build #238988: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/238988/ [13:24:17] Project beta-code-update-eqiad build #238989: 04STILL FAILING in 1 min 16 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/238989/ [13:32:39] 10Gerrit, 10Release-Engineering-Team, 10Operations: gerrit.wikimedia.org is down - https://phabricator.wikimedia.org/T218472 (10matmarex) SRE (Operations) know about the problem and are working on it right now, a few folks commented about it on IRC. It's unknown yet when it will be back up. [13:34:18] Project beta-code-update-eqiad build #238990: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/238990/ [13:44:18] Project beta-code-update-eqiad build #238991: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/238991/ [13:54:18] Project beta-code-update-eqiad build #238992: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/238992/ [14:04:04] Project mediawiki-core-doxygen-docker build #5460: 04STILL FAILING in 0.11 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/5460/ [14:04:18] Project beta-code-update-eqiad build #238993: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/238993/ [14:05:27] Reedy i apologise for speculating. [14:14:17] Project beta-code-update-eqiad build #238994: 04STILL FAILING in 1 min 16 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/238994/ [14:24:18] Project beta-code-update-eqiad build #238995: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/238995/ [14:34:17] Project beta-code-update-eqiad build #238996: 04STILL FAILING in 1 min 16 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/238996/ [14:44:18] Project beta-code-update-eqiad build #238997: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/238997/ [14:54:18] Project beta-code-update-eqiad build #238998: 04STILL FAILING in 1 min 16 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/238998/ [15:00:06] Project mediawiki-core-code-coverage-docker build #4128: 04FAILURE in 5.7 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-docker/4128/ [15:04:04] Project mediawiki-core-doxygen-docker build #5461: 04STILL FAILING in 0.25 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/5461/ [15:04:18] Project beta-code-update-eqiad build #238999: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/238999/ [15:14:17] Project beta-code-update-eqiad build #239000: 04STILL FAILING in 1 min 16 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/239000/ [15:24:17] Project beta-code-update-eqiad build #239001: 04STILL FAILING in 1 min 16 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/239001/ [15:29:38] 10Deployments, 10HHVM: mw conf cache is not properly invalidated - https://phabricator.wikimedia.org/T134448 (10demon) a:05demon→03None [15:34:17] Project beta-code-update-eqiad build #239002: 04STILL FAILING in 1 min 16 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/239002/ [15:44:18] Project beta-code-update-eqiad build #239003: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/239003/ [15:51:10] PROBLEM - Puppet staleness on deployment-db04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [15:54:18] Project beta-code-update-eqiad build #239004: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/239004/ [15:54:26] PROBLEM - Puppet errors on deployment-mx02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [3.0] [16:04:03] Project mediawiki-core-doxygen-docker build #5462: 04STILL FAILING in 0.13 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/5462/ [16:04:18] Project beta-code-update-eqiad build #239005: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/239005/ [16:14:18] Project beta-code-update-eqiad build #239006: 04STILL FAILING in 1 min 16 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/239006/ [16:23:28] 10Differential, 10Documentation: Document use of Owners in Phabricator and advertise it - https://phabricator.wikimedia.org/T128372 (10Aklapper) [16:24:18] Project beta-code-update-eqiad build #239007: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/239007/ [16:24:48] 10Differential, 10Documentation: Document use of Owners in Phabricator and advertise it - https://phabricator.wikimedia.org/T128372 (10Aklapper) [16:24:48] 10Phabricator (Upstream), 10Upstream: "Edit Paths" in Owners interface is unusable: Lists all repositories in an unsorted select box with no search - https://phabricator.wikimedia.org/T140713 (10Aklapper) 05Open→03Resolved >>! In T140713#5020738, @epriestley wrote: > Likely resolved by PROBLEM - Host integration-publishing02 is DOWN: CRITICAL - Host Unreachable (172.16.4.5) [16:29:18] 10Phabricator (Upstream), 10Upstream: Phabricator says Asia/Kolkata timezone is UTC +5 but it's not - https://phabricator.wikimedia.org/T185213 (10Aklapper) 05Open→03Resolved Thanks for the followup / ping! Confirming it's fixed on WM Phab: {F28398787} [16:30:34] 10Phabricator (Upstream), 10Upstream: Phabricator says Asia/Kolkata timezone is UTC +5 but it's not - https://phabricator.wikimedia.org/T185213 (10Krenair) @Aklapper: Hold on, isn't that screenshot saying that Kolkata is now *negative* 5:30? [16:34:18] Project beta-code-update-eqiad build #239008: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/239008/ [16:44:17] Project beta-code-update-eqiad build #239009: 04STILL FAILING in 1 min 16 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/239009/ [16:54:18] Project beta-code-update-eqiad build #239010: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/239010/ [17:04:04] Project mediawiki-core-doxygen-docker build #5463: 04STILL FAILING in 0.12 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/5463/ [17:04:18] Project beta-code-update-eqiad build #239011: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/239011/ [17:14:18] Project beta-code-update-eqiad build #239012: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/239012/ [17:24:18] Project beta-code-update-eqiad build #239013: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/239013/ [17:34:18] Project beta-code-update-eqiad build #239014: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/239014/ [17:44:18] Project beta-code-update-eqiad build #239015: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/239015/ [17:54:17] Project beta-code-update-eqiad build #239016: 04STILL FAILING in 1 min 16 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/239016/ [17:58:25] 10Gerrit, 10Release-Engineering-Team, 10Operations: Deploy multi-site plugin to cobalt and gerrit2001 - https://phabricator.wikimedia.org/T217174 (10Paladox) [17:58:31] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Patch-For-Review: Upgrade to Gerrit 2.16.6 - https://phabricator.wikimedia.org/T200739 (10Paladox) [17:59:52] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Patch-For-Review: Upgrade to Gerrit 2.16.7 - https://phabricator.wikimedia.org/T200739 (10Paladox) [18:00:48] 10Gerrit, 10Release-Engineering-Team, 10Operations: Deploy multi-site plugin to cobalt and gerrit2001 - https://phabricator.wikimedia.org/T217174 (10Paladox) [18:04:03] Project mediawiki-core-doxygen-docker build #5464: 04STILL FAILING in 0.1 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/5464/ [18:04:18] Project beta-code-update-eqiad build #239017: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/239017/ [18:14:18] Project beta-code-update-eqiad build #239018: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/239018/ [18:35:11] Project beta-code-update-eqiad build #239019: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/239019/ [18:42:12] 10Gerrit, 10Release-Engineering-Team, 10Operations: gerrit.wikimedia.org is down - https://phabricator.wikimedia.org/T218472 (10LucasWerkmeister) (The error appears to have changed from “connection refused” to “connection timed out” now, though that’s probably not very significant.) [18:42:49] 10Gerrit, 10Release-Engineering-Team, 10Operations: gerrit.wikimedia.org is down - https://phabricator.wikimedia.org/T218472 (10Paladox) Please see https://lists.wikimedia.org/pipermail/wikitech-l/2019-March/091744.html [18:47:22] Project beta-code-update-eqiad build #239020: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/239020/ [18:59:36] Project beta-code-update-eqiad build #239021: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/239021/ [19:06:14] Project mediawiki-core-doxygen-docker build #5465: 04STILL FAILING in 2 min 11 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/5465/ [19:11:46] Project beta-code-update-eqiad build #239022: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/239022/ [19:17:41] Project beta-code-update-eqiad build #239023: 04STILL FAILING in 5 min 55 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/239023/ [19:18:58] 10Gerrit, 10Release-Engineering-Team, 10Operations, 10User-greg: gerrit.wikimedia.org is down - https://phabricator.wikimedia.org/T218472 (10greg) 05Open→03Resolved a:03greg Gerrit is back, sorry for the interruption. [19:19:01] Yippee, build fixed! [19:19:01] Project beta-code-update-eqiad build #239024: 09FIXED in 1 min 19 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/239024/ [19:19:06] 10Gerrit, 10Release-Engineering-Team, 10Operations, 10User-greg: gerrit.wikimedia.org is down - https://phabricator.wikimedia.org/T218472 (10greg) a:05greg→03None [19:29:47] RECOVERY - puppet last run on contint2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:34:57] RECOVERY - puppet last run on contint1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:43:42] yay, Gerrit! Finally can continue work. Thanks everyone. :) [19:45:00] andre__: rOPUP refuses to upgrade on Phab though [19:45:04] ERR128 says [19:47:47] hauskatze: so gerrit isnt mirroring to phab on that repo? [19:48:12] Zppix: timeout errored due to the gerrit downtime - it'll be back to normal I guess [19:49:07] hauskatze: in a perfect world you would hope, ok ty [20:14:20] Yippee, build fixed! [20:14:20] Project mediawiki-core-doxygen-docker build #5466: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/5466/ [20:14:49] hauskatze, I pressed the update now button and eventually phabricator fully imported [20:17:23] Krenair: I did that a couple of times the error message was still there. Happy that this time it worked :) [20:21:17] yeah just needed patience [20:23:05] (and poke with a stick :) ) [20:23:38] it's entirely possible that our update now button presses did nothing [20:23:47] it does [20:24:00] but can take a few mins for it to process [20:24:26] and due to the sheer size of the puppet repo it could take longer to pull :) [20:25:18] it wasn't out of date by particularly many or large commits [20:25:27] paladox: but usually a 'prioritized' signal appears but not this time, anyway, it's working so I'll leave mystery to Agatha [20:25:46] oh, it didn't this time? [20:26:03] i guess because it failed during the gerrit outage? [21:05:25] PROBLEM - Host deployment-sessionstore01 is DOWN: CRITICAL - Host Unreachable (172.16.3.4) [21:10:39] (03PS1) 10Krinkle: zuul: Fix doc-publish postmerge of EventLogging [integration/config] - 10https://gerrit.wikimedia.org/r/497076 [21:11:40] James_F: ugh, ended up going to https://doc.wikimedia.org/mediawiki-extensions-EventLogging/master/ [21:11:54] I was wondering why it wasn't updating https://doc.wikimedia.org/EventLogging/master/ - I saw the job run. [21:17:12] Krinkle: Eww. [21:18:53] (03CR) 10Krinkle: [C: 04-1] "I agree keeping the job on test is useful. But having the test and postmerge use different node versions and triggers will end up being co" [integration/config] - 10https://gerrit.wikimedia.org/r/496887 (owner: 10Jforrester) [21:18:59] (03CR) 10Krinkle: [C: 03+2] zuul: Fix doc-publish postmerge of EventLogging [integration/config] - 10https://gerrit.wikimedia.org/r/497076 (owner: 10Krinkle) [21:18:59] Project beta-scap-eqiad build #241624: 04FAILURE in 9 min 12 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/241624/ [21:19:03] Let's see if this works now. [21:20:10] !log Removing doc1001:/srv/docroot/org/wikimedia/doc/mediawiki-extensions-EventLogging (created by accident) [21:20:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:20:40] (03Merged) 10jenkins-bot: zuul: Fix doc-publish postmerge of EventLogging [integration/config] - 10https://gerrit.wikimedia.org/r/497076 (owner: 10Krinkle) [21:20:57] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/497076 [21:20:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:21:21] !log krinkle@contint1001$ zuul enqueue --trigger gerrit --pipeline postmerge --project mediawiki/extensions/EventLogging --change 264494,4 [21:21:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:21:52] Krinkle: generic-node10-rundoc-docker works for me. [21:22:01] OK :) [21:22:12] I'll create the job, you update zuul? [21:22:31] Err. I can't push config to zuul. [21:22:47] Happy to re-write https://gerrit.wikimedia.org/r/496887 etc. though. :-) [21:24:09] Yeah, the zuul layout file that is [21:25:15] Where are unicodejs-node10-rundoc-docker and visualeditor-node10-rundoc-docker defined? I can't see them. [21:26:06] Oh, '{name}-node10-run{script}-docker' [21:26:24] Why do we have unicodejs and visualeditor not just use the generic? [21:26:32] (Other than that it doesn't exist. ;-)) [21:27:49] DebianJessieDocker not DebianStretchDocker? I thought our node10 images were based off stretch? (Also, when are be bumping over to buster?) [21:29:17] (03PS1) 10Krinkle: Create generic- and mwext- variants of node10-rundoc job [integration/config] - 10https://gerrit.wikimedia.org/r/497077 [21:29:24] James_F: Indeed. [21:29:47] I was thinking the same thing :) [21:30:04] Yippee, build fixed! [21:30:04] Project beta-scap-eqiad build #241625: 09FIXED in 9 min 44 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/241625/ [21:30:08] James_F: The 'node' label refers to the host, not the container. [21:30:25] The docker slaves are jessie, I don't know why but doesn't matter for the container which is indeed stretch-based for node10 [21:30:53] I assume that RelEng will switch those over based on available workers etc. not touching that :) [21:31:10] Oh, right. [21:31:46] So we'll be using mwext-node10-rundoc-docker but generic-node10-docs-docker-publish (because we don't mind about generic and mwext clashing with each other as postmerge isn't time-critical)? [21:32:04] Except postmerge is time-critical for docker pipeline images, but I guess they have high priority? [21:32:53] I don't know about the latter. But indeed postmerge there is no blocked queue. that's only in the gate [21:32:57] they can run concurrently [21:33:09] hence no split needed. [21:33:14] ugh, EventLogging is still not right. [21:33:18] (03PS2) 10Jforrester: [Flow] Replace *-jsduck-* jobs with *-node10-docs-* ones [integration/config] - 10https://gerrit.wikimedia.org/r/496887 [21:33:33] It's now publishing to doc.wm.o/EventLogging but to doc.wm.o/EventLogging/master not doc.wm.o/EventLogging/master/js, which is where it used to be. [21:33:46] We use the /js for mwext (not for libs) because they contain both PHP and JS code. [21:33:53] Yeah, just append /js. [21:33:58] "just" [21:34:02] (03PS3) 10Jforrester: [Flow] Replace *-jsduck-* jobs with *-node10-docs-* ones [integration/config] - 10https://gerrit.wikimedia.org/r/496887 [21:34:12] It breaks the model. [21:34:16] I'll need to fiddle with it [21:34:58] .. yeah, won't work with the generic one. Okay, we'll have mwext for publish as well then. [21:35:29] (03CR) 10jerkins-bot: [V: 04-1] [Flow] Replace *-jsduck-* jobs with *-node10-docs-* ones [integration/config] - 10https://gerrit.wikimedia.org/r/496887 (owner: 10Jforrester) [21:37:16] (03CR) 10Legoktm: [C: 03+2] [Disambiguator] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/496507 (owner: 10Umherirrender) [21:37:41] (03CR) 10Legoktm: [C: 03+2] [Josa] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/496524 (owner: 10Umherirrender) [21:38:16] (03CR) 10Legoktm: [C: 03+2] [GeoCrumbs] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/496520 (owner: 10Umherirrender) [21:38:45] (03CR) 10Legoktm: [C: 03+2] [Insider] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/496516 (owner: 10Umherirrender) [21:39:09] (03Merged) 10jenkins-bot: [Disambiguator] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/496507 (owner: 10Umherirrender) [21:39:23] (03CR) 10Legoktm: [C: 03+2] [Listings] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/496510 (owner: 10Umherirrender) [21:39:27] (03Merged) 10jenkins-bot: [Josa] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/496524 (owner: 10Umherirrender) [21:39:46] (03Merged) 10jenkins-bot: [GeoCrumbs] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/496520 (owner: 10Umherirrender) [21:40:20] (03Merged) 10jenkins-bot: [Insider] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/496516 (owner: 10Umherirrender) [21:40:27] (03PS2) 10Krinkle: Create generic and mwext variants of node10 doc-related jobs [integration/config] - 10https://gerrit.wikimedia.org/r/497077 [21:40:51] (03Merged) 10jenkins-bot: [Listings] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/496510 (owner: 10Umherirrender) [21:41:26] !log deploying https://gerrit.wikimedia.org/r/496507 https://gerrit.wikimedia.org/r/496524 https://gerrit.wikimedia.org/r/496520 https://gerrit.wikimedia.org/r/496516 https://gerrit.wikimedia.org/r/496510 [21:41:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:46:22] (03CR) 10Krinkle: [C: 03+2] "Deployed generic-node10-rundoc-docker, mwext-node10-docs-docker-publish, mwext-node10-rundoc-docker" [integration/config] - 10https://gerrit.wikimedia.org/r/497077 (owner: 10Krinkle) [21:48:06] ! [remote rejected] HEAD -> refs/for/master%topic=496894 (the number of pushed changes in a batch exceeds the max limit 10) [21:48:08] * James_F sighs. [21:48:16] Oh, and we need to replace anyway. [21:48:36] James_F: now don't do as I did once and push merge :P [21:48:38] (03Merged) 10jenkins-bot: Create generic and mwext variants of node10 doc-related jobs [integration/config] - 10https://gerrit.wikimedia.org/r/497077 (owner: 10Krinkle) [21:48:50] indeed. [21:48:53] annoying limit though [21:49:28] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/497077 [21:49:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:49:31] should be higher - phpcs fixes often involve more than 10 connected patches. Good thing is that most extensions are already phpcs-ized [21:50:20] Yay, https://doc.wikimedia.org/EventLogging/master/js/ is working now [21:50:56] (03PS4) 10Jforrester: Replace *-jsduck-* jobs with *-node10-docs-* ones [integration/config] - 10https://gerrit.wikimedia.org/r/496887 [21:50:58] (03PS2) 10Jforrester: Drop mwext-jsduck-publish, no longer used [integration/config] - 10https://gerrit.wikimedia.org/r/496894 [21:51:00] (03PS1) 10Jforrester: Drop extension-jsduck, no longer used [integration/config] - 10https://gerrit.wikimedia.org/r/497079 [21:51:55] I just squashed them all together. [21:51:59] James_F: cool, yeah. [21:52:00] Hmm, something's missing. [21:52:04] James_F: btw, need to define the tempalte. [21:52:16] (maybe a shorter name, too) [21:52:22] (03CR) 10jerkins-bot: [V: 04-1] Replace *-jsduck-* jobs with *-node10-docs-* ones [integration/config] - 10https://gerrit.wikimedia.org/r/496887 (owner: 10Jforrester) [21:53:01] Krinkle: oh, yeah. After my patch the only jsduck references are mediawiki-core-jsduck-*, which I'm not touching given the magic. [21:53:24] (03Abandoned) 10Jforrester: [Kartographer] Replace mwext-jsduck-publish with generic-node10-docs-docker-publish [integration/config] - 10https://gerrit.wikimedia.org/r/496888 (owner: 10Jforrester) [21:53:29] (03Abandoned) 10Jforrester: [TemplateData] Replace mwext-jsduck-publish with generic-node10-docs-docker-publish [integration/config] - 10https://gerrit.wikimedia.org/r/496889 (owner: 10Jforrester) [21:53:31] (03Abandoned) 10Jforrester: [CollabKit] Replace mwext-jsduck-publish with generic-node10-docs-docker-publish [integration/config] - 10https://gerrit.wikimedia.org/r/496890 (owner: 10Jforrester) [21:53:34] (03Abandoned) 10Jforrester: [MultimediaViewer] Replace mwext-jsduck-publish with generic-node10-docs-docker-publish [integration/config] - 10https://gerrit.wikimedia.org/r/496891 (owner: 10Jforrester) [21:53:37] (03Abandoned) 10Jforrester: [GuidedTour] Replace mwext-jsduck-publish with generic-node10-docs-docker-publish [integration/config] - 10https://gerrit.wikimedia.org/r/496892 (owner: 10Jforrester) [21:53:42] (03Abandoned) 10Jforrester: [Wikibase] Replace mwext-jsduck-publish with generic-node10-docs-docker-publish [integration/config] - 10https://gerrit.wikimedia.org/r/496893 (owner: 10Jforrester) [21:53:56] Krinkle: Can you do a template for doc-test and another for doc-test-and-publish? [21:54:10] James_F: Do we have repos tesitng with jsduck but not publishing? [21:54:11] (03CR) 10jerkins-bot: [V: 04-1] Drop extension-jsduck, no longer used [integration/config] - 10https://gerrit.wikimedia.org/r/497079 (owner: 10Jforrester) [21:54:17] Yes. [21:54:26] (03CR) 10jerkins-bot: [V: 04-1] Drop mwext-jsduck-publish, no longer used [integration/config] - 10https://gerrit.wikimedia.org/r/496894 (owner: 10Jforrester) [21:54:30] Hm.. which one? [21:54:38] 10Continuous-Integration-Config, 10Lexicographical data, 10Wikidata, 10MW-1.33-notes (1.33.0-wmf.21; 2019-03-12), and 4 others: Enable phan checks for WikibaseLexeme extension - https://phabricator.wikimedia.org/T215556 (10Umherirrender) 05Open→03Resolved [21:55:00] James_F: but yeah, that seems fine. Can replace the extension-jsduck template. [21:55:00] OOJsUIAjaxLogin. Maybe we should just publish it (or scrap that extension, but…). [21:55:16] Yeah, I'd say publish it for now. Seems like accidental, not intentional. [21:56:17] !log krinkle@contint1001:~$ zuul enqueue --trigger gerrit --pipeline postmerge --project mediawiki/extensions/OOJsUIAjaxLogin --change 490979,1 [21:56:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:56:19] Let's see if it passes. [22:01:26] yep, passes. [22:03:22] (03PS5) 10Jforrester: Replace *-jsduck-* jobs with *-node10-docs-* ones [integration/config] - 10https://gerrit.wikimedia.org/r/496887 [22:03:24] (03PS3) 10Jforrester: Drop mwext-jsduck-publish, no longer used [integration/config] - 10https://gerrit.wikimedia.org/r/496894 [22:03:26] (03PS2) 10Jforrester: Drop extension-jsduck, no longer used [integration/config] - 10https://gerrit.wikimedia.org/r/497079 [22:03:28] (03PS1) 10Jforrester: Provide extension-javascript-documentation template [integration/config] - 10https://gerrit.wikimedia.org/r/497080 [22:08:24] Krinkle: Is ^ what you were thinking of? [22:08:39] yeah, LGTM. will land once Jenkins V+2's [22:08:48] It has. :-) [22:12:11] James_F: hm.. can we enable VE for the "Obsolete:" namespace on wikitech? Seems trivial right? [22:12:33] Krinkle: It's trivial, yes. One second. [22:12:42] Seems it has to "Switch to visual editing" button, either. Cool, thanks :) [22:12:52] Well, that's because wikitech isn't using RB. [22:13:01] * Krinkle is looking at Squid stuff, gonna tackle that doc issue. [22:13:08] (03CR) 10Krinkle: [C: 03+2] Provide extension-javascript-documentation template [integration/config] - 10https://gerrit.wikimedia.org/r/497080 (owner: 10Jforrester) [22:13:12] Until wikitechwiki becomes a 'real' wiki, you can't go from mid-edit into visual mode. [22:14:29] (03CR) 10Krinkle: "-INFO:zuul.IndependentPipelineManager: ^.*\.(js|json|css)$" [integration/config] - 10https://gerrit.wikimedia.org/r/496887 (owner: 10Jforrester) [22:14:42] (03CR) 10Krinkle: [C: 03+1] Drop mwext-jsduck-publish, no longer used [integration/config] - 10https://gerrit.wikimedia.org/r/496894 (owner: 10Jforrester) [22:14:45] (03CR) 10Krinkle: [C: 03+1] Drop extension-jsduck, no longer used [integration/config] - 10https://gerrit.wikimedia.org/r/497079 (owner: 10Jforrester) [22:15:01] veaction=edit works :) [22:15:05] (03Merged) 10jenkins-bot: Provide extension-javascript-documentation template [integration/config] - 10https://gerrit.wikimedia.org/r/497080 (owner: 10Jforrester) [22:15:40] Krinkle: But discards edits. [22:16:27] I can save. [22:16:32] But yeah not switch. That's fine though. [22:16:36] * James_F nods. [22:16:41] https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/497081 [22:16:42] https://wikitech.wikimedia.org/w/index.php?title=Obsolete:Squids&action=history [22:17:24] Anyway, I should leave the office, as it's 15:15. :-) [22:17:26] * James_F waves. [22:17:38] o/ [22:43:10] (03PS6) 10Krinkle: Replace *-jsduck-* jobs with *-node10-docs-* ones [integration/config] - 10https://gerrit.wikimedia.org/r/496887 (owner: 10Jforrester) [22:43:47] (03CR) 10Krinkle: "updated to 1) remove the now-unused file-type filter (for 'gate' pipeline), and 2) update the branch filter for postmerge, that one we sho" [integration/config] - 10https://gerrit.wikimedia.org/r/496887 (owner: 10Jforrester) [23:24:19] (03PS7) 10Krinkle: Replace *-jsduck-* jobs with *-node10-docs-* ones [integration/config] - 10https://gerrit.wikimedia.org/r/496887 (owner: 10Jforrester) [23:56:29] (03CR) 10Krinkle: [C: 04-1] Drop mwext-jsduck-publish, no longer used (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/496894 (owner: 10Jforrester)