[12:44:15] 10Beta-Cluster-Infrastructure, 10Operations: Upgrade puppet in deployment-prep - https://phabricator.wikimedia.org/T243226 (10Krenair) Swapped the list in hieradata for deployment-puppetmaster03 around. Once I'm happy it's working I'll remove the old puppetdb from the list and disable command_broadcast. On th... [12:46:31] 10Phabricator, 10Pywikibot: Herald rule for adding Upstream when a task is moved to "Upstream issues" column of pywikibot-core workboard - https://phabricator.wikimedia.org/T194009 (10MarcoAurelio) This can now be done using the Triggers functionality, but you'll need to agree on which tag you'd like Phabricat... [13:07:33] 10Beta-Cluster-Infrastructure, 10Operations: Upgrade puppet in deployment-prep - https://phabricator.wikimedia.org/T243226 (10jijiki) @Krenair I have been experimenting with this host, feel free to enable puppet if needed, sorry for the inconvenience:) [13:29:15] 10Beta-Cluster-Infrastructure, 10Operations: Upgrade puppet in deployment-prep - https://phabricator.wikimedia.org/T243226 (10Krenair) No problem, just trying to avoid overwriting other's work. Am also aware (including having done it myself a few times) sometimes people just forget to re-enable it when done. W... [13:36:38] Project beta-scap-eqiad build #286228: 04FAILURE in 2 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286228/ [13:46:37] Project beta-scap-eqiad build #286229: 04STILL FAILING in 2 min 7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286229/ [13:56:36] Project beta-scap-eqiad build #286230: 04STILL FAILING in 2 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286230/ [14:06:51] Project beta-scap-eqiad build #286231: 04STILL FAILING in 2 min 20 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286231/ [14:16:42] Project beta-scap-eqiad build #286232: 04STILL FAILING in 2 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286232/ [14:26:40] Project beta-scap-eqiad build #286233: 04STILL FAILING in 2 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286233/ [14:36:39] Project beta-scap-eqiad build #286234: 04STILL FAILING in 2 min 7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286234/ [14:46:41] Project beta-scap-eqiad build #286235: 04STILL FAILING in 2 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286235/ [14:56:39] Project beta-scap-eqiad build #286236: 04STILL FAILING in 2 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286236/ [15:06:54] Project beta-scap-eqiad build #286237: 04STILL FAILING in 2 min 23 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286237/ [15:07:54] Krenair: Hi. I assume this is mediawiki07 puppet failing? [15:08:04] the failures reported above I meant [15:08:42] "deployment-mediawiki-07.deployment-prep.eqiad.wmflabs returned [255]: Host key verification failed." [15:16:44] Project beta-scap-eqiad build #286238: 04STILL FAILING in 2 min 13 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286238/ [15:26:36] Project beta-scap-eqiad build #286239: 04STILL FAILING in 2 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286239/ [15:29:59] 10Beta-Cluster-Infrastructure, 10Operations: Upgrade puppet in deployment-prep - https://phabricator.wikimedia.org/T243226 (10jijiki) @Krenair thank you, I already have a backup of the configuration, so there is no need to let this disrupt you any further [15:36:48] Project beta-scap-eqiad build #286240: 04STILL FAILING in 2 min 16 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286240/ [15:38:17] 10Beta-Cluster-Infrastructure, 10Operations: Upgrade puppet in deployment-prep - https://phabricator.wikimedia.org/T243226 (10Peachey88) [15:46:39] Project beta-scap-eqiad build #286241: 04STILL FAILING in 2 min 7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286241/ [15:56:39] Project beta-scap-eqiad build #286242: 04STILL FAILING in 2 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286242/ [16:02:31] (03CR) 10DCausse: [V: 03+2 C: 03+2] Grant `wikidata-query` people more access in wikidata/query/rdf.git [wikidata/query/rdf] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/569281 (https://phabricator.wikimedia.org/T243431) (owner: 10MarcoAurelio) [16:06:53] Project beta-scap-eqiad build #286243: 04STILL FAILING in 2 min 20 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286243/ [16:16:52] Project beta-scap-eqiad build #286244: 04STILL FAILING in 2 min 18 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286244/ [16:26:42] Project beta-scap-eqiad build #286245: 04STILL FAILING in 2 min 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286245/ [16:36:43] Project beta-scap-eqiad build #286246: 04STILL FAILING in 2 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286246/ [16:46:42] Project beta-scap-eqiad build #286247: 04STILL FAILING in 2 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286247/ [16:54:49] 10Beta-Cluster-Infrastructure, 10Browser-Support-Firefox: en.wikipedia.beta.wmflabs.org not accessible in Firefox, due to something with OCSP - https://phabricator.wikimedia.org/T243881 (10Kaartic) > I can't reproduce this in Firefox 72.0.2 on Windows 10 I have to take that back. I'm starting to get the error... [16:56:36] Project beta-scap-eqiad build #286248: 04STILL FAILING in 2 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286248/ [17:00:56] 10Beta-Cluster-Infrastructure, 10Browser-Support-Firefox: en.wikipedia.beta.wmflabs.org not accessible in Firefox, due to something with OCSP - https://phabricator.wikimedia.org/T243881 (10Krenair) a:03Krenair I'll handle this in the next few days if no one else does. A short term fix might just be copying O... [17:06:46] Project beta-scap-eqiad build #286249: 04STILL FAILING in 2 min 14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286249/ [17:16:37] Project beta-scap-eqiad build #286250: 04STILL FAILING in 2 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286250/ [17:26:38] Project beta-scap-eqiad build #286251: 04STILL FAILING in 2 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286251/ [17:36:39] Project beta-scap-eqiad build #286252: 04STILL FAILING in 2 min 7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286252/ [17:41:21] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-09 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:46:46] Project beta-scap-eqiad build #286253: 04STILL FAILING in 2 min 14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286253/ [17:47:52] PROBLEM - Parsoid on deployment-parsoid09 is CRITICAL: connect to address 172.16.5.63 and port 8000: Connection refused [17:47:52] PROBLEM - Parsoid on deployment-mediawiki-parsoid10 is CRITICAL: connect to address 172.16.0.141 and port 8000: Connection refused [17:56:39] Project beta-scap-eqiad build #286254: 04STILL FAILING in 2 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286254/ [18:07:01] Project beta-scap-eqiad build #286255: 04STILL FAILING in 2 min 29 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286255/ [18:16:46] Project beta-scap-eqiad build #286256: 04STILL FAILING in 2 min 14 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286256/ [18:21:11] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-09 is OK: HTTP OK: HTTP/1.1 200 OK - 48537 bytes in 0.658 second response time [18:26:39] Project beta-scap-eqiad build #286257: 04STILL FAILING in 2 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286257/ [18:36:44] Project beta-scap-eqiad build #286258: 04STILL FAILING in 2 min 13 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286258/ [18:39:55] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [150.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [18:46:35] Project beta-scap-eqiad build #286259: 04STILL FAILING in 2 min 3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286259/ [18:56:42] Project beta-scap-eqiad build #286260: 04STILL FAILING in 2 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286260/ [19:06:57] Project beta-scap-eqiad build #286261: 04STILL FAILING in 2 min 26 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286261/ [19:16:40] Project beta-scap-eqiad build #286262: 04STILL FAILING in 2 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286262/ [19:26:29] Project beta-scap-eqiad build #286263: 04STILL FAILING in 2 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286263/ [19:36:40] Project beta-scap-eqiad build #286264: 04STILL FAILING in 2 min 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286264/ [19:46:42] Project beta-scap-eqiad build #286265: 04STILL FAILING in 2 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286265/ [19:53:44] 10Phabricator, 10Pywikibot: Phabricator Trigger to add #Upstream tag when a task is moved to "Upstream issues" column of #pywikibot workboard - https://phabricator.wikimedia.org/T194009 (10Aklapper) 05Stalled→03Open [19:56:37] Project beta-scap-eqiad build #286266: 04STILL FAILING in 2 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286266/ [20:06:48] Project beta-scap-eqiad build #286267: 04STILL FAILING in 2 min 18 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286267/ [20:16:46] Project beta-scap-eqiad build #286268: 04STILL FAILING in 2 min 12 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286268/ [20:26:40] Project beta-scap-eqiad build #286269: 04STILL FAILING in 2 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286269/ [20:36:48] Project beta-scap-eqiad build #286270: 04STILL FAILING in 2 min 16 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286270/ [20:46:37] Project beta-scap-eqiad build #286271: 04STILL FAILING in 2 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286271/ [20:56:32] Project beta-scap-eqiad build #286272: 04STILL FAILING in 2 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286272/ [21:06:50] Project beta-scap-eqiad build #286273: 04STILL FAILING in 2 min 24 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286273/ [21:16:37] Project beta-scap-eqiad build #286274: 04STILL FAILING in 2 min 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286274/ [21:26:36] Project beta-scap-eqiad build #286275: 04STILL FAILING in 2 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286275/ [21:31:44] 10Release-Engineering-Team, 10Gerrit-Privilege-Requests, 10Wikidata, 10Wikidata-Query-Service, 10Patch-For-Review: Grant more rights to wikidata/query/rdf for the group wikidata-query (similar to search) - https://phabricator.wikimedia.org/T243431 (10MarcoAurelio) 05Open→03Resolved Patch merged so th... [21:33:59] @seen qchris [21:36:36] Project beta-scap-eqiad build #286276: 04STILL FAILING in 2 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286276/ [21:46:41] Project beta-scap-eqiad build #286277: 04STILL FAILING in 2 min 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286277/ [21:56:41] Project beta-scap-eqiad build #286278: 04STILL FAILING in 2 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286278/ [22:06:45] Project beta-scap-eqiad build #286279: 04STILL FAILING in 2 min 15 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286279/ [22:09:58] maybe we should quiet ^ until the puppet upgrade for beta cluster is done? [22:16:28] why puppet upgrade? [22:16:45] Project beta-scap-eqiad build #286280: 04STILL FAILING in 2 min 13 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286280/ [22:16:46] 22:06:12 ['/usr/bin/scap', 'pull', '--no-update-l10n', 'deployment-deploy01.deployment-prep.eqiad.wmflabs', 'deployment-deploy02.deployment-prep.eqiad.wmflabs', 'deployment-deploy01.deployment-prep.eqiad.wmflabs'] on deployment-mediawiki-07.deployment-prep.eqiad.wmflabs returned [255]: Host key verification failed. [22:20:51] paladox|UKInEU: puppet 4 has bugs, I was told; and is being upgraded [22:21:04] ok [22:21:11] see T243226 [22:21:13] T243226: Upgrade puppet in deployment-prep - https://phabricator.wikimedia.org/T243226 [22:21:46] and I guess this is puppet because mediawiki-07 was the host that had puppet down and now it's failing? [22:26:39] Project beta-scap-eqiad build #286281: 04STILL FAILING in 2 min 7 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286281/ [22:36:36] Project beta-scap-eqiad build #286282: 04STILL FAILING in 2 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286282/ [22:46:36] Project beta-scap-eqiad build #286283: 04STILL FAILING in 2 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286283/ [22:56:34] Project beta-scap-eqiad build #286284: 04STILL FAILING in 2 min 3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286284/ [23:06:52] Project beta-scap-eqiad build #286285: 04STILL FAILING in 2 min 22 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286285/ [23:16:39] Project beta-scap-eqiad build #286286: 04STILL FAILING in 2 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286286/ [23:26:32] Project beta-scap-eqiad build #286287: 04STILL FAILING in 2 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286287/ [23:36:40] Project beta-scap-eqiad build #286288: 04STILL FAILING in 2 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286288/ [23:46:32] Project beta-scap-eqiad build #286289: 04STILL FAILING in 2 min 3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286289/ [23:56:27] Project beta-scap-eqiad build #286290: 04STILL FAILING in 1 min 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286290/