[06:05:15] (03CR) 10Ori.livneh: [C: 03+2] Add the part for rebuilding array l10n cache [tools/scap] - 10https://gerrit.wikimedia.org/r/558243 (https://phabricator.wikimedia.org/T105683) (owner: 10Ladsgroup) [06:06:10] (03CR) 10Ori.livneh: [V: 03+1 C: 03+1] Add the part for rebuilding array l10n cache [tools/scap] - 10https://gerrit.wikimedia.org/r/558243 (https://phabricator.wikimedia.org/T105683) (owner: 10Ladsgroup) [06:08:30] (03CR) 10Ammarpad: [C: 03+1] "Looks good." [integration/config] - 10https://gerrit.wikimedia.org/r/560078 (owner: 10Minhducsun2002) [07:29:59] 10Gerrit, 10Operations, 10serviceops, 10Patch-For-Review: Convert Gerrit to use H2 as the database - https://phabricator.wikimedia.org/T211139 (10Peachey88) >>! In T211139#5762926, @Paladox wrote: >>>! In T211139#4798560, @Dzahn wrote: >> On one hand i would love this because it would make the gerrit codfw... [16:14:51] (03CR) 10Ori.livneh: [V: 03+1 C: 03+1] "Note that this depends on I64ef1812e getting cherry-picked to all production branches." [tools/scap] - 10https://gerrit.wikimedia.org/r/558243 (https://phabricator.wikimedia.org/T105683) (owner: 10Ladsgroup) [17:47:53] PROBLEM - Parsoid on deployment-parsoid09 is CRITICAL: connect to address 172.16.5.63 and port 8000: Connection refused [17:47:53] PROBLEM - Parsoid on deployment-mediawiki-parsoid10 is CRITICAL: connect to address 172.16.0.141 and port 8000: Connection refused [20:17:19] 10Beta-Cluster-Infrastructure, 10GlobalRename, 10MediaWiki-extensions-CentralAuth, 10User-DannyS712: Global renames aren't being processed on beta cluster - https://phabricator.wikimedia.org/T241294 (10MarcoAurelio) For debug: All events from 21 to 22 December 2019. Bunch of cpjobqueue errors: https://logs... [20:28:58] Is it posible to check if there are jobs awaiting execution for kafka on beta cluster? [20:50:56] 10Beta-Cluster-Infrastructure, 10ChangeProp, 10WMF-JobQueue: Job queue broken on Beta Cluster - https://phabricator.wikimedia.org/T241448 (10MarcoAurelio) [20:51:57] 10Beta-Cluster-Infrastructure, 10ChangeProp, 10WMF-JobQueue: Job queue broken on Beta Cluster - https://phabricator.wikimedia.org/T241448 (10MarcoAurelio) p:05Triage→03Unbreak! JobQueue as an essential feature being broken defeats the purpose of Beta Cluster as a testing platform to detect other issues b... [20:53:43] 10Beta-Cluster-Infrastructure, 10GlobalRename, 10MediaWiki-extensions-CentralAuth, 10User-DannyS712: Global renames aren't being processed on beta cluster - https://phabricator.wikimedia.org/T241294 (10MarcoAurelio) [20:53:45] 10Beta-Cluster-Infrastructure, 10ChangeProp, 10WMF-JobQueue: Job queue broken on Beta Cluster - https://phabricator.wikimedia.org/T241448 (10MarcoAurelio) [20:58:37] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10ChangeProp, 10WMF-JobQueue: Job queue broken on Beta Cluster - https://phabricator.wikimedia.org/T241448 (10MarcoAurelio) [21:02:36] 10Beta-Cluster-Infrastructure, 10GlobalRename, 10MediaWiki-extensions-CentralAuth, 10User-DannyS712: Global renames aren't being processed on beta cluster - https://phabricator.wikimedia.org/T241294 (10MarcoAurelio) I tried to unblock ~riley's rename using the script as well, but I get the same errors as i... [21:08:00] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10ChangeProp, 10WMF-JobQueue: Job queue broken on Beta Cluster - https://phabricator.wikimedia.org/T241448 (10Pchelolo) I think I fixed it. Same problem as seen with beta RESTBase. Root cause: T241263 [21:10:17] Pchelolo: hi, thanks for taking a look at ^ [21:13:42] looks like it keeps erroring with Error sending hot-shots message: Error: getaddrinfo ENOTFOUND labmon1001.eqiad.wmnet [21:17:35] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10ChangeProp, 10WMF-JobQueue: Job queue broken on Beta Cluster - https://phabricator.wikimedia.org/T241448 (10MarcoAurelio) @Pchelolo Thanks. Looking at Logstash feed it looks it keeps failing with "Error sending hot-shots message: Error: getaddrin... [23:52:52] (03CR) 10Ladsgroup: "> Patch Set 8:" [tools/scap] - 10https://gerrit.wikimedia.org/r/558243 (https://phabricator.wikimedia.org/T105683) (owner: 10Ladsgroup)