[02:53:35] 10Phabricator: Pastes on Phabricator are totally white in Dark mode - https://phabricator.wikimedia.org/T241452 (10Masumrezarock100) [02:56:06] 10Phabricator: Pastes on Phabricator are totally white in Dark mode - https://phabricator.wikimedia.org/T241452 (10Masumrezarock100) [03:36:28] 10Phabricator (Upstream), 10Upstream: Pastes on Phabricator are totally white in Dark mode - https://phabricator.wikimedia.org/T241452 (10Peachey88) Please file a task in the upstream [[ https://discourse.phabricator-community.org/ | phabricator discourse ]]. [05:55:06] 10Phabricator (Upstream), 10Upstream: Pastes on Phabricator are totally white in Dark mode - https://phabricator.wikimedia.org/T241452 (10Ammarpad) >>! In T241452#5763533, @Peachey88 wrote: > Please file a task in the upstream [[ https://discourse.phabricator-community.org/ | phabricator discourse ]]. Note, th... [06:22:26] (03CR) 10DannyS712: [C: 03+1] "Looks good" [integration/config] - 10https://gerrit.wikimedia.org/r/560049 (https://phabricator.wikimedia.org/T235286) (owner: 10N3rsti) [06:31:39] 10Phabricator (Upstream), 10Upstream: Pastes on Phabricator are totally white in Dark mode - https://phabricator.wikimedia.org/T241452 (10Masumrezarock100) >There’s very little customer interest in this mode, it doesn’t work well now, it’s a relatively high-cost feature to maintain, so it’s likely that it will... [10:51:26] 10Phabricator (Upstream), 10Upstream: Pastes on Phabricator are totally white in Dark mode - https://phabricator.wikimedia.org/T241452 (10Aklapper) p:05Triage→03Lowest [11:20:27] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Scap, 10Operations, 10serviceops: On beta, scap can't clear opcache on some mw servers - https://phabricator.wikimedia.org/T237033 (10jcrespo) @hashar based on Dzhan's comment, is that something your team could handle, sending a puppet patch fo... [12:08:03] !log projectadmin on deployment-prep for MarcoAurelio see discussion at https://gerrit.wikimedia.org/r/560607 [12:08:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:21:17] Krenair: keeps erroring: https://logstash-beta.wmflabs.org/goto/d908b9903e08a9a016bede316b5fb0e8 [12:21:32] do we need to run puppet-agent? [12:22:39] hauskatze, puppet-agent will need to have run since you updated horizon for any change to happen, yes [12:22:47] it runs automatically every... 30 minutes or so IIRC [12:23:18] will do it on -mediawiki-parsoid10 now [12:23:40] Notice: Run of Puppet configuration client already in progress; skipping (/var/lib/puppet/state/agent_catalog_run.lock exists) [12:24:54] I was trying to login :) [12:26:39] looks like it finished but maybe didn't fix the thing [12:27:34] krenair@deployment-mediawiki-parsoid10:~$ sudo service parsoid status [12:27:34] ● parsoid.service - "parsoid service" [12:27:34] Loaded: loaded (/lib/systemd/system/parsoid.service; enabled; vendor preset: enabled) [12:27:34] Active: active (running) since Mon 2019-12-16 21:46:14 UTC; 1 weeks 2 days ago [12:27:41] but then... [12:27:42] krenair@deployment-mediawiki-parsoid10:~$ sudo service parsoid reload [12:27:42] parsoid: unrecognized service [12:27:52] maybe that's fine and it just doesn't have a reload function [12:28:15] restarted it [12:28:15] Active: active (running) since Thu 2019-12-26 12:27:58 UTC; 2s ago [12:28:52] hauskatze, errors should go down now? [12:29:22] Krenair: I think we've fixed on parsoid10, so I'll move to cpjobqueue now [12:30:00] "startup finished" [12:30:09] yup, parsoid10 looks fixed [12:30:37] noticed a weird thing in -cpjobqueue syslog [12:30:41] Dec 26 12:29:51 deployment-cpjobqueue rsyslogd: could not load module 'lmnsd_tcp', errors: trying to load module /usr/lib/x86_64-linux-gnu/rsyslog/lmnsd_tcp.so: /usr/lib/x86_64-linux-gnu/rsyslog/lmnsd_tcp.so: cannot open shared object file: No such file or directory [v8.1901.0 try https://www.rsyslog.com/e/2066 ] [12:31:25] no more errors for parsoid10; last entries in logstash are CentralAuthVerbose [12:38:52] I'll see if I can find where 'changeprop' is erroring later [12:39:19] Looks to me it's eventlogging_statsd_host: labmon1001.eqiad.wmnet [12:39:44] but that's for the project puppet, not just deployment-cpjobqueue [12:39:55] yeah [12:40:26] while all instances will get it from project puppet hieradata, not all will necessarily use it [12:40:46] it's possible cpjobqueue uses that key and the others don't? [12:40:52] gotta go, good luck [12:41:20] gotta go too, family lunch (hate hate hate) [15:10:24] 10Gerrit: `mediawiki-replication` is a non-existing Gerrit group present in All-Projects.git - https://phabricator.wikimedia.org/T241461 (10MarcoAurelio) [15:18:55] 10Beta-Cluster-Infrastructure: Update references from labmon.* to cloudmetrics.* on deployment-prep (beta cluster) - https://phabricator.wikimedia.org/T241462 (10MarcoAurelio) [15:20:54] 10Beta-Cluster-Infrastructure: Update references from labmon.* to cloudmetrics.* on deployment-prep (beta cluster) - https://phabricator.wikimedia.org/T241462 (10MarcoAurelio) [15:21:12] 10Beta-Cluster-Infrastructure: Update references from labmon.* to cloudmetrics.* on deployment-prep (beta cluster) - https://phabricator.wikimedia.org/T241462 (10MarcoAurelio) p:05Triage→03Normal [15:22:47] 10Beta-Cluster-Infrastructure: Update references from labmon.* to cloudmetrics.* on deployment-prep (beta cluster) - https://phabricator.wikimedia.org/T241462 (10MarcoAurelio) Fixed mediawiki-parsoid10 via c12ee9cbb897. [15:49:06] 10Gerrit: `mediawiki-replication` is a non-existing Gerrit group present in All-Projects.git - https://phabricator.wikimedia.org/T241461 (10Paladox) The group is hidden to non project members thus you carn't see it. [15:54:43] 10Gerrit: `mediawiki-replication` is a non-existing Gerrit group present in All-Projects.git - https://phabricator.wikimedia.org/T241461 (10Krenair) Why do we have any hidden groups? [16:02:04] 10Gerrit: `mediawiki-replication` is a non-existing Gerrit group present in All-Projects.git - https://phabricator.wikimedia.org/T241461 (10Paladox) No one checked "Make group visible to all registered users". I'm not sure if it was intentional though. [16:05:16] maurelio@deployment-changeprop:~$ sudo service puppet status [16:05:16] ● puppet.service - Puppet agent [16:05:17] Loaded: loaded (/lib/systemd/system/puppet.service; disabled) [16:05:17] Active: inactive (dead) [16:05:23] great [16:05:41] hauskatze that should be intended [16:06:06] (e.g that should not be running and is intentionally stopped by puppet) [16:06:14] I'm replacing labmon -> cloudmonitor [16:07:20] fixed it on deployment-parsoid10 but other hosts with errors [16:07:31] and logstash-beta flooded with "cannot connect" etc [16:07:53] oh [16:08:26] Error sending hot-shots message: Error: getaddrinfo ENOTFOUND labmon1001.eqiad.wmnet [16:08:43] which makes sense because labmon1001 no longer exists [16:08:56] rsyslog has some weird stuff as well [16:09:12] Dec 26 15:53:45 deployment-changeprop rsyslogd[16454]: could not load module 'lmnsd_tcp', errors: trying to load module /usr/lib/x86_64-linux-gnu/rsyslog/lmnsd_tcp.so: /usr/lib/x86_64-linux-gnu/rsyslog/lmnsd_...g.com/e/2066 ] [17:00:17] 10Beta-Cluster-Infrastructure: Steward access on Beta Cluster for Tulsi Bhagat - https://phabricator.wikimedia.org/T241466 (10Tulsi_Bhagat) [17:01:00] 10Beta-Cluster-Infrastructure: Steward access on Beta Cluster for Tulsi Bhagat - https://phabricator.wikimedia.org/T241466 (10Tulsi_Bhagat) [17:23:10] maybe I should restart deployment-cpjobqueue [17:32:44] 10Beta-Cluster-Infrastructure: Update references from labmon.* to cloudmetrics.* on deployment-prep (beta cluster) - https://phabricator.wikimedia.org/T241462 (10MarcoAurelio) After replacing all usages of `labmon1001` I continue to see in Logstash-Beta the following for deployment-cpjobqueue: Error sending h... [17:48:35] 10Beta-Cluster-Infrastructure: Update references from labmon.* to cloudmetrics.* on deployment-prep (beta cluster) - https://phabricator.wikimedia.org/T241462 (10MarcoAurelio) This is also interesting: ` maurelio@deployment-changeprop:~$ sudo service rsyslog status ● rsyslog.service - System Logging Service... [20:12:08] 10Beta-Cluster-Infrastructure: Steward access on Beta Cluster for Tulsi Bhagat - https://phabricator.wikimedia.org/T241466 (10DannyS712) @Tulsi_Bhagat you appear to have admin rights on beta commons but to have never used them... what will you use these for? [22:53:10] (03PS1) 10DannyS712: Add Echo extension as a dependency for AbuseFilter [integration/config] - 10https://gerrit.wikimedia.org/r/560807 (https://phabricator.wikimedia.org/T235873) [22:53:42] (03PS2) 10DannyS712: Add Echo extension as a dependency for AbuseFilter [integration/config] - 10https://gerrit.wikimedia.org/r/560807 (https://phabricator.wikimedia.org/T235873)