[00:04:14] !log bking@cumin1001 END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) restart without plugin upgrade (3 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic cluster restart - bking@cumin1001 - T297986 [00:04:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:04:21] T297986: [Tracking task] Pair with brian king on various operational tasks - https://phabricator.wikimedia.org/T297986 [00:17:13] PROBLEM - SSH on kubernetes1002.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [00:24:29] (03PS1) 10Gergő Tisza: GrowthExperiments: Add campaign pattern for JOSA [mediawiki-config] - 10https://gerrit.wikimedia.org/r/749598 (https://phabricator.wikimedia.org/T298057) [00:52:25] PROBLEM - Prometheus jobs reduced availability on alert1001 is CRITICAL: job=sidekiq site=codfw https://wikitech.wikimedia.org/wiki/Prometheus%23Prometheus_job_unavailable https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets [00:54:43] RECOVERY - Prometheus jobs reduced availability on alert1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Prometheus%23Prometheus_job_unavailable https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets [03:20:49] RECOVERY - SSH on kubernetes1002.mgmt is OK: SSH OK - OpenSSH_7.0 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [04:06:21] RECOVERY - SSH on contint1001.mgmt is OK: SSH OK - OpenSSH_6.6 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [05:45:45] PROBLEM - Debian mirror in sync with upstream on sodium is CRITICAL: /srv/mirrors/debian is over 14 hours old. https://wikitech.wikimedia.org/wiki/Mirrors [06:24:29] PROBLEM - SSH on kubernetes1002.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [07:00:11] PROBLEM - SSH on kubernetes1004.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [07:54:04] (03PS1) 10JMeybohm: cfssl-issuer: Update to version 0.2.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/749689 (https://phabricator.wikimedia.org/T294560) [07:56:54] (03PS1) 10JMeybohm: cfssl-issuer: Update to v0.2.0 [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/749690 (https://phabricator.wikimedia.org/T294560) [08:01:15] RECOVERY - SSH on kubernetes1004.mgmt is OK: SSH OK - OpenSSH_7.0 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [08:26:49] RECOVERY - SSH on kubernetes1002.mgmt is OK: SSH OK - OpenSSH_7.0 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [09:20:42] (03CR) 10Kormat: Add MySQL upgrade cookbook (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/749176 (https://phabricator.wikimedia.org/T239814) (owner: 10Ladsgroup) [09:28:22] 10SRE, 10ops-eqiad: Rack msw2-eqiad in cab A8 for configuration - https://phabricator.wikimedia.org/T296271 (10ayounsi) a:05ayounsi→03Cmjohnson The interface msw1:et-0/1/0 is alerting about CRC errors. In the meantime I also added it to Netbox as disabled feel free to enable it for troubleshooting. [09:55:44] (03PS1) 10Kormat: wmfdb/mycnf: Improve support for in-line comments. [software/wmfdb] - 10https://gerrit.wikimedia.org/r/749694 [09:56:58] (03PS2) 10Kormat: wmfdb/mycnf: Improve support for in-line comments. [software/wmfdb] - 10https://gerrit.wikimedia.org/r/749694 [10:08:27] (03PS1) 10Ayounsi: Capirca: disable shade check [software/homer] - 10https://gerrit.wikimedia.org/r/749696 (https://phabricator.wikimedia.org/T273865) [10:09:48] (03CR) 10Volans: [C: 03+1] "LGTM, for a definition of "good" compatible with the upstream nonsense. Nits inline." [software/wmfdb] - 10https://gerrit.wikimedia.org/r/749694 (owner: 10Kormat) [10:10:16] (03CR) 10Klausman: wmfdb/mycnf: Improve support for in-line comments. (032 comments) [software/wmfdb] - 10https://gerrit.wikimedia.org/r/749694 (owner: 10Kormat) [10:11:45] (03CR) 10jerkins-bot: [V: 04-1] Capirca: disable shade check [software/homer] - 10https://gerrit.wikimedia.org/r/749696 (https://phabricator.wikimedia.org/T273865) (owner: 10Ayounsi) [10:11:59] (03CR) 10Jelto: "I updated the helmBinary field and kept the environment values files. Could you take a look again @JMeybohm ?" [deployment-charts] - 10https://gerrit.wikimedia.org/r/737034 (https://phabricator.wikimedia.org/T251305) (owner: 10Jelto) [10:17:18] (03PS3) 10Kormat: wmfdb/mycnf: Improve support for in-line comments. [software/wmfdb] - 10https://gerrit.wikimedia.org/r/749694 [10:18:35] (03PS4) 10Kormat: wmfdb/mycnf: Improve support for in-line comments. [software/wmfdb] - 10https://gerrit.wikimedia.org/r/749694 [10:19:01] (03CR) 10Kormat: wmfdb/mycnf: Improve support for in-line comments. (035 comments) [software/wmfdb] - 10https://gerrit.wikimedia.org/r/749694 (owner: 10Kormat) [10:23:40] (03CR) 10Kormat: [C: 03+2] wmfdb/mycnf: Improve support for in-line comments. [software/wmfdb] - 10https://gerrit.wikimedia.org/r/749694 (owner: 10Kormat) [10:25:06] (03Merged) 10jenkins-bot: wmfdb/mycnf: Improve support for in-line comments. [software/wmfdb] - 10https://gerrit.wikimedia.org/r/749694 (owner: 10Kormat) [10:26:47] (03PS1) 10Kormat: wmfdb/mycnf: Return number of cnf's loaded. [software/wmfdb] - 10https://gerrit.wikimedia.org/r/749697 [10:36:10] (03PS1) 10Kormat: switchover.py: Change references to db-mysql. [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/749699 (https://phabricator.wikimedia.org/T297618) [10:41:57] (03CR) 10Volans: [C: 03+1] "looks fine, although I think that the events should be properly deployed somewhere and not read blindly from gerrit." [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/749699 (https://phabricator.wikimedia.org/T297618) (owner: 10Kormat) [10:43:56] (03CR) 10Kormat: [C: 03+2] switchover.py: Change references to db-mysql. [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/749699 (https://phabricator.wikimedia.org/T297618) (owner: 10Kormat) [10:46:12] (03Merged) 10jenkins-bot: switchover.py: Change references to db-mysql. [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/749699 (https://phabricator.wikimedia.org/T297618) (owner: 10Kormat) [10:49:12] (03CR) 10JMeybohm: [C: 03+1] services: cleanup helmfile, update SAL logging [deployment-charts] - 10https://gerrit.wikimedia.org/r/737034 (https://phabricator.wikimedia.org/T251305) (owner: 10Jelto) [10:49:50] (03CR) 10Kormat: [C: 03+2] wmfdb/mycnf: Return number of cnf's loaded. [software/wmfdb] - 10https://gerrit.wikimedia.org/r/749697 (owner: 10Kormat) [10:50:26] (03PS1) 10Arturo Borrero Gonzalez: toolforge: exec_env: use default buster mono in buster grid [puppet] - 10https://gerrit.wikimedia.org/r/749703 (https://phabricator.wikimedia.org/T298089) [10:51:40] (03Merged) 10jenkins-bot: wmfdb/mycnf: Return number of cnf's loaded. [software/wmfdb] - 10https://gerrit.wikimedia.org/r/749697 (owner: 10Kormat) [10:53:04] (03CR) 10Arturo Borrero Gonzalez: [C: 03+2] toolforge: exec_env: use default buster mono in buster grid [puppet] - 10https://gerrit.wikimedia.org/r/749703 (https://phabricator.wikimedia.org/T298089) (owner: 10Arturo Borrero Gonzalez) [11:24:26] (03PS1) 10Arturo Borrero Gonzalez: wmcs: toolforge: add_grid_webgrid_generic_node: fix start instance response [cookbooks] (wmcs) - 10https://gerrit.wikimedia.org/r/749704 [11:34:06] !log upgraded spicerack to v1.1.1 on cumin1001,cumin2002 [11:34:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:39:09] (03CR) 10Jbond: [C: 04-1] "-1 relates to using from spicecrack import puppet vs the spicrack instance (see inline)" [cookbooks] - 10https://gerrit.wikimedia.org/r/749176 (https://phabricator.wikimedia.org/T239814) (owner: 10Ladsgroup) [11:40:14] (03PS1) 10Arturo Borrero Gonzalez: wmcs: toolforge: grid: add_node_to_cluster: verify the node name is a FQDN [cookbooks] (wmcs) - 10https://gerrit.wikimedia.org/r/749706 [11:40:16] (03PS1) 10Arturo Borrero Gonzalez: wmcs: toolforge: grid: add_node_to_cluster: add extra puppet run [cookbooks] (wmcs) - 10https://gerrit.wikimedia.org/r/749707 [11:42:44] (03CR) 10Jbond: [C: 03+1] "LGTM optional nit/question" [cookbooks] - 10https://gerrit.wikimedia.org/r/749590 (owner: 10Volans) [11:43:07] (03CR) 10jerkins-bot: [V: 04-1] wmcs: toolforge: grid: add_node_to_cluster: add extra puppet run [cookbooks] (wmcs) - 10https://gerrit.wikimedia.org/r/749707 (owner: 10Arturo Borrero Gonzalez) [11:43:09] (03CR) 10jerkins-bot: [V: 04-1] wmcs: toolforge: grid: add_node_to_cluster: verify the node name is a FQDN [cookbooks] (wmcs) - 10https://gerrit.wikimedia.org/r/749706 (owner: 10Arturo Borrero Gonzalez) [11:46:41] (03CR) 10Volans: "reply inline" [cookbooks] - 10https://gerrit.wikimedia.org/r/749590 (owner: 10Volans) [11:49:22] (03CR) 10Jcrespo: "I double checked by manually running" [puppet] - 10https://gerrit.wikimedia.org/r/749561 (https://phabricator.wikimedia.org/T262668) (owner: 10Jcrespo) [11:51:13] (03CR) 10Jbond: [C: 03+1] sre.hosts.provision: fix boot order (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/749590 (owner: 10Volans) [11:59:40] (03CR) 10Jbond: [C: 03+1] "lgtm" [puppet] - 10https://gerrit.wikimedia.org/r/749561 (https://phabricator.wikimedia.org/T262668) (owner: 10Jcrespo) [12:02:39] (03Abandoned) 10Nikerabbit: Set wmgULSAnonCanChangeLanguage true for commonswiki. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/345342 (https://phabricator.wikimedia.org/T161517) (owner: 10Steinsplitter) [12:03:04] (03CR) 10Volans: [C: 03+2] sre.hosts.provision: fix boot order [cookbooks] - 10https://gerrit.wikimedia.org/r/749590 (owner: 10Volans) [12:03:12] (03PS2) 10Arturo Borrero Gonzalez: wmcs: toolforge: grid: add_node_to_cluster: add extra puppet run [cookbooks] (wmcs) - 10https://gerrit.wikimedia.org/r/749707 [12:03:24] (03PS4) 10Jbond: mediabackups: Add minio port to ipv6 connections [puppet] - 10https://gerrit.wikimedia.org/r/749561 (https://phabricator.wikimedia.org/T262668) (owner: 10Jcrespo) [12:05:49] (03Merged) 10jenkins-bot: sre.hosts.provision: fix boot order [cookbooks] - 10https://gerrit.wikimedia.org/r/749590 (owner: 10Volans) [12:06:03] (03CR) 10jerkins-bot: [V: 04-1] wmcs: toolforge: grid: add_node_to_cluster: add extra puppet run [cookbooks] (wmcs) - 10https://gerrit.wikimedia.org/r/749707 (owner: 10Arturo Borrero Gonzalez) [12:07:13] (03PS3) 10Arturo Borrero Gonzalez: wmcs: toolforge: grid: add_node_to_cluster: add extra puppet run [cookbooks] (wmcs) - 10https://gerrit.wikimedia.org/r/749707 [12:09:01] (03PS1) 10Majavah: wmcs: toolforge: add_grid_webgrid_generic_node: fix description [cookbooks] (wmcs) - 10https://gerrit.wikimedia.org/r/749711 [12:11:02] (03CR) 10jerkins-bot: [V: 04-1] wmcs: toolforge: grid: add_node_to_cluster: add extra puppet run [cookbooks] (wmcs) - 10https://gerrit.wikimedia.org/r/749707 (owner: 10Arturo Borrero Gonzalez) [12:11:53] (03PS1) 10Giuseppe Lavagetto: Introduce the ClusterConfig class [mediawiki-config] - 10https://gerrit.wikimedia.org/r/749717 [12:11:55] (03PS1) 10Giuseppe Lavagetto: Simplify management of the request time limit [mediawiki-config] - 10https://gerrit.wikimedia.org/r/749718 [12:12:49] (03CR) 10jerkins-bot: [V: 04-1] Introduce the ClusterConfig class [mediawiki-config] - 10https://gerrit.wikimedia.org/r/749717 (owner: 10Giuseppe Lavagetto) [12:12:53] (03CR) 10jerkins-bot: [V: 04-1] Simplify management of the request time limit [mediawiki-config] - 10https://gerrit.wikimedia.org/r/749718 (owner: 10Giuseppe Lavagetto) [12:17:13] (03PS2) 10Giuseppe Lavagetto: Introduce the ClusterConfig class [mediawiki-config] - 10https://gerrit.wikimedia.org/r/749717 [12:17:16] (03PS2) 10Giuseppe Lavagetto: Simplify management of the request time limit [mediawiki-config] - 10https://gerrit.wikimedia.org/r/749718 [12:17:17] (03PS2) 10Arturo Borrero Gonzalez: wmcs: toolforge: grid: add_node_to_cluster: verify the node name is a FQDN [cookbooks] (wmcs) - 10https://gerrit.wikimedia.org/r/749706 [12:17:20] (03PS4) 10Arturo Borrero Gonzalez: wmcs: toolforge: grid: add_node_to_cluster: add extra puppet run [cookbooks] (wmcs) - 10https://gerrit.wikimedia.org/r/749707 [12:18:25] (03CR) 10jerkins-bot: [V: 04-1] Introduce the ClusterConfig class [mediawiki-config] - 10https://gerrit.wikimedia.org/r/749717 (owner: 10Giuseppe Lavagetto) [12:18:31] (03CR) 10jerkins-bot: [V: 04-1] Simplify management of the request time limit [mediawiki-config] - 10https://gerrit.wikimedia.org/r/749718 (owner: 10Giuseppe Lavagetto) [12:19:29] (03CR) 10Jbond: [C: 03+1] sre.hosts.provision: check if changes were applied [cookbooks] - 10https://gerrit.wikimedia.org/r/749591 (owner: 10Volans) [12:33:18] (03CR) 10Arturo Borrero Gonzalez: [C: 03+2] wmcs: toolforge: add_grid_webgrid_generic_node: fix start instance response [cookbooks] (wmcs) - 10https://gerrit.wikimedia.org/r/749704 (owner: 10Arturo Borrero Gonzalez) [12:33:34] (03CR) 10Arturo Borrero Gonzalez: [C: 03+2] wmcs: toolforge: grid: add_node_to_cluster: verify the node name is a FQDN [cookbooks] (wmcs) - 10https://gerrit.wikimedia.org/r/749706 (owner: 10Arturo Borrero Gonzalez) [12:34:16] (03CR) 10Arturo Borrero Gonzalez: [C: 03+2] wmcs: toolforge: grid: add_node_to_cluster: add extra puppet run [cookbooks] (wmcs) - 10https://gerrit.wikimedia.org/r/749707 (owner: 10Arturo Borrero Gonzalez) [12:35:41] (03PS2) 10Majavah: wmcs: toolforge: add_grid_webgrid_generic_node: fix description [cookbooks] (wmcs) - 10https://gerrit.wikimedia.org/r/749711 [12:37:49] 10SRE, 10ops-eqiad, 10DC-Ops: (Need By: 2021-03-31) rack/setup/install ms-backup100[12] - https://phabricator.wikimedia.org/T274206 (10jcrespo) 05Resolved→03Open I just realized with jbond that the 2 servers (ms-backup1001.eqiad.wmnet and ms-backup1002.eqiad.wmnet) do not have ipv6 AAAA dns records. Whi... [12:39:03] (03PS1) 10Arturo Borrero Gonzalez: sonofgridengine: grid_configurator: relax compat domain message [puppet] - 10https://gerrit.wikimedia.org/r/749730 [12:40:53] (03CR) 10Jbond: mediabackups: Add minio port to ipv6 connections (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/749561 (https://phabricator.wikimedia.org/T262668) (owner: 10Jcrespo) [12:41:38] (03CR) 10Jcrespo: "I filed an issue for dcops due to the lack of AAAA records on the clients- once that gets fixed, either the issue will go away, as Majavah" [puppet] - 10https://gerrit.wikimedia.org/r/749561 (https://phabricator.wikimedia.org/T262668) (owner: 10Jcrespo) [12:45:21] (03CR) 10Jcrespo: [C: 04-1] "Waiting on other bug to be fixed, most likely this will be abandoned (but leaving this open as a reminder)." [puppet] - 10https://gerrit.wikimedia.org/r/749561 (https://phabricator.wikimedia.org/T262668) (owner: 10Jcrespo) [12:51:52] (03PS1) 10Jbond: nodegen: interface with puppetdb via the nginx site instead of directly [software/puppet-compiler] - 10https://gerrit.wikimedia.org/r/749731 [12:53:14] (03CR) 10jerkins-bot: [V: 04-1] nodegen: interface with puppetdb via the nginx site instead of directly [software/puppet-compiler] - 10https://gerrit.wikimedia.org/r/749731 (owner: 10Jbond) [13:00:22] Are the servers ready for me to delete 17,000 revisions? [13:00:25] :P [13:03:23] (03PS2) 10Jbond: nodegen: interface with puppetdb via the nginx site instead of directly [software/puppet-compiler] - 10https://gerrit.wikimedia.org/r/749731 [13:04:37] (03CR) 10jerkins-bot: [V: 04-1] nodegen: interface with puppetdb via the nginx site instead of directly [software/puppet-compiler] - 10https://gerrit.wikimedia.org/r/749731 (owner: 10Jbond) [13:07:46] (03PS3) 10Jbond: nodegen: interface with puppetdb via the nginx site instead of directly [software/puppet-compiler] - 10https://gerrit.wikimedia.org/r/749731 [13:11:27] (03CR) 10Jbond: [V: 03+2 C: 03+2] nodegen: interface with puppetdb via the nginx site instead of directly [software/puppet-compiler] - 10https://gerrit.wikimedia.org/r/749731 (owner: 10Jbond) [13:12:39] (03Merged) 10jenkins-bot: nodegen: interface with puppetdb via the nginx site instead of directly [software/puppet-compiler] - 10https://gerrit.wikimedia.org/r/749731 (owner: 10Jbond) [13:13:19] (03PS1) 10Jbond: 2.0.1: prepare release [software/puppet-compiler] - 10https://gerrit.wikimedia.org/r/749732 [13:22:17] (03CR) 10Arturo Borrero Gonzalez: [C: 03+2] sonofgridengine: grid_configurator: relax compat domain message [puppet] - 10https://gerrit.wikimedia.org/r/749730 (owner: 10Arturo Borrero Gonzalez) [13:23:39] (03CR) 10Jbond: [C: 03+2] 2.0.1: prepare release [software/puppet-compiler] - 10https://gerrit.wikimedia.org/r/749732 (owner: 10Jbond) [13:24:51] (03Merged) 10jenkins-bot: 2.0.1: prepare release [software/puppet-compiler] - 10https://gerrit.wikimedia.org/r/749732 (owner: 10Jbond) [13:29:55] (03PS4) 10Jbond: P:puppet_compiler: update workers to use shared puppetdb [puppet] - 10https://gerrit.wikimedia.org/r/747899 [13:32:13] (03CR) 10Jbond: [C: 03+2] P:puppet_compiler: update workers to use shared puppetdb [puppet] - 10https://gerrit.wikimedia.org/r/747899 (owner: 10Jbond) [13:32:52] (03CR) 10Volans: [C: 03+2] sre.hosts.provision: check if changes were applied [cookbooks] - 10https://gerrit.wikimedia.org/r/749591 (owner: 10Volans) [13:35:26] (03Merged) 10jenkins-bot: sre.hosts.provision: check if changes were applied [cookbooks] - 10https://gerrit.wikimedia.org/r/749591 (owner: 10Volans) [13:41:02] (03PS1) 10Urbanecm: Do not set wgTrustedXffFile [mediawiki-config] - 10https://gerrit.wikimedia.org/r/749734 (https://phabricator.wikimedia.org/T298243) [13:41:04] (03PS1) 10Urbanecm: Remove trusted-xff.php from wmf-config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/749735 (https://phabricator.wikimedia.org/T298243) [13:46:43] (03PS3) 10Giuseppe Lavagetto: Introduce the ClusterConfig class [mediawiki-config] - 10https://gerrit.wikimedia.org/r/749717 [13:47:37] (03CR) 10jerkins-bot: [V: 04-1] Introduce the ClusterConfig class [mediawiki-config] - 10https://gerrit.wikimedia.org/r/749717 (owner: 10Giuseppe Lavagetto) [13:49:03] (03CR) 10Giuseppe Lavagetto: [C: 04-1] "If you remove the setting, you should also remove the php file at wmf-config/trusted-xff.php" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/749734 (https://phabricator.wikimedia.org/T298243) (owner: 10Urbanecm) [13:51:12] (03PS4) 10Giuseppe Lavagetto: Introduce the ClusterConfig class [mediawiki-config] - 10https://gerrit.wikimedia.org/r/749717 [13:55:23] (03CR) 10Urbanecm: Do not set wgTrustedXffFile (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/749734 (https://phabricator.wikimedia.org/T298243) (owner: 10Urbanecm) [13:55:43] (03PS1) 10Arturo Borrero Gonzalez: wmcs: GridConfigurator: run puppet agent in the master node when reconfiguring [cookbooks] (wmcs) - 10https://gerrit.wikimedia.org/r/749739 [13:57:36] (03CR) 10Volans: "replies inline" [software/spicerack] - 10https://gerrit.wikimedia.org/r/747116 (owner: 10Jbond) [13:59:49] (03PS1) 10Jbond: puppet_compiler: fix vhost for db vs worker [puppet] - 10https://gerrit.wikimedia.org/r/749740 [14:01:18] (03CR) 10Majavah: [C: 04-1] Introduce the ClusterConfig class (032 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/749717 (owner: 10Giuseppe Lavagetto) [14:01:34] (03CR) 10jerkins-bot: [V: 04-1] puppet_compiler: fix vhost for db vs worker [puppet] - 10https://gerrit.wikimedia.org/r/749740 (owner: 10Jbond) [14:04:42] (03PS2) 10Jbond: puppet_compiler: fix vhost for db vs worker [puppet] - 10https://gerrit.wikimedia.org/r/749740 [14:07:07] (03CR) 10Jbond: [C: 03+2] puppet_compiler: fix vhost for db vs worker [puppet] - 10https://gerrit.wikimedia.org/r/749740 (owner: 10Jbond) [14:11:07] 10SRE, 10SRE-swift-storage, 10Maps, 10Product-Infrastructure-Team-Backlog (Kanban): tegola-vector-tiles load testing and Swift throughput experiments - https://phabricator.wikimedia.org/T284440 (10Jgiannelos) [14:12:11] 10SRE, 10SRE-swift-storage, 10Maps, 10Product-Infrastructure-Team-Backlog (Kanban): tegola-vector-tiles load testing and Swift throughput experiments - https://phabricator.wikimedia.org/T284440 (10Jgiannelos) 05Open→03Resolved a:03Jgiannelos [14:12:17] (03PS1) 10Jbond: puppet_compiler: fix doc root [puppet] - 10https://gerrit.wikimedia.org/r/749743 [14:12:22] (03CR) 10Jbond: [V: 03+2 C: 03+2] puppet_compiler: fix doc root [puppet] - 10https://gerrit.wikimedia.org/r/749743 (owner: 10Jbond) [14:13:11] (03PS1) 10Arturo Borrero Gonzalez: sonofgridengine: grid-configurator: support shorter bastion prefix [puppet] - 10https://gerrit.wikimedia.org/r/749744 [14:20:01] (03CR) 10Arturo Borrero Gonzalez: [C: 04-1] "I'm undecided if this is good or not." [puppet] - 10https://gerrit.wikimedia.org/r/749744 (owner: 10Arturo Borrero Gonzalez) [14:25:19] (03CR) 10Zabe: "Please also remove the entry in createTxtFileSymlinks.sh and in phpcs.xml" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/749735 (https://phabricator.wikimedia.org/T298243) (owner: 10Urbanecm) [14:27:21] (03PS2) 10Urbanecm: Remove trusted-xff.php from wmf-config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/749735 (https://phabricator.wikimedia.org/T298243) [14:27:25] (03CR) 10Urbanecm: Remove trusted-xff.php from wmf-config (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/749735 (https://phabricator.wikimedia.org/T298243) (owner: 10Urbanecm) [14:30:24] (03PS1) 10Jbond: puppet_compiler: update version [puppet] - 10https://gerrit.wikimedia.org/r/749745 [14:30:40] (03CR) 10Jbond: [V: 03+2 C: 03+2] puppet_compiler: update version [puppet] - 10https://gerrit.wikimedia.org/r/749745 (owner: 10Jbond) [14:30:52] (03CR) 10Giuseppe Lavagetto: Introduce the ClusterConfig class (032 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/749717 (owner: 10Giuseppe Lavagetto) [14:35:14] (03CR) 10Zabe: [C: 03+1] "lgtm" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/749735 (https://phabricator.wikimedia.org/T298243) (owner: 10Urbanecm) [14:47:45] (03CR) 10Jbond: WIP: add reposync (0310 comments) [software/spicerack] - 10https://gerrit.wikimedia.org/r/747116 (owner: 10Jbond) [15:00:14] (03CR) 10Jbond: reposync: add initial repo sync class (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/747091 (owner: 10Jbond) [15:01:55] (03PS1) 10Jbond: realm.pp: this block still causes issues with pcc and puppet lookup [puppet] - 10https://gerrit.wikimedia.org/r/749746 [15:02:10] (03CR) 10Volans: WIP: add reposync (033 comments) [software/spicerack] - 10https://gerrit.wikimedia.org/r/747116 (owner: 10Jbond) [15:02:17] (03CR) 10Jbond: [C: 04-1] "-1 untill the new year" [puppet] - 10https://gerrit.wikimedia.org/r/749746 (owner: 10Jbond) [15:04:47] 10SRE, 10ops-eqiad, 10DC-Ops: (Need By: 2021-03-31) rack/setup/install ms-backup100[12] - https://phabricator.wikimedia.org/T274206 (10RobH) a:05RobH→03Cmjohnson [15:04:52] 10SRE, 10ops-eqiad, 10DC-Ops: (Need By: 2021-03-31) rack/setup/install ms-backup100[12] - https://phabricator.wikimedia.org/T274206 (10RobH) @Cmjohnson sets up dns so perhaps he knows? [15:12:33] 10SRE, 10ops-eqiad, 10DC-Ops: (Need By: 2021-03-31) rack/setup/install ms-backup100[12] - https://phabricator.wikimedia.org/T274206 (10Cmjohnson) @jcrespo I believe this would've been a script failure/issue. I would only select the skip IPv6 button if the task called for it. [15:28:49] 10SRE, 10Foundational Technology Requests, 10Traffic, 10Wikimedia Enterprise, 10Wikimedia Enterprise Discussion: Allow-Listing for Enterprise IPs - https://phabricator.wikimedia.org/T294798 (10AnnaMikla) [15:35:31] PROBLEM - SSH on kubernetes1002.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [15:35:58] (03PS5) 10Giuseppe Lavagetto: Introduce the ClusterConfig class [mediawiki-config] - 10https://gerrit.wikimedia.org/r/749717 [15:42:02] 10SRE, 10ops-eqiad, 10DC-Ops: (Need By: 2021-03-31) rack/setup/install ms-backup100[12] - https://phabricator.wikimedia.org/T274206 (10jcrespo) a:05Cmjohnson→03jcrespo Thank you for the feedback! As we all think this was not disabled for some reason (I've seen the setup scripts fail on some steps sometim... [15:53:16] 10SRE-swift-storage, 10Beta-Cluster-Infrastructure: Upgrade deployment-prep Swift cluster to Debian Buster or newer - https://phabricator.wikimedia.org/T298253 (10Majavah) [16:01:05] !log volans@cumin1001 START - Cookbook sre.hosts.provision for host mc2038.mgmt.codfw.wmnet with reboot policy FORCED [16:01:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:01:37] !log volans@cumin1001 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mc2038.mgmt.codfw.wmnet with reboot policy FORCED [16:01:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:13:58] !log volans@cumin1001 START - Cookbook sre.hosts.provision for host mc2038.mgmt.codfw.wmnet with reboot policy FORCED [16:14:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:26:33] !log volans@cumin1001 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mc2038.mgmt.codfw.wmnet with reboot policy FORCED [16:26:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:26:43] (03CR) 10MewOphaswongse: GrowthExperiments: Add campaign pattern for JOSA (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/749598 (https://phabricator.wikimedia.org/T298057) (owner: 10Gergő Tisza) [16:33:47] (03PS1) 10Volans: sre.hosts.provision: boot order, settle with 2 NIC [cookbooks] - 10https://gerrit.wikimedia.org/r/749752 [16:35:15] (03CR) 10Volans: [C: 03+2] "self-merging as I'm testing it right now" [cookbooks] - 10https://gerrit.wikimedia.org/r/749752 (owner: 10Volans) [16:37:54] (03Merged) 10jenkins-bot: sre.hosts.provision: boot order, settle with 2 NIC [cookbooks] - 10https://gerrit.wikimedia.org/r/749752 (owner: 10Volans) [16:40:38] !log volans@cumin1001 START - Cookbook sre.hosts.provision for host mc2038.mgmt.codfw.wmnet with reboot policy FORCED [16:40:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:45:29] PROBLEM - High average GET latency for mw requests on api_appserver in eqiad on alert1001 is CRITICAL: cluster=api_appserver code=200 handler=proxy:unix:/run/php/fpm-www.sock https://wikitech.wikimedia.org/wiki/Monitoring/Missing_notes_link https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red-dashboard?panelId=9&fullscreen&orgId=1&from=now-3h&to=now&var-datasource=eqiad+prometheus/ops&var-cluster=api_appserver&var-method=GET [16:46:49] !log volans@cumin1001 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mc2038.mgmt.codfw.wmnet with reboot policy FORCED [16:46:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:47:37] RECOVERY - High average GET latency for mw requests on api_appserver in eqiad on alert1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Monitoring/Missing_notes_link https://grafana.wikimedia.org/d/RIA1lzDZk/application-servers-red-dashboard?panelId=9&fullscreen&orgId=1&from=now-3h&to=now&var-datasource=eqiad+prometheus/ops&var-cluster=api_appserver&var-method=GET [17:01:08] (03CR) 10Dzahn: [C: 03+1] ssh-config: add config for gitlab.wikimedia.org [debs/wmf-sre-laptop] - 10https://gerrit.wikimedia.org/r/749509 (owner: 10Jelto) [17:09:49] 10SRE, 10Commons, 10Datasets-Archiving, 10Datasets-General-or-Unknown, and 3 others: Back up of Commons files - https://phabricator.wikimedia.org/T160229 (10jcrespo) 05In progress→03Resolved Commonswiki is now backed up on 2 geographically redundant locations within WMF infrastructure. The current arc... [17:09:57] 10SRE, 10SRE-swift-storage, 10Data-Persistence-Backup, 10media-backups, and 3 others: WMF media storage must be adequately backed up - https://phabricator.wikimedia.org/T262668 (10jcrespo) [17:10:16] !log volans@cumin1001 START - Cookbook sre.dns.netbox [17:10:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:12:19] PROBLEM - SSH on kubernetes1004.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [17:13:53] 10SRE, 10SRE-swift-storage, 10Data-Persistence-Backup, 10media-backups, and 3 others: WMF media storage must be adequately backed up - https://phabricator.wikimedia.org/T262668 (10jcrespo) Commonswiki codfw backup copy, with 91823709 files backed up and a 0.04% error rate. The number of duplicates and err... [17:14:32] !log volans@cumin1001 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [17:14:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:17:31] (03PS3) 10JHathaway: exim: add the ability to silently drop senders [puppet] - 10https://gerrit.wikimedia.org/r/748884 (https://phabricator.wikimedia.org/T298038) [17:19:32] !log volans@cumin2002 START - Cookbook sre.hosts.provision for host mc2041.mgmt.codfw.wmnet with reboot policy FORCED [17:19:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:19:37] (03CR) 10JHathaway: exim: add the ability to silently drop senders (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/748884 (https://phabricator.wikimedia.org/T298038) (owner: 10JHathaway) [17:36:47] !log volans@cumin2002 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mc2041.mgmt.codfw.wmnet with reboot policy FORCED [17:36:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:50:01] (03PS1) 10Bartosz Dziewoński: Disable querying the 'wikieditor' change tag temporarily [core] (wmf/1.38.0-wmf.13) - 10https://gerrit.wikimedia.org/r/749766 (https://phabricator.wikimedia.org/T298225) [17:50:24] (03CR) 10Legoktm: [C: 03+2] Disable querying the 'wikieditor' change tag temporarily [core] (wmf/1.38.0-wmf.13) - 10https://gerrit.wikimedia.org/r/749766 (https://phabricator.wikimedia.org/T298225) (owner: 10Bartosz Dziewoński) [17:50:47] !log mforns@deploy1002 Started deploy [airflow-dags/analytics@363651a]: (no justification provided) [17:50:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:50:53] !log mforns@deploy1002 Finished deploy [airflow-dags/analytics@363651a]: (no justification provided) (duration: 00m 06s) [17:50:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:57:08] 10SRE, 10serviceops, 10Datacenter-Switchover, 10Patch-For-Review: Use encrypted rsync for deployment::rsync - https://phabricator.wikimedia.org/T289857 (10Legoktm) a:05Legoktm→03None [17:58:37] 10SRE-Access-Requests: Google Search Console access request for Andrew Green - https://phabricator.wikimedia.org/T298262 (10AndyRussG) [18:01:22] Hi all! Many many apologies for the bother, just wondering if there's any small chance the above request ^ could somehow be expedited...? really sorry for requesting just before break, many many thanks in advance!! :) [18:06:14] AndyRussG: I'm not on clinic duty, but I don't think it's likely. [18:06:36] Also, there are some things on https://wikitech.wikimedia.org/wiki/Google_Search_Console_access like which subdomains and for how long you need access that should be included in your ticket [18:09:35] (03Merged) 10jenkins-bot: Disable querying the 'wikieditor' change tag temporarily [core] (wmf/1.38.0-wmf.13) - 10https://gerrit.wikimedia.org/r/749766 (https://phabricator.wikimedia.org/T298225) (owner: 10Bartosz Dziewoński) [18:09:35] legoktm: thanks! right oops let me add those :) [18:12:06] MatmaRex: it's on mwdebug1001 right now [18:12:33] wfm [18:13:27] !log mwdebug-deploy@deploy1002 helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [18:13:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:14:25] !log mwdebug-deploy@deploy1002 helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [18:14:26] !log mwdebug-deploy@deploy1002 helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [18:14:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:14:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:14:51] !log legoktm@deploy1002 Synchronized php-1.38.0-wmf.13/includes/changetags/ChangeTags.php: Disable querying the 'wikieditor' change tag temporarily (T298225) (duration: 00m 59s) [18:14:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:15:03] thanks [18:15:18] yeah looks as expected in prod [18:17:02] !log mwdebug-deploy@deploy1002 helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [18:17:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:22:30] 10SRE-Access-Requests: Google Search Console access request for Andrew Green - https://phabricator.wikimedia.org/T298262 (10AndyRussG) [18:23:53] legoktm: fixed ^ thanks again!! [18:40:11] did someone here request a google verification code for noc@? I'm guessing from the above messages [18:40:18] (we got email) [18:44:15] 10SRE, 10Datasets-General-or-Unknown: Mirror more Kiwix downloads directories - https://phabricator.wikimedia.org/T57503 (10Kelson) @ArielGlenn Any chance this ticket could be implemented some time? It seems as well that the Wikimedia does not mirror since at least a month because all the Wikipedia ZIM files a... [18:44:38] 10SRE, 10Datasets-General-or-Unknown, 10affects-Kiwix-and-openZIM: Mirror more Kiwix downloads directories - https://phabricator.wikimedia.org/T57503 (10Kelson) [18:45:02] nm I see that someone followed up on the thread, thanks [18:45:40] 10SRE, 10Datasets-General-or-Unknown, 10affects-Kiwix-and-openZIM: Mirror more Kiwix downloads directories - https://phabricator.wikimedia.org/T57503 (10Kelson) [18:45:42] 10SRE: Mailman3 templates with colons in filename made operations/puppet not cloneable on Windows - https://phabricator.wikimedia.org/T282308 (10Legoktm) a:05Legoktm→03None [18:48:42] (03PS1) 10Krinkle: wmf-config: Move loading/computing of $globals to a method for profiling [mediawiki-config] - 10https://gerrit.wikimedia.org/r/749762 (https://phabricator.wikimedia.org/T169821) [18:48:44] (03PS1) 10Krinkle: MWConfigCacheGenerator: Move siteFromDB() deeper down the stack [mediawiki-config] - 10https://gerrit.wikimedia.org/r/749763 (https://phabricator.wikimedia.org/T169821) [18:57:08] 10SRE, 10MW-on-K8s, 10Traffic, 10serviceops, and 2 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Legoktm) [18:57:20] 10SRE, 10MW-on-K8s, 10serviceops, 10Patch-For-Review, 10User-jijiki: Create a mwdebug deployment for mediawiki on kubernetes - https://phabricator.wikimedia.org/T283056 (10Legoktm) [18:57:26] 10SRE, 10MW-on-K8s, 10serviceops, 10MW-1.37-notes (1.37.0-wmf.20; 2021-08-23), and 2 others: Make HTTP calls work within mediawiki on kubernetes - https://phabricator.wikimedia.org/T288848 (10Legoktm) 05Open→03Resolved I will file a separate follow-up task for finishing the rollout on traditional appse... [19:05:56] (03CR) 10Jbond: [C: 04-1] exim: add the ability to silently drop senders (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/748884 (https://phabricator.wikimedia.org/T298038) (owner: 10JHathaway) [19:38:13] AndyRussG: normally this wouldn't work. but eh.. as a Christmas present.. you have been added to the all the domains (i don't know about it besides following the docs though) don't tell anyone *g:) [19:39:51] ahhhh thanks so so much mutante!! [19:40:13] RECOVERY - SSH on kubernetes1002.mgmt is OK: SSH OK - OpenSSH_7.0 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [19:40:36] Happy holidays to you too, all the best!!! :) :) [19:40:52] AndyRussG: so you should have read access on all the ones listed, hope that does it. and to you as well [19:44:35] mutante thx again!! (on my phone just now, should be to try it out in about 1.5 hrs) [19:44:41] 10SRE, 10SRE-Access-Requests: Google Search Console access request for Andrew Green - https://phabricator.wikimedia.org/T298262 (10Dzahn) 05Open→03Resolved a:03Dzahn Normally this wouldn't work adhoc like this, it's kind of a tedious process but .. DONE as a special case for the holidays. You have been... [19:45:21] AndyRussG: alright!, yep [19:45:29] 10SRE, 10MW-on-K8s, 10serviceops: Evaluate nginx-controller as an Ingress - https://phabricator.wikimedia.org/T286197 (10Legoktm) a:05Legoktm→03None [20:15:23] 10SRE, 10Maps: Allow Wikimedia Maps usage on wikijournal.org - https://phabricator.wikimedia.org/T297948 (10Aklapper) 05Stalled→03Declined I'm going to boldly decline this for the time being. [20:15:49] RECOVERY - SSH on kubernetes1004.mgmt is OK: SSH OK - OpenSSH_7.0 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [20:17:50] 10SRE: Allow Wikimedia Maps usage on wikijournal.org - https://phabricator.wikimedia.org/T297948 (10Aklapper) (@dzahn: For the records, the link on https://wikitech.wikimedia.org/wiki/Maps/External_usage intentionally only adds #SRE and not #Maps) [20:19:23] (03PS3) 10Legoktm: Merge db-codfw.php and db-eqiad.php into db-production.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/702421 (https://phabricator.wikimedia.org/T260297) [20:20:48] (03CR) 10Legoktm: "PS3: Rebase, dropped the ParserCache parts of this now that it's been moved." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/702421 (https://phabricator.wikimedia.org/T260297) (owner: 10Legoktm) [20:45:53] 10SRE: Allow Wikimedia Maps usage on wikijournal.org - https://phabricator.wikimedia.org/T297948 (10Dzahn) @Aklapper That seems wrong but "not a manager" (tm). [21:08:16] last AndyRussG [21:08:34] heh. missing / strikes again. :) [21:09:04] hehe hmmm what does last do? [21:09:29] for my client it's like search. makes it easier to follow conversations sometimes. [21:21:45] PROBLEM - Restbase LVS codfw on restbase.svc.codfw.wmnet is CRITICAL: /en.wikipedia.org/v1/page/talk/{title} (Get structured talk page for enwiki Salt article) timed out before a response was received https://wikitech.wikimedia.org/wiki/RESTBase [21:23:57] RECOVERY - Restbase LVS codfw on restbase.svc.codfw.wmnet is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/RESTBase [21:25:06] it's short for "lastlog" https://irssi.org/documentation/startup/#lastlog-and-jumping-around-in-scrollback [21:32:32] (03PS2) 10Gergő Tisza: GrowthExperiments: Add campaign pattern for JOSA [mediawiki-config] - 10https://gerrit.wikimedia.org/r/749598 (https://phabricator.wikimedia.org/T298057) [21:33:03] (03CR) 10Gergő Tisza: GrowthExperiments: Add campaign pattern for JOSA (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/749598 (https://phabricator.wikimedia.org/T298057) (owner: 10Gergő Tisza) [21:37:02] !log taavi@deploy1002 Started deploy [horizon/deploy@ff82962] (dev): (no justification provided) [21:37:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:41:08] !log taavi@deploy1002 Finished deploy [horizon/deploy@ff82962] (dev): (no justification provided) (duration: 04m 06s) [21:41:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:43:15] (03PS4) 10JHathaway: exim: add the ability to silently drop senders [puppet] - 10https://gerrit.wikimedia.org/r/748884 (https://phabricator.wikimedia.org/T298038) [22:08:57] (03CR) 10MewOphaswongse: [C: 03+1] GrowthExperiments: Add campaign pattern for JOSA [mediawiki-config] - 10https://gerrit.wikimedia.org/r/749598 (https://phabricator.wikimedia.org/T298057) (owner: 10Gergő Tisza) [22:24:22] (03PS1) 10Muehlenhoff: Record updated MOU dates [puppet] - 10https://gerrit.wikimedia.org/r/749795 [22:25:53] (03CR) 10Muehlenhoff: [C: 03+2] Record updated MOU dates [puppet] - 10https://gerrit.wikimedia.org/r/749795 (owner: 10Muehlenhoff) [22:47:39] (03CR) 1020after4: [C: 03+1] build: add mypy types [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/747104 (owner: 10Hashar) [23:06:04] mutante finally back from my errand, all good with the console access, thanks again!!! :) :) [23:34:22] is there a problem with gerrit, or is it just me? git pull is hanging forever [23:38:55] oh, it was just me! it was my ssh-agent hanging. never seen that before [23:39:07] noticed when the fans powered up :)