[00:18:28] 10Analytics-Clusters, 10Analytics-Kanban, 10DBA, 10Patch-For-Review: dbstore1004 85% disk space used. - https://phabricator.wikimedia.org/T283125 (10razzi) Hmm, the machine has been renamed and is almost operational, but doesn't have ssh keys so I can't log in. I'm not sure what to do at this point, I trie... [01:54:11] 10Analytics-Radar, 10LDAP-Access-Requests, 10SRE, 10SRE-Access-Requests: Account setup issues for jmixter-ctr - https://phabricator.wikimedia.org/T283250 (10Dzahn) When using Phabricator it is an "either or" thing. You can either use an existing SUL (global wiki projects) user or you can use an existing de... [04:32:55] 10Analytics-Clusters, 10Analytics-Kanban, 10DBA, 10Patch-For-Review: dbstore1004 85% disk space used. - https://phabricator.wikimedia.org/T283125 (10Marostegui) @razzi I have fixed the issue and the host is now accessible. The certificated wasn't signed by puppet, so I have done so manually - did you get a... [05:05:20] 10Analytics-Clusters, 10Analytics-Kanban, 10DBA, 10Patch-For-Review: dbstore1004 85% disk space used. - https://phabricator.wikimedia.org/T283125 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['dbstore1006.eqiad.wmnet'] ` The log can be foun... [05:25:16] 10Analytics-Clusters, 10Analytics-Kanban, 10DBA, 10Patch-For-Review: dbstore1004 85% disk space used. - https://phabricator.wikimedia.org/T283125 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['dbstore1006.eqiad.wmnet'] ` and were **ALL** successful. [05:29:23] 10Analytics-Clusters, 10Analytics-Kanban, 10DBA, 10Patch-For-Review: dbstore1004 85% disk space used. - https://phabricator.wikimedia.org/T283125 (10Marostegui) @razzi all went fine this time, do you remember if you used the `--new` thing? Maybe that was the issue. Anyways, I have executed: ` root@dbstore1... [06:37:44] 10Analytics-Radar, 10LDAP-Access-Requests, 10SRE, 10SRE-Access-Requests: Account setup issues for jmixter-ctr - https://phabricator.wikimedia.org/T283250 (10elukey) After a chat with Moritz I think that I get what's happening. In the email that OIT sends there is a mention of LDAP and how to change its pas... [06:55:32] 10Analytics-Radar, 10SRE, 10ops-eqiad: Try to move some new analytics worker nodes to different racks - https://phabricator.wikimedia.org/T276239 (10elukey) @Cmjohnson Hi! Do you have time next week for the A2 rack? [07:19:55] (03CR) 10Gilles: [C: 03+2] Add request context fields to CpuBenchmark schema [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/693163 (https://phabricator.wikimedia.org/T281243) (owner: 10Gilles) [07:20:31] (03Merged) 10jenkins-bot: Add request context fields to CpuBenchmark schema [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/693163 (https://phabricator.wikimedia.org/T281243) (owner: 10Gilles) [08:09:36] (03PS5) 10Martaannaj: Create wd_propertysuggester/client_ab_testing and wd_propertysuggester/server_ab_testing [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/689152 [08:30:15] 10Analytics, 10SRE, 10Traffic: Downloading from Archiva.wikimedia.org seems slower than Maven Central - https://phabricator.wikimedia.org/T273086 (10hashar) That is still hitting us from time to time, specially when the CI cache is not warmed up or has been wiped. [10:43:54] take it this channel no longer in use, rather using libera [10:44:18] probably want to change the topic [13:34:13] 10Analytics, 10SRE, 10Traffic: Downloading from Archiva.wikimedia.org seems slower than Maven Central - https://phabricator.wikimedia.org/T273086 (10Ottomata) Moving back to incoming for re-triage [14:28:18] (03CR) 10Mforns: [C: 03+1] "LGTM!" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/673128 (owner: 10Razzi) [14:47:33] hi folks, friendly reminder that the channel has been moved to #wikimedia-analytics on Libera [14:53:19] elukey: you can +if #wikimedia-libera and then clear [14:56:40] RhinosF1: what do you mean? [14:57:17] elukey: that will make anyone who joins get redirected to a channel with a nice message explaining the situation [14:57:32] ./mode #wikimedia-analytics +if #wikimedia-libera [14:57:48] Then ./cs clear #wikimedia-analytics USERS moved to libera [14:59:02] yeah but not sure if people want to be forced to Libera yet [14:59:13] we can do it later on [15:01:19] There's about 19 missing [15:01:29] Probably people who don't check often [15:01:55] I'm staying both until the last places are closed [15:08:28] 10Analytics, 10Analytics-Kanban: Generalize the current Airflow puppet/scap code to deploy a dedicated Analytics instance - https://phabricator.wikimedia.org/T272973 (10Ottomata) Airflow 2 .deb proceeding in T277012. This .deb will only install a python environment with all the Airflow dependencies. Everythi... [15:13:51] RhinosF1: there is no rush for the moment for public chans :) [15:15:04] Cool, I'm hanging round to help making sure everyone gets over [15:20:37] 10Analytics, 10Analytics-Kanban: Generalize the current Airflow puppet/scap code to deploy a dedicated Analytics instance - https://phabricator.wikimedia.org/T272973 (10Ottomata) [15:42:22] 10Analytics, 10Analytics-Kanban: Generalize the current Airflow puppet/scap code to deploy a dedicated Analytics instance - https://phabricator.wikimedia.org/T272973 (10Ottomata) There's a bunch of puppet code already that does the above, but I think we should start from scratch, since we'll be using a new pac... [17:22:07] (03CR) 10Razzi: [C: 03+2] Add working versions of node and npm to package.json engines [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/673128 (owner: 10Razzi) [17:42:29] 10Analytics-Radar, 10LDAP-Access-Requests, 10SRE, 10SRE-Access-Requests: Account setup issues for jmixter-ctr - https://phabricator.wikimedia.org/T283250 (10jmixter) @elukey thank you so much for the help and clarification. My WikiTech shell name is jmixter. [17:46:05] 10Analytics-Radar, 10SRE, 10ops-eqiad: Try to move some new analytics worker nodes to different racks - https://phabricator.wikimedia.org/T276239 (10Cmjohnson) @elukey sure I can move one of them to A2. Rack A7 is still full [17:46:53] 10Analytics-Radar, 10SRE, 10ops-eqiad: Try to move some new analytics worker nodes to different racks - https://phabricator.wikimedia.org/T276239 (10elukey) Yes perfect! Anytime is good, those nodes are not in service. [17:46:57] 10Analytics: Request for Kerberos password for kzimmerman - https://phabricator.wikimedia.org/T283386 (10kzimmerman) [18:09:47] 10Analytics, 10MediaWiki-REST-API, 10Platform Team Sprints Board (Sprint 5), 10Platform Team Workboards (Green), 10Story: System administrator reviews API usage by client - https://phabricator.wikimedia.org/T251812 (10Pchelolo) [18:09:49] 10Analytics-Radar, 10MediaWiki-REST-API, 10Patch-For-Review, 10Platform Team Workboards (Green), 10Story: Followups to access logging after envoy 1.16 - https://phabricator.wikimedia.org/T260820 (10Pchelolo) 05Open→03Resolved [18:38:45] 10Analytics, 10Analytics-Kanban: Generalize the current Airflow puppet/scap code to deploy a dedicated Analytics instance - https://phabricator.wikimedia.org/T272973 (10gmodena) @Ottomata thanks for the summary & overview of the `.deb` status. We have a dependency on the [[ https://airflow.apache.org/docs/apa... [19:29:24] 10Analytics-Radar, 10LDAP-Access-Requests, 10SRE, 10SRE-Access-Requests: Account setup issues for jmixter-ctr - https://phabricator.wikimedia.org/T283250 (10Dzahn) > I tried to submit a Phabricator question but received this notification: > 'This board is protected. Only autoconfirmed users can participate... [20:58:56] 10Analytics-Clusters, 10Analytics-Kanban, 10DBA, 10Patch-For-Review: dbstore1004 85% disk space used. - https://phabricator.wikimedia.org/T283125 (10razzi) @Marostegui glad you were able to figure that out and that it worked on a new reimage. My last attempt timed out, and I was troubleshooting some networ...