[06:27:36] 10Analytics, 10DBA, 10Data-Services: Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Marostegui) I have sanitized hiwikisource on sanitarium hosts (db1124:3313 and db2094:3313) and I have checked that the users were sanitized correctly, and my user, which I cr... [07:42:18] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban: Archive data on eventlogging MySQL to analytics replica before decomisioning - https://phabricator.wikimedia.org/T231858 (10elukey) I asked some info to @Marostegui today and we might need to do some work to make this happen. My assumption is that... [07:44:48] 10Analytics, 10Analytics-EventLogging: Disable production EventLogging analytics MySQL consumers - https://phabricator.wikimedia.org/T232349 (10elukey) >>! In T232349#5477229, @ayounsi wrote: > This look related, > https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=eventlog1002&service=Check+s... [07:46:12] fdans: o/ - if you have time later on can you add next steps to https://phabricator.wikimedia.org/T223414 ? [07:46:27] don't recall if we decided how to proceed with the team [07:47:00] elukey I think in the end marcel is going to take this next week [07:48:05] 10Analytics, 10Analytics-Kanban: Move the Analytics Refinery to Python 3 - https://phabricator.wikimedia.org/T204735 (10elukey) Nice! Thanks a lot @mforns.. I think that we might be able to quickly deploy refinery today (that wouldn't even require a deploy to hdfs) and close this task. [07:48:11] fdans: ack thanks! [07:50:49] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban: Archive data on eventlogging MySQL to analytics replica before decomisioning - https://phabricator.wikimedia.org/T231858 (10Marostegui) db1108 has also the staging database, so that might be the reason for the different sizes. Probably exploring wha... [08:12:10] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops, 10ops-eqiad: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN. - https://phabricator.wikimedia.org/T225128 (10elukey) ` elukey@asw2-a-eqiad# show | compare [edit interfaces xe-2/0/24] - description cloudvirtan1002; +... [08:18:13] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops, 10ops-eqiad: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN. - https://phabricator.wikimedia.org/T225128 (10elukey) ` elukey@asw2-b-eqiad# show | compare [edit interfaces xe-4/0/5] - description cloudvirtan1004; +... [08:33:55] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops, 10ops-eqiad: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN. - https://phabricator.wikimedia.org/T225128 (10elukey) @Cmjohnson @Jclark-ctr there is one last problem - an-presto1005: 1) is not connected to any switch... [08:34:06] I was wrong, an-presto1005 is not connected to the switch sigh [08:46:46] 10Analytics, 10Operations, 10SRE-Access-Requests: Requesting access to analytics cluster for Martin Gerlach - https://phabricator.wikimedia.org/T232707 (10MoritzMuehlenhoff) 05Resolved→03Open @MGerlach : You're using the same key for production SSH access and Cloud VPS, which is insecure as Cloud VPS all... [08:58:05] moritzm: apologies, am slightly confused. not exactly sure what to do [09:01:39] please create a separate SSH key (e.g. name it wikimedia-prod or so) following https://wikitech.wikimedia.org/wiki/Production_shell_access#Generating_your_SSH_key [09:01:52] then the config of your SSH client (following e.g. https://wikitech.wikimedia.org/wiki/Production_shell_access#Setting_up_your_SSH_config) [09:02:01] needs to point to that separate SSH key [09:02:48] the other key (the one you can see if you access https://wikitech.wikimedia.org/wiki/Special:Preferences -> Preferences - > OpenStack) [09:03:13] is the key which is used if you want to access https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS [09:03:40] does that help? [09:05:07] moritzm: yes. [09:07:00] great :-) [09:16:04] !log starting load to cassandra of mediarequests per file daily [09:16:06] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [09:16:45] https://www.irccloud.com/pastebin/1TaimUT4/ [09:17:08] moritzm: I put a new ssh-key to https://wikitech.wikimedia.org/wiki/Special:Preferences -> references - > OpenStack). [09:17:08] Does that work? [09:18:16] fdans: I guess you've started cassandra job from the past? [09:18:32] joal: not yet, about to [09:18:57] cool :) When do we start the load? [09:19:19] (informative question onl) [09:19:39] joal: I'm about to send the job to load per file from may 16 2019 [09:20:46] fdans: Let's start May 17th to have full days please :) [09:21:05] joal: but may 16th starts on the 0 hour no? [09:21:12] have you checked? [09:21:21] joal: yes just checked the partitions [09:22:06] fdans: partitions are not trustworthy - they might be empty - non-empty folders should be the way [09:22:08] joal: oh wait maybe I'm messing up [09:22:21] joal: bonjour! Do you think that we could do a quick (non-hdfs) refinery deployment to apply the python fix that marcel did yesterday? [09:22:27] even this afternoon [09:22:28] no rush [09:22:34] of course elukey [09:22:46] mgerlach: yes, that works as well. Can you please leave a comment on task about this? [09:23:27] moritzm: yes, I can do that. I will close the task then. thanks for your help. [09:23:30] joal: my bad joseph, starting from the 17th as it is the first full day [09:23:45] Thanks fdans :) [09:25:06] mgerlach: sounds good [09:25:10] 10Analytics, 10Operations, 10SRE-Access-Requests: Requesting access to analytics cluster for Martin Gerlach - https://phabricator.wikimedia.org/T232707 (10MGerlach) 05Open→03Resolved @MoritzMuehlenhoff Added separate key for Cloud VPS. [09:30:57] (03PS1) 10Fdans: Correct dataset name parameter in mediarequest cassandra jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/537936 [09:38:56] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Upgrade ua parser to latest version for both java and python - https://phabricator.wikimedia.org/T212854 (10JAllemandou) Documentation updated: * https://wikitech.wikimedia.org/w/index.php?title=Analytics%2FData_Lake%2FTraffic%2FWebrequest&type=revision&d... [10:35:24] sorry for the flood of errors a-team, should have started with a single day, which is what I'm doing now [10:37:22] (03PS2) 10Fdans: Correct parameters in mediarequest cassandra jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/537936 [10:37:59] !log manually rollback /srv/deployment/analytics/refinery/bin/refinery-drop-hive-partitions to "#!/usr/bin/env python" on stat1007 [10:38:01] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:38:02] fdans: np! [10:38:21] as long as we know what is happening, emails are ok [10:38:37] it is worse when we get a ton of alerts and we don't know what's happening :D [10:39:03] elukey: I know but I'm sorry if they are getting mixed with other, more important alerts [10:41:09] (03PS3) 10Fdans: Correct parameters in mediarequest cassandra jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/537936 [10:41:56] 10Analytics, 10Analytics-Kanban: Move the Analytics Refinery to Python 3 - https://phabricator.wikimedia.org/T204735 (10elukey) @mforns another interesting use case: ` Sep 19 10:23:17 stat1007 systemd[1]: Started Drop cirrus click logs from Hive/HDFS following data retention policies.. Sep 19 10:23:24 stat100... [10:42:35] ebernhardson: as FYI --^ [10:42:40] :) [11:43:20] elukey: do you still want a refinery-deploy after the cirrus finding? [11:43:59] joal: let's wait thanks :) [11:44:28] ok elukey :) [12:02:39] (sorry I am in an interview and lagging :( ) [12:15:28] 10Analytics, 10DBA, 10Data-Services, 10cloud-services-team: Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Marostegui) p:05Low→03Normal a:05Marostegui→03None This is ready for #cloud-services-team - the views can be created on labsdb1009-labsdb1012.... [12:21:23] joal: you think I should run the per referer loading job while the per file is running or should I wait for completion? [12:21:38] per file will probably take a couple days to complete [12:22:38] lunch break now :) [12:23:26] elukey 🇪🇸 [12:40:57] fdans: You can launch per-referer - It's a lot smaller than per-file [13:15:02] 10Analytics, 10DBA, 10Data-Services, 10cloud-services-team: Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10JHedden) Getting an error when trying to create the views. Maybe similar to what we saw in T193187? ` labsdb1009:~$ sudo /usr/local/sbin/maintain-vie... [13:17:11] 10Analytics, 10DBA, 10Data-Services, 10cloud-services-team: Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Marostegui) Probably, let me check [13:18:10] 10Analytics, 10DBA, 10Data-Services, 10cloud-services-team: Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Marostegui) Can you try again on labsdb1009? [13:22:47] 10Analytics, 10Community-Tech, 10Product-Analytics: Hash all pageTokens or temporary identifiers from the EL Sanitization white-list for Community Tech - https://phabricator.wikimedia.org/T226861 (10aezell) Thanks @nettrom_WMF. We will take a look. I don't know if we have this off the top of my head. [13:22:53] 10Analytics, 10DBA, 10Data-Services, 10cloud-services-team: Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10JHedden) Yep, it's working now. The views have been created on labsdb1009 and labsdb1012. [13:23:27] 10Analytics, 10DBA, 10Data-Services, 10cloud-services-team: Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Marostegui) Excellent, you can proceed on labsdb1010 and labsdb1011 too. I have created the DB there too. [13:27:20] 10Analytics, 10DBA, 10Data-Services, 10cloud-services-team: Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10JHedden) >>! In T219374#5506135, @Marostegui wrote: > Excellent, you can proceed on labsdb1010 and labsdb1011 too. I have created the DB there too. A... [13:29:25] 10Analytics, 10DBA, 10Data-Services, 10cloud-services-team: Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Marostegui) DNS and all the other operations done too? I still cannot access the database from the tools bastion: ` marostegui@tools-sgebastion-07:~$... [13:31:27] !log Kill-restart webrequest-load bundle to fix queue issue [13:31:29] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:44:18] 10Analytics, 10DBA, 10Data-Services, 10cloud-services-team: Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10JHedden) OK, the replica DNS entries are setup for hiwikisource now. I went through the full instructions for cloud services https://wikitech.wikimed... [13:52:24] Gone for kids team - I Will miss meetings (sent e-scrum) and I should be back when they are asleep [14:00:40] 10Analytics, 10DBA, 10Data-Services, 10cloud-services-team, 10Patch-For-Review: Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Marostegui) >>! In T219374#5506250, @JHedden wrote: > OK, the replica DNS entries are setup for hiwikisource now. > > I went th... [14:21:20] 10Analytics, 10DBA, 10Data-Services, 10cloud-services-team, 10Patch-For-Review: Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Marostegui) @JHedden after your +1, I have merged the change. Can you run puppet and try again to see if that index error is no... [14:23:21] 10Analytics, 10DBA, 10Data-Services, 10cloud-services-team, 10Patch-For-Review: Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Marostegui) For what is worth, I can access the hosts from bastion with `-- cluster analytics` and `--cluster web` and I can se... [14:26:58] 10Analytics, 10DBA, 10Data-Services, 10cloud-services-team, 10Patch-For-Review: Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10JHedden) >>! In T219374#5506354, @Marostegui wrote: > @JHedden after your +1, I have merged the change. Can you run puppet and t... [14:30:18] 10Analytics, 10DBA, 10Data-Services, 10cloud-services-team, 10Patch-For-Review: Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Marostegui) So everything done? From my side I can connect fine and query the views. [14:39:03] 10Analytics, 10DBA, 10Data-Services, 10cloud-services-team, 10Patch-For-Review: Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10JHedden) >>! In T219374#5506399, @Marostegui wrote: > So everything done? > From my side I can connect fine and query the views.... [14:39:52] 10Analytics, 10DBA, 10Data-Services, 10cloud-services-team, 10Patch-For-Review: Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Marostegui) Let's wait for @Bstorm then before closing this as resolved Thanks! [15:15:43] 10Analytics, 10Research: Parse wikidumps and extract redirect information for 1 small wiki, romanian - https://phabricator.wikimedia.org/T232123 (10leila) [15:45:33] a-team : levelling meeting is going on today, will send e-scrum [15:45:53] leveling... [15:52:29] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban: Archive data on eventlogging MySQL to analytics replica before decomisioning - https://phabricator.wikimedia.org/T231858 (10Nuria) >One question I have is whether you guys want to keep that database online (as in: it could be queryable) or just keep... [15:54:17] 10Analytics, 10DBA, 10Data-Services, 10cloud-services-team, 10Core Platform Team Workboards (Clinic Duty Team): Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Anomie) [16:00:26] 10Analytics, 10DBA, 10Data-Services, 10cloud-services-team, 10Core Platform Team Workboards (Clinic Duty Team): Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Anomie) >>! In T219374#5506301, @Marostegui wrote: > This might be related to {T233135} cc @Anom... [16:08:22] 10Analytics, 10DBA, 10Data-Services, 10cloud-services-team, 10Core Platform Team Workboards (Clinic Duty Team): Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Marostegui) Yep, it was intentional [16:14:59] a-team nobody is here today! cancelling grooming unless there is a swell of attendance all of the sudden :) [16:18:02] neilpquinn: yt? [16:19:08] ottomata: hey there! what's up? [16:19:13] heya [16:19:18] am writing some schema docs for MEP [16:19:21] WIP still [16:19:27] but would love some comments or thoughts [16:19:28] if you have time [16:19:30] https://wikitech.wikimedia.org/wiki/Event_Platform/Schemas [16:20:14] ottomata: yeah, I can do that! anything in particular you'd like my input on? [16:20:38] mostly if what makes sense, what doesn't, what is confusing, what is too much or too little info, etc. [16:21:08] i want to make it easy for e.g. product analytics to understand, but also be comprehensive [16:25:14] ottomata: +1 to canceling [16:31:55] ottomata: okay, will do today or tomorrow! [16:36:53] 10Analytics, 10Analytics-EventLogging, 10Better Use Of Data, 10EventBus, and 5 others: Modern Event Platform: Schema Guidelines and Conventions - https://phabricator.wikimedia.org/T214093 (10Ottomata) [16:37:27] 10Analytics, 10DBA, 10Data-Services, 10cloud-services-team, 10Core Platform Team Workboards (Clinic Duty Team): Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Bstorm) That meta_p and maintain-indexes thing will make the wiki non-functional for some purpos... [16:38:15] 10Analytics, 10EventBus, 10Product-Analytics: Review draft Modern Event Platform schema guidelines - https://phabricator.wikimedia.org/T233329 (10Ottomata) [16:38:27] ty! [16:39:31] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban: Archive data on eventlogging MySQL to analytics replica before decomisioning - https://phabricator.wikimedia.org/T231858 (10Ottomata) @Nuria, perhaps sqooping this into Hive wouldn't be too hard? [16:41:13] if we don't have moar meetings, I'll log off earlier and walk a bit :) [16:41:17] ttl! [16:41:43] 10Analytics, 10DBA, 10Data-Services, 10cloud-services-team, 10Core Platform Team Workboards (Clinic Duty Team): Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Bstorm) To provide context, we did a lot to prevent breakage of the tables during that refactor,... [16:42:49] 10Analytics, 10DBA, 10Data-Services, 10cloud-services-team, 10Core Platform Team Workboards (Clinic Duty Team): Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Marostegui) Thanks for providing the context Brooke! I will remove the temporary index tomorrow... [16:44:34] 10Analytics, 10DBA, 10Data-Services, 10cloud-services-team, 10Core Platform Team Workboards (Clinic Duty Team): Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Bstorm) Looks like https://gerrit.wikimedia.org/r/c/operations/puppet/+/538030 fixes it, but I d... [16:45:08] 10Analytics, 10DBA, 10Data-Services, 10cloud-services-team, 10Core Platform Team Workboards (Clinic Duty Team): Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Bstorm) Thanks @Marostegui [16:45:48] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban: Archive data on eventlogging MySQL to analytics replica before decomisioning - https://phabricator.wikimedia.org/T231858 (10Nuria) It is actually a lot of work to scoop this data as it is not doable in bulk, every table needs to have special mappings... [16:47:02] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban: Archive data on eventlogging MySQL to analytics replica before decomisioning - https://phabricator.wikimedia.org/T231858 (10Ottomata) Hm that sounds harder than I thought. Why can't sqoop map the MySQL schemas? [17:08:23] 10Analytics, 10DBA, 10Data-Services, 10cloud-services-team, 10Core Platform Team Workboards (Clinic Duty Team): Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Bstorm) The problem is apparently a lot of settings being moved from InitializeSettings.php to V... [17:11:17] 10Analytics: Add urlshortener to Turnilo - https://phabricator.wikimedia.org/T233336 (10ayounsi) p:05Triage→03Low [17:32:34] 10Analytics, 10Analytics-Kanban: Add more dimensions to netflow's druid ingestion specs - https://phabricator.wikimedia.org/T229682 (10ayounsi) https://gerrit.wikimedia.org/r/c/operations/puppet/+/531752 has been merged after talking to Luca, confirmed using `kafkacat` that the data is there. `"country_ip_src... [18:05:52] 10Analytics, 10DBA, 10Data-Services, 10cloud-services-team, and 2 others: Prepare and check storage layer for hi.wikisource - https://phabricator.wikimedia.org/T219374 (10Bstorm) 05Open→03Resolved a:03Bstorm Ok, I've run maintain-meta_p on all 4 replicas. Docs don't need an update because what I cha... [18:26:32] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Upgrade ua parser to latest version for both java and python - https://phabricator.wikimedia.org/T212854 (10Nuria) Super nice @JAllemandou Thanks for the docs [18:26:49] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Upgrade ua parser to latest version for both java and python - https://phabricator.wikimedia.org/T212854 (10Nuria) 05Open→03Resolved [18:26:51] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Eventlogging's processors stopped working - https://phabricator.wikimedia.org/T200630 (10Nuria) [18:27:14] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: third party domain data is getting refined - https://phabricator.wikimedia.org/T228557 (10Nuria) 05Open→03Resolved [18:27:43] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 3 others: Decomission eventlogging-service-eventbus and clean up related configs and code - https://phabricator.wikimedia.org/T232122 (10Nuria) 05Open→03Resolved [18:27:47] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10Core Platform Team Legacy (Watching / External), 10Services (watching): Modern Event Platform: Stream Intake Service - https://phabricator.wikimedia.org/T201068 (10Nuria) [18:28:17] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Create a spicerack recipe to reboot the hadoop worker nodes - https://phabricator.wikimedia.org/T225297 (10Nuria) 05Open→03Resolved [18:29:48] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: wmf_netflow cube in Turnilo missing bytes and packets measures - https://phabricator.wikimedia.org/T232226 (10Nuria) 05Open→03Resolved [18:30:30] 10Analytics, 10Analytics-Kanban: Add new mediatypes to media classification refinery code - https://phabricator.wikimedia.org/T225911 (10Nuria) 05Open→03Resolved [18:30:32] 10Analytics, 10Analytics-Kanban, 10Tool-Pageviews: Load media requests data into cassandra - https://phabricator.wikimedia.org/T228149 (10Nuria) [18:30:53] joal: did we deploy aqs? [18:31:00] we did nuria [18:31:08] joal: SO MUCH PROGRESSS [18:31:27] and turnilo has latest too, nice [18:31:28] nuria: not visible outside yet (no restbase PR) [18:31:51] joal: ya, will leave all items open until we can verify, is fdans doing teh restbase PR? [18:32:16] 10Analytics, 10Analytics-Kanban: Cleanup refinery artifacts folder from unneeded jars - https://phabricator.wikimedia.org/T231856 (10Nuria) 05Open→03Resolved [18:32:37] nuria: IIRC we're gonna wait for some data to be loaded, do some testing only internally, and when data is up-to-date send the PR [18:32:44] nuria: i’ll do the restbase or once data is loaded [18:32:46] joal: and the jar cleanup did not give us any issues [18:32:52] joal: tears come to my eyes [18:33:02] joal: sounds good [18:33:05] cc fdans [18:33:37] :) [18:39:05] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban: Archive data on eventlogging MySQL to analytics replica before decomisioning - https://phabricator.wikimedia.org/T231858 (10Nuria) Scoop can map column types but there are gotchas, like column types with '.' need to be quoted, many columns need an ex... [18:40:22] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban: Archive data on eventlogging MySQL to analytics replica before decomisioning - https://phabricator.wikimedia.org/T231858 (10Ottomata) Huh hm. [18:42:24] (03CR) 10Nuria: [C: 04-1] Correct parameters in mediarequest cassandra jobs (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/537936 (owner: 10Fdans) [18:43:21] joal: we can move tiziano's spark jobs to nice queue, no? [18:43:53] nuria: we can, but it'll not release resource (it'll prevent re-using some if it releases though) - Will do [18:45:48] nuria: I actually did not manage to - finished before [19:03:34] nuria: we agreed that the convention would be that datasets are singular no? [19:04:38] fdans: an yes sorry, mediarequest table name [19:05:03] fdans: but not "webrequest", right? [20:22:35] 10Analytics, 10Analytics-EventLogging, 10Operations, 10decommission, 10ops-eqiad: Decommission dbproxy1004 and dbproxy1009 - https://phabricator.wikimedia.org/T228768 (10Cmjohnson) a:03Jclark-ctr @Jclark-ctr wipe, remove the servers, update netbox and the google sheet. Please assign back to me once eve... [20:44:58] 10Analytics, 10DC-Ops, 10Operations, 10decommission, 10ops-eqiad: Decommission old Kafka analytics brokers: kafka1012,kafka1013,kafka1014,kafka1020,kafka1022,kafka1023 - https://phabricator.wikimedia.org/T226517 (10Cmjohnson) a:05RobH→03Jclark-ctr John, please wipe the servers, remove from the rack,...