[00:05:14] 10Analytics-Radar, 10Discovery-Search, 10MediaWiki-General, 10MW-1.36-notes (1.36.0-wmf.37; 2021-03-30): Proposal: drop avro dependency from mediawiki - https://phabricator.wikimedia.org/T265967 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF [06:02:17] 10Analytics-Radar, 10Dumps-Generation: Filename convention is not easy to follow for dumps using a `precombine` step - https://phabricator.wikimedia.org/T279055 (10ArielGlenn) We don't split data, what happens is that we generate pieces of the stubs in parallel for larger wikis and then "recombine" them togeth... [07:04:13] razzi: I just spent half an hour fixing the mariadb replication on db1108, something that Daniel followed up in this chan yesterday.. Next time please open a task if you not sure, the alert was acked and I noticed it by chance looking at my irssi logs. Without replication enabled with don't have backups from db1108 :) [07:06:38] also, an-worker1080 was reported down (by Daniel), and it is still down now.. Let's find a good way to follow up on alerts, otherwise we may end up into inconsistent states. In this case one worker down is not a big problem (even if this is a journal node, but we have 5), but if it was another node it may have been more problematic (say we need to depool or take other actions) [07:28:35] !log manual fix for an-worker1080's interface in netbox (xe-4/0/11), moved by mistake to public-1b [07:28:38] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [07:28:44] all right an-worker1080 is back [09:02:31] 10Quarry, 10Patch-For-Review: Mitigate broken login due to Nonce already used - https://phabricator.wikimedia.org/T275277 (10Framawiki) 05Open→03Resolved {T272319} was closed [09:03:05] (03Abandoned) 10Framawiki: Mitigate broken login due to Nonce already used [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/665472 (https://phabricator.wikimedia.org/T275277) (owner: 10Framawiki) [09:25:14] 10Analytics-Radar, 10Article-Recommendation, 10Patch-For-Review: Generate article recommendations in Hadoop for use in production - https://phabricator.wikimedia.org/T210844 (10Aklapper) @Ottomata et al: All related patches in Gerrit merged or abandoned. Is there more to do in this task? Or can task status b... [09:25:54] 10Analytics-Clusters, 10Analytics-Radar, 10Article-Recommendation: Generate article recommendations in Hadoop for use in production - https://phabricator.wikimedia.org/T210844 (10Aklapper) [10:28:27] 10Analytics-Clusters, 10Patch-For-Review: Upgrade the rest of the Hadoop test cluster to Buster - https://phabricator.wikimedia.org/T278422 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts: ` ['an-test-master1002.eqiad.wmnet'] ` The log can be found in `/... [10:38:01] 10Analytics-Clusters, 10Patch-For-Review: Upgrade the rest of the Hadoop test cluster to Buster - https://phabricator.wikimedia.org/T278422 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts: ` ['an-test-master1002.eqiad.wmnet'] ` The log can be found in `/... [11:13:29] 10Analytics-Clusters, 10Patch-For-Review: Upgrade the rest of the Hadoop test cluster to Buster - https://phabricator.wikimedia.org/T278422 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['an-test-master1002.eqiad.wmnet'] ` and were **ALL** successful. [11:14:36] * elukey lunch! [12:59:01] 10Analytics-Clusters, 10Patch-For-Review: Upgrade the rest of the Hadoop test cluster to Buster - https://phabricator.wikimedia.org/T278422 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts: ` ['an-test-master1001.eqiad.wmnet'] ` The log can be found in `/... [13:03:54] just in time! https://www.cloudera.com/downloads/paywall-expansion.html [13:04:00] I wasn't aware of this [13:04:17] "Effective January 31, 2021, all Cloudera software requires a valid subscription to that software in order to access it. This includes all legacy versions of Cloudera products, including Apache Hadoop (CDH), Hortonworks Data Platform (HDP), Data Flow (HDF/CDF), and Cloudera Data Science Workbench (CDSW). " [13:10:05] * elukey dances [13:25:47] 10Analytics-Clusters, 10Patch-For-Review: Upgrade the rest of the Hadoop test cluster to Buster - https://phabricator.wikimedia.org/T278422 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['an-test-master1001.eqiad.wmnet'] ` and were **ALL** successful. [13:42:18] 10Analytics-Clusters, 10Analytics-Radar, 10Article-Recommendation: Generate article recommendations in Hadoop for use in production - https://phabricator.wikimedia.org/T210844 (10Ottomata) @bmansurov to answer? [14:14:59] 10Analytics-Clusters, 10Patch-For-Review: Upgrade the rest of the Hadoop test cluster to Buster - https://phabricator.wikimedia.org/T278422 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts: ` ['an-test-coord1001.eqiad.wmnet'] ` The log can be found in `/v... [15:00:26] hi! quick question... does anyone know if the _previous_ value of WMF-Last-Accessed cookie currently available to client-side JS anywhere? thanks much in advance! [15:10:34] 10Analytics-Clusters, 10Patch-For-Review: Upgrade the rest of the Hadoop test cluster to Buster - https://phabricator.wikimedia.org/T278422 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['an-test-coord1001.eqiad.wmnet'] ` and were **ALL** successful. [15:16:11] AndyRussG: no idea! Maybe better to send an email to analytics@ or to open a task! [15:16:41] @elukey ah okok thanks much! :) (not urgent also btw) [15:40:04] interesting, I reimaged an-test-coord1001 to buster and the metastore doesn't come up due to libmysql-java vs libmariadb-java [15:40:09] in the classpath [15:46:22] Any recommendations for sending an email from a stat host? Need to run a script and notify myself when it's done. Is there a command line utility I can use? [15:46:37] or is the python code here https://wikitech.wikimedia.org/wiki/Analytics/Systems/Jupyter/Tips#Sending_emails_from_within_a_Notebook the best way? [15:56:00] 10Analytics: Produce a list of wiki projects ranked by number of eligible voters in Board elections - https://phabricator.wikimedia.org/T278815 (10mforns) > I have what I think are good news. :) > > While it is exciting to get more accurate results and I have been the first one proposing to fine tune the query..... [15:57:38] (03CR) 10Mforns: [V: 03+2 C: 03+2] "Cool, looks good, merging." [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/676333 (https://phabricator.wikimedia.org/T193170) (owner: 10Awight) [16:01:27] bearloga: hi! Using the localhost smtp is fine in my opinion [16:01:59] elukey: hi! :) thank you, I'll try that! [16:04:04] oh wait, smtp isn't available [16:07:31] bearloga: what do you mean? [16:09:32] elukey: there doesn't appear to be an smtp (or ssmtp) command [16:09:54] at least not on stat1008 [16:10:12] not sure if that's what you meant by localhost smtp [16:13:01] bearloga: ahhh okok no I mean to use any lib with the localhost endpoint [16:13:04] if easy [16:13:11] otherwise lemme check what command we have for mails [16:14:47] bearloga: we have mailx [16:17:11] (going afk for a bit, bbiab) [16:19:21] !log all the Hadoop test cluster on Debian Buster [16:19:24] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:20:05] (03CR) 10Mforns: "I think this patch looks good!" (031 comment) [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/676299 (https://phabricator.wikimedia.org/T193169) (owner: 10Awight) [16:20:52] Hi all, good morning [16:22:32] Thanks for looking into superset replication elukey, I'm wondering if I should have not acked the alarm, since I didn't know how to fix it [16:28:29] !log rebalance kafka partitions for webrequest_text partitions 9,10 [16:28:31] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:02:14] 10Analytics-Radar, 10Growth-Scaling, 10Product-Analytics, 10Growth-Team (Current Sprint): Growth: shorten welcome survey retention to 90 days - https://phabricator.wikimedia.org/T275171 (10nettrom_WMF) @Tgr : I checked most of the wikis that we're deployed to and noticed it was not run on French Wikipedia.... [17:53:02] razzi: hey! Yes next time please open a task if the issue doesn't auto-resolve, acking an alert has the potential risk of forgetting it [17:53:12] since it doesn't pop up in icinga [17:54:25] 10Analytics-Clusters, 10Patch-For-Review: Upgrade the rest of the Hadoop test cluster to Buster - https://phabricator.wikimedia.org/T278422 (10elukey) All the hadoop test cluster nodes should be on Buster now, leaving this open for a couple of days to see if any issue comes up. [17:57:39] razzi: same thing for an-worker1080, Daniel mentioned that it was down, open a task with me and Andrew in Cc in case it happens and you don't know what's happening [17:58:19] the issue for an-worker1080 was a little sneaky, the dcops team accidentally changed the vlan for its port on the switch (instead for a new host) [17:58:44] but opening a task is good so we can track what it has been done and future steps [17:58:57] it is also fine to ping me or Andrew in here with a sort of handober [17:59:00] *handover [17:59:27] like "I tried XYZ but this was the issue, if you can check when you are online etc..) [17:59:30] " [18:00:35] going afk, have a nice weekend folks! [18:37:44] 10Analytics-Radar, 10Better Use Of Data, 10Product-Analytics, 10Product-Data-Infrastructure, and 2 others: prefUpdate schema contains multiple identical events for the same preference update - https://phabricator.wikimedia.org/T218835 (10Edtadros) a:05Edtadros→03ovasileva === Test Result - Prod **Stat... [18:39:16] 10Analytics-Radar, 10Better Use Of Data, 10Product-Analytics, 10Product-Data-Infrastructure, and 2 others: prefUpdate schema contains multiple identical events for the same preference update - https://phabricator.wikimedia.org/T218835 (10Edtadros) [19:26:52] i comitted the cardinal sin of querying pageview_hourly w/o partitions, i have forgotten it all [19:44:30] 10Analytics-Radar, 10Growth-Scaling, 10Product-Analytics, 10Growth-Team (Current Sprint), 10Patch-For-Review: Growth: shorten welcome survey retention to 90 days - https://phabricator.wikimedia.org/T275171 (10Tgr) Oops, thanks for catching that, @nettrom_WMF. I'm adding a test to make sure the dblist inc... [20:45:24] (03CR) 10Mholloway: [WIP] Metrics Platform context attribute schema fragment (033 comments) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/676392 (https://phabricator.wikimedia.org/T276379) (owner: 10Jason Linehan) [21:15:37] 10Analytics-Radar, 10SRE, 10Traffic, 10Wikimedia-General-or-Unknown: Cookie “WMF-Last-Access-Global” has been rejected for invalid domain. - https://phabricator.wikimedia.org/T261803 (10Krinkle) [21:15:39] 10Analytics-Radar, 10Domains, 10SRE, 10Traffic, 10Wikimedia-General-or-Unknown: WMF third-party cookies rejected - https://phabricator.wikimedia.org/T262882 (10Krinkle) [21:16:50] 10Analytics-Radar, 10SRE, 10Traffic, 10Wikimedia-General-or-Unknown: Requests for /static get an invalid WMF-Last-Access cookie for wikipedia.org on non-Wikipedia requests - https://phabricator.wikimedia.org/T261803 (10Krinkle) [21:18:16] 10Analytics-Radar, 10SRE, 10Traffic, 10Wikimedia-General-or-Unknown: Requests for /static get an invalid WMF-Last-Access cookie for wikipedia.org on non-Wikipedia requests - https://phabricator.wikimedia.org/T261803 (10Krinkle) This happens because our traffic layer shares the caches for `/static` across a... [21:18:38] 10Analytics-Radar, 10SRE, 10Traffic, 10Wikimedia-General-or-Unknown: Requests for /static get an invalid WMF-Last-Access cookie for wikipedia.org on non-Wikipedia requests - https://phabricator.wikimedia.org/T261803 (10Krinkle) ` $ curl -I 'https://commons.wikimedia.org/static/favicon/commons.ico' HTTP/2 2... [22:23:15] 10Analytics, 10Data-release, 10Privacy Engineering, 10Research, 10Privacy: Evaluate a differentially private solution to release wikipedia's project-title-country data - https://phabricator.wikimedia.org/T267283 (10Nuria) Had more time to review this and appreciate the coolness of this: "I apply a flex...