[01:28:21] <razzi>	 !log rebalance kafka partitions for webrequest_upload partition 21
[01:28:24] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[02:07:12] <wikibugs>	 10Analytics-Clusters, 10Product-Analytics: Can't re-run failed Oozie workflows in Hue/Hue-Next (as non-admin) - https://phabricator.wikimedia.org/T275212 (10razzi) Looking at Hue, it seems it looks at the job's setting for `mapreduce.job.acl-modify-job` to determine if the kill button should be enabled. Excuse...
[03:10:32] <wikibugs>	 10Analytics-Clusters, 10observability, 10User-fgiunchedi: Setup Analytics team in VO/splunk oncall - https://phabricator.wikimedia.org/T273064 (10razzi) @fgiunchedi I have created a paging schedule meant to mirror Analytics SRE's waking hours with escalation policy "Test Escalation Policy" (we can rename onc...
[05:28:50] <wikibugs>	 10Analytics-Radar, 10Datasets-General-or-Unknown, 10Dumps-Generation, 10Product-Analytics, 10Structured-Data-Backlog (Current Work): Set up generation of JSON dumps for Wikimedia Commons - https://phabricator.wikimedia.org/T259067 (10ArielGlenn) Is there anything left to be done for this task? @Cparle go...
[07:37:52] <wikibugs>	 10Analytics-Clusters, 10Product-Analytics: Can't re-run failed Oozie workflows in Hue/Hue-Next (as non-admin) - https://phabricator.wikimedia.org/T275212 (10elukey) Good finding @razzi, never seen the option applied anywhere, but I suspect that it would need to be passed to the worflow.xml's job properties, ot...
[10:12:55] <icinga-wm>	 PROBLEM - Check the last execution of produce_canary_events on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit produce_canary_events https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[10:24:13] <icinga-wm>	 RECOVERY - Check the last execution of produce_canary_events on an-launcher1002 is OK: OK: Status of the systemd unit produce_canary_events https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[10:31:50] <wikibugs>	 (03CR) 10Awight: [C: 04-1] "I need to check for problems with previous results headers not matching input headers due to the synthetic date column." (031 comment) [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/667159 (https://phabricator.wikimedia.org/T193174) (owner: 10Awight)
[10:35:21] <wikibugs>	 (03PS2) 10Silvan Heintze: Track editor numbers split by namespace [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/671195 (https://phabricator.wikimedia.org/T275999)
[10:38:12] <wikibugs>	 (03PS1) 10Silvan Heintze: Remove unneeded alias [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/672988
[10:40:44] <wikibugs>	 (03CR) 10Silvan Heintze: "Does it make sense to link the phabricator ticket as 'Bug: T275999' in the commit message?" [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/672988 (owner: 10Silvan Heintze)
[10:46:20] <wikibugs>	 (03CR) 10Silvan Heintze: "Agreed, I removed the 'user' alias and also added a separate patch to change the active_user_changes.sql, where I copy-pasted this from: s" (031 comment) [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/671195 (https://phabricator.wikimedia.org/T275999) (owner: 10Silvan Heintze)
[10:55:50] <wikibugs>	 10Analytics-Clusters, 10observability, 10User-fgiunchedi: Setup Analytics team in VO/splunk oncall - https://phabricator.wikimedia.org/T273064 (10fgiunchedi) >>! In T273064#6920071, @razzi wrote: > @fgiunchedi I have created a paging schedule meant to mirror Analytics SRE's waking hours with escalation polic...
[11:16:33] <wikibugs>	 (03CR) 10Tonina Zhelyazkova: [C: 03+2] Track editor numbers split by namespace [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/671195 (https://phabricator.wikimedia.org/T275999) (owner: 10Silvan Heintze)
[11:19:07] <wikibugs>	 (03Merged) 10jenkins-bot: Track editor numbers split by namespace [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/671195 (https://phabricator.wikimedia.org/T275999) (owner: 10Silvan Heintze)
[11:29:51] * elukey lunch
[11:42:46] <wikibugs>	 (03PS13) 10Sharvaniharan: Image recommendations table for android [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/668244
[11:53:45] <wikibugs>	 (03CR) 10Sharvaniharan: Image recommendations table for android (032 comments) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/668244 (owner: 10Sharvaniharan)
[12:10:22] <wikibugs>	 (03CR) 10Tonina Zhelyazkova: "> Patch Set 1:" [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/672988 (owner: 10Silvan Heintze)
[12:10:28] <wikibugs>	 (03CR) 10Tonina Zhelyazkova: [C: 03+2] Remove unneeded alias [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/672988 (owner: 10Silvan Heintze)
[12:11:25] <wikibugs>	 (03Merged) 10jenkins-bot: Remove unneeded alias [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/672988 (owner: 10Silvan Heintze)
[12:17:56] <wikibugs>	 10Analytics-Radar, 10Cassandra, 10ContentTranslation, 10Event-Platform, and 10 others: Rebuild all blubber build docker images running on kubernetes - https://phabricator.wikimedia.org/T274262 (10MSantos)
[12:23:24] <wikibugs>	 (03CR) 10Sharvaniharan: "@MHolloway, I also took a stab at stream config.... Please review at your convenience : https://gerrit.wikimedia.org/r/c/operations/mediaw" [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/668244 (owner: 10Sharvaniharan)
[13:07:07] <wikibugs>	 10Analytics, 10Better Use Of Data: Optimize intermediate session length data set and dashboard - https://phabricator.wikimedia.org/T277512 (10mpopov) These also seem promising: - [[ https://github.com/apache-superset/superset-roadmap/issues/19 | Improve the SQL Lab to Explore flow UX ]] which talks about "In t...
[13:11:48] <tltaylor>	 Does anyone have opinions on our standard issue engineering build laptops in terms of the specs?  Grant is looking for a handful of people to review the proposed updated specs.  A person from this team would be good.
[13:12:40] <wikibugs>	 10Analytics, 10DBA, 10Event-Platform, 10WMF-Architecture-Team, 10Services (later): Reliable (atomic) MediaWiki event production / MediaWiki events as source of truth - https://phabricator.wikimedia.org/T120242 (10Ottomata) > Generate a "daily events snapshot"  Agree that this will be needed for sure!  It...
[13:17:59] <elukey>	 tltaylor: I am pretty happy with my current laptop (I got a new one recently), but my experience/focus has been always on SRE-related things. I can review the new specs for sure, maybe a data/sw engineer that does the same would be even better
[13:22:59] <wikibugs>	 (03PS1) 10Ottomata: Update aqs to 86e486a [analytics/aqs/deploy] - 10https://gerrit.wikimedia.org/r/673015
[13:25:11] <wikibugs>	 (03CR) 10Ottomata: [V: 03+2 C: 03+2] Update aqs to 86e486a [analytics/aqs/deploy] - 10https://gerrit.wikimedia.org/r/673015 (owner: 10Ottomata)
[13:28:01] <ottomata>	 !log deploy aqs as part of train - T207171, T263697
[13:28:05] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[13:28:05] <stashbot>	 T263697: Add more popular articles per country data to AQS  - https://phabricator.wikimedia.org/T263697
[13:28:05] <stashbot>	 T207171: Have a way to show the most popular pages per country - https://phabricator.wikimedia.org/T207171
[13:29:28] <ottomata>	 lexnasser: joal I have deployed AQS, will wait for you all to confirm things are good before proceeding with reifnery deploy
[13:29:52] <joal>	 ack ottomata - lucky timing :)
[13:30:05] <ottomata>	 eh?
[13:30:28] <joal>	 ottomata: I just arrived for kids siesta
[13:30:31] <ottomata>	 ahhhh :)
[13:30:51] <ottomata>	 by 'have deployed' i mean 'am currently deploying'
[13:31:06] <ottomata>	 ok now its done :)
[13:31:24] <joal>	 ok, will check schema in cassandra
[13:32:13] <wikibugs>	 10Analytics, 10Better Use Of Data: Optimize intermediate session length data set and dashboard - https://phabricator.wikimedia.org/T277512 (10elukey) Adding @razzi as well as the new owner of Superset :)  My 2c about celery - it is something that we may want to do during the next fiscal, if there is the desire...
[13:33:38] <wikibugs>	 (03CR) 10Dbrant: [C: 04-1] "A couple of minor things inline." (033 comments) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/668244 (owner: 10Sharvaniharan)
[13:35:07] <ottomata>	 fdans: can you bulk close all of the discussed Done kanban tasks?
[13:35:19] <ottomata>	 i think that may mostly clear out the Ops Week column
[13:35:48] <razzi>	 Good morning! elukey I'm going to start rebooting an-conf nodes if it's ok with you
[13:36:15] <fdans>	 ottomata: yis
[13:36:42] <elukey>	 razzi: sure! what procedure are you going to follow?
[13:36:42] <joal>	 ottomata: all good on my side for AQS - You can go for refinery
[13:37:30] <razzi>	 elukey: check that there is 1 leader and 2 followers, then reboot each of the followers then the leader one at a time using sre.hosts.reboot-single
[13:38:04] <elukey>	 razzi: 10+, +1 to proceed
[13:38:25] <razzi>	 :) here goes
[13:39:34] <ottomata>	 !log deploying refinery for weekly train
[13:39:35] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[13:41:41] <razzi>	 !log sudo cookbook sre.hosts.reboot-single an-conf1002.eqiad.wmnet
[13:41:43] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[13:42:27] <elukey>	 razzi: bonus point - what systems are using zookeeper on an-conf100x ?
[13:42:54] <wikibugs>	 (03CR) 10Dbrant: [C: 04-1] "I'm also not seeing a `suggestionSource` field in the schema, as specified in the task." [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/668244 (owner: 10Sharvaniharan)
[13:47:19] <razzi>	 !log sudo cookbook sre.hosts.reboot-single an-conf1003.eqiad.wmnet
[13:47:23] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[13:48:19] <razzi>	 elukey: I see in common.yaml "ZK Cluster dedicated to the Hadoop cluster (and its satellite systems)"
[13:51:03] <razzi>	 hmm ok the cumin output by node was not in numerical order, so I assumed 1001 would have been the leader, but 1003 was, not a big deal, had to restart the leader anyways, but will keep this in mind about cumin output
[13:51:19] <elukey>	 yep yep no problem
[13:52:08] <elukey>	 razzi: about systems - yarn Resource Managers and hdfs Namenodes use zookeeper for HA
[13:52:32] <elukey>	 the druid clusters have their own on-host zookeeper, but in the bright future an-conf could be used instead
[13:54:14] <razzi>	 !log sudo cookbook sre.hosts.reboot-single an-conf1001.eqiad.wmnet
[13:54:16] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[13:59:39] <ottomata>	 joal:  have deployed refinery and synced to hdfs.  have not started any new jobs...you will do that?  (i have interview now, i can do later if you tell me how)
[14:29:08] <mforns>	 heya a-team: I'm feeling a bit unwell today and will take the day off. Will send an e-scrum. See you tomorrow!
[14:29:23] <razzi>	 take care mforns !
[14:29:33] <mforns>	 thanks razzi :]
[14:32:37] <milimetric>	 ❤️
[14:48:19] <wikibugs>	 (03PS1) 10Silvan Heintze: Track editor numbers split by namespace [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/672823 (https://phabricator.wikimedia.org/T275999)
[15:03:14] <wikibugs>	 (03CR) 10Silvan Heintze: "This has been merged to master and now needs to go into "production" for deployment." [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/672823 (https://phabricator.wikimedia.org/T275999) (owner: 10Silvan Heintze)
[15:06:52] <elukey>	 razzi: time to chat about https://gerrit.wikimedia.org/r/c/operations/homer/public/+/672797/2/templates/cr/firewall.conf ?
[15:07:02] <razzi>	 elukey: yeah, bc?
[15:07:10] <elukey>	 sure, 5 mins for a coffe
[15:07:12] <elukey>	 *coffee
[15:07:13] <elukey>	 brb
[15:15:40] <elukey>	 razzi: (got sidetracked on another thing, do you mind if we chat later? sorryyy)
[15:15:57] <razzi>	 elukey: no problem, have time after standup
[15:21:04] <razzi>	 !log rebalance kafka partitions for webrequest_upload partitions 22 and 23
[15:21:06] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[15:33:14] <razzi>	 a-team anybody else coming to PA sync? https://meet.google.com/fts-jojx-wbm
[15:33:42] <milimetric>	 razzi: we're all in this other meeting
[15:33:46] <joal>	 razzi: a lot of us are in another meeting (Andrew, Dan, Fran an myself)
[15:33:58] <joal>	 razzi: can you please excuse us?
[15:34:11] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Better Use Of Data, 10Event-Platform, and 2 others: VirtualPageView Event Platform Migration - https://phabricator.wikimedia.org/T238138 (10SBisson) FYI: the #kaios-wikipedia-app is also logging virtual page views so let us know when we should start sending those to event...
[15:34:42] <milimetric>	 but razzi that means you can make all the decisions today
[15:35:46] <bearloga>	 milimetric: razzi just signed you up to make airflow available later today
[15:35:54] <bearloga>	 jk :)
[15:36:04] <milimetric>	 too late, I'm already submitting patches
[15:43:30] <bearloga>	 lol
[15:55:37] <wikibugs>	 10Analytics-Kanban, 10Better Use Of Data, 10Product-Analytics, 10Tracking-Neverending: Superset Updates - https://phabricator.wikimedia.org/T211706 (10nshahquinn-wmf)
[16:06:22] <elukey>	 a-team: standup?
[16:06:29] <milimetric>	 still in that meeting elukey 
[16:06:40] <fdans>	 a-team sorry a bunch of us are in a conflicting meeting 
[16:06:42] <joal>	 elukey: we'll be in that meeting until past
[16:28:42] <milimetric>	 a-team: batcave
[16:29:55] <elukey>	 milimetric: me and Tobias have the ML team meeting
[16:30:03] <milimetric>	 totally ok
[16:30:11] <milimetric>	 (optional ping)
[16:31:44] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Better Use Of Data, 10Event-Platform, and 2 others: VirtualPageView Event Platform Migration - https://phabricator.wikimedia.org/T238138 (10Ottomata) Oh!  Cool.  Q: are there any upcoming schema modificiations?  If not, we can migrate the schema ASAP, which would enable K...
[16:43:51] <wikibugs>	 10Analytics, 10Event-Platform, 10Inuka-Team (Kanban): KaiOSAppFirstRun Event Platform Migration - https://phabricator.wikimedia.org/T267346 (10SBisson) @nshahquinn-wmf client-side migration for all 3 schemas is merged. Could you check tomorrow and tell us if you see events / errors? Thanks!
[17:04:05] <joal>	 elukey: I think we can change the hdfs-balancer back - node-usage is centered
[17:06:39] <wikibugs>	 (03PS1) 10WMDE-leszek: Review access change [analytics/wmde/scripts] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/672824
[17:08:13] <wikibugs>	 (03CR) 10WMDE-leszek: [C: 03+2] Review access change [analytics/wmde/scripts] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/672824 (owner: 10WMDE-leszek)
[17:10:12] <elukey>	 joal: I was thinking that the current threshold seems fine, it is not that high, and it works better (also we have all 10g nodes basically)
[17:10:43] <wikibugs>	 (03CR) 10WMDE-leszek: [V: 03+2 C: 03+2] Review access change [analytics/wmde/scripts] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/672824 (owner: 10WMDE-leszek)
[17:15:42] <wikibugs>	 (03CR) 10WMDE-leszek: [V: 03+2 C: 03+2] "WMDE staff apparently does not have +2 rights on analytics/wmde/scripts repo which I believe is a mistake" [analytics/wmde/scripts] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/672824 (owner: 10WMDE-leszek)
[17:16:00] <wikibugs>	 (03CR) 10WMDE-leszek: "removed my votes as they do not do anything" [analytics/wmde/scripts] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/672824 (owner: 10WMDE-leszek)
[17:25:55] <lexnasser>	 ottomata: thanks for deploying!
[17:27:01] <lexnasser>	 anyone with access to aqs1004, can you run the following curl commands for me and report back the results (I don't have ssh access): ```curl 127.0.0.1:7232/analytics.wikimedia.org/v1/pageviews/top-per-country/AT/mobile-web/2021/02/24
[17:27:01] <lexnasser>	 curl 127.0.0.1:7232/analytics.wikimedia.org/v1/pageviews/top-per-country/AT/mobile-web/2021/02/93
[17:27:01] <lexnasser>	 curl 127.0.0.1:7232/analytics.wikimedia.org/v1/pageviews/top-per-country/AT/mobile-web/2021/02/245```
[17:28:39] <razzi>	 lexnasser: I'm on it!
[17:29:52] <razzi>	 ```
[17:29:52] <razzi>	 razzi@aqs1004:~$ curl 127.0.0.1:7232/analytics.wikimedia.org/v1/pageviews/top-per-country/AT/mobile-web/2021/02/24
[17:29:52] <razzi>	 {"type":"https://mediawiki.org/wiki/HyperSwitch/errors/not_found","title":"Not found.","method":"get","detail":"The date(s) you used are valid, but we either do not have data for those date(s), or the project you asked for is not loaded yet.  Please check https://wikimedia.org/api/rest_v1/?doc for more information.","uri":"/analytics.wikimedia.org/v1/pageviews/top-per-country/AT/mobile-web/2021/02/24"}
[17:29:52] <razzi>	 ```
[17:30:17] <razzi>	 ```eb/2021/02/24"}razzi@aqs1004:~$ curl 127.0.0.1:7232/analytics.wikimedia.org/v1/pageviews/top-per-country/AT/mobile-web/2021/02/93
[17:30:17] <razzi>	 {"type":"https://mediawiki.org/wiki/HyperSwitch/errors/invalid_request","method":"get","detail":["Given year/month/day is invalid date"],"uri":"/analytics.wikimedia.org/v1/pageviews/top-per-country/AT/mobile-web/2021/02/93"}```
[17:30:47] <razzi>	 ```razzi@aqs1004:~$ curl 127.0.0.1:7232/analytics.wikimedia.org/v1/pageviews/top-per-country/AT/mobile-web/2021/02/245
[17:30:47] <razzi>	 {"type":"https://mediawiki.org/wiki/HyperSwitch/errors/invalid_request","method":"get","detail":["Given year/month/day is invalid date"],"uri":"/analytics.wikimedia.org/v1/pageviews/top-per-country/AT/mobile-web/2021/02/245"}```
[17:30:47] <razzi>	 lexnasser: ^
[17:31:27] <razzi>	 elukey: I'm ready to deploy homer changes, Arzhel gave a +1
[17:31:36] <lexnasser>	 razzi: Perfect, thanks so much! Really helps
[17:32:52] <lexnasser>	 ottomata: the endpoint looks to be behaving as expected, I think refinery is ready to be deployed. no rush, though
[17:34:59] <elukey>	 razzi: wait a sec please :)
[17:35:19] <razzi>	 ack
[17:38:03] <elukey>	 razzi: ready
[17:38:40] <elukey>	 razzi: what procedure are you planning to do?
[17:40:05] <ottomata>	 lexnasser:  ok!
[17:40:39] <ottomata>	 lexnasser: joal, what needs to be done, a new oozie job to start?
[17:41:47] <ottomata>	 oh, should the pageview_top_percountry coordinator be restarted?
[17:42:27] <razzi>	 elukey: merge gerrit patch, run `homer 'cr*-eqiad*' diff` and if that looks good `homer  'cr*-eqiad*'  commit "update firewall rules to allow analytics traffic to access clouddb1021 - https://gerrit.wikimedia.org/r/c/operations/homer/public/+/672797"`
[17:42:27] <lexnasser>	 ottomata: I think the Cassandra keyspace should already have been created in AQS, so the Oozie job corresponding to pageview_top_percountry needs to be run
[17:42:37] <ottomata>	 is that a new job?
[17:42:58] <lexnasser>	 yeah, it hasn't been run before if that's what you're asking
[17:43:02] <ottomata>	 yes
[17:43:03] <ottomata>	 ok
[17:44:01] <ottomata>	 lexnasser:  what should the start time be?
[17:44:45] <elukey>	 razzi: you need to run puppet on cumin1001 to update the homer repo before
[17:45:05] <elukey>	 razzi: the commit msg should be a bit more specific, like 
[17:45:05] <lexnasser>	 ottomata: January 1, 2021
[17:45:29] <elukey>	 razzi: "Update the term mysql-sqoop in analytics-in4 to allow traffic to clouddb1021"
[17:45:32] <elukey>	 or something similar
[17:45:50] <elukey>	 without the link to the CR
[17:45:58] <elukey>	 that one is good when you !log in #operations
[17:46:02] <lexnasser>	 ottomata: but maybe we want to try running it for a single day to make sure it succeeds first? I don't know how you usually approach it
[17:46:03] <razzi>	 elukey: gotcha, makes sense
[17:46:58] <razzi>	 ready to begin?
[17:47:07] <lexnasser>	 ottomata: the thing is that I was unable to test the loading of the data into Cassandra, so I'm not 100% confident that everything will work, so I'm thinking it's best to take it slow to start
[17:48:30] <elukey>	 razzi: +1
[17:48:38] <ottomata>	 oh ok
[17:49:26] <ottomata>	 we can set a stop time and see
[17:49:59] <ottomata>	 going to run
[17:50:03] <ottomata>	 https://www.irccloud.com/pastebin/CxYQgjCY/
[17:50:30] <lexnasser>	 looks good to me
[17:53:43] <razzi>	 Homer diff looks good, committing
[17:55:42] <elukey>	 good
[17:56:21] <elukey>	 so happy that the network rules are so easy to deploy 
[17:56:36] <elukey>	 when I started we had to manually make the change and commit on every router
[17:56:39] <elukey>	 so scary
[17:56:56] <elukey>	 (amazing work from the sre infra team)
[17:57:00] <ottomata>	 lexnasser: 
[17:57:01] <ottomata>	 https://hue.wikimedia.org/oozie/list_oozie_coordinator/0030665-210222192802983-oozie-oozi-C/
[17:57:29] <ottomata>	 https://yarn.wikimedia.org/cluster/app/application_1615988861843_0807
[18:00:27] <razzi>	 elukey: great, thanks for the support, think we're all set
[18:00:42] <razzi>	 milimetric: want to test sqoop? Trying to see if I can find the command...
[18:00:53] <milimetric>	 razzi: testing now
[18:02:44] <ottomata>	 lexnasser:  job succeeded!
[18:02:44] <ottomata>	 https://hue.wikimedia.org/oozie/list_oozie_workflow/0030666-210222192802983-oozie-oozi-W/?coordinator_job_id=0030665-210222192802983-oozie-oozi-C
[18:02:58] <ottomata>	 if that looks good i'll relaunch it without a stop time
[18:05:41] <elukey>	 razzi: did you check with telnet if it works now?
[18:05:51] <elukey>	 I mean from stat100x -> clouddb1012 for example
[18:05:57] <lexnasser>	 ottomata: great! before you relaunch it, could you test a few curl commands on aqs1004 for me?```curl 127.0.0.1:7232/analytics.wikimedia.org/v1/pageviews/top-per-country/AT/mobile-web/2021/01/01
[18:05:57] <lexnasser>	 curl 127.0.0.1:7232/analytics.wikimedia.org/v1/pageviews/top-per-country/AT/all-access/2021/01/01
[18:05:57] <lexnasser>	 curl 127.0.0.1:7232/analytics.wikimedia.org/v1/pageviews/top-per-country/KP/all-access/2021/01/01
[18:05:57] <lexnasser>	 curl 127.0.0.1:7232/analytics.wikimedia.org/v1/pageviews/top-per-country/AT/mobile-web/2021/01/02
[18:05:57] <lexnasser>	 curl 127.0.0.1:7232/analytics.wikimedia.org/v1/pageviews/top-per-country/AT/mobile-web/2021/02/93```
[18:06:17] <razzi>	 elukey: just checked, telnet can connect
[18:07:48] <ottomata>	 https://www.irccloud.com/pastebin/GHehDZ6d/
[18:08:11] <ottomata>	 lexnasser:  it looks like those last two were meant to fail, right?
[18:08:15] <ottomata>	 i only loaded 01/01
[18:08:17] <ottomata>	 but
[18:08:20] <ottomata>	 not sure abuot
[18:08:20] <ottomata>	 curl 127.0.0.1:7232/analytics.wikimedia.org/v1/pageviews/top-per-country/KP/all-access/2021/01/01
[18:08:27] <ottomata>	 i guess KP is not a valid prjoject?
[18:08:29] <lexnasser>	 yeah, the last two were meant to fail
[18:08:39] <ottomata>	 or sorry, cuntry?
[18:08:40] <lexnasser>	 KP is north korea, so just wanted to make sure the country blacklist worked
[18:08:42] <ottomata>	 country*
[18:08:44] <ottomata>	 ahhhh
[18:08:44] <ottomata>	 cool
[18:08:51] <ottomata>	 ok so everything looks as expected?
[18:09:44] <lexnasser>	 yep! I think the job can be launched indefinitely now :)
[18:09:47] <ottomata>	 ok!
[18:10:53] <ottomata>	 !log started oozie/cassandra/coord_pageview_top_percountry_daily
[18:10:55] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[18:11:11] <ottomata>	 there she goes!
[18:11:11] <ottomata>	 https://hue.wikimedia.org/oozie/list_oozie_coordinator/0030682-210222192802983-oozie-oozi-C/
[18:11:16] <ottomata>	 oh, looks like i made it re-run 01/01
[18:11:17] <ottomata>	 oh well
[18:11:20] <ottomata>	 should be ok, right?
[18:12:04] <lexnasser>	 I hope so haha
[18:12:36] <elukey>	 razzi: perfect!
[18:14:51] <lexnasser>	 ottomata: well I think that's all that's needed for the AQS deployment. thanks so much for handling it all - I really appreciate it!!
[18:15:35] <ottomata>	 great glad that looks good!
[18:15:53] <ottomata>	 razzi:  you are going to do wikistats ui part of deploy right?
[18:16:01] <ottomata>	 (or...is already done?)
[18:20:11] <razzi>	 ottomata: I'm planning to, haven't gotten around to it yet
[18:21:22] <elukey>	 razzi: https://phabricator.wikimedia.org/T277633
[18:22:08] <elukey>	 I told Chris to proceed for this use case
[18:32:52] <elukey>	 already done, all good
[18:32:58] <elukey>	 lemme know if you have questions about the taks
[18:33:00] <elukey>	 *task
[18:33:14] <elukey>	 going afk folks, have a good rest of the day :)
[18:34:27] <razzi>	 ciao elukey!
[18:34:29] <ottomata>	 ok!
[18:46:54] * razzi afk for lunch
[19:18:00] <wikibugs>	 10Analytics-Clusters, 10Product-Analytics: Can't re-run failed Oozie workflows in Hue/Hue-Next (as non-admin) - https://phabricator.wikimedia.org/T275212 (10nshahquinn-wmf) I'm actually working on an Oozie job today! I'll test whether `mapreduce.job.acl-modify-job` works 😊
[19:19:44] <joal>	 ottomata, lexnasser - thanks a lot for having handled the oozie start and the checks :)
[19:19:54] <joal>	 indeed ottomata a rerun is safe for cassandra loading
[19:34:20] <ottomata>	 :)
[19:40:40] <wikibugs>	 10Analytics, 10Patch-For-Review: Prep for replacing jupyter conda migration - https://phabricator.wikimedia.org/T262847 (10Ottomata)
[20:15:11] <ottomata>	 !log install anaconda-wmf 2020.02~wmf3 on analytics cluster clients and workers - T262847
[20:15:14] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[20:15:14] <stashbot>	 T262847: Prep for replacing jupyter conda migration - https://phabricator.wikimedia.org/T262847
[20:40:31] <wikibugs>	 (03PS1) 10Razzi: Release 2.9.0 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/673101
[20:43:50] <wikibugs>	 (03CR) 10Sharvaniharan: Image recommendations table for android (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/668244 (owner: 10Sharvaniharan)
[20:44:07] <wikibugs>	 (03CR) 10Razzi: [C: 03+2] Release 2.9.0 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/673101 (owner: 10Razzi)
[20:45:14] <razzi>	 !log release wikistats 2.9.0
[20:45:16] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[20:45:30] <wikibugs>	 (03Merged) 10jenkins-bot: Release 2.9.0 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/673101 (owner: 10Razzi)
[21:31:20] <wikibugs>	 (03PS1) 10Razzi: Add working versions of node and npm to package.json engines [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/673128
[21:35:27] <wikibugs>	 (03CR) 10Legoktm: "See T277709, I believe I've fixed it a different way." [analytics/wmde/scripts] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/672824 (owner: 10WMDE-leszek)
[21:39:29] <wikibugs>	 (03CR) 10Razzi: "I got a gulp error until I downgraded to node 10. I think it's worth documenting a stable version that contributors can use" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/673128 (owner: 10Razzi)
[21:45:54] <wikibugs>	 10Analytics, 10Analytics-Kanban: AQS tests that expect error responses pass even if the response succeeds - https://phabricator.wikimedia.org/T273404 (10lexnasser) 05Open→03Resolved
[22:18:39] <wikibugs>	 10Analytics-Data-Quality, 10Analytics-EventLogging, 10Analytics-Radar, 10Editing-team, and 3 others: WikiEditor records all edits as platform = desktop in EventLogging - https://phabricator.wikimedia.org/T249944 (10MBinder_WMF)
[23:13:31] <wikibugs>	 10Analytics: ssh access to stat100x machines - https://phabricator.wikimedia.org/T277721 (10EYener)
[23:28:25] <wikibugs>	 (03PS1) 10Erin Yener: Modified whitelist request for MobileWikiAppFeed [analytics/refinery] - 10https://gerrit.wikimedia.org/r/673157
[23:31:33] <wikibugs>	 (03PS3) 10Erin Yener: MobileWikiAppiOSFeed Whitelist Request [analytics/refinery] - 10https://gerrit.wikimedia.org/r/666227
[23:31:35] <wikibugs>	 (03PS1) 10Erin Yener: Resolving problemating fields our discussion earlier this week [analytics/refinery] - 10https://gerrit.wikimedia.org/r/673161
[23:43:55] <wikibugs>	 (03PS2) 10Erin Yener: Modified whitelist request for MobileWikiAppFeed [analytics/refinery] - 10https://gerrit.wikimedia.org/r/673157
[23:49:48] <wikibugs>	 (03PS1) 10Erin Yener: Adjusting fields requested per privacy conversion earlier this week [analytics/refinery] - 10https://gerrit.wikimedia.org/r/673164
[23:56:59] <wikibugs>	 10Analytics, 10FR-Tech-Analytics, 10Fundraising-Backlog: Whitelist Portal and WikipediaApp event data for (sanitized) long-term storage - https://phabricator.wikimedia.org/T273246 (10EYener) Hi @mforns - thanks again for the help and advice earlier this week. I have updated the gerrit reviews with the inform...