[07:08:16] Good morning! This is deploy day :) [07:16:20] holaa joal and fdans [07:19:48] 10Analytics: Quantify volume of traffic on piwik with DNT header set - https://phabricator.wikimedia.org/T199928 (10Nuria) DNT is going away as a standard, probably not worth doing this, declining. [07:19:49] 10Analytics: Quantify volume of traffic on piwik with DNT header set - https://phabricator.wikimedia.org/T199928 (10Nuria) 05Open→03Declined [07:23:18] 10Analytics, 10Wikimedia-General-or-Unknown, 10good first bug: Browser and platform stats for logged-in vs. anon users for security and product support decisions - https://phabricator.wikimedia.org/T58575 (10Nuria) There is tons of data at this time to be able to evaluate browser support in eventlogging and... [07:23:33] 10Analytics, 10Wikimedia-General-or-Unknown, 10good first bug: Browser and platform stats for logged-in vs. anon users for security and product support decisions - https://phabricator.wikimedia.org/T58575 (10Nuria) 05Open→03Resolved [07:28:43] 10Analytics, 10Analytics-Wikistats, 10Operations, 10Traffic, 10Performance-Team (Radar): Piwik JS isn't cached - https://phabricator.wikimedia.org/T230772 (10Nuria) @ema So I understand: caching pass needs to be removed (will do so) and since caching response includes an eTag. Is removing caching: pass s... [07:31:23] Hi nuria - do you think of a patch not yet merged for today's deploy? Last chance :) [07:31:36] joal: nah [07:31:56] joal: we can even skip deploying refinery-source (there is only one patch) and just do refinery [07:32:10] cc fdans [07:32:50] nuria: ok [07:32:56] Deploying refinery only then [07:34:00] !log Deploying refinery using scap [07:42:51] nuria: I'm going to wait for fdans for the restarts (this afternoon IIRC), except for mediawiki-load and geoeditors-load as those 2 have failed there regular august run [07:43:04] joal: +1 [07:43:18] joal: i can help also [07:44:06] Thanks nuria :) I'll ping if needed, but normally it's all good :) [07:44:45] I'll also take some time this morning instead of this afternoon to be in sync with Francisco [07:51:08] nuria: with your approval, I'll take some time to audit and drop jars from our refinery artifacts folder [07:51:38] nuria: It has already caused disk-space issues, and is also making deploy time immensely long for no good reason [07:52:29] joal: dropping them in the hadoop folder? [07:53:04] nuria: ensuring versions are not used by jobs and removing them from the refinery/artifacts folder [07:56:24] joal: taht makes sense but it wis not going to help with speeding up deployment [07:56:39] nuria: We have 191 jar files in refinery/artifacts/org/wikimedia/analytics/refinery [07:56:55] joal: right, makes sense to remove some , what i was trying to say [07:57:15] joal: is that the deployment does not depend on the number of artifacts there, does it? [07:57:50] nuria: deploy using scap means a full repository creation, leading to downloading the 191 jar files from archiva,every time, on every machine we deploy omnto [07:58:31] joal: wait .. there are 191 in teh repo as well? [07:59:08] nuria: we use git-fat, so we don't really feel them, but refinery/artifacts/org/wikimedia/analytics/refinery is in the repo, yes [07:59:21] And deploying means dowloading from git-fat [07:59:58] It makes years we are talking about doing it with Andrew, I really wish to do it soon [08:00:01] please :) [08:00:28] For instance: Scap deploy has started 20 minutes ago, tells me 42% done [08:00:38] Not acceptable IMO [08:01:37] joal: ya, that seems odd cause i deployed 1 month ago w/o those issues , cleaning up old jars no used makes total sense [08:02:09] joal: but it seems the slowness might come from rsync rather, no? [08:03:22] nuria: actually the slowness seems to have come from somethuing else: my term was not answering correctly [08:03:56] Tryin agian [08:04:32] joal: ok, since you are deploying on top of current deploy , * i think* you are just transfering via git fat new jars created right? [08:04:54] joal: that is not to say that we should not remove old jars , we totally should [08:05:12] nuria: I lack precise knowledge here, but redeploying means not getting new jars indeed [08:06:32] nuria: This can actually lead to issues: if deploy fails on a machine because of disk space, code being present but not the jar, redeploy will succeed without downloading jars anew, leaving the version incomplete [08:06:57] This is not the case for the deploy I just did, as full transfer was finished, but it's important to know it [08:10:33] just checked nuria - jars represent 8.8Gb in the repo :( [08:10:50] !log Deploy refinery onto HDFS [08:10:51] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [08:10:57] joal: in the /srv/deployment depo? [08:11:02] yes [08:11:43] nuria: ls -la /srv/deployment/analytics/refinery/artifacts/org/wikimedia/analytics/refinery/ | awk '{s+=$5} END {print s}' [08:14:58] joal: ya, rsync (git fat) is moving all of them every time, all them have date of today [08:16:27] indeed nuria - scap works by keeping revisions of the repo, one per deploy, and makes correct simlinks [08:16:56] meaning every new deploy is a new full download of the repo, including jars [08:17:37] joal: there is still staff i do not understand, scap cache confi is set to 2 https://github.com/wikimedia/analytics-refinery-scap/blob/master/scap.cfg#L11 [08:18:03] nuria: this means scap keeps only 2 versions (revisions) of the repo [08:18:09] on the target machines [08:18:19] This is to prevent disk-space issues [08:18:27] Luca even wanted to get down to 1 [08:18:41] Thing is: those revisions are the ones used when rollbacking [08:18:50] So keeping at least 1 or 2 is important [08:21:24] !log Kill-restart mediawiki-load and geoeditors-load jobs after corrective deploy [08:21:26] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [08:21:49] joal: if i look at the scap artifacts is about ~2G? [08:21:55] https://www.irccloud.com/pastebin/z2UH6gGJ/ [08:22:28] possibly my awk is comment is not good [08:23:01] actually no - On stat 1004 it is 8.3G [08:23:28] joal: mmm.. one sec [08:23:53] nuria: I think not all jars have been downloaded on deployment - No need to! [08:28:40] 10Analytics, 10Analytics-Wikistats, 10Operations, 10Traffic, and 2 others: Piwik JS isn't cached - https://phabricator.wikimedia.org/T230772 (10ema) >>! In T230772#5460265, @Nuria wrote: > @ema So I understand: caching pass needs to be removed Yes, and the equivalent change needs to be done for ATS too. I... [08:29:10] joal: ya, i think you are right, in the deployment machine there are just pointers to the jars [08:31:10] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Operations, and 3 others: Piwik JS isn't cached - https://phabricator.wikimedia.org/T230772 (10Nuria) [08:32:31] 10Analytics: Cleanup refinery artifacts folder from unneeded jars - https://phabricator.wikimedia.org/T231856 (10JAllemandou) [08:34:42] ok - mediawiki-history and geoeditors are unlocked - dataflow will get back into moving :) [08:41:29] 10Analytics, 10Analytics-Kanban: Cleanup refinery artifacts folder from unneeded jars - https://phabricator.wikimedia.org/T231856 (10Nuria) a:03JAllemandou [08:47:28] Gone for errand - Back in ~1h [09:09:26] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban: Archive data on eventlogging MySQL to analytics replica before decomisioning - https://phabricator.wikimedia.org/T231858 (10Nuria) [09:09:51] 10Analytics, 10Analytics-EventLogging, 10good first bug: Archive and drop the MobileOptionsTracking EventLogging MySQL table - https://phabricator.wikimedia.org/T185339 (10Nuria) This data is in hadoop in event_sanitized database, there might be some data in MySQL that predates hadoop and that would be archi... [09:10:00] 10Analytics, 10Analytics-EventLogging, 10good first bug: Archive and drop the MobileOptionsTracking EventLogging MySQL table - https://phabricator.wikimedia.org/T185339 (10Nuria) 05Open→03Declined [09:11:38] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Operations, and 3 others: Piwik JS isn't cached - https://phabricator.wikimedia.org/T230772 (10Nuria) [09:42:21] 10Analytics: Check home leftovers of smalyshev - https://phabricator.wikimedia.org/T231861 (10MoritzMuehlenhoff) [09:58:45] Back - let me know when you're in fdans :) [10:41:33] 10Analytics, 10Analytics-EventLogging, 10good first bug: Archive and drop the MobileOptionsTracking EventLogging MySQL table - https://phabricator.wikimedia.org/T185339 (10phuedx) Thanks, @Nuria and the Analytics Engineering team! [10:46:45] 10Analytics: Check home leftovers of smalyshev - https://phabricator.wikimedia.org/T231861 (10Nuria) ping @EBernhardson would you transfer to your homedir any data from stas that should not be deleted? [10:48:44] Preparing for restarts - I'll stop webrequest once the currently worked hour is done (and before next has started) [10:49:30] joal: sounds good [10:50:47] !log Kill/restart webrequest bundle [10:50:51] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:51:30] fdans: yt? [10:53:12] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Operations, and 3 others: Piwik JS isn't cached - https://phabricator.wikimedia.org/T230772 (10Nuria) I think what is left here is to restart apache for settings to take place [10:55:02] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Operations, and 3 others: Piwik JS isn't cached - https://phabricator.wikimedia.org/T230772 (10Nuria) Restarted with : sudo apache2ctl restart [10:56:37] !log Hotfixing webrequest-load job to prevent redeploying [10:56:39] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:01:48] !log Kill-restart cassandra bundle (beginning of month) [11:01:49] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:05:25] !log Kill-restart data-quality bundle [11:05:26] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:10:29] !log Fixing data-quality bundle and coord for restart [11:10:30] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:16:57] !log Kill-restart pageview_hourly, aqs_hourly and apis jobs [11:16:58] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:19:05] hey team :] [11:19:26] Hi mforns - Anapocalypse has started :) [11:19:33] ?? [11:19:40] restarts :) [11:19:54] oh! thought it was tomorrow [11:20:24] do you need help restarting? what can I do? [11:22:23] mforns: So far so good, I proceed slowly [11:23:32] joal, let me know if you want to split jobs to restart [11:24:40] mforns: Thanks for the offer - I have the process started and in place, I'll finish it :) [11:24:50] mforns: I promise to ask if I need help :) [11:24:51] ok :] [11:25:35] !log Kill-restart webrequest-druid jobs (hourly and daily) [11:25:36] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:28:42] !log Kill restart projectview_hourly job [11:28:45] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:29:46] !log Kill restart projectview_geo job [11:29:47] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:33:55] !log Kill-restart pageview-druid jobs (hourly, daily, monthly) [11:33:57] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:36:45] a-team: can someone reset my virtual environments on both notebook1003 and notebook1004? Apparently, it needs to be an ops sudoer, not sure who's in that gropu https://wikitech.wikimedia.org/wiki/SWAP#Resetting_user_virtualenvs [11:38:34] Hi neilpquinn - We need one of our ops [11:38:41] neilpquinn: Andrew should be online later today [11:39:57] !log Kill-restart mediacount jobs (load and archive) [11:39:59] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:42:16] !log Kill mediarequest-hourly (more ops to do before restarting) [11:42:18] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:43:50] !log Kill-restart virtualpageview_hourly [11:43:51] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:57:21] * joal happilly looks at the full production queue on the cluster [12:04:14] (03PS1) 10Joal: Fix bugs from changes on SLA and parameters [analytics/refinery] - 10https://gerrit.wikimedia.org/r/534145 [12:08:24] !log Restart mediarequest job after hotfix (renaming) and needed ops (table change and data move) [12:08:25] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:11:40] !log Kill-restart banner-activity jobs (daily and monthly) [12:11:42] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:14:51] 10Analytics, 10ChangeProp, 10Discovery-Search, 10EventBus, and 3 others: Better way to pause writes on elasticsearch - https://phabricator.wikimedia.org/T230730 (10mobrovac) >>! In T230730#5439794, @EBernhardson wrote: > The difficulty is that the primary usage of this pause is to allow cluster restarts to... [12:17:53] !log Manually create success-files for banner_activity monthly to start [12:17:55] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:18:56] hi nuria: just got to our place in Madrid, starting the day now [12:20:06] Hi fdans - restarts are well on the way [12:20:33] !log Kill restart browser-general and clickstream [12:20:44] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:24:35] !log Kill-restart interlanguage job [12:24:36] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:28:03] !log Fix interlanguage for naming convention [12:28:05] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:29:25] joal: is there anything I can help with? I'm looking at Hue right now as jobs get restarted [12:29:40] !log Kill-restart wikidata jobs (article_placeholder, coeditors, specialentitydata) [12:29:41] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:30:32] Hey fdans - I'm slowly poking jobs - hourly ones have been done and checked, the ones I restart now will need to wait to tomorrow (or later) for the check [12:30:41] And I correct small stuff on the way [12:33:36] neilpquinn: we should all have sudo to restart jobs but we do not (will fix) Now, your env runs as yourself so you can retstart it [12:33:41] cc joal [12:34:58] !log Kill-restart virtualpageview druid jobs (daily and monthly) [12:34:59] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:39:26] neilpquinn: i have restarted your virtual env , you should also be able to do it with : systemctl restart jupyter-your-user-singleuser.service [12:39:32] cc mforns [12:39:52] also corrected docs in wikitech https://wikitech.wikimedia.org/wiki/SWAP#Resetting_user_virtualenvs cc joal mforns [12:40:28] nuria: my understanding was that neilpquinn wanted the venv to be reset, as in reinstalled - Maybe I'm wrong though [12:40:56] joal: i think he can do that too as his env runs as his user [12:41:07] Ah - maybe [12:43:47] !log Kill-restart mobile-apps jobs (app-session, uniques daily and monthly) [12:43:48] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:46:37] (03CR) 10Nuria: [C: 03+1] "Looks good , adding @fdans so he is aware of naming of mediarequest" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/534145 (owner: 10Joal) [12:47:37] nuria: there still are 2 oozie jobs run by chelsy user, one of them being suspended [12:47:49] nuria joal if we're going for full consistency, maybe we should consider calling it mediarequest_horly? [12:47:56] hourly* [12:48:10] nuria: Let's talk about how they should be handled at standuop [12:49:36] fdans: why not :) singular/plural was kinda important to me as we almost have only singular - We however have hourly and not in various places, so I'm less obsessional :) [12:49:48] fdans: i think mediarequest is fine (unless we really do not like it) , not using plurals in table names is a general db convention that makes sense to follow [12:49:49] fdans: I have no objection though :) [12:50:17] ok, let's keep it at mediarequest [12:56:10] fdans, joaL; gerrit is a bit kaput no? [12:56:20] nuria: not tested lately [12:56:29] git st [12:56:31] oops [12:57:18] (03PS2) 10Joal: Fix bugs from changes on SLA and parameters [analytics/refinery] - 10https://gerrit.wikimedia.org/r/534145 [12:57:33] nuria: yeah, there's some slowness, Tyler is looking into it currently [12:57:34] nuria: --^ seems to work [12:58:03] there are some recurring issues where Java GC activity clogs down the current server [12:58:51] moritzm: i see, ya, there must be two hosts? ( guessing cause at times works) [13:00:27] !log Kill-restart unique_devices per_domain jobs (daily, monthly, druid daily, druid daily aggregated monthly, and druid monthly) [13:00:29] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:00:34] we have two Gerrit servers, but with only one active one, the other one is just a spare for DC failovers, if it works for some requests that might be Tyler doing his magic [13:02:38] joal: I guess since the table's name is being change there is no need for us to move the mediarequests table [13:02:50] (the one with the old partition format) [13:03:23] fdans: I have moved the data (to be sure) and dropped the table [13:04:30] (03PS3) 10Fdans: (wip) Add cassandra loading job for requests per file metric [analytics/refinery] - 10https://gerrit.wikimedia.org/r/533921 (https://phabricator.wikimedia.org/T228149) [13:05:52] (03CR) 10Ottomata: "Nice find!" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/533629 (https://phabricator.wikimedia.org/T228557) (owner: 10Nuria) [13:06:44] !log Kill-restart unique_devices per_project_familly jobs (daily, monthly, druid daily, druid daily aggregated monthly, and druid monthly) [13:06:45] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:19:33] ottomata: holaaa [13:19:50] ottomata: can you take a look at https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/534114/? [13:21:15] !log Kill-restart edit jobs (hourly and druid) [13:21:16] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:24:45] 10Analytics: Rename oozie edit_hourly job - https://phabricator.wikimedia.org/T231874 (10JAllemandou) [13:26:39] !log Kill-restart geoeditors jobs (monthly, yearly and druid) [13:26:40] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:30:26] !log Kill-restart mediawiki-history jobs (denormalize, check_denormalize, reduced, metrics, wikitext) [13:30:27] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:36:20] nuria, fdans: If ok for you I'll merge my patch of fixes, as they've been applied already [13:36:49] yea joal i didn't +1 but it looked good to me :) [13:38:17] 10Analytics: Rename oozie edit_hourly job - https://phabricator.wikimedia.org/T231874 (10Nuria) This name is now been propagated to all product teams and the dataset is widely used there so I am afraid it is too late for this change. [13:38:43] joal: I only saw one [13:39:01] joal: this one: https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/534145/, is there any other one? [13:39:25] 10Analytics: Rename oozie edit_hourly job - https://phabricator.wikimedia.org/T231874 (10JAllemandou) Could we change oozie jobs and hdfs path and keep druid name? [13:39:27] joal: let's please make sure to redeploy after merge so we are not running jobs just with local changes [13:39:28] merged nuria :) [13:39:40] ottomata: grasiasss [13:39:58] nuria: changes have been propagated manually to prod folder - Ican redeploy if you prefer [13:40:09] nuria: 1 patch, multiple fixes :) [13:40:24] joal: please yes, i think we should avoid in-place changes and rather always deploy [13:40:31] joal: great, will merge [13:40:38] (03CR) 10Nuria: [C: 03+2] Fix bugs from changes on SLA and parameters [analytics/refinery] - 10https://gerrit.wikimedia.org/r/534145 (owner: 10Joal) [13:42:15] (03CR) 10Joal: [V: 03+2 C: 03+2] "Merging" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/534145 (owner: 10Joal) [13:42:51] 10Analytics: Rename oozie edit_hourly job - https://phabricator.wikimedia.org/T231874 (10Nuria) On my opinion that will cause more confusion, no less, cc @mforns for thoughts [13:43:19] !log Deploying refinery using scap for fixes [13:43:21] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:46:52] Interesting ! [13:47:21] fdans: mediarequests data is empty starting 2019-08-14T12:00 [13:47:52] Oh! Actually no [13:48:01] I didn't catch that one :) [13:48:07] how cool [13:48:52] joal: pheeeew [13:49:04] fdans: however there still wass a bug :) [13:49:08] But we haz dataz [13:50:12] joal: a bug? you have me on pins and needles [13:50:34] fdans: https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/529911/22/oozie/mediarequests/datasets.xml [13:51:10] The uri-template miss: `timestamp=` before ${YEAR}${MONTH}${DAY}${HOUR [13:51:28] oooooohhhh [13:51:37] joal: but that's not the case in the new partitioning no? [13:51:43] i remember changing that [13:51:56] fdans: There are 2 sets of path created in the main folder - One with the correct partition scheme containing data, one with the success file only [13:52:10] correct fdans, the new partition scheme looks good [13:52:28] I guess that's why at first you were thinking there was no data joal [13:52:42] indeed - plenty empty folders [13:55:50] ok - Will start backfilling this evening after standup - need to leave for kids [14:07:35] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Operations, and 3 others: Piwik JS isn't cached - https://phabricator.wikimedia.org/T230772 (10Nuria) I can see the cache-control: max-age=604800 , I think @ema needs to change something on his end so varnish /ATS settings apply? [14:17:17] joal, nuria: I do indeed want my venv totally reinstalled, but if I can do it myself, even better. I tried `systemctl restart jupyter-neilpquinn-wmf-singleuser.service` but got `Failed to restart jupyter-neilpquinn-wmf-singleuser.service: The name org.freedesktop.PolicyKit1 was not provided by any .service files` [14:19:30] neilpquinn: k, sounds like I am wrong, you cannot retstart you own virtual env cc ottomata [14:20:36] nuria: okay, thank you for trying and updating the docs :) [14:22:08] neilpquinn: i was able to restart your env, but i think you need to recreate it right? [14:22:11] 10Analytics: Superset + Turnilo access for Verena Lindner + Raja Gumienny (WMDE) - https://phabricator.wikimedia.org/T231677 (10Verena) @Nuria My wikitech account is Verena Lindner (WMDE) [14:22:19] nuria: yes, that's correct [14:23:00] 10Analytics: Superset + Turnilo access for Verena Lindner + Raja Gumienny (WMDE) - https://phabricator.wikimedia.org/T231677 (10Nuria) @Verena is that the user name you type to log in? [14:26:37] 10Analytics: Superset + Turnilo access for Verena Lindner + Raja Gumienny (WMDE) - https://phabricator.wikimedia.org/T231677 (10Verena) @Nuria Yes [14:26:55] 10Analytics: Bot from an Azure cloud cluster is causing a false pageview spike (can we identify as bot?) - https://phabricator.wikimedia.org/T137454 (10Nuria) a:05Milimetric→03None [14:28:25] 10Analytics: Superset + Turnilo access for Verena Lindner + Raja Gumienny (WMDE) - https://phabricator.wikimedia.org/T231677 (10Nuria) a:03fdans [15:20:00] nuria: do you have anything against creating a test Cassandra keyspace to test loading from hive of request per file? [15:20:22] fdans: sounds good [15:20:30] !log creating test Cassandra keyspace "local_group_default_T_request_per_file_TEST" [15:20:51] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:01:02] ping joal standdduppp [17:38:17] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops, 10ops-eqiad: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN. - https://phabricator.wikimedia.org/T225128 (10Cmjohnson) @Ottomata Do you still need the 2nd port now that you're not doing the cloud thing? If so which... [17:38:55] 10Analytics, 10Product-Analytics: Hash all pageTokens or temporary identifiers from the EL Sanitization white-list as needed for iOS - https://phabricator.wikimedia.org/T226849 (10kzimmerman) 05Open→03Resolved a:03kzimmerman Looks like the patch was merged by @mforns (correct me if I'm wrong) [17:38:57] 10Analytics, 10Product-Analytics: Hash all pageTokens or temporary identifiers from the EL Sanitization white-list - https://phabricator.wikimedia.org/T220410 (10kzimmerman) [17:40:47] Hey all. I was curious if pageview data can be used to work out the skin being used (for example, I'm curious about people using Vector on mobile devices and people using Timeless). Is my understanding correct that we don't capture skin information in page views? If not is that something that could be solved with an X-Analytics header? [17:41:29] (or maybe that's a privacy concern since it can possibly identify people on lesser used skins) [17:45:22] 10Analytics, 10Discovery-Analysis, 10Product-Analytics, 10Patch-For-Review: Productionize per-country daily & monthly active app user stats - https://phabricator.wikimedia.org/T186828 (10kzimmerman) [18:05:58] fdans: Heya - wanna chat a minute about backfilling mediarequests? [18:06:13] joal: let's do it! [18:06:22] To the cave? [18:46:24] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops, 10ops-eqiad: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN. - https://phabricator.wikimedia.org/T225128 (10Ottomata) We don't! Just Analytics VLAN for now please. [19:04:20] fdans: ook! [19:04:52] ottomata: bc? [19:05:13] let's do IRC if ok, i'm also deploying with petr too [19:05:34] this ya? [19:05:34] https://wikitech.wikimedia.org/wiki/Analytics/Systems/AQS#Deploy_new_History_snapshot_for_Wikistats_Backend [19:06:12] yesss [19:06:32] ottomata: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/534191/ [19:07:05] is that it? just merge that ? [19:14:20] ottomata: and run puppet [19:14:33] ottomata: but yea merge the thing :) [19:14:54] puppet on aqs nodes yes? [19:15:30] ottomata: yes [19:15:54] ottomata: then I think I can just run scap deploy --l aqs1004.eqiad.wmnet --service-restart [19:16:08] and once data is verified i can scap deploy on all the hosts [19:16:10] ok, done, puppet run on aqs nodes [19:18:51] !log restarting service on aqs1004 [19:19:00] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:19:49] ottomata: data is fine, restarting on all hosts [19:21:34] gr8 [19:21:44] !log finished restart of all hosts, 2019-08 snapshot deployed [19:21:48] thank you ottomata [19:21:51] good night [19:21:56] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:28:50] goodnight fdans! [20:18:46] 10Analytics, 10Product-Analytics: Get "edits hourly" on a daily basis - https://phabricator.wikimedia.org/T231938 (10JKatzWMF) [20:28:24] 10Analytics, 10Analytics-EventLogging: EventLogging MobileWebSectionUsage has backwards incompatible changes - https://phabricator.wikimedia.org/T231939 (10Ottomata) [20:34:25] ottomata: o/ [20:34:35] leila: o/ [20:34:37] ottomata: if you're around for a chat, let me know. [20:34:42] if/when [20:34:56] leila: irc now? or hangout? [20:35:06] hangout if it works. [20:39:43] ottomata: ^ [20:45:16] leila: k gimme 1 min [21:03:58] 10Analytics, 10Analytics-EventLogging: EventLogging MobileWebSectionUsage has backwards incompatible changes - https://phabricator.wikimedia.org/T231939 (10Jdlrobson) MobileWebSectionUsage is disabled and has been disabled for some time. How is this event firing? [21:04:59] 10Analytics, 10Analytics-EventLogging: EventLogging MobileWebSectionUsage has backwards incompatible changes - https://phabricator.wikimedia.org/T231939 (10Jdlrobson) Schema was deactivated in January 2016 in T124220 [21:07:37] 10Analytics, 10Analytics-EventLogging: EventLogging MobileWebSectionUsage has backwards incompatible changes - https://phabricator.wikimedia.org/T231939 (10Ottomata) 05Open→03Invalid Oh sorry! I just saw the 'August' date on those schema changes and assumed it was this August! If that is the case then I... [21:15:09] 10Analytics, 10Product-Analytics: Hash all pageTokens or temporary identifiers from the EL Sanitization white-list for Editing - https://phabricator.wikimedia.org/T226855 (10kzimmerman) a:05Neil_P._Quinn_WMF→03MNeisler Reassigning to @MNeisler to discuss with @ppelberg as they go through priorities [21:47:17] Jdlrobson: re: skins, how do those appear in a request? [21:59:21] (03PS1) 10GoranSMilovanovic: Biases [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/534245 [22:01:09] (03CR) 10GoranSMilovanovic: [V: 03+2 C: 03+2] Biases [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/534245 (owner: 10GoranSMilovanovic) [22:01:32] (03Merged) 10jenkins-bot: Biases [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/534245 (owner: 10GoranSMilovanovic) [22:12:04] (03PS1) 10GoranSMilovanovic: Dashboard [analytics/wmde/WDCM-Biases-Dashboard] - 10https://gerrit.wikimedia.org/r/534247 [22:12:17] (03CR) 10GoranSMilovanovic: [V: 03+2 C: 03+2] Dashboard [analytics/wmde/WDCM-Biases-Dashboard] - 10https://gerrit.wikimedia.org/r/534247 (owner: 10GoranSMilovanovic) [23:58:55] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops, 10ops-eqiad: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN. - https://phabricator.wikimedia.org/T225128 (10Cmjohnson) @ottomata All the servers are moved and all of them but cloudvirtan1003 are connected to the swi...