[05:46:28] community discussion about wikistats v2: https://de.wikipedia.org/wiki/Wikipedia_Diskussion:Kurier#Eine_Reise_geht_zu_Ende [05:48:24] HaeB: thanks! Is there a away to have a summary about what it is "buggy" somewhere in english? Or maybe a phab task? [05:53:33] regarding the first comment there, it seems from the responses that user aschmidt made some mistakes himself when comparing the data, so that "buggy" claim would need to be taken with a grain of salt [05:55:11] not sure about the concerns voiced by others below [05:57:44] ah okok [05:57:59] if there is anything good that we can learn from we are all ears [06:01:43] elukey: people won't automatically file a phab ticket if they find an issue [06:03:30] also, i thought it would be useful for the team to know that there is such a discussion happening (btw, also in the section above the one i linked), and what the overall sentiment about v2 is there (even though much of the criticism may be mistaken or unfair) [06:08:51] HaeB: I am aware of people not filing tasks, I was more speaking about 'us' (like Analytics or in general WMF employee interested in the project) [06:09:14] it is interesting indeed! [06:10:17] the google translate looks legit as far as I can see, so it might be reported to a phab task if something needs to be done.. [06:10:29] I'll ping Fran/Dan later on :) [07:52:47] Morning folks :) [07:53:35] 10Analytics, 10Operations, 10User-Elukey: rack/setup/install labsdb1012.eqiad.wmnet - https://phabricator.wikimedia.org/T215231 (10Marostegui) p:05Triage→03Normal [07:54:34] 10Analytics, 10Operations, 10ops-eqiad, 10User-Elukey: rack/setup/install labsdb1012.eqiad.wmnet - https://phabricator.wikimedia.org/T215231 (10Marostegui) I have suggested to use `labsdb1012` as a hostname, as this host has the same hardware as the other labsdb1009-1011 and will be setup the same way of t... [07:58:03] 10Analytics, 10Operations, 10ops-eqiad, 10User-Elukey: rack/setup/install labsdb1012.eqiad.wmnet - https://phabricator.wikimedia.org/T215231 (10elukey) >>! In T215231#4926445, @Marostegui wrote: > I have suggested to use `labsdb1012` as a hostname, as this host has the same hardware as the other labsdb1009... [08:17:01] 10Analytics, 10Analytics-Kanban, 10Operations, 10Product-Analytics, 10Patch-For-Review: dbstore1002 Mysql errors - https://phabricator.wikimedia.org/T213670 (10Marostegui) An attempt to run mydumper for T210478 on dbstore1002 made it crash. [08:29:29] joal: bonjour! [08:30:28] Hi elukey :) [08:31:15] joal: yesterday we discovered https://phabricator.wikimedia.org/T215171 [08:38:11] wow - interesting --^ [08:59:33] 10Analytics, 10Operations, 10ops-eqiad, 10Patch-For-Review, 10User-Elukey: rack/setup/install labsdb1012.eqiad.wmnet - https://phabricator.wikimedia.org/T215231 (10jcrespo) I don't think we should setup new hosts using multi-source. [09:06:40] 10Analytics, 10Operations, 10ops-eqiad, 10Patch-For-Review, 10User-Elukey: rack/setup/install labsdb1012.eqiad.wmnet - https://phabricator.wikimedia.org/T215231 (10Marostegui) >>! In T215231#4926627, @jcrespo wrote: > I don't think we should setup new hosts using multi-source. This host will act as a cu... [09:07:43] 10Analytics, 10Operations, 10ops-eqiad, 10Patch-For-Review, 10User-Elukey: rack/setup/install labsdb1012.eqiad.wmnet - https://phabricator.wikimedia.org/T215231 (10jcrespo) > current labsdb hosts Those will be on multi-instance soon(TM). [09:08:40] 10Analytics, 10Operations, 10ops-eqiad, 10Patch-For-Review, 10User-Elukey: rack/setup/install labsdb1012.eqiad.wmnet - https://phabricator.wikimedia.org/T215231 (10Marostegui) >>! In T215231#4926632, @jcrespo wrote: >> current labsdb hosts > > Those will be on multi-instance soon(TM). Yes, that was dis... [09:19:11] 10Analytics, 10Operations, 10ops-eqiad, 10Patch-For-Review, 10User-Elukey: rack/setup/install labsdb1012.eqiad.wmnet - https://phabricator.wikimedia.org/T215231 (10elukey) From our perspective, we will use the replica only for ETL (so with Sqoop) and we'll not grant any user access to the host. Having mu... [09:19:45] joal: let me know what you think about https://gerrit.wikimedia.org/r/#/c/operations/dns/+/488004/ [09:19:53] (whenever you have time) [09:22:47] 10Analytics, 10Operations, 10Performance-Team, 10Traffic: Only serve debug HTTP headers when x-wikimedia-debug is present - https://phabricator.wikimedia.org/T210484 (10Gilles) [09:24:38] 10Analytics, 10Performance-Team: Plan navtiming data release - https://phabricator.wikimedia.org/T214925 (10Gilles) p:05Triage→03Low [09:30:14] 10Analytics: Upgrade to Spark 2.3.2 - https://phabricator.wikimedia.org/T215043 (10JAllemandou) If we upgrade, let's move to 2.4.0? [09:39:18] (03PS1) 10Joal: Update mediawiki sqooped project list [analytics/refinery] - 10https://gerrit.wikimedia.org/r/488011 (https://phabricator.wikimedia.org/T215082) [09:40:02] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: Punjabi Wikisource WikiStats 2.0 - https://phabricator.wikimedia.org/T215082 (10JAllemandou) a:03JAllemandou [09:52:21] 10Analytics, 10Analytics-Kanban, 10DBA, 10Patch-For-Review, 10User-Banyek: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10Marostegui) @elukey from what I can see, the research user is used to access wikis, not only for staging. So I guess w... [09:56:44] 10Analytics, 10Analytics-Kanban, 10DBA, 10Patch-For-Review, 10User-Banyek: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10elukey) Yes exactly, basically people only use research to consult wiki replicas and to store things in staging, sorry... [10:15:19] hi hi elukey, just checking in re the db store stuff again [10:15:36] is the data already in the new place? / can scripts be migrated already (I might have already asked you this and forogtten) [10:17:45] addshore: o/ [10:18:02] we are still completing the move, the staging db is not there yet [10:18:14] but it should be done this week hopefully [10:18:21] do you use the research user? [10:18:28] (to connect I mean) [10:19:20] elukey: https://github.com/wikimedia/puppet/blob/b347052863d4d2e87b37d6c2d9f44f833cfd9dc2/modules/statistics/manifests/wmde/graphite.pp#L38-L47 [10:19:28] yes :) [10:25:50] ah ok nice, will take a note :) [10:26:04] in the bright future we'd like to move to a more granular user scheme [10:26:29] but for the moment we'll keep the 'research' user [10:57:00] updated https://wikitech.wikimedia.org/wiki/Analytics/Data_access#MariaDB_replicas [11:01:28] going afk for a bit :) [11:29:57] :) [13:28:04] 10Analytics, 10Research, 10WMDE-Analytics-Engineering, 10User-Addshore, 10User-Elukey: Provide tools for querying MediaWiki replica databases without having to specify the shard - https://phabricator.wikimedia.org/T212386 (10elukey) I had the chance to have a chat with a lot of people during all hands, t... [13:33:38] hey team :] [13:37:54] o/ [14:15:17] 10Analytics, 10Core Platform Team, 10EventBus, 10Parsoid, and 5 others: How to surface link changes as a stream? - https://phabricator.wikimedia.org/T214706 (10Samwalton9) Thanks for your input, all - it's great to see how quickly new event streams can be set up! What are the next steps for getting the ev... [14:25:40] 10Analytics, 10Operations, 10Product-Analytics, 10User-Elukey: notebook/stat server(s) running out of memory - https://phabricator.wikimedia.org/T212824 (10elukey) @aborrero thanks a lot! As far as I can see the limits are applied for each user separately, but my use case is a bit different - I'd need to a... [15:07:30] 10Analytics, 10EventBus, 10Parsoid, 10Reading-Infrastructure-Team-Backlog, and 6 others: How to surface link changes as a stream? - https://phabricator.wikimedia.org/T214706 (10CCicalese_WMF) [15:54:02] 10Analytics, 10EventBus, 10Growth-Team, 10MediaWiki-Watchlist, and 6 others: Clear watchlist on enwiki only removes 50 items at a time - https://phabricator.wikimedia.org/T207329 (10kostajh) Came across this task while looking over my team's board. `Special:EditWatchlist/clear` is still broken. [15:56:26] 10Analytics, 10EventBus, 10Parsoid, 10Reading-Infrastructure-Team-Backlog, and 6 others: How to surface link changes as a stream? - https://phabricator.wikimedia.org/T214706 (10Nuria) @Samwalton9 we still need to see if urls are url encoded or not and hook publishing to one of the mediawiki events (I think... [16:10:15] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: Punjabi Wikisource WikiStats 2.0 - https://phabricator.wikimedia.org/T215082 (10Milimetric) p:05Triage→03High [16:13:37] (03CR) 10Milimetric: [V: 03+2 C: 03+2] Update mediawiki sqooped project list [analytics/refinery] - 10https://gerrit.wikimedia.org/r/488011 (https://phabricator.wikimedia.org/T215082) (owner: 10Joal) [16:25:03] 10Analytics, 10Analytics-Kanban: Update reportupdater to be able to query the new db cluster that will substitute 1002 - https://phabricator.wikimedia.org/T215289 (10Nuria) [16:27:33] 10Analytics: update mw scooping to be able to scoop from new db cluster - https://phabricator.wikimedia.org/T215290 (10Nuria) [16:48:59] 10Analytics, 10EventBus, 10Parsoid, 10Reading-Infrastructure-Team-Backlog, and 6 others: How to surface link changes as a stream? - https://phabricator.wikimedia.org/T214706 (10Pchelolo) We have discovered that we would need to update `LinkUpdates` class in the core to support this functionality, so it wil... [17:07:59] 10Analytics: update mw scooping to be able to scoop from new db cluster - https://phabricator.wikimedia.org/T215290 (10Nuria) See: https://phabricator.wikimedia.org/T212386 for snipet to determine the correct shard. [17:08:35] 10Analytics, 10Analytics-Kanban: Update reportupdater to be able to query the new db cluster that will substitute 1002 - https://phabricator.wikimedia.org/T215289 (10Nuria) See snipet to determine the correct shard: https://phabricator.wikimedia.org/T212386 [17:15:04] 10Analytics, 10EventBus, 10Parsoid, 10Reading-Infrastructure-Team-Backlog, and 6 others: How to surface link changes as a stream? - https://phabricator.wikimedia.org/T214706 (10mobrovac) [17:34:44] 10Analytics: Reportupdater should alert if it fails over and over - https://phabricator.wikimedia.org/T213309 (10elukey) The systemd timer has beed deployed :) [17:36:14] (03CR) 10Elukey: Clarify pid file error message (031 comment) [analytics/reportupdater] - 10https://gerrit.wikimedia.org/r/485673 (https://phabricator.wikimedia.org/T213219) (owner: 10Milimetric) [17:43:36] joal: o/ [17:43:59] if you have time (even tomorrow) can we go through the process of properly clean up hive leftovers? [17:48:07] 10Analytics: Check home leftovers of user imarlier (Ian Marlier) - https://phabricator.wikimedia.org/T213702 (10elukey) Leftovers summary: ` ====== stat1004 ====== total 452 -rw-rw-r-- 1 18334 wikidev 455751 Jan 16 2018 dec_reqs.csv -rw-rw-r-- 1 18334 wikidev 370 Jan 16 2018 dec_reqs.sql ====== stat1006 =... [17:50:46] 10Analytics: Clean up home dirs for users jamesur and nithum - https://phabricator.wikimedia.org/T212127 (10elukey) Current status: * jamesur ` ====== stat1004 ====== ls: cannot access '/srv/home/jamesur': No such file or directory ====== stat1006 ====== ls: cannot access '/srv/home/jamesur': No such file or d... [17:52:20] 10Analytics: Clean up home dirs for user mkroetzsch - https://phabricator.wikimedia.org/T214501 (10elukey) Current status: ` ====== stat1004 ====== ls: cannot access '/srv/home/mkroetzsch': No such file or directory ====== stat1006 ====== ls: cannot access '/srv/home/mkroetzsch': No such file or directory ===... [17:59:12] a-team: updated https://wikitech.wikimedia.org/wiki/Analytics/Ops_week#Have_any_users_left_the_Foundation? [17:59:27] I added a simple script to check for user's leftovers that works reasonably well [17:59:32] feel free to expand/modify it [18:02:57] Heya elukey - tomorrow evening would be better for me - does it work for you? [18:03:15] super fine [18:03:21] Great :) [18:03:39] I was going through the things to do for user deletion and thought to ask about hive db drop/deletion [18:03:47] so we can update the docs and finally purge some stuff [18:04:09] sounds good elukey - There is a trick in that regard, but should figure it out :) [18:04:30] when you have time I'd need another pair of eyes when I delete stuff for T200875 [18:04:39] to avoid any pebcak [18:05:20] No problem elukey :) [18:05:31] super, whenever you have time ping me :) [18:05:40] Let's do that now [18:05:52] batcave? [18:07:21] elukey: --^? [18:08:23] joal: didn't you say tomorrow evening? :D [18:08:43] elukey: tomorrow evening is easier, but I felt you might prefer tonight :) [18:08:51] nono [18:08:53] no urgency [18:08:58] elukey: I'll drop for diner then :) [18:09:05] have a nice evening :) [18:09:12] So you elukey [18:36:38] 10Analytics: Clean up home dirs for users jamesur and nithum - https://phabricator.wikimedia.org/T212127 (10elukey) a:05fdans→03None [18:37:06] * elukey afk! byyeee [18:37:43] byeeeee elukey :] [19:13:35] mforns: I'm just going to leave a quick message on the de.wiki wikistats thread explaining our approach and offering up the list and Phabricator as places to collaborate. [19:14:06] milimetric, I read that thread [19:14:20] mforns: want to talk about it before I respond? [19:14:34] a couple of the concerns are misunderstandings, and others are legit lacks of Wikistats2 [19:14:47] but couldn't figure up an action item [19:15:03] milimetric, was going to ask joal, but he left for a moment [19:15:27] sorry I missed what was the request of Tilman [19:15:39] mforns: yeah, I think I got the same from the rough translation. I'll comment and we can go from there [19:15:47] ok [19:15:55] mforns: as i mentioned above, it was just a heads up [19:16:08] HaeB, oh thanks! ok [19:16:25] I'm here mforns - Not sure I can help though :) [19:16:51] ...haven't had the chance to delve into the thread(s) myself to extract and verify actionable bug reports [19:16:55] HaeB, didn't log in to IRC since traveling back from SF, couldn't see your message [19:17:18] don't worry joal :] [19:17:34] mforns: https://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-analytics/20190205.txt [19:19:56] thanks [19:29:21] ok, replied here: https://de.wikipedia.org/wiki/Wikipedia_Diskussion:Kurier#Eine_Reise_geht_zu_Ende [19:31:01] Thanks milimetric :) [19:31:27] Hey analytics people, how do I get into the MariaDB replicas these days? The instructions at https://wikitech.wikimedia.org/wiki/Analytics/Data_access#MariaDB_replicas are inconsistent with each other and none of them work [19:34:21] Hi RoanKattouw - We are (Luca and Emmanuel mostly) close to the end of having new analytics replicas [19:34:41] RoanKattouw: o/ - I updated them today, those are the new aliases that will work by the end of the week [19:34:53] The page you linked contains information that is not yet valid actually (using s1-analytics-replica.eqiad.wmnet for instance) [19:34:54] I can re-add a temporary note for the current scheme [19:35:06] basically ssh to analytics-store.eqiad.wmnet [19:35:10] Ah - elukey for the best answer :) [19:35:17] Thnaks elukey [19:35:38] my bad, I'll fix the documentation with a more meaningful message tomorrow [19:36:17] RoanKattouw: what you mean that the info are inconstitent and none of them work? [19:36:23] I can see a reference to analytics-store [19:36:51] So, part of it says the server naming scheme is s1-analytics-replica and part of it refers to s1-analytics-store [19:37:07] Part of it says the port number for s1 would be 3311, but the example uses 3321 [19:37:18] So I tried all 4 of those combinations, and 0 of them worked [19:37:55] Specifically, s1-analytics-store.eqiad.wmnet does not resolve [19:38:51] And trying to access s1-analytics-replica using any port that I could guess times out [19:39:37] Ooh but analytics-store still works!~ [19:40:50] RoanKattouw: analytics-store is still the way to go for now (even if doc is almost ready for the new settings) [19:42:16] RoanKattouw: ah snap you are right there is a s1-analytics-store [19:42:19] good catch :) [19:46:06] ok amended https://wikitech.wikimedia.org/wiki/Analytics/Data_access#MariaDB_replicas and added an explanation [19:46:11] I hope it is clearer now [19:46:28] sorry for the confusion RoanKattouw [19:46:50] OK yes thank you [19:46:54] (added "We are in a transition phase, from a single host to a multi-host setup. Until T210478 is open please use analytics-store.eqiad.wmnet port 3306 to query all Wiki replicas.") [19:46:55] T210478: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 [19:47:07] One more thing: the example uses port 3321 but the port numbering scheme above suggests that that should be 3311 [19:54:18] RoanKattouw: nono that is the new scheme to use with the sX-etc.. scheme [19:54:41] dbstore1002 has a mysql instance running on 3306 (should be) [20:23:46] 10Analytics: Superset is showing "no data" for event_pageissues datasource - https://phabricator.wikimedia.org/T215342 (10mforns) [20:24:18] 10Analytics, 10Analytics-Kanban, 10Page-Issue-Warnings, 10Patch-For-Review: event_pageissues Turnilo view contains no valid data from before January 5 - https://phabricator.wikimedia.org/T214136 (10mforns) Moved the superset issue to this other task: T215342 This task can be closed then. [20:24:52] elukey, can I restart superset? [20:34:41] 10Analytics, 10Analytics-Kanban, 10Research, 10Patch-For-Review: Automate XML-to-parquet transformation for XML dumps (oozie job) - https://phabricator.wikimedia.org/T202490 (10diego) Thanks @JAllemandou ! [20:51:31] milimetric, nuria: When you have a minute, can you have a look at https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Spark#Spark_tuning_for_big_jobs please? [20:51:52] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Update big spark jobs conf with better settings - https://phabricator.wikimedia.org/T213525 (10JAllemandou) [20:52:04] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Update big spark jobs conf with better settings - https://phabricator.wikimedia.org/T213525 (10JAllemandou) Doc available here: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Spark#Spark_tuning_for_big_jobs [20:53:15] Gone for tonight folks - see you tomorrow evening [21:06:09] 10Analytics, 10Analytics-Kanban, 10Page-Issue-Warnings, 10Patch-For-Review: event_pageissues Turnilo view contains no valid data from before January 5 - https://phabricator.wikimedia.org/T214136 (10Tbayer) 05Open→03Resolved [21:24:51] elukey: Yes, I know, but the example for the new scheme is inconsistent with the rules for the new scheme [21:46:17] 10Analytics: update mw scooping to be able to scoop from new db cluster - https://phabricator.wikimedia.org/T215290 (10Milimetric) you mean T212386#4905949 right? Elukey I'm going to wait until you have that in code review, just in case you have other thoughts as you make it more official. [21:48:51] 10Analytics, 10Analytics-Kanban: Reportupdater should alert if it fails over and over - https://phabricator.wikimedia.org/T213309 (10Milimetric) a:05Milimetric→03elukey [21:52:00] 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Product-Analytics, and 5 others: Modern Event Platform: Schema Guidelines and Conventions - https://phabricator.wikimedia.org/T214093 (10Milimetric) I like the `[a-z0-9_]` restriction being up-front. It's predictable and a very fair limitation co... [22:38:31] 10Analytics, 10Analytics-Kanban, 10Page-Issue-Warnings, 10Patch-For-Review: event_pageissues Turnilo view contains no valid data from before January 5 - https://phabricator.wikimedia.org/T214136 (10Nuria) >In Superset though, I still can't seem to get rid of that "No data was returned" error - any idea wha... [22:38:56] 10Analytics: Superset is showing "no data" for event_pageissues datasource - https://phabricator.wikimedia.org/T215342 (10Nuria) 05Open→03Declined [22:39:37] 10Analytics: Superset is showing "no data" for event_pageissues datasource - https://phabricator.wikimedia.org/T215342 (10Nuria) Time ranges are off, data is there if you query for time ranges for which data exists [22:41:09] RoanKattouw: sorry about docs you just read them the week we are moving everything arround [22:42:31] No worries, it's hard to keep stuff accurate mid-migration [22:42:46] The reason I even read the docs is because the s3-analytics-slave alias (which I had been using until now) didn't work anymore [22:44:39] 10Analytics: Replace analytics mailto link in analytics.wikimedia.org - https://phabricator.wikimedia.org/T215362 (10Nuria) [23:19:18] RoanKattouw: also the current replica is hardware kaput so it might not even work at times, we will have this sorted out in the next couple of weeks [23:31:25] We did notice some hiccups recently so that explains some things