[00:40:07] 10Quarry: Letter G should fit in box for title - https://phabricator.wikimedia.org/T216581 (10Zoranzoki21) [00:44:11] 10Quarry: Letter G should fit in box for title - https://phabricator.wikimedia.org/T216581 (10Zoranzoki21) [01:04:49] 10Quarry, 10User-Zoranzoki21: Letter G should fit in box for title - https://phabricator.wikimedia.org/T216581 (10Zoranzoki21) a:03Zoranzoki21 [01:24:37] 10Analytics, 10Dumps-Generation, 10Wikidata: Update wikidata-entities dump generation to fixed day-of-month instead of fixed weekday - https://phabricator.wikimedia.org/T216160 (10Nicolastorzec) I'm also interested in the specific reasons why the update frequency needs to be changed, i.e. beside streamlining... [01:26:57] 10Analytics, 10ExternalGuidance, 10Product-Analytics, 10MW-1.33-notes (1.33.0-wmf.18; 2019-02-19), 10Patch-For-Review: Measure the impact of externally-originated contributions - https://phabricator.wikimedia.org/T212414 (10chelsyx) >>! In T212414#4963359, @santhosh wrote: >>>! In T212414#4963003, @chels... [01:28:11] 10Analytics, 10CirrusSearch, 10EventBus, 10WMF-JobQueue, and 4 others: EventBus or CirrusSearch: DomainException from line 353 of /srv/mediawiki/php-1.33.0-wmf.9/vendor/firebase/php-jwt/src/JWT.php: Unknown JSON error: 5 - https://phabricator.wikimedia.org/T212335 (10Smalyshev) > In this concrete instance... [01:44:16] 10Analytics, 10CirrusSearch, 10EventBus, 10WMF-JobQueue, and 4 others: EventBus or CirrusSearch: DomainException from line 353 of /srv/mediawiki/php-1.33.0-wmf.9/vendor/firebase/php-jwt/src/JWT.php: Unknown JSON error: 5 - https://phabricator.wikimedia.org/T212335 (10Pchelolo) a:03Pchelolo This is a bug... [01:48:39] 10Analytics: Clean up home dirs for user mkroetzsch - https://phabricator.wikimedia.org/T214501 (10Smalyshev) I've copied/linked all data sets that need to be published, and I think I've preserved all things that may need to be preserved, so the only things we're now waiting answers to are: 1. We have some raw... [04:52:53] 10Analytics, 10ExternalGuidance, 10Product-Analytics, 10MW-1.33-notes (1.33.0-wmf.18; 2019-02-19), 10Patch-For-Review: Measure the impact of externally-originated contributions - https://phabricator.wikimedia.org/T212414 (10santhosh) >>! In T212414#4967425, @chelsyx wrote: > And I was trying to see wheth... [05:28:53] 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Product-Analytics, and 5 others: Modern Event Platform: Schema Guidelines and Conventions - https://phabricator.wikimedia.org/T214093 (10Nuria) >@EBernhardson, could we add a cookies (map) field and only populate it with safe cookies, e.g. WMF-Las... [05:49:31] 10Analytics, 10ExternalGuidance, 10Product-Analytics, 10MW-1.33-notes (1.33.0-wmf.18; 2019-02-19), 10Patch-For-Review: Measure the impact of externally-originated contributions - https://phabricator.wikimedia.org/T212414 (10Nuria) The graphana dashboard has a lot of problems, if you export the data to cs... [06:04:22] (03CR) 10Nuria: "I am still confused as to what an "artificial Id" is, is there anywhere in the code that gets explained?" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/485710 (https://phabricator.wikimedia.org/T213603) (owner: 10Joal) [06:33:07] 10Analytics: Deprecate Spark 1.6 in favor of Spark 2.x only - https://phabricator.wikimedia.org/T212134 (10elukey) @Ottomata should we proceed? [07:34:16] 10Analytics, 10ExternalGuidance, 10Product-Analytics, 10MW-1.33-notes (1.33.0-wmf.18; 2019-02-19), 10Patch-For-Review: Measure the impact of externally-originated contributions - https://phabricator.wikimedia.org/T212414 (10chelsyx) > You are testing it in correct way. Perhaps there is delay in getting t... [07:40:02] Good morning elukey :) [07:41:21] o/ [07:42:14] We can merge purge when you want - Thanks for keeping my `-` / `_` difference :) [07:42:20] elukey: --^ [07:42:37] yeah sorry I thought to have all '-' but then I reverted :) [07:42:40] all right merging! [07:45:31] Thanks :) [07:49:52] Wed 2019-03-20 06:00:00 UTC 3 weeks 6 days left n/a n/a mediawiki-xmldumps-pages_meta_history.timer mediawiki-xmldumps-pages_meta_history.service [07:49:56] joal: --^ [07:50:28] \o/ elukey :) [07:50:42] elukey: can we try a manual run (we are the 20th :) [07:50:43] ? [07:57:36] yep! [07:57:41] feel free to go [07:59:22] reading again the timers-list, I realize I should have had put 'drop' in the timer name :S [07:59:25] My bad [07:59:53] ah yes didn't see it as well [08:00:00] we can amend or leave it in this way [08:00:23] elukey: I'm gonna push a small patch - Will easier in further maintenance [08:00:54] ack, need to verify if puppet cleans up the old units [08:01:02] my bet is no, so I'll clean up afterwards :D [08:01:04] will be interesting ! [08:01:12] pretty sure it doesn't [08:01:34] elukey: for a single manual run, shall I use `systemctl start mediawiki-xmldumps-pages_meta_history.service` ? [08:02:16] 10Quarry: Letter G should fit in box for title - https://phabricator.wikimedia.org/T216581 (10Zoranzoki21) a:05Zoranzoki21→03None Oh, what I clicked :) [08:02:31] joal: yep that one is fine [08:02:50] you run directly the service unit since the timer itself is not useful in this case [08:04:39] elukey: Yes - I was mostly wondering about the "start" aspect of a non-daemon job [08:05:21] joal: it basically execute the ExecStart command [08:06:06] elukey: failure :( [08:06:14] The name org.freedesktop.PolicyKit1 was not provided by any .service files [08:19:29] here I am sorry [08:19:51] joal: did you use 'sudo' ? [08:20:00] I tried, but it asks me for a pass [08:20:38] ah yes makes sense [08:20:47] I just checked the sudoers [08:21:41] ok going to merge your rename change [08:23:16] so the main issue is that you are not whitelisted to sudo systemctl start anything [08:23:20] but only specific daemons [08:23:35] Ah ! Makes sense :) [08:24:27] either we ask to SRE a full systemctl usage, or we rename all the units that we want to be able to restart with a prefix and then whitelist only that [08:24:34] but it might be confusing [08:28:28] elukey@an-coord1001:~$ systemctl list-timers | grep xml [08:28:28] Wed 2019-03-20 06:00:00 UTC 3 weeks 6 days left n/a n/a mediawiki-drop-xmldumps-pages_meta_history.timer mediawiki-drop-xmldumps-pages_meta_history.service [08:28:34] (had to clean up the old one) [08:28:43] joal: for the moment I can start it [08:28:59] need to file a request for SRE Access that will be evaluated next monday [08:29:15] Thanks elukey :) I have no opinion about the 2 solutions you mentioned above [08:31:11] Feb 20 08:30:53 an-coord1001 systemd[1]: Started Drop xmldumps pages_meta_history data from HDFS after 80 days.. [08:31:14] Feb 20 08:30:53 an-coord1001 systemd[1]: mediawiki-drop-xmldumps-pages_meta_history.service: Main process exited, code=exited, status=1/FAILURE [08:33:45] joal: I am checking the script, do we need the '=' after the parameters? [08:34:16] elukey: I removed them on purpose to be homogeneous with other scripts - Thought it was ok - Maybe it's not? [08:34:27] some of them have it [08:34:30] and some not [08:34:49] Wow this is me not being attentive enough - please excuse me - will review [08:34:58] the log doesn't say much though [08:35:01] mmmm [08:35:21] also joal there is a logging change that I'd like to do [08:35:38] sure [08:35:52] am I correct in saying that without '--log-file' everything will be emitted via stdout? [08:36:13] just saw the culprit for = (`base-path`) [08:36:14] because now the systemd timer puppet logic can handle syslog log to a file etc.. [08:36:19] I think so elukey [08:36:37] and we have a convention, namely /var/log/refinery/name-of-the-timer/etc.log.. [08:36:45] but now it is directly under /var/log/refinery [08:36:52] elukey: if you prefer logs to be managed by systemd I have no objection - I used a logfile cause thr script allows for it and this how other jobs were doing [08:37:10] yeah I think it was pre-systemd-timers [08:37:18] I'll file a task for it [08:44:51] 10Analytics, 10Operations, 10Product-Analytics, 10Patch-For-Review, 10User-Elukey: notebook/stat server(s) running out of memory - https://phabricator.wikimedia.org/T212824 (10elukey) Today I checked notebook1003 using the command `systemd-cgls memory`, that should show how the cgroups for memory setting... [08:52:55] elukey: I have some brass-news for you - https://www.youtube.com/watch?v=bsq6TM41IFs [08:55:05] thanks :) [09:00:20] 10Analytics: Staging environment for upgrades of superset - https://phabricator.wikimedia.org/T212243 (10elukey) @Ottomata I have a proposal, let me know what you think about it: * we create a ganeti instance called `analytics-tool-test1001` (or whatever name we think it is appropriate :) * we deploy superset a... [09:01:30] 10Analytics, 10Analytics-Kanban: Create staging environment for superset - https://phabricator.wikimedia.org/T213923 (10elukey) [09:01:34] 10Analytics: Staging environment for upgrades of superset - https://phabricator.wikimedia.org/T212243 (10elukey) [09:01:56] 10Analytics, 10User-Elukey: Staging environment for upgrades of superset - https://phabricator.wikimedia.org/T212243 (10elukey) a:03elukey [09:03:32] PROBLEM - Check the last execution of mediawiki-drop-xmldumps-pages_meta_history on an-coord1001 is CRITICAL: CRITICAL: Status of the systemd unit mediawiki-drop-xmldumps-pages_meta_history [09:04:05] 10Analytics-Kanban, 10Patch-For-Review: Coordinate work on minor changes for Edit Data Quality - https://phabricator.wikimedia.org/T213603 (10JAllemandou) [09:04:23] Arf elukey :( --^ [09:04:44] 10Analytics, 10Analytics-Kanban, 10Product-Analytics: Superset's rolling average feature results in error message - https://phabricator.wikimedia.org/T213488 (10elukey) @jlinehan sorry for the lag in answering :( I just updated T212243 with a plan to create a staging environment for Turnilo/Superset. If my... [09:05:22] joal: yep but we knew it no? :) [09:05:38] do you want me to try removing (manually) the = and restart? [09:05:45] to see if it runs correctly [09:05:52] then we could do a follow up patch [09:06:20] if you don't mind that's be great - also, what about logging? Shall i remove it? [09:07:37] nono let's do it another time [09:09:06] ok elukey [09:09:19] elukey: waiting for you confirmation about the = sign, then sending a patch [09:10:18] mmm it doesn't seem working [09:10:24] and this time I don't even see logs [09:10:29] :( [09:10:45] lemme run it manually [09:11:32] elukey: the log file says dry-run [09:12:01] elukey: but not all logs seems to be present in the log file (weird) [09:12:06] joal: that was the first run [09:12:16] ah [09:12:45] removed the --log-file [09:12:45] elukey@an-coord1001:~$ sudo -u hdfs /usr/local/bin/refinery-drop-mediawiki-xmldumps-pages_meta_history [09:12:49] /usr/local/bin/refinery-drop-mediawiki-xmldumps-pages_meta_history: line 13: --skip-trash: command not found [09:13:19] ah no nevermind [09:13:23] it's me [09:13:26] did I forget ah [09:13:29] ok [09:14:12] elukey: I assume there must be a diff in checksum, but I don't understand why :( [09:15:05] 10Quarry: Letter G should fit in box for title - https://phabricator.wikimedia.org/T216581 (10Aklapper) 05Open→03Stalled I do not know what this bug report is about, what "box for title" and where to see a letter G. In general, please be way more specific and always read and follow https://www.mediawiki.org... [09:17:31] ahhh removing the = changed the checksum [09:17:37] but it should emit an error [09:17:41] otherwise it is confusing [09:17:53] mwarf :( [09:17:58] ok - will submit a patch [09:18:10] I'll also remove the = for the checksum param [09:19:50] wait wait joal [09:19:59] k [09:20:04] I am trying to execute it without the --execute parameter [09:20:10] it should dry-run right? [09:20:18] correct [09:20:25] but it still doesn't emit anything and return 1 [09:20:54] same for me [09:21:26] elukey: I think it needs the = sign or single-letters params [09:21:36] elukey: shall I try with =? [09:21:43] or do you prefer single letters? [09:23:06] AHHHH ! I think I know - I made a dummy mistake elukey [09:24:18] /usr/bin/python3 :) [09:24:21] Ok got it working [09:24:31] me too [09:24:36] * joal hides his tears [09:24:59] it should work though [09:25:57] ah the script uses /usr/bin/env python so python2 [09:26:03] weird though [09:26:12] there is something to follow up probably [09:26:18] we shouldn't use python2 in there [09:36:20] 10Quarry: Letter G should fit in box for title - https://phabricator.wikimedia.org/T216581 (10Zoranzoki21) 05Stalled→03Open a:03Zoranzoki21 >>! In T216581#4967910, @Aklapper wrote: > I do not know what this bug report is about, what "box for title" and where to see a letter G. > > In general, please be wa... [09:41:42] joal: [09:41:43] 2019-02-20T09:41:17 INFO Starting EXECUTION. [09:41:43] 2019-02-20T09:41:19 INFO Removing 1 directories for tree depth 1. [09:41:43] 2019-02-20T09:41:21 INFO EXECUTION finished. [09:41:45] looks good [09:42:14] \o/ !!! thanks a lot for helping me debug this elukey [09:43:02] 10Quarry: Letter G should fit in box for title - https://phabricator.wikimedia.org/T216581 (10Zoranzoki21) | Before | After | {F28250591} | {F28250593} [09:43:07] elukey: I confirm we have only 3 folders left in xmldumps folder [09:43:09] <3 [09:43:15] (03PS1) 10Zoranzoki21: view.css: Fixed problem with the display of certain letters [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/491718 (https://phabricator.wikimedia.org/T216581) [09:43:17] Many thanks again elukey [09:44:11] (03CR) 10Zoranzoki21: "See how it looks before and after this change: https://phabricator.wikimedia.org/T216581#4967952" [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/491718 (https://phabricator.wikimedia.org/T216581) (owner: 10Zoranzoki21) [09:44:24] RECOVERY - Check the last execution of mediawiki-drop-xmldumps-pages_meta_history on an-coord1001 is OK: OK: Status of the systemd unit mediawiki-drop-xmldumps-pages_meta_history [09:45:17] (03CR) 10Joal: [V: 03+2 C: 03+2] "Let's deploy this." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/491252 (owner: 10Joal) [09:46:40] joal: can I ask you a review https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/491520/ ? [09:46:54] reading elukey [09:50:01] elukey: first question: what is the x1 section? [09:51:00] it is where the extension tables live, like FlowDB, etc.. [09:51:15] Interesting !!! Thanks for that ?) [09:51:16] Interesting !!! Thanks for that :) [09:51:29] it is called extension1 IIUC [09:51:36] makes sense [09:52:02] Might knowledge of Xtensions is far from Xtensive [09:52:15] s/ight/y/g [09:56:49] elukey: I have questions for you - I'm assuming you use argparse instead of docopt to get the unknown parameters? [09:58:53] 10Quarry, 10Patch-For-Review: Letter g cut off at bottom in #title - https://phabricator.wikimedia.org/T216581 (10Aklapper) [09:59:37] joal: yep exactly, I added a comment at the top since Dan didn't find a way to use docopt [09:59:45] I personally prefer argparse though [10:00:05] 10Quarry, 10Patch-For-Review: Letter g cut off at bottom in #title - https://phabricator.wikimedia.org/T216581 (10Aklapper) Problem does not happen in Firefox 65 on Fedora29. (For future reference, please include browser information when filing CSS issues - thanks!) [10:04:37] elukey: makes sense - A couple of comment I'll do [10:20:01] sure :) [10:20:22] if the train is going to be deployed tomorrow I'll be able to ship the tool via puppet [10:20:36] (a wrapper to avoid to define PYTHONPATH etc..) [10:22:01] yessir [10:42:46] 10Analytics, 10Patch-For-Review: Give access to Superset to Pau - https://phabricator.wikimedia.org/T211036 (10elukey) Hello @Pginer-WMF, sorry for the delay in following up on this task. Once the above patch gets reviewed/merged I'll be able to add your username to superset. [10:43:04] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Give access to Superset to Pau - https://phabricator.wikimedia.org/T211036 (10elukey) a:03elukey [10:44:28] one comment (for everybody) - we'd need to be more vigilant to answer to the above access requests --^ [10:44:47] Last comment was Dec 10th [10:45:03] yes elukey :( I assume it was not in incoming anymore, and probably not in kanban? [10:46:37] yeah it was in ops excellence [10:46:59] these ones should probably go now in the ops duty column [10:47:07] so we don't loose track of them [10:47:10] (week by week) [10:49:54] (03CR) 10Joal: "Comments inline :)" (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/491520 (https://phabricator.wikimedia.org/T212386) (owner: 10Elukey) [10:50:05] makes sense elukey [10:50:18] elukey: see my comments - happy to discuss them - And sorry for the delay [10:54:37] I can include your comments yes [11:01:11] (03PS5) 10Elukey: analytics-mysql: add print-target parameter [analytics/refinery] - 10https://gerrit.wikimedia.org/r/491520 (https://phabricator.wikimedia.org/T212386) [11:02:24] (03CR) 10Elukey: [V: 03+1] "Seems working on stat1007 :)" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/491520 (https://phabricator.wikimedia.org/T212386) (owner: 10Elukey) [11:02:50] joal: done --^ [11:46:41] (03CR) 10Joal: [V: 03+2 C: 03+2] "Looks good - can be part of this week train - thanks elukey :)" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/491520 (https://phabricator.wikimedia.org/T212386) (owner: 10Elukey) [11:47:05] elukey: merged :) [11:49:08] \o/ [11:49:10] thanks! :) [12:01:28] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Give access to Superset to Pau - https://phabricator.wikimedia.org/T211036 (10elukey) Added to the wmf LDAP group, and it seems that your username is already present in superset. @Pginer-WMF you should now (hopefully) be able to use Superset. [12:07:35] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Give access to Superset to Pau - https://phabricator.wikimedia.org/T211036 (10Pginer-WMF) >>! In T211036#4968247, @elukey wrote: > Added to the wmf LDAP group, and it seems that your username is already present in superset. @Pginer-WMF you should now (hope... [12:07:40] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Give access to Superset to Pau - https://phabricator.wikimedia.org/T211036 (10Pginer-WMF) 05Open→03Resolved [12:42:04] 10Analytics, 10Dumps-Generation, 10Wikidata: Update wikidata-entities dump generation to fixed day-of-month instead of fixed weekday - https://phabricator.wikimedia.org/T216160 (10JAllemandou) I can't speak about failures and restarts as I don't know much about the dumps-generation process. @ArielGlenn would... [12:46:44] (03PS5) 10Joal: Update delete/restore in mediawiki-history [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/485710 (https://phabricator.wikimedia.org/T213603) [12:55:18] * elukey lunch! [13:33:57] 10Analytics, 10EventBus, 10MediaWiki-Core-Testing, 10Quibble, and 4 others: Flaky quibble-vendor-mysql-hhvm-docker test in Jenkins - https://phabricator.wikimedia.org/T216069 (10hashar) EventBus itself seems to be fine since a change got merged yesterday by CI https://gerrit.wikimedia.org/r/#/c/491591/ So... [14:02:02] hi joal, you wanna talk planning? [14:02:13] Yessir ! [14:02:13] I'm in the cave [14:02:24] Oh sorry - was waiting here [14:11:36] 10Analytics, 10Operations, 10Research, 10serviceops, and 4 others: Transferring data from Hadoop to production MySQL database - https://phabricator.wikimedia.org/T213566 (10akosiaris) 05Open→03Stalled Per comment above. [14:15:00] So I'm trying to use beeline to see how many hits a very specific url pattern got in the last 3 months. When I have a date range of like a month i get [14:15:13] "[Fatal Error] total number of created files now is 100010, which exceeds 100000. Killing the job." [14:15:35] Is there a way to do without doing a whole lot of queries over very specific date range? [14:15:39] bawolff: 3 month of webrequest is ~75Tb of data [14:15:42] 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Product-Analytics, and 5 others: Modern Event Platform: Schema Guidelines and Conventions - https://phabricator.wikimedia.org/T214093 (10Ottomata) @nuria yeah that was his original suggestion (i.e. `has_cookies: true`), but I suggested making it a... [14:15:48] 10Analytics, 10Product-Analytics, 10Research, 10WMDE-Analytics-Engineering, and 3 others: Provide tools for querying MediaWiki replica databases without having to specify the shard - https://phabricator.wikimedia.org/T212386 (10mpopov) Tried using `analytics-mysql` on stat1007 and got "permission denied".... [14:15:52] bawolff: would sampling work for you? [14:16:11] Just a second, PMing you [14:16:21] 10Analytics: Deprecate Spark 1.6 in favor of Spark 2.x only - https://phabricator.wikimedia.org/T212134 (10Ottomata) Yes let's! Ok on todo list for this week! [14:21:00] 10Analytics, 10Product-Analytics, 10Research, 10WMDE-Analytics-Engineering, and 3 others: Provide tools for querying MediaWiki replica databases without having to specify the shard - https://phabricator.wikimedia.org/T212386 (10elukey) @mpopov still not ready sorry, the patch will be ready after the next r... [14:22:40] 10Analytics, 10Product-Analytics, 10Research, 10WMDE-Analytics-Engineering, and 3 others: Provide tools for querying MediaWiki replica databases without having to specify the shard - https://phabricator.wikimedia.org/T212386 (10Ottomata) Ya, let's deploy refinery to notebooks, it'd be nice to have there too. [14:41:18] PROBLEM - eventbus grafana alert on icinga2001 is CRITICAL: CRITICAL: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is alerting: EventBus POST Response Status alert. [14:44:46] elukey: taking a look here but I'm super alien to eventbus alerts ^ [14:44:50] 10Analytics, 10Discovery, 10Operations, 10Research: Workflow to be able to move data files computed in jobs from analytics cluster to production - https://phabricator.wikimedia.org/T213976 (10Ottomata) Alright, I'm not familiar with Swift, but if we were to do this, here is what I think we'd need: - Netwo... [14:48:10] bawolff: Heya - when running big queries, please use the nice queue of the cluster not to bloc other users :) (see https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Queries#Run_long_queries_in_a_screen_session_and_in_the_nice_queue) [14:49:16] ok, is there a way to renice my query? Should I stop it and restart? [14:49:52] fdans: o/ [14:49:54] RECOVERY - eventbus grafana alert on icinga2001 is OK: OK: EventBus ( https://grafana.wikimedia.org/d/000000201/eventbus ) is not alerting. [14:50:05] I think that it could be a spurious alert [14:50:23] yeah that's what I was thinking looking at the graphs [14:50:53] bawolff: I niced them for you - Please keep it in mind for next ones :) [14:51:06] Will do, and thanks [14:52:04] * bawolff apologizes if I caused anybody problems [14:52:41] no problem bawolff :) I hope you'll find the answers you're after [14:53:29] elukey: that eventbus alert looks lke it was caused by a job queue problem [14:53:38] mediawiki.job.cirrusSearchElasticaWrite message too large [14:55:02] yep I was about to say that, I checked in the kafka1001 logs [14:55:24] I am wondering if it is a "known" problem :) [14:55:37] ottomata: what do you do usually to debug these issues? Check EB logs ? [14:55:44] ya [14:55:46] logstash too [14:55:48] ack thanks :) [14:55:55] although i don't know logstash query lang very well [14:56:35] wow i'm in a cafe, and there is a job interview happening [14:56:38] very awkwarrd [14:56:58] ottomata: try to make comments like "uuuhh that was not cool in my opinion" [14:57:04] :D [14:59:38] ah I forgot to tell everybody my first GH answer from the GPU issue that I have opened [14:59:44] " If you work for WMF from San Francisco you make 100.000 $ dollar a year. Can't you fix this problem simply by buying a new GPU?" [15:00:00] lovely community member [15:00:08] haha oh man [15:00:24] there was also my picture from the staff page [15:00:28] ahhahaha [15:00:44] hmmm, i have a problem with people using people's appearances as a sort of argument [15:00:45] upstream deleted the comment of course [15:00:50] "huhuhu this is your picture" [15:01:15] * ottomata running home (job interview to awkward to stay here... :p) [15:01:38] (too*) [15:01:57] harej: yeah exactly, I didn't get the why of that [15:02:12] did this happen in a Wikimedia technical space? [15:02:22] nono on github :D [15:02:31] AMD GPU drivers [15:02:35] nothing related to us [15:06:57] (03PS1) 10Fdans: Add 10 Wikipedias to sqoop list [analytics/refinery] - 10https://gerrit.wikimedia.org/r/491762 [15:11:07] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review: eventlogging fails flake8 due to new upstream version, breaking CI - https://phabricator.wikimedia.org/T212396 (10hashar) The magic failure is that tox.ini is creates the virtualenv with `flake8` as a dependency but without any... [15:16:43] 10Analytics, 10Analytics-Kanban: Deprecate Spark 1.6 in favor of Spark 2.x only - https://phabricator.wikimedia.org/T212134 (10Ottomata) [15:17:11] good morning, buenas tardes and bon après-midi! I have a wee question about the correct approach to instrument an event stream that needn't be public, but that could be, in this day of modern event platforms [15:17:36] it's just a stream of changes in CentralNotice campaign configurations [15:17:59] I'm pretty sure I can just make server-side events and consume them on the corresponding Kafka stream [15:18:27] However, I don't want to use up all the normal event _logging_ resources [15:18:47] for something that is already queryable from the database [15:19:26] Being able to consume a stream of these changes will be useful for a few CN monitoring tools [15:19:40] which for now don't need a public stream [15:20:39] however, the stream certainly could be public! It's kinda like recent changes, but for stuff that's not stored as Wiki pages (i.e., the non-banner-content aspects of CentralNotice config) [15:20:42] thoughts, anyone? [15:20:51] many thanks in advance!!!! :) [15:24:16] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review: eventlogging fails flake8 due to new upstream version, breaking CI - https://phabricator.wikimedia.org/T212396 (10hashar) eventlogging depends on `mysqlclient>=1.3.7` (see [[ https://pypi.org/project/mysqlclient/ mysqlclient on... [15:29:53] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10Core Platform Team Backlog (Watching / External), 10Services (watching): Modern Event Platform: Stream Intake Service: Implementation - https://phabricator.wikimedia.org/T206785 (10Ottomata) Here's what's left to do for this quarter's goal of porting o... [15:31:19] a-team: re: https://phabricator.wikimedia.org/T211173, nuria said we should do all the transformation using Druid transforms rather than a Hive query (which seems like it would be easier and more useful), but I don't remember the reason. Anybody know? Was it the storage space? Load from transforming a dataset that big every month? [15:32:18] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review: eventlogging fails flake8 due to new upstream version, breaking CI - https://phabricator.wikimedia.org/T212396 (10hashar) `ImportError: No module named _mysql` is due to `mysqlclient` python module, eventlogging is not compatibl... [15:32:38] AndyRussG: yes for sure! [15:32:51] Can you get the data you need in MW now from a hook? [15:33:26] AndyRussG: https://wikitech.wikimedia.org/wiki/EventBus#Usage_Example_for_MediaWiki_Developers [15:33:30] ottomata: hi! yes, or just from adding some really simple stuff to CN [15:33:42] neilpquinn, can I ask when Nuria said that, I'm trying to remember the context [15:34:08] well, we try to keep event emitting separate from code, but if you have a hook, then you can make register a handler in the EventBus extension, and emit the event to the Event Service [15:34:18] you need to design a schema first in the mediawiki/event-schemas repo [15:34:19] mforns: December or January—not too long ago [15:34:31] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review: eventlogging fails flake8 due to new upstream version, breaking CI - https://phabricator.wikimedia.org/T212396 (10hashar) TLDR: ` --- a/requirements.txt +++ b/requirements.txt @@ -3,7 +3,7 @@ jsonschema>=0.7 confluent-kafka>=0.... [15:35:00] ottomata: ok fantastic yes this sounds right [15:35:15] AndyRussG: we will also soon have Monolog support for EventBus, so you can do it without a hook, but I think we prefer the hook way, especially if one already exists [15:35:25] ok gotcha [15:35:27] what about if one day people would like the stream to be public? [15:35:49] AndyRussG: via EventStreams? Once the events are in Kafka, its only a simple puppet config change to expose it in eventstreams [15:35:55] I mean, it's not necessary, but if I can do it in a way that would eventually make it easy, that'd be nice [15:35:58] AndyRussG: how many events / second is this? [15:35:59] ah ok fantastic [15:36:08] ottomata: relatively a pittance [15:36:09] btw, bmansurov just went through this very process [15:36:13] for page-links-change [15:36:18] ah cool [15:36:24] so he (and petr p) can help you a lot too [15:36:46] yeah it's be like once every few minutes at peak, usually less [15:36:46] neilpquinn, ok, then I also don't remember her reason at that moment :] [15:37:15] mforns: haha okay, thanks for trying. I'll ping her on the ticket. [15:37:31] cool! was that in an extension or core? (Just to take a peek at how it was done) [15:38:09] extension [15:38:44] neilpquinn, although it makes sense to me to use Druid transforms if possible [15:38:52] AndyRussG: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/EventBus/+/486691/ [15:39:00] mforns: why is that? :) [15:39:06] neilpquinn, it seems to me it saves 1 computation step [15:39:13] AndyRussG: and https://gerrit.wikimedia.org/r/#/c/mediawiki/event-schemas/+/486521/ [15:39:36] also https://gerrit.wikimedia.org/r/#/c/mediawiki/event-schemas/+/490143/ :) [15:39:47] when loading data, Druid copies the data you pass it to a temporary location in JSON format to then load it [15:39:52] mforns: true—but I feel like that's balanced out by the benefits by having the transformed data in Hive as well [15:40:01] Hmm [15:40:01] ottomata: cool beans, thanks so much!!! Ah one more question [15:40:18] what about local developer setup, to be able to test the consumer code against one's local wiki? [15:40:20] So one effect of doing it in Hive is essentially doubling the storage space requirement? [15:40:36] mforns: and that's significant for mediawiki_history [15:40:37] neilpquinn, if you are going to use the specific hive table, then that is a reason to use Hive [15:40:38] AndyRussG: mw-vagrant has an eventbus role [15:40:38] I guess I'll need some local Kafka install? [15:40:46] that will install all that for ya [15:40:46] ah ok great [15:40:52] yeee [15:40:55] then you can tail out the logs/eventbus.log (I think) to see events flow [15:41:07] (or consume from Kakfa) [15:41:14] ok fantastic [15:41:18] AndyRussG: btw, this is all changing now, but that shouldn't block you [15:41:34] we are replacing eventbus with another service, but the API and the EventBus extension remain the same [15:41:41] the schemas for event metadata will slightly change [15:41:42] neilpquinn, yes, it would double the data you'd use, but that is OK I guess, given that it would be much smaller than mediawiki_history [15:41:50] ah ok [15:41:51] but we can handle porting your event over to the new service along with all the others [15:42:04] ok yeah sounds adaptable-to [15:42:07] ya [15:42:09] mforns: it would be just as tall, but yeah, it would be significantly narrower [15:42:26] neilpquinn, I think the question is: would you use the Hive table, or would it just be a intermediate step [15:42:36] yeah I did want to try to do this in a way appropriate for that big roadmap [15:42:45] if the latter, maybe it pays the maintenance [15:42:55] cost [15:43:14] AndyRussG: i'd say do the work to make it work with existing system now. the changes to make it work with the new one will be very small [15:43:25] if we get our stuff all settled before your event is ready to be deployed, we can adapt beforehand [15:43:48] if that happens we could even use your new event as as another guinea pig :p [15:43:52] if not, we'll do it along with all the other stuff :) [15:44:20] neilpquinn, sorry, the other way round: if the table is useful tou you, then it would pay the maintenance cost [15:45:34] mforns: hmm, yeah. I'm actually not sure...depends on the spec we come up with. (I just realized that you wouldn't be able to use the same Druid table for counting edits and counting editors [15:45:59] neilpquinn, yes, probably you'd need 2 tables [15:46:01] because you can't do a distinct username type thing, so it wouldn't be as useful in Hive [15:46:46] mforns: yeah, that makes sense. Thanks—I think I have a decent idea of the tradeoffs now :) [15:47:35] neilpquinn, ok cool, let us know in the task! [15:48:12] ottomata: hey fantastic, thanks so much for the help!!! :) I think I have all I need, I'll reach out if I have questions :) [15:48:36] and for sure we're completely guinea-pig-able [15:51:25] great :) [15:51:40] elukey: hm, so package oozie (server) depends on spark-core in cdh [15:51:53] so we can remove it from most places, but not from an-coord1001? [15:53:08] nice :D [15:53:21] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review: eventlogging fails flake8 due to new upstream version, breaking CI - https://phabricator.wikimedia.org/T212396 (10hashar) @mforns I have send a bunch of patches. Some might be conflicting with the one you have send to setup flak... [15:57:05] 10Analytics, 10EventBus, 10MediaWiki-Core-Testing, 10Quibble, and 4 others: Flaky quibble-vendor-mysql-hhvm-docker test in Jenkins - https://phabricator.wikimedia.org/T216069 (10Pchelolo) I have tried to resubmit the change: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/EventBus/+/491014/ with no... [16:03:08] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Patch-For-Review: eventlogging fails flake8 due to new upstream version, breaking CI - https://phabricator.wikimedia.org/T212396 (10mforns) @hashar Thanks a lot for all the dedicated work!! [16:03:41] !log removing spark 1 from Analytics cluster - T212134 [16:03:44] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:03:44] T212134: Deprecate Spark 1.6 in favor of Spark 2.x only - https://phabricator.wikimedia.org/T212134 [16:05:00] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Deprecate Spark 1.6 in favor of Spark 2.x only - https://phabricator.wikimedia.org/T212134 (10Ottomata) ` sudo cumin '(R:Class = profile::analytics::cluster::client or R:Class = profile::hadoop::worker) and not an-coord1001.eqiad.wmnet' 'apt-get -y remove... [16:07:24] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Run critical Analytics Hadoop jobs and make sure that they work with the new auth settings. - https://phabricator.wikimedia.org/T212259 (10elukey) ` hdfs@analytics1030:/mnt/hdfs/wmf/data/raw/webrequest$ ls webrequest_test_text ` So if I got it correctly,... [16:08:54] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Deprecate Spark 1.6 in favor of Spark 2.x only - https://phabricator.wikimedia.org/T212134 (10Ottomata) Also removed /etc/spark on an-coord1001 and analytics-tool1001, and analytics1030 (hue). I should have also excluded analytics1030 from my cumin comman... [16:10:41] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Deprecate Spark 1.6 in favor of Spark 2.x only - https://phabricator.wikimedia.org/T212134 (10Ottomata) [16:13:20] 10Analytics, 10Product-Analytics: "Edit" equivalent of pageviews daily available to use in Turnilo and Superset - https://phabricator.wikimedia.org/T211173 (10mforns) @kzimmerman @MNeisler Sure, we can discuss here, or have a meeting, what's better for you. I also just talked to @Neil_P._Quinn_WMF about wheth... [16:19:38] 10Quarry, 10Patch-For-Review: Letter g cut off at bottom in #title - https://phabricator.wikimedia.org/T216581 (10zhuyifei1999) This seems relevant to Google Chrome (and possibly other Chromium-based). Reproduced on Google Chrome 72.0.3626.109 (too lazy to compile Chromium just for this ticket) on Gentoo. Does... [16:20:37] (03PS1) 10Elukey: Add oozie webrequest test bundle [analytics/refinery] - 10https://gerrit.wikimedia.org/r/491791 (https://phabricator.wikimedia.org/T212259) [16:21:24] (03CR) 10Zhuyifei1999: view.css: Fixed problem with the display of certain letters (032 comments) [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/491718 (https://phabricator.wikimedia.org/T216581) (owner: 10Zoranzoki21) [16:22:55] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: LDAP login advice on https://superset.wikimedia.org/ specifies wrong kind of login name - https://phabricator.wikimedia.org/T214524 (10elukey) @Tbayer should be fixed now! Thanks for the report! [16:25:04] joal: created a patch above for the webrequest test refine, I *think* I only need a modified bundle.xml [16:25:12] lemme know how far I am from the truth :D [16:28:06] a-team anything you'd like me to mention in sos? [16:28:22] mmmmmmm [16:28:34] fdans: message in a bottle? [16:28:45] joal: nice [16:28:57] haha [16:29:07] well, I did find those instances of data loss just now [16:29:21] in the EL schemas, but maybe we should figure out what to do first [16:29:25] so yeah, don't mention anything yet [16:29:58] ok [16:30:11] fdans, can you mention that we'll sunset wikimetrics at the end of this quarter? [16:30:34] fdans, if someone has objections, can you give them the task link: https://phabricator.wikimedia.org/T211835 [16:30:42] fdans, and/or point them to me? [16:31:15] yep! [16:31:23] thanks [16:31:24] ! [16:33:14] elukey: I actually don't know how you've handled data flowing onto the test cluster [16:33:49] elukey: kafkacat + camus on webrequest_test_text source? [16:39:40] joal: correct [16:39:42] (03CR) 10Joal: "I ran mediawiki-history-check on data generated by this patch and failures are coming from expected changes:" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/491494 (https://phabricator.wikimedia.org/T216603) (owner: 10Joal) [16:39:59] makes sense :) [16:40:14] elukey: imformative question: how much downscale do we apply? [16:45:38] joal: so one partition (kafka) sampled 1000 [16:45:49] (03CR) 10Joal: "For this to work we also need to modify the oozie/webrequest/datasets.xml and oozie/webrequest/datasets_raw.xml files to represent the new" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/491791 (https://phabricator.wikimedia.org/T212259) (owner: 10Elukey) [16:46:00] elukey: great :) [16:46:16] elukey: please let me know if my comment makes sense --^ [16:54:47] ah yes you are right, the bundle.properties should be there as well, I thought to keep it somewhere else but it doesn't make sense. [16:55:03] didn't know about those datasets.xml, will study a bit. thanks for the pointers! [17:00:06] elukey: I can help with that :) [17:01:07] joal,fdans standuuupp [17:01:22] thanks elukey [17:14:49] 10Analytics, 10Operations, 10Research-management, 10Patch-For-Review, 10User-Elukey: GPU upgrade for stats machine - https://phabricator.wikimedia.org/T148843 (10elukey) Updates from https://github.com/RadeonOpenCompute/ROCm/issues/714#issuecomment-465666946 are not encouraging, gfx701 is a dead end so w... [17:17:48] (03CR) 10Milimetric: [V: 03+2 C: 03+2] Add 10 Wikipedias to sqoop list [analytics/refinery] - 10https://gerrit.wikimedia.org/r/491762 (owner: 10Fdans) [17:21:14] 10Analytics, 10Analytics-Kanban, 10Operations, 10hardware-requests: GPU upgrade for stat1005 - https://phabricator.wikimedia.org/T216226 (10elukey) In https://github.com/RadeonOpenCompute/ROCm/issues/714#issuecomment-465666946 the upstream developers of the AMD drivers told me that our GPU on stat1005 is b... [17:26:47] For those of you interested in distributed systems inner stuff: https://arxiv.org/abs/1902.06776 [17:32:41] nice! [17:33:43] last one that I've heard was raft (that should be used by etcd?) [17:36:23] 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Product-Analytics, and 5 others: Modern Event Platform: Schema Guidelines and Conventions - https://phabricator.wikimedia.org/T214093 (10Nuria) @Ottomata If we are trying to store cookie info to better identify bots we do not need the map at all w... [17:37:15] 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Product-Analytics, and 5 others: Modern Event Platform: Schema Guidelines and Conventions - https://phabricator.wikimedia.org/T214093 (10Ottomata) @nuria what about WMF-Last-Access? [17:43:34] 10Quarry, 10Wikibugs: Quarry -> #wikimedia-research & #wikimedia-labs - https://phabricator.wikimedia.org/T76467 (10Framawiki) [17:44:23] 10Quarry, 10Wikibugs: Quarry -> #wikimedia-research & #wikimedia-labs - https://phabricator.wikimedia.org/T76467 (10bmansurov) @Legoktm thanks for the reminder. I should not have been lazy. Thanks for the patch. [17:47:54] (03PS2) 10Elukey: Add oozie webrequest test bundle [analytics/refinery] - 10https://gerrit.wikimedia.org/r/491791 (https://phabricator.wikimedia.org/T212259) [17:50:38] (03PS3) 10Elukey: Add oozie webrequest test bundle [analytics/refinery] - 10https://gerrit.wikimedia.org/r/491791 (https://phabricator.wikimedia.org/T212259) [17:52:47] joal: added moar things --^ [17:52:57] MAOR ! [17:53:06] probably still not good [17:53:27] elukey: ottomata: we planned and estimated smart tools for better data, starting like 19 on the goals pad: https://etherpad.wikimedia.org/p/analytics-goals [17:53:52] take a look and let us know what you think, happy to explain more [17:54:47] milimetric: :) [17:54:51] what does "ML" means? [17:55:13] I am also a bit scared by the amount of time that it will need to implement airflow [17:55:13] Mucho Luca :) [17:55:17] * joal hides and run [17:55:20] machine learning? [17:56:02] harej: yep that part was clear, I was more referring to "what are our plans for ML" since it seems a very broad topic :_ [17:56:05] :) [17:56:26] in my experience, if you talk about machine learning with no additional details, it's to try to impress people so that they might fund your projects :) [17:56:46] ahahahah [17:56:48] lol [17:56:53] sheesh, so cynical [17:57:18] harej: I wish we get funded \l9 [17:57:20] :) [17:57:29] in this context, ML means supporting the other teams that need to do it, providing samples for working with ML in Spark, etc, and helping solve any related problems, like with ORES infrastructure [17:57:48] that project could easily take all our time, but we're more timeboxing it to 6 months, thinking it would be like half of Joseph's time [17:58:01] (03CR) 10Joal: "2 small comments inline - did not check the list of parameters in bundle.xml, oozie will complain if there is an error. Almost ready !" (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/491791 (https://phabricator.wikimedia.org/T212259) (owner: 10Elukey) [17:59:12] does that help elukey? [17:59:33] (03CR) 10Elukey: Add oozie webrequest test bundle (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/491791 (https://phabricator.wikimedia.org/T212259) (owner: 10Elukey) [18:00:12] milimetric: ah okok, if it doesn't need infra support (tools, hw, etc..) I am ok :) [18:00:36] milimetric: depending on the answer to https://phabricator.wikimedia.org/T213976#4968603, we might want to make that ticket a goal for next quarter [18:00:59] milimetric: i think ML work will need infra support, no? [18:01:20] we want it to be easy for folks to run ML jobs in Hadoop (distributed) [18:01:29] and then deploy models (and features?) to prod [18:01:39] to do that we need some conventions, and standards, and packages, and tooling [18:01:47] maybe https://mlflow.org/ will do it all for us [18:01:47] dunno [18:02:40] if this is the case then I am worried a bit :) [18:04:23] (03PS4) 10Elukey: Add oozie webrequest test bundle [analytics/refinery] - 10https://gerrit.wikimedia.org/r/491791 (https://phabricator.wikimedia.org/T212259) [18:05:04] milimetric: elukey: we'd also like the GPU stuff working in hadoop too [18:05:12] maybe that is not relevant for this annual plan [18:05:48] elukey: I think you can test :) [18:05:55] \o/ [18:06:03] ottomata: next quarter is still this fiscal, so we're ok to do other stuff like the swift work [18:06:11] joal: hmm, the two refinery source tasks were part of last week's deployment right? [18:06:21] ottomata: I think that we'd need a meeting to iron out all the details [18:06:32] if we don't plan it correctly it will be a huge amount of work [18:06:33] joal i mean the two tasks currently in ready to deploy [18:06:36] fdans: correct, they've already been deployed, but jobs not restrarted [18:06:37] ottomata / elukey: ML may need infra support, but this estimation (sorry, I should've explained) is just for Jo, me, Fran, and Marcel [18:06:49] joal: gotcha [18:07:01] joal: do we want to restart the jobs now? [18:07:39] ok, so I agree the ML support is something that could take over our whole year, so it could be something we talk with the boss about when she's back [18:08:27] 10Analytics: Old job_tracker setting in oozie properties - https://phabricator.wikimedia.org/T216519 (10Ottomata) Hm. I can't remember exactly what this was for. If there are no errors...let's remove it and see what happens? :) [18:09:32] milimetric: yes please :) My main concern if we need infra tooling support etc.. is that it will backfire and add a ton of extra ops load [18:09:46] out of curosity, what does the analytics team use ML for? [18:09:57] harej: we don't right now [18:10:22] this is work in anticipation of more teams needing to use that kind of technology and someone needing to own the platform [18:10:41] agreed elukey, it may be inevitable, but agreed [18:12:36] milimetric: I am supportive for ML, my only concern is planning, otherwise the sre load will grow too much over the year [18:13:44] yeah, we need to really get this right, and make sure either it's purely a supporting role or that we assign the resources fairly [18:17:39] !log deploying refinery [18:17:41] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:32:36] (03PS3) 10Joal: Refactor mediawiki-history core data gathering [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/491494 (https://phabricator.wikimedia.org/T216603) [18:33:16] a-team: analytics-mysql is ready to use on stat1007 [18:33:24] you have it in your path so easy to use [18:33:24] nice [18:33:29] \o/ :) [18:33:37] elukey@stat1007:~$ analytics-mysql enwiki --print-target [18:33:37] dbstore1003.eqiad.wmnet:3311 [18:34:00] elukey: it should be on stat1006 too [18:34:42] milimetric: it is currently deployed where we have refinery [18:34:59] elukey: ah, right, didn't think of that [18:35:06] yeah, most people access the replicas from 1006 [18:35:17] kind of weird but I guess we should deploy refinery there [18:35:20] we can add the refinery to stat1006 and the notebooks [18:35:38] right, seems like it'd be useful on the notebook servers too [18:36:07] 10Analytics, 10Product-Analytics, 10Research, 10WMDE-Analytics-Engineering, and 3 others: Provide tools for querying MediaWiki replica databases without having to specify the shard - https://phabricator.wikimedia.org/T212386 (10elukey) @mpopov you should be able to test analytics-mysql from stat1007 :) [18:37:18] (03CR) 10jerkins-bot: [V: 04-1] Refactor mediawiki-history core data gathering [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/491494 (https://phabricator.wikimedia.org/T216603) (owner: 10Joal) [18:42:02] ottomata: one thing about stat1006 [18:42:06] it doesn't have profile::hadoop::common::hadoop_cluster_name [18:42:12] joal: do you want me to restart any steve jobs? [18:43:22] ottomata: we can add a parameter to profile::analytics::refinery to skip hdfs stuff if needed [18:45:35] code review incoming [18:46:43] fdans: yeah - let's iRestart some jobs [18:47:22] 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Product-Analytics, and 5 others: Modern Event Platform: Schema Guidelines and Conventions - https://phabricator.wikimedia.org/T214093 (10EBernhardson) I don't think the actual date inside the WMF-Last-Access header makes any difference. Every requ... [18:48:44] (03PS1) 10Elukey: Add notebook100[3,4] and stat1006 to the scap targets [analytics/refinery/scap] - 10https://gerrit.wikimedia.org/r/491827 (https://phabricator.wikimedia.org/T212386) [18:50:10] all right leaving for dinner, will merge the above tomorrow :) [18:50:14] o/ [18:51:51] (03PS4) 10Joal: Refactor mediawiki-history core data gathering [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/491494 (https://phabricator.wikimedia.org/T216603) [18:54:00] fdans: The jobs I'd like to be restarted are the ones impacted by the change of spark-paramters: mediawiki-history, mediawiki-history-checker and mediawiki-wikitext (dump converter) - Please :) [18:54:51] And I assume you'll also need to restart mediawiki-history-reduced (for email) :) [18:55:54] (03PS5) 10Joal: Refactor mediawiki-history core data gathering [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/491494 (https://phabricator.wikimedia.org/T216603) [19:26:40] (03PS1) 10Joal: Add change_tag and change_tag_def to hive [analytics/refinery] - 10https://gerrit.wikimedia.org/r/491838 (https://phabricator.wikimedia.org/T205940) [20:05:46] 10Quarry, 10Security: Use data attributes instead of unsafe-inline var definitions - https://phabricator.wikimedia.org/T216653 (10Framawiki) [20:22:18] (03PS2) 10Zoranzoki21: view.css: Fixed problem with the display of certain letters [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/491718 (https://phabricator.wikimedia.org/T216581) [20:23:16] 10Analytics, 10Analytics-Kanban, 10Operations, 10hardware-requests: GPU upgrade for stat1005 - https://phabricator.wikimedia.org/T216226 (10EBernhardson) Another thing to take away from the upstream response is that debian is unsupported. I can't imagine deploying ubuntu to a single machine will be an acce... [20:23:18] (03CR) 10Joal: [V: 03+1] "Tested on cluster." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/491838 (https://phabricator.wikimedia.org/T205940) (owner: 10Joal) [20:50:00] (03CR) 10Zhuyifei1999: [C: 03+1] view.css: Fixed problem with the display of certain letters [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/491718 (https://phabricator.wikimedia.org/T216581) (owner: 10Zoranzoki21) [20:53:04] (03CR) 10Framawiki: [C: 03+2] view.css: Fixed problem with the display of certain letters [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/491718 (https://phabricator.wikimedia.org/T216581) (owner: 10Zoranzoki21) [20:53:29] (03Merged) 10jenkins-bot: view.css: Fixed problem with the display of certain letters [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/491718 (https://phabricator.wikimedia.org/T216581) (owner: 10Zoranzoki21) [20:54:23] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10Security-Team, and 3 others: Modern Event Platform: Stream Intake Service: AJV usage security review - https://phabricator.wikimedia.org/T208251 (10Ottomata) > lodash <= 4.17.5 It looks like the lodash is fixed by updating service-runner to 2.6.9. Done. [21:05:08] 10Quarry, 10Browser-Support-Google-Chrome, 10Patch-For-Review: Letter g cut off at bottom in #title - https://phabricator.wikimedia.org/T216581 (10Framawiki) 05Open→03Resolved Thanks for he patch @Zoranzoki21 ! [21:14:27] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10Security-Team, and 3 others: Modern Event Platform: Stream Intake Service: AJV usage security review - https://phabricator.wikimedia.org/T208251 (10Ottomata) [21:19:43] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10Security-Team, and 4 others: Modern Event Platform: Stream Intake Service: AJV usage security review - https://phabricator.wikimedia.org/T208251 (10Ottomata) [21:20:18] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10Security-Team, and 4 others: Modern Event Platform: Stream Intake Service: EventGate security review - https://phabricator.wikimedia.org/T208251 (10Ottomata) [21:21:35] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10Security-Team, and 4 others: Modern Event Platform: Stream Intake Service: EventGate security review - https://phabricator.wikimedia.org/T208251 (10Ottomata) @sbassett, I've moved your TODO list into the task description. I believe I've resolved all of t... [21:25:14] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 5 others: Modern Event Platform: Stream Intake Service: EventGate security review - https://phabricator.wikimedia.org/T208251 (10Ottomata) [21:29:33] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 5 others: Modern Event Platform: Stream Intake Service: EventGate security review - https://phabricator.wikimedia.org/T208251 (10sbassett) @Ottomata - LGTM. Ideally, we'd get some low-priority tasks filed for the service-template-n... [21:48:49] 10Analytics, 10Product-Analytics: Timestamp column in EventLogging tables have incompatible collation - https://phabricator.wikimedia.org/T216658 (10nettrom_WMF) [21:52:48] 10Analytics, 10EventBus, 10MediaWiki-Core-Testing, 10Quibble, and 4 others: Flaky quibble-vendor-mysql-hhvm-docker test in Jenkins - https://phabricator.wikimedia.org/T216069 (10hashar) The tests are being run with #quibble which should let us reproduce the failure. Specially if one reuse the Docker contai... [21:59:46] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 3 others: Modern Event Platform: Stream Intake Service: Implementation: Deployment Pipeline - https://phabricator.wikimedia.org/T211247 (10Ottomata) @akosiaris the above two patches are my best effort at copy/pasting stuff from math... [22:20:13] 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Product-Analytics, and 5 others: Modern Event Platform: Schema Guidelines and Conventions - https://phabricator.wikimedia.org/T214093 (10Tbayer) >>! In T214093#4969454, @EBernhardson wrote: > I don't think the actual date inside the WMF-Last-Acces... [22:21:58] 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Product-Analytics, and 5 others: Modern Event Platform: Schema Guidelines and Conventions - https://phabricator.wikimedia.org/T214093 (10Ottomata) > I assume the discussion here is confined to the particular CirrusSearch use case, correct? Naw, t... [22:28:55] 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Product-Analytics, and 5 others: Modern Event Platform: Schema Guidelines and Conventions - https://phabricator.wikimedia.org/T214093 (10EBernhardson) >>! In T214093#4970086, @Tbayer wrote: >>>! In T214093#4969454, @EBernhardson wrote: >> I don't... [22:38:47] 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Product-Analytics, and 5 others: Modern Event Platform: Schema Guidelines and Conventions - https://phabricator.wikimedia.org/T214093 (10Tbayer) >>! In T214093#4970109, @EBernhardson wrote: >>>! In T214093#4970086, @Tbayer wrote: >>>>! In T214093#... [22:42:05] 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Product-Analytics, and 5 others: Modern Event Platform: Schema Guidelines and Conventions - https://phabricator.wikimedia.org/T214093 (10Tbayer) >>! In T214093#4966772, @Ottomata wrote: >> technically there could be 2 different headers that differ... [22:52:34] is there by chance a hive table that maps from wiki db name to the language of the wiki? [23:06:04] ebernhardson: i believe neilpquinn has been looking into creating something like that https://phabricator.wikimedia.org/T184576 [23:08:07] 10Analytics, 10Contributors-Analysis, 10Product-Analytics: Make an Analytics Data Lake table to provide meta info about wikis - https://phabricator.wikimedia.org/T184576 (10EBernhardson) Interested in seeing this complete. In particular I could use the wiki db name -> language map portion of this. [23:08:29] HaeB: thanks! doesn't look like it's moving, but i can at least put a vote in