[06:25:29] PROBLEM - ORES web node labs ores-web-03 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:29:50] PROBLEM - https://grafana.wikimedia.org/dashboard/db/ores-extension grafana alert on einsteinium is CRITICAL: CRITICAL: https://grafana.wikimedia.org/dashboard/db/ores-extension is alerting: Failure rate alert. [06:31:50] RECOVERY - https://grafana.wikimedia.org/dashboard/db/ores-extension grafana alert on einsteinium is OK: OK: https://grafana.wikimedia.org/dashboard/db/ores-extension is not alerting. [14:33:40] o/ Amir1 [14:33:42] * halfak looks around for awight [14:33:58] halfak: Hey, as I'm SWATing today, I can't be at BS meeting :(((( [14:34:10] Gotcha. :) [14:35:18] halfak [14:35:27] Hey [14:35:34] hi [14:46:51] 10Scoring-platform-team (Current): OneVsRest Classification for revscoring - https://phabricator.wikimedia.org/T185896#3927992 (10Sumit) [14:47:01] 10Scoring-platform-team (Current): OneVsRest Classification for revscoring - https://phabricator.wikimedia.org/T185896#3928005 (10Sumit) https://github.com/wiki-ai/revscoring/pull/389 [15:07:11] 10Scoring-platform-team (Current): Refactor scripts fetching text and other metadata - https://phabricator.wikimedia.org/T181074#3928028 (10Sumit) [15:07:13] Amir1, are you coming to the sync meeting? [15:21:49] 10Scoring-platform-team: Clean up all revscoring 2.0 requirements, -> 2.1 - https://phabricator.wikimedia.org/T185897#3928041 (10awight) [15:28:09] awight https://groups.google.com/forum/#!topic/repo-discuss/B4P7G1YirdM [15:28:18] he was the founder of gerrit as it is today. [15:28:40] https://git.github.io/rev_news/2017/08/16/edition-30/#developer-spotlight-shawn-pearce [15:28:55] paladox: sorry to see it [15:29:03] yep. [15:29:22] paladox: on the bright side, congrats on landing that login dialog patch! I had the opportunity to use it the other day, it’s very classy... [15:29:32] awight :) [15:30:02] awight i think chad wants to pull in picture of the day for it [15:30:19] hehe sounds like he got inspired [15:30:47] paladox: I just loaded it on a phone btw, it needs to be more responsive. [15:31:05] awight yeh, that's gwtui fault. [15:31:09] IMO the dialog should be full-screen [15:31:19] i made a feature request to get the login page implemented in polygerrit. [15:31:24] gwtui prevents us from CSS fixing that? [15:32:11] awight yes, because we aren't using the mobile detection thingy in html [15:33:22] I don’t think it’s necessary—CSS lets you provide different rules based on screen size [15:33:30] w/o HTML or Javascript support [15:34:18] awight not really you need the meta thing [15:34:35] without setting the meta thing, it forces desktop by default. [15:34:43] Could you send a link as to what that’s about? [15:35:07] awight https://www.w3schools.com/css/css_rwd_viewport.asp [15:54:40] 10Scoring-platform-team (Current), 10ORES, 10Patch-For-Review: Make sure ORES is compatible with stretch - https://phabricator.wikimedia.org/T182799#3928108 (10Halfak) a:03Halfak [16:00:07] halfak: Feel free to offload the Done -> announcement work to me [16:00:19] Oooh. I'd like that :) [16:00:32] I've feeling a bit overloaded with org stuff right now :| [16:00:46] Most QR and planning for FY18-19 [16:01:08] planning for FY18-19 == setting up so that we can plan and draft [16:06:18] (03PS1) 10Sbisson: Enable ORES filters on RecentChangesLinked [extensions/ORES] - 10https://gerrit.wikimedia.org/r/406597 (https://phabricator.wikimedia.org/T179718) [16:06:37] Hey, here’s something fun. While I was unable to do real work, I got a high score on a personal project, https://scrutinizer-ci.com/g/adamwight/boxes/ [16:07:13] This is the more insteresting directory in that repo, https://github.com/adamwight/boxes/tree/master/boxes/commands [16:07:16] *most [16:07:32] halfak: Is there a task to assign to myself, yet? [16:09:01] paladox: This suggests the meta vierport thing isn’t necessary? https://stackoverflow.com/questions/14775195/is-the-viewport-meta-tag-really-necessary [16:09:07] *viewport [16:09:46] grabbing lunch/dinner [16:09:49] be back soon [16:10:12] hmm [16:13:07] halfak: I don’t see the quarterly review meeting, even on your calendar. Did you say it’s today, in 4 hours? [16:14:55] Nope. On Wednesday at 2000 [16:18:15] (03CR) 10Sbisson: [C: 04-2] "Pending more testing" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/406597 (https://phabricator.wikimedia.org/T179718) (owner: 10Sbisson) [16:20:23] :) [16:31:57] 10Scoring-platform-team (Current), 10ORES, 10Operations, 10Patch-For-Review: Reimage ores* hosts with Debian Stretch - https://phabricator.wikimedia.org/T171851#3928149 (10Halfak) [16:32:23] 10Scoring-platform-team (Current), 10ORES, 10Operations, 10Patch-For-Review: Reimage ores* hosts with Debian Stretch - https://phabricator.wikimedia.org/T171851#3478146 (10Halfak) @akosiaris, is this done? [16:32:38] halfak: I just verified that it is *not* done [16:32:45] Linux ores1001 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u1~bpo8+2 (2018-01-04) x86_64 [16:32:46] Debian GNU/Linux 8.9 (jessie) [16:33:04] 10Scoring-platform-team (Current), 10ORES: Preliminary deployment of ORES to new cluster - https://phabricator.wikimedia.org/T185901#3928152 (10Halfak) p:05Triage>03High [16:33:14] 10Scoring-platform-team (Current), 10ORES: Preliminary deployment of ORES to new cluster - https://phabricator.wikimedia.org/T185901#3928152 (10Halfak) [16:33:17] 10Scoring-platform-team (Current), 10ORES, 10Operations, 10Patch-For-Review: Reimage ores* hosts with Debian Stretch - https://phabricator.wikimedia.org/T171851#3928163 (10Halfak) [16:33:27] awight, thanks. [16:33:37] So we're blocked there. [16:34:08] awight, here's the status report task: https://phabricator.wikimedia.org/T185535 [16:34:15] I'll start in on some reviewing. [16:39:51] 10Scoring-platform-team, 10Wikilabels, 10editquality-modeling, 10User-Tgr, 10artificial-intelligence: Complete edit quality campaign for Hungarian Wikipedia - https://phabricator.wikimedia.org/T167968#3928182 (10Halfak) Confirmed that this is done! Thanks for your work. I'll get us moving on the next s... [16:40:12] 10Scoring-platform-team (Current), 10Wikilabels, 10editquality-modeling, 10User-Tgr, 10artificial-intelligence: Complete edit quality campaign for Hungarian Wikipedia - https://phabricator.wikimedia.org/T167968#3928183 (10Halfak) [16:41:06] 10Scoring-platform-team, 10editquality-modeling, 10artificial-intelligence: Train/test damaging and goodfaith model for Hungarian Wikipedia - https://phabricator.wikimedia.org/T185903#3928185 (10Halfak) [16:41:15] 10Scoring-platform-team (Current), 10Wikilabels, 10editquality-modeling, 10User-Tgr, 10artificial-intelligence: Complete edit quality campaign for Hungarian Wikipedia - https://phabricator.wikimedia.org/T167968#3351422 (10Halfak) [16:41:17] 10Scoring-platform-team, 10editquality-modeling, 10artificial-intelligence: Train/test damaging and goodfaith model for Hungarian Wikipedia - https://phabricator.wikimedia.org/T185903#3928195 (10Halfak) [16:46:11] OMG-mail [16:47:47] 10Scoring-platform-team (Current), 10editquality-modeling, 10User-Ladsgroup, 10artificial-intelligence: Train/test damaging and goodfaith model for Hungarian Wikipedia - https://phabricator.wikimedia.org/T185903#3928201 (10Ladsgroup) a:03Ladsgroup [16:50:52] <3 ^ Amir1 [16:51:05] codezee, just added notes re. OneVsRest [16:53:42] :) [17:05:37] halfak: wat! Amir1 already wrote the report…. I’ll just do clean-up [17:05:43] Right :) [17:06:45] Amir1: When you marked the status report task “blocked by others”, what exactly is blocking? [17:07:07] oh I see, that was on your personal board. [17:07:14] others = us :) [17:09:48] halfak: I see the notes, so you're saying we remove the code previously added for multilabel and bring OneVsRest under the multilabel flag? [17:09:59] I think so, yeah, codezee [17:10:01] what do you think? [17:13:09] yeah :P [17:19:20] halfak: we can if we don't intend to use sklearn's multilabel option. I don't exactly know what it does internally and if we might find it useful in future for sth else... [17:20:16] Seems like we did a fine evaluation. If we want to experiment again in the future, we can re-jigger. [17:24:01] okay...I'll adjust the pieces to bring this classification under the multilabel flag [17:28:50] (03CR) 10Awight: [C: 04-1] Decorator to populate the model table if it's not there (035 comments) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/406046 (https://phabricator.wikimedia.org/T184938) (owner: 10Ladsgroup) [17:30:45] Nice. Thanks. [17:31:10] awight, can you give me a quick status of your explorations of ContentHandler and MCR? [17:32:15] halfak: I abandoned MCR, it’s not mature enough and the slots aren’t a win for our use case. If the situation changes in the future, it’s easy enough to migrate the date into slots, but I doubt that will happen. [17:32:37] ContentHandler gives us a nice visualisation for JSON content, I think it’s right for us. [17:32:38] repo: [17:32:45] https://github.com/adamwight/mw-ext-JADE [17:33:31] cool. Any new thoughts on either (1) doing everything in MW or (2) having an external, event-based source of truth? [17:33:32] That’s the minimum amount of glue to support our new namespaces (JADE and JADE_Talk) [17:33:40] halfak: I want to stick with (2) [17:33:50] IMO it’s very compatible with MW’s eventbus [17:34:20] Our MW extension will receive JADE events, perform actions, then MW will emit events that we listen to [17:34:21] OK then work should continue on wiki-ai/jade, but probably with an eye towards the page/revision compatibility? [17:34:39] That’s pretty much the textbook event-based architecture as described by the Confluent guy [17:34:46] Oh... so there's two sources of events, but one source of truth? [17:34:59] It’s okay that events come from many places [17:35:01] Or maybe there's MW event truth and JADE event truth? [17:35:05] The event stream itself is the source of truth [17:35:18] I like that. [17:35:50] So, e.g. a suppression event would be emitted from MW and then JADE would emit its own suppression event in response for JADE listeners? [17:35:57] yeah your formulation is a good way to put it, because each subsystem only listens to a subset of events, it isn’t interested in truths outside of its own domain. [17:36:02] halfak: exactly. [17:36:10] cool. I'm in. Makes sense. [17:36:17] My last thoughts about it are captured in the drawing... [17:36:25] * halfak turns down stress levels about JADE slightly. [17:36:41] jaja [17:37:03] https://docs.google.com/drawings/d/1Lagl0BJWVWHNvHLy5y6RNNKvl0C1tdVrE5YniwgqFJY/edit [17:37:06] OK I'm going to do lunch and then start in on QR stuff. [17:37:11] I grew attached to ^ that [17:37:28] I think we can avoid all synchronous calls from JADE to MW [17:37:56] ah darn, except the CentralAuth one—but the users will be cached after the first access [17:38:32] Might need to talk to CentralAuth often to handle user rights changes (e.g. blocked, promoted, etc.) [17:38:36] Other than that wart, the API is just an adapter that listens to a REST interface and emits events, no tricky stuff with distributed transactions. [17:38:49] Actually, we listen to user change events [17:39:10] So once we know a user, I think we don’t need to call CentralAuth about that user ever again. [17:39:22] We do. What about if they got blocked? [17:39:28] There’s an event :D [17:39:36] mediawiki.user-blocks.change [17:39:48] Oh so we'd maintain parallel state. Hmm.. Sounds like we could make mistakes. [17:40:00] But it could work. [17:40:04] sure but it’s the same margin of error as a direct API call. [17:40:07] IMO [17:40:12] Fair enough [17:40:16] oh I see what you mean [17:40:27] If we misinterpret incoming events, that’s just a bug [17:40:35] It’s “impossible” to miss events [17:40:39] Right. [17:40:45] since Kafka guarantees at-least-once [17:41:19] OK I think I have a sense for what you are thinking. [17:41:29] Thanks for talking through it! [17:41:42] :) [17:41:48] Will be back in a bit [17:41:58] I got sort of inspired to write the MW extension, but have tried to hold off since it’s not on our roadmap [17:41:59] o/ [17:46:20] (03PS1) 10Ladsgroup: Make ScoreFetcher and SqlScoreLookup implement different interfaces [extensions/ORES] - 10https://gerrit.wikimedia.org/r/406609 (https://phabricator.wikimedia.org/T185534) [17:47:32] 10Scoring-platform-team (Current), 10User-Ladsgroup: New status report of scoring platform team, end of January 2018 - https://phabricator.wikimedia.org/T185535#3928463 (10Ladsgroup) a:05Ladsgroup>03None [17:48:15] (03CR) 10jerkins-bot: [V: 04-1] Make ScoreFetcher and SqlScoreLookup implement different interfaces [extensions/ORES] - 10https://gerrit.wikimedia.org/r/406609 (https://phabricator.wikimedia.org/T185534) (owner: 10Ladsgroup) [17:53:45] (03PS2) 10Ladsgroup: Make ScoreFetcher and SqlScoreLookup implement different interfaces [extensions/ORES] - 10https://gerrit.wikimedia.org/r/406609 (https://phabricator.wikimedia.org/T185534) [18:19:22] Amir1: Do we really run selenium under mw-vagrant? [18:19:35] Or is that supposed to be on the host machine? [18:19:49] awight: no, it's ran on new machine [18:19:52] every time [18:19:58] O_O [18:20:03] mw-vagrant is headless [18:24:58] Amir1: I want to smoke test this patch, but the selenium setup is baffling. let me know when you have a minute to help [18:25:17] Sure [18:25:41] Currently about to finish this editquality thing [18:26:49] nice [18:27:11] I have to run for 1-2 hr, fyi [18:27:27] I’m installing X on mw-vagrant just to say I did. [18:39:49] awight: I finished editquality, ended up dropping fawiki [18:40:00] Made another patch against the extension [18:40:07] good call wrt. triage [19:10:39] (03CR) 10Awight: "Not the fault of this patch, but I'm here to gripe about tests/selenium/README.md" (033 comments) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/402863 (https://phabricator.wikimedia.org/T184451) (owner: 10Etonkovidova) [19:39:58] WTF. ORES isn't working on Spanish Wikipedia [19:40:05] It's not enabled in rcfilters [19:42:27] PROBLEM - puppet on ORES-redis02.experimental is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [19:42:33] PROBLEM - puppet on ORES-worker05.experimental is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [19:43:37] ^ why is this still happening. [19:47:03] What? [19:47:14] halfak: whats up? [19:47:48] We're still getting warnings on machines that shouldn't get warnings. Maybe we've not adjusted the puppet for production icinga? [19:48:17] halfak thats beta [19:48:30] halfak: according to the host lists those exist [19:49:06] 10Scoring-platform-team (Current), 10ORES, 10Operations, 10Patch-For-Review: Reimage ores* hosts with Debian Stretch - https://phabricator.wikimedia.org/T171851#3928820 (10akosiaris) It's half done (codfw but not eqiad). I 've been stalling it on T182799 so that we don't get hosts in a non-working state al... [19:53:29] 10Scoring-platform-team (Current), 10ORES, 10Operations, 10Patch-For-Review: Reimage ores* hosts with Debian Stretch - https://phabricator.wikimedia.org/T171851#3928840 (10Halfak) Sorry for the confusion. T182799 is done. Will resolve. [19:53:54] 10Scoring-platform-team, 10ORES: Switch ORES to dedicated cluster - https://phabricator.wikimedia.org/T168073#3928845 (10Halfak) [19:53:57] 10Scoring-platform-team (Current), 10ORES, 10Patch-For-Review: Make sure ORES is compatible with stretch - https://phabricator.wikimedia.org/T182799#3928843 (10Halfak) 05Open>03Resolved [20:03:34] Zppix, I got this in an email 12 hours ago "PROBLEM alert - ores.wmflabs.org/ORES web node labs ores-web-03 is CRITICAL" [20:03:45] We don't have an ores-web-03 anymore [20:04:26] halfak is that in shinken? [20:04:34] as we doin't send emails with icinga2 [20:04:35] what's a shinken? [20:04:41] It's the prod icinga [20:04:45] a nagios fork. [20:04:50] it's similar to icinga. [20:04:57] icinga.wikimedia.org [20:07:18] halfak: whats the from: in that email [20:07:41] icinga@einsteinium.wikimedia.org [20:07:47] Thats prod [20:07:50] right [20:08:00] halfak: ill write a patch to fix prod [20:08:33] thanks :) [20:08:48] Don't forget to make it a task so that we can claim it as work that got done :) [20:09:08] 10Scoring-platform-team, 10Icinga, 10User-Zppix: Remove old ores* hosts - https://phabricator.wikimedia.org/T185929#3928916 (10Zppix) [20:09:36] halfak: way ahead of you [20:11:27] RECOVERY - puppet on ORES-worker05.experimental is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [20:11:57] RECOVERY - puppet on ORES-redis02.experimental is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [20:17:30] awight: around? [20:20:56] Amir1: guess now [20:20:58] *not [20:22:42] awight: Amir1 wanted you [20:23:06] Amir1: hey, I’m mostly AFK, fixing a friend’s phone screen >.< [20:23:24] awight: oh, cool, let me know when you have some minute to review some stuff :D [20:23:39] I installed chrome, wdio-*, xfce4, and chromedriver within mw-vagrant [20:23:44] it’s miserable :-] [20:24:19] :)))) [20:25:52] (03PS3) 10Ladsgroup: Decorator to populate the model table if it's not there [extensions/ORES] - 10https://gerrit.wikimedia.org/r/406046 (https://phabricator.wikimedia.org/T184938) [20:25:55] (03PS13) 10Ladsgroup: Create Selenium UI tests for ORES damaging and good faith filters. [extensions/ORES] - 10https://gerrit.wikimedia.org/r/402863 (https://phabricator.wikimedia.org/T184451) (owner: 10Etonkovidova) [20:29:33] (03CR) 10jerkins-bot: [V: 04-1] Decorator to populate the model table if it's not there [extensions/ORES] - 10https://gerrit.wikimedia.org/r/406046 (https://phabricator.wikimedia.org/T184938) (owner: 10Ladsgroup) [20:29:39] (03CR) 10jerkins-bot: [V: 04-1] Create Selenium UI tests for ORES damaging and good faith filters. [extensions/ORES] - 10https://gerrit.wikimedia.org/r/402863 (https://phabricator.wikimedia.org/T184451) (owner: 10Etonkovidova) [20:30:53] awight, what's the current upper-bound estimate for the new ORES cluster? 4k scores per minute? [20:32:34] (03PS14) 10Ladsgroup: Create Selenium UI tests for ORES damaging and good faith filters. [extensions/ORES] - 10https://gerrit.wikimedia.org/r/402863 (https://phabricator.wikimedia.org/T184451) (owner: 10Etonkovidova) [20:32:42] shoot [20:33:26] (03PS4) 10Ladsgroup: Decorator to populate the model table if it's not there [extensions/ORES] - 10https://gerrit.wikimedia.org/r/406046 (https://phabricator.wikimedia.org/T184938) [20:33:29] (03PS15) 10Ladsgroup: Create Selenium UI tests for ORES damaging and good faith filters. [extensions/ORES] - 10https://gerrit.wikimedia.org/r/402863 (https://phabricator.wikimedia.org/T184451) (owner: 10Etonkovidova) [20:43:43] (03CR) 10Ladsgroup: Decorator to populate the model table if it's not there (034 comments) [extensions/ORES] - 10https://gerrit.wikimedia.org/r/406046 (https://phabricator.wikimedia.org/T184938) (owner: 10Ladsgroup) [20:46:54] (03CR) 10Ladsgroup: [C: 031] "It looks fine for me but I would suggest testing this in beta cluster before moving forward (and testing rollback to make sure it's timely" [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/392682 (https://phabricator.wikimedia.org/T181071) (owner: 10Awight) [20:54:59] halfak about prod icinga... [20:55:17] halfak: unless we get a op to agree to deployment we will need to wait a week to deploy anything [20:55:39] halfak: releng canceled all swats this week [20:56:01] * mutante random op appears [20:56:07] Zppix, no sweat. I don't think we need any swat for this. [20:56:10] Zppix: what did you want to do with prod Icinga? [20:56:14] I think it just needs to get merged. [20:56:26] prod-icinga would not be affected by MW deployment [20:56:32] or did you mean Puppet swat? [20:56:32] mutante, prod icinga is pointing toward some old hostnames we have in labs (sort of a meso-level prod thing going on) [20:56:46] mutante: swat is cancel weeklog [20:56:48] Weeklong [20:56:56] which kind of swat though [20:56:59] I'd imagine mutante knows that )^_^ [20:57:03] mutante: both [20:57:13] ok [20:57:16] Zppix, I think we just need to merge the changes for puppet to be affected. [20:57:24] mutante: greg sent an email to wikitech-l [20:57:31] was there an existing change for icinga? [20:57:36] waiting in gerrit [20:57:36] mutante: no [20:57:39] ok [20:57:47] mutante: i was going to write one up here soon [20:58:10] I think the confusion is that "swat" != "all potential changes to prod" [20:58:11] to remove hosts from icinga? [20:58:18] Yes [20:58:21] mutante, switch their hostnames [20:58:34] so the way that hosts get added to icinga is automatic [20:58:39] e.g. ores-web-03/05 are gone ores-web-01/02 took their place :) [20:58:44] Oh! Hmm [20:58:45] Its not puppet mutante ? [20:58:46] just because a host is known by the puppetmaster means it gets added [20:58:58] if it should not be known, then commands need to be run on the puppetmaster [20:59:04] to revoke the puppet cert [20:59:10] mutante: oh so its not gerrit ? [20:59:18] likely not a gerrit change, yea [20:59:35] mutante: if i give you a list of changes could you do it [21:00:23] maybe :) let's start with a list of changes , yes [21:00:35] Let me get that [21:03:05] so .. ores-web are cloud VPS [21:03:15] or am i wrong [21:03:26] aka "labs instances" [21:03:35] mutante: https://wikitech.wikimedia.org/w/index.php?title=ORES/Instances&diff=1780854&oldid=1778638 [21:03:37] i see what you mean now :) [21:04:15] FWIW, I don't expect the names to change again. :) [21:04:31] so, we have a check on -03 and -05 [21:04:43] and we would replace that with -01 and -02.. gotcha [21:04:53] i see the existing ones are "WARN" in icinga prod, yes [21:05:15] maybe wasnt so obvious because it was just WARN and not CRIT [21:06:05] the things i said about hosts being added automatically are still true ... [21:06:11] BUT this still needs a Gerrit change :) [21:06:33] it's a special case [21:06:45] the "host" for icinga is "ores.wmflabs.org" [21:07:02] and the "services" on that "host" are the node on 03 and the node on 05 [21:07:09] i'll fix it. hold on [21:10:00] mutante: https://phabricator.wikimedia.org/T185929 [21:10:02] Fyi [21:10:28] ah, thanks!:) [21:11:46] 10Scoring-platform-team, 10Icinga, 10monitoring, 10Patch-For-Review, 10User-Zppix: Remove old ores* hosts - https://phabricator.wikimedia.org/T185929#3929233 (10Dzahn) [21:12:18] 10Scoring-platform-team, 10Icinga, 10monitoring, 10Patch-For-Review, 10User-Zppix: Remove old ores* hosts - https://phabricator.wikimedia.org/T185929#3928916 (10Dzahn) [21:12:51] 10Scoring-platform-team, 10Icinga, 10monitoring, 10Patch-For-Review, 10User-Zppix: Remove old ores* hosts - https://phabricator.wikimedia.org/T185929#3928916 (10Dzahn) [21:19:36] Zppix: is that ticket only about Icinga or is there more to do to remove them elsewhere [21:19:53] delete instances in horizon f.e. [21:21:11] awight, am I in the wrong call? [21:21:17] Are we talking docs now? [21:21:28] Maybe it's good to skip a week after allhands :) [21:22:00] [21:22:00] ORES web node labs ores-web-02 [21:22:00] [21:22:22] paste fail. the point was that it's OK and green now [21:32:33] Thanks for your help mutante [21:32:35] And Zppix :) [21:32:48] Glad to have that squared away :) [21:34:46] i am just waiting for -01 to also report OK.. maybe puppet didnt run yet [21:34:52] Ok [21:35:07] Zppix: is it resolved after they are gone from prod Icinga? [21:35:14] Yes [21:35:17] or do they still exist in labs [21:35:19] or soemthing [21:35:23] They are gone [21:35:26] They dont exist [21:35:32] (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [extensions/ORES] - 10https://gerrit.wikimedia.org/r/406735 (owner: 10L10n-bot) [21:36:43] awight https://wiki.org.uk/landing/ heh [21:37:18] https://wiki.org.uk/landing/#team [21:37:27] Lol staph [21:37:32] Its not done paladoz [21:37:37] oh [21:38:06] 10Scoring-platform-team, 10Icinga, 10monitoring, 10Patch-For-Review, 10User-Zppix: Remove old ores* hosts - https://phabricator.wikimedia.org/T185929#3929367 (10Luke081515) a:05Zppix>03Dzahn [21:38:12] Im joking paladoz [21:38:14] Paladox* [21:38:20] lol [21:38:22] ok [21:46:16] 10Scoring-platform-team, 10Icinga, 10monitoring, 10Patch-For-Review, 10User-Zppix: Remove old ores* hosts - https://phabricator.wikimedia.org/T185929#3929404 (10Dzahn) 05Open>03Resolved a:05Dzahn>03Zppix Notice: /Stage[main]/Icinga/Nagios_service[einsteinium ores_web_node_labs_ores-web-03]/ensure... [21:46:50] halfak: done ^ [21:46:51] 10Scoring-platform-team, 10Icinga, 10monitoring, 10Patch-For-Review, 10User-Zppix: Remove old ores* hosts - https://phabricator.wikimedia.org/T185929#3929408 (10Dzahn) a:05Zppix>03Dzahn [21:47:10] mutante: choose who your gonna assign :P [21:48:38] Zppix: ok :) yep, prod icinga clean now [21:49:56] ty [21:56:50] \o/ [21:57:05] I'm stepping away for a bit. Should be on a little later. [23:05:04] halAFK: The actual maximum thoroughput of the old cluster was 4k/min, and we also maxes out at about 4.5k/min on the new cluster. The theoretical upper bound is much higher, but various bottlenecks are throttling us. [23:05:58] In case you want the reference, https://phabricator.wikimedia.org/T169246 [23:12:12] thanks awight [23:12:21] Almost put something stupid in our QR deck :) [23:13:24] * halAFK runs away again [23:13:27] I thought the strategy was usually to put as many stupid things in the deck as can be managed, to see if anyone calls it out [23:17:27] Amir1: Are you still around? It looks like this UI test is the last unreviewed patch, unless I missed something? [23:19:21] (03CR) 10Awight: "Sure, I'll try this on beta before we merge." [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/392682 (https://phabricator.wikimedia.org/T181071) (owner: 10Awight) [23:53:47] (03PS6) 10Awight: Build venv into versioned source dir [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/392682 (https://phabricator.wikimedia.org/T181071)