[02:55:17] halfak [09:53] PROBLEM - Puppet errors on deployment-ores-redis-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [04:23:42] (03Draft2) 10Felipe L. Ewald: Adjusting the punctuation. [extensions/ORES] - 10https://gerrit.wikimedia.org/r/365544 [06:29:04] PROBLEM - https://grafana.wikimedia.org/dashboard/db/ores grafana alert on einsteinium is CRITICAL: CRITICAL: https://grafana.wikimedia.org/dashboard/db/ores is alerting: 5xx rate (Change prop) alert. [08:02:14] 10Scoring-platform-team, 10ORES, 10User-Zppix: Extend icinga check to catch 500 errors like those of the 20170613 incident - https://phabricator.wikimedia.org/T167830#3442795 (10akosiaris) https://gerrit.wikimedia.org/r/#/c/363890/ has been merged on Jul 10th. Is the merged change above enough or is there mo... [12:51:02] 10Scoring-platform-team, 10Wikilabels, 10User-Ladsgroup: Apply mediawiki core styling convention on javascript files of wikilabels - https://phabricator.wikimedia.org/T169578#3443297 (10AnotherLadsgroup) https://github.com/wiki-ai/wikilabels/pull/192 [12:55:11] 10Scoring-platform-team, 10Wikilabels: Wikilabels uses deprecated api calls - https://phabricator.wikimedia.org/T170758#3443299 (10AnotherLadsgroup) Found this: https://lists.wikimedia.org/pipermail/mediawiki-api-announce/2017-June/000134.html [14:07:00] !!! [14:07:08] Looks like ORES was upset [14:27:19] halfak: I can't find anything [14:27:21] what's up [14:28:10] Oh! We got a ping from our icinga alert. [14:28:14] Some 500s happened. [14:28:30] https://grafana.wikimedia.org/dashboard/db/ores?panelId=23&fullscreen&orgId=1 [14:28:56] It happened at 0630 UTC [14:29:16] It was very brief. [14:29:42] I only saw the problem start but no recovery [14:31:02] it seems it happens from time to time [14:31:08] 10Scoring-platform-team, 10ORES, 10User-Zppix: Extend icinga check to catch 500 errors like those of the 20170613 incident - https://phabricator.wikimedia.org/T167830#3443812 (10Halfak) It looks like we just got an Icinga ping for a 500 response spike (very brief overload event) so it is certainly working :)... [14:31:25] 10Scoring-platform-team, 10ORES, 10User-Zppix: Extend icinga check to catch 500 errors like those of the 20170613 incident - https://phabricator.wikimedia.org/T167830#3346179 (10Halfak) 05Open>03Resolved [14:32:23] Amir1, do you know why updates aren't getting posted from https://gerrit.wikimedia.org/r/#/c/365289 to https://phabricator.wikimedia.org/T170709 ? [14:32:31] I must be doing something wrong. [14:32:47] 10Scoring-platform-team, 10ORES, 10Puppet: Add greek dict to ores puppet base - https://phabricator.wikimedia.org/T170709#3440098 (10Halfak) 05Open>03Resolved [14:32:49] 10Scoring-platform-team, 10ORES, 10editquality-modeling, 10artificial-intelligence: ORES deployment - Mid July, 2017 - https://phabricator.wikimedia.org/T170485#3443824 (10Halfak) [14:33:47] Oh yeah, there should be no new lines after Bug: T#### [14:38:26] Oh so Bug: ### should be the last thing in the message? [14:38:29] After the ChangeId? [14:39:38] no before that [14:39:52] gerrit add it automatically [14:40:19] Yeah. OK so it's "Bug: T####\nChangeId: ...." [14:40:29] yup [14:40:31] halfak: https://gerrit.wikimedia.org/r/#/c/299963/ [14:40:34] see this for example [14:40:37] The problem was that I had \n\n instead of \n [14:40:43] yeah [14:40:52] Aha! Thanks :) [14:40:53] halfak: also I have a pr for review in wikilabels :D [14:40:57] Also WTF damn it. ;) [14:41:05] OK will look at that :D [14:41:21] Just finished the copyright statement on the Keilana paper. It's always nice to be submitting the photo-ready for a paper :) [14:41:35] Congrats! [14:41:59] Here it is in its CC-BY-SA glory: https://github.com/halfak/keilana-effect/blob/master/halfaker2017keilana.with_fonts.pdf [14:42:18] It cost me $700 USD to make it open access :( [14:42:24] We don't have the budget to cover it. [14:43:09] lol I just saw I have some broken refs in there. Time to fix more things. [14:43:36] WTF [14:43:54] Really, WMF should have budget for that [14:44:22] Right. I'll be digging to see if I can find something [15:13:04] OKAY! Time to start working on the deployment of ORES! [15:13:07] So much new stuff! [15:13:17] PROBLEM - puppet on ores-web-04 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [15:13:36] Oh shaddup. We should probably just kill ores-web-04 and start from scratch. [15:13:40] It's not event pooled right now [15:13:42] *even [15:14:02] halfak :) [15:14:13] you can do it, by deleting it and recreating it :) [15:14:31] it seems that it is intermittent errors as it recovers shortly after [15:16:08] https://twitter.com/FixmeBot used the ORES logo :D [15:20:17] 10Scoring-platform-team, 10Documentation: Add ORES technical documentation - https://phabricator.wikimedia.org/T170069#3444001 (10Sumit) The above page is complete in documentation of technical details and is linked from https://www.mediawiki.org/wiki/ORES hence closing. [15:20:43] 10Scoring-platform-team, 10ORES, 10Documentation, 10Epic: [Epic] Clean up ORES service documentation - https://phabricator.wikimedia.org/T148974#3444004 (10Sumit) [15:20:45] 10Scoring-platform-team, 10Documentation: Add ORES technical documentation - https://phabricator.wikimedia.org/T170069#3418525 (10Sumit) 05Open>03Resolved [15:40:04] 10Scoring-platform-team, 10Wikidata, 10editquality-modeling, 10User-Ladsgroup, 10artificial-intelligence: Add basic bad word check to Wikidata feature set - https://phabricator.wikimedia.org/T170834#3444124 (10AnotherLadsgroup) [15:42:05] Confirmed that web-04 is not pooled https://wikitech.wikimedia.org/wiki/Hiera:Ores [15:42:27] halfak: I'd say just remove it and don't make it unless needed [15:42:56] Amir1, OK. Was thinking the same. [15:43:13] This gives us a little overhead in our quota in case we want to remake a celery worker or something like that. [15:43:18] PROBLEM - puppet on ores-web-04 is UNKNOWN: connect to address 10.68.23.45 port 5666: No route to hostconnect to host ores-web-04.ores.eqiad.wmflabs port 5666: No route to host [15:43:22] PROBLEM - ping4 on ores-web-04 is UNKNOWN: check_ping: Invalid hostname/address - ores-web-04.ores.eqiad.wmflabsUsage:check_ping -H -w ,% -c ,% [-p packets] [-t timeout] [-4 [15:43:37] paladox, ^ [15:43:37] :) [15:43:42] ah [15:43:48] halfak are you recreating it? [15:43:52] na. [15:43:55] Not for now anyway [15:43:55] ok [15:43:57] PROBLEM - check users on ores-web-04 is WARNING: Could not resolve hostname ores-web-04.ores.eqiad.wmflabs: Name or service not known [15:43:57] PROBLEM - Host ores-web-04 is DOWN: check_ping: Invalid hostname/address - ores-web-04.ores.eqiad.wmflabsUsage:check_ping -H -w ,% -c ,% [-p packets] [-t timeout] [-4 [15:44:06] 10Scoring-platform-team, 10Wikidata, 10editquality-modeling, 10User-Ladsgroup, 10artificial-intelligence: Add entropy-related and uppercase-related measures to comments - https://phabricator.wikimedia.org/T170835#3444142 (10AnotherLadsgroup) [15:44:09] halfak i guess you want it removed? [15:44:19] Unless you are recreating a new instance with a different name? [15:45:32] Right. Let's just remove it [15:45:39] ok [15:46:20] 10Scoring-platform-team, 10Wikidata, 10editquality-modeling, 10User-Ladsgroup, 10artificial-intelligence: Add entropy-related and uppercase-related measures to comments - https://phabricator.wikimedia.org/T170835#3444159 (10Halfak) How would this measurement look in practice? Something like `UPPERCASE... [15:48:08] halfak https://gerrit.wikimedia.org/r/#/c/365620/ [15:48:51] thanks paladox :) [15:48:58] your welcome :) [15:52:03] 10Scoring-platform-team, 10Wikidata, 10editquality-modeling, 10User-Ladsgroup, 10artificial-intelligence: Add entropy-related and uppercase-related measures to comments - https://phabricator.wikimedia.org/T170835#3444211 (10AnotherLadsgroup) It will be one of the features, but things like "Longest repeat... [15:53:02] Arg. And now it is meeting time and I didn't accomplish any deploy prep. [15:53:13] But I did do a bunch of outreachy stuff and email. [15:53:15] And web-04 [15:53:19] So that's not bad. :) [15:54:32] 10Scoring-platform-team, 10ORES, 10articlequality-modeling, 10artificial-intelligence: Deploy new trwiki article quality model - https://phabricator.wikimedia.org/T170838#3444222 (10Halfak) [15:55:10] I'm going to self-merge https://gerrit.wikimedia.org/r/#/c/365354/ [15:55:15] Amir1, ^ fyi [15:55:30] This adds albanian language stuffs [15:55:41] And fixes some issues with dictionaries. [15:56:15] FYI: https://github.com/wiki-ai/revscoring/tree/1.3.x [15:56:29] I've been working from this and I'm hoping we can rebase master on it eventually. [15:57:00] +2 on my side, don't have access yet [15:57:00] Next time we'll do a 2.0 ( or I guess 3.0) branch and work from that until we're ready to convert everything [15:57:06] cool :) thanks :) [15:58:18] o/ armalcolite :) [15:58:37] halfak: O/ [15:58:40] Does your research project involves some AI work? :D [15:58:54] ORES is <3 [15:58:59] \o/ [15:59:14] I am working mostly with machine learning & deep learning, so sort of YES! [15:59:36] Gotcha! I'm pretty sure the anti-harassment team is going to be interested in your studies then :) [16:00:37] I am sure I can contribute to ORES as well! Will surely find time to contribute something. [16:00:53] That would be great and very welcome. [16:01:05] I have to step away now. I have 3 hours of meetings! [16:01:27] Sure [16:01:43] and Thanks for a warm welcome. :) [16:05:41] o/ halfak [16:23:51] i wonder what this check_command = "hostalive" does heh [16:28:43] 10Scoring-platform-team, 10MediaWiki-extensions-ORES: Make list of features and locations of ORES Review Tool for handoff - https://phabricator.wikimedia.org/T167911#3444507 (10awight) [16:47:44] i think its similar to ping4 or something like that [17:15:03] RECOVERY - https://grafana.wikimedia.org/dashboard/db/ores grafana alert on einsteinium is OK: OK: https://grafana.wikimedia.org/dashboard/db/ores is not alerting. [17:15:37] question is the repo grafana dashboards what controls grafana.wikimedia.org/dashboard/db/ores? halfak? [17:18:42] grafana-admin.wikimedia.org [17:19:41] so what does the grafana dashboards repo on our github for halfak? [17:20:06] Unused. I think it can be deleted. Amir1 created that I think [17:20:15] Are we still wanting to use that to record snapshots? [17:20:20] Zppix: That's a dump of configuration, for history and interest. [17:20:38] halfak if amir is okay if with that ill delete it [17:20:51] Amir1 do you still need the grafana-dashboards repo on our github? [17:20:57] Maybe we should start getting better with taking snapshots and not deleted it :) [17:21:05] +1 [17:21:21] The dashboard admin UI has no "undo" or "view history" capability [17:21:37] so we want to keep the repo? im confused [17:22:18] My vote is to keep, because it's the only record we have of somewhat intricate config. If the config is inadvertently damaged, it'll be difficult to reconstruct without some history. [17:24:42] ok then [17:26:59] The README could use some love, to better explain the repo's purpose. [17:28:42] tell me what to put in there and i will do it awight [17:29:55] Zppix: yes, I need that [17:30:12] Amir1 ok [17:30:34] Zppix: thanks! If you choose to accept this mission, we could use a few sentences that explain that the files are a dump of current settings, that changes are made with grafana-admin.wmo, which is the authoritative source, and that you are welcome to run php fetch.php to grab the latest settings... [17:31:08] big bold letters, configuration is not read from this repo, any changes made here will be overwritten. [17:32:13] 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10MW-1.30-release-notes (WMF-deploy-2017-06-27_(1.30.0-wmf.7)), 10Patch-For-Review, and 2 others: [Discuss] Make ORES Review Tool preferences more prominent - https://phabricator.wikimedia.org/T167910#3444829 (10awight) There's been some related discuss... [17:35:18] ok will do awight just give me a min cause i got to brb [17:39:30] Zppix: We want this change *yesterday* :p [17:39:44] take your time, of course! [17:40:53] awight i will do it ASAP i just go to do osme quick irl things first [17:41:16] lol I was just messing around [17:41:30] Thanks again for noticing that the repo needs more context. [17:42:40] np [17:46:59] awight what should be used instead to change the grafana dashboard settings? [17:48:17] They can only be changed via the UI, https://grafana-admin.wikimedia.org/ [17:51:09] awight done [17:55:25] halfak can i directly merge a link syntax fix to grafana-dashboard its a minor fix so i think a pr is a waste of time [17:55:56] Zppix: either way works for me [17:56:36] ill just self-merge since its so minor [17:57:15] hehe paperwork to avoid paperwork [17:57:32] hey, that worked. [17:57:52] * awight stares hard at Markdown link syntax [17:58:11] awight yeah but you all have better things to do rather then look at pr that swaps parenthesis and brackets [17:58:33] awight markdown lang for links i rarely get wrong thats the first time i've screwed up links [17:58:33] I've come to accept that programming is just text processing. [17:59:25] so is that readme better awight? [17:59:58] Zppix: It reads well, that notice at the bottom certainly makes the requisite point. [18:00:35] I made sure if noone reads the whole readme they should atleast catch that notice ... [18:01:39] NOTE: you must read this readme to the end to find our tl;dr ;) [18:01:46] I think it's perfect for now. [18:06:08] 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10User-Ladsgroup: Make list of features and locations of ORES Review Tool for handoff - https://phabricator.wikimedia.org/T167911#3445041 (10awight) [18:24:02] halfak: Want to guide me through the deployment, or would you prefer to do it yourself? [18:35:27] 10Scoring-platform-team, 10Research-and-Data-Archive, 10Research-management: Reading-Research collaboration on ORES (start: Q3-FY17) - https://phabricator.wikimedia.org/T151259#3445321 (10ggellerman) 05Open>03Resolved [18:36:17] 10Scoring-platform-team, 10Data-release, 10Research-and-Data, 10articlequality-modeling, 10artificial-intelligence: Formal publication of article quality score dataset - https://phabricator.wikimedia.org/T145332#3445376 (10ggellerman) [18:36:19] 10Scoring-platform-team-Backlog, 10Data-release, 10Research-and-Data-Archive, 10articlequality-modeling, 10artificial-intelligence: Ask Figshare to remove file upload limit for Article Quality Score dataset - https://phabricator.wikimedia.org/T146708#3445374 (10ggellerman) 05Open>03Resolved [18:54:16] o/ awightfud. I forget, did you do the last deployment with me? [19:01:31] 10Scoring-platform-team, 10Edit-Review-Improvements-RC-Page, 10ORES, 10Collaboration-Team-Triage (Collab-Team-Q4-Apr-Jun-2017), and 2 others: Conform ORES sensitivity levels to the new ERI standards - https://phabricator.wikimedia.org/T160575#3445603 (10Etonkovidova) Re-tested with the ORES scoring ranges... [19:28:46] halfak: I watched some screensharing [19:28:53] You were typing awfully fast... [19:30:58] Gotcha. Yeah, let's have you do it this time then :) [19:31:09] what's the schedule? [19:31:27] https://wikitech.wikimedia.org/wiki/Deployments [19:31:29] * halfak looks too [19:31:46] 10 PDT [19:31:49] 12 UTC [19:31:52] *CDT [19:31:59] 17 UTC [19:32:00] :D [19:32:29] * awight blinks [19:32:32] I see 20:00 UTC [19:32:39] https://wikitech.wikimedia.org/wiki/Deployments#Tuesday.2C.C2.A0July.C2.A018 [19:32:47] ah, tomorrow. [19:32:54] :P :D [19:33:06] That's like a whole 'nother day. [19:33:09] So let's work together to get the deploy ready today and then we'll be all set for tomorrow. :D [19:33:13] OK great [19:33:19] I'll have you do the beta deployment today then too. [19:33:25] + [19:33:26] 1 [19:34:38] OK so I'm part way through a PR for ores-wmflabs-deploy [19:34:42] I'll have you review it shortly. [19:34:51] o/ [19:34:53] Then you get to make a similar set of changes to our deploy repo [19:34:58] *prod deploy repo [19:35:04] and I'll review. [19:35:32] awight, is a deploy to beta at 1500 PDT going to work for you? [19:35:37] yep [19:35:45] Cool [19:35:49] * halfak rushes out a PR [19:51:45] I don't even see how to make that patch in ores-prod-deploy. [19:51:59] Must not be what to do. [19:52:37] awight iirc you just pr from ores master to master in ores-prod-deploy [19:54:38] ? [19:54:49] Oh! We use gerrit for the prod deploy repo [19:55:53] halfak: sorry, there's a simple explanation. You haven't pushed the ores-wmflabs-deploy change yet :) [19:55:59] Oh yes [19:56:00] right [19:56:07] I will officialy hold horses [19:56:57] We have a crappy verion issue problem. I'm working on it now :( [19:57:28] halfak what version problem? [19:57:46] wait for the pr [20:10:58] halfak i got a question, you have a min? [20:43:01] halfak: Looking at the fiwiki flaggedrevs task for a minute. You were mentioning that I should merge the flaggedrevs-approved revisions with the existing autolabeled set? [20:43:06] That confuses me. [20:43:45] halfak i figured it out nevermind [20:43:45] Autolabeled revisions won't have anything for "approved", so I'm not sure how mixing them in can help. [20:44:21] Sorry meeting meeting meeting [20:44:23] Zppix, ^ [20:45:23] halfak: Should I review+merge, then prepare a merge to production? [20:45:40] Yes. But I messed something up. [20:45:43] Fixing :| [20:45:50] halfak meeting this late? wow [20:47:11] awight, OK ready for you at https://github.com/wiki-ai/ores-wmflabs-deploy/pull/84 [20:47:18] Zppix, yeah... today was a rough one. [20:47:29] Just about to start another meeting [20:47:50] We like to keep the managers out of our hair by giving them lots of "very important" meetings [20:48:56] lol [20:49:33] But wait, I'm a manager-researcher-engineer. [20:49:53] Scientific engi-manager [20:50:16] sorry that we have to ghettoize you with your people [20:51:37] Alright. Now for the real test. [20:51:51] Now that you have the code in master [20:51:58] I want you to pip install fabric [20:52:33] Then do the staging step for https://wikitech.wikimedia.org/wiki/ORES/Deployment#Labs_.28ores.wmflabs.org.29 [20:52:47] Which is just going into the base dir of the repo and running "fab stage" [20:53:01] kk [20:53:04] https://ores-staging.wmflabs.org will be waiting for you once the process finishes. [20:53:12] * awight waits for submodules to loooad [20:56:18] Docs mention "ores-wikimedia-config", is that outdated? [20:56:20] 10Scoring-platform-team, 10ORES, 10editquality-modeling, 10artificial-intelligence: ORES deployment - Mid July, 2017 - https://phabricator.wikimedia.org/T170485#3446025 (10Halfak) https://github.com/wiki-ai/ores-wmflabs-deploy/pull/84 was merged by @awight [20:56:29] outdated yeah [20:56:45] renamed to ores-wmflabs-deploy [20:56:49] fixing. [20:58:24] Starting next meeting. I'll be back for 1500 PDT beta-deploy [21:08:09] * awight furiously deletes things from hard drive to fit ores-wmflabs-deploy [21:12:43] Any idea why my submodules have a .git directory but no checked-out files? [21:13:51] I ran "git checkout" in each dir, but that shouldn't have been necessary. [21:18:29] halfak: fabric doesn't support python3! [21:21:10] "git submodule update --init" [21:21:17] In the base dir of the deploy repo [21:21:31] I did. it was a weird scene [21:21:40] :) [21:21:48] could have been caused by the initial call failing due to disk fullness [21:22:39] don't worry about my python3 comment, obviously I can work around. It was just an involuntary expression of shock. [21:22:50] or git clone --recursive :) [21:23:37] I'll play around with that when I'm in a good position to download 3GB again, but I did follow pretty reasonable steps. [21:27:05] o/ revi [21:27:13] I need to talk to you about a terrible troll from kowiki [21:28:30] * halfak does his meeting to-dos while still in the meeting [21:30:46] I gotta read the logs, but I'm getting a 500 error from http://ores-staging.wmflabs.org/v3/scores/bnwiki/reverted/2521799 [21:33:14] r[24158]: Hspell: can't open /usr/share/hspell/hebrew.wgz.sizes. [21:33:27] must be unrelated, but looks fishy [21:36:20] We have three layers of staging? ores-staging, ores.wmflabs, and ores-beta? [21:38:06] right [21:38:11] sort of [21:46:54] halfak: Just to spare us some low-bandwidth IRC scroll, my preoccupation at the moment is diagnosing the staging fail. Where are the logs? syslog had unrelated errors. [21:47:57] /var/log/ores/main.log hasn't been written to since Jul 14th. [21:52:55] ooh that must have been jul 14th of last year. [21:53:16] awight do ls -la [21:53:23] hmm where? [21:53:27] ls -la /var/log/ores/main.log [21:53:32] ls -la /var/log/ores/ [21:53:51] it will list the datas will also list all the files in there [21:53:57] No dotfiles present [21:54:06] or do ls -la /var/log/ [21:54:10] fwiw I was using ls -ltr to show the most recent changes last [21:54:20] oh [21:56:40] logging_config.yaml has some clues, but I'm suspecting it doesn't do anything unless I paste into 00-config.yaml? [21:56:46] OK back [21:56:48] So. [21:56:56] No worries about that weird hspell thing [21:57:11] good timing! /me replaces most recent chunk of torn hair [21:57:14] lol [21:57:17] * halfak looks at staging [21:58:13] Here's what I did. Recent change on bnwiki, https://bn.wikipedia.org/w/index.php?title=%E0%A6%AA%E0%A7%8D%E0%A6%B0%E0%A6%A7%E0%A6%BE%E0%A6%A8_%E0%A6%AA%E0%A6%BE%E0%A6%A4%E0%A6%BE&diff=2521799&oldid=2433094 [21:58:26] Where I assumed I might find it on staging, http://ores-staging.wmflabs.org/v3/scores/enwiki/reverted/2521799?features [21:58:41] oops. http://ores-staging.wmflabs.org/v3/scores/bnwiki/reverted/2521799?features <-- that one [21:59:16] Working URL for comparison, https://ores-staging.wmflabs.org/v2/scores/enwiki/reverted/641962088?features [21:59:29] * halfak does pushups [21:59:44] awight http://ores-staging.wmflabs.org/v3/scores/enwiki/reverted/ [21:59:47] shows 500 [21:59:57] so /reverted/ is throwing the 500 on the v3 api [22:00:51] Nice sleuthing! https://ores-staging.wmflabs.org/v3/ [22:01:21] ah, also 500s on https://ores-staging.wmflabs.org/v2/ [22:02:02] halfak: um, once you are pumped up, lmk if I should copy logging_config into 00-main? [22:02:58] awight, you shouldn't have to copy anything [22:03:23] OK so this isn't supposed to 500 [22:03:25] Hmm [22:03:52] lol @ this isn't supposed to 500 [22:05:23] arg! well I see some errors in the log that are related to config issues [22:07:55] https://github.com/wiki-ai/ores/blob/master/config/00-main.yaml#L7 [22:07:57] Line 7 to 10 [22:08:26] https://github.com/wiki-ai/ores/blob/master/config/00-main.yaml#L7-L10 btw [22:08:34] :P [22:08:44] I'm looking. What's the problem? http vs https? [22:09:05] https://github.com/wiki-ai/ores-wmflabs-deploy/pull/85 [22:09:20] It wants to show a 404 error and fails [22:09:25] lol [22:09:28] A standard wikimedia 404 error [22:09:41] brb [22:10:01] I'll deploy that... [22:11:50] OK so, you asked earlier about staging --> wmflabs --> beta --> prod [22:12:01] So wmflabs is meant for experimental models + prod models. [22:12:11] E.g. we have some text complexity models that are only in wmflabs. [22:12:21] It's also intended to offset prod for big, batch scoring jobs. [22:12:33] How did you find this issue btw? Were there logs, or you just guessed correctly? [22:12:38] wmflabs deploy is essentially "experimental" prod [22:12:46] tail main.log [22:12:50] <_< [22:13:14] -rw-r----- 1 www-data www-data 6064737 Jul 14 2016 /var/log/ores/main.log [22:13:19] wat. [22:13:22] directory? [22:13:25] sudo tail main.log [22:13:28] :D [22:13:40] sudo -u www-data tail main.log [22:13:41] :) [22:13:59] bollox. What directory? [22:14:15] awight bollocks* [22:14:18] lol [22:14:30] :D thanks for the support [22:14:49] /srv/log/ores/ [22:14:51] awight i cant have typos man it leads to even more errors in the code xD [22:14:54] awkward. I haven't read the dictionary for that until now. [22:15:02] lol [22:15:13] Right. It is weird. [22:15:15] aha. okay thanks, documenting somethere... [22:15:26] I'll also move the decoy file [22:16:02] root@ores-staging-02:/var/log/ores# ls -l [22:16:03] total 5928 [22:16:03] lrwxrwxrwx 1 root root 21 Jul 17 22:15 app.log -> /srv/log/ores/app.log [22:16:05] lrwxrwxrwx 1 root root 22 Jul 17 22:15 main.log -> /srv/log/ores/main.log [22:16:08] -rw-r----- 1 www-data www-data 6064737 Jul 14 2016 main.log-2016 [22:19:14] http://ores-staging.wmflabs.org/v2/scores/bnwiki/reverted/2521799 looks good [22:19:26] For v3, I must have had the wrong URL [22:20:13] There it is. http://ores-staging.wmflabs.org/v3/scores/bnwiki/2521799 [22:20:25] halfak: OK i'm happy with staging. Now deploy to beta? [22:21:29] Now, I would usually do a deploy to ores.wmflabs.org but let's skip that for now and do beta. [22:21:42] But in order to do beta, we need to update a separate repo! [22:21:46] DRY, HA! [22:21:48] :D [22:21:57] We'll show those CS fools [22:22:25] https://phabricator.wikimedia.org/source/ores-deploy/ [22:22:38] You want to submit a patchset to this via gerrit [22:22:49] That roughly does the same things we did to ores-wmflabs-deploy [22:23:06] origin https://gerrit.wikimedia.org/r/mediawiki/services/ores/deploy (fetch) [22:23:09] ? [22:24:04] yeah. [22:24:04] https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/services/ores/deploy [22:24:15] and do you have a recommended way to copy the patch over, or shall I freeclimb? [22:24:24] git clone ssh://halfak@gerrit.wikimedia.org:29418/mediawiki/services/ores/deploy [22:26:54] * awight gets angsty reading git logs for both repos [22:27:29] fyi https://gerrit.wikimedia.org/r/#/c/359224/1/config/00-main.yaml [22:28:33] diff -ur ./ ~/work/ores-prod-deploy/ | wc -l [22:28:33] 4905 [22:29:17] ok. I'm going to limit self-inflicted mental damage, and just port over the exact PR we're working with today. [22:31:05] halfak: argh. also, I now see that I never rebased the labs deploy branch, instead I deployed master. [22:32:14] Oh wait. Did you do a deploy in labs? [22:32:34] You should have just staged [22:32:54] deploy = "fab deploy_web" and "fab deploy_celery? [22:32:56] I did "fab stage" [22:33:05] Oh just fine then. That comes from master [22:33:08] I see what you mean now [22:33:21] argh, sorry I'm not doing anything like following these instructions [22:33:28] No worries :D [22:33:40] but now I've gone ahead and deployed "deploy" to staging which seems to be wrong [22:34:04] Shall I leave it like that, or is it worth putting staging back on master? [22:35:39] How did you manage to deploy "deploy" to staging? [22:36:00] how did I do it technically, or why did my brain malfunction? [22:36:08] How did you do it technically ;) [22:36:10] It was easy :) [22:36:29] git checkout deploy; git rebase origin/master; fab stage [22:36:37] Is there supposed to be a safety interlock? [22:36:47] Oh you still didn't deploy "deploy" :) [22:36:57] O_o [22:37:01] what do you mean? [22:37:11] Look, who is on first [22:37:29] fab doesn't send your current repo, it tells the remote server to update the report and checkout a specific branch. Staging knows it is supposed to get "master" [22:37:41] oho tricky [22:37:47] /srv/ores/config/ has master checked out right now [22:40:47] who's on first was a cultural ref btw, not actually trying to be a tough guy [22:41:53] So where was that wrecking ball. k I'm doing something ad-hoc to copy changes over to the prod repo. [22:42:11] awight, no worries, I know that comedy bit :D [22:42:34] me usually reviews a diff and makes the changes manually. [22:43:20] I'm trying format-patch [22:43:43] * halfak types "/" better [22:55:52] halfak: need help. why doesn't ores/deploy have a .gitreview file? [22:56:04] Where do I read about how to push changes to that repo? [22:56:55]