[00:03:32] me too [00:09:07] (03PS2) 10Andrew Bogott: Keystone: Publish credentials for novaobserver account [labs/private] - 10https://gerrit.wikimedia.org/r/326979 (https://phabricator.wikimedia.org/T150092) [00:13:04] (03CR) 10Andrew Bogott: [V: 032 C: 032] Keystone: Publish credentials for novaobserver account [labs/private] - 10https://gerrit.wikimedia.org/r/326979 (https://phabricator.wikimedia.org/T150092) (owner: 10Andrew Bogott) [01:54:04] 06Labs, 10Tool-Labs: Set up and use exported resources for Tool Labs's shared knowledge - https://phabricator.wikimedia.org/T153163#2871558 (10scfc) [05:25:40] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [06:00:42] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [06:35:28] PROBLEM - Puppet run on tools-worker-1014 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [06:38:44] PROBLEM - Puppet run on tools-exec-1411 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [06:43:52] 06Labs, 10Tool-Labs, 13Patch-For-Review: Offer Korean Locales "ko_KR.euckr" and "ko_KR.utf8" on Tool Labs - https://phabricator.wikimedia.org/T130532#2871754 (10Dzahn) merged on prod puppetmaster, ran puppet on tools-exec-1401, tried to confirm that ko locales package is installed now, but not yet... [06:45:22] 06Labs, 10Tool-Labs, 13Patch-For-Review: Offer Korean Locales "ko_KR.euckr" and "ko_KR.utf8" on Tool Labs - https://phabricator.wikimedia.org/T130532#2871768 (10Dzahn) ``` root@tools-exec-1401:~# puppet agent -tv .. Info: Caching catalog for tools-exec-1401.tools.eqiad.wmflabs Info: Applying configuration ve... [06:47:29] PROBLEM - Puppet run on tools-exec-1409 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [06:53:11] PROBLEM - Puppet run on tools-cron-01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [07:10:28] RECOVERY - Puppet run on tools-worker-1014 is OK: OK: Less than 1.00% above the threshold [0.0] [07:18:43] RECOVERY - Puppet run on tools-exec-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [07:22:28] RECOVERY - Puppet run on tools-exec-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [07:28:12] RECOVERY - Puppet run on tools-cron-01 is OK: OK: Less than 1.00% above the threshold [0.0] [08:51:33] (03Draft1) 10Paladox: Make bot ignore jenkins-bot v+2 again [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/327152 [08:51:38] (03Draft2) 10Paladox: Make bot ignore jenkins-bot v+2 again [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/327152 [10:53:05] RECOVERY - Puppet staleness on tools-worker-1017 is OK: OK: Less than 1.00% above the threshold [3600.0] [11:58:21] (03PS3) 10Paladox: Make bot ignore jenkins-bot v+2 again [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/327152 [11:59:57] (03CR) 10Paladox: [] Make bot ignore jenkins-bot v+2 again (031 comment) [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/327152 (owner: 10Paladox) [12:10:30] (03CR) 10Paladox: [] "@MarkTraceur hi, I know you doint like me doing self merges but I would like to self merge this as it fixes jenkins keeps commenting on ir" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/327152 (owner: 10Paladox) [12:10:49] (03CR) 10Paladox: [] "I can clear up the code later if you doint like it and think there's a better way." [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/327152 (owner: 10Paladox) [12:11:28] (03CR) 10Paladox: [C: 032] Make bot ignore jenkins-bot v+2 again [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/327152 (owner: 10Paladox) [12:11:33] (03CR) 10jenkins-bot: [] Make bot ignore jenkins-bot v+2 again [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/327152 (owner: 10Paladox) [12:12:03] (03Merged) 10jenkins-bot: Make bot ignore jenkins-bot v+2 again [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/327152 (owner: 10Paladox) [12:43:44] 06Labs, 10Tool-Labs, 13Patch-For-Review: Offer Korean Locales "ko_KR.euckr" and "ko_KR.utf8" on Tool Labs - https://phabricator.wikimedia.org/T130532#2872327 (10Ykhwong) Thanks for your help. ko_KR.utf8 could be found. However, I still cannot find EUC-KR for my application. Could you consider to add it as we... [13:06:41] paladox: I was hoping to be in a good mood today, but I see we're off to a bad start [13:07:11] marktraceur Not sure what you mean, i did explain my action. I also tested it. [13:07:58] paladox: Your explanation seems pretty thin. The noise cannot have been so bad that it needed a fix with no review. [13:08:59] Well one user did tell me that he got pinged by jenkins (wasent complaning about it, but still said something about it.) It was spamming irc with [13:09:00] (CR) jenkins-bot: [] Revert "mariadb: Depool db1092 for schema change" [mediawiki-config] [13:10:06] that was jenkins commenting that it will add it to the gate and submit pipeline which we doint allow but since gerrit 2.13 it now says all the values. [13:11:02] There were four messages, several minutes apart, one of which was caused by the user... [13:11:10] Hardly spam. [13:12:17] marktraceur i was trying to do a follow up to https://phabricator.wikimedia.org/rTGRTb34fc5c35c133cf1298ef2c5b9f54c22e86fd206 [13:12:36] i was trying to fix that ^^ without needing a revert. [13:14:14] I still don't see the pressing need that pushed your self-review [13:39:09] paladox: Just so I'm clear, you recognised the problem of self-merging, but overrode your better judgement because it was "spamming"? Did Freenode staff K-line the bot? [14:01:40] paladox: line 119 looks like an noop? [14:07:30] 06Labs, 06Discovery, 13Patch-For-Review: Upgrade postgres on labsdb1004 / 1005 to 9.4, and PostGis 2.1 - https://phabricator.wikimedia.org/T101233#2872512 (10akosiaris) 05Open>03Resolved labsdb1005 is not longer a postgres slave. I 've removed both comments to fix that and no longer give that impression... [14:31:54] 06Labs, 10Tool-Labs, 13Patch-For-Review: Offer Korean Locales "ko_KR.euckr" and "ko_KR.utf8" on Tool Labs - https://phabricator.wikimedia.org/T130532#2872563 (10Dzahn) I could generate it and then it appeared in the list of locales afterwards: Doesn't that solve it? ``` root@tools-exec-1401:~# locale-gen k... [14:40:46] 06Labs, 10Labs-Infrastructure, 06Operations, 06Reading-Web-Backlog, and 3 others: https://wikitech.m.wikimedia.org/ serves wikimedia.org portal - https://phabricator.wikimedia.org/T120527#2872577 (10Dzahn) [15:02:30] (03PS3) 10Zppix: Adding ##zppix-wikipedia [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/327121 [15:05:48] I see that tools ssh no longer asks for password if no pub key is provided [15:07:39] paladox: Since you're in the habit of self-merging to fix trivial bugs, could you please fix the stupid empty [] when no scores are applied to a comment? [15:08:21] paladox: While you're at it remove the no-op that zhuyifei1999_ pointed out and that would have been caught by any sane reviewer [15:08:43] Obviously both of these are incredibly high priority [15:10:59] marktraceur: oh lord. im gone 12 hours and everything has gone wrong i see' [15:11:26] Zppix: You're emphatically not to blame for this. [15:11:39] i know [15:11:46] link to changes ill fix it right now [15:12:41] Zppix: What changes? [15:13:00] https://thepasteb.in/p/P1hv8oXQEvRil [15:13:24] Zppix: Those are...the changes I requested, yes [15:13:31] They have no valid URL [15:13:35] Except the one you just created [15:15:00] ok [15:15:17] do you know what changes this started with [15:16:19] Zppix: You mean the self-merged patch that caused the problem? Yeah, https://gerrit.wikimedia.org/r/327152 [15:17:42] I caught the no-op if statement while blearily looking at my email from bed, so hopefully you or paladox can find it and fix it [15:18:37] i may just end up reverting the whole change [15:19:01] Zppix: Given that the patch actually fixed something, we are stuck in the MORONIC situation where that is not an option [15:22:48] Zppix: yah, the password never worked anyway [15:23:00] it did actually ... yuvipanda [15:23:17] Zppix: huh? what password did you use? [15:23:23] shell [15:23:38] this was long ago though [15:23:42] what do you mean by 'shell'? [15:24:13] my password from wikitech' [15:25:17] Zppix: when was the last time that worked? I don't see how that could've happened... [15:26:09] about 2 months ago [15:40:30] On https://grafana-labs.wikimedia.org/dashboard/db/labs-project-board?var-project=tools&var-server=All&from=now-15m&to=now , most graphs error out for me after loading for a while. Same for everyone? [15:40:47] https://grafana-labs.wikimedia.org/dashboard/db/labs-project-board?var-project=tools (which is the default link form wikitech) is more or less the same here. [16:01:00] !ping [16:01:01] !pong [16:28:34] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Initial setup and provision of labsdb1009, labsdb1010 and labsdb1011 - https://phabricator.wikimedia.org/T140452#2872866 (10chasemp) >>! In T140452#2872812, @Stashbot wrote: > {nav icon=file, name=Mentioned in SAL (#wikimedia-operations), href=https... [16:33:27] 06Labs, 10Tool-Labs, 13Patch-For-Review: Add yanker redirect - https://phabricator.wikimedia.org/T136924#2872902 (10scfc) 05Open>03Resolved [16:40:52] 06Labs, 10Tool-Labs, 13Patch-For-Review: Offer Korean Locales "ko_KR.euckr" and "ko_KR.utf8" on Tool Labs - https://phabricator.wikimedia.org/T130532#2872927 (10Ykhwong) I've just opened a new session for testing, but I don't see the euckr when running locale -a. Have you tried "dpkg-reconfigure locales" tha... [16:47:19] 06Labs, 10Labs-Infrastructure, 10DBA, 13Patch-For-Review: Initial setup and provision of labsdb1009, labsdb1010 and labsdb1011 - https://phabricator.wikimedia.org/T140452#2872950 (10jcrespo) Just one minor comment- you want to comment this on the open T141097 or T147051 or T147052. This is an old ticket ju... [16:50:53] ERROR 1356 (HY000): View 'enwiki_p.aft_article_answer' references invalid table(s) or column(s) or function(s) or definer/invoker of view lack rights to use them [16:51:15] ^^^ Looks like the aft_article_answer view is broken and that is breaking SQL Alchemy for me :( [16:51:25] Is this a known issue? [16:57:04] halfak: I don't know about this, it could be old stale views? [16:57:46] Indeed. I'll file a task about it and see if I can find any others. [17:00:11] (03PS1) 10Andrew Bogott: Add labs.yaml to labs/private [labs/private] - 10https://gerrit.wikimedia.org/r/327231 (https://phabricator.wikimedia.org/T150092) [17:00:42] 06Labs, 10DBA: Fix broken views in labs DB "ERROR 1356 -- references invalid table(s) or column(s)" - https://phabricator.wikimedia.org/T153213#2872990 (10Halfak) [17:11:36] 10Tool-Labs-tools-Other: Coordinate links should remember user preference for the mapping service and not ask to choose every time - https://phabricator.wikimedia.org/T153211#2873032 (10MarcoAurelio) As `geohack` don't seem to have a dedicated project. CCing @Dispenser, @Magnus and @Kolossos as tool mantainers,... [17:16:44] 06Labs, 10DBA: Fix broken views in labs DB "ERROR 1356 -- references invalid table(s) or column(s)" - https://phabricator.wikimedia.org/T153213#2873056 (10Halfak) Looks like `povwatch_log` has problems too [17:18:21] 10Tool-Labs-tools-Other: Coordinate links should remember user preference for the mapping service and not ask to choose every time - https://phabricator.wikimedia.org/T153211#2873073 (10Yurivict) For example, the link is in the top right corner of https://en.wikipedia.org/wiki/San_Francisco_Bay The fact that th... [17:34:39] 06Labs, 10DBA: Fix broken views in labs DB "ERROR 1356 -- references invalid table(s) or column(s)" - https://phabricator.wikimedia.org/T153213#2873113 (10Halfak) Here's a related issue I've filed in SQLAlchemy: https://bitbucket.org/zzzeek/sqlalchemy/issues/3871/metadatareflect-should-skip-tables-that [18:01:58] 10Tool-Labs-tools-Other: watroles is misleading and should be disabled - https://phabricator.wikimedia.org/T153216#2873173 (10Krenair) [18:05:06] marktraceur hi sorry for late reply, the [] is intentional for now otherwise it will fail, im looking into that. would you like me to revert my patches + revert https://phabricator.wikimedia.org/rTGRTb34fc5c35c133cf1298ef2c5b9f54c22e86fd206 ? so that i can have help in doing a proper fix since it currently works but still has [] which i need to find a way to remove. [18:06:01] also sorry for the self merges [18:06:32] also nope no k-line. [18:08:33] ah you want me to remove line 119? zhuyifei1999_ marktraceur [18:08:46] since i see now. [18:09:04] 06Labs, 10DBA: Fix broken views in labs DB "ERROR 1356 -- references invalid table(s) or column(s)" - https://phabricator.wikimedia.org/T153213#2872990 (10Krenair) The underlying table simply no longer exists: ```mysql:wikiadmin@db1089 [enwiki]> show create table povwatch_log; ERROR 1146 (42S02): Table 'enwiki... [18:11:18] (03Draft1) 10Paladox: Fix a no-op [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/327248 [18:11:21] (03Draft2) 10Paladox: Fix a no-op [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/327248 [18:11:49] zhuyifei1999_ marktraceur ^^ [18:12:16] 06Labs, 10DBA: Fix broken views in labs DB "ERROR 1356 -- references invalid table(s) or column(s)" - https://phabricator.wikimedia.org/T153213#2873207 (10Krenair) Related tickets: T103011, T92739, T54924 [18:14:37] 06Labs, 10Labs-Infrastructure, 10DBA: LabsDB replica service for tools and labs - issues and missing available views (tracking) - https://phabricator.wikimedia.org/T150767#2873214 (10Krenair) [18:14:40] 06Labs, 10DBA: Fix broken views in labs DB "ERROR 1356 -- references invalid table(s) or column(s)" - https://phabricator.wikimedia.org/T153213#2873213 (10Krenair) [18:24:02] (03CR) 10Zppix: [C: 032] Fix a no-op [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/327248 (owner: 10Paladox) [18:25:43] marktraceur: sorry for pinging you every 2 seconds, but... I've made it clear that the errors you mentioned are to be done before anything else gets merged. [18:26:40] (03Merged) 10jenkins-bot: Fix a no-op [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/327248 (owner: 10Paladox) [18:27:53] anything i can do to help out labs team noticing you guys are short handed [18:28:32] (03PS4) 10Zppix: Adding ##zppix-wikipedia [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/327121 [18:29:45] 10Tool-Labs-tools-Other, 06Release-Engineering-Team, 13Patch-For-Review: Jouncebot: Add functionality to change Nick from Jouncebot_ to Jouncebot automatically - https://phabricator.wikimedia.org/T150916#2873288 (10Zppix) 05Open>03Resolved [18:31:35] should i do a revert? [18:33:19] on? [18:38:13] 06Labs, 10Tool-Labs, 13Patch-For-Review: Offer Korean Locales "ko_KR.euckr" and "ko_KR.utf8" on Tool Labs - https://phabricator.wikimedia.org/T130532#2873342 (10Dzahn) @Ykhwong i am on tools-exec-1401 , maybe it's because you are on another exec node / we have to generate them on multiple nodes? [18:38:48] needs a list of all exec nodes [18:38:58] i guess [18:42:45] !log toollabs tools-bastion-03 - locale-gen ko_KR.EUC-KR [18:42:45] Unknown project "toollabs" [18:43:02] !log tools tools-bastion-03 - ran 'locale-gen ko_KR.EUC-KR' for T130532 [18:43:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:43:06] T130532: Offer Korean Locales "ko_KR.euckr" and "ko_KR.utf8" on Tool Labs - https://phabricator.wikimedia.org/T130532 [18:45:28] Zppix https://phabricator.wikimedia.org/rTGRTb34fc5c35c133cf1298ef2c5b9f54c22e86fd206 [18:46:00] 06Labs, 10Tool-Labs, 13Patch-For-Review: Offer Korean Locales "ko_KR.euckr" and "ko_KR.utf8" on Tool Labs - https://phabricator.wikimedia.org/T130532#2873382 (10Dzahn) @Ykhwong I think part of the confusion was i only looked at -exec nodes (because the patch we merged above installs them there), while you te... [18:49:50] paladox: Why would the bot break if you removed the [] part? [18:50:41] marktraceur since thats what https://phabricator.wikimedia.org/rTGRTb34fc5c35c133cf1298ef2c5b9f54c22e86fd206 did causing https://phabricator.wikimedia.org/T152932 [18:51:36] paladox: I don't think that's even slightly an answer to my question, but let me look at it [18:51:48] this https://bugs.chromium.org/p/gerrit/issues/detail?id=3220 caused the stream-event to always show vote info. [18:52:03] https://gerrit-review.googlesource.com/#/c/71051 [18:52:42] paladox: OK, let me ask you this [18:52:58] ok [18:53:01] paladox: You have a string that I give you, let's say, "CR [] 123 blah blah blah" [18:53:09] yep [18:53:15] paladox: Why would removing the first instance of '[]' from that string break anything? [18:53:51] it wouldent but implementing it might as i am looking at removing [] but carn't find a way to do it. [18:54:04] Wouldent huh [18:54:16] would not. [18:54:22] paladox: OK, explain to me what method exactly you are using to remove the []. [18:54:53] ret.approvals = undefined; [18:57:02] paladox: And how does that break exactly? [18:57:47] marktraceur it dosen't but with https://phabricator.wikimedia.org/rTGRTb34fc5c35c133cf1298ef2c5b9f54c22e86fd206 it did because of the use of Object.key. [18:58:07] paladox: OK, that sounds like a fixable problem. Can you fix that problem? [18:58:26] i am looking into it but i doint know [18:58:57] paladox: Could you perhaps check if ret.approvals exists before running Object.keys on it? [18:59:58] oh maybe but Object.keys was causing patches that were merged (what i mean is i would write a comment on a merged change but it would not send the notification to irc whereas an open change did). [19:00:58] paladox: OK, that sounds like a totally different problem that may have more to do with other parts of the code. I can take a look at it, though. [19:01:26] oh thanks, also again sorry for the self merges. [19:01:36] paladox: Don't be sorry. Don't do it again. [19:01:42] ok [19:02:00] paladox: You say OK, but I'm pretty sure you said OK last time, too. [19:03:01] paladox: So where is the patch that fixes the [] problem? [19:03:14] yes but if you look at my history in that repo, most (i think not 100% sure was c+2 by someone else). I did it this once and yesturday to fix something that well i found annoying. [19:03:22] I doint have a patch that fixes [] [19:04:25] paladox: I thought you were working on it? [19:04:38] Yes, i am trying the code changes on the test bot [19:04:43] paladox: Being annoyed is not a good reason to self-merge. [19:04:58] For example I am annoyed by UploadWizard's mascot. [19:05:07] Oh sorry. [19:05:10] I don't write a patch removing the tutorial and self-merge it. [19:05:14] wont happen again. [19:05:21] Because that would significantly harm the value of the extension for other people. [19:05:30] yep. [19:05:48] And even if there was a miniscule chance that others would disagree with me, I would want them to have the opportunity to tell me I'm being stupid. [19:05:53] Because sometimes, I am being stupid. [19:05:59] And, being stupid, I don't realize it. [19:06:11] oh [19:07:12] 06Labs, 10Wikimedia-Extension-setup, 10wikitech.wikimedia.org, 07Mobile: Install MobileFrontend on wikitech - https://phabricator.wikimedia.org/T87633#2873423 (10Dzahn) for completeness: we deleted wikitech.m from DNS today. [radon:~] $ host wikitech.m.wikimedia.org Host wikitech.m.wikimedia.org not found... [19:08:03] 06Labs, 10Labs-Infrastructure, 06Operations, 06Reading-Web-Backlog, and 3 others: https://wikitech.m.wikimedia.org/ serves wikimedia.org portal - https://phabricator.wikimedia.org/T120527#2873427 (10Dzahn) [radon:~] $ host wikitech.m.wikimedia.org Host wikitech.m.wikimedia.org not found: 3(NXDOMAIN) This... [19:08:37] 06Labs, 10Labs-Infrastructure, 06Operations, 06Reading-Web-Backlog, and 3 others: https://wikitech.m.wikimedia.org/ serves wikimedia.org portal - https://phabricator.wikimedia.org/T120527#2873428 (10Dzahn) 05Open>03Resolved a:03Dzahn [19:10:16] oldValue [19:11:01] Woops, thats from https://gerrit-review.googlesource.com/#/c/71051/24/Documentation/config-hooks.txt [19:17:16] (03CR) 10Andrew Bogott: [V: 032 C: 032] Add labs.yaml to labs/private [labs/private] - 10https://gerrit.wikimedia.org/r/327231 (https://phabricator.wikimedia.org/T150092) (owner: 10Andrew Bogott) [19:34:16] 06Labs, 10Labs-Infrastructure, 06Operations: Silver anomalies - https://phabricator.wikimedia.org/T151486#2873604 (10Dzahn) [19:35:01] 06Labs, 10Labs-Infrastructure, 06Operations: silver: /dev/md2 mounted twice - https://phabricator.wikimedia.org/T151489#2873610 (10Dzahn) [20:50:24] 06Labs, 06Release-Engineering-Team: Request for CI staging project - https://phabricator.wikimedia.org/T150772#2873967 (10hashar) [21:04:05] PROBLEM - Puppet run on tools-puppetmaster-02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [21:05:13] PROBLEM - Puppet run on tools-webgrid-lighttpd-1206 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [21:06:15] PROBLEM - Puppet run on tools-precise-dev is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [21:06:41] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [21:07:01] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [21:07:31] PROBLEM - Puppet run on tools-static-11 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [21:07:57] PROBLEM - Puppet run on tools-flannel-etcd-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [21:08:56] PROBLEM - Puppet run on tools-prometheus-02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [21:09:08] PROBLEM - Puppet run on tools-exec-1407 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [21:09:30] PROBLEM - Puppet run on tools-worker-1007 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [21:09:32] PROBLEM - Puppet run on tools-webgrid-lighttpd-1403 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [21:09:34] PROBLEM - Puppet run on tools-exec-1414 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [21:09:40] PROBLEM - Puppet run on tools-exec-1417 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [21:09:41] PROBLEM - Puppet run on tools-mail-01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [21:09:44] PROBLEM - Puppet run on tools-exec-1411 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [21:09:45] PROBLEM - Puppet run on tools-proxy-02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [21:10:20] PROBLEM - Puppet run on tools-prometheus-01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [21:10:26] PROBLEM - Puppet run on tools-checker-02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [21:10:41] PROBLEM - Puppet run on tools-exec-1406 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [21:10:42] ^ those alerts are a known problem being worked on [21:10:53] PROBLEM - Puppet run on tools-exec-1214 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [21:12:07] PROBLEM - Puppet run on tools-worker-1023 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [21:12:15] PROBLEM - Puppet run on tools-redis-1002 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [21:12:53] PROBLEM - Puppet run on tools-worker-1013 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [21:13:19] PROBLEM - Puppet run on tools-worker-1001 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [21:13:21] PROBLEM - Puppet run on tools-exec-1410 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [21:13:27] PROBLEM - Puppet run on tools-exec-1409 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [21:13:55] PROBLEM - Puppet run on tools-worker-1002 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [21:14:11] PROBLEM - Puppet run on tools-webgrid-lighttpd-1412 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [21:15:44] PROBLEM - Puppet run on tools-bastion-02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [21:17:55] You can now find out the "revert status" of an edit using a single function call against a database connection. See https://pythonhosted.org/mwreverts/ [21:18:02] Also added functionality for finding reverts in deleted revisions (revisions to deleted pages) for those with the API rights or access to the tool labs DB. [21:18:19] what kinds of reverts can that detect? [21:18:23] I'm using this to do an analysis of the activities of newcomers to English Wikipedia. [21:18:30] Just identity reverts -- with matching sha1s [21:18:59] You give it a rev_id and it tells you if the edit is "reverting", "reverted", or was "reverted_to" along with details about how those reverts played out. [21:19:45] The best way to see how this works is to check out the demos in the repo [21:19:46] https://github.com/mediawiki-utilities/python-mwreverts/blob/master/demo_api_check.py [21:19:50] https://github.com/mediawiki-utilities/python-mwreverts/blob/master/demo_db_check.py [21:20:46] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Wurgl was created, changed by Wurgl link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Wurgl edit summary: Created page with "{{Tools Access Request |Justification=There are a few tools from the User APPER, but that user does not have time. So give me a chance to take a look at the tools to allow..." [21:21:43] halfak, so wait [21:21:51] * halfak waits [21:22:20] halfak, this can detect a single revision being undone by a single revision immediately following it? [21:22:42] Within a "radius" and bound within a time "window" [21:26:43] Krenair, we use this utility to flag reverts for building machine learning models. Or for analyzing the activities of newcomers. [21:27:03] When I say "Identity revert", I'm drawing from language here: https://meta.wikimedia.org/wiki/Research:Revert [21:30:47] halfak, interesting [21:30:59] halfak, I vaguely recall someone asking for something like this recently [21:40:03] halfak: https://phabricator.wikimedia.org/T152434 [21:43:54] RECOVERY - Puppet run on tools-prometheus-02 is OK: OK: Less than 1.00% above the threshold [0.0] [21:44:04] RECOVERY - Puppet run on tools-puppetmaster-02 is OK: OK: Less than 1.00% above the threshold [0.0] [21:44:32] RECOVERY - Puppet run on tools-worker-1007 is OK: OK: Less than 1.00% above the threshold [0.0] [21:44:36] RECOVERY - Puppet run on tools-exec-1414 is OK: OK: Less than 1.00% above the threshold [0.0] [21:44:42] RECOVERY - Puppet run on tools-exec-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [21:44:42] RECOVERY - Puppet run on tools-proxy-02 is OK: OK: Less than 1.00% above the threshold [0.0] [21:45:14] RECOVERY - Puppet run on tools-webgrid-lighttpd-1206 is OK: OK: Less than 1.00% above the threshold [0.0] [21:46:14] RECOVERY - Puppet run on tools-precise-dev is OK: OK: Less than 1.00% above the threshold [0.0] [21:47:09] RECOVERY - Puppet run on tools-worker-1023 is OK: OK: Less than 1.00% above the threshold [0.0] [21:47:17] RECOVERY - Puppet run on tools-redis-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [21:47:29] RECOVERY - Puppet run on tools-static-11 is OK: OK: Less than 1.00% above the threshold [0.0] [21:47:49] RECOVERY - Puppet run on tools-worker-1013 is OK: OK: Less than 1.00% above the threshold [0.0] [21:47:57] RECOVERY - Puppet run on tools-flannel-etcd-01 is OK: OK: Less than 1.00% above the threshold [0.0] [21:48:17] RECOVERY - Puppet run on tools-worker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [21:48:53] RECOVERY - Puppet run on tools-worker-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [21:49:07] RECOVERY - Puppet run on tools-exec-1407 is OK: OK: Less than 1.00% above the threshold [0.0] [21:49:33] RECOVERY - Puppet run on tools-webgrid-lighttpd-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [21:49:39] RECOVERY - Puppet run on tools-exec-1417 is OK: OK: Less than 1.00% above the threshold [0.0] [21:49:40] RECOVERY - Puppet run on tools-mail-01 is OK: OK: Less than 1.00% above the threshold [0.0] [21:50:20] RECOVERY - Puppet run on tools-prometheus-01 is OK: OK: Less than 1.00% above the threshold [0.0] [21:50:26] RECOVERY - Puppet run on tools-checker-02 is OK: OK: Less than 1.00% above the threshold [0.0] [21:50:38] RECOVERY - Puppet run on tools-exec-1406 is OK: OK: Less than 1.00% above the threshold [0.0] [21:50:52] RECOVERY - Puppet run on tools-exec-1214 is OK: OK: Less than 1.00% above the threshold [0.0] [21:53:20] RECOVERY - Puppet run on tools-exec-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [21:53:28] RECOVERY - Puppet run on tools-exec-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [21:54:10] RECOVERY - Puppet run on tools-webgrid-lighttpd-1412 is OK: OK: Less than 1.00% above the threshold [0.0] [21:55:45] RECOVERY - Puppet run on tools-bastion-02 is OK: OK: Less than 1.00% above the threshold [0.0] [22:01:31] Krenair, thanks for linking. This has existed for a long time. [22:01:38] I've just added new functionality. [22:01:41] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:02:01] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [22:44:04] PROBLEM - Puppet staleness on tools-worker-1017 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [43200.0] [23:06:32] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 06Developer-Relations, and 4 others: Set up process / criteria for taking over abandoned tools - https://phabricator.wikimedia.org/T87730#2874758 (10bd808) The [[https://meta.wikimedia.org/w/index.php?title=Requests_for_comment/Abandoned_Labs_tools&oldid=161... [23:18:36] If any of you folks have ideas about Tool owner who we be interested in being part of the committee to deal with abandoned tools -- https://wikitech.wikimedia.org/wiki/Help_talk:Tool_Labs/Tool_Labs_standards_committee#December_2016_committee_nominations [23:34:32] 06Labs, 07Tracking: Existing Labs project quota increase requests (Tracking) - https://phabricator.wikimedia.org/T140904#2874853 (10Aklapper) [23:34:34] 06Labs: Increase resource quota for dwl - https://phabricator.wikimedia.org/T152456#2874852 (10Aklapper) [23:43:53] (03PS5) 10Paladox: Replace git.wikimedia.org links with diffusion [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/305293 (https://phabricator.wikimedia.org/T139089) [23:45:36] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Wurgl was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=1150908 edit summary: [23:48:25] yuvipanda: Are procedures not allowed in quarry? [23:48:46] I'm trying to figure out a way to query multiple wikis at once without copy-pasting it multiple times and using union all [23:48:54] but instead writing a procedure. [23:48:59] Just an idea :) [23:49:06] It seems it doesn't like the syntax