[00:01:23] 06Labs, 10Tool-Labs, 06Developer-Relations: Make sure abandoned useful tools are properly advertised so potentially interested new maintainers could find them - https://phabricator.wikimedia.org/T159595#3086206 (10Quiddity) 1 more example: * Guillom is crazy busy right now, and doesn't have time to maintain... [00:18:41] PROBLEM - Puppet run on tools-worker-1029 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [00:21:48] PROBLEM - Puppet run on tools-worker-1028 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [00:25:49] https://quarry.wmflabs.org/query/runs/all gives an 502 [01:00:49] madhuvishy: do you know how the quarry bits work? The 'recent queries' page is throwing a 502. I'm not sure if that means that redis is busted or something else [01:08:50] bd808: I'm at gym right now, I can look in a little bit [01:09:27] is yuvipanda around? I suppose he knows how it works too ;) [01:10:55] He's in a Bart with a bike :) [01:11:28] I'm on bart with a bike [01:11:44] I hope it isn't MySQL corruption because I'll be so so sad [01:12:06] i'm on laptop [01:12:08] looking [01:12:27] quarry-main-01 or something like that. [01:12:35] Uwsgi logs should tell you what's wrong [01:15:54] https://www.irccloud.com/pastebin/PLob3ugF/ [01:16:16] Disk full /tmp [01:16:52] /dev/vda1 ext4 7.4G 7.2G 0 100% / [01:17:17] yuvipanda: ^ [01:17:24] Ah thank God [01:17:33] Clean out some logs maybe? [01:20:27] yuvipanda: can i truncate killer.log [01:21:36] Yes [01:30:15] yuvipanda: oh man i have been cleaning up logs [01:30:21] and /var is on a different volume [01:30:25] is full [01:30:29] / is full [01:30:49] i'm like aaah why won't it go down from 100% [01:30:55] madhuvishy :( I'll move it to tools soon hopefully [01:31:05] yuvipanda: not sure what to kill [01:31:12] from / [01:31:27] lib is 5.1G [01:31:35] of 7.4G [01:31:44] Might be MySQL filling it up [01:32:17] Well, it might just have to be down until I get to it tonight [01:32:27] I can move MySQL to labsdb [01:32:46] I've wanted to do that anyway [01:34:03] yuvipanda: okay then [01:34:42] I'll go back to working out [01:39:29] Madhuvishy thanks for looking [03:22:59] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/BinaryDude was created, changed by BinaryDude link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/BinaryDude edit summary: Created page with "{{Tools Access Request |Justification=Writing small tools for auto-generating lists such as 'list of wikipedians by number of edits' for siwiki. |Completed=false |User Name=Bi..." [04:36:25] (03CR) 10Legoktm: [C: 032] Add RecentChanges and Watchlist to #wikimedia-collaboration [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/340806 (owner: 10Mattflaschen) [04:37:59] (03Merged) 10jenkins-bot: Add RecentChanges and Watchlist to #wikimedia-collaboration [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/340806 (owner: 10Mattflaschen) [04:38:09] (03CR) 10jenkins-bot: Add RecentChanges and Watchlist to #wikimedia-collaboration [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/340806 (owner: 10Mattflaschen) [05:02:26] !log tools.wikibugs Updated channels.yaml to: 4015b6415b4aaf2bfd13e6ce25bbd113624d1379 Add RecentChanges and Watchlist to #wikimedia-collaboration [05:31:47] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/BinaryDude was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=1667762 edit summary: [06:21:56] PROBLEM - Puppet run on tools-webgrid-generic-1402 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [06:39:51] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, 13Patch-For-Review: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3086795 (10Marostegui) It has actually done some recovering as the file it is scanning now has changed since last night: ``` postgres 7189 0.0 0.... [06:53:18] (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/341974 (owner: 10L10n-bot) [06:56:54] RECOVERY - Puppet run on tools-webgrid-generic-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [08:34:57] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, 13Patch-For-Review: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3086995 (10Marostegui) Not sure from which time this is: ``` FATAL: the database system is starting up FATAL: terminating walreceiver process due... [08:45:18] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations, 13Patch-For-Review: labsdb1006/1007 (postgresql) maintenance - https://phabricator.wikimedia.org/T157359#3086996 (10jcrespo) That is me killing the replication, which will not work anyway. @akosiaris can you point us to the osm load process, do you have... [08:52:31] 06Labs, 10Tool-Labs, 10Tools-Kubernetes: Reassign service/pod IP ranges for kubernetes on tool labs - https://phabricator.wikimedia.org/T152399#3087017 (10yuvipanda) My understanding of the service controller code is that it won't validate against the range in its control loop - so our test should *fail*. Bu... [10:59:10] 06Labs, 06Operations: Remove linux kernel 3.16 from the jessie image on labs - https://phabricator.wikimedia.org/T159990#3087240 (10MoritzMuehlenhoff) @Paladox : That's entirely unrelated, the Launchpad entry refers to a bug in Upstart, which jessie doesn't use at all. @yuvipanda : I don't think there's a gene... [11:16:36] hi andrewbogott, bd808 let me know when you are around :P [11:16:46] This user is now online in #wikimedia-operations. I'll let you know when they show some activity (talk, etc.) [11:16:46] @notify andrewbogott [11:16:50] This user is now online in #wikimedia-dev. I'll let you know when they show some activity (talk, etc.) [11:16:50] @notify bd808 [13:37:14] (03PS3) 10Zppix: Fix typo preventing phab notifications to ##Zppix-Wikipedia [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/341928 (https://phabricator.wikimedia.org/T159612) [13:58:39] 10Tool-Labs-tools-Zppixbot, 10Wikibugs, 10QuarryBot-enwiki, 13Patch-For-Review: Wikibugs doesnt send phab updates to ##Zppix-Wikipedia - https://phabricator.wikimedia.org/T159612#3087671 (10Zppix) [13:58:54] 10Tool-Labs-tools-Zppixbot, 10Wikibugs, 10QuarryBot-enwiki, 13Patch-For-Review: Wikibugs doesnt send phab updates to ##Zppix-Wikipedia - https://phabricator.wikimedia.org/T159612#3073166 (10Zppix) https://gerrit.wikimedia.org/r/#/q/Idb50c5ea2efa7bb8eaf662ccea52ac20caf47df6 [14:07:08] 06Labs, 10Tool-Labs, 10Tools-Kubernetes: Reassign service/pod IP ranges for kubernetes on tool labs - https://phabricator.wikimedia.org/T152399#3087708 (10chasemp) root@k8s-master-01:~# kubectl get svc --all-namespaces NAMESPACE NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE default kuber... [14:11:04] 10Tool-Labs-tools-Zppixbot, 10Wikibugs, 10QuarryBot-enwiki, 13Patch-For-Review: Wikibugs doesnt send phab updates to ##Zppix-Wikipedia - https://phabricator.wikimedia.org/T159612#3087710 (10Aklapper) @Zppix: How is your link related to this task? [14:11:59] 10Tool-Labs-tools-Zppixbot, 10Wikibugs, 10QuarryBot-enwiki, 13Patch-For-Review: Wikibugs doesnt send phab updates to ##Zppix-Wikipedia - https://phabricator.wikimedia.org/T159612#3087711 (10Zppix) @Aklapper It isnt let me try sending the correct link... [14:12:30] 10Tool-Labs-tools-Zppixbot, 10Wikibugs, 10QuarryBot-enwiki, 13Patch-For-Review: Wikibugs doesnt send phab updates to ##Zppix-Wikipedia - https://phabricator.wikimedia.org/T159612#3087726 (10Zppix) https://gerrit.wikimedia.org/r/341928 heres the correct one [14:15:44] 06Labs, 10Tool-Labs, 10Tools-Kubernetes: Reassign service/pod IP ranges for kubernetes on tool labs - https://phabricator.wikimedia.org/T152399#3087730 (10chasemp) chasetestproject.testuserone@k8s-bastion:~$ kubectl get pods NAME READY STATUS RESTARTS AGE chase2-1667214045-6ubsp... [14:41:39] 06Labs, 10Tool-Labs, 10Tools-Kubernetes: Reassign service/pod IP ranges for kubernetes on tool labs - https://phabricator.wikimedia.org/T152399#3087811 (10chasemp) [14:45:56] (03CR) 10Samtar: [C: 031] Fix typo preventing phab notifications to ##Zppix-Wikipedia [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/341928 (https://phabricator.wikimedia.org/T159612) (owner: 10Zppix) [14:46:32] Zppix: annoying that wasn't caught first time 'round eh? ^ [14:58:57] selam [16:02:00] yuvipanda: in regards to the grid job scheduler, what settings have the grid automatically respawn a script that terminated with a non-zero exit code, and what settings has the grid ignore it? [16:13:29] Cyberpower678: I know jstart (continuous jobs) does that [16:13:47] zhuyifei1999_: What about the -once flag? [16:13:48] Cyberpower678: `jsub -continuous` will try to restart on a non-zero exit [16:13:58] bd808: ^^ [16:14:08] -once just keeps 2 jobs wiht the same name from getting scheduled [16:14:18] once is making sure you don't have two jobs with the same name [16:14:43] Is there a flag I can set where the automatic respawning is disabled? [16:15:04] don't use -continuious [16:15:11] that's literally all it does [16:15:33] So just omit the flag and it won't auto-respawn? [16:16:12] Yes. and use jsub not jstart. jstart is shorthand for jsub -once -continuous [16:16:26] I always use jsub. :p [16:16:57] this is what -continuous does to try and restart on failure -- https://github.com/wikimedia/labs-toollabs/blob/master/jobutils/bin/jsub#L647-L661 [16:17:20] it traps the exit code and loops on failure [16:19:56] petan: what's up? [16:30:09] 06Labs, 10wikitech.wikimedia.org, 15User-bd808: Fix or delete user accounts showing on wikitech but not found in LDAP after T149109 - https://phabricator.wikimedia.org/T159986#3088212 (10bd808) @Tgr and @Physikerwelt: These two accounts are completely broken for authentication. They have rows in the MediaWik... [16:36:34] (03PS1) 10Lokal Profil: Prepare monument_tables for wikidata and add tests [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/342038 [16:36:40] (03CR) 10jerkins-bot: [V: 04-1] Prepare monument_tables for wikidata and add tests [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/342038 (owner: 10Lokal Profil) [16:40:35] (03PS2) 10Lokal Profil: Prepare monument_tables for wikidata and add tests [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/342038 [16:43:37] (03CR) 10Lokal Profil: "Note my question on line 74. I cannot understand why this was done. If we stop doing it then extra_cols can be handled entirely in process" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/342038 (owner: 10Lokal Profil) [16:44:58] (03CR) 10Lokal Profil: "For testing I simply regenerated the sql files and did a meld of the old sql directory and the new. Other than " 0" becoming " 0" there a" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/342038 (owner: 10Lokal Profil) [17:02:34] petan: still there? [17:07:32] 06Labs, 10wikitech.wikimedia.org, 15User-bd808: Fix or delete user accounts showing on wikitech but not found in LDAP after T149109 - https://phabricator.wikimedia.org/T159986#3088312 (10Tgr) `nick` I think? Probably best to just ask. [17:09:49] 06Labs, 10Tool-Labs, 10Tools-Kubernetes: Reassign service/pod IP ranges for kubernetes on tool labs - https://phabricator.wikimedia.org/T152399#3088329 (10chasemp) **testing to make sure that a new range is at least honored for new deployments regardless of context for held-over services** > KUBE_SERVICE_AD... [17:16:54] Hey folks. I'm trying to load a dataset into a DB on one of the replica servers. it's a pretty small one, so I don't think it'll be an issue. My goal is to have the dataset queryable in quarry. It seems that the docs say I must "become mytool" which I think is news to me. Is it true that I can't just create a database on a replica server with my tool user account? [17:18:14] 06Labs, 10Tool-Labs, 10Tools-Kubernetes: Reassign service/pod IP ranges for kubernetes on tool labs - https://phabricator.wikimedia.org/T152399#3088373 (10chasemp) [17:18:34] I suppose that chasemp is the right person to ping for ^ [17:20:26] Oh! Nevermind, it does work from my user account. I'd just missed a required underscore :) [17:23:28] halfak: I think right now that new Tools users aren't getting db creds, but yeah prior grants should still work [17:23:38] Gotcha. That makes sense. [17:23:50] Maybe I should make a tool for "datasets" [17:23:54] Would that be crazy? [17:23:57] and the lack of new creds is a bit of a bug [17:24:04] maybe "wikiscience" or something [17:24:12] nope. tool accounts are the awesome [17:24:23] makes handing things off to a larger group much easier [17:24:31] Can you link me to some docs on the process for requesting that? [17:24:44] halfak: requesting tools? Self-service! [17:24:57] https://wikitech.wikimedia.org/w/index.php?title=Special:NovaServiceGroup&action=addservicegroup&projectname=tools [17:25:10] there is a link from https://tools.wmflabs.org/ [17:25:21] just type a "tool name" into the "Service group name" box thignie? [17:25:27] yup [17:25:31] the UX is horrible! [17:25:37] TOo easy. Do not believe O.o [17:25:47] bd808 will be making everything better! ;-) [17:26:30] valhallasw`cloud: as soon as I get done with annual planning for the next fiscal year. That's eaten by whole quarter so far :/ [17:26:31] bd808, I want to talk to you about some geopolitical bugs if you're taking on "everything" ;) [17:26:45] Seriously annual planning is rough. [17:26:56] halfak: write them up in phab and I'll see what I can do ;) [17:28:31] * halfak holds his political comments [17:28:34] :D [17:29:00] * bd808 hides his anarchist newsletters [17:29:40] * valhallasw`cloud puts everything on fire [17:43:11] 06Labs, 06Operations: Remove linux kernel 3.16 from the jessie image on labs - https://phabricator.wikimedia.org/T159990#3088459 (10Andrew) I just built four different jessie instances, ran 'apt get update && apt-get upgrade' on them and rebooted. All four came up, no problems. [17:47:29] 06Labs, 06Operations: Remove linux kernel 3.16 from the jessie image on labs - https://phabricator.wikimedia.org/T159990#3088472 (10Paladox) Oh, i wonder why mine failed. [17:52:55] PROBLEM - Puppet run on tools-webgrid-generic-1402 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [18:10:32] 06Labs, 10Tool-Labs, 10Tools-Kubernetes: Reassign service/pod IP ranges for kubernetes on tool labs - https://phabricator.wikimedia.org/T152399#3088548 (10yuvipanda) Just deleting the service IP doesn't seem to work in giving us a new one either. I get an error saying `# * spec.clusterIP: Invalid value: "":... [18:13:47] 06Labs, 10Tool-Labs, 10Tools-Kubernetes: Reassign service/pod IP ranges for kubernetes on tool labs - https://phabricator.wikimedia.org/T152399#3088554 (10yuvipanda) lololol, current range is 192.168.0.0/24 not /16 as I was thinking - the /24 is the reason we've to do the rejig at all lol! ok, brainfreeze...... [18:27:55] RECOVERY - Puppet run on tools-webgrid-generic-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [19:06:04] (03CR) 10Yuvipanda: [C: 031] "Finally!" [labs/toollabs] - 10https://gerrit.wikimedia.org/r/341666 (https://phabricator.wikimedia.org/T94792) (owner: 10BryanDavis) [19:26:03] hey yuvipanda! k8s-related puppet modules question(s) for ya if you have a sec [19:27:58] ^ or chasemp possibly [19:36:07] (03CR) 10Legoktm: [C: 032] Fix typo preventing phab notifications to ##Zppix-Wikipedia [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/341928 (https://phabricator.wikimedia.org/T159612) (owner: 10Zppix) [19:36:34] (03Merged) 10jenkins-bot: Fix typo preventing phab notifications to ##Zppix-Wikipedia [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/341928 (https://phabricator.wikimedia.org/T159612) (owner: 10Zppix) [19:36:56] (03CR) 10jenkins-bot: Fix typo preventing phab notifications to ##Zppix-Wikipedia [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/341928 (https://phabricator.wikimedia.org/T159612) (owner: 10Zppix) [19:38:47] !log tools.wikibugs Updated channels.yaml to: ece2d4e9ef1da1ab682971262812baac737e9293 Fix typo preventing phab notifications to ##Zppix-Wikipedia [19:58:00] I got this warning when I used become "groups: cannot find name for group ID 550". Any idea what that is? [20:11:15] marxarelli: ask away :) [20:11:57] marxarelli: become for what tool? [20:12:10] chasemp: rad. thanks. we're trying to get a barebones k8s cluster up and running for use in our ci experiments [20:12:21] multichill: become for what tool? [20:12:31] marxarelli: sure [20:12:47] i'm referencing the toollabs modules as a starting point [20:12:50] got etcd up and running [20:13:02] now i'm working on applying the docker registry stuff [20:13:58] marxarelli: I think chasemp just setup a new k8s cluster by himself from scratch. [20:14:09] yuvipanda: we skipped a registry tho [20:14:15] but I shoudl do that... [20:14:30] that's true. i sent my message before I saw that line [20:15:01] looks like an ssl cert is required [20:15:20] can i get away with a self-signed cert if it's all on a single host? [20:15:45] marxarelli: there's an option to just use the puppet cert i think [20:15:45] looking at docker::registry::web [20:16:35] yuvipanda: I think that option was only put in place for the master role? [20:16:37] hi, are we aware of any cases where the replicas contain out-of-date data? as in, for some reason the new data wasn't copied over, unrelated to replag? [20:16:37] marxarelli: look at profile::docker::registry [20:16:41] that's the config prod uses [20:16:50] halfak: ping :) [20:16:52] chasemp: nope, prod's registry also uses puppet certs [20:17:07] o/ TheresNoTime [20:17:07] musikanimal: that's a really broad question to which I would say maybe [20:17:09] What's up? [20:17:32] halfak: apparently you may be able to help with access to the detox API? I believe you're working on adding it to ORES? [20:17:33] okay, I guess I was mostly wanting to know if it was possible, because I think I've found such a case [20:17:35] yuvipanda: i see the expose_puppet_certs stuff in role::toollabs::k8s::master [20:17:43] but not for the docker registry [20:18:02] TheresNoTime, yeah. That's right. It's not prioritized right now, but we'll have it in ORES some time in the next couple of months. [20:18:03] maybe i'm just missing it [20:18:14] There's a public API that is available now. [20:18:15] but role::toollabs::docker::registry references the star.tools.wmflabs.org cert [20:18:17] marxarelli: look at profile::docker::registry, not inside the toollabs module :) [20:18:36] musikanimal: there are a bunch of scattered tickets around for data integrity on the labsdb cluster but without specifics I don't know [20:18:37] we're in confusing transition time! [20:18:46] TheresNoTime, https://tools.wmflabs.org/detox/ [20:18:54] yuvipanda: ah! i see it [20:18:56] yuvipanda: is role::toollabs::docker::registry used? [20:19:01] use_puppet_certs :) [20:19:03] w00t [20:19:27] halfak: Oh? That was the crux of the question really - I tried getting hold of Ellery but haven't had any luck.. is there any way to get at that programmatically? [20:19:54] chasemp: in tools. it's one of the prod / tools things that need unification. partially caused by the 'profile vs role' thing [20:20:12] chasemp: so prod uses the profile thing, tools uses the role thing [20:20:23] TheresNoTime, I just reverse engineered this: https://tools.wmflabs.org/detox/api?model=attack&input_type=rev_id&input=34567890 [20:20:24] yuvipanda: anything wrong with putting etcd, the docker registry, and the k8s master services all on a single host? because that's what i'm doing :) [20:20:29] Not sure if that's the best way to hit the API or not. [20:20:32] But it works :) [20:20:44] right, i was confused as he said he was using all teh tools stuff as a starting point but we said don't mimic tools in this one place [20:21:00] halfak: oh you clever soul, that's exactly what I need - thank you very much! [20:21:01] yuvipanda: is that somethign you are planning to untangle in the next few weeks? [20:21:04] * halfak read through the little js snippet that makes it go [20:21:05] chasemp: yup! [20:21:08] k [20:21:14] TheresNoTime, godspeed :) [20:21:28] chasemp: I've a ticket for it with TODOs. similar to the docker untangling from earlier [20:21:40] yuvipanda: where is taht out of curiosity? [20:21:53] am trying to find now [20:22:12] ok, https://phabricator.wikimedia.org/T158452. there's no TODO there tho [20:22:26] I'm planning on working on it next week, now that akosiaris is back [20:22:51] k [20:32:28] 06Labs, 10PAWS: Move PAWS nfs onto its own share - https://phabricator.wikimedia.org/T160113#3089093 (10yuvipanda) [20:35:54] 06Labs, 10PAWS: Track PAWS user storage - https://phabricator.wikimedia.org/T160114#3089110 (10yuvipanda) [20:54:13] musikanimal: yes, the replicas can have stale rows. The new DB servers that are in testing right now should fix that problem by changing the type of replication that is done from the masters. [20:56:21] Cool, thanks. I ran into one case the other day where a categorylink was missing. I just removed and re-added the category then it showed up [20:57:35] yeah. there are a variety of weird things that can cause the replicas to get out of sync today. The DBAs are close to having that fixed though :) [20:58:42] musikanimal: if you find a horrible case, file a phab task and tag it for the DBAs [20:58:56] but if it's something like a missing category, meh [20:59:15] will do. This one was no biggie, just confusing [21:03:40] 06Labs, 10Gerrit, 06Operations, 06Release-Engineering-Team, 07LDAP: Remove user gerrit2 from ldap - https://phabricator.wikimedia.org/T160122#3089287 (10Paladox) [21:03:43] 06Labs, 10Gerrit, 06Operations, 06Release-Engineering-Team, 07LDAP: Remove user gerrit2 from ldap - https://phabricator.wikimedia.org/T160122#3089299 (10Paladox) p:05Triage>03High [21:09:41] chasemp: Sorry, missed your reply. tools.multichill . My screen session is quite old (100+ days I guess) so it might just be some changed enviroment thing [21:17:32] 06Labs, 10Gerrit, 06Operations, 06Release-Engineering-Team, 07LDAP: Remove user gerrit2 from ldap - https://phabricator.wikimedia.org/T160122#3089347 (10Paladox) [23:11:28] Hi. I'm trying to run update.php on our labs instance of mwvagrant and it says "php: command not found". Same if I use mwscript instead of php. [23:25:37] Niharika: install php5-cli? [23:26:31] Platonides: Sorry, it works fine. I was being silly. Didn't ssh into the VM. [23:28:25] ! [23:28:26] hello :) [23:28:47] well, it's good when it all was a silly mistake:) [23:28:49] /13 [23:30:16] !log tools.validator Stopped and started webservice to clear weird state where grid thought both precise and trusty versions were running [23:30:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.validator/SAL [23:53:00] 10Tool-Labs-tools-Xtools, 06Community-Tech: [Epic] Rewrite XTools: Top edits - https://phabricator.wikimedia.org/T160137#3089933 (10kaldari) [23:56:54] 10Tool-Labs-tools-Xtools, 06Community-Tech: [Epic] Rewrite XTools: Top edits - https://phabricator.wikimedia.org/T160137#3089948 (10MusikAnimal) [23:57:17] 10Tool-Labs-tools-Xtools, 06Community-Tech: [Epic] Rewrite XTools: Top edits - https://phabricator.wikimedia.org/T160137#3089949 (10kaldari)