[06:31:53] 10Wikibugs: Wikibugs has problems with usernames - https://phabricator.wikimedia.org/T132271#2194215 (10Legoktm) [06:31:56] 10Wikibugs: Assigning a task to someone with a name beginning with a number causes IRC colour issues - https://phabricator.wikimedia.org/T111214#2194216 (10Legoktm) [10:43:53] 6Labs: labtestcontrol2001 cronspam - https://phabricator.wikimedia.org/T122931#2194464 (10elukey) [11:53:46] 6Labs, 10Tool-Labs, 6Commons, 10pywikibot-core, and 2 others: Pywikibot : Fix Commons scripts broken by toolserver.org to labs migration - https://phabricator.wikimedia.org/T78462#2194615 (10jayvdb) [12:08:13] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Gabriel.sofronie was created, changed by Gabriel.sofronie link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Gabriel.sofronie edit summary: Created page with "{{Tools Access Request |Justification=Contribute to Tool Labs for developing and maintaining Wiki projects. |Completed=false |User Name=Gabriel.sofronie }}" [13:52:16] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Gabriel.sofronie was modified, changed by Andrew Bogott link https://wikitech.wikimedia.org/w/index.php?diff=427937 edit summary: [14:02:54] RECOVERY - Puppet run on tools-web-static-01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:15:08] 6Labs, 10Labs-Infrastructure: I/O on labmon1001 is very slow - https://phabricator.wikimedia.org/T127957#2059611 (10fgiunchedi) I suspect I/O being pegged by graphite, sda/sdb report ~300 iops each. Not sure about the history of this machine but there seem to be two disks practically unused ``` labmon1001:~$... [14:20:26] 6Labs, 10Wikimedia-Stream: Provide useful diffs to high-volume consumers of RCStream - https://phabricator.wikimedia.org/T100082#2194897 (10chasemp) p:5Triage>3Normal [14:20:45] !log tools moving tools-bastion-mtemp to labvirt1009 [14:20:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, dummy [14:23:01] 6Labs, 10Tool-Labs: Puppet fails on all Precise execution nodes - https://phabricator.wikimedia.org/T132282#2194906 (10Joe) The mediawiki doesn't support precise anymore, as multiple parts of it are now HHVM and trusty and newer only. Why do we run precise in tools? do we really need mediawiki classes there? [14:23:49] (03CR) 10MarcoAurelio: "Will try again." [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/275190 (https://phabricator.wikimedia.org/T128503) (owner: 10MarcoAurelio) [14:23:57] (03Restored) 10MarcoAurelio: Continuous Integration Python config for labs/tools/stewardbots [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/275190 (https://phabricator.wikimedia.org/T128503) (owner: 10MarcoAurelio) [14:25:16] (03PS6) 10MarcoAurelio: Continuous Integration Python config for labs/tools/stewardbots [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/275190 (https://phabricator.wikimedia.org/T128503) [14:25:41] PROBLEM - Host tools-bastion-mtemp is DOWN: CRITICAL - Host Unreachable (10.68.19.117) [14:30:22] (03CR) 10MarcoAurelio: [C: 04-2] Continuous Integration Python config for labs/tools/stewardbots [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/275190 (https://phabricator.wikimedia.org/T128503) (owner: 10MarcoAurelio) [14:30:52] (03CR) 10MarcoAurelio: Continuous Integration Python config for labs/tools/stewardbots (035 comments) [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/275190 (https://phabricator.wikimedia.org/T128503) (owner: 10MarcoAurelio) [14:37:26] RECOVERY - Host tools-bastion-mtemp is UP: PING OK - Packet loss = 0%, RTA = 0.32 ms [14:39:37] 6Labs, 6Operations, 13Patch-For-Review, 15User-bd808: Setting up bulk proxies pointing to a multiwiki mediawiki-vagrant setup running on a labs vm - https://phabricator.wikimedia.org/T132216#2194966 (10faidon) [14:41:30] PROBLEM - Puppet run on tools-webgrid-generic-1405 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [14:43:22] PROBLEM - Puppet run on tools-webgrid-lighttpd-1412 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [14:44:34] PROBLEM - Puppet run on tools-k8s-master-01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [14:44:44] PROBLEM - Puppet run on tools-bastion-02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [14:46:58] Labs down? [14:47:01] https://tools.wmflabs.org/slumpartikel/ 502's [14:48:15] I think it might be just that tool [14:48:24] PROBLEM - Puppet run on tools-worker-1009 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [14:48:30] was working a second ago... [14:48:54] PROBLEM - Puppet run on tools-k8s-bastion-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [14:49:12] hmm..seems ok now...nvm [14:50:20] PROBLEM - Puppet run on tools-worker-1012 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [14:51:00] PROBLEM - Puppet run on tools-webgrid-lighttpd-1411 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [14:51:42] PROBLEM - Puppet run on tools-elastic-02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [14:57:57] PROBLEM - Puppet run on tools-worker-1004 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [14:58:38] PROBLEM - Puppet run on tools-webgrid-lighttpd-1413 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:00:56] ^ !!! [15:01:00] PROBLEM - Puppet run on tools-bastion-mtemp is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:01:39] labs dns is down I think [15:02:44] PROBLEM - Puppet run on tools-webgrid-generic-1404 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:03:21] kk [15:04:34] PROBLEM - Puppet run on tools-exec-1403 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:05:04] PROBLEM - Puppet run on tools-flannel-etcd-03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:05:16] PROBLEM - Puppet run on tools-docker-builder-03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:05:44] PROBLEM - Puppet run on tools-grid-shadow is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:07:07] PROBLEM - Puppet run on tools-exec-1401 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [15:07:39] PROBLEM - Puppet run on tools-flannel-etcd-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:09:10] labs DNS seems to be exploding [15:09:22] as halfak mentioned [15:09:32] Yeah. I got that from chasemp :) [15:09:35] PROBLEM - Puppet run on tools-proxy-02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:09:43] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [15:09:49] yeah [15:09:53] :D [15:10:23] PROBLEM - Puppet run on tools-mail-01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:12:57] PROBLEM - Puppet run on tools-exec-1410 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:17:04] PROBLEM - Puppet run on tools-elastic-01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:22:18] Hi! I'm hosting a PHP tool on labs. The errors from compilation are not getting logged to error.log. That file is perpetually blank. What could be the problem? [15:25:20] RECOVERY - Puppet run on tools-worker-1012 is OK: OK: Less than 1.00% above the threshold [0.0] [15:25:22] Niharika, how are you running the PHP file? [15:26:14] tom29739: The files are in public_html. I am just accessing the tool online. [15:26:26] Not sure of where the actual compile magic happens. [15:27:55] Niharika, it should appear after a few minutes, how long have you waited for it to appear? [15:28:33] tom29739: I've been observing this since a week now. The error file is always empty while there are definitely compile errors. [15:28:55] RECOVERY - Puppet run on tools-k8s-bastion-01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:29:12] Niharika, and the error.log file exists? [15:29:29] tom29739: Yes. [15:29:44] Try deleting the file and restarting the webservice. [15:30:00] When you restart it the error.log file should be created. [15:30:34] tom29739: Okay. [15:31:49] RECOVERY - Puppet run on tools-elastic-02 is OK: OK: Less than 1.00% above the threshold [0.0] [15:35:20] tom29739: That worked! Thank you so much. [15:36:03] RECOVERY - Puppet run on tools-webgrid-lighttpd-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [15:36:40] Niharika, no problem, it unhooks the file or something if you delete it and recreate it with the webservice running. [15:37:49] Ah. Okay. I didn't delete it but I switched from using Python to PHP. That must have caused it. [15:38:01] RECOVERY - Puppet run on tools-worker-1004 is OK: OK: Less than 1.00% above the threshold [0.0] [15:40:23] RECOVERY - Puppet run on tools-docker-builder-03 is OK: OK: Less than 1.00% above the threshold [0.0] [15:42:46] RECOVERY - Puppet run on tools-webgrid-generic-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [15:43:44] RECOVERY - Puppet run on tools-webgrid-lighttpd-1413 is OK: OK: Less than 1.00% above the threshold [0.0] [15:44:38] RECOVERY - Puppet run on tools-proxy-02 is OK: OK: Less than 1.00% above the threshold [0.0] [15:44:38] RECOVERY - Puppet run on tools-exec-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [15:44:38] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:47:12] RECOVERY - Puppet run on tools-exec-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [15:47:43] RECOVERY - Puppet run on tools-flannel-etcd-01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:48:36] (03CR) 10Jforrester: [C: 031] Add #wikimedia-ai channel [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/282364 (owner: 10Ladsgroup) [15:51:01] 6Labs, 10Labs-Infrastructure: I/O on labmon1001 is very slow - https://phabricator.wikimedia.org/T127957#2195233 (10yuvipanda) Ah, interesting. So I suppose that we can fix this by reconfiguring the disks to spread the load more evenly? [15:53:13] (03PS75) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 [15:53:25] RECOVERY - Puppet run on tools-webgrid-lighttpd-1412 is OK: OK: Less than 1.00% above the threshold [0.0] [15:54:39] RECOVERY - Puppet run on tools-bastion-02 is OK: OK: Less than 1.00% above the threshold [0.0] [15:55:35] 6Labs, 10Tool-Labs, 6Operations, 7Icinga, 13Patch-For-Review: tool labs instance distribution monitoring is broken - https://phabricator.wikimedia.org/T119929#2195282 (10Andrew) 5Open>3Resolved Test is fixed and passing. [15:56:40] (03PS1) 10Alex Monk: Try to fix display of colours appearing directly before numbers [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/282719 (https://phabricator.wikimedia.org/T111214) [15:57:03] RECOVERY - Puppet run on tools-elastic-01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:57:49] RECOVERY - Puppet run on tools-exec-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [15:59:38] RECOVERY - Puppet run on tools-k8s-master-01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:03:46] (03CR) 10Ricordisamoa: "PS75 moves the bulk of UI creation to the client side" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [16:05:31] (03PS76) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 [16:08:16] 6Labs, 10DBA, 13Patch-For-Review: Move labs pdns database off of m5-master - https://phabricator.wikimedia.org/T128737#2195351 (10Andrew) [16:08:18] 6Labs: pdns trying to resolve wikimedia.org.eqiad.wmflabs - https://phabricator.wikimedia.org/T128123#2195352 (10Andrew) [16:08:20] 6Labs, 13Patch-For-Review: Periodic internal labs dns outages - https://phabricator.wikimedia.org/T124680#2195350 (10Andrew) [16:11:57] (03CR) 10Ricordisamoa: "PS76 saves some bytes" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [16:18:22] 6Labs, 13Patch-For-Review: Periodic internal labs dns outages - https://phabricator.wikimedia.org/T124680#2195402 (10chasemp) This outage pattern happened again today with a similar pattern and time of day. 2:47 UTC or so first reported and then it was flaky until 3:15 UTC or so. We restarted pdns in there a... [16:19:11] WARNING: POSSIBLE DNS SPOOFING DETECTED! [16:19:17] <-- what's that? [16:19:18] ? [16:19:30] In what? [16:19:47] login.tools.wmflabs.org [16:19:50] when I ssh [16:20:08] When did you last login to that? [16:20:19] maybe last week [16:20:24] (03PS77) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 [16:20:30] https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/tools-login.wmflabs.org [16:20:40] The fingerprints are here [16:20:46] Do they match? [16:22:04] Nope, SHA256 don't match [16:22:26] mafk: the bastion key changed last week or so [16:22:31] there was an email notice to labs-l [16:22:46] I'm not subscribed [16:22:47] 10Wikibugs: Wikibugs comments twice, once to the wrong comment - https://phabricator.wikimedia.org/T132354#2195428 (10Luke081515) [16:22:58] 10Wikibugs: Wikibugs comments twice, once to the wrong comment - https://phabricator.wikimedia.org/T132354#2195440 (10Luke081515) [16:23:02] so chasemp, what shall I do? [16:23:14] Ignore the warning. [16:23:20] change the key in my known_hosts directory? [16:24:01] delete the old entry mafk [16:24:22] chasemp: done, and successfully logged via tools-login.wmflabs.org [16:30:06] (03CR) 10Ricordisamoa: "PS77 changes two @return to @param" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [16:33:06] (03PS1) 10MarcoAurelio: Modify help URL [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/282725 [16:38:57] (03CR) 10Luke081515: [C: 031] Modify help URL [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/282725 (owner: 10MarcoAurelio) [16:42:04] (03CR) 10Luke081515: "The page itself should be modified too, it shows still "powered by toolserver"" [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/282725 (owner: 10MarcoAurelio) [16:42:43] (03PS2) 10MarcoAurelio: Modify help URL [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/282725 [16:45:16] (03CR) 10MarcoAurelio: "There are a lot of outdated stuff on this project. I'm trying to get some things updated, and the help pages were next on my list. I've al" [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/282725 (owner: 10MarcoAurelio) [16:45:50] (03CR) 10Luke081515: [C: 031] "Ok. This set looks good too." [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/282725 (owner: 10MarcoAurelio) [16:46:05] Luke|Busy: some empty spaces cleanup [16:46:15] yeah, I saw that ;) [16:46:39] (03CR) 10MarcoAurelio: [C: 032] Modify help URL [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/282725 (owner: 10MarcoAurelio) [16:47:14] (03Merged) 10jenkins-bot: Modify help URL [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/282725 (owner: 10MarcoAurelio) [16:47:45] MarcoAurelio: I guess you need to restart the bot after deploying ;) [16:47:53] no need [16:48:00] sure? [16:48:17] I don't care if people can't find the help page for a day or two xD [16:48:24] ah, ok :D [16:48:37] it's kinda messy that I'll update it, and after that, I'll restart, etc. [16:49:03] pitty that jenkins can't check python code because I wasn't able to get that python texts/lints running :| [16:49:13] mafk: You can clean up a lot for more tabs: https://gerrit.wikimedia.org/r/#/c/282725/2/StewardBot/StewardBot.py => goto line 291 [16:49:29] maybe asking hashar helps? [16:50:13] o-O, wtf [16:50:22] notepad++ failure [16:50:27] gerrit displays ~30-40 [17:06:16] (03PS1) 10MarcoAurelio: Attempting to clean Stewardbot's code from unneeded TAB and empty spaces [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/282727 [17:07:35] Luke|Busy: ^^ [17:07:45] 6Labs, 10Labs-Infrastructure, 6Operations: labnet1002 can't talk to webproxy.eqiad.wmnet:8080, puppet fails to install designateclient - https://phabricator.wikimedia.org/T129623#2195699 (10Dzahn) on labnet1002, running puppet seems fine now: Notice: Finished catalog run in 18.62 seconds [labnet1002:~] $ [17:08:12] ah [17:08:52] let me see if you got all [17:09:17] 6Labs, 10Labs-Infrastructure, 6Operations: labnet1002 can't talk to webproxy.eqiad.wmnet:8080, puppet fails to install designateclient - https://phabricator.wikimedia.org/T129623#2195709 (10Dzahn) 5Open>3Resolved a:3chasemp @chasemp resolved, right? [labnet1002:~] $ nc webproxy.eqiad.wmnet 8080 GET in... [17:09:25] please compare with the previous patch merged [17:09:59] ok [17:12:16] (03CR) 10Luke081515: [C: 031] "Looks like you got all. Congrats!" [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/282727 (owner: 10MarcoAurelio) [17:17:28] mafk: it would be a good idea to subscribe to labs-l -- there's more relevant updates there. [17:18:24] 6Labs, 10Labs-Sprint-100, 10Tool-Labs: Deploy new unified webservice code - https://phabricator.wikimedia.org/T98440#2195796 (10yuvipanda) a:3yuvipanda [17:20:53] PROBLEM - Host tools-worker-1011 is DOWN: PING CRITICAL - Packet loss = 100% [17:21:36] valhallasw`cloud: will subscribe later then [17:38:18] (03PS4) 10Ladsgroup: Add #wikimedia-ai channel [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/282364 (https://phabricator.wikimedia.org/T132359) [17:40:10] (03PS1) 10MarcoAurelio: Temporary web page for stewardbot documentation. [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/282734 [17:41:00] (03CR) 10MarcoAurelio: [C: 032] Temporary web page for stewardbot documentation. [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/282734 (owner: 10MarcoAurelio) [17:46:05] (03Merged) 10jenkins-bot: Temporary web page for stewardbot documentation. [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/282734 (owner: 10MarcoAurelio) [17:47:12] Luke|Busy: http://tools.wmflabs.org/stewardbots/StewardBot/StewardBot-temp.html <-- at least that's an improvement! :D [17:47:19] many things to fix though [17:47:58] (03PS2) 10MarcoAurelio: Attempting to clean Stewardbot's code from unneeded TAB and empty spaces [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/282727 [17:48:32] valhallasw`cloud: hey, do you have a min to check this? https://gerrit.wikimedia.org/r/282364 [17:52:33] mafk: Yeah, looks better [17:53:02] but larger letter whould not be a disadvantage ^^ [17:56:26] (03CR) 10Merlijn van Deen: [C: 032] Add #wikimedia-ai channel [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/282364 (https://phabricator.wikimedia.org/T132359) (owner: 10Ladsgroup) [17:56:43] 6Labs, 10Labs-Infrastructure, 6Operations: Migrate labsdb1005/1006/1007 to jessie - https://phabricator.wikimedia.org/T123731#2195914 (10Dzahn) [17:56:45] (03CR) 10Merlijn van Deen: [C: 032] "lgtm" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/282719 (https://phabricator.wikimedia.org/T111214) (owner: 10Alex Monk) [17:57:01] (03Merged) 10jenkins-bot: Add #wikimedia-ai channel [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/282364 (https://phabricator.wikimedia.org/T132359) (owner: 10Ladsgroup) [17:57:19] !log tools.wikibugs Updated channels.yaml to: 4acf9e002ad00a8af3553833f99b754bfc0e189c Merge "Add #wikimedia-ai channel" [17:57:20] 6Labs, 10Labs-Infrastructure, 10DBA, 6Operations: Migrate labsdb1005/1006/1007 to jessie - https://phabricator.wikimedia.org/T123731#2195917 (10Dzahn) [17:57:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL, Master [17:57:25] (03Merged) 10jenkins-bot: Try to fix display of colours appearing directly before numbers [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/282719 (https://phabricator.wikimedia.org/T111214) (owner: 10Alex Monk) [17:57:55] thanks :) [17:57:56] Luke|Busy: http://tools.wmflabs.org/stewardbots/StewardBot/StewardBot-temp.html ? [17:58:55] mafk: Yeah, better :) [17:58:57] !log tools.wikibugs valhallasw: Deployed 170e3ace519867782ecb709eb095059b063d1cd1 Merge "Try to fix display of colours appearing directly before numbers" wb2-irc [17:58:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL, Master [17:59:37] mafk: Maybe you should update the example for @help too :P It still shows the old link [18:00:04] time to time [18:00:38] mafk: Or you can remove it, because the people who opened the page already tried this command :D [18:03:24] twentyafterfour: could you try commenting on https://phabricator.wikimedia.org/T1152 to see if we fixed the wikibugs color issue? [18:03:49] I was about to assign a task for that [18:03:59] but wikibugs still appears to be restarting [18:04:05] uuuh [18:04:06] hm [18:04:07] that's not good [18:04:09] * valhallasw`cloud missed that [18:04:12] indeed :p [18:04:50] * valhallasw`cloud frowns [18:12:19] 10Wikibugs: wikibugs test bug - https://phabricator.wikimedia.org/T1152#2195992 (10valhallasw) 👍 [18:12:50] 10Wikibugs: Test fix for T111214 - https://phabricator.wikimedia.org/T132368#2195994 (10Krenair) a:05Krenair>0320after4 [18:13:09] 10Wikibugs: Test fix for T111214 - https://phabricator.wikimedia.org/T132368#2195930 (10Krenair) a:0520after4>03Krenair [18:13:16] valhallasw`cloud, ^ looks good [18:13:19] Krenair: seems to work! thanks :-) [18:13:34] 10Wikibugs: Test fix for T111214 - https://phabricator.wikimedia.org/T132368#2195930 (10Krenair) 05Open>03Invalid works [18:13:46] 10Wikibugs, 13Patch-For-Review: Assigning a task to someone with a name beginning with a number causes IRC colour issues - https://phabricator.wikimedia.org/T111214#2195999 (10Krenair) 05Open>03Resolved a:03Krenair [18:19:20] 06Labs, 06Operations, 13Patch-For-Review: Kill the 'puppet' module with fire, make self hosted puppetmasters use the puppetmaster module - https://phabricator.wikimedia.org/T120159#2196048 (10Krenair) [18:34:40] (03PS1) 10MarcoAurelio: Stewardbot's new webpage code fixes [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/282742 [18:35:59] (03CR) 10MarcoAurelio: [C: 032] Stewardbot's new webpage code fixes [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/282742 (owner: 10MarcoAurelio) [18:39:50] (03Merged) 10jenkins-bot: Stewardbot's new webpage code fixes [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/282742 (owner: 10MarcoAurelio) [18:53:06] RECOVERY - Puppet run on tools-bastion-mtemp is OK: OK: Less than 1.00% above the threshold [0.0] [19:06:03] 06Labs: mwscriptwikiset broken when using all.dblist on terbium - https://phabricator.wikimedia.org/T132383#2196393 (10aaron) [19:08:11] 06Labs, 10wikitech.wikimedia.org: mwscriptwikiset broken when using all.dblist on terbium - https://phabricator.wikimedia.org/T132383#2196412 (10Krenair) [19:15:12] 06Labs, 10DBA: Querying the logging table on labs is slow - https://phabricator.wikimedia.org/T131266#2196441 (10Sigma) Hi @jcrespo Thank you for your response. I've looked over the view definition. I think this slowness could be circumvented by creating a new index log_type_deleted_title_time on (log_type,... [19:16:17] YuviPanda, does https://phabricator.wikimedia.org/T87199 need puppet changes or is there some command to run on each instance? [19:28:34] I'll just reply on-task [19:29:48] Krenair: iiuc there is logic in puppet to setup salt "grains" teh equiv of facts and in beta it is not specific [19:29:53] should be a puppet or hiera change or both [19:33:40] chasemp, I'm struggling to understand [19:34:04] chasemp, you're saying there is a way to set this on beta hosts via hiera? [19:35:30] backtracking a bit then, so salt allows targeting / grouping by grains and we have a defined type I think [19:35:30] modules/salt/manifests/grain.pp [19:35:32] in ^ [19:35:34] to set thees [19:35:42] and a lot of the source of them flows from a heira lookup [19:35:43] like [19:35:56] hieradata/role/eqiad/lvs/balancer.yaml [19:36:04] debdeploy::grains: [19:36:05] debdeploy-lvs-eqiad: [19:36:05] value: standard [19:36:16] I think the work is done by modules/salt/files/grain-ensure.py [19:36:25] so somewhere things in beta end up w/ a misc cluster grain [19:36:25] I saw those debdeploy entries but how is debdeploy relevant? [19:36:44] I'm trying to explain how grains are set from hiera so it's just an example [19:36:54] are those salt grains? [19:36:58] right [19:37:04] yes? [19:37:09] yes [19:37:13] ok.... [19:38:14] ah, modules/base/manifests/debdeploy.pp does some magic to take that data and tell salt [19:38:39] yes debdeploy piggybacks on grains for targeting [19:39:07] So we could do something similar for labs? [19:40:29] 06Labs, 10DBA: Querying the logging table on labs is slow - https://phabricator.wikimedia.org/T131266#2161630 (10Volans) @Sigma: FYI jcrespo is on vacation, he will be back at the end of the week. [19:40:44] sure I imagine, although this is within beta afaik and that has its own salt setup [19:40:50] I think [19:41:58] modules/role/manifests/salt/minions.pp: cluster => hiera('cluster', $::cluster), [19:42:27] It sounds like we can just set the 'cluster' key in hiera? [19:42:49] that seems like it [19:42:51] and then look at [19:42:52] modules/puppetmaster/files/labs.hiera.yaml [19:42:58] to kind of understand how the lookup will work out [19:45:49] so I could make a hiera page on wikitech for cache-text04 [19:45:54] set cluster: cache_text [19:47:11] something like [19:47:11] https://wikitech.wikimedia.org/wiki/Hiera:Tools/host/tools-worker-1011 [19:47:18] but for that key I imagine yes [19:47:25] I don't actually know what the right cluster values are [19:48:32] yeah I'm going to get someone to dump the result of that salt command on the ticket for prod [19:51:23] RECOVERY - Puppet run on tools-mail-01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:52:34] Krenair: yes, you can do that per-host now, but gets unweildy for large number of hosts [19:52:45] YuviPanda, so what should we do instead? [19:52:59] Krenair: make 'role' work for labs as well [19:53:09] Krenair: so you can define them once only [19:53:10] isn't that a separate ticket? [19:53:39] yeah but I think that should block this [19:53:55] So completing that ticket won't get us this for free? [19:54:01] ok [19:54:28] Krenair: oh, that depends on how that is completed. if we load things from ops/puppet's hieradata/role stuff in labs too, then yes it'll get us it for free. [19:54:52] Krenair: the problem with setting it up per-host is that someone will have to keep it up to date, and if you want to change one thing you've to remember to change it manually everywhere. gets out of hand fast [19:55:27] 06Labs, 10Labs-Infrastructure: Make labs wikitech role aware - https://phabricator.wikimedia.org/T127771#2196561 (10Krenair) [19:55:34] indeed [19:55:45] Krenair: however, setting up role based lookup, even if we only use wikitech, would be still far better, since we can jsut do it once. it'll also make life easier in other areas too [20:06:10] 06Labs, 10Labs-Infrastructure: Make labs wikitech role aware - https://phabricator.wikimedia.org/T127771#2196582 (10yuvipanda) p:05Low>03Normal This would also make things like setting up k8s worker nodes easier, since we can set hiera variables by the roles they've applied rather than one per host, which... [20:19:22] RECOVERY - Puppet run on tools-worker-1009 is OK: OK: Less than 1.00% above the threshold [0.0] [22:29:14] 06Labs, 10Tool-Labs, 10DBA: Disabling general.confirmeduser from dbreports for using up too much db resources - https://phabricator.wikimedia.org/T131956#2197353 (10Danny_B) [22:58:24] 06Labs, 10Tool-Labs, 06Collaboration-Team-Backlog, 06Community-Tech-Tool-Labs, and 2 others: Enable Flow on wikitech (labswiki and labtestwiki), then turn on for Tool talk namespace - https://phabricator.wikimedia.org/T127792#2197425 (10Mattflaschen) [22:59:10] 06Labs, 10Tool-Labs, 06Collaboration-Team-Backlog, 06Community-Tech-Tool-Labs, and 2 others: Enable Flow on wikitech (labswiki and labtestwiki), then turn on for Tool talk namespace - https://phabricator.wikimedia.org/T127792#2054159 (10Mattflaschen) See update to the description regarding External Store.