[00:29:38] 6Labs: Test Ceph for instance storage - https://phabricator.wikimedia.org/T90364#1056976 (10Andrew) 3NEW a:3Andrew [01:34:24] 10Wikimedia-Labs-wikitech-interface, 6operations: fix wikitech-static updates - https://phabricator.wikimedia.org/T83596#1057022 (10Andrew) 5Open>3Resolved a:3Andrew This is fixed. It broke because the mw version on -static had a version incompatibility with the version running on wikitech. Of course,... [01:37:23] 10Wikimedia-Labs-wikitech-interface, 6operations: fix wikitech-static updates - https://phabricator.wikimedia.org/T83596#1057027 (10Andrew) This is fixed. It broke because the mw version on -static had a version incompatibility with the version running on wikitech. Of course, we need to monitor. https://ph... [02:04:12] 6Labs, 10MediaWiki-extensions-OpenStackManager: Wikitech 'manage instances' displays "PHP Fatal error: Call to a member function getImageName() on a non-object" - https://phabricator.wikimedia.org/T89856#1057048 (10Andrew) p:5Triage>3Low [02:16:02] 10Wikimedia-Labs-wikitech-interface: Many "(Cannot contact the database server: Too many connections (208.80.154.18))" errors - https://phabricator.wikimedia.org/T78492#1057056 (10Andrew) 5Open>3Resolved This has stopped happening now that wikitech is on silver, safely away from puppet's vandalism. [05:12:58] 10Tool-Labs: Decide how to handle gridengine on Debian Jessie - https://phabricator.wikimedia.org/T90369#1057104 (10scfc) 3NEW [05:16:56] RECOVERY - Puppet staleness on tools-exec-15 is OK: OK: Less than 1.00% above the threshold [3600.0] [05:52:48] 10Tool-Labs: log files not written - https://phabricator.wikimedia.org/T85775#1057118 (10scfc) I think `qmod -rj` will keep the memory settings, but nevertheless it will kill the "previous" job with `-TERM`, if the process does not react to the "milder" signals. However, it also kills my desire to debug this f... [06:44:37] (03PS1) 10Yurik: parsoid/* and various services/ mapping [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/192175 [09:23:03] the tools of Magnus do not function [09:23:16] can someone do something about it please ? [11:04:09] PROBLEM - Host tools-webproxy-jessie is DOWN: CRITICAL - Host Unreachable (10.68.17.147) [12:42:49] (03PS1) 10Merlijn van Deen: Add Fabric deploy tools [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/192183 [12:43:00] (03CR) 10Merlijn van Deen: [C: 032] Add Fabric deploy tools [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/192183 (owner: 10Merlijn van Deen) [12:43:03] (03Merged) 10jenkins-bot: Add Fabric deploy tools [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/192183 (owner: 10Merlijn van Deen) [12:43:37] !log tools.lolrrit-wm valhallasw: Deployed ca0427db5c7482a4a825c67db7e098d5047d53cd Set password [12:43:40] Logged the message, Master [12:43:47] that's almost useful [12:44:34] (03PS1) 10Merlijn van Deen: Use HEAD^ because 'show password' is usless [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/192184 [12:44:47] (03CR) 10Merlijn van Deen: [C: 032] Use HEAD^ because 'show password' is usless [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/192184 (owner: 10Merlijn van Deen) [12:44:49] can a labs admin restart magnus's tools ? [12:44:50] (03Merged) 10jenkins-bot: Use HEAD^ because 'show password' is usless [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/192184 (owner: 10Merlijn van Deen) [12:45:06] can a labs admin restart magnus's tools? [12:45:09] (03CR) 10Merlijn van Deen: [C: 032] parsoid/* and various services/ mapping [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/192175 (owner: 10Yurik) [12:45:12] (03Merged) 10jenkins-bot: parsoid/* and various services/ mapping [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/192175 (owner: 10Yurik) [12:45:17] (03PS2) 10Merlijn van Deen: Add MoodBar, WikiLove to #wikimedia-collaboration [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/188302 (owner: 10Mattflaschen) [12:45:18] stupid bot :/ [12:45:25] (03CR) 10Merlijn van Deen: [C: 032] Add MoodBar, WikiLove to #wikimedia-collaboration [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/188302 (owner: 10Mattflaschen) [12:45:28] (03Merged) 10jenkins-bot: Add MoodBar, WikiLove to #wikimedia-collaboration [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/188302 (owner: 10Mattflaschen) [12:46:01] !log tools.lolrrit-wm valhallasw: Deployed f4a809b6b6ddb5dac25c27d26e1d20967ee6816c Add MoodBar, WikiLove to #wikimedia-collaboration [12:46:04] Logged the message, Master [12:46:42] * valhallasw`cloud frowns [12:47:48] what the heck [12:49:25] !log tools-lolrrit-wm for some reason two grrrit-wm's were running, even though there was only a single job. After qdel, both lolrrits died, and a fab start_job later everything looks OK again [12:49:26] tools-lolrrit-wm is not a valid project. [12:49:34] !log tools.lolrrit-wm for some reason two grrrit-wm's were running, even though there was only a single job. After qdel, both lolrrits died, and a fab start_job later everything looks OK again [12:49:36] Logged the message, Master [13:23:19] can somone PLEASE restart the tools of Magnus [13:23:27] they have been down for almost a day [13:28:16] 10Tool-Labs, 10Tool-Labs-tools-Other: Restart webservice for /magnustools/ - https://phabricator.wikimedia.org/T90384#1057358 (10valhallasw) 3NEW [13:30:54] GerardM-: creating a task is a much better option than dropping in here every now and then hoping someone is around [13:31:10] GerardM-: I don't think magnus is even online, so he might not even be aware of the outage [13:31:39] It should not depend on Magnus [13:31:44] GerardM-: it's magnus tool. [13:32:01] yes ... MediaWiki was started by Magnus as well [13:32:17] and MW is on production level service, while reasonator is not. [13:33:17] Magnus is the only one who has access to the /magnustools/ tool, apart from admins who are on their weekend and thus unlikely to continuously monitor this channel [13:34:20] Yes, we need better monitoring, but we don't have that yet. [13:34:41] So in the meanwhile, stuff will break, and if you want it fixed quickly, you need to make sure the people who can fix it actually know it's broken. [13:37:16] valhallasw this is not the first time ... not at all [13:37:41] GerardM-: I know. [13:38:16] it is not only Wikidata, or Labs that is sufereing [13:39:38] GerardM-: and this is relevant to solving the current issue how? [13:39:57] GerardM-: Yes, there should be better monitoring. Yes, the stability is not good enough. [13:40:06] It is not about monitoring [13:40:11] it is much deeper [13:40:23] it is neglicence [13:40:32] and not of the Labs team' [13:40:48] GerardM-: and how does complaining here solve that? [13:41:12] GerardM-: Every time something breaks down, file a bug [13:41:16] Over and over again [13:41:50] Maybe make a tracking bug 'things that broke down on labs that actively broke my ability to contribute to wmf projects' [13:48:17] I was used to Bugzilla and it is no longer there [13:48:30] now there is something else and it is so different [13:49:48] Get used to it. I already like it more than good old bugzilla ;-) [14:12:55] !log tools.magnustools restarted webservice, had died many many times [14:12:57] Logged the message, Master [14:13:16] actually no [14:14:32] valhallasw`cloud: ^ I restarted [14:18:22] welcome back YuviPanda [14:18:37] GerardM-: am still on vacation :) [14:18:43] for another 24h at least [14:20:55] do you know the tool that has not started for https://tools.wmflabs.org/toolscript/index.html?pastebin=x9TXBfZi to work ? [14:23:01] GerardM-: nope, it references wikidata-todo and that seems up [14:23:42] anyway, am off on my vacation again [14:27:56] Thanks [14:57:12] I am trying to recall the URL to the tool that nicely summarizes a Wikipedian's edit count globally and on the home wiki and says when the Wikipedian started editing... [15:23:54] 6Labs, 10Wikimedia-Labs-wikitech-interface: Unable to see or delete existing web proxy - https://phabricator.wikimedia.org/T90391#1057452 (10Yurik) 3NEW [17:22:56] PROBLEM - Puppet staleness on tools-exec-15 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [43200.0] [17:27:52] 10Tool-Labs, 10Tool-Labs-tools-Other: Restart webservice for /magnustools/ - https://phabricator.wikimedia.org/T90384#1057632 (10scfc) 5Open>3Resolved a:3scfc Someone or something seems to have restarted the webservice at 14:12Z: ``` tools.magnustools@tools-dev:~$ qstat job-ID prior name user... [19:36:02] 10Tool-Labs, 10Tool-Labs-tools-Other: Restart webservice for /magnustools/ - https://phabricator.wikimedia.org/T90384#1057843 (10GerardM) There is a difference between the webservice being up and rht functionality being up. The webservice was up but that is only half the story. Thanks, GerardM [19:52:28] 10Tool-Labs, 10Tool-Labs-tools-Other: Restart webservice for /magnustools/ - https://phabricator.wikimedia.org/T90384#1057859 (10scfc) The `bigbrother` process was (re-)started after it last restarted `magnustools`: ``` root@tools-submit:~# ls -dl /proc/$(pidof perl bigbrother) dr-xr-xr-x 8 root root 0 Feb... [20:01:16] 10Tool-Labs, 10Tool-Labs-tools-Other: Restart webservice for /magnustools/ - https://phabricator.wikimedia.org/T90384#1057864 (10scfc) And now `bigbrother` restarted my tool's webservice. So a) now everything is fine and b) strange. [20:07:51] 6Labs, 10Wikimedia-Labs-wikitech-interface: Unable to see or delete existing web proxy - https://phabricator.wikimedia.org/T90391#1057883 (10DLindsley) [20:40:42] 10Tool-Labs-tools-Database-Queries, 10Analytics, 10MediaWiki-User-preferences: DBQ-197 User preferences usage on Portuguese Wikipedia - https://phabricator.wikimedia.org/T61480#632685 (10He7d3r) [21:35:42] (03CR) 10Legoktm: "<3" [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/192183 (owner: 10Merlijn van Deen) [21:39:10] (03PS1) 10Legoktm: Volunteer? --> Lowest (priority) [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/192245 [21:41:58] (03CR) 10Merlijn van Deen: [C: 032] Volunteer? --> Lowest (priority) [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/192245 (owner: 10Legoktm) [21:42:30] valhallasw`cloud: I was going to wait until the puppet change was merged but ok :P [21:43:28] (03Merged) 10jenkins-bot: Volunteer? --> Lowest (priority) [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/192245 (owner: 10Legoktm) [21:43:50] * legoktm deploys [21:44:23] !log tools.wikibugs legoktm: Deployed 1f579477957417a693308fa9a23d2080821eb551 Volunteer? --> Lowest (priority) wb2-irc [21:44:27] Logged the message, Master [21:46:26] Legoktm: :-P just don't fab deploy [21:46:39] too late! [21:46:42] Oh, you already did [21:46:45] :D [21:49:04] (03PS2) 10Legoktm: Add MoodBar and WikiLove to #wikimedia-collaboration [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/182869 (owner: 10Mattflaschen) [21:49:10] (03PS4) 10Legoktm: Add more repos maintained by Fundraising Tech [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/177378 (owner: 10Awight) [21:50:09] (03CR) 10Legoktm: [C: 032] Add more repos maintained by Fundraising Tech [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/177378 (owner: 10Awight) [21:50:12] (03Merged) 10jenkins-bot: Add more repos maintained by Fundraising Tech [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/177378 (owner: 10Awight) [21:50:29] (03CR) 10Legoktm: [C: 032] Add MoodBar and WikiLove to #wikimedia-collaboration [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/182869 (owner: 10Mattflaschen) [21:50:36] (03Merged) 10jenkins-bot: Add MoodBar and WikiLove to #wikimedia-collaboration [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/182869 (owner: 10Mattflaschen) [21:51:17] [tools-login.wmflabs.org] out: Fetching actuallygerrit [21:51:19] >.> [21:52:00] !log tools.lolrrit-wm legoktm: Deployed c06fb25dbd4ece9e8493393bae7081e42ba9ddbe Merge "Add MoodBar and WikiLove to #wikimedia-collaboration" [21:52:03] Logged the message, Master [21:53:26] valhallasw`cloud: hmm, the other grrrit-wm didn't die? [21:54:29] [04:49:34] !log tools.lolrrit-wm for some reason two grrrit-wm's were running, even though there was only a single job. After qdel, both lolrrits died, and a fab start_job later everything looks OK again [21:54:30] exciting [21:55:18] * legoktm does that [21:56:11] well I qdel'd the job but only one quit........ [21:56:26] :o [21:56:27] okay [21:57:26] !log tools.lolrrit-wm for some reason two grrrit-wm's were running, even though there was only a single job. After qdel, both lolrrits died, and a fab start_job later everything looks OK again [21:57:28] Logged the message, Master [21:57:36] * legoktm will file a bug for that in a bit [22:07:56] Legoktm: yeah, not sure what's happening [22:08:32] Especially because it /is/ killed on qdel [22:08:52] So sge does know how to kill it [22:15:50] hm [22:16:04] valhallasw`cloud: is there a bug for it already? [22:16:17] No [22:16:32] I thought it might be a fluke [22:44:27] legoktm, you have an encycopedic knowledge of the available wikitools, right? ;) [22:44:40] maybe, what's up? [22:44:45] "08:57:12] I am trying to recall the URL to the tool that nicely summarizes a Wikipedian's edit count globally and on the home wiki and says when the Wikipedian started editing... " [22:45:21] this is its sample output, as I have saved it, starts this way, anyway: "Statistics: 30,482 edits on 135 wikis. [22:45:22] Most edits on pl.wikipedia.org (17,937) [22:45:50] except the number is tweaked by hand [22:46:01] as I could not locate it to run it today [22:46:26] 10Tool-Labs, 10Tool-Labs-tools-Other: Restart webservice for /magnustools/ - https://phabricator.wikimedia.org/T90384#1058024 (10valhallasw) >>! In T90384#1057632, @scfc wrote: > Someone or something seems to have restarted the webservice at 14:12Z: > > ``` > tools.magnustools@tools-dev:~$ qstat > job-ID p... [22:47:07] legoktm: meh, I think Krinkle|detached is right about the underscores. It is a bit distracting for the amount of information it adds [22:50:42] (03PS1) 10Merlijn van Deen: Remove underline from matched projects [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/192250 (https://phabricator.wikimedia.org/T89632) [22:50:44] (03PS1) 10Merlijn van Deen: Use brown instead of red for orange (=release) projects [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/192251 (https://phabricator.wikimedia.org/T89632) [22:52:01] valhallasw`cloud: I haven't found them useful yet, but it's not distracting to me. Removing is fine [22:52:25] yeah, it's pretty client-dependent [22:52:34] it's OK in irssi, but more distracting in irccloud [22:53:09] (03CR) 10Merlijn van Deen: [C: 032] Fix incorrect 'and N others' message. [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/192069 (https://phabricator.wikimedia.org/T90324) (owner: 10Merlijn van Deen) [22:53:25] (03CR) 10Merlijn van Deen: [C: 032] Remove underline from matched projects [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/192250 (https://phabricator.wikimedia.org/T89632) (owner: 10Merlijn van Deen) [22:53:29] (03CR) 10Legoktm: [C: 032] Remove underline from matched projects [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/192250 (https://phabricator.wikimedia.org/T89632) (owner: 10Merlijn van Deen) [22:53:31] (03Merged) 10jenkins-bot: Fix incorrect 'and N others' message. [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/192069 (https://phabricator.wikimedia.org/T90324) (owner: 10Merlijn van Deen) [22:53:35] (03CR) 10Merlijn van Deen: [C: 032] Use brown instead of red for orange (=release) projects [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/192251 (https://phabricator.wikimedia.org/T89632) (owner: 10Merlijn van Deen) [22:53:38] (03Merged) 10jenkins-bot: Remove underline from matched projects [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/192250 (https://phabricator.wikimedia.org/T89632) (owner: 10Merlijn van Deen) [22:53:48] (03Merged) 10jenkins-bot: Use brown instead of red for orange (=release) projects [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/192251 (https://phabricator.wikimedia.org/T89632) (owner: 10Merlijn van Deen) [22:54:35] !log tools.wikibugs valhallasw: Deployed 6da78462504cd023e0c31babb5cc56a7eae3a88a Merge "Use brown instead of red for orange (=release) projects" wb2-irc [22:54:38] Logged the message, Master [22:55:10] * valhallasw`cloud prods wikibugs [22:56:54] 10Wikibugs, 5Patch-For-Review: Problems with the new color scheme - https://phabricator.wikimedia.org/T89632#1058043 (10valhallasw) After trying the current formatting for a bit longer, I agree with @Krinkle that the underline is too distracting for the small amount of information it provides. The impact of no... [22:57:01] 10Wikibugs, 5Patch-For-Review: Problems with the new color scheme - https://phabricator.wikimedia.org/T89632#1058044 (10valhallasw) 5Open>3Resolved [22:57:25] legoktm: I liked the bold, but that failed in irssi :( [23:05:49] 10Wikimedia-Labs-Infrastructure: Make the configuration file of labs console public - https://phabricator.wikimedia.org/T38242#1058063 (10Hydriz) 5Open>3Resolved a:3Hydriz Wikitech has been moved into puppet and the configuration for the wiki has moved to mediawiki-config as well. The configuration file is... [23:24:50] 10Wikibugs, 10Bot-Requests, 10Code-Review, 6Phabricator, 10Phabricator.org: wikibugs test bug - https://phabricator.wikimedia.org/T1152#1058072 (10valhallasw) [23:25:41] meh, overflow is still broken [23:26:49] 10Wikibugs: wikibugs test bug - https://phabricator.wikimedia.org/T1152#20004 (10valhallasw) [23:27:05] 10Wikibugs: Wikibugs should not say ", and 0 others" - https://phabricator.wikimedia.org/T90324#1058075 (10valhallasw) The message is gone, but the overflow is still broken: > 00:24 Wikibugs, Bot-Requests, Code-Review, Phabricator, Phabricator.org: wikibugs test bug - https://phabricator.wikimedia.or...