[00:00:48] 06Labs, 10Tool-Labs: define a prebaked way to temporarily disable access to a tool - https://phabricator.wikimedia.org/T147242#2686515 (10chasemp) [00:25:50] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [01:35:34] 503 on http://tools.wmflabs.org/splinetools/whois - does anyone know if the tool is abandoned? [03:47:20] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Tobias47n9e was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=877983 edit summary: [03:48:41] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/G was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=877985 edit summary: [06:22:40] deployment-phab02 puppet-agent[518]: Could not request certificate: getaddrinfo: Name or service not known [06:22:53] is labs dns acting up or is it just something wrong with my instance? [06:23:28] * twentyafterfour configured everything through horizon for the first time, not sure if that is the reason ...but I can't log into any instance I create in deployment-prep [06:25:03] on a different instance I instead get Error 400 on SERVER: DNS lookup failed for deployment-tin.deployment-prep.eqiad.wmflabs [06:25:17] it's acting like dns issues in both cases ... [06:27:41] 06Labs, 10Labs-Infrastructure, 10DBA, 07Upstream: mysqld process hang in db1069 - S2 mysql instance - https://phabricator.wikimedia.org/T145077#2687248 (10Marostegui) Makes sense - I will closed it as resolved and will report back if Percona/MariaDB report some findings. [06:27:53] 06Labs, 10Labs-Infrastructure, 10DBA, 07Upstream: mysqld process hang in db1069 - S2 mysql instance - https://phabricator.wikimedia.org/T145077#2687249 (10Marostegui) 05Open>03Resolved [06:44:23] 06Labs, 10Labs-Infrastructure, 10DBA, 10MassMessage, and 2 others: mysqld process hang in db1069 - S2 mysql instance - https://phabricator.wikimedia.org/T145077#2687274 (10Legoktm) The queries listed in the bug so far all seem to occur for the MassMessage system user "MediaWiki message delivery" (by lookin... [06:47:34] PROBLEM - Puppet run on tools-webgrid-lighttpd-1412 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [06:49:16] 06Labs, 10Labs-Infrastructure, 10DBA, 10MassMessage, and 2 others: mysqld process hang in db1069 - S2 mysql instance - https://phabricator.wikimedia.org/T145077#2687278 (10Marostegui) Thanks a lot @Legoktm for spending time on this. Hopefully by changing the code as you just did and converting them to Inn... [07:27:36] RECOVERY - Puppet run on tools-webgrid-lighttpd-1412 is OK: OK: Less than 1.00% above the threshold [0.0] [08:31:08] 06Labs, 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Replace all class imports on Labs with role imports - https://phabricator.wikimedia.org/T147233#2686048 (10hashar) Would it be possible to have for each class the list of instances having the class applied ? Would ease the migration toward roles :] [09:33:25] PROBLEM - Puppet run on tools-worker-1014 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [10:08:24] RECOVERY - Puppet run on tools-worker-1014 is OK: OK: Less than 1.00% above the threshold [0.0] [10:16:38] 06Labs, 10Labs-project-other, 10Tool-Labs, 06WMDE-Analytics-Engineering: Add http://tools.wmflabs.org/grafana-json-datasource as a datasource to labs grafana instance - https://phabricator.wikimedia.org/T141265#2687838 (10Addshore) 05Open>03Resolved a:03Addshore [10:16:42] 06Labs, 10Labs-project-other, 10Tool-Labs, 06WMDE-Analytics-Engineering: Add http://tools.wmflabs.org/grafana-json-datasource as a datasource to labs grafana instance - https://phabricator.wikimedia.org/T141265#2492254 (10Addshore) [10:16:44] 06Labs, 10Labs-project-other, 10Tool-Labs, 06WMDE-Analytics-Engineering, 15User-Addshore: Add simple-json-datasource plugin to labs grafana instance - https://phabricator.wikimedia.org/T141636#2687841 (10Addshore) 05Open>03Resolved a:03Addshore [12:15:52] 06Labs, 10Labs-Infrastructure, 10DBA: prepare storage layer for olo.wikipedia - https://phabricator.wikimedia.org/T147302#2688126 (10Dzahn) [12:16:47] 06Labs, 10Labs-Infrastructure, 10DBA: prepare storage layer for olo.wikipedia - https://phabricator.wikimedia.org/T147302#2688140 (10Dzahn) [12:17:22] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: prepare storage layer for olo.wikipedia - https://phabricator.wikimedia.org/T147302#2688126 (10Dzahn) [13:37:10] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: prepare storage layer for olo.wikipedia - https://phabricator.wikimedia.org/T147302#2688485 (10jcrespo) This is not a blocking step at this point- the process can continue but this must be kept open until the production side of filtering is run. [13:59:51] twentyafterfour, it's a weird error that appears every so often [14:00:07] The fix is to recreate the instance. [14:17:07] (03CR) 10Dzahn: "I can assume that a phab project has been renamed, but neither the commit message nor the linked task say anything about why we are doing " [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/313761 (https://phabricator.wikimedia.org/T142851) (owner: 10Paladox) [14:20:32] twentyafterfour: are you still having login issues? I have time to look if so [14:21:53] (03PS3) 10Paladox: Change project Project-Creators to Project-Admins [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/313761 (https://phabricator.wikimedia.org/T142851) [14:23:50] 10Wikibugs, 07Easy, 13Patch-For-Review: Change Project-Creators to Project-Admins in channels.yaml - https://phabricator.wikimedia.org/T142851#2548434 (10Dzahn) Where has the project name been changed and why? [14:24:05] (03PS4) 10Paladox: Change project Project-Creators to Project-Admins [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/313761 (https://phabricator.wikimedia.org/T142851) [14:26:21] 10Wikibugs, 07Easy, 13Patch-For-Review: Change Project-Creators to Project-Admins in channels.yaml - https://phabricator.wikimedia.org/T142851#2688695 (10Aklapper) See https://phabricator.wikimedia.org/project/manage/835/#19216 [14:36:31] (03CR) 10Dzahn: [C: 032] Change project Project-Creators to Project-Admins [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/313761 (https://phabricator.wikimedia.org/T142851) (owner: 10Paladox) [14:39:36] 10Wikibugs, 07Easy, 13Patch-For-Review: Change Project-Creators to Project-Admins in channels.yaml - https://phabricator.wikimedia.org/T142851#2688753 (10Dzahn) Thank you for confirming. I merged that. [14:40:19] hashar: any objection to https://gerrit.wikimedia.org/r/#/c/314007/? It looks to me like that class is only used in beta... [14:41:41] 10Wikibugs, 07Easy, 13Patch-For-Review: Change Project-Creators to Project-Admins in channels.yaml - https://phabricator.wikimedia.org/T142851#2688756 (10Dzahn) @Rithika I added a +2 on that and then jenkins merged it. I wonder if you know about additional steps to deploy it and have the needed permissions. [14:42:03] (03Merged) 10jenkins-bot: Change project Project-Creators to Project-Admins [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/313761 (https://phabricator.wikimedia.org/T142851) (owner: 10Paladox) [14:48:58] 10Wikibugs, 07Easy, 13Patch-For-Review: Change Project-Creators to Project-Admins in channels.yaml - https://phabricator.wikimedia.org/T142851#2688799 (10Paladox) >>! In T142851#2688756, @Dzahn wrote: > @Rithika I added a +2 on that and then jenkins merged it. I wonder if you know about additional steps to d... [14:53:12] andrewbogott: yeah still can't log in to deployment-phab01 or deployment-phab02 [14:53:32] twentyafterfour: and those are new instances, or ones that used to work? [14:53:41] new [14:53:56] I have never been able to log in to either [14:54:07] or the other 2 that I created and deleted already [14:55:52] Error 400 on SERVER: DNS lookup failed for deployment-tin.deployment-prep.eqiad.wmflabs [14:55:59] andrewbogott: yeah that [14:56:21] and before rebooting them I had an error getting the puppet cert [14:56:28] another dns lookup error [14:56:34] have instances with these names previously existed? [14:56:43] Krenair: not that I know of [14:57:01] I feel like I heard yuvi debugged one with that case recently [14:57:16] twentyafterfour, does the instance have a project puppetmaster? [14:57:26] yes tom29739 these are deployment- instances [14:57:44] what's the name of the puppetmaster there? [14:57:50] deployment-puppetmaster [14:58:08] I remember something like this happening a bit ago. [14:59:16] andrewbogott: I was thinking this might be related to the scap stuff that you and thcipriani changed yesterday. [14:59:55] do you have a theory for how it's connected? Other than proximity? [15:00:31] mostly the proximity but I really don't know [15:00:33] ok [15:00:38] here's what I suspect is happening... [15:00:52] 1) Brand new instances get their first puppet run on the labs puppet master (labcontrol1001) always [15:01:09] 2) Something in the puppet config for that instance assumes that it will only ever have deployment-puppetmaster as the puppetmaster [15:01:14] I was trying out the puppet config in horizon which I never used before, so that might also be related [15:01:16] so, labcontrol1001 is trying to do that host lookup [15:01:36] which it should be able to do, unless [15:01:41] it's using the production dns system [15:01:48] can you link me to the change? [15:02:01] why would labcontrol1001 /not/ be using the production dns system? [15:02:25] if that particular lookup command was set to use the labs dns servers instead [15:02:39] hmmm [15:02:54] maybe I can remove the puppet configs that are causing that to happen [15:04:08] typically I'd advise letting an instance come up and set up ssh and such before applying roles [15:04:37] Okay, no link, but I found it anyway [15:04:46] $deployment_ip = ipresolve($deployment_host) [15:04:52] in role::beta::mediawiki [15:05:52] doesn't that need to be ipresolve($deployment_host, 4, $::nameservers[0]) ? [15:06:22] yeah, that should fix it [15:06:44] hnnn [15:06:46] hmmmm [15:07:54] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: prepare storage layer for olo.wikipedia - https://phabricator.wikimedia.org/T147302#2688832 (10Dzahn) Thank you, i will go ahead with adding it to DNS. [15:16:21] https://gerrit.wikimedia.org/r/#/c/314011/ [15:21:30] andrewbogott, ^ [15:21:31] lgtm [15:21:40] twentyafterfour, hey did you see https://phabricator.wikimedia.org/D401 ? [15:24:07] twentyafterfour: try rebooting those boxes and see if they're happier now [15:24:40] Krenair: no didn't see that, looking [15:24:44] andrewbogott: rebooting, thank you! [15:33:21] andrewbogott: except that wasn't the actual culprit. :-/ https://gerrit.wikimedia.org/r/#/c/314014/ [15:36:24] so yeah removing scap::target makes the instances boot at least [15:36:57] I mean, they run puppet now and I bet I can log in [15:39:09] twentyafterfour: merged [15:41:21] Warning: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: deployment-puppetmaster.deployment-prep.eqiad.wmflabs] [15:41:23] hmm [15:45:06] andrewbogott: that fixed puppet! yay [15:45:18] Krenair: thanks for helping also [15:46:01] oh, we did it in the wrong class first time [15:46:01] I see [15:53:24] hmm still getting puppet errors about self-signed cert but at least I was able to log in finlly [15:57:51] twentyafterfour, probably due to the failed first run [15:57:57] you should be able to fix it manually now you're in [16:00:58] 06Labs, 06Operations, 13Patch-For-Review: Phase out the 'puppet' module with fire, make self hosted puppetmasters use the puppetmaster module - https://phabricator.wikimedia.org/T120159#1846901 (10ema) On my self-hosted puppetmaster using `role::puppet::self` I've ended up having two stanzas for `[agent]`, o... [16:03:38] 06Labs, 06Operations, 13Patch-For-Review: Phase out the 'puppet' module with fire, make self hosted puppetmasters use the puppetmaster module - https://phabricator.wikimedia.org/T120159#2689030 (10AlexMonk-WMF) >>! In T120159#2689020, @ema wrote: > On my self-hosted puppetmaster using `role::puppet::self` I'... [17:38:52] hi bd808. are you planning to run the ToolLabs survey again this year? [17:39:16] we have just passed its anniversary, so I was wondering if I can help with something, bd808. It would be good to collect this data once a year. [17:40:42] leila: yeah we should do it again. It dropped off my todo list :/ [17:41:08] bd808: np, and honestly running it is not the hard part, the hard part is what you did after that. ;) [17:41:32] bd808: if you want, create a task and assign it to me, I'll send you the details from last year's email, and you can take it from there? [17:41:32] heh. I hope doing it a second time won't eat as much time [17:41:45] sounds like a plan [17:45:16] I bet it won't need nearly as much. You have the whole infrastructure ready for analysis [17:54:22] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs: 2016 Tool Labs user survey - https://phabricator.wikimedia.org/T147336#2689440 (10bd808) [17:54:35] leila: ^ that should get us started [17:54:45] edit as needed obviously [17:55:51] On wikitech.wikimedia.org what should I put for Service group name ? [17:56:46] tobias47n9e-c, you're creating a service group? choose a descriptive name... [17:57:50] Krenair: I just want to create a new tool. Or more precise a website for https://commons-app.github.io/ [17:58:00] okay [17:58:03] so choose a name [17:58:18] I pressed on "create a new tool". [17:58:27] Okay I will :) [17:58:41] tobias47n9e-c: the service group name will be part of the url to your tool. i.e. https://tools.wikimedia.org// [17:58:43] Do i use dashes? [17:58:50] "commons-app-web" [17:59:04] yeah that sounds fine [17:59:55] thanks bd808, re the phab task [18:01:43] And then "bastion" or "tools"? I don't understand what the bastion page is about. [18:06:34] tobias47n9e-c: you need to "hop" through the bastion to get to the tools server shell host. So you can either ssh to bastion.wmflabs.org and from there ssh to login.tools.wmflabs.org or you can setup your ssh client to use bastion.wmflabs.org as a transparent proxy -- https://wikitech.wikimedia.org/wiki/Help:Access#Accessing_instances_with_ProxyCommand_ssh_option_.28recommended.29 [18:07:03] our docs on this seem horrible :/ [18:08:34] If your client can't do the ProxyCommand then you can use agent forwarding: https://wikitech.wikimedia.org/wiki/Help:Access#Accessing_instances_using_agent_forwarding [18:09:17] using a proxy command is more secure in that it doesn't expose your ssh-agent to possible attacks [18:11:05] tobias47n9e-c: it would be awesome if you took notes about what docs are confusing and what actually works and open a phab task or just send me an email about things we should make more clear. [18:11:23] and be WP:BOLD and fix what you can on wikitech! [18:22:51] bd808: Thanks will do that. [18:26:56] bd808, why would you do that when there's the tools bastion? [18:53:34] I think I just realized that i know not enough about SSH to do this O_o [18:57:07] is the tools bastion accessible to the outside world tom29739 ? [18:57:22] Krenair, yep. [18:57:36] login.tools.wmflabs.org [18:58:08] bd808, yeah that particular host might not need to go via the labs bastion? [19:00:18] When I use "ssh commons-app-web.esams.wmflabs" I get "channel 0: open failed: administratively prohibited: open failed" [19:00:51] esams? [19:01:01] esams doesn't have any labs infrastructure [19:01:04] labs is all 100% in eqiad [19:01:24] unless you count labtest, which is in codfw. but that's small [19:03:47] Now i see this: [19:03:56] ssh commons-app-web.eqiad.wmflabs [19:03:56] channel 0: open failed: administratively prohibited: open failed [19:03:56] stdio forwarding failed [19:03:57] ssh_exchange_identification: Connection closed by remote host [19:05:15] I thought you were making a service group in tools rather than your own instance [19:05:19] what project is this instance in? [19:07:26] Krenair: I did make a service group: tools.commons-app-web [19:07:40] okay so you need to ssh to login.tools.wmflabs.org [19:07:43] there is no commons-app-web.eqiad.wmflabs [19:11:32] Krenair: [19:11:42] It doesn't seem to like my password [19:11:59] I type it 3 times and then it disconnects [19:12:01] you shouldn't be giving it a password [19:12:16] Permission denied (publickey,keyboard-interactive,hostbased). [19:12:23] the password you gave it - if you use that password elsewhere, go and change it [19:14:26] then set up your public ssh key in wikitech preferences [19:17:22] !log tools.wikidata-exports Added Markus Kroetzsch and Guenthermi as project maintainers [19:17:42] 06Labs, 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Replace all class imports on Labs with role imports - https://phabricator.wikimedia.org/T147233#2689877 (10Andrew) [19:31:10] !log toolsbeta puppet is broken due to incorrect certificates. Cleaning up ('puppet cert clean toolsbeta-webgrid-lighttpd-1406.toolsbeta.eqiad.wmflabs' on puppetmaster3, 'rm -f /var/lib/puppet/client/ssl/certs/toolsbeta-webgrid-lighttpd-1406.toolsbeta.eqiad.wmflabs.pem' on host, for all hosts that I got emails for) [19:31:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL, Master [19:33:23] !log quarry removed myself as admin [19:33:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL, Master [19:35:31] o/ valhallasw`cloud [19:35:42] bd808: *waves* [19:37:32] 06Labs, 10Tool-Labs: "exim paniclog on tools-cron-01.tools.eqiad.wmflabs has non-zero size" - https://phabricator.wikimedia.org/T145524#2689980 (10valhallasw) 05Open>03Resolved a:03valhallasw Probably. It's odd that the message is from `tools-cron-01.tools.eqiad.wmflabs` (rather than tools-mail). I have... [19:39:44] PROBLEM - SSH on tools-webgrid-generic-1404 is CRITICAL: Server answer [19:39:52] 06Labs, 10Tool-Labs: Setup Failoverable Puppetmasters for tools - https://phabricator.wikimedia.org/T145883#2644019 (10valhallasw) Or, alternatively, a recovery plan for when it does die. The only thing that is tools-specific on the puppetmaster are the certificates of all the hosts, right? [19:41:09] 06Labs, 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Replace all class imports on Labs with role imports - https://phabricator.wikimedia.org/T147233#2689985 (10Andrew) [19:49:46] Hmm I must be doing something wrong. Now I generated a new key without a password, and ssh login.tools.wmflabs.org is still asking me for a password [19:52:49] 06Labs, 10Tool-Labs: Change Python hashbang to `#! /usr/bin/env python -E -s` for user-facing tools - https://phabricator.wikimedia.org/T147350#2690003 (10valhallasw) [19:53:41] tobias47n9e-c, what's your username? [19:53:52] tobias47n9e [19:55:51] 06Labs, 10Tool-Labs: Change Python hashbang to `#! /usr/bin/env python -E -s` for user-facing tools - https://phabricator.wikimedia.org/T147350#2690097 (10valhallasw) In addition, `-E -s` makes `webservice` start much faster if `PYTHONPATH` refers to a directory on NFS; the user-packages directory is always on... [19:55:57] 06Labs, 10Tool-Labs: BUB 503: AttributeError: 'module' object has no attribute 'python_2_unicode_compatible' - https://phabricator.wikimedia.org/T144554#2690100 (10valhallasw) 05Open>03Resolved a:03valhallasw If you don't want to remove your `PYTHONPATH` setting, you can use ```python -E -s `which webse... [20:02:01] tobias47n9e-c, and your public key is ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDC6BAs7onAWVxElBU2MH9/QjgpulsU/k9IBeMolNO/aj/Q5hM/TPylDwtgQhew+iqLELHlkJJHOmGMI49BKvt3PbgL/ssDMxjceASIXK+Qs2UDiMnQz/OutWVVO1Xsxk8iY7kzcJXWINiob4zbbdBpjyigAgRCjCEdIjgorFa05MUwBFXq/+d+XTSuLNO+ysJ7J26C1CVnzXf0iLKSOYLchcG2NIhbagon9EFF7mQRJumysFPgPuFrG5mf6JkwXlR4U+zS/2P0K1rkjViiUT0meCwsu9m8DuU/M0rtn1q+2102GEtHnhaR6UKrA4yxUfYpVOGL7RE69I3a1pXWFM77 tobias@two ? [20:02:18] Krenair: now it worked. I am an idiot :) [20:02:28] what did you do wrong? [20:02:38] ssh-add was missing after creating the new key [20:02:41] ah [20:02:43] yeah [20:02:59] I could add that to the "Gotchas" section ;) [20:04:29] there's a gotchas section? [20:04:51] Krenair: https://wikitech.wikimedia.org/wiki/Help:Tool_Labs#Gotchas [20:05:36] yep [20:05:48] there is already 'You might need to use ssh-add after creating a new key.' [20:06:06] .. which you just added. right [20:06:11] thanks tobias47n9e-c [20:06:48] Krenair: Thanks you for your patience and help (Also the rest of the channel) ;) [20:13:35] RECOVERY - Host tools-secgroup-test-102 is UP: PING OK - Packet loss = 0%, RTA = 0.62 ms [20:16:03] PROBLEM - Host tools-secgroup-test-102 is DOWN: CRITICAL - Host Unreachable (10.68.21.170) [20:17:12] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 07Tracking: Issues with 'webservice' kubernetes backend (tracking) - https://phabricator.wikimedia.org/T139107#2690203 (10valhallasw) >>! In T139107#2673477, @Samwilson wrote: > There's no old gridengine webservice running (`qstat` returns empty), `webservice status`... [20:17:19] 06Labs, 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Replace all class imports on Labs with role imports - https://phabricator.wikimedia.org/T147233#2690204 (10Andrew) [20:24:51] 06Labs, 10Tool-Labs, 10Pywikibot-core: Running a core script fails with 'permission denied' creating a logfile folder - https://phabricator.wikimedia.org/T146996#2690229 (10valhallasw) This has nothing to do with pywikibot -- the issue is with the permissions on that directory: ``` valhallasw@tools-bastion-... [20:29:58] RECOVERY - Host tools-secgroup-test-103 is UP: PING OK - Packet loss = 0%, RTA = 0.89 ms [20:32:20] PROBLEM - Host tools-secgroup-test-103 is DOWN: CRITICAL - Host Unreachable (10.68.21.22) [20:33:33] RECOVERY - Host secgroup-lag-102 is UP: PING OK - Packet loss = 0%, RTA = 0.78 ms [20:39:29] 06Labs, 10Tool-Labs: define a prebaked way to temporarily disable access to a tool - https://phabricator.wikimedia.org/T147242#2686515 (10valhallasw) What should and shouldn't this do? Clear yes: - block logins of the tool, and Maybe: - stop all running jobs? - except webservices? - disable all cronjobs?... [20:41:24] 10Wikibugs, 07Easy, 13Patch-For-Review: Change Project-Creators to Project-Admins in channels.yaml - https://phabricator.wikimedia.org/T142851#2690318 (10valhallasw) 05Open>03Resolved a:03valhallasw Channel changes are automatically deployed after merging via the post-merge jenkins hook. [20:47:39] PROBLEM - Host tools-exec-1211 is DOWN: PING CRITICAL - Packet loss = 80%, RTA = 3233.84 ms [20:48:09] PROBLEM - Host secgroup-lag-102 is DOWN: CRITICAL - Host Unreachable (10.68.17.218) [20:49:15] PROBLEM - Puppet staleness on tools-worker-1018 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [20:52:05] 10Tool-Labs-tools-Other, 07Tracking: merl tools (tracking) - https://phabricator.wikimedia.org/T69556#2690379 (10bd808) [20:52:06] RECOVERY - Host tools-exec-1211 is UP: PING OK - Packet loss = 0%, RTA = 0.63 ms [20:52:07] 06Labs, 10Tool-Labs, 10DBA: s51127__dewiki_lists (merlbot) database using 13G on labsdb1001 (enwiki) - https://phabricator.wikimedia.org/T133325#2690378 (10bd808) [21:01:20] (03PS1) 10Jean-Frédéric: Pass positional arguments to nosetests via tox [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/314172 [21:24:37] 06Labs, 10Labs-Infrastructure, 10DBA, 06Operations: Prepare storage layer for olo.wikipedia - https://phabricator.wikimedia.org/T147302#2690580 (10MarcoAurelio) p:05Triage>03Normal [21:34:15] 06Labs, 10Tool-Labs: define a prebaked way to temporarily disable access to a tool - https://phabricator.wikimedia.org/T147242#2690679 (10chasemp) [21:38:08] 06Labs, 10Labs-Kubernetes, 10Tool-Labs, 07Tracking: Issues with 'webservice' kubernetes backend (tracking) - https://phabricator.wikimedia.org/T139107#2690688 (10Samwilson) Thanks @valhallasw — that's strange though, because I had also restarted it under SGE and last I knew it was running! But I guess some... [21:49:52] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [21:54:03] (03CR) 10Jean-Frédéric: "I tried to incorporate your suggested change (complaining for lack of countrycode but with a lang), and went down a rabbit hole of debuggi" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/313451 (owner: 10Jean-Frédéric) [21:56:45] (03Abandoned) 10Jean-Frédéric: Allow to set lang parameter in update_database [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/313451 (owner: 10Jean-Frédéric) [21:59:05] (03PS2) 10Jean-Frédéric: Expand ReadMe on development environment [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/313452 [21:59:12] (03CR) 10Jean-Frédéric: Expand ReadMe on development environment (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/313452 (owner: 10Jean-Frédéric) [22:15:42] 06Labs, 10Tool-Labs, 06Collaboration-Team-Triage, 06Community-Tech-Tool-Labs, and 5 others: Enable Flow on wikitech (labswiki and labtestwiki), then turn on for Tool talk namespace - https://phabricator.wikimedia.org/T127792#2690863 (10Dereckson) We need `$wgFlowDefaultWikiDb` to be set before to be able t... [22:24:52] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:35:21] 06Labs, 10Tool-Labs, 06Collaboration-Team-Triage, 06Community-Tech-Tool-Labs, and 5 others: Enable Flow on wikitech (labswiki and labtestwiki), then turn on for Tool talk namespace - https://phabricator.wikimedia.org/T127792#2690926 (10Mattflaschen-WMF) a:03Dereckson [23:28:54] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Maynich was created, changed by Maynich link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Maynich edit summary: Created page with "{{Tools Access Request |Justification=For reanimation of connectovity project for Russian part of wikipedia |Completed=false |User Name=Maynich }}"