[00:12:12] 6Labs: Create labs project for NonFreeWiki - https://phabricator.wikimedia.org/T108167#1523117 (10Green_Giant) In Accessing instances with ProxyCommand ssh option (recommended), what exactly is "~/.ssh/config" and the "local machine"? [01:04:43] phabricator is down at the moment [01:45:12] 6Labs, 7Tracking: New Labs project requests (Tracking) - https://phabricator.wikimedia.org/T76375#1523158 (10scfc) [01:45:14] 6Labs: Create labs project for NonFreeWiki - https://phabricator.wikimedia.org/T108167#1523156 (10scfc) 5Resolved>3Open Ha. https://wikitech.wikimedia.org/wiki/Nova_Resource:Nonfreewiki says members are @Andrew and novaadmin. @Andrew, I think you forgot to add @Green_Giant to the project. @Green_Giant: `~... [08:16:58] 6Labs, 10Analytics, 10Labs-Infrastructure, 3Labs-Sprint-108, 5Patch-For-Review: Set up cron job on labstore to rsync data from stat* boxes into labs. - https://phabricator.wikimedia.org/T107576#1523508 (10akosiaris) Hey @Halfak, glad you joined us on this one. >>! In T107576#1519641, @Halfak wrote: > He... [08:21:45] 6Labs, 10Analytics, 10Labs-Infrastructure, 3Labs-Sprint-108, 5Patch-For-Review: Set up cron job on labstore to rsync data from stat* boxes into labs. - https://phabricator.wikimedia.org/T107576#1523509 (10yuvipanda) My understanding is that halfak is suggesting that we automatically sync datasets.wikimed... [08:22:45] 6Labs, 10Analytics, 10Labs-Infrastructure, 3Labs-Sprint-108, 5Patch-For-Review: Set up cron job on labstore to rsync data from stat* boxes into labs. - https://phabricator.wikimedia.org/T107576#1523511 (10akosiaris) >>! In T107576#1523509, @yuvipanda wrote: > My understanding is that halfak is suggesting... [09:49:12] 6Labs, 10pywikibot-core: pywikipedia.org down? - https://phabricator.wikimedia.org/T106311#1523595 (10siebrand) Please make a change request without a question mark. Current DNS settings are: pywikipedia.org CNAME - wikimedia-lb.wikimedia.org 86400 *.pywikipedia.org CNAME - wikimedia-lb.wikimedia.org... [09:51:10] 6Labs, 10pywikibot-core: pywikipedia.org down? - https://phabricator.wikimedia.org/T106311#1523596 (10siebrand) >>! In T106311#1523595, @siebrand wrote: > register pywikibot.org, which is currently unclaimed. I registered the domain in anticipation of a yes to this at least... [10:13:59] 6Labs, 10Analytics, 10Labs-Infrastructure, 3Labs-Sprint-108, 5Patch-For-Review: Set up cron job on labstore to rsync data from stat* boxes into labs. - https://phabricator.wikimedia.org/T107576#1523627 (10ArielGlenn) which directories do you want synced over? [10:24:43] All of xtools seem to now be non-responsive :-/ [10:33:25] 6Labs, 10pywikibot-core: pywikipedia.org is not responding; pywikibot.org is not registered - https://phabricator.wikimedia.org/T106311#1523650 (10jayvdb) p:5Triage>3Low [10:35:10] 6Labs, 10pywikibot-core: pywikipedia.org is not responding; pywikibot.org is not registered - https://phabricator.wikimedia.org/T106311#1464523 (10jayvdb) >>! In T106311#1523596, @siebrand wrote: >>>! In T106311#1523595, @siebrand wrote: >> register pywikibot.org, which is currently unclaimed. > > I registere... [10:54:14] 6Labs: Create labs project for NonFreeWiki - https://phabricator.wikimedia.org/T108167#1523662 (10Steinsplitter) >>! In T108167#1514528, @Bugreporter wrote: > Comment: > # Some extensions are not needed, like GWToolset. others includes all CentralAuth/global* extensions, CheckUser, MassMessage, OAIRepository,... [11:00:21] Is it possible to store large (~15mb) text blobs in mariadb on tool labs? I'm using a longtext field that gets cut off around 1.1mb [11:03:24] jakob_WMDE: I think that should be technically possible, but I'm not sure if it's a good idea [11:07:24] 6Labs, 10pywikibot-core: pywikipedia.org is not responding; pywikibot.org is not registered - https://phabricator.wikimedia.org/T106311#1523674 (10valhallasw) >>! In T106311#1523595, @siebrand wrote: > Please make a change request without a question mark. After checking with #wikimedia-operations: the correc... [11:08:24] valhallasw`cloud: since the longtext limit is much much higher i'm guessing that there are some configurations limiting the size. is it still possible to get those into mariadb somehow? [11:08:27] jakob_WMDE: not sure why it gets cut off there, but the docs say "The effective maximum length of LONGTEXT columns also depends on the configured maximum packet size in the client/server protocol and available memory." [11:08:58] what happens when the text is larger? does it get cut off, or do you get an error? [11:09:24] it gets cut off without any errors [11:11:07] jakob_WMDE: odd. No warnings either (if you run SHOW WARNINGS; afterwards?) [11:11:09] max_allowed_packet is 16777216, so 16mb. i'm not sure what else could go wrong to cut it off at 1.1mb already [11:13:07] and/or sql_mode='TRADITIONAL' [11:15:41] if that doesn't work, please create a bug with #labs and #database tags, and please add jcrespo as cc [11:16:21] please note also that bytes != characters [11:16:55] check also client side [11:17:03] valhallasw`cloud: alright, i will try that and see if i can find anything out. thanks [11:18:03] even in non-strict mode, when the field is shorter than the content, a warning is sent [12:58:31] we have some small lag on enwiki db, it will take some minutes to fully recover [13:11:13] (03PS1) 10Sitic: Add Extension:PageTriage logevents support [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/230532 (https://phabricator.wikimedia.org/T108547) [13:13:16] (03CR) 10Sitic: [C: 032 V: 032] Add Extension:PageTriage logevents support [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/230532 (https://phabricator.wikimedia.org/T108547) (owner: 10Sitic) [13:23:03] 6Labs: Create labs project for NonFreeWiki - https://phabricator.wikimedia.org/T108167#1523921 (10Andrew) oops, yeah, username is green-giant. Apparently wikitech doesn't notify me if I request an invalid member when creating a project. @Green_Giant, you should be able to select your project in the project fil... [13:38:41] 6Labs, 10Analytics, 10Labs-Infrastructure, 3Labs-Sprint-108, 5Patch-For-Review: Set up cron job on labstore to rsync data from stat* boxes into labs. - https://phabricator.wikimedia.org/T107576#1523941 (10Ottomata) datasets.wikimedia.org lives on stat1001. The contents of it are rsynced from various /sr... [13:44:19] 6Labs, 10Labs-Infrastructure: Enable arp on the labs vlan - https://phabricator.wikimedia.org/T108581#1523950 (10Andrew) 3NEW a:3Andrew [13:46:59] 6Labs, 10Labs-Infrastructure: Switch to a multi_host nova network - https://phabricator.wikimedia.org/T107731#1523969 (10Andrew) Updated, tentative plan: 0) switch to arp on the labs network (rather than explicitly routing everything to labnet1001 as now) 1) Disable puppet on labnet1001 so it doesn't restart... [13:47:51] 6Labs: Create labs project for NonFreeWiki - https://phabricator.wikimedia.org/T108167#1523975 (10Green_Giant) Ahh. Thank you very much, it now makes sense. [13:48:22] 6Labs, 7Tracking: New Labs project requests (Tracking) - https://phabricator.wikimedia.org/T76375#1523979 (10Green_Giant) [13:48:22] 6Labs: Create labs project for NonFreeWiki - https://phabricator.wikimedia.org/T108167#1523978 (10Green_Giant) 5Open>3Resolved [13:48:43] 6Labs, 10Labs-Infrastructure: Enable arp on the labs vlan - https://phabricator.wikimedia.org/T108581#1523982 (10Andrew) a:5Andrew>3mark [14:12:58] YuviPanda, shinken shows an HTTP 502 [14:33:03] Krenair: YuviPanda is on holidays... [14:33:15] not sure who else has access [14:34:48] Krenair: https://wikitech.wikimedia.org/wiki/Nova_Resource:Shinken [15:21:30] (03PS1) 10Giuseppe Lavagetto: Adding ganglia fake key [labs/private] - 10https://gerrit.wikimedia.org/r/230557 [15:22:14] (03CR) 10Giuseppe Lavagetto: [C: 032] Adding ganglia fake key [labs/private] - 10https://gerrit.wikimedia.org/r/230557 (owner: 10Giuseppe Lavagetto) [15:22:23] (03CR) 10Giuseppe Lavagetto: [V: 032] Adding ganglia fake key [labs/private] - 10https://gerrit.wikimedia.org/r/230557 (owner: 10Giuseppe Lavagetto) [15:33:16] andrewbogott: can you add valhallasw`cloud and Krenair to the shinken project? Tim L too, for good measure... :) [15:34:14] YuviPanda: hey, go back on vacation ;-) [15:34:46] valhallasw`cloud: I'm not on vacation till tomorrow I think. [15:34:54] ah, okay :-) [15:51:33] YuviPanda: yep, one moment... [15:54:32] YuviPanda: as admins? [15:55:53] valhallasw`cloud: you should have access now [15:56:01] andrewbogott: yup [15:56:35] YuviPanda: ok, done. btw, I’d appreciate a review of https://gerrit.wikimedia.org/r/#/c/229458/ before you depart :) [15:57:06] andrewbogott: aaah yes I'll do so in an hour or so [15:57:14] I'm working today :) [15:57:34] And tomorrow except tomorrow its from Oslo [15:58:27] Oh man, once the sun stops going down your productivity is going to skyrocket! [15:59:38] Going up you mean [15:59:46] But I'll be in Norway for only 8h [16:00:32] Hm? Isn’t it summer in Norway? [16:01:14] andrewbogott: oh I mean I will be more productive if it goes down and stays down :) [16:01:21] But I won't be leaving the airport anyway [16:03:18] apparently everything shinken was broken [16:03:40] !log shinken scheduler, poller, reactionner, broker, receiver and arbiter were all down. service shinken start'ed [16:03:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Shinken/SAL, Master [16:03:57] ok, seems back alive [16:07:29] 6Labs, 10Tool-Labs: Use instances with resources customized for Tools use - https://phabricator.wikimedia.org/T96714#1524320 (10scfc) @Andrew, looking at your considerations regarding disk space on virtual nodes, is the required space on the virtual node defined by the possible sizes of the hosted instances or... [16:13:13] 6Labs, 10Tool-Labs: Use instances with resources customized for Tools use - https://phabricator.wikimedia.org/T96714#1524327 (10Andrew) Best I can tell, the nova scheduler isn't very smart about this (partly because COW is a hack that we're using but isn't really understood by the upstream.) The effect, as I... [16:30:16] 6Labs, 10Tool-Labs, 3Labs-Sprint-108: Investigate the cause of puppet failures on Tools - https://phabricator.wikimedia.org/T107782#1524358 (10coren) 5Open>3Resolved I've examined the logs for the puppet failures and it does seem that the current causes are mostly genuine issues that should be looked in... [16:35:36] 6Labs, 3Labs-Sprint-107, 3Labs-Sprint-108: Setup monitoring and reporting for disk space usage of each project on NFS - https://phabricator.wikimedia.org/T106476#1524370 (10coren) I have a working script right now in my dev VM which is able to surface outliers in disk usage based on UID at very little I/O co... [16:37:52] andrewbogott: can you be valhallasw`cloud and tim's +2 puppet for the week? :) they basically have full +2 on the toollabs module, just need someone to press buttons... [16:38:15] yep :) [16:38:26] andrewbogott: awesome! [16:38:41] andrewbogott: usually I do it only when they are around just in case it requires revert [16:38:55] oh, yeah, we should discuss those logistics with scfc [16:39:05] andrewbogott: thank ls :) [16:39:09] valhallasw`cloud: yeah... [16:39:15] Yeah, I’ll most likely only merge in response to a specific ping, but I’m happy to be pinged. [16:39:24] ok, great :-) [16:39:32] Cool :) [16:40:12] 6Labs, 6operations, 3Labs-Sprint-102, 3Labs-Sprint-103, and 5 others: Reinstall labstore1001 and make sure everything is puppet-ready - https://phabricator.wikimedia.org/T107574#1524393 (10coren) Labstore1001 has been reinstalled (and is pristine from the puppet manifest) and all I/O tests are fine, but th... [16:42:17] 6Labs, 6operations: Investigate whether to use Debian's jessie-backports - https://phabricator.wikimedia.org/T107507#1524403 (10coren) [16:42:19] 6Labs, 6operations, 3Labs-Sprint-102, 3Labs-Sprint-103, and 5 others: Reinstall labstore1001 and make sure everything is puppet-ready - https://phabricator.wikimedia.org/T107574#1498287 (10coren) [16:43:05] 6Labs, 3Labs-Sprint-107, 3Labs-Sprint-108: Switch NFS server back to labstore1001 - https://phabricator.wikimedia.org/T107038#1524404 (10coren) A draft of the planned announcement is on etherpad at https://etherpad.wikimedia.org/p/labs-maintenance-aug-2015-draft for comments and adjustments. This is pending... [16:43:56] 6Labs, 6operations, 3Labs-Sprint-102, 3Labs-Sprint-103, and 5 others: Reinstall labstore1001 and make sure everything is puppet-ready - https://phabricator.wikimedia.org/T107574#1524417 (10coren) [16:43:58] 6Labs, 3Labs-Sprint-107, 3Labs-Sprint-108: Switch NFS server back to labstore1001 - https://phabricator.wikimedia.org/T107038#1524416 (10coren) [16:47:48] YuviPanda: what's the pupswat etherpad again? [16:48:18] oh, puppetswat [16:48:22] right on the second guess [16:48:43] :) [16:51:58] 6Labs, 6operations, 3Labs-Sprint-107, 3Labs-Sprint-108, 3ToolLabs-Goals-Q4: Investigate kernel issues on labvirt** hosts - https://phabricator.wikimedia.org/T99738#1524449 (10Andrew) Proposed reboot schedule here: https://wikitech.wikimedia.org/wiki/Virt_node_upgrade_schedule [17:12:40] 6Labs, 3Labs-Sprint-107, 3Labs-Sprint-108: Switch NFS server back to labstore1001 - https://phabricator.wikimedia.org/T107038#1524523 (10coren) The (more) detailed plan: * Coordinate with @cmjohnson as switching requires moving cabling around At the selected window: * power down 1001 * `service nfs-kernel-... [17:48:57] 6Labs, 3Labs-Sprint-107, 3Labs-Sprint-108, 3Labs-Sprint-109: Evaluate kubernetes for use on Tool Labs - https://phabricator.wikimedia.org/T107993#1524702 (10yuvipanda) [17:50:27] 6Labs, 3Labs-Sprint-109, 7Monitoring, 5Patch-For-Review: Monitor nova services - https://phabricator.wikimedia.org/T90784#1524711 (10Andrew) [17:50:50] 6Labs, 3Labs-Sprint-109: Make a menu of potential new labs features, invite comments from users - https://phabricator.wikimedia.org/T101769#1524714 (10Andrew) [17:51:36] 6Labs, 3Labs-Sprint-109, 5Patch-For-Review: Make a fact for project_id on labs instances - https://phabricator.wikimedia.org/T93684#1524715 (10Andrew) (This is almost totally done, needs verification before closing) [17:53:59] 6Labs, 3Labs-Sprint-105, 3Labs-Sprint-108, 3Labs-Sprint-109, 5Patch-For-Review: Archive NFS data for projects that no longer have NFS - https://phabricator.wikimedia.org/T104857#1524721 (10yuvipanda) [17:54:06] 6Labs, 3Labs-Sprint-108, 3Labs-Sprint-109, 5Patch-For-Review: Simple method to have a per-project debian repository - https://phabricator.wikimedia.org/T104194#1524722 (10yuvipanda) [17:54:31] 6Labs, 3Labs-Sprint-107, 3Labs-Sprint-108, 3Labs-Sprint-109: Setup monitoring and reporting for disk space usage of each project on NFS - https://phabricator.wikimedia.org/T106476#1524725 (10coren) [17:54:33] 6Labs, 3Labs-Sprint-108, 3Labs-Sprint-109: Have checkpoint checks for all labs services (Tracking) - https://phabricator.wikimedia.org/T107058#1524726 (10coren) [17:54:35] 6Labs, 3Labs-Sprint-107, 3Labs-Sprint-108, 3Labs-Sprint-109: Switch NFS server back to labstore1001 - https://phabricator.wikimedia.org/T107038#1524727 (10coren) [17:55:25] 6Labs, 10Incident-20150617-LabsNFSOutage, 3Labs-Q4-Sprint-2, 3Labs-Sprint-100, and 4 others: Disable LDAP and enable admin puppet module on labstore100[12] - https://phabricator.wikimedia.org/T95559#1524735 (10coren) [17:55:52] 6Labs, 10Incident-20150617-LabsNFSOutage, 3Labs-Q4-Sprint-2, 3Labs-Sprint-100, and 3 others: Disable LDAP and enable admin puppet module on labstore100[12] - https://phabricator.wikimedia.org/T95559#1194445 (10coren) [17:56:05] 6Labs, 3Labs-Q4-Sprint-1, 3Labs-Q4-Sprint-2, 3Labs-Q4-Sprint-4, and 2 others: Labs NFSv4/idmapd mess - https://phabricator.wikimedia.org/T87870#1524739 (10coren) [17:58:28] 6Labs, 3Labs-Q4-Sprint-1, 3Labs-Q4-Sprint-2, 3Labs-Q4-Sprint-4, 3ToolLabs-Goals-Q4: Labs NFSv4/idmapd mess - https://phabricator.wikimedia.org/T87870#1524747 (10coren) [18:11:06] 6Labs, 10Tool-Labs, 6Design Research Backlog, 6Learning-and-Evaluation, 6Research-and-Data: Organize a (annual?) toollabs survey - https://phabricator.wikimedia.org/T95155#1524807 (10leila) [18:45:14] 6Labs, 3Labs-Sprint-107, 3Labs-Sprint-108, 3Labs-Sprint-109: Evaluate kubernetes for use on Tool Labs - https://phabricator.wikimedia.org/T107993#1524938 (10yuvipanda) Proxying between pods is already managed by services, no? What we need is a https terminator / http proxy that does tools.wmflabs.org/ 6Labs, 10Tool-Labs: Set up A-based SPF for tools.wmflabs.org - https://phabricator.wikimedia.org/T104733#1524976 (10coren) a:3coren [18:57:43] 6Labs, 3Labs-Sprint-107, 3Labs-Sprint-108, 3Labs-Sprint-109: Evaluate kubernetes for use on Tool Labs - https://phabricator.wikimedia.org/T107993#1524983 (10coren) >>! In T107993#1524938, @yuvipanda wrote: > Not sure if we want to put the SSL cert in a pod? I, on the other hand, am quite certain we absolu... [18:58:48] 6Labs, 3Labs-Sprint-107, 3Labs-Sprint-108, 3Labs-Sprint-109: Evaluate kubernetes for use on Tool Labs - https://phabricator.wikimedia.org/T107993#1524988 (10yuvipanda) Indeed, so we'll need a proxy / ssl termination instance. It'll probably be a lot simpler than our current setup of course since we can jus... [18:59:18] 6Labs, 3Labs-Q4-Sprint-1, 3Labs-Q4-Sprint-2, 3Labs-Q4-Sprint-4, 3ToolLabs-Goals-Q4: Labs NFSv4/idmapd mess - https://phabricator.wikimedia.org/T87870#1525003 (10faidon) What's the status of this? [19:09:22] 6Labs, 3Labs-Q4-Sprint-1, 3Labs-Q4-Sprint-2, 3Labs-Q4-Sprint-4, 3ToolLabs-Goals-Q4: Labs NFSv4/idmapd mess - https://phabricator.wikimedia.org/T87870#1525031 (10coren) It's high on my radar but Mark asked me to priorize other things (I had just brought it up this week for the sprint). I'll still try to... [19:39:10] YuviPanda: should we disable the queues for the exec hosts that will be restarted next wednesday? [19:39:22] wait, that's in two days [19:39:24] the answer is yes [19:39:34] valhallasw`cloud: yes [19:39:44] valhallasw`cloud: coordinate with andrewbogott? [19:39:52] will do [19:40:15] Thanks for taking care of that :) [19:40:36] andrewbogott: I will disable the queues now, will check for running non-continuous jobs tomorrow and send an e-mail about them, and then we can reboot safely on wednesday [19:41:00] valhallasw`cloud: do we have enough capacity to live without them for two days? I figured disabling them would be more of a last-minute thing [19:41:18] (Also, note that I’m going to reboot labvirt1002 on Thursday, and labvirt1003 on Friday…) [19:41:21] andrewbogott: the problem is that it's hard to see what is a long-running job and what not [19:41:25] hrm. [19:44:09] valhallasw`cloud: unless you think one per day is too frequent? [19:46:15] I'm trying to come up with a smart solution [19:46:55] for the long term, I'm thinking we might want different queues with different guarantees. One 'week' queue that will be disabled a week before planned downtime, one 'day' queue that will be disabled a day before planned downtime [19:46:59] We could spread then out more... [19:47:06] The reboots that is [19:47:19] it doesn't matter much, though, for long-running jobs [19:47:21] Yeah, I’m happy to slow down the reboots if it will be less disruptive. [19:47:29] I can try to do some stats on that [19:47:42] Are long-running jobs ever really needed, or are they just examples of poor design? [19:48:10] it's a bit of both. Most of them would probably be restartable (e.g. interwiki bots) [19:48:23] or irc bots [19:48:33] irc bots should be in the continuous queue anyway [19:54:14] andrewbogott: I suppose we can also do t-24h disable queue, t-23h send email about still-running (non-continuous) jobs, t-1h restart continuous jobs, t restart [19:54:37] I'm preparing a data file to build some graphs, but it's already 200M in just wall clock times O_o [19:55:06] 8.5M entries and counting :D [19:55:15] Is it easy to corellate running jobs with email addresses? [19:55:21] yes [19:55:45] just username@tools.wmflabs.org [19:55:54] and the exim config will take care of it [19:57:04] I think the 24-23 hour thing is probably good enough. The case 'how can I start a job that takes > 24h' is something we can address later [19:57:26] I think the main answer there is 'make sure it listens to signals and saves state to a file where it can recover when restarted' [19:58:08] ok, I think this plan sounds good. [19:58:31] But maybe you can write out the steps for me so I’m not relying on you being here day after day :) [19:59:25] valhallasw`cloud: this answer is … not want i want to hear >.> [20:07:45] gifti: what's the issue specifically? [20:08:12] andrewbogott: that's assuming I know the steps >_> but I'll try to write some stuff down. It's mostly https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Admin#Draining_a_node_of_Jobs [20:08:50] writing scripts that save their own state seems hard to do [20:08:56] Yeah, drain/reschedule I know, but the detect-noncontinuous-jobs-and-email-owners seems harder... [20:09:23] Can email originate from a tools node? [20:09:26] yes [20:09:35] that part should be ok. parsing qstat is harder [20:11:01] gifti: it is in some cases, yes, but there's also limits as to how much continuous uptime we can guarantee [20:12:26] (I'm not sure what's reasonable there. 24h sounds short, 7 days on the long side?) [20:28:45] valhallasw`cloud: ive got a tool for that [20:31:08] valhallasw`cloud: http://tools.wmflabs.org/betacommand-dev/cgi-bin/sge_status.py [20:37:46] 6Labs, 3Labs-Sprint-109, 5Patch-For-Review: Make a fact for project_id on labs instances - https://phabricator.wikimedia.org/T93684#1525408 (10Andrew) ok, the fact is in place and works. It depends on the insertion of tenant_id into metadata by sink_nova_ldap. In Libery and later, metadata will provide pro... [20:39:01] 6Labs, 3Labs-Sprint-109, 5Patch-For-Review: Make a fact for project_id on labs instances - https://phabricator.wikimedia.org/T93684#1525412 (10yuvipanda) Can we also then get rid of the ldap variable from wikitech puppet groups? [20:40:20] 6Labs, 3Labs-Sprint-109, 5Patch-For-Review: Make a fact for project_id on labs instances - https://phabricator.wikimedia.org/T93684#1525417 (10Andrew) ...I think so? But I'm confused by the phrase 'wikitech puppet group' [20:44:33] YuviPanda: if you have a moment (and I know you do!) can you grep instanceproject modules/shinken/files/shinkengen [20:45:10] and either s/instanceproject/labsproject or suggest how I could do that? [20:45:15] Maybe it’s trivial, I haven’t really read that code [20:45:32] andrewbogott: looking [20:46:09] andrewbogott: ugh, it requires LDAP variable to be present to work... [20:46:28] will we keep populating the LDAP variable? [20:46:46] Does that script run on the instance that we want to know the project for? [20:46:50] Or on an aggregator? [20:47:24] We can keep populating it, but it sucks to have the same value declared in two places... [20:47:26] andrewbogott: it's an aggregator. [20:47:39] andrewbogott: it can be rewritten to use the wikitech API [20:47:58] Does it run on a prod box or in labs? [20:47:59] andrewbogott: but that's a bit lame, since then it'll hit the wikitech API and then LDAP [20:48:01] andrewbogott: labs [20:48:09] dammit [20:48:33] ok, well, I’ll open a bug and leave it to rot for eternity then [20:48:54] andrewbogott: heh, I can rewrite it in a couple of weeks. [20:49:02] don’t rewrite it to use the wikitech api, I hate that too [20:49:05] oh [20:49:15] everything shoult just talk to nova [20:49:19] novaobserver! [20:49:21] which… blah blah blah easy in kilo [20:49:30] this bug, like every bug, is blocked by the network upgrade [20:49:38] which is why I bring that up every damn week [20:49:58] +1 [20:50:46] 6Labs, 3Labs-Sprint-109, 5Patch-For-Review: Remove reliance on ldap $::projectid from shinkengen - https://phabricator.wikimedia.org/T108625#1525456 (10Andrew) 3NEW a:3Andrew [20:51:10] 6Labs, 10Labs-Infrastructure: Give 'novaobserver' keystone account rights to read everything, everywhere, write or change nothing - https://phabricator.wikimedia.org/T104588#1525474 (10Andrew) [20:51:10] 6Labs, 3Labs-Sprint-109, 5Patch-For-Review: Remove reliance on ldap $::projectid from shinkengen - https://phabricator.wikimedia.org/T108625#1525456 (10Andrew) [20:51:25] andrewbogott: sorry I can't help much with the network stuff. [20:51:51] I should’ve just gone ahead and caused the outage a month ago :) [20:52:12] heh [20:54:07] Betacommand: thanks [21:06:25] 6Labs, 10Tool-Labs, 10Tool-Labs-tools-Global-user-contributions, 10Labs-Infrastructure, and 3 others: meta_p.wiki table corrupt (contains many NULL entries for 'url' field) - https://phabricator.wikimedia.org/T106897#1525506 (10Joergens.mi) will there be any fix soon? [21:11:10] 6Labs, 10Tool-Labs: Use instances with resources customized for Tools use - https://phabricator.wikimedia.org/T96714#1525533 (10scfc) 5Open>3declined a:3scfc Okay, then indeed it doesn't make sense to save on "potential" space for Tools instances. Thanks for the explanation. [21:16:24] 6Labs, 10Tool-Labs, 10Tool-Labs-tools-Global-user-contributions, 10Labs-Infrastructure, and 3 others: meta_p.wiki table corrupt (contains many NULL entries for 'url' field) - https://phabricator.wikimedia.org/T106897#1525559 (10Krenair) This was already fixed: ```MariaDB [meta_p]> select * from wiki where... [21:25:58] andrewbogott: /home/valhallasw/mailer/runningtasks.py [21:26:14] I'll send an e-mail about the jobs [21:45:10] valhallasw`cloud: the email will explain about the script too? [21:46:07] Sorry, i meant the email I just sent to labs-l [21:52:08] valhallasw`cloud: labvirt1009 has already been rebooted. So we could create an exec node there to hold long-lived jobs. [21:53:46] Yes, but we'd need to create a specific queue so jobs get sent there specifically :/ [21:54:11] Not sure how to do that. Coren should ;) [21:54:25] * valhallasw`cloud is off to bed - good night [22:00:31] YuviPanda: Still around? [22:00:56] Coren: sup [22:01:01] * YuviPanda is in Bart [22:08:55] Ah. When you get in a better spot, I'd appreciate you looking at the timer changeset for the backups so I can close that whole thing. (You're in the reviewers) [22:15:21] Coren: ah ok [22:15:28] Coren: I'll try doing so before takeoff [22:16:40] If not, I'll just rope someone else for it, but you already know what it's about. [22:16:53] Coren: yeah, I'll take a look. :) [22:17:00] I'll be working from Stockholm tomorrow [22:17:14] That's gonna be fun timezonewise [22:30:22] YuviPanda: airplane internet ftw [22:30:58] Not sure if the type of airlines I usually fly have such things [22:31:24] same here and it costs a small fortune [23:10:30] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Lt910001 was created, changed by Lt910001 link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Lt910001 edit summary: Created page with "{{Tools Access Request |Justification=Proposing and making (with consensus) code changes to VeblenBot per WP en requests accepted by the two current maintainers |Completed=fal..." [23:15:00] 6Labs, 3Labs-Sprint-109, 5Patch-For-Review: Make a fact for project_id on labs instances - https://phabricator.wikimedia.org/T93684#1526060 (10scfc) AFAIU `operations/puppet`'s `modules/shinken/files/shinkengen`, #Shinken uses the LDAP data to get a list of instances for a project, so that needs to be update...