[01:20:39] 3Wikimedia Labs / 3Infrastructure: Replica MySQL: Wiki ViewStats databases completely missing! - 10https://bugzilla.wikimedia.org/71043 (10Sean Pringle) a:3Sean Pringle [02:42:52] 3Wikimedia Labs / 3tools: Rows missing from enwiki_p on s1, c1 - 10https://bugzilla.wikimedia.org/71084#c1 (10Brad Jorsch) Since it seems we're moving from "s" names to "c", I tested those too: tools.anomiebot@tools-login:~$ echo 'select * from enwiki_p.page where page_id in (43871050,43874575);' | mysql -... [04:28:48] so... [04:29:34] does editing the members of a project's projectadmins usually update the SMW template automatically? [04:30:52] i have an idea [04:34:24] Coren: 31.3 seconds [04:35:35] (was writing in #-operations before and I thought I was here so 31.3 is a followup on that) [04:54:24] 3Wikimedia Labs / 3wikitech-interface: add [[wikitech:Release Engineering/SAL]] to [[wikitech:mediawiki:sidebar]] - 10https://bugzilla.wikimedia.org/71165 (10jeremyb) 3NEW p:3Unprio s:3normal a:3Marc A. Pelletier add [[wikitech:Release Engineering/SAL]] to [[wikitech:mediawiki:sidebar]]. [04:59:56] huh [05:11:22] (03PS1) 10Jeremyb: add OpenStackManager component to #wikimedia-labs [labs/tools/pywikibugs] - 10https://gerrit.wikimedia.org/r/162201 [05:11:41] legoktm: YuviPanda|afk: ^^ [12:35:55] 3Wikimedia Labs / 3(other): hikebikemap utf8 miscoding - 10https://bugzilla.wikimedia.org/71173 (10Peter Gervai (grin)) 3NEW p:3Unprio s:3critic a:3None Created attachment 16551 --> https://bugzilla.wikimedia.org/attachment.cgi?id=16551&action=edit http://tiles.wmflabs.org/hikebike/12/2263/1423.png... [14:35:01] I'm seeing what may be some corruption on the enwiki_p.pagelinks table, possibly related to the recent database shenannigans. Has anyone got a minute or two to double-check for me before I raise a ticket? [14:35:20] "corruption"? [14:35:35] And no, enwiki was basically completely unrelated to any of this. [14:36:35] select * from pagelinks where pl_namespace = 0 and pl_title = 'And_Now_The_Screaming_Starts!' shows a record which (unless I am mistaken) does not reflect the live enwiki. This looks very much like the results of replication prblems on the old toolserver. [14:38:54] Hm, you're right that that row doesn't seem to be there in production. [14:39:29] Okay thanks - labs has been so reliable to date that I was doubting myself. Will ticket it. [14:40:16] I know Sean is tracking down an issue with some queries causing OOM on the database that can create small desyncs like this; the extra data point may end up being useful. [14:43:57] Coren: Now that virt1006 is free of instances, what shall we do with it? It's easy to do a fresh OS install, but I wonder if we're better off just scrapping it, since Rob is getting quotes for modern hardware anyway... [14:44:52] andrewbogott: I'd have like to reinstall and test given time and availability. Losing a whole host hurts even if we get new ones later. [14:45:38] Does it hurt more than a few hundred dollars in hardware costs? [14:45:44] Well, $1000 [14:45:44] andrewbogott: But the important auxilary question is "do we actually have the time to test for reals?" [14:46:12] 3Wikimedia Labs: Discrepancy between enwiki_p.pagelinks on labs and production - 10https://bugzilla.wikimedia.org/71176 (10tb) 3UNCO p:3Unprio s:3normal a:3None Intention: Was tracking down a problem with one of my tool-labs tools and found a discrepancy in the enwiki_p database. Steps to Reproduce: t... [14:46:13] Coren: Since I never had a clear understanding of what was wrong or a clear path to reproduce the issues… I think the answer is no :( [14:46:30] andrewbogott: I'd have expected those boxen to be more expensive than that. If the amount is that (relatively) low, then I agree that scrap and order one more is the wiser choice. [14:46:46] OK -- at the very least we can wait and see what the bids are for new hardware. [14:47:12] I know that if we bought a new virt1006 today it would cost about a grand. But that may include a steep discount due to obsolescence. [14:56:00] Coren: Is c3.labsdb supposed to have all wikis available on it? [14:56:11] anomie: It should. [14:56:40] Coren: enwiki_p is empty on c3 [14:58:09] Coren: Also, what tb__ was mentioning earlier may have the same underlying cause as whatever is causing bug 71084. [14:59:18] anomie: Oh! So it is. [14:59:24] * Coren grumbles. [14:59:33] It's the views that are out of date, enwiki is. [15:00:39] ahhh... maybe not. Perhaps sean forgot to turn s1 replication on. [15:01:11] I see c3 has all the databases that used to be on the individual slices, not the other ones. [15:01:45] So, for the moment, only use c3 for the databases actually pointed to it. [15:02:48] (enwiki.labsdb points at c1, where enwiki_p works as expected) [15:04:24] Coren: I suppose we need to wait for springle to figure out why rows are missing from enwiki on c1 but not c2? [15:04:59] Oh, I'm pretty sure the why is "c3 was merged but replication of 'everything else' to it hasn't been started yet". [15:05:13] Oh, missing rows. [15:05:45] That's a known issue atm; some queries cause the DB to go OOM and miss some replication statements. It's not a *big* effect but he's working on a fix. [15:06:34] And intermediate patch has been/will shortly be deployed that sets a cap memory used by user queries to circumvent the issue. [15:16:26] jeremyb: Eh. I would +2 and deploy, but somehow I've lost +2 rights for that repo… [15:18:07] wikibugs? [15:18:37] I don't think I even added you as a reviewer? [15:18:51] so I wonder where you saw it :) [15:19:00] andrewbogott: wanna help a bunch of small changes? [15:19:08] https://gerrit.wikimedia.org/r/#/q/owner:%22Yuvipanda+%253Cyuvipanda%2540gmail.com%253E%22+project:operations/puppet+status:open,n,z [15:19:28] hmm [15:19:36] listing order not in dependency order :( [15:19:40] need to merge https://gerrit.wikimedia.org/r/#/c/161952/ first [15:29:04] YuviPanda: Have you run some tests on that modularization? Or shall I do that? [15:29:19] andrewbogott: I ran them on shinken and puppet is ok [15:29:28] ok [15:29:52] andrewbogott: all these patches don't do anything different other than move files around, and the new location is set as cfg_dir in icinga.cfg as well (and already has merged patches putting things there) [15:29:54] so should be ok [15:36:08] Hello, I am searching for somebody I could ask about the grid engine [15:36:37] guenthermi: Plenty of people here who can help. What's your issue? [15:38:10] I tried to submit a job which needs to run the java vm. But there eccuring errors everytime. because there is not enough heap storage to run it. But that should not be the problem, because there is [15:38:33] If I run free there seems to be enough storage [15:39:20] I tried to reduce the inital heap storage to 16mb but it dosn't help. I can not even run java -v [15:39:38] guenthermi: Jobs have a fairly conservative default maximum memory. See https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Allocating_additional_memory [15:40:45] https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Allocating_additional_memory also has some extra hints for Java; in particular jamvm helps a lot. [15:41:04] I already tried out this [15:42:05] Can you show me what commandline you are using, exactly? [15:42:26] yes, but wait a minute [15:44:08] stdout: "Error occurred during initialization of VM Could not reserve enough space for object heap", stderr: "Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit." [15:44:30] I need your command line, not the error output. :-) [15:44:48] sorry, what do you mean with jamvm? [15:45:57] It's a different VM handler for java that's less of a memory hog; but lemme see your commandline first. :-) [15:48:22] "jsub java -version" I also tried to do this in an sh script. and tried out to add "-mem 2000m" and "-Xms16m" to reduce the heap size [15:50:38] First I tried to run a java helloworld program. After it does not work, I used java -version, which comes to the same error. [15:50:39] Well, "jsub java -version" will definitely not work; the default VM for java needs no less than 3.2G to start up by default. (Yes, it's kinda insane). With -jamvm, you can reduce this so that an -Xms and -Xmx of 300M only needs a -mem 750m. The default VM has about 1G of overhead, and starts with -Xmx at, iirc, 2G [15:54:12] I think -jamvm does not working ("Error: missing `jamvm' JVM at `/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/jamvm/libjvm.so'. Please install or use the JRE or JDK that contains these missing components.") [15:54:30] Oh, hm. It's no longer installed by default. Interesting. [15:54:37] Give me a sec. [15:57:27] Try that again? [16:02:07] ok thank you, its working :) [16:10:30] aude: are you involved in the upcoming Amsterdam hackathon? Either in attending it or organizing? [16:12:31] andrewbogott: possibly [16:13:44] aude: I'm wondering if they need a Labs and/or Vagrant support person to attend or on call during the event or if that's already covered. [16:14:15] really a question for multichill [16:14:16] aude: you already have rights to give out shell accounts and such on Labs, right? [16:14:22] i don't think so [16:14:58] aude: would you like to? Since you're on NDA it might make sense to just designate you as the labs liason if Coren or I can't make it. [16:15:03] sure [16:15:04] * bd808 perks up ears at hackathon+vagrant [16:15:08] :D [16:15:28] aude: ok. Will figure out first if one of us has budget to show up in person :) [16:15:38] we'll want a wikibase+commons role or something in vagrant [16:15:42] for structured data [16:16:00] maybe want that done before hackathon [16:16:03] I built a server in labs to generate the usb stick image for wikimania but haven't updated it since [16:16:08] andrewbogott: cool [16:16:21] aude: We don't have that combo yet? [16:16:39] afaik, the hackathon will be more experienced people and limited number [16:16:57] aude: what's a good way to contact multichill? mail/irc/talk page? (I don't see them in #wikimedia-tech) [16:17:02] irc [16:17:08] or mail [16:17:59] aude: Doesn't `vagrant enable-role wikidata commons` setup commons with wikibase? [16:18:31] Or do we still have the interwiki links issue [16:18:51] bd808: probably more settings needed [16:18:56] like the correct namespace [16:19:21] e.g. we can't occupy the main namespace with items :) [16:19:36] I don't know what the setup needs to be, but if you run into problems wedging it into mwv I can help with that aspect [16:20:14] * bd808 has a boss who is excited to play with wikidata in mwv [16:20:49] speaking of which, i have a couple patches to submit for vagrant [16:21:04] andrewbogott: Given the usual pattern to date, I expect I'll be shipped there. [16:21:21] andrewbogott: Wait, "upcoming"? This fall? [16:21:44] yeah -- the 'normal' hackathon is in Paris in the spring, but there's a little wikidata thing in November in ams [16:21:53] glam/wikidata [16:22:04] Anyway, I'll write them and ask, will cc you [16:22:12] kk [16:24:05] Do we know it's in Paris or is that still an hypothetical? The only news I heard was that wmit might propose a hackaton. [16:24:28] Yeah, the spring one is still tentative. [16:24:43] Just trying to clarify that this thing in November is a niche event, not the main annual thing [16:27:24] andrewbogott: labs_vagrant is broken because of the changes I got you to merge last week. :( I have a fix in https://gerrit.wikimedia.org/r/#/c/162160/ but getting a +1 from Ori would probably be good. [16:27:39] bd808: funny, it worked for me. Does it break existing installs? [16:28:30] There's a battle between ops/puppet and the mwv code. The perms on /srv/vagrant end up with all "others" access bits removed [16:28:51] This breaks the wiki badly as www-data can't read the php files [16:29:15] bd808: https://gerrit.wikimedia.org/r/#/c/162284/ [16:29:25] makes the client wiki be able to access the repo [16:29:41] i would like 'enwiki' to be configurable at some point, but baby steps :) [16:31:00] oh, of course I boldly say 'works for me' but totally didn't try to access the wiki [16:31:05] aude: Cool. That patch needs a tiny bit of work to play nicely with hiera vars. That could let you change the siteGlobalID too. I can touch it up and merge this evening. [16:31:21] andrewbogott: ha. Yeah. It provisions but then melts at runtime. [16:31:46] merged [16:31:48] bd808: yay [16:32:13] * aude needs to read about hiera [16:32:25] are there docs on wikitech or some place? [16:32:47] not yet [16:33:55] bd808: and i needed https://gerrit.wikimedia.org/r/#/c/162285/ for debugging in vagrant [16:34:05] * aude realizes it was removed at some point [16:35:28] aude: That's a good one. Memcached was removed to trim down the runtime memory footprint, but I totally see the need to use it for some debugging stuff. [16:35:50] yep [16:36:02] we tend to have issues with memcached, as in last deployment [16:36:04] The hiera docs thing for MWV is "on my list" but unlikely to get fixed any time soon. [16:36:24] and then debug in combo with hhvm, vagrant is the best option [16:38:44] The short version of hiera and mwv is that there are yaml files in puppet/hieradata which are used to fill in parameters when classes are applied. There is a resolution order for each lookup that allows local customizations of the defaults. This replaces a lot of the configuration that would have happened in a "role" class previously. [16:39:05] probably works for me to look there and see what it does [16:39:49] This doc is a pretty good overview -- https://docs.puppetlabs.com/hiera/1/puppet.html [16:40:21] "Automatic Parameter Lookup" is the magic we are using [16:40:25] bd808: also https://gerrit.wikimedia.org/r/#/c/149901/ was super helpful [16:40:31] for using hhvm debugger [16:40:41] and the explanation on how [16:40:54] should be on wikitech or mediawiki.org [16:41:55] Yeah. That probably belongs on https://www.mediawiki.org/wiki/MediaWiki-Vagrant/Advanced_usage or something [16:42:00] yep [16:42:04] i see there is hhvmsh also [16:42:11] only just noticed [16:42:57] Yeah. dirty hack I whipped up [16:44:21] i'm not quite sure how to use with gdb though [16:44:39] * aude wants to investigate the class alias issue we have [16:44:51] in addition to killing them in wikibase [16:46:43] or need to do http://stackoverflow.com/questions/22781124/debugging-hhvm-segmentation-faults [17:00:39] aude: ori may have some useful notes about using gdb to trace through hhvm dumps. He's been doing quite a bit of that. [17:01:37] ok [17:01:58] i think what's suggested in the stackoverflow is what to try next [17:02:08] and am looking at the test cases [17:02:42] thing is that it seems to work but then is a memory leak perhaps [17:08:54] 3Wikimedia Labs / 3(other): hikebikemap utf8 miscoding - 10https://bugzilla.wikimedia.org/71173 (10Andre Klapper) 5NEW>3UNCO [17:08:54] 3Wikimedia Labs / 3(other): hikebikemap utf8 miscoding - 10https://bugzilla.wikimedia.org/71173#c1 (10Andre Klapper) Created attachment 16557 --> https://bugzilla.wikimedia.org/attachment.cgi?id=16557&action=edit Screenshot Cannot reproduce on http://www.hikebikemap.org/ as the name "Bernecebaráti" is dis... [17:10:37] 3Tool Labs tools / 3[other]: hikebikemap utf8 miscoding - 10https://bugzilla.wikimedia.org/71173 (10Andre Klapper) s:5critic>3major [17:37:26] !log Initialized bloom cache on betalabs, enabled it, and populated it for enwiki [17:37:26] Initialized is not a valid project. [17:37:42] !log deployment-prep Initialized bloom cache on betalabs, enabled it, and populated it for enwiki [17:37:44] Logged the message, Master [17:45:10] AaronSchulz: you're not in #-multimedia? [17:45:48] no [17:46:33] I know :) [17:47:08] →-dev then I guess [18:13:43] Coren: thanks for the detailed report on the db issues :) I got a bit wiser [18:18:05] I tried to make it also not boring. :-) [18:23:41] Coren: curious, is ldaplist broken for you again? [18:24:06] andrewbogott: Not on tools-login at least. [18:24:13] hm, works for eqiad but not for codfw. [18:24:25] I wonder if privs aren't replicated...? [18:24:32] That would _suck_ [18:27:23] hm, nope, the priv is there [18:55:40] hullo [18:56:06] o/ [18:56:35] Is there a delay between the time you change your SSH key and the time you can use it to log in? [18:56:57] Yes, but it's very short - a minute or two at the most normally. [18:57:26] ok, thanks. I'll wait a bit and retry, then :) [18:58:00] In the meantime I'll create a tool account [19:01:29] Success! [19:02:04] Patience is a virtue, they say. :-P [19:02:08] héhé [19:02:45] Coren: So far I'm very impressed by how easy the process has been, and how fantastic the documentation is. [19:03:16] ... "fantastic"? It's a horrid tl;dr wall of text that could stand serious reorganization. Your standards are low. :-P [19:03:35] It's mostly all there, mind you, which is a good thing. [19:03:48] Coren: Are my questions answered? Definitely yes :) [19:04:25] Yeah, it's a bit unappealing, but it's very detailed, and I appreciate the step-by-step. [19:04:50] Well, I'm glad its functional at least. :-) [19:05:24] 3Wikimedia Labs / 3Infrastructure: Replica MySQL: Wiki ViewStats databases completely missing! - 10https://bugzilla.wikimedia.org/71043#c4 (10metatron) Database is now accessible through c3 again, but unusable because of a continuing lock. Waiting for table metadata lock | SELECT * FROM p50380g50769__wvs2... [19:07:42] Coren: While I have you, is it within accepted practices to use a couple GBs of space for a tool? [19:08:20] It's not ridiculous, though if you tell me what you're using it for I may be able to give you alternatives with better performance. [19:09:45] Coren: I have a Python script that generates static HTML files; and it generates many of them (a few thousands). [19:11:03] guillom: That's cool. You may want to be selective about backups though. https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help/Backups [19:11:27] Coren: Cool; I'll look :) Thanks for the help! [19:13:03] yeah, I can back up the important files, but I can re-generate the HTML files if all goes to hell [19:15:53] 3Wikimedia Labs / 3Infrastructure: need "agree to ToS" checkbox in the Labs createaccount page - 10https://bugzilla.wikimedia.org/48768#c4 (10Sumana Harihareswara) 5NEW>3RESO/FIX https://wikitech.wikimedia.org/w/index.php?title=Special:UserLogin&returnto=Main+Page&type=signup now includes a disclaimer re... [19:24:39] 3Wikimedia Labs / 3Infrastructure: add useful content to Wikitech:Shell - 10https://bugzilla.wikimedia.org/54697 (10Sumana Harihareswara) a:5Ryan Lane>3None [19:26:08] 3Wikimedia Labs / 3Infrastructure: add Central Logging Service documentation - 10https://bugzilla.wikimedia.org/54702 (10Sumana Harihareswara) a:5Ryan Lane>3None [19:26:23] 3Wikimedia Labs / 3Infrastructure: Encouraging people to do the right thing re bastion - 10https://bugzilla.wikimedia.org/54719 (10Sumana Harihareswara) a:5Ryan Lane>3None [19:34:55] 3Tool Labs tools / 3[other]: bring back missing-from-wikipedia - 10https://bugzilla.wikimedia.org/70199#c4 (10Sumana Harihareswara) a:5Sumana Harihareswara>3sumanah Thanks Daniel, and thank you Britta! I'll take a look in October; am changing the assignee to my volunteer account. [19:52:23] 3Wikimedia Labs / 3deployment-prep (beta): deployment-graphite.eqiad.wmflabs went away? - 10https://bugzilla.wikimedia.org/71031#c3 (10C. Scott Ananian) Ok, I've changed the statsd configuration from deployment-graphite.eqiad.wmflabs to labmon1001.eqiad.wmnet on both deployment-pdf01 and deployment-pdf02. F... [19:58:06] 3Wikimedia Labs / 3deployment-prep (beta): Mobile redirect goes to wrong domain name on beta labs - 10https://bugzilla.wikimedia.org/71079#c1 (10Greg Grossmeier) p:5Unprio>3High Reedy: can you take a look here? There's also https://bugzilla.wikimedia.org/show_bug.cgi?id=70145 which is "Safari sets forceH... [19:58:21] 3Wikimedia Labs / 3deployment-prep (beta): Safari sets forceHTTPS=deleted incorrectly, causing login failure on Beta Cluster - 10https://bugzilla.wikimedia.org/70145 (10Greg Grossmeier) [19:59:54] 3Wikimedia Labs / 3deployment-prep (beta): Mobile redirect goes to wrong domain name on beta labs - 10https://bugzilla.wikimedia.org/71079 (10Greg Grossmeier) [19:59:55] 3Wikimedia Labs / 3deployment-prep (beta): Mobile redirect goes to wrong domain name on beta labs - 10https://bugzilla.wikimedia.org/71079#c2 (10Greg Grossmeier) And https://bugzilla.wikimedia.org/show_bug.cgi?id=70948 "Beta Cluster isn't redirecting en.wikipedia.beta.wmflabs.org correctly" but I thought tha... [19:59:55] 3Wikimedia Labs / 3deployment-prep (beta): Beta Cluster isn't redirecting en.wikipedia.beta.wmflabs.org correctly - 10https://bugzilla.wikimedia.org/70948 (10Greg Grossmeier) [20:14:37] !log deployment-prep updated OCG to version 1cf9281ec3e01d6cbb27053de9f2423582fcc156 [20:14:40] Logged the message, Master [20:22:07] 3Wikimedia Labs / 3deployment-prep (beta): Determine first pass list of icinga-alerting data from graphite.wmflabs - 10https://bugzilla.wikimedia.org/70141#c17 (10Greg Grossmeier) 5PATC>3ASSI p:5High>3Normal a:3Yuvi Panda Yuvi: Thanks for the first pass work! Once you remove yourself from the list... [20:27:23] 3Wikimedia Labs / 3deployment-prep (beta): deployment-graphite.eqiad.wmflabs went away? - 10https://bugzilla.wikimedia.org/71031#c4 (10C. Scott Ananian) 5NEW>3RESO/FIX Seems like it's working now. [20:32:08] 3Tool Labs tools / 3[other]: hikebikemap utf8 miscoding - 10https://bugzilla.wikimedia.org/71173#c2 (10Peter Gervai (grin)) Have you tried to retrieve the specific tile I've shared? http://tiles.wmflabs.org/hikebike/12/2263/1423.png I actually tried to find a part which was not cached (Z20 somewhere around... [20:51:07] 3Tool Labs tools / 3[other]: hikebikemap utf8 miscoding - 10https://bugzilla.wikimedia.org/71173#c3 (10Peter Gervai (grin)) I am using the tile server with leaflet and not the referenced website. Still, try to find some tiles not yet generated since it is probably a recent problem (so most of the old cached... [21:29:07] 3Tool Labs tools / 3[other]: hikebikemap utf8 miscoding - 10https://bugzilla.wikimedia.org/71173#c4 (10Andre Klapper) (In reply to Peter Gervai (grin) from comment #2) > Have you tried to retrieve the specific tile I've shared? > > http://tiles.wmflabs.org/hikebike/12/2263/1423.png Yeah, but how would I "n... [21:36:53] 3Tool Labs tools / 3[other]: hikebikemap utf8 miscoding - 10https://bugzilla.wikimedia.org/71173#c5 (10Andre Klapper) 5UNCO>3NEW Uh. I see the problem now when going to http://tiles.wmflabs.org/osm/slippymap.html (e.g. with cities in Western Czech Republic) The problem seems related to the zoom level (a... [21:47:03] andrewbogott: can please get a new project on labs ? [21:48:05] matanya: sure… name? [21:48:11] video [21:48:36] i'm going to use it for intensive video re-encoding [21:49:11] matanya: done, please document: https://wikitech.wikimedia.org/wiki/Special:FormEdit/Nova_Project_Documentation/Nova_Resource:Video/Documentation [21:51:18] Thanks! done [22:08:23] 3Tool Labs tools / 3[other]: hikebikemap utf8 miscoding - 10https://bugzilla.wikimedia.org/71173#c6 (10Peter Gervai (grin)) Oh I quickly realised that you're not familiar with the OSM internals, sorry. :-) Thanks for trying to find someone involved. Probably something was upgraded which broke the strings, i... [22:53:19] !log integration Killed all pending jobs for deployment-bastion in Jenkins queue ("waiting for executor" bug/issue) [22:53:21] Logged the message, Master [22:55:36] !log integration Took deployment-bastion offline to try and reset master's view of it's workers [22:55:37] 3Wikimedia Labs / 3deployment-prep (beta): Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - 10https://bugzilla.wikimedia.org/70597#c7 (10Greg Grossmeier) Sept 23rd: 22:51 bd808: Jenkins stuck trying to update database in beta again with the dumb "waiting for executors" bug/pro... [22:55:38] Logged the message, Master [22:57:08] 3Wikimedia Labs / 3tools: Setup an icinga instance to monitor tools on tool-labs - 10https://bugzilla.wikimedia.org/51434#c7 (10Sumana Harihareswara) Good luck, Yuvi! [22:58:53] 3Wikimedia Labs / 3Infrastructure: Make instance creation failures more verbose - 10https://bugzilla.wikimedia.org/45483 (10Sumana Harihareswara) a:5Ryan Lane>3None [23:07:45] !log integration Jenkins and deployment-bastion talking to each other again after six (6!) disconnect, cancel jobs, reconnect cycles [23:07:47] Logged the message, Master [23:16:55] 3Wikimedia Labs / 3wikitech-interface: Not possible to delete files - 10https://bugzilla.wikimedia.org/71208 (10Tim Landscheidt) 3NEW p:3Unprio s:3normal a:3None Working through [[wikitech:Category:Page deletion queue]], for example, trying to delete [[wikitech:File:2-roller crusher.jpg]] (https://wi... [23:18:23] 3Wikimedia Labs / 3deployment-prep (beta): Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - 10https://bugzilla.wikimedia.org/70597#c8 (10Bryan Davis) (In reply to Bryan Davis from comment #6) > > I have manually changed the config for the beta-update-databases-eqiad job > in... [23:22:07] 3Wikimedia Labs / 3wikitech-interface: Not possible to delete files - 10https://bugzilla.wikimedia.org/71208#c1 (10Tim Landscheidt) * "Other/additional reason" = "also spam.".