[01:41:13] !ping [01:41:13] !pong [01:47:23] hmm, how to avoid that OperationalError: (2006, 'MySQL server has gone away') ? [01:48:23] phe: that might be due to the database migration? [01:48:32] you could always just catch it and try to reconnect [01:49:07] I keep the connection open for a long time too, that doesn't help surely [02:03:01] (03PS1) 10Legoktm: Initial commit [labs/tools/gblrenamemon] - 10https://gerrit.wikimedia.org/r/150754 [02:07:35] legoktm: lol :) [02:07:44] :P [02:07:58] Coren: I just created a new tool ("gblrenamemon") and there's no replica.my.cnf... [02:08:04] wait [02:08:04] legoktm: wait for a bit [02:08:06] there it is [02:08:10] lol [02:08:13] legoktm: it's a cron yo [02:08:17] Coren: sorry, I'll be less impatient :P [02:08:19] * YuviPanda sends legoktm a yo [02:08:23] haha [02:08:30] (03CR) 10Legoktm: [C: 032 V: 032] Initial commit [labs/tools/gblrenamemon] - 10https://gerrit.wikimedia.org/r/150754 (owner: 10Legoktm) [02:08:35] which reminds me, let me fix a couple of privacy policy issues [02:09:08] :P [02:09:08] legoktm: I'm doing that ip ignore thing on quarry, so I'll do it on extdist right after [02:09:16] * legoktm hugs YuviPanda [02:10:58] legoktm: :D [02:13:21] legoktm: oh, heh, logs don't record XFF by default, so as long as we don't expliclty record them, we're safe :) [02:13:32] oh great. [02:13:33] yay! [02:13:42] legoktm: :) [02:13:55] (03PS1) 10Legoktm: Fix typo [labs/tools/gblrenamemon] - 10https://gerrit.wikimedia.org/r/150755 [02:13:57] (03PS1) 10Legoktm: Add example crontab file [labs/tools/gblrenamemon] - 10https://gerrit.wikimedia.org/r/150756 [02:14:06] (03CR) 10Legoktm: [C: 032 V: 032] Fix typo [labs/tools/gblrenamemon] - 10https://gerrit.wikimedia.org/r/150755 (owner: 10Legoktm) [02:14:13] (03CR) 10Legoktm: [C: 032 V: 032] Add example crontab file [labs/tools/gblrenamemon] - 10https://gerrit.wikimedia.org/r/150756 (owner: 10Legoktm) [02:14:36] and it's running! [02:21:30] Coren: Im still seeing a lot of the tweetme spider activity [03:12:40] 3Wikimedia Labs / 3deployment-prep (beta): populateBacklinkNamespace script causing massive slave lag on beta - 10https://bugzilla.wikimedia.org/68349#c24 (10Kevin Israel (PleaseStand)) (In reply to Greg Grossmeier from comment #22) > It already happened in production. Which is the only reason why it was > m... [04:29:41] wmflabs services that access the database seem to be experiencing MySQL errors. (x-posted from #wikimedia-operations) [04:29:41] https://tools.wmflabs.org/sigma/editorinteract.py and other tools give errors like "Access denied for user 's51469'@'10.68.17.123' (using password: YES)" [05:08:13] !log deployment-prep HHVM hung on both boxes. Grabbed core and backtrace before restarting [05:08:15] Logged the message, Master [07:37:52] (03PS1) 10Hashar: Jenkins job validation (DO NOT SUBMIT) [labs/tools/gblrenamemon] - 10https://gerrit.wikimedia.org/r/150773 [07:38:13] (03Abandoned) 10Hashar: Jenkins job validation (DO NOT SUBMIT) [labs/tools/gblrenamemon] - 10https://gerrit.wikimedia.org/r/150773 (owner: 10Hashar) [07:39:46] legoktm: flake8 pass on gblrenamemon :-D [07:39:52] :D [07:59:44] hi guys: I need your help to run php bot on wikipedia. I have installed xampp which contains: Apache, mysql and php. but I dont know how to run php bot, will you please give me codes like ones used in python (like User-config.py, Login.py), Im new in php> Now I need to login to my account bot through php. Thank you [09:02:51] 3Tool Labs tools / 3Quentinv57's tools: SUL info tool fails with unicode usernames - 10https://bugzilla.wikimedia.org/65144#c1 (10YurB) Same problem here, cyrillic username in my case. [09:41:16] Tools are still unable to access databases [09:41:30] ERROR 1045 (28000): Access denied for user 'p50380g50692'@'10.68.16.7' (using password: YES) [09:44:23] anybody here who can address database access issue??? [10:06:54] springle: is inabiilty to access enwiki related to the MariaDB migration? [10:15:51] Coren? [10:15:55] petan? [10:16:13] russblau: checking [10:25:41] russblau: any change? [10:26:10] yes, i can connect now [10:26:30] tools seem to be working; thank you! [10:26:58] np [10:27:07] something wnet wrong with grants for p50380g50692__DPL [10:27:32] * springle pokes around [10:28:39] springle: i don't think it was just that tool; there's a report in the channel log from LFaraone a few hours ago with a similar issue [10:30:26] that report was before migration was completed [10:31:55] now I'm getting errors like 'PROCEDURE p50380g50692__DPL.ch1_prep_for_join does not exist'; did all stored procedures get wiped? [10:41:23] !log deployment-prep Taking gdb traces of hhvm on mediawiki01 and mediawiki02. Restarting hhvm [10:41:25] Logged the message, Master [10:51:13] Yeah, things are back to normal for the tools I was using. [11:02:23] 3Wikimedia Labs / 3Infrastructure: Database upgrade MariaDB 10: Key: user_timestamp missing from archive table - 10https://bugzilla.wikimedia.org/68912 (10metatron) 3UNCO p:3Unprio s:3normal a:3None According to https://bugzilla.wikimedia.org/show_bug.cgi?id=63777 Key user_timestamp has been added... [11:08:21] 3Wikimedia Labs / 3tools: Install lighttpd binary mod_magnet.so - 10https://bugzilla.wikimedia.org/68614 (10metatron) 5PATC>3RESO/FIX [11:17:52] !ping [11:17:52] !pong [12:07:37] 3Wikimedia Labs / 3Infrastructure: Database upgrade MariaDB 10: Key: user_timestamp missing from archive table - 10https://bugzilla.wikimedia.org/68912 (10Sean Pringle) a:3Sean Pringle [12:21:09] 3Tool Labs tools / 3[other]: merl tools (tracking) - 10https://bugzilla.wikimedia.org/67556 (10merl) [12:21:09] 3Wikimedia Labs / 3(other): (Tracking) Database replication services - 10https://bugzilla.wikimedia.org/48930 (10merl) [12:21:09] 3Wikimedia Labs / 3tools: deletion queries joined with tokudb replication tables are really slow - 10https://bugzilla.wikimedia.org/68918 (10merl) 3NEW p:3Unprio s:3major a:3Sean Pringle DELETION queries which are joining with replication databases for selection are really slow on mariadb 10. exampl... [12:21:09] 3Wikimedia Labs / 3tools: Missing Toolserver features in Tools (tracking) - 10https://bugzilla.wikimedia.org/58791 (10merl) [12:43:34] Coren: can you bump the quota for the quarry project by a bit? [12:44:06] Coren: I have 4 instances, 2 that'll be running master and two that'll be 'production', and current quota only lets me get 3 [12:44:10] I'll try to get to it today if I get a minute. [12:44:57] Coren: ok [12:57:35] 3Wikimedia Labs / 3tools: deletion queries joined with tokudb replication tables are really slow - 10https://bugzilla.wikimedia.org/68918#c1 (10Sean Pringle) I'm fairly sure this is a manifestation of the same bug in the github report, which relates to TokuDB fractal indexes not (yet) using a bulk prefetchin... [12:58:51] 3Wikimedia Labs / 3tools: Provide resource for db access in grid - 10https://bugzilla.wikimedia.org/68881 (10Sam Reed (reedy)) [13:10:33] !ping [13:10:33] !pong [13:41:49] hey. there seems to be a problem between labs and packagist.org (16% packet loss). this is killing the CI infrastructure for Wikidata (because reasons). [13:41:58] is anyone around to help & investigate? [13:42:36] the problem doesn't exist between the wmde office and packagist, nor does it exist between labs and, say, google [13:42:41] it seems to be pretty specific [13:42:47] any idea how to pin it down? [13:42:56] (maybe they are just throtteling us?) [13:48:07] !log integration updating puppet master [13:48:10] Logged the message, Master [13:49:15] hashar: hey! does Jenkins now support composer installs? [13:49:21] no [13:49:28] if so, we could stop maintaining our own jenkins for wikidata, which is a pain [13:49:50] yeah I am working on something :-] [13:49:53] hashar: too bad :) someone told me it does... i wonder where that rumor came from [13:50:03] aka a way to easily fetches multiple repositories [13:50:09] one of them holding composer dependencies [13:50:17] gotta need that for mediawiki/core [13:50:20] mm [15:54:29] scfc_de: do you have a link to the patch that installed the android sdk dependencies? [15:54:31] on toollabs? [15:54:35] might be resurrected [16:17:03] YuviPanda: Not at hand, but "owner:yuvipanda status:abandoned"? [16:17:25] scfc_de: heh, ok [16:19:31] YuviPanda: I think https://gerrit.wikimedia.org/r/#/c/125241/ [17:16:52] Accessing the wmflabs I get: Abfrage konnte nicht ausgeführt werden Denkmlalliste: Disk full (/tmp/#sql_5b7_4); waiting for someone to free some space... ; who can help? [17:18:00] Herzi_Pinki: what machine are you connecting to? [17:18:45] I just called one of the wmflabs tools: https://tools.wmflabs.org/denkmalliste/index.php?action=EinzelID&ID=94029 [17:18:57] No idea about the machine [17:19:10] Ah, I see, thought you were using ssh. [17:19:18] Might be best to contact the owner of that tool. [17:20:05] the owner of the tool can free space on the temp partition? Does she have a private instance of a VM running, she can administer herself? [17:21:04] the owner is away since a fortnight, vacancies I suppose [17:22:31] No, but they might better understand the problem. I'll look a bit harder once I finish up what I'm doing now... [17:22:51] thanks andrewbogott [17:23:24] I will tell the tool owner so that she can fix the root cause [17:34:59] All webgrid nodes seem to have plenty of space in /tmp. [17:39:36] 3Wikimedia Labs / 3deployment-prep (beta): beta labs not responding; API shows 503 from varnish - 10https://bugzilla.wikimedia.org/68574 (10Bryan Davis) [17:42:18] andrewbogott: Looking at denkmalliste's source, the "Disk full (/tmp/#sql_5b7_4); waiting for someone to free some space..." is the output of mysql_error(), and it connects only to tools-db, so /tmp on tools-db could be full. [17:43:22] scfc_de: you mean the prod db boxes? [17:44:38] andrewbogott: I don't know exactly where tools-db is located; I think one of the labsdb*. [17:44:44] hm [17:44:59] 10.64.37.9 [17:49:26] Coren, springle, labsdb1005 has a full root partition. Is that somehow expected? [17:49:49] andrewbogott: No. Lemme go check. [17:53:30] Something odd going on; while table data is (as expected) in /srv, there is nearly 1T of stuff in /var/lib/postgresql [17:54:43] And there is both a ./main and a ./main.old both of which seem to be in use [17:54:57] that's… concerning. [17:55:06] Is that one of the boxes that springle has already upgraded? [17:55:20] No. [17:57:34] mysqld runs with the correct --datadir [17:57:50] I can free a little bit of log space, but that's not going to hold. [17:59:06] Not even enough to make a dent, actually. [17:59:39] I don't understand… is postgres related to mysql? Or is a whole different database that we're also hosting? [18:00:13] andrewbogott: Coren is this the postgis database that akosorias was playing with? [18:00:15] Is labsdb1005 the PostgreSQL slave? IIRC akosarios worked on something PostgreSQL-related recently. [18:00:17] (I never get the name right) [18:00:39] scfc_de: It is. [18:00:51] DOH! [18:00:56] Ha, copy & paste helps: akosiaris. [18:00:56] He put the database on the root FS! [18:01:42] * Coren fixes that. [18:01:50] It's a slave, I should be able to stop it politely. [18:02:53] !ping [18:02:53] !pong [18:05:59] * Coren creates a volume for postgres, moves data there. [18:17:46] !ping [18:17:46] !pong [18:38:13] I am not able to create/ delete instances in my mediawiki-verp labs project [18:38:29] deletion shows something like :- the requested host do not exists [18:39:20] tonythomas: try logging out and logging back in? [18:39:36] YuviPanda: checking [18:40:00] quit [18:40:19] YuviPanda: now looks ok :) [18:40:23] tonythomas: :) [18:40:30] * tonythomas wonders how that helped [18:42:49] chrismcmahon, greg-g: i broke beta; fixing [18:43:05] tonythomas: turning it off and on always helps [18:43:59] ha :) true taht [18:44:34] chrismcmahon, greg-g: fixed [18:45:13] ori: that was quick, thanks :) [18:54:20] are tools-db dead ? I got a ERROR 1114 (HY000) ... The table 'hocr' is full when creating an index on an empty table ? [18:54:41] phe: Known issue, fix in progress. [18:55:02] phe: It'll take a bit, there's a lot of data to move. [18:58:54] !log deployment-prep re-enabled puppet on deployment-mediawiki{01,02} [18:58:56] Logged the message, Master [18:59:23] 3Wikimedia Labs / 3Infrastructure: Database upgrade MariaDB 10: Engine / Option mismatch on table `user_properties` - 10https://bugzilla.wikimedia.org/68942 (10metatron) 3UNCO p:3Unprio s:3normal a:3None My optimizer throws mismatch error on dump/recreation for table user_properties. Compression does... [18:59:39] bd808: ^^ [18:59:49] (not the bug, the puppet on the beta app servers !log) [18:59:54] ori: \o/ [19:00:10] it's puppetized now; i cherry-picked https://gerrit.wikimedia.org/r/#/c/150873/ [19:05:29] ori: You may want to look at local commit 572c5da3de501df6a7beb3fbb18c2f159b8e0e5b in deployment-salt:/var/lib/git/operations/puppet and see if that hack commit can be removed now. [19:05:44] * ori nods [19:10:14] andrewbogott: wikitech is acting all weird now :| [19:10:25] andrewbogott: I ran into the 'missing things in Special:Nova*' thing again [19:10:29] andrewbogott: logged out, told me I had logged out [19:10:39] andrewbogott: now I try to log in again, it tells me I'm already logged in [19:12:18] YuviPanda: try to log in again? [19:12:30] andrewbogott: now it says I'm logged out [19:12:33] I'm so confused [19:12:45] so… working correctly now? [19:13:36] andrewbogott: seems to be [19:13:42] after enabling role::mediawiki-install::labs and sudo puppet agent -tv, I get this error Error: Failed to apply catalog: Could not find dependent Package[apache2-mpm-worker] for Apache::Mod_conf[mpm_worker] at /etc/puppet/modules/apache/manifests/mpm.pp:46 [19:13:42] …ok... [19:13:45] andrewbogott: hmm, no, I got a blank page on login submission [19:13:56] andrewbogott: hard refresh gives me the login form again [19:14:16] I just logged out and in again, no problems. [19:14:22] But, ok, I'm watching the log now, try again [19:14:25] andrewbogott: I typed things in again, and could log in again. [19:14:40] dang [19:14:42] well, ok [19:14:42] andrewbogott: while you're there, can you grep logs for OAuth fatals? [19:14:50] which logs? Apache? [19:15:01] andrewbogott: OAuth is broken on wikitech, since any API request made with the authorized headers just seems to give a blank page [19:15:02] andrewbogott: ya [19:15:08] andrewbogott: wherever php fatals go to [19:16:52] So… did it work before? I thought you confirmed that it was fine [19:17:36] andrewbogott: most of the steps worked, so I assumed it did [19:17:44] andrewbogott: except actually making requests as authenticated users :( [19:18:08] ah [19:18:14] andrewbogott: let me setup a test instance up so I can hit it again [19:18:18] I don't see anything, but I don't really know where to look. [19:18:35] Maybe you should mess with wikitech-test and see if you can get it working there? [19:18:51] andrewbogott: where's wikitech-test? is it a labsproject? [19:18:54] andrewbogott: does it have OAuth installed? [19:19:01] not until you install it [19:19:05] andrewbogott: or alternatively, can I put in an access request for virt1xxx? [19:19:05] it's in testlabs [19:19:12] I already have one for labmon [19:19:16] and it uses wikitech vagrant [19:20:43] I don't know -- you can put in a request but it probably needs to be discussed at an Ops meeting, which, there won't be one for ten days [19:21:08] andrewbogott: can you check now? I just got a blank page there again [19:21:50] Hm, try once more? [19:22:38] andrewbogott: moment [19:23:11] andrewbogott: hitting now [19:23:17] hit [19:23:19] waiting [19:23:25] andrewbogott: blank page! [19:23:30] PHP Fatal error: Call to a member function isSpecial() on a non-object in /srv/org/wikimedia/controller/wikis/slot0/extensions/OAuth/api/MWOAuthAPI.setup.php on line 55 [19:23:31] well, blank API response [19:23:33] boom [19:23:56] anomie: ^ OAuth fatal [19:24:02] anomie: do you know what might be wrong? [19:24:43] legoktm: ^ [19:25:08] uhhhhhh [19:25:16] abuse of an ExtensionFunction [19:25:21] that's what it was last time [19:25:25] hmm? [19:25:27] YuviPanda: Somehow RequestContext::getMain() is returning a context with a null title? [19:25:45] hmm, this is in response to an API request [19:26:04] $wgTitle should be set in the API [19:26:49] what non-standard extensions does wikitech have? OSM and SMW? [19:27:06] legoktm: yeah [19:27:18] It also uses ldap auth [19:27:39] legoktm: Reedy says getTitle can also return null, according to the doc [19:27:52] but it never should in the API [19:27:58] [20:27:02] ofc $context->getTitle() does say @return Title|null [19:28:42] that should fallback to $wgTitle which api.php sets [19:29:12] andrewbogott: is both mw and OAuth on master? or something else? [19:29:32] YuviPanda: Special:Version [19:29:43] ah, right [19:29:43] wmf/1.24wmf15 [19:30:07] https://bugzilla.wikimedia.org/show_bug.cgi?id=58705#c2 [19:30:54] andrewbogott: ^ heh, exact same error, but we don't have flow :( [19:30:55] https://bugzilla.wikimedia.org/show_bug.cgi?id=58380 [19:31:11] YuviPanda: the issue was that something in an ExtensionFunction was trying to access context [19:33:14] legoktm: andrewbogott so it's either OSM or SMW [19:33:17] and I hope it's OSM [19:33:26] I didn't see anything in OSM :\ [19:34:19] legoktm: uh oh [19:34:43] what branch of SMW is this? [19:34:46] So, what should I grep for to see if the issue is in SMW? [19:34:53] * legoktm checks [19:38:38] andrewbogott: \$wgExtensionFunctions I guess [19:41:56] legoktm: here's the full grep: https://dpaste.de/sJNd [19:42:30] andrewbogott: do we have all those enabled? [19:42:36] andrewbogott: can you list the enabled extensions list as well? [19:43:09] Is there a maintenance command or that? [19:43:12] *for [19:43:33] YuviPanda: that's Special:Version... [19:44:17] andrewbogott: I don't see anything in those extensions that would cause it :/ [19:45:51] andrewbogott: can you post the full backtrace? [19:46:38] legoktm: you have all that I have. [19:47:26] uhhh [19:47:43] can you enable more debug logging :) [19:47:54] I can trigger it pretty trivially [19:48:20] I'd prefer not, on a working production system. I'd encourage you to try to reproduce this on a labs instance. [19:49:25] andrewbogott: heh, yeah. I'll setup OAuth there [19:50:02] andrewbogott: what instance / proect you were telling me about? [19:50:02] thanks [19:50:09] testlabs project [19:50:25] lemme make sure you're a member [19:50:40] andrewbogott: cool, and projectadmin? :) [19:51:30] oh, wait, that's the wrong project, hang on... [19:52:06] ok, try instance wikitech-test-frontend in project 'openstack' [19:52:14] that has labs-vagrant with the wikitech role [19:52:24] and also a bunch of openstack stuff and ldap bits set up, which, please don't break those parts :) [19:54:12] andrewbogott: moment [19:59:46] !log deployment-prep Granted sudo right to cscott (under NDA). Will let him reboot OCG service [19:59:48] Logged the message, Master [20:01:22] (03PS1) 10Legoktm: nightly.py: Don't bail if updating an extension fails [labs/tools/extdist] - 10https://gerrit.wikimedia.org/r/150903 [20:01:30] YuviPanda: ^ [20:02:53] (03CR) 10Legoktm: [C: 032] nightly.py: Don't bail if updating an extension fails [labs/tools/extdist] - 10https://gerrit.wikimedia.org/r/150903 (owner: 10Legoktm) [20:09:41] !log deployment-prep updated OCG to version d2919c59eb09e09fc87777696411a070620aef45 [20:09:43] Logged the message, Master [20:10:38] legoktm: heh, lol at just skipping [20:10:57] YuviPanda: someone deleted a git repo that is still referenced in a REL1_22 branch! [20:11:05] legoktm: heh [20:11:14] I'm not gonna handle that nicely [20:11:40] https://github.com/wikimedia/mediawiki-extensions-Wikidata/tree/REL1_22 <-- the "DataValues" repo doesn't exist anymore [20:26:41] andrewbogott: bah, no OAuth role for vagrant :( This will probably have to wait a while [20:26:53] Where's the page/list to report problems with a specific tool? (Specifically, categorycount is getting an MySQL error, when I run https://tools.wmflabs.org/erwin85/categorycount.php?lang=en&family=wikipedia&category=Wikipedia+noticeboards&subcats=1&d=10&submit=Submit [20:26:59] YuviPanda: adding one should only take 5 minutes though... [20:27:08] YuviPanda: o_0 write it [20:27:58] bd808: andrewbogott head esplode from too many things :) but yeah, I'll do that. [20:28:47] YuviPanda: I understand. I've been working on too many different things as well [20:29:16] bd808: :) seen quarry.wmflabs.org? [20:29:45] I saw you talking aobut having it up but I haven't poked it yet [20:40:15] (03CR) 10Platonides: "Wow, merged when turning 1 :)" [labs/toollabs] - 10https://gerrit.wikimedia.org/r/70110 (owner: 10Platonides) [20:57:36] i'm getting mysql errors as well [20:57:42] Query: UPDATE `page` SET page_touched = '20140731205545' WHERE page_id = '28437' [20:57:46] Error: 2013 Lost connection to MySQL server during query (10.68.16.193) [20:57:56] (from enwiki beta) [21:00:29] `show processlist` on the master looks plenty normal, although maybe my user account just doesn't have access to see much :P [21:01:33] sudo! [21:02:56] ebernhardson: using sql.php doesn't show anything weird [21:03:31] hi, I'm getting some error while encoding u"(Lefèvre_d'Étaples,_Jacques)" : """UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 60: ordinal not in range(128)""" Why is that? [21:08:10] rohit-dua: sounds like the first example for python unicode handling: https://docs.python.org/2/howto/unicode.html#the-unicode-type [21:10:16] ebernhardson: that'll be for decoding. well the encoding works in python terminal but not in the program. [21:16:01] rohit-dua: whats the actual code that its erroring out at ?py [21:16:55] oh yuuuuuuuvi [21:16:59] 17:08 < bd808> That box doesn't run blocking tests does it? [21:16:59] 17:09 < greg-g> this one https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/ [21:17:00] 17:09 < greg-g> hasn't run for a while [21:17:03] 17:14 < bd808> Here's the wikitech page for the instance -- https://wikitech.wikimedia.org/wiki/Nova_Resource:I-00000387.eqiad.wmflabs [21:17:06] 17:14 < bd808> I'm not a member of the puppet3-diffs project so It won't let me see the console output [21:17:09] 17:15 < bd808> One of the labs folks (andrew, coren, yuvi) might be able to poke it [21:17:09] what... did I do? [21:17:12] YuviPanda: ^ [21:17:29] YuviPanda: summary: this job is blocking because it only runs on that box: https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/ [21:17:44] greg-g: ah, no, sadly. I don't have access rights yet (and probably won't for a couple of months). you want andrewbogott or Coren_away [21:17:55] https://integration.wikimedia.org/ci/computer/puppet-compiler02.eqiad.wmflabs/log [21:17:56] ebernhardson: """source = ia_info['response']['docs'][i]['source'].encode("utf-8")""" ia_info is json got from web. [21:18:29] andrewbogott: hey, sorry to interrupt, can you take a look at the above (my convo with yuvi, just since XX:16) ^ [21:18:41] greg-g: sure, reading... [21:19:10] that last link is probably the most helpful bit right now [21:19:51] andrewbogott: : My guess is that puppet isn't running there (ironic?) and thus the ldap connectivity was broken by an apt package upgrade. [21:19:58] b.d808 thinks it could be related to a security update of a package that we need to apply a special confi..... [21:20:09] That happened to 4 hosts in beta yesterday [21:20:28] um… happen to know the name of that instance (other than the i-000whatever ec2 name?) [21:20:41] andrewbogott: puppet-compiler02.eqiad.wmflabs [21:20:42] oh, nm, that's on the page [21:20:44] rohit-dua: not sure, havn't seen it error like that before :( [21:21:40] greg-g: puppet is explicitly switched off on that box [21:22:04] hmm, this is what I get when I go poking at things [21:22:18] matanya, if you are awake, can you comment? [21:22:30] or maybe just wait until hashar is awake tomorrow [21:22:33] hm, all admins sleeping [21:22:47] I'm reluctant to switch puppet back on since whatever's happening is clearly on purpose :) [21:22:49] * greg-g emails hashar [21:22:57] Are there other things I should look at vs. just puppet not running there? [21:24:45] andrewbogott: that's it, probably, honestly this got more complex than I was intending with that auth failure in the log :) [21:24:55] greg-g: ok… waiting for hashar to wake up is OK? [21:26:20] andrewbogott: if you don't know what this job does, probably https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/ [21:26:40] (ie: it must not be *that* important?) [21:27:14] I sort of know what it does, but I don't know how the pieces of zuul and jenkins fit together exactly [21:27:36] Generally if a job won't compile there, it won't compile on a server either... [21:28:49] curious why it only runs very infrequently [21:28:52] * greg-g shrugs [21:31:25] yeah, best to wait and get the full story from joe or hashar [21:34:05] andrewbogott: comment on what ? [21:34:25] matanya: greg-g is blocked (or at least puzzled) by a frozen puppet-compiler job. [21:34:40] I logged into puppet-compiler02 and note that puppet is disabled. [21:34:50] Since you're an admin for that project I thought you might know what's up [21:34:51] of course it is [21:35:06] that box is for puppet tests [21:35:15] ah, because it runs the puppet compiler on itself? Not in a jail? [21:35:23] i think so [21:35:32] didn't check in a long time [21:35:39] hm, ok then. I still don't entirely understand what was thwarting greg-g [21:35:43] joe would know for sure [21:35:59] that box should have nothing to do with him [21:36:19] it is an internal puppet changes testing box [21:36:35] ok. [21:36:39] greg-g: want to shed some light on this ? [21:36:43] I'm going to ignore this unless greg-g wants to appear and engage :) [21:36:53] I just innocently was curious why this was shown as blocked https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/ , then rabbit hole :/ [21:37:18] * YuviPanda sings 'welcome to the rabbit hole' to the tune of 'welcome to the machine' [21:37:40] where have you been? it's alright we know where you've been! (because we saw the link) [21:38:27] if I should care about it, that's fine :) [21:38:32] shouldn't* [21:38:40] greg-g: FYI: This node is offline because Jenkins failed to launch the slave agent on it. See log for more details [21:39:03] matanya: yes, bad ssh key or somesuch [21:39:17] i can fix it if it bothers you, but it is almost 1am and i have other things to fix. is it critical for you ? [21:39:23] greg-g: btw, ori ran into it a while ago and was trying to file an RT ticket, I think? [21:39:40] matanya: not at all [21:39:42] * matanya needs root on jenkins boxes [21:39:47] unless that job is critical to run [21:39:53] it is not [21:39:58] then go do other things :) [21:40:01] thanks [21:40:04] just a puppet compilation differ [21:40:19] YuviPanda: not sure about either [21:40:37] file a ticket, i'll handle [21:40:37] greg-g: just a fyi from IRC memory :) [21:40:38] Now I'm worried that when I typed 'matanya' in IRC, it cause a klaxon to sound and Matanya's bed to tip upright, spilling him into a desk chair. [21:40:57] * YuviPanda should setup such a setup [21:41:11] wallas and grommit! [21:41:21] just don't put on the wrong trousers [21:41:38] andrewbogott: in fact that is what happened :) [21:41:41] night all [21:41:55] well, ok, I will be more conscious of timezones in the future, sorry :) [21:51:07] 3Wikimedia Labs / 3Infrastructure: Database upgrade MariaDB 10: Engine / Option mismatch on table `user_properties` - 10https://bugzilla.wikimedia.org/68942 (10Sean Pringle) a:3Sean Pringle [22:02:21] 3Wikimedia Labs / 3Infrastructure: Database upgrade MariaDB 10: Engine / Option mismatch on table `user_properties` - 10https://bugzilla.wikimedia.org/68942#c1 (10Sean Pringle) 5UNCO>3ASSI The table is intentionally, though hopefully temporarily, InnoDB, due to a TokuDB issue we're still investigating in... [22:17:23] andrewbogott: https://gerrit.wikimedia.org/r/#/c/150971/ should fix OAuth on wikitech \o/ [22:18:01] YuviPanda: ok! Um… it will be a couple of weeks before I roll out updates though :( [22:18:13] andrewbogott: even for something that's already fataling? [22:18:22] hm… is it merged? [22:18:33] will it be added to core in the next release? [22:18:48] andrewbogott: core? it's on the OAuth extension [22:19:24] andrewbogott: and yeah, it'll roll into the next wmf branch [22:19:27] Sorry, what I mean is -- will the wmf version of... [22:19:30] yes, that's what I meant :) [22:19:37] OK, I can try to hotfix, hang on... [22:19:47] YUSSSS [22:21:28] YuviPanda: um… I'd really like to see it get merged first though... [22:21:41] andrewbogott: yeah, that's fine. I'll poke around to get it merged soon [22:21:53] still, YAY [22:21:55] ok, thanks [22:30:56] andrewbogott: how do I regenerate the Pacakges.gz file in /data/project/repo with labsdebrepo? [22:32:30] andrewbogott: also, the patch has been merged and backported into wmf15 :) [22:32:58] dpkg-scanpackages . /dev/null | gzip -9c > binary/Packages.gz [22:33:05] andrewbogott: ah, cool, thanks [22:33:12]