[02:10:29] leloiandudu: T143504 [02:10:30] T143504: Require signing of OAuth request body - https://phabricator.wikimedia.org/T143504 [02:16:10] tgr: thanks, that's some great analysis! :) [02:23:10] (03CR) 10BryanDavis: [C: 032] Add initial database schema [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/305046 (https://phabricator.wikimedia.org/T142545) (owner: 10BryanDavis) [02:23:17] (03Merged) 10jenkins-bot: Add initial database schema [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/305046 (https://phabricator.wikimedia.org/T142545) (owner: 10BryanDavis) [02:34:13] (03CR) 10Dzahn: "in the README it says "git clone https://phabricator.wikimedia.org/diffusion/TCRW/labs-tools-crosswatch.git" as the repo URL to clone from" [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/305293 (https://phabricator.wikimedia.org/T139089) (owner: 10Paladox) [03:14:52] PROBLEM - Puppet run on tools-exec-1208 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [03:27:45] PROBLEM - Puppet run on tools-puppetmaster-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [03:29:13] PROBLEM - Puppet run on tools-worker-1015 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [03:30:03] PROBLEM - Puppet run on tools-webgrid-generic-1404 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [03:32:27] PROBLEM - Puppet run on tools-flannel-etcd-03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [03:40:32] PROBLEM - Puppet run on tools-worker-1023 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [03:42:44] RECOVERY - Puppet run on tools-puppetmaster-01 is OK: OK: Less than 1.00% above the threshold [0.0] [03:54:51] RECOVERY - Puppet run on tools-exec-1208 is OK: OK: Less than 1.00% above the threshold [0.0] [03:55:33] RECOVERY - Puppet run on tools-worker-1023 is OK: OK: Less than 1.00% above the threshold [0.0] [03:56:57] PROBLEM - Puppet run on tools-k8s-etcd-03 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [04:06:58] RECOVERY - Puppet run on tools-k8s-etcd-03 is OK: OK: Less than 1.00% above the threshold [0.0] [04:09:12] RECOVERY - Puppet run on tools-worker-1015 is OK: OK: Less than 1.00% above the threshold [0.0] [04:10:01] RECOVERY - Puppet run on tools-webgrid-generic-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [04:12:26] RECOVERY - Puppet run on tools-flannel-etcd-03 is OK: OK: Less than 1.00% above the threshold [0.0] [05:44:23] PROBLEM - Puppet staleness on tools-webgrid-lighttpd-1208 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [05:54:54] PROBLEM - Puppet staleness on tools-exec-1211 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [05:58:59] PROBLEM - Puppet staleness on tools-exec-1213 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [05:58:59] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Facenapalm was created, changed by Facenapalm link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Facenapalm edit summary: Created page with "{{Tools Access Request |Justification=Bot hosting (https://ru.wikipedia.org/wiki/User:NapalmBot). |Completed=false |User Name=Facenapalm }}" [06:01:37] PROBLEM - Puppet run on tools-worker-1003 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [06:14:34] PROBLEM - Puppet run on tools-webgrid-lighttpd-1412 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [06:16:22] PROBLEM - Puppet run on tools-webgrid-lighttpd-1210 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [06:41:36] RECOVERY - Puppet run on tools-worker-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [06:54:33] RECOVERY - Puppet run on tools-webgrid-lighttpd-1412 is OK: OK: Less than 1.00% above the threshold [0.0] [06:56:22] RECOVERY - Puppet run on tools-webgrid-lighttpd-1210 is OK: OK: Less than 1.00% above the threshold [0.0] [06:59:14] PROBLEM - Puppet run on tools-webgrid-lighttpd-1414 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [07:34:12] RECOVERY - Puppet run on tools-webgrid-lighttpd-1414 is OK: OK: Less than 1.00% above the threshold [0.0] [07:35:44] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Facenapalm was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=817890 edit summary: [07:41:58] PROBLEM - Puppet run on tools-exec-1409 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [08:05:12] PROBLEM - Puppet staleness on tools-exec-1204 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [43200.0] [08:15:49] PROBLEM - Puppet run on tools-exec-1208 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [08:27:00] RECOVERY - Puppet run on tools-exec-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [08:49:33] 10PAWS: Paws display 504 - Bad gateway time-out - https://phabricator.wikimedia.org/T143493#2570099 (10yuvipanda) I've restarted it for you again just now. [08:55:49] RECOVERY - Puppet run on tools-exec-1208 is OK: OK: Less than 1.00% above the threshold [0.0] [09:12:58] PROBLEM - Puppet run on tools-exec-1409 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [09:53:00] RECOVERY - Puppet run on tools-exec-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [09:54:05] (03CR) 10Paladox: "Yes, that is the clone link, but this changes the other links to correctly link to the repo." [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/305293 (https://phabricator.wikimedia.org/T139089) (owner: 10Paladox) [10:00:20] (03PS1) 10Jérémie Roquet: Add README.md [labs/tools/WikiConvFR-training-2016] - 10https://gerrit.wikimedia.org/r/305851 [10:05:31] (03CR) 10Arktest: [C: 031] "Looks good to me." [labs/tools/WikiConvFR-training-2016] - 10https://gerrit.wikimedia.org/r/305851 (owner: 10Jérémie Roquet) [10:06:01] (03CR) 10Jérémie Roquet: [C: 032 V: 032] "Ok." [labs/tools/WikiConvFR-training-2016] - 10https://gerrit.wikimedia.org/r/305851 (owner: 10Jérémie Roquet) [10:20:57] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/தமிழ்க்குரிசில் was created, changed by தமிழ்க்குரிசில் link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/%e0%ae%a4%e0%ae%ae%e0%ae%bf%e0%ae%b4%e0%af%8d%e0%ae%95%e0%af%8d%e0%ae%95%e0%af%81%e0%ae%b0%e0%ae%bf%e0%ae%9a%e0%ae%bf%e0%ae%b2%e0%af%8d edit summary: Created page with "{{Tools Access Request |Justification=To create a bot for maintenance/Cleanup in Tamil wikipedia and other wikimedia projects. |Completed=false |User Name=தமிழ்க..." [12:14:47] Arkanosis: *waves* [12:21:51] Hi :) [12:26:43] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Nicolas NALLET was created, changed by Nicolas NALLET link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Nicolas_NALLET edit summary: Created page with "{{Tools Access Request |Justification=WikiConvention training |Completed=false |User Name=Nicolas NALLET }}" [12:26:56] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Shonagon was created, changed by Shonagon link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Shonagon edit summary: Created page with "{{Tools Access Request |Justification=tools for creating and editing artworks for Wikidata |Completed=false |User Name=Shonagon }}" [12:27:19] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Assassas77 was created, changed by Assassas77 link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Assassas77 edit summary: Created page with "{{Tools Access Request |Justification=To try developping tools in Mediawiki. It is part of the Wikiconvention France 2016. |Completed=false |User Name=Assassas77 }}" [12:27:30] * valhallasw`cloud goes off to click some links [12:27:33] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Trizek was created, changed by Trizek link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Trizek edit summary: Created page with "{{Tools Access Request |Justification=I want to do stuff with wikidata and frwiki config (I'm a sysop there). #WikiConfFR |Completed=false |User Name=Trizek }}" [12:27:42] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Nojhan was created, changed by Nojhan link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Nojhan edit summary: Created page with "{{Tools Access Request |Justification=I'm at wikiconvfr and I want to improve the timeline extension for wikidata. |Completed=false |User Name=Nojhan }}" [12:27:52] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/தமிழ்க்குரிசில் was modified, changed by Merlijn van Deen link https://wikitech.wikimedia.org/w/index.php?diff=817906 edit summary: [12:27:53] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/GAllegre was created, changed by GAllegre link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/GAllegre edit summary: Created page with "{{Tools Access Request |Justification=Learning dev in Wikiconvention-fr Develop wikidata extraction tools from wikisource documents |Completed=false |User Name=GAllegre }}" [12:28:02] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Envlh was created, changed by Envlh link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Envlh edit summary: Created page with "{{Tools Access Request |Justification=I discover the tools with the workshop at WikiConventionFr. I intend to use it with Wikidata. |Completed=false |User Name=Envlh }}" [12:28:11] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Mah3110 was created, changed by Mah3110 link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Mah3110 edit summary: Created page with "{{Tools Access Request |Justification=Contributing to the source code creation |Completed=false |User Name=Mah3110 }}" [12:28:19] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Sorcierbob was created, changed by Sorcierbob link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Sorcierbob edit summary: Created page with "{{Tools Access Request |Justification=Wikiconvention Paris |Completed=false |User Name=Sorcierbob }}" [12:29:44] Arkanosis: how many people are there? [12:30:06] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Nicolas NALLET was modified, changed by Merlijn van Deen link https://wikitech.wikimedia.org/w/index.php?diff=817914 edit summary: [12:30:12] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Shonagon was modified, changed by Merlijn van Deen link https://wikitech.wikimedia.org/w/index.php?diff=817916 edit summary: [12:30:21] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Assassas77 was modified, changed by Merlijn van Deen link https://wikitech.wikimedia.org/w/index.php?diff=817918 edit summary: [12:30:32] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Trizek was modified, changed by Merlijn van Deen link https://wikitech.wikimedia.org/w/index.php?diff=817920 edit summary: [12:30:43] Seven people, thank you very much! :) [12:31:08] All requests have been submitted. [12:31:43] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Sorcierbob was modified, changed by Merlijn van Deen link https://wikitech.wikimedia.org/w/index.php?diff=817923 edit summary: [12:31:48] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Mah3110 was modified, changed by Merlijn van Deen link https://wikitech.wikimedia.org/w/index.php?diff=817925 edit summary: [12:31:53] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Envlh was modified, changed by Merlijn van Deen link https://wikitech.wikimedia.org/w/index.php?diff=817928 edit summary: [12:31:57] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/GAllegre was modified, changed by Merlijn van Deen link https://wikitech.wikimedia.org/w/index.php?diff=817930 edit summary: [12:32:02] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Nojhan was modified, changed by Merlijn van Deen link https://wikitech.wikimedia.org/w/index.php?diff=817932 edit summary: [12:32:33] Thank you so much for your help, Merlijn! :) [12:32:38] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Csisc was created, changed by Csisc link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Csisc edit summary: Created page with "{{Tools Access Request |Justification=Contributing to source code creation |Completed=false |User Name=Csisc }}" [12:32:48] Arkanosis: wikitech is throwing some errors, but I think it should all work [12:33:43] Awesome! I owe you a drink when we met ;-) [12:34:07] Arkanosis: please let me know if anyone has issues logging in. One non-obvious thing to remember: shell names are case-sensitive [12:34:42] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Csisc was modified, changed by Merlijn van Deen link https://wikitech.wikimedia.org/w/index.php?diff=817935 edit summary: [12:38:50] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Drongou was created, changed by Drongou link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Drongou edit summary: Created page with "{{Tools Access Request |Justification=Wp:fr stuff |Completed=false |User Name=Drongou }}" [12:39:13] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Drongou was modified, changed by Merlijn van Deen link https://wikitech.wikimedia.org/w/index.php?diff=817940 edit summary: [13:56:31] (03PS1) 10GAllegre: initialisation du fichier gallegre [labs/tools/WikiConvFR-training-2016] - 10https://gerrit.wikimedia.org/r/305864 [13:58:09] 10PAWS: Paws display 504 - Bad gateway time-out - https://phabricator.wikimedia.org/T143493#2570397 (10Ivanhercaz) Thank you @yuvipanda! If it happens again I will notify you. [14:09:25] Hmm, db metawiki_p "SELECT site_domain FROM sites" gives domain names in reverse order (gro.yranoitkiw.ym.). [14:26:19] he1kki: you're probably looking for meta_p [14:27:14] (meta_p is a table with metadata, metawiki_p is the table for meta.wikimedia.org) [14:27:22] wrt metawiki_p.sites: https://www.mediawiki.org/wiki/Manual:Sites_table#site_domain [14:37:58] (03PS1) 10Xavier Combelle: init readme [labs/tools/WikiConvFR-training-2016] - 10https://gerrit.wikimedia.org/r/305865 [14:39:29] (03CR) 10Xavier Combelle: [C: 031] "well done !" (031 comment) [labs/tools/WikiConvFR-training-2016] - 10https://gerrit.wikimedia.org/r/305864 (owner: 10GAllegre) [21:11:51] PROBLEM - Puppet run on tools-exec-1208 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [21:13:59] PROBLEM - Puppet run on tools-exec-1409 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [21:15:35] PROBLEM - Puppet run on tools-webgrid-lighttpd-1412 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [21:27:40] valhallasw`cloud: around? I'm trying to follow your steps at https://phabricator.wikimedia.org/T143375#2567885 [21:28:17] I'm not sure what 15733 is with "sudo gdb /data/project/musikbot/.rbenv/versions/2.2.1/bin/ruby 15733" [21:28:45] a pid? [21:28:57] or a ruby script ? [21:30:48] right, I guess I need to get into the exec node [21:31:13] I tried `ssh musikanimal@tools-exec-1406` using my LDAP password, no dice [21:31:32] while I was in bastion, that is [21:32:03] musikanimal, just do `ssh tools-exec-1406` [21:32:18] beautiful :) [21:36:30] am I able to use gdb without sudo? I'm getting "Could not attach to process. If your uid matches the uid of the target" but I am logged in as tools.musikbot [21:38:36] ^ the rest of that output says "...try again as the root user" [21:39:09] iirc this is disabled for normal users [21:39:39] you could use it when launching the process [21:39:43] gdb program [21:39:51] but not for a running one (gdb -p pid) [21:40:26] well, https://phabricator.wikimedia.org/T143375 is still a problem :( I disabled my log rotation as valhallasw recommended, and reverted the only code changes I had since beginning of August when this started [21:42:15] musikanimal: sorry, I don't have my laptop nearby. [21:42:25] no problem, will comment on phab [21:42:33] Ok! [21:55:32] RECOVERY - Puppet run on tools-webgrid-lighttpd-1412 is OK: OK: Less than 1.00% above the threshold [0.0] [21:56:50] RECOVERY - Puppet run on tools-exec-1208 is OK: OK: Less than 1.00% above the threshold [0.0] [21:58:59] RECOVERY - Puppet run on tools-exec-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [22:01:15] 06Labs, 10Tool-Labs, 13Patch-For-Review: Tool Labs jobs locking up - https://phabricator.wikimedia.org/T143375#2570764 (10MusikAnimal) This is still happening regularly with my three jobs that run with the highest frequency: `copypatrol`, `fix_pp` and `perm_clerk`. I think I have confirmed it is continually... [22:03:39] I guess I don't need gdb anymore, I'm now seeing the same backtrace in my .err files since I made the change to the logging [22:34:24] PROBLEM - Puppet run on tools-exec-1202 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [22:45:15] PROBLEM - Puppet run on tools-exec-1410 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [23:09:07] (03PS1) 10BryanDavis: Use utf8mb4 charset with MySQL backend [labs/striker] - 10https://gerrit.wikimedia.org/r/305940 (https://phabricator.wikimedia.org/T142545) [23:09:09] (03PS1) 10BryanDavis: Add Logstash logging support [labs/striker] - 10https://gerrit.wikimedia.org/r/305941 [23:09:22] RECOVERY - Puppet run on tools-exec-1202 is OK: OK: Less than 1.00% above the threshold [0.0] [23:13:16] (03CR) 10BryanDavis: [C: 032] Use utf8mb4 charset with MySQL backend [labs/striker] - 10https://gerrit.wikimedia.org/r/305940 (https://phabricator.wikimedia.org/T142545) (owner: 10BryanDavis) [23:13:57] (03Merged) 10jenkins-bot: Use utf8mb4 charset with MySQL backend [labs/striker] - 10https://gerrit.wikimedia.org/r/305940 (https://phabricator.wikimedia.org/T142545) (owner: 10BryanDavis) [23:25:16] RECOVERY - Puppet run on tools-exec-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [23:35:23] PROBLEM - Puppet run on tools-exec-1202 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]