[00:04:59] 10PAWS: Dead kernel after exec a Python notebook - https://phabricator.wikimedia.org/T155381#2941986 (10Ivanhercaz) [00:08:41] PROBLEM - Puppet run on tools-exec-1411 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [00:48:40] RECOVERY - Puppet run on tools-exec-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [05:44:38] folks, sorry to bug, but we're seeing a lot of 500s on the wikipedia IFTTT channel labs tool [05:44:47] our dashboard is well over 50% red at this point [05:45:33] yuvipanda: you in here? [05:46:04] it's been deteriorating over the last week, consistently bad for the last day [05:46:19] mhashemi: i'll look [05:46:31] thank you madhuvishy [06:00:32] mhashemi: it looks like the article of the day endpoint is the only one that throws 500s [06:00:38] is that what you're seeing? [06:01:03] what is the dashboard you're looking at? [06:01:14] article revisions is throwing 500 too [06:01:21] this is IFTTT's internal dashboard for channel owners [06:01:31] article of the day is just way more popular of a recipe, so it has high traffic [06:04:05] mhashemi: hmmm [06:04:10] I see [06:04:12] [pid: 12|app: 0|req: 1944624/6718495] 192.168.37.0 () {46 vars in 869 bytes} [Mon Jan 16 06:03:13 2017] POST /ifttt/v1/triggers/article_revisions => generated 29224 bytes in 420 msecs (HTTP/1.1 200) 2 headers in 89 bytes (1 switches on core 0) [06:04:26] HTTP/1.1 200 [06:04:53] i'm not sure this is a tools issue - something seems to have changed with the article_of_the_day end point [06:08:59] https://www.irccloud.com/pastebin/dOrCb3ub/Article%20of%20the%20day%20ifttt%20error [06:10:48] ahh, ok, when i get back to my computer i'll take a closer look [06:10:49] thank you! [06:13:13] mhashemi: okay, let me know if you need any help [09:31:46] 06Labs, 10DBA: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#2942371 (10Marostegui) We need to decide a host per shard to be the master for db1095 as it needs to stream row based replication to db1095. Possible candidates s2: (they are a... [10:28:40] 06Labs, 10Tool-Labs: Several users and tools have invalid credentials in replica.my.cnf - https://phabricator.wikimedia.org/T154933#2942457 (10Marostegui) I am going to remove the DBA tag from this ticket as it doesn't look related to a DB problem at the moment (if you guys feel it needs to be, feel free to ad... [12:54:53] !log wikispeech Deploy latest from Git master: 46f0ef0 (T144156) [12:54:53] Unknown project "wikispeech" [12:54:54] T144156: Use correct language when reciting - https://phabricator.wikimedia.org/T144156 [12:55:37] !log Wikispeech Deploy latest from Git master: 46f0ef0 (T144156) [12:55:37] Unknown project "Wikispeech" [13:05:41] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [13:39:01] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [14:15:42] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:44:00] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [15:06:41] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [16:25:42] 10Tool-Labs-tools-Other: svgtranslate tool - https://phabricator.wikimedia.org/T154866#2943202 (10Mikey641) Thank you very much! [16:25:44] (03PS6) 10Paladox: Connect wikibugs to irc over ssl [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/328663 (https://phabricator.wikimedia.org/T141089) [16:44:00] !log wikilabels deploying 4fe40ef into staging [16:44:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL [17:14:38] 06Labs, 10Labs-Infrastructure: set apt to Wikimedia mirror instead of http://nova.clouds.archive.ubuntu.com/ubuntu/ - https://phabricator.wikimedia.org/T68121#2943242 (10Dzahn) p:05Low>03Normal [17:20:22] yuvipanda: about? [18:11:44] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:07:41] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [19:22:05] !log wikilabels deployed baf1558 to master [19:22:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL [20:10:02] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [20:22:39] !log wikilabels u_wikilabels=> select * from campaign where wiki = 'plwiki'; [20:22:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL [20:36:17] 10PAWS: Dead kernel after exec a Python notebook - https://phabricator.wikimedia.org/T155381#2943583 (10Ivanhercaz) It seem to be some error of the Python 3 notebook because if I exec the same script through bash commands it seems to work fine. It also happens if I exec the script in a terminal. [21:15:01] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [21:19:01] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Rich Farmbrough was created, changed by Rich Farmbrough link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Rich_Farmbrough edit summary: Created page with "{{Tools Access Request |Justification=Automating support for WikiProjects. And other cool stuff. |Completed=false |User Name=Rich Farmbrough }}" [21:36:02] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [21:42:12] 10PAWS: Dead kernel after exec a Python notebook - https://phabricator.wikimedia.org/T155381#2943681 (10Ivanhercaz) Python 3 notebook seems to work again. [21:42:23] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Rich Farmbrough was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=1329816 edit summary: [22:46:03] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [23:27:46] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Rich Farmbrough was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=1330075 edit summary: