[04:08:33] 6Labs: Document labs SSH Fingerprints in sha256 format - https://phabricator.wikimedia.org/T112993#1717803 (10Krenair) a:3Krenair I made a VM with OpenSSH v6.9p1. How's this format, @Platonides/@hashar/@he7d3r? ```alex@ubuntu-15-10-wily-b2:~$ ./server-fingerprints.py bast1001.wikimedia.org ;RSA: * MD5:ee... [04:21:29] 6Labs: Automate population of floating->internal IP aliasing in labs pdns recursor. - https://phabricator.wikimedia.org/T108063#1717810 (10Krenair) Is this addressed fully by https://gerrit.wikimedia.org/r/#/c/243357/ ? [04:21:59] 6Labs, 3Labs-Sprint-109, 5Patch-For-Review: Remove reliance on ldap $::projectid from shinkengen - https://phabricator.wikimedia.org/T108625#1717812 (10Krenair) Sounds like T85612 would be necessary? [06:43:58] hm still debug1: key_parse_private2: missing begin marker [06:44:41] it could be me .. but I dont see any clues to that effect [12:16:51] 6Labs, 7Database: Lots of rows are missing from enwiki_p.`revision` - https://phabricator.wikimedia.org/T115207#1717995 (10Glaisher) 3NEW [12:17:40] 6Labs, 7Database: Lots of rows are missing from enwiki_p.`revision` - https://phabricator.wikimedia.org/T115207#1718005 (10Glaisher) [12:17:42] 6Labs, 10Wikimedia-Labs-General, 6operations, 7Database, 7Tracking: (Tracking) Database replication services - https://phabricator.wikimedia.org/T50930#1718004 (10Glaisher) [14:13:26] Hello [14:13:44] Can i ask any question at here? [14:13:59] ask [14:14:54] I insert commend at Putty "jsub -once python ~/data/project/altobot/core/pwb.py replace -family:wiktionary -transcludes:예문 "{{예문}}" "{{풀기:예문}}" [14:15:02] But it is not worked [14:15:23] "Your job 467500 ("python2") has been submitted" [14:15:41] It can find job 467500 http://tools.wmflabs.org/?status [14:15:49] can -> can`t [14:16:05] hm, sorry I can not help you with that, and I have to go, but the others may help you. [14:16:31] OK [14:16:37] maybe the file phython2.out can help you? [14:16:45] There is maybe the abort reason [14:17:14] look for files ending with .out and .err in your tools home [14:17:55] Go to root than insert like "python2.out"? [14:18:08] Have you pasted the original command above? [14:18:09] then [14:18:15] look above [14:18:16] no, this file will be automaticaly inserted, when you run your job [14:18:49] i just insert "jsub -once python ~/data/project/altobot/core/pwb.py replace -family:wiktionary -transcludes:예문 "{{예문}}" "{{풀기:예문}}" [14:19:45] I can see a file named python2.out and python2.err, but I can' read them. Read them, there you can find the error [14:20:01] Where is that file? [14:20:11] /data/project/altobot [14:20:34] The "~" is wrong, since you like to have the path executed relative to root [14:21:00] I think I works without writing /data/project... [14:21:04] just try: [14:21:17] There is filename "core python2.err replace.err replica.my.cnf public_html python2.out replace.out" [14:21:30] try python2.err [14:24:01] tools.altobot@tools-bastion-01:~$ python python2.err File "python2.err", line 1 /usr/bin/python2.7: can't open file '/data/project/altobot/replace.py': [Errno 2] No such file or directory ^ SyntaxError: invalid syntax [14:24:58] tools.altobot@tools-bastion-01:~$ python replace.err File "replace.err", line 1 Traceback (most recent call last): ^ SyntaxError: invalid syntax [14:26:03] That is correct work "python python2.err"? [14:27:34] Sorry, don't try to execute it with python, just paste the content of the files here (command: cat python2.err) [14:29:14] "/usr/bin/python2.7: can't open file '/altobot/core/pwb.py': [Errno 2] No such file or directory" [14:29:37] i search file at core folder "ls" [14:29:41] with "ls" [14:29:50] but i find that pwb.py [14:33:36] I remeber I had the same problem a few weeks ago [14:34:29] My solution was a little shellscript just calling python with the script [14:36:12] Look at /data/project/jogobot/charts.sh [14:37:36] There is no file "charts.sh" [14:39:16] what do you get when executing "cat /data/project/jogobot/charts.sh" ? [14:39:40] May i change "jogobot" to "altobot"? [14:40:17] "cat: /data/project/altobot/charts.sh: No such file or directory" [14:40:51] No, do not change it! [14:40:58] ....ok [14:41:18] I just want to give you an example for a script [14:41:40] i had executed file you said [14:42:02] #!/bin/bash source /data/project/jogobot/.bash_profile echo $( date +"%Y-%m-%d %H:%M:%S" ) UTC >> /data/project/jogobot/cron.log /usr/bin/python3.4 /data/project/jogobot/bot/charts/charts.py -page:"Benutzer:JogoBot/Charts" [14:42:32] Then it worked it i want to work? [14:43:45] No, its an example, you can copy it, remove everything except for the first and the last line and put your python command in last line [14:45:32] so you are saying copy your file to my folder "altobot"? [14:45:41] yes [14:46:39] then? [14:46:55] tools.altobot@tools-bastion-01:~$ cp /data/project/jogobot/charts.sh charts.sh [14:47:00] tools.altobot@tools-bastion-01:~$ ls charts.sh public_html python2.out replace.out core python2.err replace.err replica.my.cnf [14:47:20] open the file with an editor, nano for example [14:47:56] i opened charts.sh "cat charts.sh" [14:48:51] cat just prints the file to your terminal, for editing it you need another tool [14:49:12] a simple editor is nano, execute "nano charts.sh" [14:49:27] i did it [14:50:04] ok, now navigate to the second line with the arrow keys [14:50:21] Ok, and then? [14:50:46] replace jogobot with altobot [14:51:14] all word "jogobot" have to replace to "altobot"? [14:51:27] no, just in this line now [14:51:37] Ok. i write that [14:52:02] then you can remove the line beginning with echo, we don't need it now [14:52:20] remove line "echo"? [14:52:35] i tried [14:53:34] ok, before continuing press [CTRL] + [O] [14:53:57] then save name with previous name? [14:54:23] giv it another one, for example "replace-test.sh" [14:54:56] then go to out? [14:55:09] No, wait a moment pls [14:55:34] What can i do now? [14:57:06] okay, then go to last line and replace this with your command [14:57:07] /usr/bin/python /data/project/altobot/core/pwb.py replace -family:wiktionary -transcludes:예문 "{{예문}}" "{{풀기:예문}} [14:58:36] line "/usr/bin/python3.4 /data/project/jogobot/bot/charts/charts.py -page:"Benutzer:J$" have to deleted? [14:58:53] yes, replace it with you command [14:59:02] your* [14:59:16] i worked [14:59:40] press [CTRL] + [O], and save it [14:59:50] i saved [15:00:24] okay now exit nano with [CTRL] + [X] [15:01:04] and then? [15:01:11] then run "chmod +x replace-test.sh to make it executable [15:03:19] I run "chmod +x replace-test.sh" [15:04:48] a bit strange, it has overwritten all other permission on this file [15:05:48] you can correct that with "chmod 755 replace-test.sh" [15:06:34] it run [15:07:23] ok, now we can jsub it [15:08:15] tools.altobot@tools-bastion-01:~/core$ jsub -once pwb.py replace -family:wiktionary -transcludes:예문 "{{예문}}" "{{풀기:예문}}" [15:08:22] Your job 468528 ("pwb") has been submitted [15:08:22] jsub - N replace_test -mem 256m /data/project/altobot/replace-test.sh [15:10:29] This should not have worked right, pls try my command [15:10:44] just copy "jsub - N replace_test -mem 256m /data/project/altobot/replace-test.sh"? [15:10:52] yes [15:11:35] it not worked and displayed how to work jsub [15:12:21] i think you said not "- N", so "-once" [15:12:34] sorry, my mistake: "jsub -N replace_test -mem 256m /data/project/altobot/replace-test.sh" [15:13:08] Your job 468630 ("replace_test") has been submitted [15:13:24] it display that [15:14:36] There is no job 468630 at http://tools.wmflabs.org/?status [15:14:56] and i insert commend qstat [15:15:07] but it not display anything [15:15:22] ok, lets checkout replace_test.err, could you execute "chmod o+r replace_test.err" so I can reed it [15:17:23] i think you did not teach about ".bash_profile" [15:17:42] "/data/project/altobot/replace-test.sh: line 2: /data/project/altobot/.bash_profile: No such file or directory" [15:18:15] "/data/project/altobot/replace-test.sh: line 4: unexpected EOF while looking for matching `"'" [15:18:22] "/data/project/altobot/replace-test.sh: line 5: syntax error: unexpected end of file" [15:18:59] sorry, i don't expected you have none [15:19:38] so, what is .bash_profile? [15:19:51] create a file called .bash_profile and add following line to it [15:19:53] export PYTHONPATH=/shared/pywikipedia/core:/shared/pywikipedia/core/externals/httplib2:/shared/pywikipedia/core/scripts [15:20:22] create ".bash~" and edit? [15:20:37] bash_profile is a preconfiguration for your terminal environment [15:20:57] no, you can type "nano .bash [15:21:11] sorry stop [15:21:30] nano .bash_profile [15:23:39] but, wait you have installed pywikibot localy, you don't need it [15:24:01] you can just remove the second line from replace-test.sh [15:24:55] And there is a missing " at the end of replace-test.sh [15:27:37] i edit error line then execute what you said [15:27:51] The error displayed is...so long [15:28:11] The summary message for the command line replacements will be something like: 로봇: 자동으로 텍스트 교체 (-{{예문}} +{{풀기:예문}}) Press Enter to use this automatic message, or enter a description of the changes your bot will make: Traceback (most recent call last): File "/data/project/altobot/core/pwb.py", line 248, in if not main(): File "/data/project/altobot/core/pwb.py", line 242, [15:28:41] and blablabla... [15:28:55] Can i upload message at phab? [15:29:36] I can read the file [15:29:54] ok [15:30:13] See my replace_test.err [15:32:15] I'm not sure, but I suppose the problem is that we try to run the script in an interactive mode, which is not compatible with sending it to the grid with jsub [15:33:44] Wheatley_: can you pastebin the full traceback? [15:33:58] my error message? [15:34:08] Yeah [15:34:13] at phab? [15:34:26] or pastebin.com or whereever [15:35:02] http://pastebin.com/CRLPRAmg [15:35:58] Wheatley_: root cause is your running the bot in manual mode, via grid, which will not work [15:36:49] I think there is an -always flag or similar that need to be used for automatic replacements [15:38:03] i insert " jsub -N replace_test -mem 256m /data/project/altobot/replace-test.sh" [15:38:53] you have to add it in replace-test.sh [15:39:26] What add and where? [15:39:46] Wheatley_: how much python do you understand? [15:39:59] beginner [15:40:22] Wheatley_: can you pastebin the contents of the .sh ? [15:40:37] replace_test.sh? [15:40:45] yes [15:40:52] Betacommand: You can read it /data/project/altobot/replace-test.sh [15:40:57] Ill walk you thru what the command mean [15:41:21] see http://pastebin.com/B6wMJfkZ [15:41:40] Jogo-obb: That would require me to have an active SFTP connection, which I dont have ATM [15:42:07] Ok, We added the shell script since calling python directly via grid is ugly [15:42:38] Jogo-obb: No its not, its really really easy [15:43:06] I never get it to work correctly for me oo [15:44:04] Jogo-obb: here is a crontab entry: [15:44:06] 0 20 * * * jsub -once -mem 1G -N NFC9 python /data/project/betacommand-dev/svn_copy/sql_nfcc9.py [15:44:36] has been working since the early days of labs without much issue [15:45:01] Also having your bot directory world readable can be a security issue [15:45:21] Wheatley_: first thing is first there are several changes I would make [15:45:56] you do not have an edit summary defined, the default summary isnt that useful [15:46:06] Betacommand: well, OK, I've tried it with python [15:46:23] python3, and didn't get it to work [15:46:27] i have to add tag -summary:? [15:46:32] Ah, python3 [15:46:41] Wheatley_: yes [15:47:30] Wheatley_: pwb.py replace -family:wiktionary -transcludes:예문 "{{예문}}" "{{풀기:예문}}" -summary:"Reason for edit" -always [15:48:08] Wheatley_: its called a parameter, not a tag [15:48:24] i don`t want to insert any summary [15:48:51] Wheatley_: that will violate the bot policy [15:48:58] ok [15:49:58] Wheatley_: why? [15:50:02] Jogo if you have any python 2 questions let me know [15:50:14] Every edit should has one especially bot edits must have one [15:50:57] I just try jsub -once, not -N "jsub -once python pwb.py replace -family:wiktionary -transcludes:예문 "{{예문}}" "{{풀기:예문}} -summary:"위키낱말사전 틀 체계를 걷어내기 위함" -always" [15:51:10] And it display ">" [15:51:54] it worked? [15:51:59] No [15:52:17] what it means? [15:52:36] Wheatley_: you have an invalid command that hasnt been finished yet [15:53:39] jsub -once -N replace_test jsub -once python pwb.py replace -family:wiktionary -transcludes:예문 "{{예문}}" "{{풀기:예문}}" -summary:"위키낱말사전 틀 체계를 걷어내기 위함" -always [15:56:30] When i run "jsub -once python pwb.py replace -family:wiktionary -transcludes:예문 "{{예문}}" "{{풀기:예문}}" -summary:"위키낱말사전 틀 체계를 걷어내기 위함" -always" [15:56:32] Wheatley_: would you like me to break down that string to help you understand what it means? [15:57:01] ?? [15:57:35] Wheatley_: what happens when you run that command? [15:57:50] Traceback (most recent call last): File "/data/project/altobot/core/scripts/replace.py", line 147, in import pywikibot ImportError: No module named pywikibot [15:58:26] Sorry, I have to go now [15:59:32] Wheatley_: jsub -once -cwd -N replace_test jsub -once python pwb.py replace -family:wiktionary -transcludes:예문 "{{예문}}" "{{풀기:예문}}" -summary:"위키낱말사전 틀 체계를 걷어내기 위함" -always [16:00:21] 6Labs, 7Database: Lots of rows are missing from enwiki_p.`revision` - https://phabricator.wikimedia.org/T115207#1718098 (10Krenair) Query in P2180 returns 1185 rows in labs but only 16 in production. [16:03:04] it display " -bash: /usr/bin/jsub: No such file or directory" [16:04:23] what does pwd return? [16:05:03] what pwd? [16:05:09] its a command [16:06:01] it was not ask any pwd [16:06:23] Wheatley_: pwd isnt about a password [16:06:43] its Path to Working Directory command built into linux [16:07:47] You said like this " tools.altobot@tools-bastion-01:~$"? [16:08:03] Ah [16:08:05] thats why [16:08:34] Wheatley_: http://pastebin.com/2EWcakwS is what pwd returns for me [16:08:54] Wheatley_: when you run a bot you must call it from the directory where the file is located [16:08:59] "/data/project/altobot" [16:10:28] Wheatley_: can you run "ls" and pastebin the results? [16:10:54] http://pastebin.com/yq9JXaU7 [16:11:12] Ok [16:11:15] so: [16:11:19] cd core [16:11:31] jsub -once -cwd -N replace_test jsub -once python pwb.py replace -family:wiktionary -transcludes:예문 "{{예문}}" "{{풀기:예문}}" -summary:"위키낱말사전 틀 체계를 걷어내기 위함" -always [16:13:03] When i open replace_test.err [16:13:12] it display "-bash: /usr/bin/jsub: No such file or directory" again [16:13:45] Wheatley_: can you add me to your tool? [16:14:29] how to add? [16:16:33] https://wikitech.wikimedia.org/w/index.php?title=Special:NovaServiceGroup&action=managemembers&projectname=tools&servicegroupname=tools.altobot&returnto=Special%3ANovaServiceGroup [16:17:13] i add [16:17:20] Betacommand [16:17:44] yes [16:17:46] i add [16:18:57] see https://wikitech.wikimedia.org/wiki/Special:NovaServiceGroup and press ctrl+f "altobot" then you can find i add your name [16:24:03] oh [16:24:06] Wheatley_: fixed [16:24:09] jsub -once -cwd -N replace_test python pwb.py replace -family:wiktionary -transcludes:예문 "{{예문}}" "{{풀기:예문}}" -summary:"위키낱말사전 틀 체계를 걷어내기 위함" -always [16:24:11] How did you do that? [16:24:15] that is all you need to run now [16:24:34] Wheatley_: you had a few extra files that where causing an issue [16:24:45] cleaned up the mess and ran the command [16:25:07] Ok. Thank you [16:25:34] You remove word "jsub -once" [16:25:59] that was the problem? [16:27:53] it was duplicated [16:29:53] what duplicated? [16:30:57] Wheatley_: jsub -once was in the command two times [16:32:14] ah [16:34:36] See your talk page betacommand [16:35:16] So can i remove your name from my group now? [16:35:38] Ill do it [16:36:04] ah [16:36:07] Just waiting to ensure that was all you needed :) [16:36:08] Thx [16:37:50] Maybe that work will not be end soon [16:39:40] You see Korean wiktionary, There article`s composed is disparate with other wiktionary projects [16:40:08] I will replace to standard [16:40:14] all of articles [16:46:10] Wheatley_: do you have consensus to do that? [16:46:39] Yes [16:50:17] I have to go now [16:50:19] bye bye [16:51:48] Hi, after enabling the role role::labs::mediawiki_vagrant I can not run puppet anymore ( sudo puppet agent -tv) Error: Could not retrieve catalog from remote server: Error 400 on SERVER: OS ubuntu >= trusty required. does labs-vagrant require ubuntu? [16:53:16] ok "https://wikitech.wikimedia.org/wiki/Help:MediaWiki-Vagrant_in_Labs" says it MediaWiki-Vagrant will not run on the default debian-8.1-jessie base image. [17:04:37] (03PS1) 10Alex Monk: Add blank mx secrets [labs/private] - 10https://gerrit.wikimedia.org/r/245139 (https://phabricator.wikimedia.org/T87848) [17:14:14] (03CR) 10Alex Monk: "Cherry-picked on deployment-puppetmaster" [labs/private] - 10https://gerrit.wikimedia.org/r/245139 (https://phabricator.wikimedia.org/T87848) (owner: 10Alex Monk) [17:14:36] Coren, ? https://tools.wmflabs.org/wsexport/ since 2 days, can you restart the webservice? [18:42:53] 10Tool-Labs-tools-Other: wsexport webservice down; restarts by webservicewatcher fail - https://phabricator.wikimedia.org/T115221#1718293 (10valhallasw) 3NEW [18:44:22] 10Tool-Labs-tools-Other: wsexport webservice down; restarts by webservicewatcher fail - https://phabricator.wikimedia.org/T115221#1718300 (10valhallasw) But no relevant info in error.log: ``` 2015-10-10 08:00:16: (mod_fastcgi.c.2673) FastCGI-stderr: PHP Stack trace: 2015-10-10 08:00:16: (mod_fastcgi.c.2673) Fast... [18:45:19] 10Tool-Labs-tools-Other: wsexport webservice down; restarts by webservicewatcher fail - https://phabricator.wikimedia.org/T115221#1718302 (10valhallasw) And resolved by ``` tools.wsexport@tools-bastion-01:~$ webservice restart Webservice not running, starting. tools.wsexport@tools-bastion-01:~$ qstat job-ID pr... [19:14:10] 6Labs, 10Tool-Labs: webservicemonitor: AttributeError: 'module' object has no attribute 'utcnow' - https://phabricator.wikimedia.org/T115225#1718359 (10valhallasw) 3NEW [19:15:06] 6Labs, 10Tool-Labs: webservicewatcher does not log exceptions to user log - https://phabricator.wikimedia.org/T115226#1718369 (10valhallasw) 3NEW [19:16:17] 6Labs, 10Tool-Labs: create monitoring for webservicewatcher - https://phabricator.wikimedia.org/T115227#1718375 (10valhallasw) 3NEW [19:19:28] 6Labs, 10Tool-Labs: webservicemonitor: AttributeError: 'module' object has no attribute 'utcnow' - https://phabricator.wikimedia.org/T115225#1718391 (10valhallasw) I have now manually restarted these webservices with `webservice restart`. [21:20:19] 6Labs, 10Tool-Labs: Webservice on Tools Labs fails repeatedly - https://phabricator.wikimedia.org/T115231#1718509 (10russblau) 3NEW [21:32:18] 6Labs, 10Tool-Labs: webservicemonitor: AttributeError: 'module' object has no attribute 'utcnow' - https://phabricator.wikimedia.org/T115225#1718529 (10yuvipanda) p:5Triage>3Unbreak! This breaks webservicemonitor. /cc @Coren [22:17:10] 6Labs, 10Tool-Labs, 5Patch-For-Review: webservicemonitor: AttributeError: 'module' object has no attribute 'utcnow' - https://phabricator.wikimedia.org/T115225#1718570 (10yuvipanda) This patch should be good enough, need to merge, rebuild package and deploy it. [22:17:19] 6Labs, 10Labs-Infrastructure: Can't delete NovaProxy instance with malformed DNS hostname - https://phabricator.wikimedia.org/T69927#1718572 (10Krenair) Confirmed this is still an issue [22:17:33] 10MediaWiki-extensions-OpenStackManager: Can't delete NovaProxy instance with malformed DNS hostname - https://phabricator.wikimedia.org/T69927#1718574 (10Krenair) [22:31:21] hi! [22:31:36] i'm trying to get .cgi to work on labs - is there something i'm missing? [22:38:59] 6Labs, 10Tool-Labs, 5Patch-For-Review: webservicemonitor: AttributeError: 'module' object has no attribute 'utcnow' - https://phabricator.wikimedia.org/T115225#1718584 (10yuvipanda) I merged, rebuilt package and hand copied it to tools-services-02 and restarted it. Things seem to work now. [22:42:04] 6Labs, 10Tool-Labs, 5Patch-For-Review: webservicemonitor: AttributeError: 'module' object has no attribute 'utcnow' - https://phabricator.wikimedia.org/T115225#1718587 (10yuvipanda) Tested by killing the wikibugs webservice manually, it got restarted. [22:45:54] 6Labs, 10Tool-Labs, 5Patch-For-Review: webservicemonitor: AttributeError: 'module' object has no attribute 'utcnow' - https://phabricator.wikimedia.org/T115225#1718604 (10yuvipanda) Still left to do is to add the built package to aptly and verify that it installs from there properly. You can find the newly b... [22:46:06] anyopne? [22:54:19] !log tools delete service.manifest for tool wikiviz to prevent it from attempting to be started. It set itself up for nodejs but didn't actually have any code [22:54:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [22:54:38] rschen7754: I'm on a flight so can't help much. I'll reccomend *not* using CGI, however. if you're using python there's the uwsgi setup. [22:54:41] oh well [22:54:44] I just missed him I guess [22:54:47] s/him/them/ [22:55:38] 10Tool-Labs-tools-Other: wsexport webservice down; restarts by webservicewatcher fail - https://phabricator.wikimedia.org/T115221#1718615 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Should be fixed now, see T115225 for followup [22:57:10] 6Labs, 10Tool-Labs: Webservice on Tools Labs fails repeatedly - https://phabricator.wikimedia.org/T115231#1718622 (10yuvipanda) This is possibly related to T115225? How long has this been happening? [23:00:40] 10MediaWiki-extensions-OpenStackManager: Can't delete NovaProxy instance with malformed DNS hostname - https://phabricator.wikimedia.org/T69927#1718627 (10Krenair) I was thinking this is just an OSM issue, but I've been playing around with code and curl requests etc. from silver and this might be an issue with d... [23:02:30] 10MediaWiki-extensions-OpenStackManager: Can't delete NovaProxy instance with malformed DNS hostname - https://phabricator.wikimedia.org/T69927#1718629 (10yuvipanda) It could also be an issue where in OSM where it tries to delete the entry on dynamicproxy-api, succeeds, and then tries to remove teh DNS entry its... [23:02:47] 10MediaWiki-extensions-OpenStackManager: Can't delete NovaProxy instance with malformed DNS hostname - https://phabricator.wikimedia.org/T69927#1718630 (10yuvipanda) p:5High>3Normal [23:30:39] 6Labs, 6Discovery, 10Wikidata, 10Wikidata-Query-Service: Wikidata Metrics - https://phabricator.wikimedia.org/T115120#1718647 (10Christopher) After researching this, I have discovered that the Munger that processes the RDF dump removes several ontology types (wikibase:Item, wikibase:Statement, wikibase:Ref... [23:32:44] 6Labs, 10Tool-Labs, 5Patch-For-Review: webservicemonitor: AttributeError: 'module' object has no attribute 'utcnow' - https://phabricator.wikimedia.org/T115225#1718650 (10coren) Err - As I said on IRC I was on it already; with a ~/tools-manifest_0.6_all.deb in //my// homedir. [23:34:17] 6Labs, 10Tool-Labs: Webservice on Tools Labs fails repeatedly - https://phabricator.wikimedia.org/T115231#1718651 (10russblau) I'm not entirely sure because I have been mostly offline myself for several days, but I noticed it definitely on Saturday 10 Oct. [23:39:34] PROBLEM - Puppet failure on tools-services-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]