IRC log of #schooltool for Monday, 2008-06-09

*** aelkner has joined #schooltool00:46
*** tdoggette has quit IRC00:53
*** didymo has joined #schooltool02:33
*** tdoggette has joined #schooltool04:22
*** tdoggette has quit IRC06:13
*** jfroche has quit IRC06:13
*** tdoggette has joined #schooltool06:20
*** jfroche has joined #schooltool06:20
*** tdoggette has quit IRC06:33
*** jfroche has quit IRC07:09
*** jfroche has joined #schooltool07:11
*** jfroche has quit IRC07:17
*** jfroche has joined #schooltool07:26
*** jfroche has quit IRC07:33
*** jfroche has joined #schooltool07:43
*** jfroche has quit IRC07:48
*** jfroche has joined #schooltool07:57
*** tdoggette has joined #schooltool08:21
*** jfroche has quit IRC08:28
*** jfroche has joined #schooltool08:38
*** tdoggette has quit IRC10:10
*** tdoggette has joined #schooltool11:53
*** tdoggette has quit IRC11:56
*** tdoggette has joined #schooltool11:57
*** didymo has quit IRC12:46
*** tdoggette has quit IRC14:01
*** ignas has joined #schooltool14:21
*** mgedmin has joined #schooltool14:31
*** th1a has joined #schooltool15:31
ignasth1a: hi15:47
th1ahi ignas.15:47
ignasth1a: should I just go and fix the CAS authentication? or are we going to leave it to aelkner?15:48
ignasit seems he is having a lot of problems with it :/15:48
th1aWell, I assume we'll be discussing that today.15:49
th1aCertainly if you can help him get unstuck, that would be good.15:49
th1aignas: Do you know what the specific problem is?15:51
*** wbrady has joined #schooltool16:01
ignasth1a: no, but i know that the code is complex, and the tests are missing16:04
ignasnot the code, but the CAS magic dnce16:05
th1aYeah... I think we have to have tests.16:05
ignaswell - adding tests is complicated too16:06
th1aI ran into the same kind of thing with SIF.16:06
th1aIf you don't know if the bug is in the client or server, you're humped.16:06
ignasyep, and testing with a server outside is giving you messy logs16:08
ignasbecause one part of the logs is coming from rubycas16:08
ignasand the other part from your application16:08
ignaswhich means - you don't know what happend first16:09
aelknerth1a, ignas: morning16:21
th1aGood morning aelkner.16:22
th1aA CAS gameplan seems to be the first order of business here.16:22
aelkneryeah, i figured16:22
aelkneri should give you some details of what i've discovered16:23
th1aGo ahead.16:23
aelkneri didn't want to get into it in my report to the SLA folks16:23
aelknerso first, if you are not signed on to CAS and you start the app16:24
aelknerit will correctly take you to the CAS login screen16:24
aelknerand if you put the right credentials in, it will take you back to the app16:24
aelknerso that part always worked16:24
aelkneronly problem with that part was what i found out was going on in the traces i wrote16:24
aelknernamely, i fond out that zope is weird about redirects16:25
aelknerwhen you say, request.response.redirect(some_url)16:25
aelkneri always assumed that zope would take that as a signal to abort that transaction16:26
aelknerand redirect16:26
*** jelkner has joined #schooltool16:26
ignasemm, why?16:26
aelknerwell, how would you know if you were really going to redirect to where you wanted to16:26
ignasi mean - you won't be able to store anything in the session or in the database and then redirect then ...16:26
aelknerif it just keeps going on executing more tranversal handlers16:27
aelknerin other words16:27
aelknerit starts traversing the app object16:27
aelknerit calls the authenticate method of your plugin16:27
aelknerwe say, "redirect to CAS server with this query string"16:28
aelknerit doesn't redirect at first16:28
aelknerit traverses to say, the person container, for the sake of an example16:28
aelknerthat causes it to call our authenticate method again16:29
aelknercausing us to redirect again!16:29
aelknerwith a different query string16:29
ignasare you sure it's happening in the same request then?16:29
aelkneryes, i'm sure16:29
aelknerbecuase i put a trace at the beginning of zope/publichser/publish.py16:30
aelknerin the publish method16:30
ignaspdb trace?16:30
ignasi see16:30
aelknerno way could i use pdb16:30
aelknerwell, in that case i could16:30
ignasif query strings are different - it's probably the second request, and you even though you have access to the app, just don't have access to the person container16:30
ignasif it's print16:30
ignasit can be comming from 2 different requests16:30
aelknercuase it only stopped there a couple of times16:30
ignashow do you know it's the same request if you are printing ?16:31
aelkneri don't understand the question16:31
aelkneri hit just one button in the browser16:31
aelkneradn watch the print statements16:31
ignasit still generates 2x requests16:31
ignaswith CAS authentication16:31
ignasSchooltool -> CAS -> Schooltool16:32
aelkneri understand that16:32
aelkneri expect that, in fat16:32
ignas2 redirects - 2 different print statements16:32
aelknerit's the calling of authenticate ten times that got me confused16:32
aelknerand agravated actgually16:32
aelknerbecause why would it not redirect to CAS when i ask it to16:32
aelknerinstead of coming back to authenticate once for each object in the traversal16:33
aelknerbefore leaving the publich method16:33
ignasis it really comming back for each object?16:33
ignasbecause a redirect is not "invoked" before the end of traversal16:33
aelkneryes, that's the benefit of my traces, that i can see what it's doing16:33
ignasso if it hits CAS 10 time16:33
ignashits CAS actually, not tries to redirect 10 times16:33
aelknerit doesn't hit the CAS server ten times16:34
aelknerit hits the authenticate method ten times16:34
aelknerad maybe i'll have to live with that16:34
ignashmm, interesting16:34
aelknerit's just really disturbing that you redirect only to find that you havce to redirect again16:34
ignaswell - that's the way CAS works, most of the time16:35
aelknerthat's not a CAS thing, it's a zope thing16:35
ignasand yes - at the moment schooltool authentication strategy is not optimal16:35
ignasit's schooltool thing actually16:35
aelknerwell, yes, that's the other issue16:35
aelknerno, i'm talking zope traversal16:35
aelknerwhen zope traverses16:35
aelknerit should notice that it's been redirected16:36
aelknerand stop!16:36
ignasno it should not16:36
ignasbecause there might be more operations to be performed16:36
ignasto the database even16:36
aelknermore travesal?16:36
aelknerif i'm traversing and object16:37
aelknerand i can't allow travesal to the root16:37
aelknerwhy should it keep traversing below the root16:37
aelknerif i say, redirect16:37
aelknerit should redirect16:37
ignasredirect does not mean - you can't access what's below16:38
aelknerwhat does it mean when you can't access the root?16:38
aelkneronly that?16:39
ignasredirect is not "signaling that"16:39
ignasthe question is - why is our authentication utility not saying "you can't access" properly, or why Zope is ignoring that16:39
ignasit's not related to redirects16:39
aelknerok, well, take the situation where the user just signs in16:40
aelknerthe first traversal is the root16:40
aelknerschooltool.cas authenticate method redirects to CAS16:40
ignasaelkner: do you know the difference between authentication and authorization?16:40
aelkneryeah, why?16:41
ignasunauthenticated does not mean - can't access16:41
aelkneri'm only talking authenticate method of the pluging16:41
ignasit just means - we don't know who you are16:41
aelknerright, the principal is None16:42
aelknerbecuase it needs to redirect to CAS16:42
ignasso it's our problem16:42
aelknerand get back the ticket16:42
ignasto say - we know we don't know who you are16:42
ignas10 times16:42
ignasand redirect only once16:42
ignasthough - we don't have to do that16:42
aelknerwe can just redirect the ten times as we do16:43
ignasit will only redirect once16:43
aelknerwell, i can live with that16:43
ignaswith the last value of redirect16:43
aelknerit just was not how i would expect redirect to work16:43
ignasthe redirect call is not "do a redirect"16:43
aelknerso even our success case was confusing16:43
ignasit's "remember - render a redirect http response"16:43
aelkneris saw the code16:44
aelknerit was stting up headers and all16:44
aelknerbut just not finishing the transaction16:44
aelkneri'm just not used to that paradigm16:44
aelkneranyway, i can live with that16:45
ignasbelieve me - you would not think that redirect acting as an exception is a good idea too16:45
* ignas had to deal with an http server that does that16:45
ignasdoing anything meaningful and then redirecting suddenly becomes a major pain16:45
aelknerand you had problems cleaning up the transaction, right?>16:46
ignasno, i had problems making the server not cleanup the transaction for things like "Add a person, then redirect to his information view" things16:46
aelknerright, you didn't want to abort16:46
ignasbut that was in a different language, though - it made me understand why this approach was picked16:47
aelknerthat's cool, i can see the benefit of the way it is now16:47
th1aSo does something else not work?16:47
aelknerwell, yes16:47
aelknerthe next thing to report is that when i click on the logout button16:47
aelknerit doesn't go to schooltool.cas logout view16:48
aelknerbut that is probably because16:48
ignasthat view "redirects immediately"16:48
ignasso - you can't see it16:48
aelknerit reaches the unauthenticate method16:48
aelknerwhich redirects to CAS16:48
aelknerit never reaches the logout view16:49
ignasyou mean the __call__ of it?16:49
aelkneri'm not sure of this one yet16:49
aelknerit may be a registration problem16:49
aelkneri was going to look into that soon16:49
aelkneri'll tell you what my trace revealed16:50
aelknerit calls the unathenticate method16:50
aelknerwhich redirects to CAS with login?...16:50
aelknerand the CAS sever omes up16:50
aelknerwhich is what we want, except16:50
ignasi am not sure really16:50
ignasit might be the second go16:50
aelknerit says, "you are already logged in as aelkner"16:51
ignasthen it's not working properly16:51
aelknerit shoudl say, "log in"16:51
ignasmaybe permissions on the logout view are broken16:51
ignasor something like that16:51
aelknercould be16:51
aelknerit's registered for *16:51
aelknermaybe it needs to be registered for the plugin16:51
aelknermaybe it would get found then16:51
ignasshould not be affecting it16:52
aelkneri was going to check that out today16:52
aelknerit's always a guessing game for me with registrations16:52
aelknerbecause there isn;'t any trace16:52
aelkneras to what zope is trying and failing at16:52
aelknerand then succeeding16:52
aelknerthat would be so nice16:52
aelknerpersonally, i am always able to get complex systems of my own creation to work16:53
aelknerand be trouble-shootable16:53
aelknerby using trace debugging16:53
aelknerthat i can turn on and off16:53
aelknerso if a problem happens that i didn't anticipate16:54
aelkneri can look at the trace and say, "oh, i see"16:54
aelknerbut zope doesn't provide such a thing for registration lookup16:54
aelknerthat's too bad16:54
ignasbelieve me - you don't want traces of "Adapting Foo to IBar got BarFooAdapter"16:54
aelknerwhy not16:55
ignasyou'd get thousands of them ...16:55
aelknerfor once transaction?16:55
ignasjust tracing SecurityPolicy gets you more data than you can handle most of the time, like a few hundered16:56
aelknerwell, perhaps16:56
aelknerthe third problem i wanted to report is16:57
aelknerthat when schooltool gets an authorized exception16:57
aelknerit wants to redirect to a login page16:58
aelknerour CAS plugin redirects to CAS16:58
aelknerbut CAS has no problem with the user16:58
aelknerso it just redirects back to us16:58
aelknerand we redirect to it16:58
ignasweird, must be a missing parameter then16:58
aelknerand this infinite loop gets stopped by the browser16:59
ignaswe should "force" the authentication16:59
ignasand I was sure we were ...16:59
aelknerafter about ten round trips16:59
Lumiereis it possible to tell cas 'insufficient' authorization?16:59
aelkneris that what "renew" is for?16:59
ignasLumiere: yes, you can "force" authentication16:59
ignasyes i think it's renew16:59
ignasor something like that17:00
ignasyeah, it's "renew"17:00
ignasyou know17:00
Lumierewhat does renew do if the person isn't authenticated to zope?17:00
ignasmaybe we are making the mistake of redirecting to CAS server17:00
aelknerlet me try it right now and tell you what my traces yield17:00
ignasinstead of redirecting to schooltool login page17:00
aelknergo ahead17:00
ignasyeah, it seems that "unauthorized" redirects to CAS17:01
ignaswhile it should probably redirect either to schooltool login or at least ask for a renew17:01
ignasthough i am not sure how to do it properly while maintaining the form information17:02
aelknerwell, redirecting to schooltool login would not work17:02
*** tdoggette has joined #schooltool17:02
aelknerbecuase then they get the schooltool auth17:02
aelknerinstead of cas auth17:02
*** wbrady has quit IRC17:02
ignasaelkner: the first redirect loses form information17:03
ignasunless you store it17:03
aelkneri understand your store form dat logic17:03
ignasthus - you should be very careful about what's being fone there17:03
aelknerit works fine17:03
aelknerperhaps it jsut isn;'t being used there17:03
ignasit is being used at the moment17:03
ignasbut if you redirect to schooltool login page - you will lose the form data i think17:04
th1aWhich form information?17:04
aelkneryou post something17:04
aelkneradn you want it to be there after you resolve the authentication17:04
aelknerthat was good code that ignas wrote17:04
aelknerto handle that situation17:04
ignasth1a: we are handling the case where you enter all the data for let's say person, click submit and get to a login page (session timeouts, insufficient privileges)17:04
ignasyou log in, and your form get's properly submitted17:04
ignasinstead of being lost17:05
th1aI guess timeout is a particularly important case there.17:05
aelknerthat's not even cas code, that's schooltool auth code17:05
aelknerthe cas plugin subclasses schooltool's17:05
th1aotoh, I'm not sure that insufficient privileges is.17:06
ignasI think losing data is bad, no matter why...17:06
th1aWell, it just depends on how many hoops we're creating for ourselves.17:07
ignasthis hoop is solvable from CAS plugin anyway17:07
ignasjust adding "renew" as a parameter in unauthorized method17:07
ignasshould work17:07
aelknerwell, i have to run that trace again17:08
aelknergive me a couple of minutes17:08
aelknerok, i see that it's not redirecting to CAS with renew17:09
aelknerso maybe just changing that would fix the infinite loop problem17:09
aelknerlet me look into that after the meeting17:10
ignasyeah, it should17:10
aelknerwhat about the logout view not getting called17:10
aelknerdo you have a recommendation17:10
aelknerfor how to trouble-shoot that17:10
aelknerignas: i'm guessing you're looking at schooltool.cas configure.zcml17:12
aelknerdo you see how the login and logout differ17:13
aelkneronly with the for being * for logout17:13
aelknershouldn't that be the same as for the login?17:13
ignasno difference17:13
aelknerwhat do mean17:14
aelknerthere is no difference or it's ok for them to be different?17:14
ignasok, replace it now that i think about it17:14
ignasi'd look for other logout views17:14
ignassee what they are on17:14
ignassee whether they are registered17:14
ignasor called17:14
ignasit might be that logout is on * for backwards compatibility17:15
ignasbut at the same time * will not conflict with a more specific logout view17:15
*** wbrady has joined #schooltool17:15
ignasso if there is a more specific logout view even if that view is registewred for IGenericAuthnticationPlugin17:15
ignasthe more specific one will get used17:16
ignasignoring the * view17:16
ignasso yeah - find out whether it's - problem with other view being used17:16
ignasor problem with permissions for logout view17:16
aelkneri think you answered my question anout the registration lookup logic of zope17:17
aelknerand i found the registration of schooltool's logout view17:17
aelknerit's for
aelknerwhich is in fact more specific than *17:17
aelkneradn since ICASAuthenticationPlugin is a subclass of it17:18
ignasyeah, it seems so17:18
ignasso yes - it will "win"17:18
aelkneri bet changing the reg will fix it17:18
aelkneri can do that after the meeting as well17:18
aelknerso i've reported the three issues now17:19
aelknerand i have a good feeling for how to address them for now17:19
aelknerfirst one was don't worry17:19
aelknerabout the mutiple redirectsd17:19
aelknerjust maek sure it works17:19
aelknersecond one is fix the registration17:19
aelknerthird one is fix the unathorized method to use renoew17:19
aelknerso that's all i have for now17:20
ignasth1a: should I report?17:21
ignasso my plan from add "SchoolYear" went to "fix timetables" and then to "fix timetable REST views"17:21
ignaswhich is what i am doing17:21
ignasthe REST views either have to get fixed17:21
ignasor I have to rewrite a bunch of functional tests17:22
ignasdon't know which one is easier at the moment, actively looking at tit17:22
ignasthe problem is - as i will add SchoolYear, and put terms in it, referring to terms by term_id17:23
ignaswill not be viable, nor will it be a good idea17:23
ignasso i have to fix timetable -> term/school_timetable relationships first17:23
ignaswhich means - getting rid of the "relationship is encoded in __name__ of the timetable"17:23
ignasthat was bothering me for ages17:24
ignasfirst i will make __name__ used only through term and school_tt attributes of the timetable17:24
ignasto have only 1 point in code that has to be fixed17:24
ignasthen i will convert them to direct references (temporarily)17:24
ignasthen will make __names__ not important17:25
ignaslike section __names__ that are 1,2,3 ...17:25
ignasand then add relationships so we'd avoid direct references17:25
ignasand would get backreferences too17:25
ignasas in - terms would know which timetables are using them directly17:25
ignasthen i will add schoolyears and move terms into schoolyears17:26
ignasthat's kind of it, spent a lot of time finding out which parts of the system are relying on terms being in a flat namespace17:26
ignasand refactoring timetable code17:26
ignasthat's it17:27
th1aGlad you're digging into it.17:28
th1aLumiere: Anything to add?17:28
Lumierenot really17:30
Lumiereeldar is making great progress on his tasks17:30
th1aOK then.  Sounds like ignas and aelkner have plenty to sink your teeth into this week.17:30
ignasLumiere: what is he working on?17:30
th1aHave a good one!17:30
* th1a drops the bag of gravel.17:30
Lumierehe's still working through replacing all the competency back end17:30
aelkneri attend the wed meeting each week17:31
Lumiereit's one of those things that changes everything17:31
aelknerso i'm on top of what eldar is up to17:31
wbradyjelkner: ayt?17:31
Lumiereright now he's finishing up rewriting the import of competencies into courses17:31
Lumiereand then rewriting importing into skill drivers17:31
th1aLumiere: I'd guess it is making him feel like the whole thing is getting more sane.17:32
Lumierefrom there he'll do gradebook likely17:32
aelknerimporting is not the right term17:32
aelknerit's assinging17:32
Lumiereaelkner: yes17:32
Lumiereimporting is easier to type ;)17:32
aelknerit's one of your habits, using that term :)17:32
Lumiereth1a: yes, it also will hopefully make it more bulletproof17:32
Lumiereold terms die hard17:33
aelknerit wil be sane17:33
aelknerand reliable17:33
LumiereI still have trouble calling jr developers anything but interns17:33
aelkneryeah, we all have that one17:33
aelkneralthough i've been getting better17:33
aelknerit just takes time17:33
aelknerbut you shoudl stop thinking about importing17:33
Lumiereimport is similar for welsh and I... we both have called it that for 3-5 years17:33
aelknerit will take time i guess17:33
aelkneri oticed that welsh does the same thing17:34
aelknerit's a major change to the way cando works17:34
Lumiereanyways... good progress is being made17:34
Lumierewe start our sprint in a couple weeks17:34
aelkneryes, we have a good plan, and we keep on top of the progress each week17:34
Lumiereand hopefully will come out of that with substantial functional testing17:34
Lumiereand unit testing where possible17:35
aelknerit's nice to have such a report17:36
aelknerbut i don't have time to look into how it's created right now17:37
aelknerso i can't say confidently, that 100% coverage means just that17:37
aelknerbut we can get into this during the week leading up to the sprint17:37
aelkneri don't want to distract myself from the CAS work in the meantime17:38
*** jboisture has joined #schooltool18:12
jboistureI'm having a problem with the zope test browser. Can anyone help?18:13
jboisturewhen I try to run'') I get an error that says zope.configuration.exceptions.ConfigurationError: No registered publisher found for (GET/)18:14
jboisturedoes anyone know what this means18:15
Lumiereremove the /18:19
jelknerignas: ping18:21
ignasjelkner: pong18:21
jelknerwe have a new developer who has been studying zope3 furiously for the past several weeks18:21
ignasjboisture: i think that there is more than one browser class18:21
ignasand the default class18:21
ignasonly works with "local" hosts18:21
ignaswhich is - your Zope3 application18:22
ignasyou must use a different implementation18:22
ignasif you want to test a remote server18:22
jboistureok I'm going to test this ok again thanks18:22
jboistureI tried'http://localhost:8999') and got the same problem18:24
ignashow and where from are you running testbrowser?18:25
ignasis it inside of a functional test?18:25
jboistureI'm just running it in a python interpreter in the home folder of my time clock instance18:26
ignasyou see18:27
ignasin functional tests18:27
ignasthe publisher gets setup by all the fancy functional test machinery18:27
ignasso it works in there18:27
ignasthe "test testbrowser"18:27
ignasand if you want it to run in a non functional test environment18:27
ignasyou have to use a different TestBrowser class18:28
Lumieremy suggestion is to write a functional test18:28
Lumiereand call it18:28
Lumiereignas: would pdb's settrace work in a functional test?18:28
Lumiereor start trace (don't remember name offhand)18:29
th1ajelkner: The new developer is jboisture?18:29
jelknerth1a: si18:29
ignasLumiere: nope i don't think so18:30
jelknerth1a: did you see my email to david trask?18:30
th1aYes, thanks.18:31
aelknerignas: i'm going to need to use TestBrowser to visit pages outside of schooltool18:31
aelknerwhat implementation are you refering to?18:32
jelkneri hope they follow up with the XO idea18:32
ignasand no - you DON'T!18:32
ignasunless you really want to make tests depend on outside servers and internet connection18:32
ignaswhich is BAD18:32
aelknerone of my projects for this summer will be to propogate changes to schooltool to Moodle and Drupal18:32
aelknerso if you create a person in schooltool, it gets created in the other apps18:33
aelknerhwo would you suggest doing that?18:33
ignasyou mean - testing that18:33
ignasit is tricky18:33
aelkneri mean doing it18:33
*** mattva01 has joined #schooltool18:33
ignasaelkner: emm, no idea, i mean - the perfect case is having them all use the same data structure for persons18:33
ignasaelkner: so the totally perfect case would be having SChoolTool use the sql table when storing persons18:34
aelknerthat was my first idea18:34
aelknerbut th1a didn't like it18:34
ignasaelkner: then if you can't do that, you opt for synchronization using crontab18:34
ignaswhich is simpler but not as reliable18:34
ignasthan you can go for webservices and zc.deferred18:35
th1aWe're going for very simple first.18:35
ignasso it really depends on what's available18:35
aelknerth1a: ignas suggests it's impossible to use TestBrowser18:35
ignaswell - i am suggesting that it's a major pain18:35
ignasyou have to find out how to do "teardown" for moodle18:35
ignasand "teardown" for drupal18:35
th1aYou aren't going to write tests for the whole thing.18:36
ignasif you want to functionally test it18:36
ignasnow if it's crontab approach18:36
ignasthen you only test your new script18:36
ignasthat lists all persons (part that connects to schooltool)18:36
ignasand adds these persons to SQL18:36
ignasin both cases - you stub out the SQL server and schooltool server18:36
ignasand test your utility without using the real stuff18:37
th1aI would rather avoid directly sticking our fingers into the database.18:38
ignasth1a: well, i guess yes, you can do that using moodle and drupal html UI18:38
ignasor some API if they have one18:38
aelknerso that brings us back to ,my question about TestBrowser18:39
th1ahtml is probably better.18:39
aelkneryou said it isn't something that can be used for external sites?18:39
ignasth1a: xml or jason API is better18:39
ignasaelkner: it can be used18:39
th1aWell, if those were well documented.18:39
ignaswell - HTML UI is always less documented than XML API's are ...18:40
ignasaelkner: i was saying - you should not do that in a functional test, unless you really really know what you are doing18:40
th1aNot necessarily.18:40
*** jboisture has quit IRC18:40
th1aNot necessarily documented, that is.18:40
aelkneri don't see how to functionally test something that interacts with another app18:40
aelknerbecause the state of the other app is not predictable18:41
aelkneror even its existense18:41
aelknerthe same goes for testing CAS18:41
aelknerunless we have a fake CAS server that is part of schooltool.cas18:42
ignasaelkner: precisely, a fake server18:42
aelknerthat gets started in the functional tests for schooltool.cas18:42
ignasor a fake utility that immitates server18:42
ignasbetter - fake utility18:42
aelkneryou keep mentioning utilities18:42
ignaswell yes - you add a utility that does all the real http calls18:43
aelknerbut we're talking to servers18:43
ignasand use that in an application18:43
ignasin tests - you replace it with one that does no real http calls18:43
ignasthe point is18:43
ignasif you call an external service18:43
ignasyou have to start it up18:43
ignasin your test setup18:44
ignasdo you want to write a functional test18:44
ignasthat starts up a *clean* moodle instance?18:44
ignasand then tears it down when the test is over?18:44
aelknercertainly not18:44
ignas(we have done some simmilar things in PoV)18:44
ignasso yes it's possible18:44
ignaswe used Zope3 testbrowser to test roundup18:44
ignasso test setup was starting up a clean roundup instance18:44
ignasthat we did our testing on18:45
ignasand was deleting it after the test18:45
aelknerso that test would take hours, right?18:45
ignasquite fast actually18:45
aelknerwhat is roundup18:45
ignaswe used it before launchpad18:46
ignasfor schooltool18:46
aelknerso what did it take to make it clean?18:46
ignascan't recall, maybe deleting the database, or was it just removing all items18:47
th1aI would say the goal is to test it enough that you confirm that you're actually sending out the message you think you're sending,18:49
th1aand that if you get the response you expect, that you do what you're supposed to do.18:49
aelknerso you're suggesting the idea of a utility that fakes talking to Moodle and Drupal18:50
aelknerand just gives responses of success or failure18:50
ignassomething like that18:50
ignasi am suggesting a separate application18:50
ignasand crontab18:50
ignasunless you want to learn zc.deferred18:50
ignasbecause as you know Zope3 has only 4 worker threads by default18:51
ignasso if your http calls will timeout or hang up18:51
ignas4 stalled requests will make the application unusable18:51
ignasfor anyone18:51
aelknerall the more reason to talk to the sql database directly18:52
ignasaelkner: that one is tough18:52
ignasbecause you have to handle double commit18:52
*** tdoggette has quit IRC18:52
ignasas in - if your zope code fails18:52
ignasyou must revert both18:53
ignasschooltool ZODB transaction18:53
ignasAND SQL transaction18:53
ignasif ZODB transaction works, but SQL transaction fails  - you must revert ZODB transaction18:53
th1awhat about zc.deferred?18:53
ignasor else you get broken state again18:53
ignasth1a: not sure i understand your question...18:54
th1aWhat would that do for us?18:54
ignaswell - i have no idea how it works18:54
ignasi have no idea whether it's stable18:54
ignasbut what it would allow us to do18:54
ignasis make HTTP calls18:54
ignasto outside applications and have them not affect schooltool in any way18:55
ignasit is used to run looong running things in the background, like "generate a bunch of PDF documents"18:55
ignasor make 20 http requests18:55
ignasin the background18:55
ignasso Zope keeps on going, and the operations that add persons get done on their own pace18:56
aelknerand error recovery?18:56
aelknerif Moodle gives and error?18:56
ignasaelkner: well - then you are in trouble anyway, you have to implement it manually18:57
ignasaelkner: because it depends on when you got the error18:57
aelknerthat's what i was getting at18:57
ignasand whether the person was added18:57
aelknerthere's no way to protect against erros18:57
ignasor only half added18:57
aelknerin a system other than the one you're in18:57
th1aaelkner: You should probably investigate zc.deferred.18:57
aelknerthat's why things need to be rerunnable18:58
ignasso yeah, a separate script that is executed by crontab that "syncs" both schooltool and moodle18:58
aelkneri probably will need it18:58
ignasis something that you can just rerun18:58
ignasand if it fails - you fix it, and run it again18:59
ignasit's brute and simple18:59
aelknerby crontab, what is the job in question doing?18:59
aelknervisiting a schooltool page18:59
aelknerthat does the sync18:59
ignasit lists all persons (like the xml page that lists all of them)18:59
ignasthen lists all persons in moodle19:00
ignasthen compares them and comes up with an action plan19:00
ignaslike - delete 2 persons, add 3 persons19:00
ignasand then executes that plan19:00
aelknerlet me see if i understand your meaning19:00
aelknerthe cron job is a python script19:00
aelknerthat visits schooltol, say using urllib19:00
aelknerand visits Moodle19:01
aelknerthe same way19:01
aelknerthen does the comparison19:01
aelknerand then visits the appropriate app, navigating the UI to insert the data19:01
aelknerdo i get your meaning?19:02
*** alga has joined #SchoolTool19:02
ignasit's simple, and you know what to do if it fails, and if you run it twice - nothing bad happens19:03
ignasthe downside is lag19:03
ignasbetween one system getting updated and the other seeing changes19:03
ignasso you might have to wait 5 minutes, and hour, a day19:04
ignasor whatever the amount you want19:04
aelkneri've developed a system that does the same thing19:04
ignasuntil changes propagate19:04
th1aAs long as it works, nobody will really complain about lag.19:04
aelknerit moved data from a mainfrmae to a pc sql database19:04
aelknerand, yes, there was a time lag19:04
aelknerand, yes, people didn't complain19:04
aelkneri had it run every two minutes19:05
ignaswell - you should lag a bit more19:05
ignasschooltool will not like you listing 1000 users19:05
ignasevery 2 minutes19:05
th1aOnce a night would probably be fine.19:05
aelknermy system didn't list all users19:05
aelknerit had a table of changed records19:05
aelknerand only updated those changed since the last successful update19:05
aelkneri could do the same in schooltool19:06
aelknerhave event subscribers that write to a change queue19:06
aelknerthe cron job will only ask for those records that changed since a certain date/time19:06
ignasi'd suggest not complicating it19:06
aelknerif nothing has changedd in last two minutes that would be nothing19:06
aelknerit was a very reliable systgem19:07
aelknerand only had time lag of two minutwes19:07
ignasi do understand it, but only having integration working IF that changelog is in the system19:07
ignasand only working with users added after the changelog activation19:07
ignas+ problems with reruns19:07
Lumierethe other thing you could do19:07
ignasif you list all of them all the time19:07
ignasit can't fail19:07
Lumiereis let it run once a night19:07
Lumiereand let the sysadmin run it19:08
ignasif you only react to "added" "removed" events19:08
Lumierewhen there was a need19:08
ignasyou can mess it up19:08
ignasif you script fails19:08
ignasto add a person19:08
ignasfor example, you don't know which "added" are "added" and which ones are "old"19:08
aelkneradd and change should not be distinguished19:08
aelknerin either case19:08
aelknerthe record is sought out in the target19:09
aelknerif it doesn't exist19:09
aelknerit gets added19:09
aelknerotherwise, it's changed19:09
aelknerthat's how you get around that19:09
ignasas i have said - full sync IMHO is more reliable and more simple19:09
ignasit works even when any part of the system fails19:09
ignasbut - it's up to you19:09
aelknerit wouldn't have been acceptable to the advertising agencies that used the product19:10
ignaszc.deferred is better if you want changelogs i'd say19:10
aelknertwo minute turnaround was necessary19:10
ignasaelkner: i know, if it's 2 minutes, then it pretty much assumes that it won't fail19:10
ignasbecause if it fails - you won't be able to fix it in 2 minutes19:10
aelknerit failed19:11
aelknerbut it was rerunnable19:11
aelknerso the cuase of failure could be fixed19:11
aelknerand it would pick up where it left off19:11
ignasyou see19:11
ignasthat requires schooltool to have additional state19:11
ignasthe synchronizer to have additional state19:11
ignaswhich adds complexity19:11
ignasprematurely i would say19:12
ignasand yes - i understand that you already did it once19:12
ignasjust that if we could avoid the "moodle integration" module in schooltool, while still having moodle integration solution19:13
ignasit would be a bit more useful ;)19:13
aelknerthat's why i like your crontab idea19:13
ignasand that's why i don't like your "changelog" idea19:13
aelkneri don't understand you objection to having a changelog19:14
aelkneras a matter of fact19:14
ignaswell - then changelog has to be a part of schooltool19:14
aelknerthat could be a general solution to any number of problems19:14
ignasno changelog - no integration19:14
ignasand it's not a "view"19:14
ignasit's a data structure19:14
aelkneri'm suggesting a data structure19:15
aelkneractually, it could be used for any object type we wanted to support19:15
aelknerit would have a datetime together with the object that was added or changed19:15
ignasand would make evolution difficult, sql integration difficult, refactoring difficult ... and disable ZODB packing19:16
aelknera view that says, "give me all changes since Jun 09, 2008 1:00PM" could devlier xml19:16
ignasin some cases19:16
ignasand the xml would include what? object __names__, UID's, URL's, object exports in XML?19:17
aelknerobject exports19:17
aelknerids would be part of it, yes19:17
aelkneranything usefult to anyone outside of schooltool19:17
aelknerlike Moodle or Drupal19:17
ignasemm, so REST revisited19:18
ignasfor everything19:18
th1aI think we're drifting off topic...19:18
aelknerwhat's the topic, then?19:18
th1aCertainly not a general object export.19:19
th1aOr general changelog.19:19
aelknerthe task at hand requires a combination of things19:20
aelkneri think you like jsut having schooltool use TestBrowser immediately19:20
aelknerso exports and changelog doesn't com into play19:20
aelknerbut ignas suggested that this approach would be frought with peril19:20
aelkneras the transaction could time out or something19:21
th1aI defer to his judgement.19:21
aelknerwell, that's what got us discussing the other things19:22
aelkneri'm not going to think about this anymore until i get CAS working19:23
ignasit's zc.assync i think not zc.deferred19:26
ignasaelkner: even if you go for changelog - i'd suggest adding person.log to schooltool, and adding 3 subscribers for IObjectAdded/Modified/Removed19:28
ignasthat would use python logging19:28
ignasto log the person __name__ + operation19:28
ignasand interact with the text script19:28
th1aIt certainly seems like zc.async would be useful in a variety of contexts.19:28
ignas - is the answer "No." ?19:30
*** mattva01 has quit IRC19:31
th1aI'll answer it.19:31
*** jelkner has quit IRC19:40
ignasth1a, Lumiere:, if you will have some free time - read it, and make the interns read it ;)20:44
Lumiereignas: looks interesting20:54
ignasRead it. It will make it easier for you to understand me, and things I am worried about...20:55
LumiereI forwarded it to the cando dev mailing list20:56
LumiereI'll take a look at some point20:56
Lumierethat lp question20:57
Lumiereis there any reason not to have a package for cal-only?20:57
ignasyes, we don't have time for that20:57
Lumierenot that it matters20:58
ignasyep, no time for that, sorry, i can explain you how to do it, I can even help you do it... I can help anyone do that20:58
ignasbut everyone comes and says - "can you do it?"20:58
ignasand runs away as soon as I am mentioning "You could do that, i can explain how!"20:59
ignasso apparently - nobody needs schoolbell enough to write some code :/20:59
ignasi mean - an intern could hack a buggy but working schoolbell in a week ;) without tests ;)21:00
ignaswithout touching schooltool code even ;)21:00
ignas(every few of months this task gets easier and easier to do)21:01
th1aJeez... is down again?21:01
th1a...and back.21:02
ignasooh "Http/1.1 Service Unavailable"21:02
th1aBouncing up and down.21:03
*** ignas has quit IRC21:33
*** alga has quit IRC21:56
*** wbrady has quit IRC22:13
*** mgedmin has quit IRC22:23

Generated by 2.15.1 by Marius Gedminas - find it at!