IRC log of #schooltool for Monday, 2012-07-02

*** menesis has quit IRC00:21
*** yvl has joined #schooltool09:50
*** menesis has joined #schooltool09:58
*** menesis has quit IRC12:41
*** yvl has quit IRC13:15
*** yvl has joined #schooltool13:15
*** menesis has joined #schooltool13:54
*** ignas__ has joined #schooltool14:12
*** pgulley has joined #schooltool15:12
*** pgulley has quit IRC15:14
*** th1a has joined #schooltool16:08
*** pgulley has joined #schooltool16:10
*** pgulley has quit IRC16:12
*** replaceafill has joined #schooltool16:23
th1ahi replaceafill, aelkner, menesis, yvl.16:30
replaceafillgood morning/afternoon16:31
*** pgulley has joined #schooltool16:31
yvlgood morning16:32
th1ahi yvl.  Would you like to start us off?  What's the timeline for long running processes at this point?16:33
yvlI'm not sure16:33
yvlit's not bogged, but I don't know when it will be done16:33
yvlI was bogged last week though16:34
yvland took the Thu off16:34
yvlbtw, this Fri is also a bank holiday here16:34
th1aThis Wednesday is here.16:35
yvlgood to know, thanks16:35
aelknerth1a, are you taking off wed?16:35
aelknerfunny that welsh said he was meeting with glenda then16:36
th1aOh, we may have a meeting.  Let's see what is going on.16:36
yvlso it feels like there's "a lot" to do, but not "*that* much"16:36
th1aWhere are you now?16:36
yvlI think I got the Zope machinery starting up correctly in spawned processes when using process pool16:37
th1aActually, what's the 50,000 foot view of how this is going to work?16:38
yvlso ST is now split into services controlled by supervisor16:39
yvlwe have schooltoo, zeo and celery16:39
yvlzeo provides us many connections to our db16:39
yvloh, and we have redis DB for temporary task storage16:39
yvlschooltool (or some other app, if we want to) pushes the task to celery16:40
yvlthat task gets routed around, queued and processed at some point16:41
yvlschooltool meanwhile stores the id of the task16:41
yvland once in a while, if user is looking via web view, it, say, polls celery for the status of the task16:42
th1aWhat triggers the poll?16:42
yvltimer :)16:42
yvlor anything else16:43
yvlthere are techniques16:43
yvlmost likely - javascript timer16:43
yvlthere are of course open questions on what for we actually use the long running processes16:44
th1aWe can send tasks to celery from a cron job.16:44
yvlor we can use celery beat for that16:44
yvlit has "cron jobs" built in16:44
yvlif we allow scheduling an import to happen at night, but scheduled from the web16:45
yvlwe'll need a place to show what happened and so on16:45
th1aSo if we want to print a big report, what happens?16:45
yvlif we do "offline" large pdf generation, we'll need UI to show when PDF is generated16:45
yvluser may, and will, close the browsers16:45
th1aYes, but just walk me through here.16:46
th1aA task is sent to celery.16:46
yvlI imagine - but we'll need to develop that -16:46
yvlso, a task is sent to celery, it generates a file, saves to ZODB Blob16:46
yvland knows that it's finished16:46
yvlthe user needs UI to find the download link for that PDF16:47
th1aYes, but celery...16:47
yvland the download link should expire at some point maybe?16:47
th1aCelery adds it to its task list...16:47
yvlwhen time comes, or immediately, it starts executing the task16:47
*** ignas__ is now known as ignas16:47
th1aAnd then probably (but not necessarily) starts on the task.16:48
th1aOK, so it spawns a new process?16:48
th1aWhat kind of process?16:48
yvlit depend on a configuration, but let's say yes16:48
yvlkeep in mind, that here you are asking me16:48
yvlhow precisely I am going to configure a service16:48
yvl(and that's usually a sysadmins job, and they might and will change that)16:49
yvl(at will)16:49
yvlbut yes, it has a pool of available processes, by default the same amount as CPUs on the server16:49
yvland those workers handle the tasks one by one16:50
yvland handling the task...16:50
yvlwell, it connects to ZEO, reads what it wants from our DB16:50
yvlmakes a PDF16:50
yvlsame code as normal SchoolTool16:51
yvlif you wonder about memory consumption... it should use the same as normal schooltool threads16:52
yvlwe have 17 of those I think16:52
yvlbut then again, the memory should get freed after, say, the imports16:53
th1aI'm not particularly worried about it.  I just should know what is going on.16:54
th1aSo we'll probably want, say, a little printer icon that might appear in the gray bar with a number next to it when reports are ready.16:54
yvlbut I think it's best to think about that when we actually have the code for long running processes working16:56
yvleasier to prototype16:56
th1aYes, of course.16:56
th1aDo you have any vacation plans, yvl?16:58
yvlthe week of Aug 8th16:58
yvlor 6th16:59
yvlI don't remember precisely now :)16:59
yvlpoint being - not this month16:59
th1aAs long as we're not imminently going to crash into it.16:59
yvlbut I plan to take a few days off this month16:59
th1abtw, I'm going to be at the parents/in-laws for a good part of this month, but mostly working.17:00
yvlah, ok17:01
th1aAnything else, yvl?17:02
yvlno, that is it17:02
yvlinteresting to work on this though ;)17:02
th1aAs long as it isn't so interesting that you work on in forever.17:03
th1amenesis:  Anything to report?  Philippines going ok?17:04
menesisth1a: yes17:04
menesisyes about the Philippines17:04
menesisfixed several bugs with table sorting, broken objects appearing in catalogs, etc.17:05
menesisnow have more features to work on17:06
th1aOK, so those are things we'd like to have in core?17:06
th1aOr, I mean, did you do those changes for us or them?17:06
menesisno, those were bugs in that plugin17:06
menesissome small things can go in core17:07
menesisor already are there17:08
menesise.g. after removing a person its demographics were left in the db17:08
menesisnow fixed17:08
th1aDon't mix in code they paid for until we have a signed agreement from them.17:08
th1a(which is kind of waiting for a clear list of things we want)17:08
menesisI have been able to make everything we needed in the plugin17:09
menesiswithout changing core17:09
menesisbut some things are workarounds for bugs in core17:09
menesisso I'm fixing them in core as well17:10
th1aOK.  That's fine.17:10
th1aAnything else, menesis?17:12
menesisdon't remember anything important17:12
th1aThanks menesis.17:13
aelknerok, so i got the three sections sheets we discussed to import and export17:13
aelkneri'm currently working on tests for them, including round-tripping17:14
aelkneri figured out how to take the result of a xls export download and upload it to the importer17:15
th1aAh, good.17:15
aelknerand it proved that it wouldn't crash17:15
th1aCan we do tests for unexpected data types?17:15
aelknerah, yes, thanks for reminding me, i already coded checking for them17:16
aelkneri'll make sure to include them in the tests17:16
aelknerone thing i didn't do yet, is the sheet that we discussed that wouldn't export, LinkedSectionImport17:16
aelknerdo we definitely still want that sheet?17:17
th1aThat's the one with start and end?17:18
aelkneri can still do it if you want, it just can't be exported17:18
aelknerand thus, can't be round-tripped17:18
th1aYes, I think we should have it.17:18
aelknerok, i have some useful code to the side, so that won't take long17:18
aelkneri think i want to merge what i already have to trunk so replaceafill can get17:19
aelknerthe resource import bug fix into his demo instance17:19
aelknerthe one that i did last week to put the descriptions in17:20
aelkneri just think it would look better in the demo to glenda to have the building descriptions17:20
aelkneri'll coordinate with replaceafill on getting that into his demo instnace17:20
aelkneri also added code to schooltool.virginia to auto-create the demos on AppInit/Startup17:21
aelknerso replaceafill can pull that change to his demo instance, too17:21
aelknerone thing that happened as a result is that i can't get schooltool.export tests to pass17:22
aelknerif the buildout.cfg has schooltool.virginia in it17:22
aelknerbecause in adding the demos fields, it breaks the default schooltool setup of the demos17:23
aelknerand thus, the schooltool.export tests fail17:23
aelknerthis is not a problem as long as one tests schooltool without schooltool.virginia in the buildout17:23
aelknerjust thought i'd mention that17:24
aelkneranyway, i have a request of yvl, replaceafill17:24
aelknercan we change the buildout.cfg that is in trunk to have the selenium setup and other things17:24
aelknerthat i currently have different in my own buildout.cfg?17:25
aelknerit's a pain to have to always bzr revert buildout.cfg in order to commit changes17:25
replaceafilli've been doing it for a long time17:25
yvlyou can do "bzr commit src" you know17:26
yvland bzr diff src17:26
replaceafilli don't think it's safe to set selenium in trunk's buildout from the buildbot POV?17:26
aelkneryeah, but i'd prefer not to have to do that17:26
aelknerah, i see17:27
yvlmaybe we can remove the buildout.cfg and add buildout.cfg.tmpl17:27
aelknerok, never mind, i'll deal with it17:27
yvland when running make just copy it17:27
aelkneroh, that would be nice17:27
yvland set bzr to ignore buildout.cfg17:27
replaceafilli always make a buildout.cfg.bak file17:27
replaceafillrevert before commiting17:27
replaceafillrestore after committing17:27
yvland set make realclean to remove buildout.cfg17:28
replaceafillbetter make clean :)17:28
replaceafilli wouldn't want to lose my test db ;)17:28
aelknermy point was i was hoping to not have to do separate steps, that it would be automatic17:29
aelknerbut that seems like a pipe dream, so i guess i will do wthout it17:29
yvlI think realclean is better17:29
yvlaelkner, yes, that's what we're discussing17:29
aelkneroh, ok, sorry, didn't understand what you were saying17:30
aelknerbut yes, i wouldn't want the db to be wiped out as part of the auto process17:30
aelkneri tend to keep a db around wh all the packages loaded in it17:30
yvlit's more that... buildout.cfg would never be refreshed17:30
yvlunless you delete it17:30
yvlso make clean it is17:31
aelknerand bzr ignored, right?17:31
yvlmenesis, opinions?17:31
yvlyes, aelkner17:31
yvl(I have to think about it is an opinion also ;) )17:31
menesisI have done something about buildout.cfg17:32
menesissplit it into two files: run.cfg and develop.cfg17:33
menesison 'make' if buildout.cfg does not exist, run.cfg is copied to buildout.cfg17:33
menesisbuildout.cfg is ignored so whatever you change in it, does not show up in bzr diff17:33
aelknerthat sounds great, can we get that into trunk?17:34
menesisto enable all the developer tools, run 'make develop'17:34
menesisthat will replace the line in buildout.cfg from 'extends = run.cfg' to 'extends = develop.cfg'17:35
menesisthat will make omelette, coverage, selenium, ipdb, zodbbrowser, etc. available17:36
menesisthat are not needed just to run schooltool17:36
yvlthank you very much17:36
menesisI had this working, but not committed anywhere17:37
menesisI can bring it back from the shelf and commit soon17:37
yvlyes please17:37
aelknerthanks menesis17:37
replaceafill+1 menesis :)17:38
th1aOK, anything else aelkner?17:39
aelknerno, that's it for me, other than preparing for the glenda demo17:40
aelknerdo we have a meeting planned with welsh?17:40
th1aWhen is the glenda demo?17:41
aelknerwhen welsh meets with her, this wed i think, she will hopefully be very impressed17:41
aelknerand that would mean more money down the line if we're lucky17:42
th1aWell, it doesn't actually mean more or less money.17:42
th1aBut yes, it would be good if she's happy.17:43
aelkneranyway, that's it for me17:43
th1aCan we shoot for having the import/export and tests done Wednesday?17:44
aelkneryes, that sounds right17:44
th1aOK.  We need to put this behind us.17:44
aelknerso are we meeting wed?17:45
th1aI'd say we need to.17:45
aelknerok, i'll be available17:45
th1aThanks aelkner.17:45
replaceafillcan you log in as manager at:17:46
replaceafillso, the attributes that work right now17:47
replaceafillAlternate ID for courses17:47
replaceafillCourse for layer17:47
replaceafilland Description for node attributes17:47
replaceafillafter clicking search with those values you should see the skillsets17:47
replaceafillto match17:47
th1a We should have a cancel button.17:48
replaceafillah! true :)17:48
replaceafilli'll add one17:48
th1aAnd we can add some hints to the form after the meeting.17:48
replaceafilli have a couple of questions for yvl17:48
replaceafillbut maybe he's out of gas?17:48
yvlno harm in asking ;)17:49
replaceafillis there any way to have one single skill in a skillset related to a course?17:49
replaceafilli guess not, right?17:49
replaceafillsince you can only use skillsets in those relationships17:49
yvlwow, I really don't remember anymore17:50
yvlwould need to look at the code17:50
yvlbut one skill in skillset does not sound right17:50
replaceafilldwelsh has come up with the "core" and "additional" concepts17:51
replaceafilland sometimes you just want to select a few, but not all skills in a skill set17:51
aelknercan't we give him a worksheet to put those skills in?17:51
th1aYou can add one skill from a skillset to a project, right?17:51
replaceafillth1a, we will (after i fix that view)17:52
yvldidn't we discuss that this winter?17:52
replaceafillbut i don't think that's what dwelsh wants17:52
replaceafillhe wants to see the "additional" ones in the skills gradebook17:52
yvlwill it kill him to see the additional ones in projects gradebook?17:53
* yvl is not objecting, just asking17:53
replaceafilli can show him that i think17:53
yvlbecause that was kind of the concept17:53
replaceafilli know17:53
yvlthe skills is something that is, well, almost enforced by somebody17:53
yvlthe projects is something that is, well, additional :)17:54
yvland teacher has more control there17:54
th1aI would have preferred if the relationship was between individual skills and courses all along.17:55
* yvl definitely has been out of the game for a while17:55
th1aI never had any particular interest in the concept of skillsets.17:56
th1aIt is ok though.17:56
th1aSo... is this something replaceafill should just add to course/skill relationships or should we punt on it for now?17:57
replaceafillth1a, i think i should fix the "add skill" view in projects and show that to dwelsh17:58
replaceafilland explain him this ^17:58
aelknernot to throw a wrench into the works, but i can guess he'll be disappointed17:58
th1aCan we connect projects to courses (that is, add that feature?)17:58
aelkneri think he wants the teachers to get these things automatically17:58
aelknerwithout having to think about projects17:58
aelknerhis goal is to set everything up as an administrator17:59
aelknerand just have the teachers show up and see everything in their gradebooks17:59
aelknernot for them to have to create projects or anything17:59
yvltechnically you can duplicate a skillset, add an additional skill to it, and assign it to the course17:59
th1aI'm not necessarily against it, but perhaps we should start writing reports first.18:00
replaceafilli could try the "duplicate the skillset" story18:01
*** pgulley has quit IRC18:02
replaceafillyvl's last message18:03
replaceafill"technically you can duplicate a skillset, add an additional skill to it, and assign it to the course"18:03
th1aWhy can't you just associate a skill with a course?18:04
th1aIt doesn't seem like it would be very hard to add that.18:05
yvlkind of intentionally18:05
yvlthink about the gradebook18:05
replaceafillimho that would break the whole model18:05
yvlit has to display skills in *some* worksheets18:05
yvlas in somewhere18:05
yvlso that somewhere is a "skillset"18:05
yvlor a worksheet in normal gradebook18:05
aelknerit could help to get welsh to be more specific about how he would want it to work18:06
aelknerlike, what the ui for adding these additional comps would be18:06
aelknerperhaps in so doing, he might change his mind and decide that projects is the way to go?18:06
yvland I think it would help us if we knew actual live examples where Welsh wants that18:07
th1aOK, so why not just have an "additional" skillset defined for each course?18:07
yvlI think he should know those by heart18:07
aelknerth1a, that may be it18:07
yvlbery doable th1a18:08
aelknerbery, bery indeed :)18:08
th1aOK.  Try that route.18:08
replaceafillok, where does that "additional" skillset live?18:08
* yvl shrugs. In the skillset container, "in the wild" if that makes sense18:09
yvlout of document model probably18:09
replaceafillcourses annotations?18:09
th1aSure -- definitely not in the document model.18:09
yvlnah, global skillset container is fine18:09
yvlunless it clutters the searches or something18:10
replaceafillthat's what i was thinking18:10
aelkneryeah, that's the only question, how to search, assign18:10
replaceafillevery additional skillset would show up there18:10
replaceafillin /skills18:10
yvlis this a bad thing, if the skillset is named somehow properly?  Like math - A - additional or something?18:11
yvlif it's a bad thing, you can probably put those in annotations18:11
yvlshould additional skills always be there?18:12
yvlor created on demand?18:12
yvlis this a common case, or it's one of those best handled by importer?18:12
yvlas in - do we need explicit UI maybe?18:12
replaceafillyvl, i think all dwelsh wants is the "old cando" way18:13
yvlI'm pretty sure about that18:13
replaceafillthe ui for adding additional skills was the same as adding "regular" skills18:13
yvlbut if those are handled by imports by admins... ;)18:13
yvlso I guess the final call is to be made by th1a, as usual18:14
yvlafter some heated discussions with David :)18:14
th1aI guess we'll need another importer.18:14
yvlit's the question of "how common of the usecase is this"18:14
yvlmaybe it can even be handled by the current one?18:15
th1aIt could be fairly common.18:15
replaceafillwe don't have importers for assigning skillsets to courses...?18:15
th1aIn terms of UI, don't we just need a "Add Additional Skills" action in courses?18:16
th1aWe do have a skillset/course importer, iirc.18:16
aelknerreplaceafill, we do have an iporter for assigning skillsets to courses18:17
replaceafillwhat sheet is that?18:17
aelknerlook in cando's importer for CourseSkillsImporter18:18
replaceafilldo we have a CourseSkills sample xls?18:19
replaceafilli mean for dwelsh?18:19
aelkneri have on, but i'm not sure what it matches18:20
replaceafillcould you send me that one please?18:21
aelknerscratch that, it matches the live data, not the autoism demo18:21
aelkneri can send that to you, but it only has two courses in it18:21
aelknerreally, what we should do is change the to generate it18:21
replaceafillthat's ok, i just want a sample one18:21
aelkneri'll send it to you18:22
replaceafillthanks aelkner18:22
replaceafillok, i'll work on the additional skillset18:22
replaceafillth1a, that's it from me18:22
aelkneremail sent18:22
replaceafillwe should put these sample xls in a single place :)18:23
replaceafillmaybe bzr is not the best18:23
th1aAll right.  Thanks guys.18:23
replaceafillbut the vps18:23
* replaceafill done18:23
th1aOK, so we'll meet Wednesday.18:24
aelknerreplaceafill, yes, you and i need to coordinate that18:24
th1aOh, send me your invoices pls.18:24
th1aHave a good one.18:24
* th1a drops the bag of gravel.18:24
aelknerthanks guys, cya wed18:24
yvlgood luck guys!18:24
replaceafillimho CourseSkills should be CourseNodes :)18:25
replaceafillthe match should be with nodes, not skillsets18:25
replaceafilli mean, the same way my view works18:26
replaceafillbut maybe it's not the same use case18:26
aelkneryeah, imports tend to need to be precise18:26
aelknerand flexible, too18:26
aelknerwe are, in the data model, matching courses to skillsets18:26
replaceafillyeah, but nodes have skillsets too18:27
* yvl is running a commandline script to asynchronously set school name :)18:27
aelknerso the importer, and at some point, and exporter, needs to reflects that18:27
aelkneryvl, sounds cool18:27
replaceafillaelkner, i think ST courses should ask nodes about skillsets18:28
replaceafillthat's all18:28
aelkneryeah, but you didn't say anything when yvl built the data model18:28
aelknerthat would have been the time to raise that point18:29
aelknerand i'm not sure it would be best anyway18:29
replaceafillaelkner, come on, THAT'S IN THE DATA MODEL18:29
th1aIt is a data model issue.18:29
aelknerassigning nodes to courses?!18:29
replaceafillthat's the way the "assign-courses-skills.html" view works18:29
replaceafillnodes have skillsets18:29
replaceafillcourse nodes have the proper skillsets18:30
replaceafillthe courseskills importer should ask the course nodes18:30
aelknerah, you're not talking about a model change, just a process change18:32
aelkneryes, the importer could have a column for nodes rather than skillsets18:33
aelknerand the processor could find the nodes, get their skillsets and assign them18:33
aelknerbut, then what do we do about additional skills?18:33
aelknerthey need skillsets, too, not attached to the document nodes18:34
aelknerperhaps we could have two columns in the importer18:34
replaceafilli think so18:34
replaceafillbut you see my point anyway18:34
aelknerthe nodes, and additonal skillsets18:34
replaceafillnot need to change that now18:34
replaceafillaelkner, something like that, yes18:34
aelkneri do see your point, so we could make a change of that sort at some point18:35
aelknersorry, i jumped to the conclusion that you were suggesting a change to the model18:35
replaceafillno man, you hurt my feelings :''''(18:35
replaceafill(joking) :D18:35
*** menesis has quit IRC19:28
*** menesis has joined #schooltool19:29
*** pgulley has joined #schooltool19:34
*** menesis has quit IRC19:46
*** paulproteus has quit IRC20:07
*** paulproteus has joined #schooltool20:10
*** ignas has quit IRC20:39
*** menesis has joined #schooltool20:44
*** pgulley has quit IRC21:01
*** menesis has quit IRC22:14
*** menesis has joined #schooltool22:23
*** menesis has quit IRC23:29
*** replaceafill has quit IRC23:30

Generated by 2.15.1 by Marius Gedminas - find it at!