IRC log of #schooltool for Wednesday, 2012-08-01

replaceafillaelkner, th1a ext act work great for jelkner :)00:06
replaceafillimho, ext act shouldn't show "Points" in the column header in the gradebook00:07
replaceafillit's kind of confusing00:07
replaceafillsince it's a multiplier, not a max00:07
replaceafilljelkner's ext act shows "42" for points00:08
replaceafilland scores with 10100:08
th1aIt should be the max points.00:35
*** replaceafill has quit IRC01:13
*** th1a has quit IRC02:38
*** paulproteus has quit IRC05:13
*** paulproteus has joined #schooltool05:17
*** menesis has joined #schooltool09:57
*** yvl has quit IRC12:14
*** menesis has quit IRC14:05
*** ignas has joined #schooltool15:15
*** menesis has joined #schooltool15:58
*** th1a has joined #schooltool16:06
*** dreich has joined #schooltool16:25
*** yvl has joined #schooltool16:28
th1ahi yvl, menesis, aelkner.16:30
* th1a isn't sure if aelkner is actually here.16:31
yvlgood morning16:31
* th1a assumes replaceafill is having connection issues...16:31
th1aHe's got the permanent excuse.16:31
th1aProbably is just sleeping in more.16:31
*** replaceafill has joined #schooltool16:31
th1aOne thing I was reminded of this morning is that one loose task yvl might do before the October release is checking out compatibility with LDAP in Zentyal.16:33
th1aThat is, exhuming our LDAP login support, seeing if it works, deciding if we want to add support for more person/contact info.16:34
th1aBut even just checking basic login and making a package for it would be good and might only take a couple days.16:34
yvlseems worthwhile16:34
th1aSo... keep that in mind yvl.16:35
th1aOtherwise, what's up yvl?16:35
yvlI can look at that after vacation16:36
yvlwell, wrapping up unfinished things16:36
yvlyesterday I implemented i18n in progress bars16:37
yvltoday I worked on...16:37
yvlso we could register things not in zcml, but in python16:38
yvlwell, I worked on that for past few hours actually, and we might keep zcml configs anyway16:39
yvlfor plugins16:39
yvlfor celery / bottle16:39
th1abtw, does schooltool's process show up as celeryd in this system?16:40
yvlthe microframework that serves task status as  json16:40
yvlas four processes16:40
yvland schooltool I think16:41
yvlwhen mega import is happening, celeryd eats up most of the processor / RAM16:41
th1aOh, ok, because it is...?16:41
yvlwell, because the import is first done in memory, then saved to ZODB16:42
yvland that task is done not in schooltool's zope server16:42
yvlbut in celeryd process16:42
yvlthat actually gives performance boost :)16:42
yvlbecause if you server is dual-core or more16:42
th1aSo... did you re-write part of the import code to allow that or what?16:43
th1aI did notice that I was using 100% cpu on my dual core laptop.16:43
yvlI did rewrite some of the import code, so it would notify "progress"16:44
yvlbut ignas did importer - view separation well16:44
yvlso I removed the "import this xls" call from the view, and put it into a celery task method16:45
yvlwriting remote task is somewhat similar to writing a view16:45
yvljust instead of a class you commonly write a single method16:46
yvloh, and it also stores the xls itself in ZODB16:47
yvlbut since we enabled blobs, it just stores that in the filesystem16:47
yvlso if you come back tomorrow and see errors, you can download that file :)16:48
yvl(that's a small bonus)16:48
yvlI don't know if it's useful16:49
yvlbut I did need to pass store that XLS "for later"16:49
yvlwhen the actual import happens16:49
yvlare you interested in any particular details?16:50
yvlI can talk about this for hours :)16:50
th1aThat's enough for me for now.  You're eventually going to have to explain how to use it to aelkner and replaceafill.16:51
yvlI can give an overview then16:51
th1aSo because of ZEO celery is able to run multiple import processes.16:52
th1aIS that also relevant to report generation?16:52
yvlwell, because of ZEO we can connect to the database from multiple processes16:52
yvlI think I'll limit imports themselves to a single queue16:52
yvlso that only one import would happen at a time16:53
yvland reports can be generated on another queue(s)16:53
th1aOK, so we aren't literally doing parallel processing here.  ;-)16:53
yvlat the moment we are16:54
yvlI want to limit that for tasks like imports16:54
yvlbut that is not necessary16:54
yvlone of questions that is still unanswered is what celery worker model we should use (configure)16:55
yvlcurrently it's greedy - processes16:55
yvlthat fires as many processes as there are cores on the server16:55
yvldownside - in theory it may consume (n_processors + 1) times more RAM than single schooltool before16:56
th1aWould that go back down when it is done?16:57
yvlgood point, I should look, maybe we can just flush the cache16:58
th1aIt is a little different between import and reports, too.16:58
yvla bit of a "doh!" moment for me here :)16:59
th1aFor reports you *don't* want to seriously slow down other users.16:59
th1aIn a big import, you probably don't care.16:59
th1aAn administrator is more likely to just run a big report in the middle of the day randomly.17:00
yvlbut the beauty of having separate processes is:17:00
th1aI don't know if you can configure that per-task.17:00
yvlif we have an import that is CPU bound17:00
yvlit will roughly eat up-to-one core worth of CPU17:00
yvlas schooltool will17:00
yvlbefore, schooltool *and* imports were bound to the same core17:01
yvlbecause our server runs on multiple threads, not processes17:01
yvlbut now, the import happens in a separate process17:02
yvlso on multi-core servers schooltool could be more responsive17:02
th1aSo we essentially can have reports and imports use cores-1 and then SchoolTool will get all it would use anyhow, right?17:02
th1aI think that's probably the best default approach.17:03
th1aesp since 4 cores isn't that exotic at this point.17:03
yvlI'm just a bit worried about RAM at this point17:04
yvl4 processes instead of one connecting to ZODB means 4 times the RAM consumption17:04
yvlwell, that's worst case scenario17:05
th1aAlthough people shouldn't be running memory starved quad core servers.17:05
yvlI'll also play with eventlets... maybe next week17:05
yvlthe idea is to run... a kind of "threaded" celery worker17:05
yvlon a single process17:05
yvlbut if we set up "read", "write" and "non-db" task queues17:06
th1aWhat's the advantage of that?17:06
yvlthere will be little-to-no zodb clashes17:06
yvland 2x RAM consumption17:06
yvlbut all write processes like imports would happen one-by-one of course17:07
*** pgulley has joined #schooltool17:07
yvlreports and such could happen in parallel17:07
th1aI would think you'd start to get a lot of clashes.17:07
yvlwell, clashes happen on "write" tasks17:09
yvlwhen transaction is committed, but database changed in a way that it cannot be merged17:09
yvlthen the whole import needs to be re-started17:09
th1aWOuldn't you get a lot if three processes were importing at the same time?17:10
yvlI haven't tested with real life scenarios17:10
th1aWell... are we prematurely optimizing here?17:11
yvla bit :D17:11
yvlI ran stupid simple checks with a threaded version, and it gave many clashes as expected17:11
yvlbasically I just bombarded it with DB writes17:12
yvlbut that's not a real life check17:12
th1aOh, you're saying make it one threaded process AND create a specific "write" queue?17:13
th1aSo writes would go in sequence with each other but parallel to reads and other tasks?17:13
th1aCouldn't you do that with processes?17:13
yvlI can and I will17:14
yvlstill, processes means x*cores RAM usage17:14
th1aAll right, this sounds reasonably worthwhile.  ;-)17:15
yvlthanks! :)17:15
th1aAll right.  Thanks yvl.17:16
th1aWe should probably move on.17:16
th1amenesis:  Have you gotten to start packaging yet?17:17
menesisth1a: not yet.17:18
th1aDid you reboot or did that happen by itself?17:19
menesisth1a: it rebooted itself17:21
menesisi did not do anything17:21
th1aOK.  We REALLY need to get off that thing when we've got time (which we don't now).17:22
pgulleyreplaceafill, I have a quick question about the workflow for competencies, if you have a moment.17:23
th1aAnything else to add menesis?17:23
menesisbut choosing a hosting provider should not take long17:23
th1apgulley:  replaceafill is giving is report in a moment.17:23
replaceafillpgulley, we can talk when we finish the devs meeting17:24
menesisno, was still busy.17:24
pgulleythla, alright, thanks.17:24
th1aHow's the rest of the week look?17:24
* th1a does not miss the days when all the SchoolTool developers had other jobs going on...17:25
menesisi finish one task today and start packaging cando & virginia straight away17:27
th1aOK. Cool.17:27
th1aThanks menesis.17:27
replaceafillReports -> VA Section Report17:29
replaceafillit still looks funny, but...17:29
replaceafillit's a start :)17:29
th1aIt is a start.17:30
replaceafilli'm writing an email for Glenda, asking her about something17:30
replaceafillsee this part of the paragraph:17:30
replaceafill"...satisfactory rating (one of the three highest marks) on the..."17:30
replaceafillnote the *three* part17:30
replaceafilland in the code i found:17:30
replaceafill                    # XXX: Hard coded 2 asked by Welsh17:30
replaceafill                    if value >= 2:17:30
replaceafillwe didn't use the passing score back then17:30
replaceafillalso, the Rated columns used 2/3/417:31
replaceafillso I'm asking Glenda if we should keep that17:31
replaceafillbecause i've used the passing score for the moment17:31
th1aYeah, well, we're going to have to reprogram your brain here.17:31
th1aIt is good to get a quick version of the old form out, but the new one is going to be really different.17:32
replaceafillah ok17:32
replaceafillsome kind of user control?17:32
th1aBasically, there's going to be a big search/filter/modify form at the top.17:32
th1aSo the passing score could be editable.17:33
replaceafillgot it17:33
th1aIf they really don't want to be dependent on the score system.17:33
th1aI'd say the whole thing should toggle between required and all competencies.17:33
replaceafillshow only one type17:34
replaceafillnot both17:34
th1aAnd then also it should be mostly graphical.17:34
th1aI think.17:34
th1aWe should probably do the bars first though.17:34
replaceafilli could create a new graphical version to work on that17:35
replaceafilli mean a separate view17:35
th1aI guess this one doesn't really need search/filter since you know it is for the section.17:35
replaceafilli think only the rating score17:35
th1aThe pdf version will probably be denser.17:35
th1a(more stuff jammed in)17:35
replaceafilldropdown with all the score system options, right?17:35
replaceafilland Submit button17:36
replaceafilluser selects minimum rating score17:36
replaceafillclicks Submit, report gets updated17:36
replaceafillit could go between the paragraph and the table?17:36
replaceafillor between the course details and the paragraph?17:37
replaceafillif they still want the paragraph, it may need to be rewritten or something17:37
th1aI'd take it out and only put it back in if they insist.17:38
replaceafillah ok17:38
th1aMaybe we can also just replace that message with actual data, like color coding if that condition is met, or something.17:39
replaceafilli'm trying to figure out the bars we would need for a chart17:40
replaceafilllongest in total, right?17:40
replaceafillthen one for evaluated17:40
replaceafillone for rated17:40
replaceafillbut i don't see the %completed one17:40
th1aWe can talk about that in a minute.17:41
replaceafillah ok17:41
th1aWe can wrap up the official meeting I think.17:41
th1aThanks guys!  Have a good rest of the week and weekend.17:41
* th1a drops the bag of gravel.17:41
th1a"rated" means "passing" here, right?17:43
replaceafillwell, kind of ;)17:43
replaceafillbecause of the hardcoded 2 :)17:43
th1atotal is shown as a number.17:43
th1aEverything else is %.17:43
th1aSo one line for evaluated.17:44
th1aOne line for passing.17:44
th1a(those are horizontal)17:44
replaceafillthis type , or separated ones?17:44
replaceafilllike stacked17:44
th1aI think I used a small vertical bar for the % of scored that are passing.17:45
th1aOh... just a sec...17:45
th1aAh, shit.17:46
th1aI have a mockup I drew somewhere...17:47
* replaceafill discards Glenda's email17:47
th1aI need to look at the other computer...17:47
replaceafillsince now i know what we'll do17:47
replaceafillth1a, sure17:47
replaceafilli'll be here17:48
replaceafilli'll add the rating filter in the meantime17:48
*** th1a_ has joined #schooltool17:49
replaceafillpgulley, wanted to talk?17:52
th1a_Yeah, this is going to take a minute...17:53
replaceafillpgulley, i thought about your services request from yesterday17:53
replaceafillfor the list of skills17:53
replaceafilli think dreich mentioned the method returning:17:54
*** th1a_ has quit IRC17:54
replaceafill(ids and names)17:54
replaceafilljust fyi, skills are contained in skillsets17:54
dreichin projects, no?17:54
replaceafilland you will need those to fully identify them17:54
replaceafilldreich, it depends17:54
replaceafillwe have two gradebooks in cando17:55
replaceafillskills and projects17:55
replaceafillskills is fully implemented and kind of stable17:55
dreichthis seems really complicated17:55
dreichI have a mockup of the functions made17:55
replaceafilldreich, ah cool17:55
dreichbut i didn't know anything about skillsets, only projects17:55
replaceafillcould you point me to it?17:55
replaceafilldreich, you could use projects if you want17:55
dreichit's in the cando/schooltool.pyquiz branch17:56
replaceafillit's only that they need more work17:56
th1areplaceafill:  Found it.17:56
dreichwell, I don't know what jelkner wants17:56
pgulleywhat is the difference between skillsets and projects?17:56
dreich(and I don't think he does either)17:56
pgulley(we can pause this if you need to go back to other things)17:56
replaceafillthe difference is that skilsets come automatically when you associate skills to a *course*17:57
replaceafillthe section gets them automatically17:57
replaceafillprojects are customized by the user17:58
replaceafillthey don't depend on anything related to the section17:58
pgulleyoh wow, alright.17:58
replaceafillsuppose the section is for a course related to programming17:58
replaceafillthe skills gradebook would probably show skills related to that17:58
replaceafillbut a teacher could select any kind of skill for a project17:59
replaceafilllike: "for this project i need pgulley to know how to speak well publicly"17:59
replaceafillor something like that17:59
replaceafillso he could search the skills tree looking for something like that17:59
replaceafilland add it to his project17:59
* replaceafill is really bad at explaining things :(18:00
pgulleyno no, we get it!18:00
dreichok, sounds like it would make a lot more sense to tie it to skillsets then, rather than projects18:00
replaceafilli think you could start with skills18:00
dreichand yeah, it makes sense!18:00
replaceafilllet me look for your mock18:00
pgulleywe just need to figure out what the workflow works like on the pyquiz side, and we hadn't been counting on this :p18:00
dreichat the bottom of xmlrpc.py18:01
replaceafillsure, something like that18:01
replaceafillproject_id becomes the skillset18:02
*** yvl has quit IRC18:02
dreichit still requires a section_id to get skillsets?18:02
replaceafillit's like the gradebook case18:02
replaceafillit's supposed to be the same skill globally18:03
replaceafillbut i think it's safer to think they're related to a section18:03
replaceafilli can have these ready today18:04
replaceafillso you can use them tomorrow18:04
dreichawesome, thanks!18:04
replaceafilldo you have a cando instance around?18:04
replaceafilli mean18:04
replaceafillcando data18:04
dreichwe don't have any sample data or anything like that18:05
replaceafilli don't remember if i sent matt or you guys jelkner's skills18:05
replaceafillah ok18:05
replaceafillwill sent then18:05
replaceafilluse the mock up for now18:05
replaceafilland we can meet tomorrow morning to explain you how to load the skills18:05
replaceafillso you get a skills gradebook functioning18:05
replaceafilla funtional skills gradebook i mean :)18:06
replaceafillmy english is really bad these days18:06
replaceafillth1a, checking the wireframe18:06
replaceafillth1a, that was image editing right?18:07
replaceafilli mean, you don't have the d3 code for that, correct?18:08
dreichcomputers are hard to talk about no matter how good your english is, and ok, great!18:08
replaceafilldreich, :D18:08
dreichfeel free to do whatever you think'll be best with the functions18:08
dreichif there's data you think might be useful to send over that I didn't mention in there, or if i ask for something impossible, change them a bit18:09
replaceafillsure, but you got it right i think18:10
pgulleyreplaceafill, one last thing- is it possible to get section-independent skillsets?18:10
replaceafillpgulley, not yet :(18:11
pgulleygosh darn, alright.18:11
replaceafillyou mean skillsets owned only by a single section, correct?18:11
replaceafillor related18:11
pgulleyno, skillsets which are not owned by any section18:12
pgulleyglobal skills18:12
replaceafillah sure, they are global18:12
replaceafillyou have this container with skillsets18:12
replaceafilllots of them18:12
replaceafilland associate some with a course18:12
replaceafilllog in as manager:schooltool here:18:13
pgulleythe issue is that we want to be adding competencies to quizzes before we associate them with sections18:14
replaceafillwhy before?18:14
replaceafillbecause the create -> deploy part?18:15
pgulleyyes, that.18:15
pgulleythey aren't associated with anything until they're deployed, and at that point it's too late to add competencies.18:15
dreichi mean, we could do it then, but it wouldn't be good18:16
replaceafillso, you're going to create an quiz item that is related to a single skill in cando?18:16
pgulleywell each question is18:16
* replaceafill is thinking...18:17
replaceafillwe know the teacher's sections at creation time...18:18
replaceafillyou could limit the skills the teacher relates to18:19
replaceafillto the ones in his sections18:19
pgulleyThat may be the only option18:20
pgulleyother than giving him access to all of the skills.18:20
replaceafillor you could use projects in cando18:20
replaceafillthey're like blank sheets18:20
pgulleybut what happens if he deploys a quiz testing skill A to a section which doesn't include that skill?18:21
replaceafillthat's whay i think projects could be better for that usecase18:21
replaceafillteachers could put any skill in them18:22
replaceafillone project per deployed quiz maybe?18:22
dreichhm, i like that idea18:22
dreichalthough of course multiple deployed quizzes can point to the same project18:22
replaceafillwell, you can tweak the deployment part however you want18:23
replaceafillbut maybe projects is the way to go18:23
replaceafillsince you need flexibility18:23
dreichmakes sense18:24
pgulleyI think so.18:24
replaceafillbtw, did you log in as manager to the url above?18:25
replaceafilljust want to show you how skillsets look like18:26
dreichyeah, i did18:26
replaceafillSchool - >Courses18:27
replaceafillselect Computer Applications18:27
replaceafilland then Skills18:27
replaceafillthere you can assign skillsets18:27
replaceafillbut it may be irrelevant now :)18:27
dreichi see18:28
dreichyeah, that's pretty cool!18:28
replaceafillimho you do need to know:18:28
replaceafillSchool -> Skills (in the sidebar)18:29
replaceafilli think you'll need more services if you want to allow the user navigate this tree18:29
replaceafillgo to Documents -> Virginia CTE18:29
replaceafillthen Information Technology18:31
replaceafillyou will se a table Course list18:31
replaceafillsee 6617 computer applications18:32
replaceafillthose are the same skillsets you just saw in the course skills view18:32
replaceafillthat's the way user usually navigate skills in cando18:32
replaceafilllooking through clusters18:32
replaceafillthen through courses18:33
replaceafillthen skillsets18:33
replaceafilluntil they get to the skill18:33
replaceafillthat would seem hard to implement in the cando side imho18:33
replaceafilli mean18:33
replaceafillin the pyquiz side18:34
replaceafilleven with the all the required services available18:34
replaceafillthere's also a search option18:34
dreichso just isolating them into projects seems effective18:34
dreichit makes the teacher do all the work on the schooltool side, creating the projects18:35
replaceafilldreich, but again, you face the "knowing skills available at creation time" problem18:36
replaceafillyou won't allow a teacher to associate a quiz item to anything that is not already in his cando gradebooks18:36
replaceafilli'm fine with that btw :)18:37
replaceafillbut you should ask Joe or Jeffrey about it18:37
pgulleyright. I think for now quiz deployments will create projects- the skills available at creation will be any skills in sections the teachers teach18:39
replaceafillpgulley, if you're oke with limiting the skills18:39
replaceafillwe could use the skills gradebook directly18:39
pgulleyeventually they should be able to search all the skills at creation to match them up18:39
replaceafilland you require one service less18:40
replaceafill(the one to "create" projects)18:40
pgulleyI'm reluctant to limit them.18:40
replaceafillpgulley, that would be ideal, yes18:40
replaceafillwell, your call18:40
pgulleybut we're running out of time.18:40
replaceafilland the skills gradebook is more stable18:40
replaceafillthat's why i recommend it18:40
pgulleyfor now, i think we should just go with the gradebook.18:41
replaceafillhow long do you have for this task?18:41
pgulleywe will want to look into expanding it later, but It's too big.18:41
pgulleythe rest of the week.18:41
replaceafilllet's meet tomorrow at 10am your time18:42
dreichsounds good!18:42
replaceafilli'll send you an email when i push my changes18:42
replaceafillyou can use:18:42
replaceafillto try cando18:42
replaceafillyou can edit that one18:42
replaceafillthe first one we're using for demo'ing stuff ;)18:43
replaceafillth1a, updated version:19:29
replaceafillthe 2/3/4 in the column is dinamic :)19:29
th1aCool.  That's generally useful, too.  Not just for VA.19:36
replaceafillso we should move it to cando?19:36
th1aI don't intend for this to be VA specific.19:36
* replaceafill should have thought of that ;)19:36
th1aYou can leave it there for the moment though.19:36
replaceafilldo you have the d3 code for the wireframe?19:37
th1aWell, that's what makes it a wireframe.19:38
th1aIt is a drawing.19:38
th1aI don't know about the white tickmark lines.19:38
th1aDon't worry about that part.19:39
replaceafillah ok19:39
th1aAlso, I'm not married to the outline bar and the full bar.19:39
th1aThey maybe should just be different shades.19:39
th1aSo if one is more difficult than the other, do the easier one.19:39
th1aDon't get hung up on the small graphic details in the first pass.19:40
replaceafillah, we'll need a new dropdown in the filter right?19:40
replaceafillfor selecting "Required" or "All"19:40
th1auh, let's try side by side first19:41
replaceafillah ok19:42
th1ait is a toss up19:42
replaceafillfor some reason i think that longer bars are easy to interpret19:42
replaceafillbut d3 has hover, correct?19:42
replaceafillfor that19:42
replaceafillholy s**t:
replaceafillthis is going to be fun! ;)19:48
* replaceafill wants the "Update" button functionality!19:49
th1aOh yeah.20:06
th1aThe section report should default to the score system's passing value, btw.20:07
replaceafillit does20:07
dreichhey, schooltool people, does schooltool save users email addresses?20:14
replaceafilldreich, not by default20:14
replaceafillyou could add a demographics fields for that20:15
*** ignas has quit IRC20:15
replaceafillSchool -> Demographics (sidebar)20:15
replaceafillif your code depends on the demographics field being available you can add a subscriber to create it if it's not there20:16
dreichwe'll do it pyquiz-side20:17
replaceafilli'd mark it as "move to demographics field in ST" ;)20:17
replaceafillthe less pyquiz knows the better imho20:17
dreichbut for now, things working is good20:18
replaceafillit always is! ;)20:18
replaceafillth1a, bar for evaluated20:18
replaceafillbar for rated20:19
replaceafillwhat about the other two percentages?20:19
replaceafillrating and % completed20:19
th1aPlease use passing rather than rated.20:19
th1aThat makes no sense.20:19
replaceafillah sorry20:19
th1aI know you're just saying what VA said.20:20
th1abar for % evaluated20:20
th1abar for % (of all) passing20:20
th1aCan't you just see rated/evaluated20:21
th1aCompare the bars?20:21
replaceafillhhmm well yes20:22
*** menesis has quit IRC20:25
dreichalso, replaceafill, we just talked to jeff and sorted it out20:26
replaceafillah cool20:26
dreichwe're just dealing with Khan Academy stuff for now20:26
replaceafillah ok20:26
dreichas such, we're going with projects rather than skillsets, and do you think you could make xmlrpc methods for creating projects and skills within those projects?20:27
replaceafillah, sure20:27
dreichgreat, thanks20:30
*** dreich has quit IRC20:59
*** pgulley has quit IRC21:00
replaceafillth1a, you coming?21:05
th1aI don't see an invite.21:05
*** menesis has joined #schooltool21:05
replaceafillth1a, don't puke:21:27
replaceafilljust testing ;)21:27
* replaceafill goes to get lunch, bb in ~4521:31
*** th1a has quit IRC22:32
*** th1a has joined #schooltool22:50
th1areplaceafill:  Well... those certainly are bars!23:14

Generated by 2.15.1 by Marius Gedminas - find it at!