Author Archives: sws587

April 10th Meeting Minutes

ChipX86 talked about his game project that he was working on this past week: http://pyweek.org/e/mop_and_bucket/

Mike mentioned that everyone did a good job on the screencasts & thanked Mengyun for uploading/converting them.

Today’s meeting was focused on the feedback of how things were run this semester.

Generally speaking:

  • Django tutorials online weren’t that helpful
  • Django book was a good resource: http://www.djangobook.com/en/2.0/ (first 5 chapters especially)
  • GIT guide specific to reviewboard would of been very helpful
  • Useful cheatsheet: http://ktown.kde.org/~zrusin/git/git-cheat-sheet-medium.png
  • Start of term instructions for setup would be helpful
  • Mini-sessions on the various languages
  • Orientation to Review Board during the sprint weekend.

Feedback on weekly meetings:

  • Generally good & to the point

Feedback on mentors:

  • Generally available & nice they are in different timezones
  • Good that there were three mentors
  • Difficult to know when the mentors were available on IRC
  • Some improvement needed for having reviews completed more frequently

UCOSP Administration:

  • Some felt expectations were unclear (no outline)
  • Others felt the evaluation process was unclear

Projects this term:

  • ChipX86 hopes to get all projects into RB’s 1.6 release
  • The mentor’s were happy with what was accomplished
  • Students are welcome to keep on with their projects or other projects and the IRC chat will still be there
  • Project sizes were about right, students were involved in testing and debugging if their projects were finished early
  • Pairing up in projects would have been nice, but might have involved some overhead (including some GIT confusion/clobbering)

Review Board hopes to offer a hosting service: https://www.rbcommons.com/plans/

FINAL TASK:

  • Email CHIPX86 GITHUB Addresses
  • Branch name & Descriptions
  • Update your branches & push it to your github

 

Final Milestone

It’s been a long few weeks but finally reached the last leg of the race!

For my final project we decided to try to finish one of the projects that didn’t get completed last year.  The infamous file attachment (file uploader) program, a program that would allow the user to upload any kind of file to Review Board.  This project was started by Laila Agaev, but not quite finished.  Unfortunately, we didn’t have access to the original repository where it was located, so Christian had to do a merge based off of Review Board.  This turned out to be a bit of a challenge.  After Christian had finished the merge I began to run through the changes listed on Review Board.  Part way through, though, we ran into some difficulties.  So, once again, we had to do a second merge, and run through all the changes a second time.  This was pretty tedious since we lost a lot of the code during the merges.  I had to open up each diff on Review Board and walk through the code so I was able to get some context for each of the comments.  (Ironically, a day later Laila was back online & able to push her repository to the github. It was too late at that point, but it did help to verify that I had the missing sections of code in the their right place).

After we got through those changes I was really looking forward to just seeing the program work, but it didn’t.  So the next few days were spent trying to track down why the program wouldn’t function at all.  In the end, we finally got it working, and viola! The program actually loaded a file! and that was about it.  The text from ‘Review File’ just seemed to disappear whenever we tried to post  a review or comment.

We had a few ways to tackle this but I thought best if we started with the dialog boxes.  Laila had used some custom dialog box function for the comments, where we needed to implement the default Review Board comment box.  This should be straightforward right? And help to track down the lost text?  Well easier said then done.  I spent a lot of time trying to get that thing to work, following the examples in the diffviewer and screenshot viewer, but neither of them were quite the same as what I was trying to do.  I had written some code that just should have worked on several occasions.  I would always get a TextField variable not defined.  So time for firebug, which I used a bit before this, but no where near as much.  I spent a day just figuring out the features of firebug so I could try to track down why this field wasn’t working.  Unfortunately that didn’t yield any results.

Frustration!  Our mentors were very busy with other projects at this point so it was pretty tough to get a hold of anyone for some guidance, but finally David was able to point me in the right direction.  There is a special form (html) that needed to be in with the original code.  So in review_requests_box.html we needed to add a handy {% include “reviews/comments_dlg.html” %}. Eureka! The comment dialog box worked! Except…

Of course! The missing text.  With the new dialog box working I was able to track down where the missing text flew off too.  Apparently there was a mix up on the entire page. Some comments & reviews (depending which button you pressed) sent the text to a body_tag of a reviews table, and the ‘review file’ button sent the text off to the fileuploadercomment table.  Which probably wouldn’t have been so bad in itself, except that the page loaded the text from the body_tag of the reviews table.  Unfortunately at this point the mentors were really swamped with other work and I wasn’t able to figure out the problem on my own.

So we didn’t quite get Laila’s implementation done, sorry Laila.  I think its pretty close, someone more familiar with the languages should be able to finish it up in a few hours I would think.

The remainder of my time was spent writing a report for the U of S, preparing for a presentation, and working on a webcast for the project.  Since Mike emphasized that the audio quality needed to be good on the webcast I spent a fair bit of time recording an audio track and trying to level the volume as best I could.  Once I merged the audio & video I sent it off to youtube where it took about 24 hours to upload.  In this video I largely covered the use of the fill-database script, which might be handy for someone who is trying to use it that isn’t familiar with what it does.  In the second part of the webcast I covered some of the interesting parts of the code, mostly parts that are not clearly described in the Django docs, such as the exception you need to raise for transaction rollbacks, or the need to empty out the database queries to prevent memory errors.  I also talked about uploading custom diffs, so those wishing to add more diffs to fill-database’s random sampling should be able to do so easily.

Well, that’s it for me, a special thanks to those at the UCOSP (Andrew, Michelle, Karen, Eleni, Greg) who made all this possible. A big thanks to Dave, Mike and Christian for their guidance in figuring out Review Board & helping with our projects.  Thanks to Kevin for filling in for the mentors while they were away for the sprint weekend.  Thanks to Dr. Makaroff at the U of S who did all the organizing for our presentations.  And a special thanks to my supervisors at the U of S, Dr. Neilson & Dr. Roy, who supported us through this project.

 

fill-database now filling the database (sort of)

I finally have the first completely working prototype of the fill-database program, sort of…

This past week was spent trying to get the diff-comments to properly work, tidying up the code & solving an indentation problem.  There was some difficulty getting the diff-comments to work since I used the UploadDiff function I didn’t have a direct handle on any diff (fileDiff) itself, I would only get back a diffSet object, something very different than a fileDiff.  However, there is something called a Foreign Key that allows the recursive reference of one object through a many-to-one relationship.  I read about the Foreign Key before, but I didn’t realize its full potential.  Since there is a foreign key in the FileDiff object to the DiffSet I can access the appropriate FileDiff object through a DiffSet object.  So by accessing diffset.files I’m able to get a hold of the fileDiff objects that I needed.  The only clincher is accessing this data, since this type of access returns a Manager Object you can’t simply access the diffSet.files.etc, you can only retrieve the data through a limited number of Manager Object functions.  So in order to solve this issue I created a loop that simply exits on the first iteration, which will give the first and appropriate fileDiff object (the remaining objects in the manager are variations of the first one).  The only bad thing about this is that it looks like a pretty bad cheat, but it works quite well until a better solution can be found.  I did a bunch of looking and its the only solution I could find.

Since then I resolved the indentation problem using some subfunctions, tidied up the code, and started testing.  Simple tests succeeded fine but anything with a significant bit of data (users=100, reviews=10, diffs=10, …) took many hours to run & I would run out of memory.  Christian mentioned that django uses transactions for the database interaction and that I’d need to flush the transactions every so often.  So I did a bit of reading here and discovered that there are several ways to have a transaction commit itself, which others noted sped up their performance quite a bit.  Defining an autocommit at the start of the function made a great deal of difference, taking seconds to do 10 diff-comments rather than minutes.  Unfortunately there are still memory errors that are only detectable by running large tests for a number of hours.  After much more reading I’ve found what some have termed ‘django memory leaking’.  I suspect its actually the django queries that are being stored since I can’t seem to find any information on something like a transaction cache, but I’ll have to run some long-running tests to see for sure.

So in the mean time I’ve written up some test cases and creating a bunch of large diff files that are needed for testing purposes.  Then, at last, it will be time to post it on the review-board & get some much needed feedback!  I look forward to the review, it will be nice to make sure this program is up-to-snuff & do a final commit!  I suspect there will be a number of changes needed so I’m not going to be surprised if I see a lot of comments on review-board.

Status Report – March 6

Crystal

Status

  • Finished  up the feature where when clicking the suggestion in my autocomplete, it redirects to the suggestion’s URL
  • Fixed a bug where this redirecting URL should not happen for other autocomplete fields
  • more testing!!!!

Next Steps

  • Clean up my code
  • More testing
  • Get ready to review request for my code
  • Find a new project work on?

Roadblocks

  • in California right now, no rain please!

Kahlil

Status

  • Started to work on a feature to allow our resources to handle non-unique elements. Made quite a bit of progress here
  • Continued on working on getting all the information needed for creating the XML
  • Completed the XML for a reviews comments, as well as all of the general information in a review

Next Steps

  • Complete new side project feature
  • Continue parsing through XML. Next step is code snippets

Roadblocks

  • Reviewing code for the new sub feature. Not sure what is going on with the JSON part
  • Getting the code snippet and putting it into a text format. Not sure if this will be a roadblock, but definitely has the potential to be.

Mengyun

Status

  • some reading on RESTful API
  • trying to understand djblets/webapi/resources.py and reviewboard/webapi/resources.py
  • begin to add a dictionary for spell checking

Next Steps

  • have a clear idea of how API works
  • add resources for spell checking

Roadblocks

  • midterms and 2nd midterms….
  • not sure how API work and how to implement
  • have no idea of how to test (django and resources)

Mark

Status

  • Researching OAuth to implement for the web API.
  • Finishing the banner javascript to make it a bit prettier.
  • Wrote a blog post for UCOSP.

Next Steps

  • Write some javascript.
  • Try to implement OAuth.

Roadblocks

  • Mid-terms.

Teresa

Status

  • template loader is up for review (searches for extension before looking for a default)
  • still breaking apart templates & doing more testing on the loader as I go (making sure the broken templates still load correctly & that the components can be extended)

Next Steps

  • finish up with the templates
  • testing
  • documentation

Roadblocks

  • I’ve found the odd template that doesn’t seem to ever be used (or at least not that I’ve been able to find) – not sure what I should do with these
  • Using the {% extends filename %} command seems to work (when loading an extension that has no extends it displays a blank page, with an extends it displays the original page), but I haven’t been able to get the extension files to actually add any new content, which I would like to try out to make sure the extension documentation is accurate

Steve

Status

  • diff comments now work! User can add any number of comments to a diff
  • random line commenting implemented
  • attempted to solve the lack of room caused by indentations

Next Steps

  • found a solution to the indentation problem just need to implement it
  • do some testing with the program
  • tidy up the program & output, make a final copy to post on review board!!! yay!
  • create a bunch of diffs for examples
  • upload test repository to git hub (i think)

Roadblocks

  • not sure the best method for pushing up the test repository without creating a bunch of files on post-review (minor issue)
  • slipped on some ice at the University & tore the ligaments in my ankle & hurt my back (I hate winter!) so got behind on homework this week & its tough to work at the computer for long periods of time.

Meeting Minutes – February 27

Opening Remarks

Mike lead the meeting mentioning these items:

  • A few people have some minor roadblocks to clear up.
  • Mentioned that Karen has finished the paperwork and that the cheques should be in the mail soon for reimbursement
  • Asked if anyone needed help after the meeting (several replies)
  • Noted that there was about 6 weeks left in the course & some will be completing their projects before then so need to pick some new ones

Finishing projects on Reviewboard

Kahlil asked if we can continue to work on our projects after the deadline if we don’t finish?

Mike replied “of course”:

  • Reviewboard is an opensource project, so all future contributions are welcomed
  • However, we are only graded on the work we have completed by the end of the semester

GSoC

GSoC is coming, and Review Board has a history of being a GSoC participant. If anyone is interested in doing GSoC, RB might be a good fit if you want to continue your work (or want to dive into something deeper in RB) and get paid for it!

Your experience is an advantage but no guarantees as there are a high number of applications.

They are, however, looking for mentors that will allow more students to be accepted

Adjournment

Concluded the meeting with individual support.

Uploading Diffs: So close & yet so far

I’ve been working on adding diffs to the fill-database script for the last two weeks now.  This part has been a lot more challenging that I thought it would be, it definitely hasn’t been as easy as the other two parts.  I started by examining the diffviewer/forms.py file for an example on how to upload diffs & reworked that code into my own.  Here I created a subfunction that first found the working directory of the repository:

diff_dir = str(os.path.abspath(sys.argv[0] +’manage.py/../scmtools/testdata’))

After which I filled out the appropriate fields (diffset, diffset_history, etc).  It was exciting to see this work, and discovering that I could just pass in the diffset_history field so that one review request could have several diffs variations.  Things finally were starting to make sense.

Unfortunately this wasn’t the greatest way to do things, as Christian pointed out.  Instead I could just have used an instance of the forms.py, which made the process a whole lot simpler.  At least I have a better understanding of the forms.py now.  So all I had to do was call an instance of the form.py and follow the general creation process.  Fortunately this seemed to work pretty easily, with one small exception.

For some reason I couldn’t upload a modified diff, it would just give the error ‘not found in the repository’.  Christian let me know that I needed to setup a test repository, like a repository inside a repository.  I didn’t really understand at the time but now I do.  In a separate directory I created a clone (git clone) of the test repository I wanted to use from my existing repository version of reviewboard.  For instance, my existing repository of reviewboard has some test repositories in /scmtools/testdata (such as git_repo).  Cloning this repository to a completely different location on my harddrive gave me a ‘new’ repository.  From here I could make changes, commit them, & then push the changes back up to my original repository, where the fill-database program could now see the modified file.  Its actually kind of neat, in my test repository I actually have a master branch that I can commit to. It gives the illusion of having great power! (wa-ha-ha)

Now my fill-database program has been posted to the global reviewboard awaiting commenting & critiques.  Once any of those changes are done I will be able to move on to adding comments to the existing diffs.  Then it will be on to a new project.   I’m a little worried time-wise about starting another project, since by the time I’m done this one I’ll only have about a month left before our April 8th screen capture posts are due, but hopefully the next project won’t be too difficult conceptually.

toboggan run

Well I’m starting to feel that this project is more like a toboggan run at times.  Periodically I’ll be making great progress in a limited amount of time.  But once I’m done one part, the toboggan run is over and I have to climb up the hill for the next part.  It seems like each new feature requires a hill to climb before I can start making great progress.

This week I added review-requests to the fill-database script.  It took while to get going in this section since the implementation in Django was slightly different than I was expecting.  Most database entries can be made by making use of the fields declared in their respective models.py file.  Sometimes its like an easter-egg hunt trying to find witch models.py file you need.    Once you find the appropriate models.py file its a pretty simple task of filling the fields with the appropriate values inside the script, unless, of course, there is a handy ‘objects = ‘ line in there somewhere.  That means that the fields need to be filled through a call to a function inside a different class in a different file.  Its a little confusing as it seems a bit backwards to a language like Java.  Trying to bypass this function throws some ‘field not found’ error.  I can see the benefits to this but these sub-functions aren’t that obvious unless you are looking for them.

This next week I’ll be trying to add diffs to the review requests which present its own unique challenges (time to climb up the hill again!).