This course will become read-only in the near future. Tell us at community.p2pu.org if that is a problem.
This is an introduction to contributing to the open source platform that P2PU runs on: Lernanta. Lernanta is based on (also referred to as a fork of) Mozilla's Batucada project and is a Python/Django web application whose code is on GitHub.
We'll walk through the whole process together from getting the code, running it locally, finding a bug to fix, fixing the bug, and contributing the fix to the Lernanta project. Please see full description for more information.
Setting up the enviroment:
We are using vagrant for our development environments. You can find the instructions to get up and running here.
If for some reason you don't want to use vagrant or you prefer to not use a virtual machine, you can find the old setup instructions here.
Once you are done setting up, the virtual machine should already be running lernanta on port 8001. Create an account and login.
Communication:
You can communicate with other P2PU developers using this course or via our mailing list.
If you would like to be involved in other parts of the P2PU project, there are also the community mailing list.
Writing Task:
Then, let us know how it went below (or on your blog and link to that below). Run into any problems? How did you solve it? What worked well with the documentation and what didn't? What suggestions do you have for others starting to develop on Lernanta?
In case you have problems migrating/setting up your database, you might want to look into switching to SQLite for development.
I suggested that we switch to that as the default since it's so much easier and say how to do so here: http://lists.p2pu.org/pipermail/p2pu-dev/2011-August/001628.html
More information about SQLite: http://www.sqlite.org/faq.html
I've been trying to set up the environment and have been stumped by something in one way or the other. But I've been able to get through them because of the posts in here that came before I did. However, when I tried the sync:
python manage.py syncdb --noinput --migrate
I couldn't. It gives me: ImportError: No module named django.core.management
I think, it's about the PYTHONPATH but I'm not sure how to find django. I'm not even sure I know what to search for. Also, while setting up those stuff with pip, I get a bunch of warnings like
warning: no files found matching '*' under directory 'bin'
no previously-included directories found matching 'tests/*.pyc'
no previously-included directories found matching 'docs/*.pyc'
no previously-included directories found matching 'contrib/*.pyc'
no previously-included directories found matching 'celery/*.pyc'
no previously-included directories found matching 'bin/*.pyc'
no previously-included directories found matching 'docs/.build'
warning: no files found matching 'CHANGELOG'
warning: no files found matching '*.css' under directory 'doc'
warning: no files found matching '*.html' under directory 'doc'
Help, please.
Thanks.
Yes, could be path. Could try ./manage.py instead of python manage.py
Or were you working in the virtualenv? I remember you're on Ubuntu, so your prompt would be like:
> (lernanta)jledbetter@ubuntu:~/projects/lernanta$
And if not in the environment, there's no (lernanta) at the beginning.
When I'm not in the virtualenv, I get import errors.
Yes, I'm on the virtualenv. There's the (lernata) indicator at the beginning.
What do those warnings mean?
I just got to the part about configuring the shell, how do i do this in Ubuntu 11.04?
> cd
The above will take you to your home directory where the file resides or you can edit it from wherever if you know the path.
> sudo gedit .bashrc
Sudo is making sure you have permission to edit it. Gedit is a default text editor that comes with Ubuntu.
In 11.04 we had a few problems making sure that it was there so if you have problems you can try this:
> ls /usr/local/bin/virtualenvwrapper.sh
If it's not there, reinstall virtualenvwrapper like Faheem did or doing the --upgrade route like I had to do: http://new.p2pu.org/en/groups/introduction-to-contributing-to-lernata/content/task-two-create-development-environment/#897
How do I completely remove all of the dependinces and what not and start from scratch?
You can reset the virtualenv environment by:
> mkvirtualenv --clear lernanta
then you can redo pip install -r requirements/compiled, etc.
Okay will try but I am still unsure as to how I use that bash script can you please elaborate?
If you're going to continue with developing in Linux, I highly recommend taking an intro to Linux session. Linux is great to develop on so I hope you do! Looks like there's Introduction to Linux which might help :) Vladimir also has a course in Spanish.
Now, how to use the bash script... There are a few ways.
Since the .bashrc file is used when new shells are created, you can close that terminal shell window and start it fresh to run it.
If you want to do it all without closing that window, then you just do all the commands in the terminal shell window you're in. This is a great way to test the commands before putting them into the .bashrc file too. You'll need to do the exports so that when you run the virtualenvwrapper.sh script that it has everything it needs. And to run the virtualenvwrapper.sh script, you use the source command.
> export WORKON_HOME=$HOME/.virtualenvs
> export PIP_VIRTUALENV_BASE=$WORKON_HOME
> export PIP_RESPECT_VIRTUALENV=true
> source /usr/local/bin/virtualenvwrapper.sh
There are probably other ways but those are the two I tend to use.
So save those command in a file called .bashrc and call it via terminal by ".bashrc"?
http://www.hypexr.org/bash_tutorial.php -- This link is mentioned on http://uk.answers.yahoo.com/question/index?qid=20100520102922AAzcm19 . Maybe it can help.
http://pastie.org/1979224
I am getting some sort of operational error why is this?
Saw you asked in IRC about http://pastie.org/1979224
Looks like the db isn't set up to match the settings_local.py
Default settings_local.py has no password for the user lernanta for the database lernanta. Did you create a password for your lernanta user when you created the mysql database?
Are you able to log in to the mysql database alright using that user?
> mysql -u lernanta
logs in using user lernanta
mysql> use lernanta;
which says to use the database lernanta
mysql> show tables;
lists the tables
mysql> exit
to get back to the shell's command line
no i get this
cd lernanta
wesley@wesley-Studio-1558:~/lernanta$ mkvirtualenv --no-site-packages lernanta
New python executable in lernanta/bin/python
Installing setuptools.............done.
Installing pip...............done.
(lernanta)wesley@wesley-Studio-1558:~/lernanta$ workon lernanta
(lernanta)wesley@wesley-Studio-1558:~/lernanta$ sh/rmpyc
(lernanta)wesley@wesley-Studio-1558:~/lernanta$ mysql -u lernanta
ERROR 1045 (28000): Access denied for user 'lernanta'@'localhost' (using password: NO)
(lernanta)wesley@wesley-Studio-1558:~/lernanta$ mysql> use lernanta;
ERROR 1045 (28000): Access denied for user 'wesley'@'localhost' (using password: NO)
(lernanta)wesley@wesley-Studio-1558:~/lernanta$
Hm. I should have specified that mysql> meant that we'd be in the mysql environment instead of your basic shell. If you couldn't login with mysql -u lernanta then the rest was not needed.
You said you didn't create it with a password but maybe it was anyway?
Here's how to create a user in mysql: http://www.databasef1.com/tutorial/mysql-create-user.html
If you create the lernanta user with a password, you'll need to update the local settings file to have that password.
Installation went well on OS X. One problem I've run into before when trying to develop with Python on a Mac before is that it can be very difficult to get things up and running. To get around this, I decided to try and set up my development environment using homebrew this time.
#installing mysqlbrew install mysql#installing Python prerequisites per http://youshoulddoityourself.blogspot.com/2010/11/test.htmlbrew install readline sqlite gdbm#installing Pythonbrew install python --universal --framework
#installing pipbrew install pip#using pip to install virtualenv and virtualenvwrapperpip install virtualenvpip install virtualenvwrapper
3. Set up symlink for virtualenv
ln -s /usr/local/Cellar/python/2.7.1/Frameworks/Python.framework/Versions/2.7/bin/virtualenv virtualenv
export WORKON_HOME=$HOME/virtualenvs source /usr/local/Cellar/python/2.7.1/Frameworks/Python.framework/Versions/2.7/bin/virtualenvwrapper.sh
Thank you so much, Julia! I've added this to the wiki so that other Mac users can easily (hopefully) find it!
https://github.com/p2pu/lernanta/wiki/Setting-up-the-development-environment-on-Mac
Please edit as you need :)
I'm having problems when trying to configure the .bashrc file, the message I get after typing in what I should is "bash: /usr/local/bin/virtualenvwrapper.sh: No such file or directory". I don't know what I should do at this point so any help would be greatly appreciated. (I'm on 11.04)
You could try going to the directory and seeing if it's there like
> cd /usr/local/bin/
> ls virtualenvwrapper.sh
Doesn't seem to be there, strange, I did install it.
Fixed it by reinstalling virtualenvwrapper :D
This just happened to me with Natty. I had to --upgrade to get the file to show up because reinstalling said it was up-to-date and good.
Just in case anyone else is on Natty and has the problem and reinstalling didn't work :)
My dev environment setup on ubuntu 10.04 went fairly smoothly - although there were a couple of points which didn't work:
Running the first command:
sudo apt-get install virtualenvwrapper libxml2-dev libxslt-dev mysql-client mysql-server libmysqlclient-dev python-dev
sudo easy_install virtualenv
sudo easy_install pip
pip install virtualenvwrapper
Which seemed to sort it out.
Second was the database sync error which I already mentioned in a previous comment.
Then I'm also not clear on what running the test framework does (the command python manage.py test), and at what point I should be using the test framework (when I tried it it didn't seem to work).
Perhaps this is obvious when coming from a python/django background?
I'd suggest is that we put in a bit more of an explanation of what to expect once one has reached certain points,. For example, just before saying "Finally, start the development server to take it for a spin." it would be useful to say something like - "at this point, you should now have an empty development from which to work". Same thing for the "To run the test framework" and "To recreate the test database before running the tests." I'm not sure what the function of the test database is, or what it contains, and whether or not I should recreate the test database, and also, what these tests are ;-)
I'm happy to update the readme.rst file but I would need to understand the last points a bit better.
10.04, eh? I haven't tried it on that. Glad that the later easy_install virtualenv worked.
Updating the README sounds great. :)
Nothing happened when you ran the test framework? Odd. It should spam your console as it runs through the various tests. There are some pull requests on github and some of it fixes some migrations. So, I'm wondering if the tables are not right. If you have an error, that might be it. I'd have to poke around more to see why.
Are you able to run the server (./manage.py runserver or python manage.py runserver) OK? Everything showing up in the browser or are you getting errors there too?
The test framework is what we use after doing some changes to make some changes. It will go through unit tests and assert that we haven't broken anything. It might be out of date again though.
You're right. It should be moved or written differently since it's optional when starting the environment. Maybe it should go with the wiki documentation since we should test our changes and make new tests for new code. The other suggestions are great to about "take it for a spin."
Thanks Jessica. Yes, I'm able to run the dev server fine :-)
Sorry - it did actually output messages to the console - I guess I just wasnt' sure what to expect, and what was good/bad...so, this was the command I ran:
(lernanta)charles@ubuntu:~/projects/lernanta$ python manage.py test Traceback (most recent call last): File "manage.py", line 26, in execute_manager(settings) File "/home/charles/.virtualenvs/lernanta/lib/python2.6/site-packages/django/core/management/__init__.py", line 438, in execute_manager utility.execute() File "/home/charles/.virtualenvs/lernanta/lib/python2.6/site-packages/django/core/management/__init__.py", line 379, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/home/charles/.virtualenvs/lernanta/lib/python2.6/site-packages/django/core/management/__init__.py", line 261, in fetch_command klass = load_command_class(app_name, subcommand) File "/home/charles/.virtualenvs/lernanta/lib/python2.6/site-packages/django/core/management/__init__.py", line 67, in load_command_class module = import_module('%s.management.commands.%s' % (app_name, name)) File "/home/charles/.virtualenvs/lernanta/lib/python2.6/site-packages/django/utils/importlib.py", line 35, in import_module __import__(name) File "/home/charles/.virtualenvs/lernanta/src/django-nose/django_nose/management/commands/test.py", line 24, in TestRunner = get_runner(settings) File "/home/charles/.virtualenvs/lernanta/lib/python2.6/site-packages/django/test/utils.py", line 122, in get_runner test_module = __import__(test_module_name, {}, {}, test_path[-1]) File "/home/charles/.virtualenvs/lernanta/src/test-utils/test_utils/__init__.py", line 8, in from django.test.client import RequestFactory as DjangoRequestFactory ImportError: cannot import name RequestFactory
Yes that is bad and not in a good way. I totally see what you mean about going into what the tests should look like because it looks different if there are just test fails vs that stack trace :)
I made a bug report:
http://p2pu.lighthouseapp.com/projects/71002-lernanta/tickets/282-test-framework-is-broken
In settings_local.py, does anyone know what the difference between NAME and TEST_NAME for the drupal_users database settings:
'drupal_users': {
_mysql_exceptions.OperationalError: (1045, "Access denied for user 'drupal_db_user'@'localhost' (using password: NO)")
I comment out the drupal_users database information in settings_local.py when developing.
But yes, your database name has to be the same so when you make the mysql db it will need to be "drupal_user_data." Then you can update the user name and password to be what you have created for use.
Thanks Jessica - I created an empty drupal user database and used this database as the 'NAME' and 'TEST_NAME' in the drupal_users configuration section, and the db sync script seemed to work well.
Most of the developers are not using the drupal database. I have a copy of one of the dev drupal databases in my setup to test the integration with the drupal auth (as it happens on production).
I'm having a lot of problems getting the development framework set up on my Mac, several of the pip installs fail. Have anyone installed it on a Mac, and could tell me any additional steps beyond the README? I struggled with it for a long time, and finally got it going on my old computer - now I have a new laptop, and I don't know what i did last time to make it work :)
In which step are you and what error do you receive?
The compilation of lxml is failing, here is the output: http://pastie.org/1869099
Thanks,
Stian
Do you have Xcode installed? It adds the gcc-4.2 command. It's available on the OS CD but not installed by default.
I have XCode installed, bought it through App Store. It does compile for quite a while, it's just complaining that I don't have architecture PPC, which of course has been phased out by Apple. But I don't think that's where the problem lies.
What is the content of the log file? /Volumes/Home/stian/.pip/pip.log
Pasted it here: http://pastie.org/1869369
Not much new. The error is:
src/lxml/lxml.etree.c:161954: fatal error: error writing to -: Broken pipe
In my machine the compilation of this file (part of lxml) requires a lot of time and memory.
Try with as much memory as possible (close all applications) from the terminal run only:
workon lernanta pip install lxml
If it fails, check "ulimit -a" to see if your bash has limits and if there are try with ulimit to increase them (see man ulimit) and run again these commands, you could also try with sudo since the root user could have greather limits.
If the compilation still fails you could try to use a precompiled lxml, you can dowload one egg for your architecture from:
http://pypi.python.org/pypi/lxml/2.3beta1
However I don't know where you have to install it, since it should be inside the virtual environment for lernanta, in my case it is in my home directory it goes at ~/.virtualenvs/lernanta/lib/python2.6/site-packages/lxml-2.3beta1-py2.6.egg-info/ and ~/.virtualenvs/lernanta/lib/python2.6/site-packages/lxml/
I think you would have to find where should be the location, and install there.
Not sure, probably If you download http://pypi.python.org/packages/2.5/l/lxml/lxml-2.3beta1-py2.5-macosx-10.3-i386.egg#md5=3917603babc9cc0b1c27df288dab22ec you would run something like
sh xml-2.3beta1-py2.5-macosx-10.3-i386.egg --prefix=~/.virtualenvs/lernanta/lib/python2.6/site-packages
Probably the instructions at http://peak.telecommunity.com/DevCenter/EasyInstall#custom-installation-locations could be useful also.
Install Xcode 3.x or follow this:
http://stackoverflow.com/questions/5280887/how-to-disable-easy-install-or-pip-building-ppc-by-default-on-snow-leopard-10-6-6
Stian,
I was able to get my development environment set up on a Mac but ended up using homebrew instead of overwriting the OS X install.
Here's the homebrew tutorial I used to get this working.
I ended up using two different paths than the author mentions:
## Create symlink ln -s /usr/local/Cellar/python/2.7.1/Frameworks/Python.framework/Versions/2.7/bin/virtualenv virtualenv
## Add the virtualenvwrapper to your path export WORKON_HOME=$HOME/virtualenvs source /usr/local/Cellar/python/2.7.1/Frameworks/Python.framework/Versions/2.7/bin/virtualenvwrapper.sh
I'm planning to complete the assignment and write more about my setup after I get past this error:
Caught OperationalError while rendering: (1054, "Unknown column 'schools_school.image' in 'field list'")
Julia
Great Julia
,The error you report occurrs because some migrations are still missing in the newest lernanta.
I already sent a pull request to fix that, however while it is applied, I suggest two options:
sh/schemamigration ./manage.py syncdb --migrate --noinputWhen it asks:
? The field 'Project.created_by' does not have a default specified, yet is NOT NULL. ? Since you are removing this field, you MUST specify a default ? value to use for existing rows. Would you like to: ? 1. Quit now, and add a default to the field in models.py ? 2. Specify a one-off value to use for existing columns now ? 3. Disable the backwards migration by raising an exception. ? Please select a choice:Select 2 and whe it asks
? Please enter Python code for your one-off default value. ? The datetime module is available, so you can do e.g. datetime.date.today()Give 0. After that you can run
./manage.py syncdb --migrate --noinput ./manage.py runserver
mkdir ~/experimental cd ~/experimental git clone https://github.com/vtamara/lernanta.git cd lernanta cp ~/lernanta/settings_local.py . ./manage.py syncdb --migrate ./manage.py runserver
Thank you Vladimir! I followed #1 in your options and now having a working site.
Since I would like that more spanish speakers could help to develop lernanta I wrote instructions in that language to set up the environment with:
http://dhobsd.pasosdejesus.org/?id=Desarrollar+Lernanta+en+OpenBSD+adJ
It is in the public domain (the fourth section are acknowledgments for Jessica and Zuzel among other).
The third section, also describes updates to sources and model, after the patch for migrations that I sent today (if I did everyting right). I wrote instructions about that in english at:
https://github.com/p2pu/lernanta/pull/26/files#diff-31
(it is visible there unitl the pull become accepted or rejected).
I hope someone could help with translation to english of the rest of the instructions for OpenBSD.
That's so great! Let's link to that from the readme file or at least the wiki we have on github. Where do you think it should go?
I tried doing migrations this commit, as well, but it didn't patch correctly. I guess because there was a migration missing between 002 and 003 -- 003 was the one I did. I'm not sure. I like that you are also looking into migrations. I'm glad it worked for you!
Well it could go at least at: http://new.p2pu.org/es/groups/introduction-to-contributing-to-lernata/content/links/
If portability becomes important there could be a page in the wiki referenced from README. IMHO for the moment there could be a link in README saying it is possible to use it under OpenBSD with PostgreSQL and ksh.
Regarding migrations I closed the pull request yesterday when I found a bug, it is already solved and there is a new pull request polished today. Hope some people can test it --I tested with PostgreSQL, MySQL and SQLite and migrations are working in all of them.
If someone in the course wants to test migrations, you can clone the branch migrations from my repository:
mkdir test cd test git clone https://github.com/vtamara/lernanta.git -b migrations cd lernanta
And follow the instructions in doc/migrations.txt
Collecting links to dev documentation on https://github.com/p2pu/lernanta/wiki/Development-Documentation