Skip to content

CIRCLE / cloud

Go to a project
Commit 2ca7eb56 by Őry Máté
Options

Merge branch 'feature-improve-docs-rebased' into 'master' 

Improve Docs

🚧
parents f7d7d5ac 1bbbf0ff
Showing with 225 additions and 63 deletions
docs/deploy.rst
Deploy
========
======
This tutorial describes the installation of a production environment. To
have a fully working environment, you have to set up the other components
as well. The full procedure is included in the :doc:`Puppet recipes
<puppet>` available for CIRCLE Cloud.
This component should normally deployed to a single head node.
This is the web-based entry point to the end users, and also the manager of
the compute and storage nodes.
Preparation
-----------
To get the project running, launch a new Ubuntu 14.04 machine, and
log in to it over SSH.
.. warning::
If the first character of the hostname should not be a digit, because
RabbitMQ won't work with it.
The machine should have an :abbr:`fqdn (fully qualified domain name)`,
which shoud be correctly printed by :kbd:`hostname -f`. You can achieve
this with an IP address (e.g. 127.0.1.1) in :file:`/etc/hosts` having the
short hostname as first, and the fqdn as second alias).
Setting up required software
----------------------------
Update the package lists, and install the required system software::
sudo apt-get update
sudo apt-get install --yes virtualenvwrapper postgresql git \
python-pip rabbitmq-server libpq-dev python-dev ntp memcached \
libmemcached-dev gettext wget pwgen nginx
Set up *PostgreSQL* to listen on localhost and restart it::
sudo sed -i /etc/postgresql/9.1/main/postgresql.conf -e '/#listen_addresses/ s/^#//'
sudo /etc/init.d/postgresql restart
Also, create a new database and user::
pwgen 12 >pgpw
sudo -u postgres createuser -S -D -R circle
sudo -u postgres psql <<<"ALTER USER circle WITH PASSWORD '$(cat pgpw)';"
sudo -u postgres createdb circle -O circle
Configure RabbitMQ: remove the guest user, add virtual host and user with
proper permissions::
pwgen 12 >rmqpw
sudo rabbitmqctl delete_user guest
sudo rabbitmqctl add_vhost circle
sudo rabbitmqctl add_user cloud $(cat rmqpw)
sudo rabbitmqctl set_permissions -p circle cloud '.*' '.*' '.*'
Set up nginx to serve the CIRCLE portal. ::
sudo tee /etc/nginx/conf.d/default.conf <<END
ignore_invalid_headers on;
server {
listen 443 ssl default;
ssl on;
ssl_certificate /etc/ssl/certs/ssl-cert-snakeoil.pem;
ssl_certificate_key /etc/ssl/private/ssl-cert-snakeoil.key;
location /static {
alias ${PWD}/circle/static_collected; # your Django project's static files
}
location / {
uwsgi_pass unix:///tmp/uwsgi.sock;
include /etc/nginx/uwsgi_params; # or the uwsgi_params you installed manually
}
location /vnc/ {
proxy_pass http://localhost:9999;
proxy_set_header X-Real-IP \$remote_addr;
proxy_set_header Host \$host;
proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;
# WebSocket support (nginx 1.4)
proxy_http_version 1.1;
proxy_set_header Upgrade \$http_upgrade;
proxy_set_header Connection "upgrade";
}
}
server {
listen 80 default;
rewrite ^ https://\$host/; # permanent;
}
END
sudo /etc/init.d/nginx restart
.. warning::
For a production deployment, you should use certificates issued by a
recognized certificate authority. Until you get it, you can use a
self-signed one automatically generated by the package.
Setting up Circle itself
------------------------
Clone the git repository::
git clone https://git.ik.bme.hu/circle/cloud.git circle
Set up *virtualenvwrapper* and the *virtual Python environment* for the
project::
source /etc/bash_completion.d/virtualenvwrapper
mkvirtualenv circle
Set up default Circle configuration and activate the virtual environment::
cat >>/home/cloud/.virtualenvs/circle/bin/postactivate <<END
export DJANGO_SETTINGS_MODULE=circle.settings.production
export DJANGO_DB_HOST=localhost
export DJANGO_DB_PASSWORD=$(cat pgpw)
export DJANGO_FIREWALL_SETTINGS='{"dns_ip": "152.66.243.60", "dns_hostname":
"localhost", "dns_ttl": "300", "reload_sleep": "10",
"rdns_ip": "152.66.243.60", "default_vlangroup": "publikus"}'
export AMQP_URI='amqp://cloud:$(cat rmqpw)@localhost:5672/circle'
export CACHE_URI='pylibmc://127.0.0.1:11211/'
END
workon circle
cd ~/circle
You should change DJANGO_FIREWALL_SETTINGS to your needs.
Install the required Python libraries to the virtual environment::
pip install -r requirements.txt
Sync the database and create a superuser::
circle/manage.py syncdb --all --noinput
circle/manage.py migrate --fake
circle/manage.py createsuperuser
Copy the init files to Upstart's config directory and start the manager and
the portal application server::
sudo cp miscellaneous/mancelery.conf /etc/init/
sudo start mancelery
sudo cp miscellaneous/portal-uwsgi.conf /etc/init/
sudo start portal-uwsgi
This is where you describe how the project is deployed in production.
docs/install.rst
......@@ -3,31 +3,40 @@ Installation of a development machine
.. highlight:: bash
This tutorial describes the installation of a development environment. To
have a fully working environment, you have to set up the other components
as well. The full procedure is included in the :doc:`Puppet recipes
</puppet>` available for CIRCLE Cloud.
Preparation
-----------
To get the project running on a development machine, create a new Ubuntu 12.04
instance, and log in to it over SSH.
To get the project running on a development machine, launch a new Ubuntu
14.04 machine, and log in to it over SSH.
To use *git* over *SSH*, we advise enabling SSH *agent forwarding*.
On your personal computer check if *ssh-agent* is running (the command should
print a process id)::
$ echo $SSH_AGENT_PID
1234
.. info::
To use *git* over *SSH*, we advise enabling SSH *agent forwarding*.
On your terminal computer check if *ssh-agent* is running (the command
should print a process id)::
$ echo $SSH_AGENT_PID
1234
If it is not running, you can configure your dektop environment to
automatically launch it.
If it is not running, you should set up your login manager or some other
solution to automatically launch it.
Add your private key to the agent (if it is not added by your desktop
environment)::
Add your private key to the agent (if it is not added by your desktop
environment)::
ssh-add [~/.ssh/path_to_id_rsa]
$ ssh-add [~/.ssh/path_to_id_rsa]
You can read and write all repositories over https, but you will have to
provide username and password for every push command.
Log in to the new vm. The :kbd:`-A` switch enables agent forwarding::
$ ssh -A cloud@host
ssh -A cloud@host
You can check agent forwarding on the vm::
......@@ -38,55 +47,55 @@ You can check agent forwarding on the vm::
If the first character of the hostname of the vm is a digit, you have to
change it, because RabbitMQ won't work with it. ::
$ old=$(hostname)
$ new=c-${old}
$ sudo tee /etc/hostname <<<$new
$ sudo hostname $new
$ sudo sed -i /etc/hosts -e "s/$old/$new/g"
old=$(hostname)
new=c-${old}
sudo tee /etc/hostname <<<$new
sudo hostname $new
sudo sed -i /etc/hosts -e "s/$old/$new/g"
Setting up required software
----------------------------
Update the package lists, and install the required system software::
$ sudo apt-get update
$ sudo apt-get install --yes virtualenvwrapper postgresql git \
python-pip rabbitmq-server libpq-dev python-dev ntp memcached \
libmemcached-dev
sudo apt-get update
sudo apt-get install --yes virtualenvwrapper postgresql git \
python-pip rabbitmq-server libpq-dev python-dev ntp memcached \
libmemcached-dev
Set up *PostgreSQL* to listen on localhost and restart it::
$ sudo sed -i /etc/postgresql/9.1/main/postgresql.conf -e '/#listen_addresses/ s/^#//'
$ sudo /etc/init.d/postgresql restart
sudo sed -i /etc/postgresql/9.1/main/postgresql.conf -e '/#listen_addresses/ s/^#//'
sudo /etc/init.d/postgresql restart
Also, create a new database and user::
$ sudo -u postgres createuser -S -D -R circle
$ sudo -u postgres psql <<<"ALTER USER circle WITH PASSWORD 'circle';"
$ sudo -u postgres createdb circle -O circle
sudo -u postgres createuser -S -D -R circle
sudo -u postgres psql <<<"ALTER USER circle WITH PASSWORD 'circle';"
sudo -u postgres createdb circle -O circle
Configure RabbitMQ: remove the guest user, add virtual host and user with
proper permissions::
$ sudo rabbitmqctl delete_user guest
$ sudo rabbitmqctl add_vhost circle
$ sudo rabbitmqctl add_user cloud password
$ sudo rabbitmqctl set_permissions -p circle cloud '.*' '.*' '.*'
sudo rabbitmqctl delete_user guest
sudo rabbitmqctl add_vhost circle
sudo rabbitmqctl add_user cloud password
sudo rabbitmqctl set_permissions -p circle cloud '.*' '.*' '.*'
Enable SSH server to accept your name and address from your environment::
$ sudo sed -i /etc/ssh/sshd_config -e '$ a AcceptEnv GIT_*'
$ sudo /etc/init.d/ssh reload
sudo sed -i /etc/ssh/sshd_config -e '$ a AcceptEnv GIT_*'
sudo /etc/init.d/ssh reload
You should set these vars in your **local** profile::
$ cat >>~/.profile <<'END'
cat >>~/.profile <<'END'
export GIT_AUTHOR_NAME="Your Name"
export GIT_AUTHOR_EMAIL="your.address@example.org"
export GIT_COMMITTER_NAME="$GIT_AUTHOR_NAME"
export GIT_COMMITTER_EMAIL="$GIT_AUTHOR_EMAIL"
END
$ source ~/.profile
source ~/.profile
Allow sending it in your **local** ssh configuration::
......@@ -100,17 +109,23 @@ Setting up Circle itself
Clone the git repository::
$ git clone git@git.cloud.ik.bme.hu:circle/cloud.git circle
git clone https://git.ik.bme.hu/circle/cloud.git circle
If you want to push back any modifications, it is possible to set SSH as the
push protocol::
cd circle
git remote set-url --push origin git@git.ik.bme.hu:circle/cloud.git
Set up *virtualenvwrapper* and the *virtual Python environment* for the
project::
$ source /etc/bash_completion.d/virtualenvwrapper
$ mkvirtualenv circle
source /etc/bash_completion.d/virtualenvwrapper
mkvirtualenv circle
Set up default Circle configuration and activate the virtual environment::
$ cat >>/home/cloud/.virtualenvs/circle/bin/postactivate <<END
cat >>/home/cloud/.virtualenvs/circle/bin/postactivate <<END
export DJANGO_SETTINGS_MODULE=circle.settings.local
export DJANGO_DB_HOST=localhost
export DJANGO_DB_PASSWORD=circle
......@@ -120,32 +135,32 @@ Set up default Circle configuration and activate the virtual environment::
export AMQP_URI='amqp://cloud:password@localhost:5672/circle'
export CACHE_URI='pylibmc://127.0.0.1:11211/'
END
$ workon circle
$ cd ~/circle
workon circle
cd ~/circle
Install the required Python libraries to the virtual environment::
$ pip install -r requirements/local.txt
pip install -r requirements/local.txt
Sync the database and create a superuser::
$ circle/manage.py syncdb --all --noinput
$ circle/manage.py migrate --fake
$ circle/manage.py createsuperuser --username=test --email=test@example.org
circle/manage.py syncdb --all --noinput
circle/manage.py migrate --fake
circle/manage.py createsuperuser --username=test --email=test@example.org
You can now start the development server::
$ circle/manage.py runserver '[::]:8080'
circle/manage.py runserver '[::]:8080'
You will also need to run a local Celery worker::
$ circle/manage.py celery worker -A manager.mancelery
circle/manage.py celery worker -A manager.mancelery
.. note::
You might run the Celery worker (and also the development server) in GNU
Screen, or use Upstart::
$ sudo cp miscellaneous/mancelery.conf /etc/init/
$ sudo start mancelery
sudo cp miscellaneous/mancelery.conf /etc/init/
sudo start mancelery
Building documentation
----------------------
......@@ -153,14 +168,14 @@ Building documentation
To build the *docs*, install *make*, go to the docs folder, and run the building
process. ::
$ sudo apt-get install make
$ cd ~/circle/docs/
$ make html
sudo apt-get install make
cd ~/circle/docs/
make html
You might also want to serve the generated docs with Python's development
server::
$ (cd _build/html && python -m SimpleHTTPServer 8080)
(cd _build/html && python -m SimpleHTTPServer 8080)
Configuring vim
---------------
......@@ -168,16 +183,16 @@ Configuring vim
To follow the coding style of the project more easily, you might want to
configure vim like we do::
$ mkdir -p ~/.vim/autoload ~/.vim/bundle
$ curl -Sso ~/.vim/autoload/pathogen.vim \
https://raw.githubusercontent.com/tpope/vim-pathogen/master/autoload/pathogen.vim
$ cd ~/.vim; mkdir -p bundle; cd bundle && git clone \
git://github.com/klen/python-mode.git
$ cat >>~/.vimrc <<END
mkdir -p ~/.vim/autoload ~/.vim/bundle
curl -Sso ~/.vim/autoload/pathogen.vim \
https://raw.github.com/tpope/vim-pathogen/master/autoload/pathogen.vim
cd ~/.vim; mkdir -p bundle; cd bundle && git clone \
git://github.com/klen/python-mode.git
cat >>~/.vimrc <<END
filetype off
call pathogen#infect()
call pathogen#helptags()
filetype plugin indent on
syntax on
END
$ sudo pip install pyflakes rope pep8 mccabe
sudo pip install pyflakes rope pep8 mccabe
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or sign in to comment