In the last few weeks I’ve noticed that some issues came up to surface in one’s attempt to set up a local environment for testing purposes. That’s mostly my fault as I should’ve added more documentation on that. I sincerely apologize. This blog post is a late answer to that.
During late January this year, Blobber has been officially announced as a full-functioning tool in Mozilla’s infrastructure as part of the RelEng harness. I was supposed to write a nice blog post about it since it was my baby project as intern during last summer, but failed to do so at that time. Mea culpa. I’ll save the fireworks for another time :-) Until then, for a broader scope introduction, you can watch my intern presentation as it is mainly focused on it and can be found here.
Chris wrote the announcement here. On short, the goal of this project was to make it easier for developers to get important data out of the test machines reporting to TBPL. Previously the only real output from a test job was the textual log. With Blobber successfully running, it’s now possible to upload extra files from your test runs on TBPL. Things like screen shots, minidump logs and zip files are already supported. Full list of allowed content-types can be found here.
Long story short, the next figure depicts the Blobber schema.
The architecture consists of two main components:
- Blobuploader (client side): a Python package that gets invoked by RelEng existing harness at the end of running a test. If the environment variable MOZ_UPLOAD_DIR is set in the test’s environment, one can simply send/copy/put files there and they will be uploaded after the test run is complete. The resulting URLs will be both shown in the logs and highlighted on the front page. Its codebase lies here.
- Blobber (server side): a web app written in Bottle that lies on top of an Amazon EB Server farm. It filters the upcoming files from blobuploader, uploads them to Amazon S3 storage and returns the results to TBPL. Its codebase lies here .
In order to set up a local environment to test various changes/add more features follow the next.
A) Blobber – server side setup
1. Create a separate virtual environment for it.
mtabara@MacBookPro:~/work/tmp$ mkvirtualenv blobber-server New python executable in blobber-server/bin/python Installing Setuptools [...]
2. Clone the Github repository:
(blobber-server)mtabara@MacBookPro:~/work/tmp$ git clone https://github.com/catlee/blobber Cloning into 'blobber'... remote: Reusing existing pack: 458, done. remote: Total 458 (delta 0), reused 0 (delta 0) Receiving objects: 100% (458/458), 115.47 KiB | 0 bytes/s, done. Resolving deltas: 100% (234/234), done. Checking connectivity... done [...]
3. cd into repo dir and install repo-dependencies:
(blobber-server)mtabara@MacBookPro:~/work/tmp$ cd blobber/ (blobber-server)mtabara@MacBookPro:[master]~/work/tmp/blobber$ pip install -r requirements.txt Downloading/unpacking IPy==0.81 (from -r requirements.txt (line 1)) Downloading IPy-0.81.tar.gz Running setup.py egg_info for package IPy Downloading/unpacking Werkzeug==0.9.4 (from -r requirements.txt (line 2)) [...]
4. Set the username and password that the client must provide for authentication:
(blobber-server)mtabara@MacBookPro:[master]~/work/tmp/blobber$ export CLIENT_USERNAME='user' (blobber-server)mtabara@MacBookPro:[master]~/work/tmp/blobber$ export CLIENT_PASSWORD='passwd'
5. Run the application on localhost awaiting connections (it defaults to 8080 port):
(blobber-server)mtabara@MacBookPro:[master]~/work/tmp/blobber$ python application.py Bottle v0.11.6 server starting up (using WSGIRefServer())... Listening on http://0.0.0.0:8080/ Hit Ctrl-C to quit. [...]
Server is ready. Now let’s see how the client should look like.
B) Blobuploader – client side setup
1. Create a separate virtual environment for it.
mtabara@MacBookPro:~/work/tmp$ mkvirtualenv blobber-client New python executable in blobber-client/bin/python Installing Setuptools [...]
2. Get blobuploader from PyPI:
(blobber-client)mtabara@MacBookPro:~/work/tmp$ pip install blobuploader Downloading/unpacking blobuploader Downloading blobuploader-1.1.4.tar.gz Running setup.py egg_info for package blobuploader Downloading/unpacking requests==1.2.3. (from blobuploader) Downloading requests-1.2.3.tar.gz (348kB): 348kB downloaded Running setup.py egg_info for package requests [...]
3. Blobber has several security constraints, one of which is authentication within HTTPS protocol. Therefore, to be eligible to send any files, you will need to provide an authentication file containing a username and password matching those you specified in your blobber server config. I suggest you name it <BuildSlaves.py.template> to fit with the production environment use :-)
(blobber-client)mtabara@MacBookPro:~/work/tmp$ touch BuildSlaves.py.template (blobber-client)mtabara@MacBookPro:~/work/tmp$ vim BuildSlaves.py.template (blobber-client)mtabara@MacBookPro:~/work/tmp$ cat BuildSlaves.py.template blobber_username = 'user' blobber_password = 'passwd'
5. Generate a dummy file to test the connection:
(blobber-client)mtabara@MacBookPro:~/work/tmp$ echo 'Hello world from local blobber environment!' > test.txt (blobber-client)mtabara@MacBookPro:~/work/tmp$ cat test.txt Hello world from local blobber environment!
6. Send the dummy file to the server:
(blobber-client)mtabara@MacBookPro:~/work/tmp$ blobberc.py --url http://0.0.0.0:8080 --auth BuildSlaves.py.template --branch releng --verbose test.txt (blobuploader) - INFO - Uploading test.txt ... (blobuploader) - INFO - Using http://0.0.0.0:8080 (blobuploader) - INFO - Uploading, attempt #1. (blobuploader) - DEBUG - Uploading file to http://0.0.0.0:8080/blobs/sha512/4a0d209df54ede2509f7d2255d0831d751bfdba3d473ba1760f1775d002860a64dac6d32c4692eebd6c0fb6b9ab62e45649389e014994936e5fdc73b012418d4 ... (blobuploader) - CRITICAL - Failed uploading to S3 (blobuploader) - INFO - Upload failed. Trying again ... (blobuploader) - INFO - Done attempting.
Conclusion: Note that the file has been successfully transferred to Blobber server who eventually failed to upload it to Amazon. The error is expected as on local environment we don’t have production Amazon S3 credentials set. In order for the transfer to take place another three environment variables must be set server-side:
- AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to gain access to Amazon S3 through Boto client.
- S3_UPLOAD_BUCKET – bucket name where to upload the files.
That should be it! Now you’re having a local blobber enviroment on your local machine. Should you have any issues or feedback of any sort, please let me know in the comments section.