From Planet PostgreSQL. Published on May 25, 2013.

From Planet PostgreSQL. Published on May 25, 2013.

From Planet PostgreSQL. Published on May 24, 2013.

A while back I posted some SQL which helps track of changes to the PostgreSQL settings file. I've found it useful when benchmarking tests with different settings, but unfortunately the pg_settings_log() function needs to be run manually after each setting change. However that sounds like something which a custom background worker (new in 9.3) could handle - basically all the putative background worker would need to do is execute the pg_settings_log() function whenever the server starts (or restarts) or receives SIGHUP .

This turned out to be surprisingly easy to implement. Based off the example contrib module and Michael Paquier's excellent posts , this is the code . Basically all it does is check for the presence of the required database objects (a function and a table) on startup, executes  pg_settings_log() on startup, and adds a signal handler for SIGHUP which also calls pg_settings_log() .

more...

From Planet PostgreSQL. Published on May 24, 2013.

From Planet PostgreSQL. Published on May 24, 2013.

This post was originally posted on Medium, a new blogging platform made up mostly of people who aren’t necessarily subscribed to Planet. So, please forgive the obvious statements, as the target audience are people who don’t know very much about Postgres.

Wednesday May 23, with no fanfare, Tom Lane’s move to Salesforce.com was made public on the Postgres developer wiki.

For 15 years, Tom has contributed code to Postgres, an advanced open source relational database that started development around the same time as MySQL but has lagged behind it in adoption amongst web developers. Tom’s move is part of a significant pattern of investment by large corporations in the future of Postgres.

For the past few years, Postgres development has accelerated. Built with developer addons in mind, things like PLV8 and an extensible replication system have held the interest of companies like NTT and captured the imagination of Heroku.

Tom has acted as a tireless sentry for this community. His role for many years, in addition to hacking on the most important core bits, was to defend quality and a “policy of least surprise” when implementing new features.

Development for this community is done primarily on a mailing list. Tom responds to so many contributor discussions that he’s been the top overall poster on those mailing lists since 2000, with over 85k messages.

Really, he’s a cultural touchstone for a community of developers that loves beautiful, correct code.

Someone asked: “What does [Tom’s move] mean for Postgres?”

You probably don’t remember this:

Salesforce.com bases its entire cloud on Oracle database,” Ellison said, “but its database platform offering is PostgreSQL. I find that interesting.

When I read that last October, I was filled with glee, quickly followed by terror. I love my small database community, my friends and my job. What if Oracle shifted it’s attention to our community and attacked it, directly? So far, that hasn’t happened.

Instead, Salesforce advertised they were hiring “5 new engineers…and 40 to 50 more people next year” for a “huge PostgreSQL project.”

Tom’s move probably won’t change much for the day-to-day operation of Postgres itself. Hopefully, things are about to get real at Salesforce.

I’m a major contributor to Postgres. I started in 2006, learning about relational databases through work at a small bike parts manufacturer and ERP. My contributions include code, starting conferences, encouraging user group leaders and introducing Postgres to communities that otherwise would never hear from us. I’m a data architect at Mozilla.

From Planet PostgreSQL. Published on May 23, 2013.

From Planet PostgreSQL. Published on May 23, 2013.

I delivered my presentation "Nulls Make Things Easier?" today at PGCon, so I have placed my slides online. The presentation is based on a series of eleven blog posts about NULLs I did a few months ago.

From Planet PostgreSQL. Published on May 23, 2013.

Running accurate database benchmark tests is hard.  I’ve managed to publish a good number of them without being embarrassed by errors in procedure or results, but today I have a retraction to make.  Last year I did a conference talk called “Seeking PostgreSQL” that focused on worst case situations for storage.  And that, it turns out, had a giant error.  The results for the Intel 320 Series SSD were much lower in some cases than they should have been, because the drive’s NCQ feature wasn’t working properly.  When presenting this talk I had a few people push back that the results looked weird, and I was suspicious too.  I have a correction to publish now, and I think the way this slipped by me is itself interesting.  The full updated SeekingPostgres talk is also available, with all of the original graphs followed by an “Oops!” section showing the next data.

Native Command Queueing is an important optimization for seek heavy workloads.  When trying to optimize work for a mechanical disk drive, it’s very important to know where the drive is currently at when deciding where to go next.  If you have a read for that same area of the drive in the queue, you want to read that one now, get the I/O out of the way while you’re nearby, and then move to another physical area of the disk.

However, on a SSD, you might think that re-ordering commands isn’t that important.  If reads are always inexpensive, taking a constant and small period of time on a flash device, their order doesn’t matter, right?  Well, that’s wrong on a few counts.  The idea that reads always take the same amount of time on SSD is a popular misconception.  There’s a bit of uncertainty around what else is happening in the drive.  Flash cells are made of blocks larger than a single database read.  What happens if you are reading 8K of a cell that is being rewritten right now, because someone is updating another 8K section?  Coordinating that is likely to pause your read for a moment.  It doesn’t take much lag at SSD speeds to result in a noticable slowdown.  Partially due to contention concerns, and partially due to nature of I/O, keeping the command queue full is still very important to keeping the drive usefully busy all of the time.

On the 120GB Intel 320 Series drive I used for testing, the drive tops out at around 28MB/s of transfers if you’re not pipelining requests via NCQ.  It goes a whole lot faster than that once the queue is full:

You might think such a huge difference would be immediately obvious in all test results, right?  It’s not though, and that’s how the error slipped by me.  Normally all of my tests are done by two similar machines, and then I validate they match.  I did that for some of the Seeking Postgres results, such as the write heavy tests.  For comparison, here are results from database’s pgbench tool executing its standard, TPC-B-like write test:

The write rate test is barely impacted by whether NCQ is turned on or off, so it wasn’t obvious that one drive had the feature enabled while the other didn’t.  I was using this to validate my test server was operating similar to a second system with one of these drives.  But I picked the one test here where NCQ doesn’t really matter.

The general conclusion of the original presentation is that the Intel SSDs are much faster than regular disk, but still a good bit slower than the more expensive FusionIO flash.  That I knew to be true from real-world workloads, so I’d have been surprised if things didn’t turn out that way.  But it turns out that is true whether or not NCQ is working.  The Intel 320 line in these results is better with NCQ than without, but the relative ranking isn’t any different now.  It’s just the case that the Intel SSD is more competative in some tests than I gave it credit for.

The seeking read results show a much large gap with NCQ enabled:

You might notice a small drop in TPS on that brown line at low scales.  That’s a test error I can’t correct for at this point.  The original server I used for these tests was gone before I figured out what was wrong.  The replacement has the same type of CPU chip, but it’s clocked a bit slower.  (Was an Intel i7 870, now is an Intel i7 860)  That’s why the CPU limited results at low scales dropped.  On any of the I/O limited tests, that original CPU and the slower new one are almost identical, so I still think I’m being fair here.

Finally, I turned the random seek throughput into a business oriented question by asking how long it would take to refill all of RAM after something like a server reboot.  My original test placed the Intel drive as taking 5 minutes to read 16GB of random data with 32 clients reading.  This is exactly what NCQ helps with, and the correctly working drive only takes 1 minute to refill cache:

Thankfully, I don’t have to say I was completely wrong before.  The relative ranking of the various storage options is still the same.  The FusionIO drive I tested was and still is at the top of heap, especially if you need high write throughput.  But the worst case for reading on the Intel 320 series drives (and the very similar 710 series) is much closer to specifications than my tests showed.

With this old territory sorted out, next up I’m testing Intel’s latest enterprise drive, the DC S3700, which replaces the 710 drives in their lineup.  Initial test results look great so far; detailed ones are coming soon.

From Planet PostgreSQL. Published on May 23, 2013.

From Planet PostgreSQL. Published on May 23, 2013.

Thanks to Shaun M. Thomas, I have been offered a numeric copy of the “Instant PostgreSQL Backup” book from Packt publishing, and was provided with the “Instant PostgreSQL Starter” book to review. Considering my current work-situation, doing a lot of PostgreSQL advertising and basic teaching, I was interested in reviewing this one…

Like the Instant collection ditto says, it’s short and fast. I kind of disagree with the “focused” for this one, but it’s perfectly fine considering the aim of that book.

Years ago, when I was a kid, I discovered databases with a tiny MySQL-oriented book. It teaches you the basis : how to install, basic SQL queries, some rudimentary PHP integration. This book looks a bit like its PostgreSQL-based counterpart. It’s a quick travel through installation, basic manipulation, and the (controversy) “Top 9 features you need to know about”. And that’s exactly the kind of book we need.

So, what’s inside ? I’d say what you need to kick-start with PostgreSQL.

The installation part is straight forward : download, click, done. Now you can launch pgadmin, create an user, a database, and you’re done. Next time someone tells you PostgreSQL ain’t easy to install, show him that book.

The second part is a fast SQL discovery, covering a few PostgreSQL niceties. It’s damn simple : Create, Read, Update, Delete. You won’t learn about indexes, functions, advanced queries here. For someone discovering SQL, it’s what needs to be known to just start…

The last part, “Top 9 features you need to know about”, is a bit more hard to describe. PostgreSQL is a RDBMS with included batteries, choosing 9 features must have been a really hard time for the author, and I think nobody can be blamed for not choosing that or that feature you like : too much choice… The author spends some time on pg_crypto, the RETURNING clause with serial, hstore, XML, even recursive queries… This is, from my point of view, the troublesome part of the book : mentioning all these features means introducing complicated SQL queries. I would never teach someone how to do recursive queries before teaching him joins, it’s like going from elementary school to university in fourty pages. But the positive part is that an open-minded and curious reader will have a great teaser and nice tracks to follow to increase his knowledge of PostgreSQL. Mentioning hstore is really cool, that’s one of the PostgreSQL feature one have to know…

To sum up my point of view about this book : it’s a nice book for beginners, especially considering the current NoSQL movement and people forgetting about SQL and databases. It’s a bit sad we don’t have more books like this one about PostgreSQL. I really hope Packt publishing will try to have a complete collection, from introduction (this book) to really advanced needs (PostgreSQL High Performance comes to mind) through advanced SQL queries, administration tips and so on… They have a book about PostgreSQL Server Programming planned next month, I’m really looking forward to this one.

From Planet PostgreSQL. Published on May 23, 2013.

From Planet PostgreSQL. Published on May 23, 2013.

From Planet PostgreSQL. Published on May 23, 2013.

I’ve been hacking on a tool to allow resynchronizing an old master server after failover. Please take a look: https://github.com/vmware/pg_rewind.

From Planet PostgreSQL. Published on May 22, 2013.

We just concluded the PgCon Developer Meeting. The two big items from me were that EnterpriseDB has dedicated staff to start work on parallelizing Postgres queries, particularly in-memory sorts. I have previously expressed the importance (and complexity) of parallelism. Second, EnterpriseDB has also dedicated staff to help improve Pgpool-II. Pgpool is the swiss army knife of replication tools, and I am hopeful that additional development work will further increase its popularity.

The Developer Meeting meeting notes (summary) has lots of additional information about the big things coming from everyone next year.

From Planet PostgreSQL. Published on May 22, 2013.

By DjangoCon Europe from Planet Django. Published on May 22, 2013.

Wow, what an incredible week. We still can’t believe we actually pulled it off.

We spent the last year planning, organizing and working to deliver you the best possible experience. More importantly, we were having a lot of fun and we took a lot of risks that paid off. It’s been an amazing year and we will never forget this last week with you.

We owe a huge THANK YOU to everyone who helped us along the way and words can’t express how grateful we are. We couldn’t make it without you. Thank you for trusting our ideas and backing us so generously from the very beginning. We were just crazy kids without any background who wanted to do fun things for Django community and your trust gave us power to do the impossible.

You are DjangoCon. You made the last week unforgettable for us and everyone else. The conference would not even be near this level of awesomeness if it wasn’t for you. Thanks for being so awesome. We think that since last week, the word ‘awesome’ belongs to DjangoCon forever.

If you still can’t fight your post-DjangoCon-depression, make sure to remember Django Circus Story, watch videos and see all the photos. We will keep you posted with new fun videos this week and publish all the talks in 3 weeks on our Youtube channel. If you posted a review or blogpost about DjangoCon somewhere, do let us know about that! 

Other than that, we still think we can do even better. If you want to see us raising the bar even higher, make sure to follow Makerland.

See you soon, somewhere!

By Andrew Godwin from Planet Django. Published on May 22, 2013.

A new release of an old friend, and more news on django.db.migrations.

I've wanted to get a new release of South out for ages, so I'm delighted that I've finally done so. South 0.8 is now available on PyPI - there's not a great many new changes, the most notable (and the reason for the major version bump) being Python 3 support.

Aymeric Augustin was instrumental in getting that support implemented, so I'd like to thank him for his work on it. On a related note, support for Python 2.5 is being dropped - if you still need that, you'll need to stick with the 0.7.x series.

The other notable change is support for index_together, one of the new improvements in Django 1.5 and something that should have been released a while ago. There's still no first-party support for AUTH_USER_MODEL - it'll work fine as long as you're not distributing third-party apps with migrations. The overall solution to that is something that will have to be implemented in the rewrites that are underway.

with connection.schema_editor() as editor:
    editor.create_model(Foo)
    editor.delete_model(Bar)

From Planet PostgreSQL. Published on May 22, 2013.

From Planet PostgreSQL. Published on May 21, 2013.

Last week VMware released vFabric Data Director 2.7. Among the many new features for various database, I wish to give a little more insight into my favorite ones which are regarding Postgres.

One of the big feature add from a broad perspective is support of Postgres 9.2 based managed database servers along with replication.  Lets look at how it is done in brief.

First vFabric Postgres 9.2 base DBVM needs to be uploaded into Data Director system resource pool and then converted into template. Note this template is different from vFabric Postgres 9.2 Virtual appliance and is available on the same location where Data Director 2.7 download exists.  Also this DBVM template is based on the latest PostgreSQL 9.2.4 core.

Next once the template is  loaded into the system resource pool it will show up in Base DBVMs section in System->Manage and Monor-> Templates-> BaseDBVMs

Then you would then  right click on the base DBVM and select “Convert to Base DB Template”.  Here you also have a new feature to add more disks (think Tablespaces in PostgreSQL) to the template. In Data Director 2.7, disks are added at the template level.

This process take some time and then it should show up in Base DB Templates. On the Base DB Template section right click on the newly created template and select “Validate”. This one creates a test database to see if it is successful or not before it can be rolled out to the Organizations and Database groups . Once the validation is a success, right click and select “Assign it to the resource bundle” and select the resource bundle that will have access to the template or create a new resource bundle to use the template.

Note in the Resource bundle creation steps lies yet another new  way to separate out IO on separate datastores since these may need different IO characteristics

The above image shows how the various types of datastores for your OS, Backup, Data and Logs (Data can be multiple location if you need multiple tablespaces)

Now all the Orgs/Database groups using the resource bundle will see the new vPostgres 9.2 template.

In a sample demo I created a database called MyDB using few wizard questions.

Now on the DB List we do a right click on the database to create more replicas

Couple of more quick wizard questions and a slave is ready. Similarly you can create more slaves of the same master. Now actually behind the scenes it does all the plumbing right from creating a new clone virtual machine from the master and setting it up using standard PostgreSQL replication on the newly created VM but from a user point of view it it is all point , click and done.

The database dashboard also has a new portlet to show bit more information about the replication.

To make a slave as a new master and it will give an option to  move all the other slaves to the new master select as follows:

After completion of the failover, the status shown will be similar to:

Where the original slave replica is now redirected to the new Master. The original master is kept as it is for Port-Mortem or other activities. This also gives a way to use it as new use cases like separating a slave for test purpose/DR purpose or separating “mirrors” etc. Best of all it uses all Standard PostgreSQL replication.

Of course one of the thing that it does not do is re-purpose the original Master as a slave of the new Master. For more details there is a perfect opportunity to find Heikki Linnakangas from VMware at PGCon in Ottawa this week and ask him the question after his session .. WHY??? :-)

From Planet PostgreSQL. Published on May 21, 2013.

One of the clients of OmniTI requested help to provide sample application to insert JSON data into Postgres using Java JDBC driver . I’m not Java expert so it took a while for me to write a simple java code to insert data. TBH, I took help to write test application from one of our Java engineers at OmniTI. Now, test application is ready and next step is to make it work with JSON datatype ! After struggling a little to find out work around for string escaping in JAVA code, I stumbled upon data type issue! Here is the test application code to connect to my local Postgres installation and insert JSON data into sample table:

postgres=# \d sample
Table "public.sample"
Column | Type | Modifiers
--------+---------+-----------
id | integer |
data | json |
denishs-MacBook-Air-2:java denish$ java -cp $CLASSPATH PgJSONExample
-------- PostgreSQL JDBC Connection Testing ------------
PostgreSQL JDBC Driver Registered!
You made it, take control your database now!
Something exploded running the insert: ERROR: column "data" is of type json but expression is of type character varying
Hint: You will need to rewrite or cast the expression.
Position: 42


After some research , I found out that there is no standard JSON type on java side so adding support for json to postgres jdbc is not straight forward ! StackOverflow answer helped me for testing out the JSON datatype handling at psql level. As Craig mentioned in the answer that the correct way to solve this problem is to write a custom Java mapping type that uses the JDBC setObject method. This can be a tricky though.  A simpler workaround is to tell PostgreSQL to cast implicitly from text to json:
postgres=# create cast (text as json) without function as implicit;
CREATE CAST


The WITHOUT FUNCTION clause is used because text and json have the same on-disk and in-memory representation, they’re basically just aliases for the same data type. AS IMPLICIT tells PostgreSQL it can convert without being explicitly told to, allowing things like this to work:

postgres=# prepare test(text) as insert into sample (data) values ($1);
PREPARE
postgres=# execute test('{}');
INSERT 0 1
postgres=# select data from sample;
data
----
{}
(1 row)


Awesome ! That worked Let’s try similar approach in Java application code.

denishs-MacBook-Air-2:java denish$ export CLASSPATH=/usr/share/postgresql/java/postgresql-9.2-1002.jdbc4.jar:
denishs-MacBook-Air-2:java denish$ javac -classpath $CLASSPATH PgJSONExample.java
denishs-MacBook-Air-2:java denish$ java -cp $CLASSPATH PgJSONExample
-------- PostgreSQL JDBC Connection Testing ------------
PostgreSQL JDBC Driver Registered!
You made it, take control your database now!
postgres=# select * from sample;
id | data
----+------------------------------------------------------------------------
1 | {"username":"denish","posts":10122,"emailaddress":"denish@omniti.com"}
(1 row)

Yay! It worked as well

Next in my list to figure out installing PL/Java on Mac and/or Linux !! Let me know, if you have instructions for installation and test application using PL/Java.

From Planet PostgreSQL. Published on May 21, 2013.

One of the things that really frustrated me about the KNN GIST distance box box centroid operators that came in PostgreSQL 9.1 and PostGIS 2.0 was the fact that one of the elements needed to be constant to take advantage of the index. In PostGIS speak, this meant you couldn't put it in the FROM clause and could only enjoy it in one of two ways.

From Planet PostgreSQL. Published on May 21, 2013.

Having recently posted some thoughts on Shaun Thomas' " "PostgreSQL Backup and Restore How-to" review ", Packt asked me if I'd like to review the new " Instant PostgreSQL Starter " by Daniel K. Lyons and kindly provided me with access to the ebook version. As I'm happily in a situation where I may need to introduce PostgreSQL to new users, I was interested in taking a look and here's a quick overview.

It follows the same "Instant" format as the backup booklet, which I quite like as it provides a useful way of focussing on particular aspects of PostgreSQL without being bogged down in reams of tl;dr documentation.  " Instant Pg Starter " is divided into three sections:

more...

From Planet PostgreSQL. Published on May 21, 2013.

It occurs to me I forgot to congratulate the winners of the free ebooks. So without further adieu:

• SAB, who seems to host a nice blog geared toward server administration.
• Stephan, who’s looking to improve existing strategies.
• Jeff and his growing PostgreSQL cluster.
• Pierre, who apparently has an experimental PostgreSQL backend for MySQL. Interesting.

Congrats to the winners. But more, I call upon them to pay it forward by contributing to the community, either by corresponding with the excellent PostgreSQL mailing lists, or maybe submitting a patch or two to the code. There’s a lot of ground to cover, and more warm bodies always helps.

Thanks again, everyone!

From Planet PostgreSQL. Published on May 21, 2013.

1. install Redis, either from packages or multiplatform binaries.  No root access is required for the binaries.
2. read a 1-page tutorial
3. run redis-server
4. run redis-cli or install drivers for your programming language
5. start developing
6. when your app works, deploy to production
7. in production, tune how much RAM Redis gets.

1. install PostgreSQL from packages or the one-click installer.  Root/Admin access is usually required.
2. search the documentation to figure out how to get started. 
3. figure out whether or not your packages automatically start Postgres.  If not, figure out how to start it.  This may require root access.
4. Install drivers for your programming language.
5. Figure out how to connect to PostgreSQL.  This may require making changes to configuration files.
6. Read more pages of documentation to learn the basics of PostgreSQL's variety of SQL, or how to program an ORM which works with PostgreSQL.
7. Start developing.
8. Deploy to production.
9. Read 20 pages of documentation, plus numerous blogs, wiki pages and online presentations in order to figure out how to tune PostgreSQL.
10. Tune PostgreSQL for production workload.  Be unsure if you've done it right.

• better new user tutorials, such as the ones on postgresguide.org
• better autotuning, made a lot easier to implement as of version 9.3.
• a "developer mode PostgreSQL"

By Reinout van Rees from Planet Django. Published on May 20, 2013.

By Steve Schwarz from Planet Django. Published on May 20, 2013.

From Planet PostgreSQL. Published on May 20, 2013.

I did a video interview while I was at ConFoo, and it is now online.

From Planet PostgreSQL. Published on May 20, 2013.

• Table scan
• Index scan
• Bitmap scan
• In-memory sort
• On-disk sort
• Hashing
• Merge Join
• Nested loop join
• Aggregation
• Framework for parallel functions

From Planet PostgreSQL. Published on May 20, 2013.

There’s a fantastic set of blog posts about distributed databases and network partitioning, starting with this post explaining the perils of trying to “communicate with someone who doesn’t know you’re alive.”

The next post is about Postgres and 2-phase commit. And there are four additional posts in the series.

The whole series worth reading for anyone interested in data stores, consistency and Postgres!

By Adrian Holovaty from Planet Django. Published on May 20, 2013.

On Friday, we migrated Soundslice from Heroku to direct use of Amazon Web Services (AWS). I'm very, very happy with this change and want to spread the word about how we did it and why you should consider it if you're in a similar position.

My Heroku experience

Soundslice had been on Heroku since the site launched in November 2012. I decided to use it for a few reasons:

• Being a sysadmin is not my thing. I don't enjoy it, and I'm not particularly good at it.
• Soundslice is a two-man operation (developer and designer), and my time is much better spent working on the product than doing sysadmin work.
• Heroku had the promise of easy setup and easy scaling in cases of high traffic.

While I was getting Soundslice up and running on Heroku, I ran into problems immediately. For one, their automatic detection of Python/Django didn't work. I had to rejigger my code four or five times ("Should settings.py go in this directory? In a subdirectory? In a sub-subdirectory?") in order for it to pick up my app -- and this auto-detection stuff is the kind of thing that's very hard to debug.

Then I spent a full day and a half (!) trying to get Django error emails working. I verified that the server could send email, and all the necessary code worked from the Python shell, but errors just didn't get emailed out from the app for some reason. I never did figure out the problem -- I ended up punting and using Sentry/Raven (highly recommended).

These experiences, along with a few other oddities, made me weary of Heroku, but I kept with it.

To its credit, Heroku handled the Soundslice launch well, with no issues -- and using heroku:ps scale from the command line was super cool. In December, Soundslice made it to the Reddit homepage and 350,000 people visited the site in a period of a few hours. Heroku handled it nicely, after I scaled up the number of dynos.

But over the next few months, I got burned a few more times.

First, in January, they broke deployment. Whenever I tried to deploy, I got ugly error messages. I ended up routing around their bug by installing a different "buildpack" thanks to a tip from Jacob, but this left a sour taste in my mouth.

Then, one April evening, I deployed my app, and Heroku decided to upgrade the Python version during the deploy, from 2.7.3 to 2.7.4. (In itself, that's vaguely upsetting, as I didn't request an upgrade. But my app code worked just as well on the new version, so I was OK with it.) When the deployment was done, my site was completely down -- a HARD failure with a very ugly Heroku error message being shown to my users. I had no idea what happened. I raced through my recent commits, looking for problems. I looked at the Heroku log output, and it just said some stuff about my "soundslice" package not being found. I ran the site locally to make sure it was working. It was working fine. I had deployed successfully earlier in the day, and I had made no fundamental changes to package layout.

After several minutes of this futzing around, with the site being completely down, after I had just sent the link to some potential partners who, for all I know, were evaluating the site that very moment -- I deployed again and the site worked again. So it was nothing on my end. Clearly just something busted with the Heroku deployment process.

That's when Heroku lost my trust. From then on, whenever I deployed, I got a little nervous that something bad would happen, out of my control.

Around the same time, Soundslice began using some Python modules with compiled C extensions and other various non-Python code that was not deployable on Heroku with their standard requirements.txt process. Heroku offers a way to compile and package binaries, which I used successfully, but it was more work using that proprietary process than running a simple apt-get command on a server I had root access to.

With all of this, I decided it was time to leave Heroku. I'm still using Heroku for this blog, and I might use it in the future for small/throwaway projects, but I personally wouldn't recommend using it for anything more substantial. Especially now that I know how easy it is to get a powerful AWS stack running.

My AWS setup

I'm lucky to be friends with Scott VanDenPlas, who was director of dev ops for the Obama reelection tech team -- you know, the one that got a ton of attention for being awesome. Scott helped me set up a fantastic infrastructure for Soundslice on AWS. Despite having used Amazon S3 and EC2 a fair amount over the years, I had no idea how powerful Amazon's full suite of services really were until Scott showed me. Unsolicited advertisement: You should definitely hire Scott if you need any AWS work done. He's one of the very best.

The way we set up Soundslice is relatively simple. We made a custom AMI with our code/dependencies, then set up an Elastic Load Balancer with auto-scaling rules that instantiate app servers from that AMI based on load. I also converted the app to use MySQL. In detail:

Step 1: "Bake" an AMI. I grabbed an existing vanilla Ubuntu AMI (basically a frozen image of a Linux box) and installed the various packages Soundslice needs with apt-get and pip. I also compiled a few bits of code I needed that aren't in apt-get, and I got our app's code on there by cloning our Git repository. After that instance had all my code/dependencies on it, I created an AMI from it ("Create Image (EBS AMI)" in the EC2 dashboard).

Step 2: Set up auto-scaling rules. This is the real magic. We configured a load balancer (using Amazon ELB) to automatically spawn app servers based on load. This involves setting up things called "Launch configurations" and "scaling policies" and "metric alarms." Check out my Python code here to see the details. Basically, Amazon constantly monitors the app servers, and if any of them reaches a certain CPU usage, Amazon will automatically launch X new server(s) and associate them with the load balancer when they're up and running. Same thing applies if traffic levels go down and you need to terminate an instance or two. It's awesome.

Step 3: Change app not to use shared cache. Up until the AWS migration, Soundslice used memcache for Django session data. This introduces a few wrinkles in an auto-scaled environment, because it means each server needs access to a common memcache instance. Rather than have to deal with that, I changed the app to use cookie-based sessions, so that session data is stored in signed cookies rather than in memcache. This way, the web app servers don't need to share any state (other than the database). Plus it's faster for end users because the app doesn't have to hit memcache for session data.

Step 4: Migrate to MySQL. Eeeek, I know. I have been a die-hard PostgreSQL fan since Frank Wiles showed me the light circa 2003. But the only way to use Postgres on AWS is to do the maintenance/scaling yourself...and my distaste for doing sysadmin work is greater than my distate for MySQL. :-) Amazon offers RDS, which is basically hosted MySQL, with point-and-click replication. I fell in love with it the moment I scaled it from one to two availability zones with a couple of clicks on the AWS admin console. The simplicity is amazing.

Step 5: Add nice API with Fabric. Deployment was stupidly simple with Heroku, but it's easy to make it equally simple using a custom AWS environment -- I just had to do some upfront work by writing Fabric tasks. The key is, because you don't know how many servers you have at a given moment, or what their host names are, you query the Amazon API (using the excellent boto library) to get the hostnames dynamically. See here for the relevant parts of my fabfile.

Ongoing: Update AMI as needed. Whenever there's a new bit of code that my app needs -- say, a new apt-get package -- I make a one-off instance of the AMI, install the package, then freeze it as a new AMI. Then I associated the load balancer with the new AMI, and each new app server from then on will use the new AMI. I can force existing instances to use the new AMI by simply terminating them in the Amazon console; the load balancer will detect that they're terminated and, based on the scaling rules, will bring up a new instance with the new AMI.

Another approach would be to use Chef or Puppet to automatically install the necessary packages on each new server at instantiation time, instead of "baking" the packages into the AMI itself. We opted not to do this, because it was unnecessary complexity. The app is simple enough that the baked-AMI approach works nicely.

Put all this together, and you have a very powerful setup that I would argue is just as easy to use as Heroku (once it's set up!), with the full power of root access on your boxes, the ability to install whatever you want, set your scaling rules, etc. Try it!

By Andrew Godwin from Planet Django. Published on May 20, 2013.

A field, a tent, and a large amount of Polish food - the makings of a great conference.

DjangoCon and I have a long history. The very first DjangoCon, back in 2008, was also my very first conference - and I've achieved the slightly dubious honour of having attended every single one.

They are not, of course, the only conferences that I go to; these days I try to speak at a variety of events. I've seen a lot of venues and they're all variations on a theme. That theme, of course, is large rooms full of chairs.

DjangoCon EU 2013, hosted last week in Warsaw, bucked that trend and was probably the best yet - and that's not something I say lightly. Ola Sitarska and the rest of her team went for an inspired gamble that really paid off.

The stage, and Craig Kerstiens. From flickr.com/photos/patrick91

When I first heard of the plans to host this year's DjangoCon EU in a circus tent, I was a little sceptical - after all, conference venues have evolved over many decades to serve the many needs of a large-scale event. Seating, airflow, power, networking, A/V, catering and toilets are all needs of a modern conference population.

The end result, however, was impressive. The circus tent had been outfitted with power, WiFi, lighting, a stage, projectors, audio and even chillers full of drinks, and beats many indoor venues I've spoken in. One entire side of the tent was open to the outside, providing easy airflow and access without making the inside too bright.

There were a couple of small niggles - the flight path of the nearby airport had changed the week before, meaning planes would occasionally interrupt talks, there wasn't quite enough toilet capacity at peak times, and WiFi was sluggish - which is normal for tech conferences. These were all outweighed by the positives, though - and what positives there were.

One of my tweets describes the conference as "like a music festival", which gives some idea of the wonderful attitude everyone had. Hammocks, deckchairs and bean bags were some of the seating options, there was a plentiful supply of free popcorn, and in between talks you could wander down to the fountain, relax on picnic blankets under the trees or dip into an entire freezer of ice cream.

Everyone was in a very good mood, and very relaxed. DjangoCon has always gone for a laid back approach, and here it worked incredibly well. I mostly come to DjangoCon to socialise and meet new people, rather than to learn from the talks, and in that environment it worked very well.

Danny's handstand lessons are almost a DjangoCon staple. From flickr.com/photos/patrick91

I'd like to highlight the catering in particular - it makes such a difference to have snacks and drinks available throughout the day rather than at set times. It was possible at any point in the afternoon to go and get some ice cream, fruit juice or even one of the sandwiches from breakfast.

Not only that, but the catering during the sprints wasn't the usual case of just ordering pizza or sandwiches for everyone - there were proper hot meals, with desserts. Portion size suffered a little since the sprints were so popular, but it was still very tasty, and I'm pleased to see healthier food at a sprint event.

As a speaker, the slick execution of this conference began before I even arrived. As DjangoCon is a community-run conference, staffed entirely by volunteers, speakers are generally expected to pay their own way, sometimes including some of the ticket price. This time, however, not only was admission free but the organisers picked us up from the airport and sorted out the hotel.

Of course, it's not that unusual for a conference to do this at all - I've experienced it many times before - but to do it while still keeping the prices low was impressive, and Ola and her team did it well, keeping us up to date and taking feedback into account very quickly.

One thing I wish more conferences did for their speakers is providing a local SIM card with data - this is especially useful in non-English-speaking countries, where getting one can be tricky. DjangoCon provided one right in the welcome basket, and I used it all week - a data connection is invaluable for navigating a foreign city.

Approaching the circus through the trees

I continue to be really impressed by the way the Django community evolves each year. DjangoCon EU itself gets more impressive every year - and previous years have set a high standard indeed.

Conferences are a very important part of bringing a community together and fostering the cohesion that really helps keep a project like Django going. I'm so glad that DjangoCon exists, and that each one helps push forward projects both old and new. It's somewhat unfortunate that the tech scene is mostly governed by who you know, rather than what, but events like these offer a way to improve both at once.

I'm also amazed how each year another group of volunteers tirelessly steps up to host, and this year was no different - a team of French volunteers stepped up to host it in the French Rivera next year. I can't wait - I wish them the best of luck.

I have many friends who I only ever see at conferences, and so the return of conference season each year is always a delight. It's a privilege to be able to attend and enjoy all these events each year, and to the organisers - not only of DjangoCon EU but all similar events - I'd like to say one thing: dziękuję!

By Horst Gutmann from Planet Django. Published on May 19, 2013.

DjangoCon EU has always been something special. The very first European DjangoCon back in 2009 in Prague was a great start and over the years with Berlin, Amsterstam and Zurich in between in grew and grew and got better and better. This year, DjangoCon made a stop in Warsaw and, to be honest, it was the greatest conference I've ever been to.

Back when the first plans for this year's event started to leak I was all "what on Earth???", but the local team and all the helpers including Ola Sitarska, Ola Sendecka, Kuba Kucharski, Tomek Paczkowski and Jarek Piotrowski made it happen: A tech conference in a park, in a tent, that worked!!!

The conference

Over the years, conferences did evolved into those perfectly organized events where usually only one thing breaks: the WiFi. They usually take place in perfectly AC'd hotels or conference centers and depending on the price you pay you get served hot and cold beverages by people in black & white & bow-ties. Well, DjangoCon EU has always been different being a conference by the community for the community, but this year was like off a whole different planet with the organizers taking a huge risk and it working out perfectly.

The DjangoCon Circus in all its glory!

The tent is probably the most risky venue you can have. If the weather changes and you end up getting hit by rain from all directions, your conference venue might become a nightmare (depending naturally also to some degree on your attendees). But the exact opposite happened last week: Summer!

All week was perfect summer weather. Naturally, the tent became slighted heated up but still quite bearable and there was ice cream at the buffet whenever you wanted some :D

Then there were the talks. My main objectives when going to conferences are to meet up with people I haven't seen for a while and getting new ideas. No point in talking about the fun-part since ... heck, we were in a tent in a park in Warsaw! And right on the first day I moved PyGraz.org from being managed by supervisord to circus after seeing Tarek Ziadé's talk about it. So much for new ideas ;-)

Thematically probably the biggest theme here were databases, though, with at least 5 talks being about how to store your data (mostly in PostgreSQL) the right way:

• Advanced PostgreSQL in Django
• Migrating the Future
• Taming multiple databases with Django
• Getting past the Django ORM limitations with Postgres
• Enterprise Django: transactions for web developers

Good stuff! :-)

So, the conference was awesome, but so was the after-show party: There was a large grill with exceptionally long queues for food and beer. Sadly the local fauna took that as an opportunity to assault us with all the mosquitos it had. Seems like for most this wasn't really a problem but Ulrich and myself were sadly hit rather hard so we left early while the other still had a blast into the morning. But at least I still saw Rob Spectre making the crowd go wild ;-)

Rob Spectre doing a live performance ;-)

The sprints

After everyone had gotten out of the bed on Saturday, the sprint was the next point on the agenda. I guess, more than 200 people attending a Django sprint will gets its own big entry in the Django history-book. Venue in the heart of Warsaw at the HardGamma offices, enough power and WiFi for everyone and despite being nearly twice as many people as anticipated there was also enough food for everyone!

Big thanks to the organizers for such an awesome week! And see you all again next year in France!

Travel appendix

• Get a Play prepaid data SIM card. 20 PLN for 1GB is a steal!
• For 30 PLN per day for a room for two the Hostel Słuzewiec is quite nice.
• OMG, the burger at Meat Love!
• ... while the burger at the Hard Rock Café is overrated ;-)
• Besides TripAdvisor FourSquare is surprisingly useful to find good food in a city you don't know :-)

"DjangoCon EU 2013" was written by Horst Gutmann and is licensed for reuse under Creative Commons Attribution-Share Alike 3.0 Austria License.

From Planet PostgreSQL. Published on May 19, 2013.

From Planet PostgreSQL. Published on May 19, 2013.

In our previous article we went through describing what retention policies are and how they can be enforced on your PostgreSQL server backups with Barman 1.2. In this post, we will go through the configuration aspects.

For the sake of simplicity, we assume a typical scenario which involves taking full backups once a week through the “barman backup” command. Suppose you want to automatically keep the latest 4 backups and let Barman automatically delete the old ones (obsolete).

The main configuration option for retention policies in Barman is “retention_policy” which can be defined both at global or server level. If you want all your servers by default to keep the last 4 periodical backups, you need to add in the general section of Barman’s configuration file the following line:

[barman]
... // General settings
retention_policy: REDUNDANCY 4

When the next “barman cron” command is executed (every minute if you installed Barman using RPMs or Debian/Ubuntu packages), Barman checks for the number of available full periodical backups for every server, order them in descending chronological order (from the most recent to the oldest one) and deletes backups from the 5th position onwards.

In case you have several servers backed up on the same Barman host and you want to differentiate the retention policy for a specific server, you can simply edit that server configuration section (or file, see “Managing the backup of several PostgreSQL servers with Barman“) and define a different setting:

[malcolm]
description = Malcolm Rocks
ssh_command = ssh malcolm
conninfo = host=malcolm port=5432 user=postgres dbname=postgres
retention_policy: REDUNDANCY 8

However, Barman allows systems administrators to manage retention policies based on time, in terms of recovery window and point of recoverability. For example, you can set another server to allow to recover at any point in time in the last 3 months:

[angus]
description = Angus Rocks
ssh_command = ssh angus
conninfo = host=angus port=5432 user=postgres dbname=postgres
retention_policy: RECOVERY WINDOW OF 3 MONTHS

Make sure you have enough space on the disk to store all the WAL files for every server you back up, and always monitor “barman check” through your alerting tools (such as Nagios/Icinga/Zabbix/etc.).

Current implementation of retention policies in Barman has some limitations: retention policies are managed only automatically (not manually – this would require to create a “barman delete –obsolete” command, for example) and there is no decoupling yet between full backups and WAL archive transactional logs (we have already thought of the “wal_retention_policy” option, but at the moment it is not handled).

More detailed information on retention policies can be found on Barman’s documentation website.

From Planet PostgreSQL. Published on May 18, 2013.

By DjangoCon Europe from Planet Django. Published on May 18, 2013.

It has been brought to our attention that there may have been a violation of the code of conduct at the DjangoCon speakers dinner. We have started an investigation of the incident. If our investigation reveals that a violation did occurr, we will make a further announcement regarding the action we will take.

From Planet PostgreSQL. Published on May 17, 2013.

I’ve got a Beer & Tell to give about alembic. Alembic is a migration tool that works with SQLAlchemy. I’m using it for database migrations with PostgreSQL.

So, here’s what I want to say today:

• Written by SQLAlchemy wiz Mike Bayer
• Here’s the tutorial. Socorro is now using alembic in production with SQLAlchemy 0.6.x. I’m hoping to get us upgraded to 0.8.x soon.
• Here’s what running an upgrade in production for Socorro looks like. Awesome right?
• Here’s what a migration looks like.
• Here’s a configuration file.
• Generating a migration from the command line might look something like:
  alembic revision -m "bug XXXXXX Add a new table" --autogenerate

The most difficult thing to deal with so far are the many User Defined Functions that we use in Socorro. This isn’t something that any migration tools I tested deal well with.

Happy to answer questions! And I’ll see about making a longer talk about this transition soon.

From Planet PostgreSQL. Published on May 17, 2013.

Do you use PostgreSQL and truly believe it’s “the world’s most advanced open source database” and that its upcoming 9.3 release will make it even more awesome?

Do you also use Python and believe it’s “an easy to learn, powerful programming language” with “elegant syntax” that makes it an ideal language for developing applications and tools around PostgreSQL, such as Pyrseas?

Then we could use your help. For starters, we want to add support for the MATERIALIZED VIEWs and EVENT TRIGGERs coming up in PG 9.3.

We have also been requested to add the capability to load and maintain “static data” (relatively small, unchanging tables) as part of yamltodb, so that it can be integrated more easily into database version control workflows.

And for the next release, Pyrseas 0.7, we’d like to include the first version of the database augmentation tool which will support declarative implementation of business logic in the database–starting off with audit trail columns. Some work has been done on this already, but it needs integration with the current code and tests.

Or perhaps coding is not your forte, but you’re really good at explaining and documenting technical “stuff”. Then you could give us a hand with revamping the docs, maybe writing a tutorial so that users have a smooth ride using our tools.

Or maybe you have your own ideas as to how improve the PostgreSQL version control experience. We’d love to hear those too.

If you’d like to help, you can fork the code on GitHub, join the mailing list and introduce yourself, or leave a comment below.

By Reinout van Rees from Planet Django. Published on May 17, 2013.

view.request = request
view.args = args
views.kwargs = kwargs

By Reinout van Rees from Planet Django. Published on May 17, 2013.

Dicts

This works now:

>>> from collections import defaultdict
>>> data = defaultdict(list)
>>> data['key'].add(42)

It was added in python 2.5. Previously you'd do a manual check whether the key exists and create it if it misses.

Sets

Sets are very useful. Sets contain unique values. Lookups are fast. Before you'd use a dictionary:

>>> d = {}
>>> for each in list_of_things:
...     d[each] = None
>>> list_of_things = d.keys()

Now you'd use:

>>> list_of_things = set(list_of_things)

Sorting

You don't need to turn a set into a list before sorting it. This works:

>>> something = set(...)
>>> nicely_sorted = sorted(something)

Previously you'd do some_list.sort() and then turn it into a set.

Sorting with cmp

This one is old::

>>> def compare(x, y):
...     return cmp(x.something, y.something)
>>> sorted(xxxx, cmp=compare)

New is to use a key. That gets you one call per item. The comparison function takes two items, so you get a whole lot of calls. Here's the new:

>>> def get_key(x):
...     return x.something
>>> sorted(xxxx, key=get_key)

Conditional expressions

This one is very common!

This old one is hard to debug if blank_choice also evaluates to None:

>>> first_choice = include_blank and blank_choice or []

There's a new syntax for conditional expressions:

>>> first_choice = blank_choice if include_blank else []

Constants and loops

Put constant calculations outside of the loop:

>>> const = 5 * a_var
>>> result = 0
>>> for each in some_iterable:
...     result += each * const

Someone suggested this as an old-dated pattern. You can put it inside the loop, python will detect that and work just as fast. He tried it out and it turns out to depend a lot on the kind of calculation, so just stick with the above example.

String concatenation

Which of these is faster:

>>> ''.join(['some', 'string'])
>>> 'some' + 'string'

It turns out that the first one, that most of us use because it is apparently faster, is actually slower! So just use +.

Where does that join come from then? Here. This is slow:

>>> result = ''
>>> for text in make_lots_of_tests():
...     result += text

And this is fast:

>>> result = ''.join(make_lots_of_tests())

The reason is that in the first example, the result text is copied in memory over and over again.

So: use .join() only for joining lists. This also means that you effectively do what looks good. Nobody will concatenate lots of separate strings over several lines in their source code. You'd just use a list there. For just a few strings, just concatenate them.

That's the nice thing of Python: if you do what looks good, you're mostly ok.

By Mercurytide: Django articles from Planet Django. Published on May 17, 2013.

By Reinout van Rees from Planet Django. Published on May 17, 2013.

By Reinout van Rees from Planet Django. Published on May 17, 2013.

By Reinout van Rees from Planet Django. Published on May 17, 2013.

By Reinout van Rees from Planet Django. Published on May 17, 2013.

By Reinout van Rees from Planet Django. Published on May 17, 2013.

By Reinout van Rees from Planet Django. Published on May 17, 2013.

By Reinout van Rees from Planet Django. Published on May 17, 2013.

By Reinout van Rees from Planet Django. Published on May 17, 2013.

By Reinout van Rees from Planet Django. Published on May 16, 2013.

By Reinout van Rees from Planet Django. Published on May 16, 2013.

By Reinout van Rees from Planet Django. Published on May 16, 2013.

By Reinout van Rees from Planet Django. Published on May 16, 2013.

By Reinout van Rees from Planet Django. Published on May 16, 2013.

By Reinout van Rees from Planet Django. Published on May 16, 2013.

>>> name = 'ExampleClass'
>>> bases = (object,)
>>> attrs = {'__init__': lambda self: print('Hello from __init__')}
>>> ExampleClass = type(name, bases, attrs)
>>> example = ExampleClass()
Hello From __init__
>>> type(example)
<class '__console__.ExampleClass'>

By Reinout van Rees from Planet Django. Published on May 16, 2013.

By Reinout van Rees from Planet Django. Published on May 16, 2013.

By Reinout van Rees from Planet Django. Published on May 16, 2013.

By Reinout van Rees from Planet Django. Published on May 16, 2013.

By Reinout van Rees from Planet Django. Published on May 15, 2013.

By Reinout van Rees from Planet Django. Published on May 15, 2013.

By Reinout van Rees from Planet Django. Published on May 15, 2013.

By Reinout van Rees from Planet Django. Published on May 15, 2013.

By DjangoCon Europe from Planet Django. Published on May 15, 2013.

We are thrilled to announce that during a DjangoCon sprint Daniele Procida will lead a workshop “Don’t be afraid to commit”. The workshop is addressed to all of you who want to contribute to open source projects but are not sure how to do it.

The workshop will take participants through the complete cycle of identifying a simple issue in a Django or Python project, writing a patch with tests and documentation, and submitting it.

The workshop will take about 3h starting at 11:00 on Saturday in the same place as Sprints. Since the number of attendes is limited (12 people), please make sure to sign up here: http://djangocon-workshops.eventbrite.com as soon as possible!

By Reinout van Rees from Planet Django. Published on May 15, 2013.

By Reinout van Rees from Planet Django. Published on May 15, 2013.

By Reinout van Rees from Planet Django. Published on May 15, 2013.

By Reinout van Rees from Planet Django. Published on May 15, 2013.

By Reinout van Rees from Planet Django. Published on May 15, 2013.

By Reinout van Rees from Planet Django. Published on May 14, 2013.

By Ed Menendez from Planet Django. Published on May 14, 2013.

Here's a gist for a drop-in Django manager class that allows you to return a random row.

Model.objects.random()

It can be used in your models.py like this:

class QuoteManager(RandomManager):
    def random_filter(self):
        return self.filter(is_active=True)

class Quote(models.Model):
    quote = models.TextField()
    by = models.CharField(max_length=75)
    is_active = models.BooleanField(default=True)

    objects = QuoteManager()

    def __unicode__(self):
        return self.by

Advantages over using the order_by('?') is performance. Random sort at the database seems to be extremely slow on most databases even if the table only has a few thousand rows. Note that the count of records is cached for 5 minutes, so if the table changes often you may want to change that. A limitation is that it only returns one row.

By Reinout van Rees from Planet Django. Published on May 14, 2013.

By DjangoCon Europe from Planet Django. Published on May 12, 2013.

With a few days to DjangoCon, we thought it’d be nice to let you know how we are going to make your stay in Warsaw even NICER. 

The Nicer app is here! Available for both Android and iOS, smartphones and tablets. By downloading this app, you will always be connected to us, organizers. We will update you with hot news, unexpected agenda changes and post you little tips so you can spend awesome time in Warsaw.

Get the app of your choice here: http://getnicer.com/apps
Then simply follow DjangoCon Europe 2013. Make sure to turn on push notifications so you won’t miss an important event!

Other than that, make sure to follow us on twitter: @djangocon, tweet using the #DjangoCon hashtag, check out our pictures on instagram and videos on Vine to get a full coverage :) 

See you in Warsaw really SOON!

By Torsten Engelbrecht from Planet Django. Published on May 12, 2013.

I was recently experimenting with building an API with django-tastypie and make it accessible via CORS, so it can be used from a different host from an AngularJS app.

For the Django part it was relatively straightforward. I could have either written my own Middleware, dealing with incoming CORS requests, but decided to use django-cors-headers in the end. Following the instructions in the github repo and adding my host where AngularJS is hosted to the CORS_ORIGIN_WHITELIST setting did enable the Django server to handle CORS.

With AngularJS it was a little more tricky, mainly because information is spread all over the web. Beside the fact that I was trying to implement a service using ngResource to communicate with the API, the following did enable AngularJS to send its requests with the appropriate CORS headers globally for the whole app:

var myApp = angular.module('myApp', [
    'myAppApiService']);

myApp.config(['$httpProvider', function($httpProvider) {
        $httpProvider.defaults.useXDomain = true;
        delete $httpProvider.defaults.headers.common['X-Requested-With'];
    }
]);

_

Just setting useXDomain to true is not enough. AJAX request are also send with the X-Requested-With header, which indicate them as being AJAX. Removing the header is necessary, so the server is not rejecting the incoming request.

By DjangoCon Europe from Planet Django. Published on May 12, 2013.

Mozilla hardly requires introduction. They make Firefox. And Thunderbird. And Persona. And other web stuff. They also fight for the users, by keeping the Web open and diverse. All this while being non-profit organization. They also help others, like, you know, sponsoring conferences :-)

What are they up to now? Firefox OS seems to be the new, hot project. A mobile operating system running Linux kernel that you can write applications in technologies you already know: HTML and JavaScript. Combine it with Firefox Marketplace and you have complete mobile ecosystem. Check them out at Mozilla Developer Network.

We are super excited to have two speakers from Mozilla: Andy McKay will speak about processing payments in Marketplace and Tarek Ziadé will talk about… circus!

By Thierry Schellenbach from Planet Django. Published on May 11, 2013.

Django Facebook now officially supports Django 1.5 and custom user models! Go try it out and upgrade to pip version 5.1.1. It’s backwards compatible and you can choose if you want to keep on using profiles, or migrate to the new custom user model. Installation instructions can be found on github.

Contributing

Thanks for all the contributions! My startup (Fashiolista) depends on a reliable Facebook integration and maintaining it would not be possible without all the pull requests from the community. Contributions are strongly appreciated. Seriously, give Github a try, fork and get started :)

About Django Facebook

Django Facebook enables your users to easily register using the Facebook API. It converts the Facebook user data and creates regular User and Profile objects. This makes it easy to integrate with your existing Django application.

I’ve built it for my startup Fashiolista.com and it’s currently used in production with thousands of signups per day. For a demo of the signup flow have a look at Fashiolista’s landing page (fashiolista.com)

After registration Django Facebook gives you access to user’s graph. Allowing for applications such as:

• Open graph/ Timeline functionality
• Seamless personalization
• Inviting friends
• Finding friends
• Posting to a users profile

Django Facebook helps you quickly develop Facebook applications using Django.
Let me know what features or issues you are encountering!

By Daniel-Greenfeld from Planet Django. Published on May 10, 2013.

In the summer of 2010 Frank Wiles of Revsys exposed me to what I later called the "Easy Form Views" pattern when creating Django form function views. I used this technique in a variety of places, including Django Packages and the documentation for django-uni-form (which is rebooted as django-crispy-forms). At DjangoCon 2011 Miguel Araujo and I opened our Advanced Django Forms Usage talk at DjangoCon 2011 with this technique. It’s a pattern that reduces the complexity of using forms in Django function-based views by flattening the form handling code.

def my_view(request, template_name="my_app/my_form.html"):

    if request.method == 'POST':
        form = MyForm(request.POST)
        if form.is_valid():
            do_x() # custom logic here
            return redirect('home')
    else:
        form = MyForm()
    return render(request, template_name, {'form': form})

def my_view(request, template_name="my_app/my_form.html"):

    form = MyForm(request.POST or None)
    if form.is_valid():
        do_x() # custom logic here
        return redirect('home')
    return render(request, template_name, {'form': form})

By Reinout van Rees from Planet Django. Published on May 10, 2013.

By DjangoCon Europe from Planet Django. Published on May 09, 2013.

You probably thought one party is good enough for a conference. We’re here to prove you wrong! Awesome guys from Base are helping us organize Base Party on Wednesday, 15th May at 9pm in Klub Balsam.

Base is the only CRM built for people. They believe that by 2020, business software will be radically different. Base is paving the way by building the next generation of CRM software. Their mission is to make you and your team 10x more productive.

Most importantly, they’re building the best tech team in Europe. With a big vision and small, highly talented team, Base is creating an amazing place to work for self-driven and dynamic people. Watch this short video to know them better:

They’re looking for great python developers to join their team. Make sure to drop by Base Party at 9pm, Wednesday in Klub Balsam! 

By Andrew Godwin from Planet Django. Published on May 09, 2013.

Welcome to the first of my Django Diaries, where I'll be detailing the progress I'm making on my Schema Alteration project.

After a very successful Kickstarter, I had the unfortunate situation of a couple of successive trips abroad, and so initial work has been a bit more delayed than I would have liked. However, thanks to securing more time to work on the project every week, progress should be faster than planned from now on.

The plan is that these diaries will contain a rough summary of the work I've been doing; they're here both to help engage you (the slightly-too-interested public) in the work I'm doing, as well as providing some transparency.

If you want to hear more about a certain issue, feel free to get in touch with me - see the About page for my contact details. I'd love to explain as much as I can to those who are interested!

new_app_cache = BaseAppCache()

new_app_cache = BaseAppCache()
class Author(models.Model):
    class Meta:
        app_cache = new_app_cache

By DjangoCon Europe from Planet Django. Published on May 08, 2013.

Who doesn’t know yet that we’re doing DjangoCon Europe in a circus tent? But circus is no place for DjangoCon Sprints! We were looking for the kind of space that will accommodate 150 people and at the same time would be located in the heart of Warsaw (it will be weekend so it’s better to be closer to the epicenter of parties, right?).

Thankfully, we found the perfect venue! It’s called GammaFactory - a startup hub/co-working center!! It is based in the.. cheese factory :) Ok, it was a cheese factory, but now it’s a place with a vibrant community around it. If you happen to be a rock climber - the biggest bouldering center in the centre of Warsaw is next door to GammaFactory! So don’t forget your climbing shoes!

HardGAMMA Ventures, which owns GammaFactory, is one of the leading early stage VC funds in Poland (

Visit their site for more info! http://www.hardgamma.com/  

They also run a startup accelerator program for technology entrepreneurs called GammaRebels. It focuses on accelerating and developing startups through mentoring, advising and sharing both business & technical knowledge.

See you soon! :)

By DjangoCon Europe from Planet Django. Published on May 07, 2013.

Let me introduce you to New Relic, our Platinum Sponsor.

New Relic is application performance management company. They offer web and mobile application monitoring with support for many languages, including Python, of course, as well as Ruby, PHP, .NET, Java, Android and iOS.

With their great product they make life easier for more than forty thousands clients monitoring staggering number of 1.4 million of application instances.

Using New Relic you can easily dive into your application performance breakdown and see what parts are slow. It’s very easy to use and you can install it in matter of minutes, as we did for our conference website!

If that sounds interesting, be sure to check out Amjith’s talk during DjangoCon!

Be sure to check out their website at http://newrelic.com/

By David Cramer from Planet Django. Published on May 07, 2013.

By Adrian Holovaty from Planet Django. Published on May 06, 2013.

My friend and fellow Chicagoan Evan Miller wrote an excellent blog post over the weekend: Why I Develop For The Mac. It's full of great reasons why his software (which is also excellent, by the way) was written for the desktop, despite the fact that he's a web developer, even the creator of an Erlang web framework.

But I'm compelled to respond to it, specifically his statements about <canvas>:

large <canvas> areas seem laggy on most browsers

So I'm left with <canvas>, and <canvas> is slow.

I have become intimately familiar with <canvas> while developing Soundslice. I'd even venture to say Soundslice is one of the most advanced uses of <canvas> on the web that's not a tech demo -- i.e., it's an application that normal people use. The site uses not one, but nine <canvas> elements stacked on top of each other to make a very rich UI, sort of like Photoshop for guitar tabs. (For a flashy demo of how those canvases interact, watch the tech talk I gave at 37signals, specifically the bit starting at 10:20.)

Here's what I've learned: <canvas> is not slow. In fact, I've been continually surprised by how fast it is -- as long as you take care to do things right. Evan's article mentions the "magical" sensation of instantaneous feedback; I invite you to play with the zoom slider on any Soundslice page (example) to experience this same magic, all drawn dynamically with <canvas>.

Of course, <canvas> is certainly not as fast as the lower-level drawing routines that you can use if you develop a desktop app. No question. But it's fast enough that, unless you're doing something relatively insane, you'll be totally fine.

On Soundslice, we're drawing guitar-chord charts completely on the fly (again, see an example), which is a relatively involved drawing routine -- and it's still near-instant performance. That's across all modern browsers (Chrome, Safari, Firefox and IE 10).

Here are some specific tips I've picked up to make <canvas> performance really shine.

Use requestAnimationFrame

Above all else, do this.

It's a JavaScript API designed to fix a very specific problem: your computer screen can only be redrawn a certain number of times per second (the "refresh rate"), so any calculations that redraw more often than your refresh rate are wasteful.

For example, say you have an event such as mousemove that results in a <canvas> redraw. A mousemove might happen hundreds of times per second, but your screen might only refresh, say, 75 times per second (75 Hz). That means, if your code is naively written, it will try to redraw several times within each actual opportunity to redraw (hundreds of times per second vs. 75 actual redraw opportunities per second).

The requestAnimationFrame API solves this by letting you say, "Execute this code the next time a redraw happens." Which saves your browser from having to do unnecessary work.

When I added this to Soundslice, the site became dramatically faster and more responsive. Here's more info about how to use the API.

Stack canvases

Above, I linked to a video of a tech talk I gave about Soundslice. In that talk, I demonstrated how Soundslice uses several different <canvas> elements, stacked on top of each other as layers, for maximum performance -- and for nice, clean code. Definitely watch the demo at around 10:20 in the video to get a sense of it.

I'm planning to write a separate blog post about this, but the Cliff's Notes version is that you can stack transparent <canvas> elements on top of each other so that you only have to redraw the ones that need to change.

For example, on Soundslice, there's a separate <canvas> for the playhead -- the vertical orange line that tracks the currently played moment of the video. That's a separate <canvas> with a z-index above the other ones, so that redrawing it doesn't require redrawing any of the other stuff. The less you have to redraw, the better.

Bunch calls to fillStyle

When you draw on <canvas>, you first have to tell it which color you're using. You can do that by setting the "fillStyle." It turns out that, each time you change the fillStyle, there's a slight performance penalty. Therefore, you can squeeze out some extra performance by bunching your calls to fillStyle -- that is, rather than drawing a gray thing, then an orange thing, then a gray thing again, you should draw all the gray things, then draw all the orange things.

For example, Soundslice, which is all about annotating YouTube videos, needs to draw dozens, sometimes hundreds, of annotations on the screen at a time. Each annotation might use several different colors -- the text color, the border color, the line color, etc.

My original implementation looped over each annotation and drew each one independently, which resulted in two to five fillStyle calls for each annotation. I changed this to bunch the fillStyles across all annotations -- so that all of the light grays were drawn at the same time, then all the dark grays, etc. -- and the drawing got a few dozen milliseconds faster.

For more background, see the "Avoid unnecessary canvas state changes" section in this great HTML5 Rocks article.

Cache text rendering

In profiling, I've found that rendering text on <canvas> is my next big rendering-related bottleneck on Soundslice. I haven't done this yet, but I'm planning to come up with a way of caching the results of fillText, possibly using this technique.

Final thoughts

A decent argument in Evan's favor is: "Well, if <canvas> is only fast if you use these various hacks, it's not really fast, then, is it?"

Two thoughts on that.

First, well, sure! I'd love it if <canvas> was super fast right out of the box, without needing to use these techniques. No doubt about it. But the reality is, it is fast enough, if you put in the work.

Second, there's the bigger question -- a defining question for the current generation of web developers -- which is: web or native app? I am squarely in the web camp, both for philosophical reasons (such as openness) and practical reasons (such as the fact that Soundslice has only one developer and one designer, and we can't justify building separate apps for separate platforms).

What I love about <canvas> is that it lets us make desktop-quality apps right in the browser, so we can get the benefits of being "of the web" along with the benefits of amazing, fast graphics. Fear not, my friends: <canvas> is great.

UPDATE, May 7, 2013: Evan has posted a thoughtful follow up, reacting to this.

By DjangoCon Europe from Planet Django. Published on May 06, 2013.

Guess who is throwing the best party of 2013?:) 

We’re super excited to announce that SendGrid is a part of our DjangoCircus family!! :) Moreover, they were one of the first companies to back us making all this possible! Kudos to all Sendgriders, especially to Swift :)

We’re planning BBQ with Polish KIEŁBASA ;) a lot of great beer and.. the DjangoCon FlipCup tournament! The party will take place in the circus and hopefully outside - if the weather is good. Join us on Friday night at 7pm.

SendGrid is the leader in email deliverability. SendGrid’s cloud-based platform increases email deliverability, provides actionable insight and scales to meet any volume of email, relieving businesses of the cost and complexity of maintaining custom email infrastructures. 

For more information, visit www.sendgrid.com.

See you at SendGird Party!

By Arne Brodowski from Planet Django. Published on May 05, 2013.

Das nächste Treffen der Django-UserGroup Hamburg findet am Mittwoch, den 08.05.2013 um 19:30 statt. Dieses Mal treffen wir uns wieder in den Räumen der intosite GmbH im Poßmoorweg 1 (3.OG) in 22301 Hamburg.

Die Organisation der Django-UserGroup Hamburg findet ab jetzt über Meetup statt. Um automatisch über zukünftige Treffen informiert zu werden, werdet bitte Mitglied in unserer Meetup-Gruppe: http://www.meetup.com/django-hh

Da wir in den Räumlichkeiten einen Beamer zur Verfügung haben hat jeder Teilnehmer die Möglichkeit einen kurzen Vortrag (Format: Lightning Talks oder etwas länger) zu halten. Konkrete Vorträge ergeben sich erfahrungsgemäß vor Ort.

Eingeladen ist wie immer jeder der Interesse hat sich mit anderen Djangonauten auszutauschen. Eine Anmeldung ist nicht erforderlich, hilft aber bei der Planung.

Weitere Informationen über die UserGroup gibt es auf unserer Webseite www.dughh.de.

By DjangoCon Europe from Planet Django. Published on May 05, 2013.

DjangoCon is in a less than two weeks. Are you already excited? We definitely are!

Since a lot of you will be in Poland for a first time in their life, we prepared some tips for you. We really want to make sure you will enjoy being in Warsaw this year!

We already covered some basics here: http://djangocircus.com/getaround/ (taxis, buses, trains, sim cards, internet).

Here’s some more.

Currency

Polish currency is PLN (złoty). 1PLN is around ~ €0.24 or ~$0.32. You will find a lot of ATMs (bankomat) in Warsaw, so you don’t need to have a lot of cash with you. It is possible to pay with debit or credit cards in most of the shops (MasterCard, Visa, Visa Electron, Maestro, PolCard etc. are widely used), but there are places where only cash is accepted or it is possible to pay with card when you spend more than 10 or 20PLN.

You can also exchange some money in a bank or in a kantor (you will find a lot of kantors in the city center). If it’s possible ask for banknotes with lower denomination (10, 20, 50).

Power plugs

In Poland we use Type E power plugs. If you use different kind of power plug, remember to buy an adapter. We have also an extra tip for you: if you bring a power strip you’ll be able to charge a lot of your devices using only one adapter :).

Tap water

If you ask Poles if Polish tap water is good enough to drink, they will tell you that you shouldn’t do it. However it’s no longer true. Water in Warsaw (we cannot assure it is a case in other places in Poland) according to newest water tests is safe to drink. It is probably not the tastiest water in the world though, so drinking still water which you can buy in many shops around Warsaw is a better option in our opinion.

Food

Make sure to try a little bit of Polish cusine while being in Warsaw. Some of Polish specialities are: bigos, pierogi, gołąbki, barszcz, żurek, zupa ogórkowa and many, many other.

Language

Young people speaks English quite good in Warsaw. People 50+ may have some troubles with understaning English, but it’s not a rule - in this case try German or Russian if you know one of those languages. In most of restaurants and pubs (and in many shops) people speak English, so we are sure you’ll be fine.

If you want to know how to pronounce Polish words (for example “Służewiec”) here are some useful pages you can read: Polish alphabet, Polish phonology :).

Safety

Be sure to watch your belongings while you are in a crowded place. In many places there are pitpockets. Loosing your wallet and documents won’t be a perfect memory from Poland.

By Daniel Lindsley from Planet Django. Published on May 04, 2013.

By Inka Labs from Planet Django. Published on May 03, 2013.

The django unit test framework is great, it allows you to test everything you code in python. The "TestCase" class also supports fixtures. Most people writes fixtures as XML or JSON and loads them in every test. This can make running the test suite a very very slow process. You shuold consider that every time you run a test, the fixtures are loaded in the database, then they are deleted and loaded again by the next test. 

We had a simmilar problem some weeks ago. We had all our fixtures in json files (some of them were huge). So we decided to avoid those fixtures. That is why we decided to create our objects (per test) in pure python. Something like:

obj = SomeModel.objects.create(
          attr1=1,
          attr2=2,....)
related = OtherModel.objects.create(
            fk=obj,
            other_field=...)

#use related and obj

But, let's be reallistic, this is very boring and tedious. We don't want to spend soo many time creating database objects. Also consider you add a required field in a model, then you should go to all your tests using that model and add a value for that field, making your tests difficult to maintain.

Here is where django dynamic fixture comes to the rescue. This great application allows you to create python database objects very very easy.

# create a database object with random data
obj = G(SomeModel)

# Give some field values 
# and fill the remaining fields with random data
obj = G(SomeModel, attr1="field1", attr2="Field2")

# ignoring fields
obj = G(SomeModel, ignore_fields=['field1'])

This is just a very small example of all features of this great application. If you want your tests to run fast and be maintainable you shall give it a try. Check the docs here

By Daniel-Greenfeld from Planet Django. Published on May 03, 2013.

Because of the ubiquitousness of reStructuredText in the lives of Python developers and the advocacy of it, it's not uncommon for people to assume we used it to write our book. However, that's not really the case.

The short Answer is we used:

• reStructuredText (RST)
• Google Documents
• Apple Pages
• LaTeX

The long answer is the rest of this posting. Since writing the book was broken up into three major stages 'alpha', 'beta', and 'final', so have I broken up blog article.

\subsection{Model Inheritance in Practice: The TimeStampedModel}
It's very common in Django projects to include a \inlinecode{created} and \inlinecode{modified} timestamp field on all your models. We could manually add those fields to each and every model, but that's a lot of work and adds the risk of human error. A better solution is to write a \inlinecode{TimeStampedModel} \index{TimeStampedModel} to do the work for us:

\goodcodefile{chapter_06/myapp/core/timestampedmodel.py}

Take careful note of the very last two lines in the example, which turn our example into an abstract base class: \index{abstract base classes}

\goodcodefile{chapter_06/myapp/core/class_meta.py}

By defining \inlinecode{TimeStampedModel} as an abstract base class \index{abstract base classes} when we define a new class that inherits from it, Django doesn't create a \inlinecode{model\_utils.time\_stamped\_model} table when syncdb is run.

By chrism from plope. Published on Dec 24, 2012.

By chrism from plope. Published on Jun 04, 2012.

By chrism from plope. Published on May 15, 2012.

By chrism from plope. Published on Mar 03, 2012.

By chrism from plope. Published on Jan 01, 2012.

By chrism from plope. Published on Dec 24, 2011.

By chrism from plope. Published on Dec 19, 2011.

By chrism from plope. Published on Nov 10, 2011.

By chrism from plope. Published on Oct 17, 2011.

By chrism from plope. Published on Oct 15, 2011.

By chrism from plope. Published on Jun 12, 2011.

By chrism from plope. Published on May 04, 2011.

By chrism from plope. Published on Apr 14, 2011.

By chrism from plope. Published on Mar 19, 2011.

By chrism from plope. Published on Feb 02, 2011.