Django on Steroids

self.__doc__

• Co-founder and CTO at DoSelect.
• Using Python since 2010, Django in production since 2014.
• Rant at sanketsaurav.com.
• Rant within 140 characters at @sanketsaurav.
• Spaces over tabs. Vim over Emacs. (⌐■_■)

Motivation

• DoSelect is primarily built on Python and Django.
• We're currently in the middle of a scaling frenzy.
  • Serving millions of unique assessment sessions, several millions of code executions.
  • Grown into a heterogenous stack, all glued together with Python.
  • Scaling up our engineering team at the moment.
• We've solved numerous scaling problems in the past 3 years.

What is web scale?

• Traffic increases exponentially.
• Fleet architecture and resource consumption grows at far-greater-than-linear rates.
• Application complexity increases at greater-than-linear rates.
• Team size increases substantially.

Optimize for product velocity.

Optimize for developer happiness.

Scaling teams is harder than scaling products.

Why choose Django?

• Django is easy to become productive with.
• Common sense design patterns. Simple, clean, and favors pragmatism.
• Opinionated and comes with batteries. But easy to break out of the box when needed.
• Great community support. Matured language and framework.

Frameworks don't scale. Applications do.

How does the stack scale?

Re-think stack architecture

• Too many apps in one project is not a good idea.
• Keep your models together.
• Your most critical business logic should be accessible from outside Django.
• Optimize for stack heterogeneity.

What's your core reliability metric?

Case Study: DoSelect's Async Fleet

When we began...

• Most of the application logic was a Django monolith.
• Entire async fleet was on Celery.
• Python and Django cannot solve all problems. We needed multiple other technologies to solve different problems. e. g. Go, NodeJS, Docker, Kubernetes etc.
• HTTP-based services were very difficult to scale reliably for our use case.

We had to optimize for reliability.

What we did...

• Django auth service can be used by any other component in any other language.
• Django models can be used by any other Python-based project.
• Celery tasks can now be written in Go, and can be invoked from Python (and vice versa).
• Async all the things (that we could).

Digging deeper into details...

Tuning your WSGI server

• Use Gunicorn. It's awesome.
• Introspect your application to understand the nature of hosts you need.
• Determine worker_class. Know when to use sync and async workers.
• Adjust params like workers, worker_connections, keepalive, etc. wisely.
• There is such a thing as too many workers.

Tuning your proxy server

• Use NGINX for fun and profit.
• worker_processes auto;.
• Adjust keepalive_timeout.
• Turn on tcp_nopush and tcp_nodelay.
• gzip all the things!

What's taking so long?

• Use Chrome Developer Tools to analyze and find out the rate-limiting step. Rinse. Repeat.
• EXPLAIN ANALYZE the complex queries.
• Python is most likely not your performance bottleneck. Everything else might just be.
• Use caching often, and intelligently.

Caching

• Cache All That You Can But No More^TM
• Use Redis as your primary cache. Cache very often:
• All user sessions.
• User model lookups.
• Resource-level caching.
• Use a CDN for all things the user cannot change.
• Static files, images, etc.
• HTML templates, if possible.
• Varnish.

Django ORM

• ORM is awesome.
• It's not always the silver bullet. Break away when you need to.
• Tail database logs to see what your ORM queries actually mean.
• Automatic relationship access can bite you. Expand relationships carefully.
• Add extra indices where needed.
• Offload heavy-lifting on the database whenever possible all the time.

Django ORM (contd.)

Django ORM (contd.)

• Understand how querysets work.
• If your're serving an API, tune your resources for list and detail views.
• Dehydrate finite data.
• Denormalize infinite data.
• Know when to switch to a full-fledged search engine, like Elasticsearch.

Scaling your database

• Use database-level connection pooling.
• Use pgBouncer if you're using Postgres.
• Monitor all your queries. Find what's holding you back.
• If you're using replication, it's a good idea to segregate reads and writes.

Let's do some Q & A!

• Follow me on @sanketsaurav
• We're hiring developers at DoSelect in Bangalore.
• Come talk to me about Django!