It was my first, and I had an amazing time meeting so many people and learning from both the talks and the hallway track! Our team had a heap of fun at the recent P圜on and PyData. We are always interested in doing more for the community, so please get in touch if there is something we can help with. We sponsor conferences such P圜on and Euro Python, provide support to local groups such as the Stockholm Python User Group and NYC PyLadies, host hackathons and contribute back to open source projects. Spotify is involved in the community in a number of ways. Part of what makes Python so special is the community around it. There is even a Django app or two! Community Python is also used for prototyping services, quick scripts, build processes and more. To simplify the integration with our Python services, we ported the GraphWalker runner to Python. Spotify squads often use GraphWalker to do model based testing of both user facing clients as well as some APIs. When it’s going all out we have seen over 6000 Python processes running over the hundreds of nodes in our Hadoop cluster. Simpler jobs power things like our top lists.Īround 90% of our map reduce jobs are written in Python. We use Luigi, along with a range of machine learning algorithms, to power our Radio and Discover features, as well as recommendations for people you may want to follow. This means you can quickly prototype complex data jobs. It handles the bundling of required libraries, and brings back any error logs to your local machine. Luigi allows you to quickly build complex pipelines of batch jobs from your own machine.
To simplify interactions with Hadoop, we use our Luigi package. Spotify teams make heavy use of analytics, both in decision making and within the product itself. This has included performance testing, profiling, cython, and native libraries. Some services are compute bound, and we’ve tried a range of strategies for how to handle this in Python. Earlier services were written using Twisted, and in the last few years we’ve preferred gevent. We also make heavy use of Python async frameworks to help services that are IO bound. Python fits well into this mindset, as it gets us big wins in speed of development. The non-Python services are typically written in Java, although we do have a few using C or C++.
Around 80% of these services are written in Python. Spotify’s backend consists of many interdependent services, connected by own messaging protocol over ZeroMQ. Python has a habit of turning up in other random places, as most of our developers are happy programming in it. Hopefully this post answers the question!Īt Spotify the main two places we use Python are backend services and data analysis. The most frequent question we heard at P圜on this weekend, was how do we use Python at Spotify.
0 kommentar(er)
