How to debug Python Azure Worker Roles

Two months ago I wrote a blog post on Python 3 Adoption. If you want to read the first post you can find it here.

In the two months since that blog post was written we have more data to process and I got to thinking: why shouldn’t we scale this up and increase the performance a bit? You’ll recall that I used a worker role on Azure to process incoming data,and with worker roles it’s really easy to scale up resources as needed toprocess updates faster. To learn how to Scale your Azure Service, I suggest you read the Azure Documentation on Scaling Cloud Services.

So I scaled my service to 3 VMs. I dialed the instance count up and awaited my reward of more throughput. But when I scaled things didn’t go as planned; instead, I noticed the instances were failing to start. What had gonewrong?!? Time to debug!

[Azure Worker Roles Failing To Start]

The easiest way to debug what is going wrong is to Enable Remote Desktop Access to your Azure Cloud Service and to look into the logs. So after following the documentation and RDP-ing you are on the machine. But where do you find the interesting information? Python Worker Roles store a few log files that will show issues that occur. Below you can see the log directory, which you can find under C:ResourcesDirectory. The LaunchWorker.txt and LaunchWorker.err.txt file are likely more interesting as they will show any errors in your process. And sure enough, LaunchWorker.err.txt shows an issue!

[Viewing Log Files On Worker Role]

Failed to import the site module
Traceback (most recent call last):
  File "F:approotenv34Libsite.py", line 703, in
    main()
  File "F:approotenv34Libsite.py", line 670, in main
    virtual_install_main_packages()
  File "F:approotenv34Libsite.py", line 553, in virtual_install_main_packages
    f = open(os.path.join(os.path.dirname(__file__), 'orig-prefix.txt'))
FileNotFoundError: [Errno 2] No such file or directory: 'F:\approot\env34\Lib\orig-prefix.txt'

This message seemed rather odd to me. My role ran locally, and nothing was wrong. Why is it missing a file from my virtual environment after deployment?

Well, as it turns out the issue is that the worker role isn’t really set up to use virtualenv environments – we designed it to install packages on deployment so that you have a smaller upload. Before deploying, generate a requirements.txt file from the environment you’re using, then in Visual Studio, activate a system Python install. This way you won’t be trying to copy a virtualenv on the Python Azure Worker Role. (Sometimes you may really need to copy the virtualenv, which is why you can do it, but most of the time it is easier to avoid it.)

We have more info about deploying to cloud service on our wiki, as well as some hints for deploying packages that need a compiler.

Hopefully this helps you debug your cloud service deployments in the future!

Oh, I nearly forgot to share the updated results of my PyPI analysis.

[PyPI Data May 5, 2016]

The original hypothesis was that by May 20, 2016, Python 3 would be better supported than Python 2. As it turns out, when we rerun the analysis with data through April we find that by April 30, 2016 we should anticipate Python 3 to be better supported. This is in the past now so we can see that this isn’t exactly how things have went. Though the data a bit of a gap now, they have crossed at one point, so the original prediction of May 20 looks pretty good – Python 3 will soon be better supported!