Unix Permissions 101

1. Multiple servers, multiple configs
- 1.1. Debugging
- 1.2. Analysis
2. sudo
- 2.1. sudo justExecuteThis
- 2.2. sudo mode
3. Unix Permissions 101

DRAFT

1 Multiple servers, multiple configs

Couple of days ago we discovered a nasty bug in one of our systems. Before getting to the bug let me explain the setup very briefly:

We have a master node which wakes up several AWS instances, delegates series of jobs, from a queue, to them and after all of the jobs are completed switches them off. Each slave node, after completing a job, stores the result in a common database.

Each of the event is logged in the same common database.

The system was working as expected until we found that the results calculated by the slave nodes are being stored in a development database and not in the production database. Initially it looked like we plugged in wrong configuration file, which of course wasn't the case. I checked the code that talked to the database to see if someone hard coded the Dev DB configuration and later forgot to revert. Nothing helped.

I even considered the possibility of multiple configuration files and tried grepping the whole repository for Dev DB credential. Nothing was mentioned anywhere about the Dev DB.

1.1 Debugging

The master node packs the Python scripts, configuration files and other data files nicely in an egg (zip file) and deploys it to the slave nodes. I sshed into one of the slaves and to my horror saw that the job did have configuration of the Dev DB! This was surprising since while the master node didn't have a trace of the incorrect configuration, the egg contained it.

My job was now a bit easier, I just needed to figure out how the unwanted configuration slipped into the egg. I went over the code and double checked the path of configuration declared for making the egg. Ultimately, I manually executed the script that does the zipping of code and deployment. I found that the zipping operation failed due to insufficient permissions. ls -l revealed that the build folder was owned by the root and was readonly for others. The script couldn't clean the temporary files and failed. Since this operation was done using Python subprocess and the exception was handled in a way that didn't stop the script, the script went on failing to deploy to each slave.

When the slaves were ordered to execute the job, they executed the last available version of the egg, which happened to be a job which was being tested on development server and thus contained development configuration.

1.2 Analysis

After the file permissions were fixed, it was time for examining what went wrong. And what could have been done so that finding the cause would have been easier. The fatal mistake was to handle the subprocess exception and not stop. This is exactly opposite of the Unix Philosophy¹:

Rule of Repair: Repair what you can — but when you must fail, fail noisily and as soon as possible.

Instead of silently consuming the exception, the script should have failed and log it. Still this isn't the root cause of the bug. The root cause is setting up of incorrect file permissions.

2 `sudo`

2.1 `sudo` justExecuteThis

The main problem is not technical but, like most of the other problems in software development, a social one. We want to get things working, fast. Unable to save a file? - use sudo. Unable to install package? - use sudo. I can bet most of the engineer would repeat the last command, if it failed, prefixing it with sudo without giving it a thought. For most of us, sudo means:

I don't really understand (or want to know) why you failed but I don't want you to fail.
Here, I give you the power of sudo

Again, the problem is not technical. Fixing the permission is not very difficult. But why would one get into the trouble of actually reading up the commands and doing it the right way, when one can just sudo.

2.2 `sudo` mode

After one has successfully executed couple of commands using sudo, which were failing earlier, it is very tempting to get into sudo mode – su. Switch to superuser and one never have to type in sudo again.

Using su (login as superuser) we can go anywhere and do anything we wish, there is no one to stop us. But whenever we create a new file, other ordinary users may only be able to read it and not write it.

Most of the code we write is executed by non privileged users.

3 Unix Permissions 101

There are just 3 commands one needs to know to avoid 90% of the unnecessary use of sudo.

ls -l (list directory with a long listing format²):

This is the usual ls with -l flag which outputs extra information about the files in the directory.

-rw-r--r--  1 anuragpeshne  staff  1080 Feb  2 20:52 LICENSE
-rw-r--r--  1 anuragpeshne  staff    51 Feb  2 20:52 README.md
drwxr-xr-x  7 anuragpeshne  staff   238 May  1 20:57 assets
drwxr-xr-x  3 anuragpeshne  staff   102 May 21 16:15 css
drwxr-xr-x  9 anuragpeshne  staff   306 Jun  5 13:45 essays
-rw-r--r--@ 1 anuragpeshne  staff    53 Feb 28 20:41 googleb7e54271b39d6266.html
-rw-r--r--  1 anuragpeshne  staff  8556 Apr  2 00:11 index.html
drwxr-xr-x  3 anuragpeshne  staff   102 Apr  1 22:32 js
-rw-r--r--  1 anuragpeshne  staff  6565 Feb 20 23:08 now.html
drwxr-xr-x  8 anuragpeshne  staff   272 Apr  2 00:11 org-files
drwxr-xr-x  4 anuragpeshne  staff   136 Feb 21 23:02 quotes
drwxr-xr-x  3 anuragpeshne  staff   102 Feb 16 20:09 todo

The first 10 characters denote file permissions. The first character denotes whether the file is regular file or directory³. 3 Sets of 3 characters follow that. Each character in the set denotes whether it is [w]ritable, e[x]ecutable or [r]eadble. The sets represents permission of Owner, Group and Other. The third, fourth column gives the information about owner and group.

Thus in the above example, only anuragpeshne can write to file README.md while users in group staff and others can just read it.

chown (change owner⁴):

If you are working in some folder above your home folder, most probably it won't be owned by you; It will be owned by root. If you are editing files in your repository and going to edit those many times in a day, it would be a good idea to own the repository folder.

sudo chown -R $(whoami) ./
chmod (change mode⁵):

There are couple of ways to change file permissions, but I'm going to mention just one way. chmod ugo file, where ugo are octal values representing permission for user, group and others. For calculating these octal value, just add up the number corresponding to the permission you want to give.

read 4

write 2

execute 1

Examples:
- chmod 600 d.txt will give permission to read and write only to the user (owner).
- chmod 755 app will give permission to read and execute to everyone but only the user can write to it.

And that's it! From now on whenever you are tempted to use sudo, give yourself a moment to think if you are going to edit the file often, if so, change the ownership to you or add yourself to the owner group.