Stefan Heule's Blog

On the Importance of Safe Interfaces

Sun, 05 Mar 2017 00:00:00 -0800

Some features like footnotes work best when reading this post at stefanheule.com.

The recent Amazon S3 outage is a good reminder that humans make errors, and sometimes issue commands that they didn't mean to issue. On February 28th, Amazon's cloud offering S3 was partially offline for over four hours, because a human ran a routine command and got some of the arguments wrong. It's important to realize that it's not the human that is to blame, but rather the tool that did not sufficiently check the human's arguments. "Be more careful" is not a viable strategy to ensure that a software system works reliably, as humans will always eventually make errors. Instead, we need to design software with safe interfaces that check for bad arguments. In this post, I'll show another example of a badly designed interface, and how to fix it.

Safe Interface

I don't have a bullet-proof definition of what safe interfaces look like and it depends a lot on the context. But some guiding principles might be:

User inputs are validated, and invalid ones rejected. For valid inputs that are uncommon, the user might be asked to confirm.
Actions that have big effects have to be confirmed, and the expected effects are shown to the user. For instance, if a command takes all of your data center offline, then asking for confirmation is the least that should be required.
Whenever possible, actions can be undone, so that errors can be easily recovered from.

Deleting Files on the Command Line

Another example of a tool without a safe interface is rm. It is an ancient tool on Unix systems to delete files, and programmers and system administrators often use it on a daily basis. However, files deleted this way are not meant to be recovered¹, but instead deleted permanently. This leaves the door wide open for data loss if the arguments to rm are ever entered incorrectly. Unsurprisingly, the internet is full of people asking how to recover files inadvertently deleted this way. I don't blame any of these people, as rm is poorly designed to be used on a regular basis; its operation is permanent, and there are no safeguards against deleting even the most important files. Using rm -rf /, you can even delete everything on your computer (including rm).

So, we need a better alternative, and trash-cli is just that. It provides the command trash-put with roughly the same interface as rm, but moves files to the trashcan rather than permanently deleting them. This allows inadvertently deleted files to be easily restored. To avoid having to remember not to use rm, we can instead create an alias by adding the following line to our ~/.bashrc or ~/.zshrc:

alias rm='trash-put'

Possibly an even better solution is to use a new name, and override rm to remind you not to use it:

alias rm='echo "use d instead to avoid losing files."; false'
alias d='trash-put'

If for some reason we really need to run rm instead of trash-put, we can still do so via \rm to bypass our alias. But doing so should only be done in very rare cases and after carefully checking the command. And because the alias only shadows the rm binary in the shell, we can still run scripts that rely on rm.

No more lost files, as they can be easily restored with trash-restore. While this solution still leaves room for further improvement (e.g., checking against deleting files that aren't meant to be deleted, such as system files), it's a big step towards a safe interface to deleting files.

Sometimes, files can still be recovered with specialized recovery tools, though this is unreliable and only works well if the error is detected relatively quickly, before the deleted files get overwritten. ↩

Weather Information for Obsidian

Sat, 18 Jun 2016 00:00:00 -0700

Some features like footnotes work best when reading this post at stefanheule.com.

See the original blog post if you want to learn about Obsidian in general.

By far the most requested feature for Obsidian, my watch face for the Pebble smartwatch, was to add weather information. Over the last few days, I've implemented this feature. It's fully configurable and the weather information moves out of the way of watch hands so that it is never obstructed.

Here are a few screenshots of what the weather information looks like on the Pebble Time

and on the Pebble Time Round

The source of the weather information can be configured to be openweathermap.org or forecast.io, the temperature can be displayed in Celsius or Fahrenheit, and the color can be fully customized. By default Obsidian will display the current weather conditions, but this can be changed to today's conditions. One interesting feature that I haven't seen in other watch faces is that Obsidian allows for a mix between current conditions and today's weather in the following way: In the first part of the day, Obsidian will display today's weather, and then at 2pm it will switch to the current conditions. That allows the user to get a sense of today's weather in the morning with the day's maximum temperature (e.g., to decide what to wear), and then around the time of the highest temperature, the watch will switch to current conditions as today's high is no longer useful information. Finally, weather information is displayed for the current location by default, but it is possible to set a fixed location.

I have borrowed some of the fonts from Minimalin, another great watch face for the Pebble. Thanks to the people behind Minimalin for making everything available. The latest version of Obsidian can be downloaded here:
obsidian-3.6.pbw.

Furthermore, Obsidian is also source, and you can check it out at github.com/stefanheule/obsidian.

How Many x86-64 Instructions Are There Anyway?

Mon, 07 Mar 2016 00:00:00 -0800

Some features like footnotes work best when reading this post at stefanheule.com.

x86 is an enormously popular instruction set that is used on most desktop computers and servers (but usually not on mobile devices like phones). Given this wide use, it might seem like an easy question to ask how many x86 instructions there are, but it turns out this is more intricate than it looks.

Background

x86 is the name of a whole family of so-called instruction set architectures (ISA for short). The ISA defines the available instructions of a CPU, but also the registers, the memory architecture, how interrupts are handled, and much more. It is different from the microarchitecture, though, which determines how the ISA is implemented. Different microarchitectures can implement the same ISA, which is what happens with different AMD and Intel microarchitectures implementing essentially the same ISA. Since the introduction of x86 by Intel in 1978, the architecture has evolved extensively, but with almost full backward compatibility. Partly for this reason, the instruction set is full of weird corner-cases and historical artifacts. Here, I will look at the 64-bit variant (sometimes called x86-64) of the ISA and attempt to count the number of instructions. Specifically, the ISA from the Haswell microarchitecture, one of the latest ISA available at the time of writing. Due to the backward compatibility, this instruction set still includes most of the instructions from the very first generation of x86, and anything in between.

Attempts 1 and 2: Mnemonics

Machine instructions (the ones and zeros that tell your computer what to do next) have a portion called the opcode that determines what operation is performed. A mnemonic is a symbolic name for these opcodes. For instance, to add one to the eax register, one could use the instruction addl $1, %eax. Here, addl is the mnemonic, and so we can count the number of unique mnemonics to determine the number of instructions. I've written a small program to do this:

$ ./instr-count --mode "mnemonic_att"
1279

Great, so we are done and there are 1,279 instructions, right? Actually, the story is more complicated. Those familiar with assembly will have realized that the l in addl is a suffix that indicates the bit width of the operands. For instance, to add one to a 16-bit register (instead of the 32-bit register eax), one would use addw $1, %ax. Both of these instructions are doing addition, maybe we should not count them separately. In fact, had we used Intel syntax for the instructions, then the same operation would be written as add eax, 1 (note the absence of a width suffix, and the inverted ordering of operands). Let's try that:

$ ./instr-count --mode "mnemonic_intel"
981

Attempt 3: Mnemonics and Operand Types

If we look a little closer at mnemonics, we find that the situation is actually not so simple. Let's again look at the two instructions to add a constant to a 16 and 32-bit register, addw $0x1, %ax and addl $0x1, %eax, respectively¹. It turns out, that on 64-bit machines we consider here, they actually have relatively different semantics. The former just adds one to ax. On the other hand, addl $0x1, %eax also does the addition, but additionally zeros out the upper 32 bits of rax. Here is an example illustrating this behavior:

Note that the 32-bit addition on the right zeroes out the upper 32 bits of rax (marked in red), but the 16-bit addition (as well as the 8-bit addition not shown here) don't do that. That doesn't really seem like the same instruction any longer, even if both have the same (Intel) mnemonic. For another example why mnemonics can be misleading consider the instruction to perform a 32-bit signed multiplication: imull. There is a variant that multiplies two registers, and there is a variant that multiplies two registers and a constant². Again, even though both of these share the same mnemonic, they hardly seem like the same instruction (one multiplies two numbers, the other three numbers).

Therefore, maybe we should really count all the different combinations of mnemonic as well as the argument types. For instance, addl $1, %eax would be add_imm32_r32. This ensures that we correctly identify the examples shown as different instructions, and leaves us with 3,683 instructions:

$ ./instr-count --mode "operand_type"
3683

Attempt 4: Mnemonics and Operand Widths

Now we have a lot of instructions, and some of them seem like they are not really different instructions. For instance, the instruction to add a constant to a 32-bit register and the instruction to add a constant to a 32-bit memory location are counted separately, when the operation they perform could be argued to be identical. Thus, maybe a better way would be to consider the mnemonic in addition to the operand width. Our running example would be add_32_32, regardless of whether the operand is a register, memory location or constant. This way, we end up with 2,034 instructions:

$ ./instr-count --mode "operand_width"
2034

Attempt 5: Counting everything (added 2018-08-09)

All the attempts so far assume that we actually know what instructions a given CPU has, and that there is a textual representation for these instructions. However, these assumptions might be less well-founded than it seems, as people have discovered hidden and undocumented instructions in modern CPUs. The most thorough examination of the space of all instructions has been done by Christopher Domas with sandsifter, where he systematically explores the space of bit-level encoded instructions. Through clever tricks he is able to tell which bit sequences make up valid instructions, and which don't. This has the advantage of making very few assumptions and in principle allows us to count all bit sequences that denote valid instructions. Note that this will count addl $1, %eax and addl $2, %eax as separate instructions, because assembled as bits they are 0x83, 0xC0, 0x01 and 0x83, 0xC0, 0x02, respectively. And there is another problem: there are actually too many instructions to count them all this way. On x86-64, instructions can be up to 15 bytes long, making far to many to efficiently count them.

But even if we don't get an actual number here, it is another interesting way to (at least in principle if not in practice) count instructions. Christophers whitepaper is worth a read for much more details on his approach and findings.

It's All Wrong

Unfortunately, there are even more weird corner cases and historical artifacts in x86 that make counting instructions hard. For instance, we used mnemonics to group the same "mathematical operation" regardless of the operands, but it turns out that doing a bitwise xor operation is done using xor (in Intel syntax), but also pxor, vpxor, vxorpd, vxorps, xorpd, and xorps depending on the operand size and other things (and even more in AT&T syntax)³. So if attempt 2 tries to group the same operation regardless of the operation, then that count is actually too high.

And on the other end of the spectrum, we tried to count every different operation separately in attempt 3. However, it turns out there are instructions that really are a family of instructions. For instance, consider cmpsd that compares two floating point numbers. A third operand, an 8-bit constant, determines if the comparison is for equality, inequality, if the first number is larger, and 5 more variants. These clearly seem like different instructions, yet we have counted them as a single one.

Unfortunately, I can't think of an easy automatic way of counting such nuances, so I will leave it at the numbers I have.

Even more to the extreme we could go with attempt 5, where I suggested counting every valid bit sequeence and point to some work that has attempted to move in that directly. Unfortunately there are too many to actually get a count, but it is an interesting way to think about the problem nonetheless.

Conclusions

As I have hopefully convinced you by now, this innocent looking question is more difficult to answer than it might seem. It depends on your notion of what an instruction is, and is complicated by the many intricacies of the complex x86-64 instruction set. Here is an overview of possible answers:

Measure	Count	Comment
AT&T mnemonic (e.g., `addl`)	1,279	Counts the number of unique mnemonics in AT&T syntax.
Intel mnemonic (e.g., `add`)	981	This is a rough estimate of the number of different kinds of operations the x86 instruction set can perform, ignoring the operand type and size. There are various caveats as described earlier, such as some operations having multiple different mnemonics.
Mnemonic and operand types (e.g., `add_r32_imm32`)	3,683	Distinguishes instructions in every way possible and is thus a sort of upper bound on the number of instructions. If you are looking for a very fine-grained notion of instruction, this is a good measure.
Mnemonic and operand width (e.g., `add_32_32`)	2,034	This is an estimate of the number of different kinds of instructions, if an operation on 8 bits is considered to be different from the same operation on 16 bits (because sometimes, these actually have different semantics, as shown with the zeroing of the upper 32 bits for the `addl` instruction). Does not distinguish instructions that operate on registers vs. constants vs. memory locations.
Every valid bit sequence (e.g., `0x83, 0xC0, 0x01`)	?	Counts every bit seqeuence that makes a valid x86 instruction. Unfortunately there are too many to count.

Personally, the 4th attempt seems to capture my intuitive understanding of what an instruction is the closest. So, if I ever have to cross the bridge of death and get asked how many x86-64 instructions there are, I shall answer 2,034.

All the numbers in this blog post (except for attempt 5) have been obtained through a small program making use of our awesome C++11 library for working with x86-64 assembly. Just like the library, my program is available as open source on GitHub.

From here on I will use AT&T syntax because that's what I'm used to. It will not have any effect on the numbers. ↩
Actually, the situation is even more complicated for imull (no surprise at this point). There are seven different variants: The first one, imull_r32, just takes a single register and uses eax as an implicit input, and stores 64 bits of output. Next, imull_r32_r32 and imull_r32_r32 multiply two 32-bit numbers and store 32 bits of output, and finally imull_r32_m32_imm8, imull_r32_r32_imm8, imull_r32_r32_imm32, and imull_r32_m32_imm32 multiply two registers (or memory locations) as well as a constant (8 or 32 bits) and result in 32 output bits. ↩
Some of these different mnemonics to perform a bitwise exclusive or operation are due to the operand size. It turns out that for newer instructions that work on the xmm and ymm vector registers, different mnemonics depending on the operand size, both in AT&T as well as in Intel syntax. For the bitwise operations, there are also different mnemonics depending on the type of data (integer, single precision floating point or double precision floating point), even though the operation is on the bit level and, therefore, the same for all types (yes, different mnemonics perform semantically exactly the same operation). This is because internally, processors may have several pipelines for different data types for efficiency reasons, and thus using a bitwise mnemonics of the wrong type will compute the result correctly, but may be slower because the data has to be moved from one pipeline to another. ↩

Obsidian: A Usable and Elegant Analog Watch Face for the Pebble Time

Wed, 09 Sep 2015 00:00:00 -0700

Some features like footnotes work best when reading this post at stefanheule.com.

Update (2018-08-09): I have been informed that the Pebble app store is no longer accessible. Thus, I'm making the watchface available directly from this site, see below for instructions on how to install.

Update (2016-06-18): Obsidian now also supports weather information, see the corresponding blog post about it. All screenshots here have been updated to include weather information.

Update (2015-11-30): Obsidian now supports the original Pebble and the new Pebble Time Round. See the end of this post for more.

I have written a watch face application for the Pebble Time that I call Obsidian and I'd like to introduce it here. The main goal behind Obsidian is to put utility and practicality above all else, and only attempt to improve aesthetics if it does not interfere with that goal. Some of the main features are simplicity, high contrast colors and a date display that is never obstructed by the watch hands.

In fact, the reason I wrote the watch face in the first place was my frustration when I used other analog watch faces and wasn't able to tell today's date, just because it was 3:14 pm (this caused the watch hands to cover the date display as it would on traditional watches). Instead, I've chosen to move the date a little if necessary in Obsidian¹, so that it is visible no matter the position of the watch hands.

Another important aspect for a smartwatch is to warn the user if the connection to the phone is lost (or regained). Obsidian does this through two short vibrations, followed by a message on screen that is shown for five seconds. After that, the message disappears and only a small icon remains to keep the watch fully usable even without a smartphone connected².

The Pebble Time has a surprisingly long battery life compared to other smartwatches, but at some point it will need to be re-charged, too. To not miss that point, Obsidian has a small battery icon in the top right corner, and turns the background yellow, orange and finally red as the battery level reaches 30%, 20%, and 10%, respectively.

Configurability

I like the default settings, but I acknowledge that not everybody has the same preferences as I have, which is why I made most aspects of the watch face configurable. It's possible to change any color, hide the battery or Bluetooth icons and not vibrate upon losing the phone connection. Here are a few examples of the possibilities:

Downloading Obsidian

Pebble's app store is no longer functional after the company shut down. Instead, download the installation file here: obsidian-3.6.pbw, then install the file on your smartphone.

Furthermore, it is open source and available on Github, where you can also see more screenshots as well as an overview of all available settings. Feel free to give it a try, and let me know what you think!

Updating Obsidian for Aplite and Chalk

(added on 2015-11-30) Pebble offers smartwatches of three different hardware platform. They started with the original Pebble (aplite platform), which was slightly larger and black-and-white only. They then added the Pebble Time (basalt platform), for which Obsidian was originally written. Recently the Pebble Time Round (chalk platform) got added to the collection, which is a round smartwatch that is thinner and lighter. Since writing this blog post, I've added support for all three platforms to Obsidian. Because Obsidian is an analog watch face, it looks great on Chalk:

On Aplite, Obsidian looks a bit boring, because only two colors are available. Additionally, the drawing APIs are more limited and don't support anti-aliasing, making edges look a bit rough.

Obsidian remains fully configurable on all platforms.

The name Obsidian is a type of rock and thus a play on Pebble. ↩
Of course, the Bluetooth icon also cannot be covered by the watch hands, in case you wondered. ↩

The Surprising Reasons I Like the Pebble Time and Smartwatches in General

Mon, 31 Aug 2015 00:00:00 -0700

Some features like footnotes work best when reading this post at stefanheule.com.

I was skeptical about smartwatches and wasn't sure whether they are useful enough to offset the annoyance of having yet another expensive device that requires charging. Especially the thought of owning a smartwatch that doesn't even do the watch part well sounded terrible; I don't want to have to press a button first to turn on the screen, or turn my wrist in just the right way so that the sensors pick it up and turn on the screen. Luckily, I found the Pebble Time, a smartwatch that features an always-on e-ink display which solves at least the watch problem. Furthermore, I actually like my Pebble Time more than I thought I would, and I found many of the reasons unexpected. In this post, I'd like to highlight some of these. Some are obvious in hindsight, but maybe there are other people out there who equally don't have a good sense of the usefulness of smartwatches. In essence, this is the article that I would have wanted to read when I was deciding if I should get a smartwatch.

However, this is not a Pebble Time review; and, in fact, many of the reasons (but not all) are not specific to the Pebble Time, but apply to smartwatches in general. There are enough articles that describe its great features like the always-on display and the very long battery life, and that talk about the not-so-great aspects like the low resolution of the display or the lack of a GPS or heart-rate sensors.

Silent and Noticeable Notifications

The Pebble Time allows you to show (some or all) notifications from your smartphone on your watch and allows some interactions with them. For instance, it can show arriving emails and you can archive them, or dismiss the notification. The built-in microphone even allows you to respond, though to me that seems more of a gimmick¹. Having some of the notifications from your phone on your wrist is great, but I find another aspect even better. But let's first look at why notifications on phones are annoying:

Smartphones are surprisingly bad at notifying the user about incoming emails, texts, calls or other things; they either are too loud, or not noticeable enough. We can choose to use a ringtone and risk having the phone go off at inconvenient times like during a meeting or a presentation. Alternatively we can turn the phone silent, and miss possibly important notifications. Neither is great, and even the vibrate-only option doesn't work well; it's still common to miss notifications, even if the phone is carried on person, and yet the vibration can be clearly audible depending on the background noise level and the exact position of the phone (if it touches, say, a chair through my pocket in just the right way, the vibration is seemingly louder than even a ringtone?!).

With smartwatches like the Pebble Time, notifications are finally silent and noticeable. Because the watch touches the skin directly, the vibration can be soft yet clearly noticeable to the user (and only the user, as opposed to other people in close proximity). No more missed notifications and no more embarrassing moments when notifications are heard by others at inappropriate times.

Notifications Everywhere

Another source of missed notifications used to be when I'm close to my phone, but not actually carry it on me. Maybe I'm in my office and the phone sits on the desk, or I'm at home cooking dinner, and the phone is in my room charging. With the Pebble Time, I have the freedom to move around without my phone and not miss notifications.

Never Forget Your Phone

Like most smartwatches, the Pebble Time is connected to your phone via Bluetooth. If that connection is lost, then the watch can notify you by vibrating and showing a short notification on the display. This turns your watch into a great ally to prevent forgetting your phone: If you ever accidentally leave your phone behind, the watch will remind you as soon as you step outside the Bluetooth range, which is typically around 10 meters.

Fitness and Sleep Tracking on your Terms

Fitness trackers like the devices from FitBit are getting a lot of attention lately, and thanks to the accelerometer the Pebble Time can track your steps, too. It's nice to be able to collect information like this and also fully be in control of where that data is stored and with whom it is shared. Unlike FitBit, which stores all the information on their servers, you can choose a fitness tracking app and therefore decide what happens with your data. The same applies for tracking your sleep; you pick the app, and you own the data.

Silent Alarm

The vibration function of the Pebble Time can also be used as a silent alarm clock. This is convenient if you need to get up before somebody sleeping in the same room, but I'm now using it even if I'm by myself. I like waking up to a gentle vibration on the wrist rather than a blaring alarm clock.

Water Resistance

Most smartwatches are water resistant². For instance, the Apple Watch or the Moto 360 are rated as water resistant according to the IPX7 standard, which should allow these devices to stay submerged for 30 minutes at up to 1m in fresh water. However, this essentially only protects the device from light splashes and the like. In fact Apple specifically tells users to avoid submerging the watch for long times, swimming and showering. In particular showering is a bad idea, because the water hits the watch at a velocity that IPX7 is not meant to protect from. The Pebble Time on the other hand is rated as water resistant at up to 30m (according to ISO 22810), and can be worn while showering or swimming. This turns out to be super convenient, as I can keep the watch on for essentially everything I do. I wouldn't take it diving, but that's not something I do daily.

Automatic Timezone

One thing that smartphones are good at is getting the correct local time from the network automatically, no matter what timezone you are in. Since smartwatches connect to the phone, your watch will also always show the correct local time, just as it should. This is in contrast to most regular wrist watches.

Programmable in C

Pebble watches are programmable in C³, and this may not excite most people, but I found it quite fun to play with. There is something exciting about programming an embedded device with restricted resources in a low-level programming language like C, and even non-experts can write a useful app. I wrote an analog watch face that I now use daily. You can find out more about it here.

Wrapping Up

For me, there is no single killer application (though the silent yet noticeable notifications come close), but rather it's the combination of all the small nice features that makes the Pebble Time smartwatch great.

Answering via voice works but feels slow and cumbersome. Also, due to the device size, the only way to fix an error in the voice recognition is to repeat the whole message. ↩
Note that no watch is water-proof. If you submerge any watch deep enough, it will stop working. The proper term is water resistance, in addition to a measure of how resistant it is (e.g., how deep it can be submerged, or for how long). ↩
It is actually also possible to write simple apps in JavaScript, but they are fairly limited. ↩

Installing Missing Packages the Easy Way on Linux

Thu, 30 Jul 2015 00:00:00 -0700

Some features like footnotes work best when reading this post at stefanheule.com.

This is a quick tip to making the installation of missing packages in certain Linux flavors easier.

Many Linux distributions come with a program called command-not-found pre-installed (e.g., Ubuntu or Debian), that helps to discover missing packages. Say you just freshly installed Linux and type hg status in a shell. Because hg is not installed yet, you instead see the following:

$ hg status
The program 'hg' is currently not installed. You can install it by typing:
sudo apt-get install mercurial

Now this very useful, especially when the package has a different name from the command you are trying to run (like with mercurial). But why do I have to type (or copy) that message if I want to install the package? Turns out, there is an environment variable which when set, asks to install the package directly, like this:

$ hg status
The program 'hg' is currently not installed. You can install it by typing:
sudo apt-get install mercurial
Do you want to install it? (N/y)

Entering y will install the package. Very convenient. To enable this feature, add the following line to ~/.bashrc (or ~/.zshrc, or wherever your shell initialization is):

export COMMAND_NOT_FOUND_INSTALL_PROMPT=1

Life just got so much better!

The Most Dangerous Code in the Web Browser

Sun, 28 Jun 2015 00:00:00 -0700

Some features like footnotes work best when reading this post at stefanheule.com.

Did you know that the web browser extension you installed a long time ago (say, AdBlock), can probably see all your passwords, look at any website you visit using your credentials and could trivially send all that information to an arbitrary web server? That's pretty scary, and in this blog post I will explain how security for extensions currently works. I will also outline research towards a better extension security model for browsers that protects your sensitive information.

Background

Web applications are ubiquitous and many tasks that traditionally have been achieved using dedicated desktop applications are carried out in web browsers today. Furthermore, these applications increasingly handle sensitive data such as banking information, passwords or medical data. This poses security challenges to protect this data from malicious entities, and luckily the web platform and, as part of this, web browsers have evolved to achieve this. For instance, the same-origin policy (or SOP for short) roughly ensures that information from one website (say, bank.com) cannot be accessed by a malicious site like evil.com, because they have different origins¹.

Mechanisms like the SOP have worked relatively well in protecting user data in web browsers². However, users often want to extend web browser functionality by installing extensions, e.g., to block advertisements or to extend the functionality of certain websites. Unfortunately, the security story for extensions is less rosy.

Extension Security

Let's look at Google Chrome, the most popular browser and the one with the most sophisticated security model for extensions: Unlike websites, extensions need access to more sensitive APIs to implement their functionality, such as access to the list of open tabs, the browsing history, or the cookies. At the same time, extensions also need to deeply interact with the websites code, so that they can change their behavior or looks. Since such APIs give access to rather sensitive data, Chrome uses two main mechanisms to try and ensure the information remains private. Firstly, extensions are split into a content script and a core script, and only the core gets access to these APIs (like your browsing history), but cannot directly interact with any web pages. In contrast, the content script can interact with web pages, and, for instance, access and change what is displayed (by accessing the so-called DOM). Message passing is then used between the content script and the extension core to try and prevent malicious websites from getting access to the sensitive browser APIs, even in the presence of badly written or buggy extensions. This is important because the sensitive APIs are not governed by, say, the SOP. Secondly, extensions need to declare a list of permissions, and extensions only get access to the APIs allowed by the declared permissions.

In the Chrome security model, every extension is split into a content script that has access to the website (via it's DOM), and the core, which has access to the sensitive browser APIs. The two parts communicate via message passing.

At this point, you might be able to guess what the problem is: The sensitive APIs are protected from malicious websites, but what about malicious extensions? The only protection of the sensitive APIs is through the permissions, which are declared by the extension. The user can either grant all requested permissions at install time, or not install the extension. We have looked at the top 500 extensions by number of users in the Chrome Web Store to get a sense of what permissions are being requested. We found that more than 71% of extensions require permission to "read and change all your data on the websites you visit". And it only gets worse if we look at the top 100 (with 82%) or top 50 of extensions, where a staggering 88% of extension require this permission. And these extensions are widely used, too, with the most popular having more than 10 million users (how many exactly we don't know, as Google caps the reported number at 10 million), and even the extension in place 500 still has more than 70,000 users. Clearly, users are willing to give these very broad permissions, and most extensions do require powerful permissions.

A full view of what percentage of the most popular extensions require permission to read and change all your data on the websites you visit. At the left, we consider the top 500 most popular extensions (by number of users), and we restrict it to the more popular extensions as we go to the right. For instance, all of the top 7 most popular extension show this message on installation.

This means that most of these popular extensions can interact with web pages, and for instance learn the password entered by the user, or see the bank statement. At the same time, they can send all that data to wherever they choose, clearly putting the user's privacy at risk.

This is a highly unsatisfactory situation, and while Google is trying to remove malicious extensions from their online store³, this inevitably is an arms race that cannot guarantee the safety of all users. Thus, in the remainder of this blog post, I will outline a possible solution to the problem of extensions needing access to sensitive information to implement their functionality, yet ensuring the user's privacy is not at risk.

Towards a Solution

The key to a solution is the insight that extensions which deal with sensitive information are perfectly safe as long as they do not disseminate this sensitive information arbitrarily, for instance by sending it to evil.com. Let's look at an example: The extension Google Mail Checker gives the user an icon with the number of unread emails in Gmail.

To implement this functionality, the extension requires permission to look at any information from google.com sites. Clearly, this is potentially dangerous to the user's privacy. In fact, there is nothing stopping the extension from leaking all emails, or stealing the user's Google password (it probably isn't, but how do you know?)⁴.

But as we observed earlier, the user's data would be safe if the extension can access the emails, but not leak them arbitrarily. This idea is pervasive in mandatory access control (MAC) based confinement systems, where it is not just limited who can access information, but also how that information can be further shared.

We can implement this idea by tracking where data originates from by using a label. The labels correspond to origins, and in our example the extension would be tainted with the label mail.google.com after it accessed the unread count. This label is then used to limit with whom the extension can communicate. In particular, in the example, the extension is now limited to communicate with mail.google.com, and couldn't leak the user's email to evil.com⁵. With this, even if it reads all your email or looks at your password, the extension has no way of sending the information anywhere. However, it can still perform its main feature of displaying the unread count as an icon to the user (which can always be done, regardless of the label). And most importantly, it is possible to do this without requiring a permission at all! Accessing sensitive information does not require permission if the information cannot be leaked.

Of course not all extensions are this simple, and so we need more sophisticated APIs to allow more complex extensions to be safe. A common problem in MAC-based confinement systems is that sometimes information that is sensitive (with regard to a particular label) is actually okay to be leaked to another origin. For instance, extensions like the Evernote Web Clipper allow its users to save parts of a web page (that might contain sensitive information) to the popular note-taking app at evernote.com. Formally, this problem is known as declassification; conceptually the label needs to be removed of a piece of data before it can be sent to the destination origin. Clearly extensions cannot be trusted to make these declassification decisions, as that would allow them to just arbitrary leak sensitive information again. However, for extensions we can leverage user intent: Already in current extensions the user clicks a context menu entry to share the information with the extension. Thus, if we provide a sharing API, then extensions can register such a context menu in a trusted UI, and every time the user shares information explicitly, this corresponds to a declassification (by the user) in our system. The user is in full control of what information gets leaked, again without having to use any permissions.

Another application enabled by our approach is a secure password storage in the cloud. This is not a new idea and services like LastPass allow users to only remember a single master password, and have all other passwords be stored in the cloud. This functionality is great, but allows anybody with access to LastPass servers to read all the user's password hashes. Unfortunately it's not enough to trust LastPass to not peak them: If somebody manages to hack into their systems, then that person might gain access to user's password hashes⁶. So, how can we help? If an extension reads the password on the accounts.google.com page when you log into your Google account, then that password is labeled with the Google origin and couldn't be sent to lastpass.com. We wouldn't want to declassify the password, as that reveals the password to LastPass and has all the aforementioned problems. Instead, we can take advantage of cryptography: In our system, we allow extensions to conceptually remove a label from a piece of data (like the password) by encrypting it through a browser API. With this, an extension can access the password, encrypt it and then hold the encrypted string without being tainted at all. This allows the extension to share the encrypted password with, say, lastpass.com. While the password is encrypted, the extension isn't tainted, but it also cannot see or use the password (e.g., to log the user in with previously saved credentials). There is a second API call that decrypts the information and allows the extension to see the password. However, the decryption now taints the extension with the corresponding label. In our example, the extension would be tainted with accounts.google.com and couldn't share the password arbitrarily any longer, but can perform it's legitimate function of filling in the login form. Essentially, this mechanism allows extensions to trade taint with encryption.

Conclusions

These are just a few of the ideas from a research paper I have written with Devon Rifkin and Deian Stefan. The paper contains more technical details and has been presented at the Workshop for Hot Topics in Operating Systems. While we have not yet implemented such a secure extension system and only outlined a solution, we do believe this to be promising direction towards a practical solution. We hope that this encourages browser vendors to rethink extension systems and raise awareness in users who may not be aware of the capabilities of even simple browser extensions.

For now, the best recommendation for users is to be careful what extensions you install and only do so through trustworthy sources.

Press

This blog post has been discussed on Hacker News. Furthermore, Eric Limer from Popular Mechanics has written an article with the title Reminder: Your Browser Extensions Have Absurd Access To Everything You Do Online about this work.

Technically, an origin is defined as an URI scheme, a hostname, and a port number. For simplicity, I only use the domain in my examples. ↩
Here we are only looking at client side security, i.e., protecting user data when it is in the web browser. Security on the server is a separate problem, and a lot of reports about privacy breaches are about servers getting hacked. ↩
For example, Google recently removed almost 200 extensions that affected a large number of users. The extensions range from injecting ads to outright stealing private information such as passwords. ↩
This particular extension is actually better than most extensions, and only requires permission to all google.com sites (but more than just mail.google.com). For this reason, it couldn't directly leak the information to, say, evil.com. However, it could send it via the user's Gmail account, and then delete the incriminating email. ↩
By default, only GET requests are possible, which only allow reading a website. Changing information on a website or sending an email requires a POST request, which prevents the extension from sending the users information via Gmail (the attack outlined earlier). ↩
In fact, LastPass did get hacked recently, though the saved passwords for other sites have not been compromised according to their analysis. ↩

smartless: a better pager for small and large inputs

Sun, 07 Jun 2015 00:00:00 -0700

Some features like footnotes work best when reading this post at stefanheule.com.

A pager is a computer program that makes viewing long output or long files on the terminal more pleasant. Instead of dumping all the output at once, a pager shows the text one page at a time, allowing the user to scroll up and down, or search through the text. For instance, viewing a large diff with git is much more pleasant with a pager. The most common pager on Unix systems is probably less, and while it works great in many cases, viewing only very short output in a pager is a bit silly. If all of the text is only a few lines long and fits on the screen, then no pager is required in the first place.

However, it is often hard to know upfront whether a pager will be necessary, and so it would be more convenient to always use a pager and have the pager return immediately if the text is short, and just print it instead.

The pager less has a command line option -F that does exactly this. Unfortunately, it comes at a price: scrolling is no longer possible. For this reason, I have built my own pager I call smartless, that avoids exactly this problem. For small output, smartless will just print the text directly and exit.

For larger output (more than 15 lines by default, but this is configurable), the pager less is invoked. Note that the first 15 lines are still printed directly, before less is called. This is a feature and not a bug, as it gives a little bit of information about the output, even after exiting less. Here's a little demo:

Quick demo of smartless in action. (Unfortunately the recording software I used introduced some artifacts and non-smooth transitions.)

This works great for me, and I have completely replaced less with smartless on my system. For instance, to change git to use the new pager, run

$ git config --global core.pager 'smartless'

Similarly, I have configured an alias so that I can keep typing less, but actually use smartless instead:

alias less='/path/to/smartless'

Download

smartless is a small Bash script, and available on GitHub. Give it a try if it sounds useful, and let me know what you think!

git clone https://github.com/stefanheule/smartless.git

Finally, credit where credit is due: When I originally searched for a solution I found this StackExchange post that my script is based on.