Storage for the web

Internet connections can be flakey or non-existent on the go, which is why
offline support and reliable performance are common features in
progressive web apps. Even in perfect wireless
environments, judicious use of caching and other storage technique…

Internet connections can be flakey or non-existent on the go, which is why
offline support and reliable performance are common features in
progressive web apps. Even in perfect wireless
environments, judicious use of caching and other storage techniques can
substantially improve the user experience. There are several ways to cache
your static application resources (HTML, JavaScript, CSS, images, etc.), and
data (user data, news articles, etc.). But which is the best solution? How
much can you store? How do you prevent it from being evicted?

What should I use?

Here’s a general recommendation for storing resources:

IndexedDB and the Cache Storage API are supported in every modern browser.
They’re both asynchronous, and will not block the main thread. They’re
accessible from the window object, web workers, and service workers, making
it easy to use them anywhere in your code.

What about other storage mechanisms?

There are several other storage mechanisms available in the browser, but they
have limited use and may cause significant performance issues.

SessionStorage is tab specific, and scoped to the
lifetime of the tab. It may be useful for storing small amounts of session
specific information, for example an IndexedDB key. It should be used with
caution because it is synchronous and will block the main thread. It is
limited to about 5MB and can contain only strings. Because it is tab specific,
it is not accessible from web workers or service workers.

LocalStorage should be avoided because it is synchronous
and will block the main thread. It is limited to about 5MB and can contain
only strings. LocalStorage is not accessible from web workers or service
workers.

Cookies have their uses, but should not be used for storage.
Cookies are sent with every HTTP request, so storing anything more than a
small amount of data will significantly increase the size of every web request.
They are synchronous, and are not accessible from web workers. Like
LocalStorage and SessionStorage, cookies are limited to only strings.

The File System API and FileWriter API provide methods for
reading and writing files to a sandboxed file system. While it is asynchronous,
it is not recommended because it is
only available in Chromium-based browsers.

The File System Access API was designed to make it
easy for users to read and edit files on their local file system. The user
must grant permission before a page can read or write to any local file, and
permissions are not persisted across sessions.

WebSQL should not be used, and existing usage should be migrated to
IndexedDB. Support has been removed from almost all major
browsers. The W3C stopped maintaining the Web SQL spec in 2010,
with no plans to further updates planned.

Application Cache should not be used, and existing usage should be
migrated to service workers and the Cache API. It has been
deprecated and support will be removed from browsers in
the future.

How much can I store?

In short, a lot, at least a couple of hundred megabytes, and potentially
hundreds of gigabytes or more. Browser implementations vary, but the amount
of storage available is usually based on the amount of storage available on the
device.

  • Chrome allows the browser to use up to 80% of total disk space. An origin can
    use up to 60% of the total disk space. You can use the StorageManager
    API
    to determine the maximum quota available. Other Chromium-based
    browsers may allow the browser to use more storage. See
    PR #3896 for details about
    Chrome’s implementation.
  • Internet Explorer 10 and later can store up to 250MB and will prompt the
    user when more than 10MB has been used.
  • Firefox allows the browser to use up to 50% of free disk space. An
    eTLD+1
    group (e.g., example.com, www.example.com and foo.bar.example.com)
    may use up to 2GB. You can use the
    StorageManager API to determine how much space is still
    available.
  • Safari (both desktop and mobile) appears to allow up to 1GB. When the limit
    is reached, Safari will prompt the user, increasing the limit in 200MB
    increments. I was unable to find any official documentation on this.

In the past, if a site exceeded a certain threshold of data stored, the
browser would prompt the user to grant permission to use more data. For
example, if the origin used more than 50MB, the browser would prompt the user
to allow it to store up to 100MB, then ask again at 50MB increments.

Today, most modern browsers will not prompt the user, and will allow a site
to use up to its allotted quota. The exception appears to be Safari, which
prompts at 750MB, asking permission to store up to 1.1GB. If an origin
attempts to use more than its allotted quota, further attempts to write data
will fail.

How can I check how much storage is available?

In many browsers, you can use the
StorageManager API to determine the amount of storage
available to the origin, and how much storage it’s using. It reports the total
number of bytes used by IndexedDB and the Cache API, and makes it possible
to calculate the approximate remaining storage space available.

if (navigator.storage && navigator.storage.estimate) {
const quota = await navigator.storage.estimate();
// quota.usage -> Number of bytes used.
// quota.quota -> Maximum number of bytes available.
const percentageUsed = (quota.usage / quota.quota) * 100;
console.log(`You've used ${percentageUsed}% of the available storage.`);
const remaining = quota.quota - quota.usage;
console.log(`You can write up to ${remaining} more bytes.`);
}

The StorageManager isn’t implemented in all browsers yet, so you
must feature detect it before using it. Even when it is available, you must
still catch over-quota errors (see below). In some cases, it’s possible for
the available quota to exceed the actual amount of storage available.

Other Chromium-based browsers may factor in the amount of free space when
reporting the available quota. Chrome does not, and will always report 60% of
the actual disk size. This helps to reduce the ability to determine the size
of stored cross origin resources.

Inspect

During development, you can use your browser’s DevTools to inspect the
different storage types, and easily clear all stored data.

Storage test tool.

While working on this article, I wrote a simple tool to
attempt to quickly use as much storage as possible. It’s a quick and easy way
to experiment with different storage mechanisms, and see what happens when
you use all of your quota.

How to handle going over quota?

What should you do when you go over quota? Most importantly, you should
always catch and handle write errors, whether it’s a QuotaExceededError or
something else. Then, depending on your app design, decide how to handle it.
For example delete content that hasn’t been accessed in a long time, remove
data based on size, or provide a way for users to choose what they want to delete.

Both IndexedDB and the Cache API both throw a DOMError named
QuotaExceededError when you’ve exceeded the quota available.

IndexedDB

If the origin has exceeded its quota, attempts to write to IndexedDB will
fail. The transaction’s onabort() handler will be called, passing an event.
The event will include a DOMException in the error property. Checking the
error name will return QuotaExceededError.

const transaction = idb.transaction(['entries'], 'readwrite');
transaction.onabort = function(event) {
const error = event.target.error; // DOMException
if (error.name == 'QuotaExceededError') {
// Fallback code goes here
}
};

Cache API

If the origin has exceeded its quota, attempts to write to the Cache API
will reject with a QuotaExceededError DOMException.

try {
const cache = await caches.open('my-cache');
await cache.add(new Request('/sample1.jpg'));
} catch (err) {
if (error.name === 'QuotaExceededError') {
// Fallback code goes here
}
}

How does eviction work?

Web storage is categorized into two buckets, "Best Effort" and "Persistent".
Best effort means the storage can be cleared by the browser without
interrupting the user, but is less durable for long-term or critical data.
Persistent storage is not automatically cleared when storage is low. The user
needs to manually clear this storage (via browser settings).

By default, a site’s data (including IndexedDB, Cache API, etc) falls into
the best effort category, which means unless a site has
requested persistent storage, the browser may evict
site data at its discretion, for example, when device storage is low.

The eviction policy for best effort is:

  • Chromium-based browsers will begin to evict data when the browser runs out
    of space, clearing all site data from the least recently used origin first,
    then the next, until the browser is no longer over the limit.
  • Internet Explorer 10+ will not evict data, but will prevent the origin from
    writing any more.
  • Firefox will begin to evict data when the available disk space is filled up,
    clearing all site data from the least recently used origin first, then the
    next, until the browser is no longer over the limit.
  • Safari previously did not evict data, but recently implemented a new
    seven-day cap on all writable storage (see below).

Starting in iOS and iPadOS 13.4 and Safari 13.1 on macOS, there is a
seven-day cap on all script writable storage, including IndexedDB, service
worker registration, and the Cache API. This means Safari will evict all
content from the cache after seven days of Safari use if the user does not
interact with the site. This eviction policy does not apply to installed
PWAs
that have been added to the home screen. See
Full Third-Party Cookie Blocking and More on the WebKit
blog for complete details.

You can request persistent storage for your site to
protect critical user or application data.

Bonus: Why use a wrapper for IndexedDB

IndexedDB is a low level API that requires significant setup before use,
which can be particularly painful for storing simple data. Unlike most modern
promise-based APIs, it is event based. Promise wrappers like
idb for IndexedDB hide some of the powerful features but more
importantly, hide the complex machinery (e.g. transactions, schema versioning)
that comes with the IndexedDB library.

Conclusion

Gone are the days of limited storage and prompting the user to store more and
more data. Sites can store effectively all of the resources and data they
need to run. Using the StorageManager API you can
determine how much is available to you, and how much you’ve used. And with
persistent storage, unless the user removes it, you
can protect it from eviction.

Additional resources

Thanks

Special thanks to Jarryd Goodman, Phil Walton, Eiji Kitamura, Daniel Murphy,
Darwin Huang, Josh Bell, Marijn Kruisselbrink, and Victor Costan for reviewing
this article. Thanks to Eiji Kitamura, Addy Osmani, and Marc Cohen who wrote
the original articles that this is based on. Eiji wrote a helpful tool
called Browser Storage Abuser that was useful in validating
current behavior. It allows you to store as much data as possible and see the
storage limits on your browser. Thanks to Francois Beaufort who did the digging
into Safari to figure out its storage limits.

The hero image is by Guillaume Bolduc on
Unsplash.


Print Share Comment Cite Upload Translate
APA
Pete LePage | Sciencx (2024-03-28T14:48:37+00:00) » Storage for the web. Retrieved from https://www.scien.cx/2020/04/27/storage-for-the-web/.
MLA
" » Storage for the web." Pete LePage | Sciencx - Monday April 27, 2020, https://www.scien.cx/2020/04/27/storage-for-the-web/
HARVARD
Pete LePage | Sciencx Monday April 27, 2020 » Storage for the web., viewed 2024-03-28T14:48:37+00:00,<https://www.scien.cx/2020/04/27/storage-for-the-web/>
VANCOUVER
Pete LePage | Sciencx - » Storage for the web. [Internet]. [Accessed 2024-03-28T14:48:37+00:00]. Available from: https://www.scien.cx/2020/04/27/storage-for-the-web/
CHICAGO
" » Storage for the web." Pete LePage | Sciencx - Accessed 2024-03-28T14:48:37+00:00. https://www.scien.cx/2020/04/27/storage-for-the-web/
IEEE
" » Storage for the web." Pete LePage | Sciencx [Online]. Available: https://www.scien.cx/2020/04/27/storage-for-the-web/. [Accessed: 2024-03-28T14:48:37+00:00]
rf:citation
» Storage for the web | Pete LePage | Sciencx | https://www.scien.cx/2020/04/27/storage-for-the-web/ | 2024-03-28T14:48:37+00:00
https://github.com/addpipe/simple-recorderjs-demo