Cheat Sheet: Mapping out a Powerful File Handling System

Are you launching an app with user-generated or rich product content? At some point, you’ll need a steady infrastructure for receiving, processing, and delivering files. To save you time on feature planning and research, we’ve drafted a high-level tech…

Are you launching an app with user-generated or rich product content? At some point, you’ll need a steady infrastructure for receiving, processing, and delivering files. To save you time on feature planning and research, we’ve drafted a high-level tech spec with all the traps and pitfalls.

A tech spec? Seriously? Yep! As you will soon understand, implementing a “Choose a file” button is just the tip of the iceberg: the whole project will need an architecture overview. Use our cheat sheet to create your own tech/product spec or simply glance through it before story mapping.

? This cheat sheet will bring the most value to media-heavy platforms like website builders, CMSs, marketplaces, and marketing automation platforms.



Intro: Giving the Whole Thing a Real-World Vibe

Let’s imagine you’re building a next-gen CMS. You will need a fully fledged file management system to enable your customers to upload content to their websites and landing pages.

Let’s call this functionality UploadUFO and map it out using a popular tech spec template from Lyft Engineering, as their approach seems simple and well thought out. There are eight key sections, and we’ll cover all of them right now ?

Alt Text
Meet UploadUFO, a steady infrastructure for uploading any files without having a cow.



1. Creating a File Handling System: Overview

Here you should describe all the “what”s and “why”s of the project. What kind of uploading functionality is UploadUFO? What user jobs will this feature be implemented for? Why do you need to build it now? Wrap your answers into a concise summary.

UploadUFO is a reliable file handling system that enables publishers and designers to fill their websites with rich content. Apart from receiving and storing, UploadUFO will deliver the content at the drop of a hat and tailor it to any screen and device. Efficient visual content optimization and fast load times will differentiate us from other CMS providers and bring our customers the maximum value.

Got it? Here we’ve described our vision, making the feature a part of the product and business reality.



2. Goals and Impact Measurement

List all the desired outcomes so that you can evaluate the results at the end of the day. The more specific your goals are, the better. That’s why it’s crucial to introduce relevant metrics and their approximate values.

✔️ UploadUFO will make file uploads and delivery 5X faster than our competitors (XYZ), ensuring a great user experience and page load times.

✔️ Our customers will be happy with the functionality. We’ll track their happiness via NPS (should be above 50) and the average number of support tickets (should be below 10 per month).

✔️ UploadUFO will work like a charm for 10,100, or 100,000 users, scaling as we grow.

✔️ The functionality should be easy to maintain (less than X hours per month).



3. Identifying Non-Goals

Identify what you are intentionally not building to formalize the scope and prevent feature creep.

❌ We are not building our own storage or a CDN; we’ll use third-party vendors instead.

❌ We are not investing in any AI elements, as file handling is not at the core of our product.

❌ We are not letting our users upload cows, as they are too big and can’t be split into chunks.’



4. Feature Planning

Here you will need to identify your engineering approach together with key infrastructure elements. This part will be the longest one and should be based on in-depth research.

First, define the big picture: how will the whole file handling system function? What are the crucial elements, and how are they connected?

The UploadUFO file management system will be composed of 3 main elements: an uploading UI or File Uploader, cloud storage, and a CDN. When a user uploads a file, it goes to the storage and gets delivered upon request, tailored to the individual’s context.

Alt Text
Use schemes and simple diagrams to show the big picture and main infrastructure elements.

Then you’ll need to describe each element. We gathered all the possible requirements and split them into four main parts: uploads, storage, processing, and delivery. To decide what’s relevant for your project, go through the points below or download the checklist in PDF format.



Uploads

Decide whether you’ll need to support all sorts of files. Consider multipart and resumable uploads, especially if you plan to receive large files like videos or high-res graphics. Think about possible security issues, as the uploading process is the weakest part in terms of potential hazards.

File types: UX: Performance and security:
Audio files
Video files
Image files
Documents
GIFs
ZIPs
Any others
Crazy large files
3rd party sources like Dropbox or Instagram
Dev experience
User experience
Image compression
Cross-browser UX
Cross-platform UX
Lazy uploads/proxy
I18n and L10n

Encryption
Upload acceleration
Resumable uploads
Error handling
Multiupload
Multipart uploads
Malware protection
Network issue handling
Timeout handling
Monitoring
Large files
Custom validation
File shrinking upon upload
Upload signing

All the possible parts and features of a file uploading process.



Storage

Your storage will need to be secure and able to scale as you grow. If you’re planning to handle personal or health data, you’ll have to make sure it’s HIPAA and GDPR compliant.

Performance: Security:
Scalability
Compliance
Encryption
Performance
Dev experience
Backup
Management
Search
Durability
Monitoring
Metadata handling
Availability
Automation
HIPAA etc. compliance
Signed URLs

All you need to take into account when planning file storage.



Processing

For this part of the scope, the sky’s the limit: you could even include AI-powered features like object and color recognition to automate content management. But be careful to avoid feature creep.

Performance and security: Content insights: Content optimization: Content processing:
Virus checking
Malware checking
Explicit content checking
Async processing
DDOS
Color recognition
Face and object recognition
Identity check
OCR
Object tagging
Trend analytics
Spike prediction
Categorization
Duplication checks
Similarity checks
Advanced external APIs
Smart cropping
Image compression
Video encoding
GIF to video
Monitoring
AI-based compression
Conversion to next-gen formats
Image enhancement
Video enhancement
Doc conversion
Upscaling
Cropping
Watermarks/Overlays
Thumbnails
RGB←→CMYK
Filters
Text extraction

Here is what you’ll need for setting up file processing.



Delivery

To deliver the content fast and tailor it to the individual’s context, you’ll have to think about serving it from different locations and building automated responsiveness: on-the-fly destination properties detection, resizing, and cropping.

Performance: Security:
Efficiency
Global presence
Monitoring
Resilience
Cost effectiveness
Precision
Auto format switch (e.g. WebP, GIF to Video)
Auto size detection
Network detection
Access control
Encryption
Security
Attack protection
Audit
HIPAA etc. compliance

All the nuts and bolts of the content delivery process.



5. Security, privacy, risks

File uploading is one of the riskiest parts of the process in terms of potential hazards and malware, so ensure your infrastructure is bulletproof.

UploadUFO will have a signed uploads option to help our customers control who is uploading content to their website. We’ll make sure we have a serverless backup for all the user content. On top of that, we’ll consider GDPR and HIPAA compliance requirements, as we have different types of customers on board and use cases related to confidential information.

Another potential risk can be related to scaling: a basic in-house infrastructure will endure the MVP stage, but what if you grow to thousands of customers? To foresee all the risks, it’s recommended to ask your stakeholders for a tech spec review.



6. Other Considerations

In this section, you should describe alternative ways of solving the problem and why you chose the approach you described in the Planning section.

For example, here you can list your ideas around the classic build or buy dilemma: should the infrastructure be built from scratch, be composed of different outsourced components, or fully outsourced?

If you opt for a completely DIY solution, a fully fledged infrastructure with all the above elements covered can take up to 2-3 years to build. Deciding on what to buy or build could reduce this time frame dramatically.

In-house Open source solutions and partial outsourcing Out-of-the-box

Time to market 2-3 years 1-2 months Less than 1 day
Maintenance 50+ hours per year 50+ hours per year 0
Possible bugs Many Many Few
Scalability Low Low High

A comparison of different types of delivery.

UploadUFO will be an out-of-the-box solution, as we need to ship the feature faster and focus on the core functionality. We also aren’t ready to solve all the possible issues of open-source solutions, as we’d have to bring many moving parts together, and support and maintain the infrastructure in the future.



7. Milestones

Plan the project milestones in line with the development strategy you choose, defining particular dates. Make this planning true to life and change dates if you move faster or slower than expected.

Since UploadUFO is an out-of-the-box solution, we’ll have the following implementation flow:

• Making a shortlist of 3rd party solutions: May 1st
• Product demos and choosing a vendor: May 10th
• Setup, configuration, and tests: May 14th
• Feature release: May 15th



8. Open Questions

If there are still blind spots or areas you lack expertise in, list the questions you could ask your stakeholders and industry experts.

❓What do we need to do to meet GDPR compliance?

❓How well can we optimize content and what will be the impact on page load speed and hosting costs? What other perks are there for the end users?



What’s Next?

If you decide to use this tech spec, refine it by showing it to your teammates, stakeholders, and other interested parties. In case of critics, don’t give your reviewers the cold shoulder: they’re helping you to create the best possible solution.

If you’re a tech spec renegade and hate waterfall-ish approaches, just take all this information to the story mapping session and get your kudos from the team.



P.S. Laying the Cards on the Table

The UploadUFO functionality we described is actually inspired by a real-world project that was released by Shogun, a powerful drag-and-drop page builder for online stores. The team needed to implement file handling functionality that would:

• Outsource the image handling burden
• Reduce image bandwidth and hosting costs
• Speed up production and time to market
• Ensure reliable delivery

As a result, Shogun cut 86% of image bandwidth costs, achieved 1.5% page load acceleration, and saved three months of development time, resulting in a $200,000 overall savings.

? Read the full story


Print Share Comment Cite Upload Translate
APA
Uploadcare | Sciencx (2024-03-28T20:45:45+00:00) » Cheat Sheet: Mapping out a Powerful File Handling System. Retrieved from https://www.scien.cx/2021/04/29/cheat-sheet-mapping-out-a-powerful-file-handling-system/.
MLA
" » Cheat Sheet: Mapping out a Powerful File Handling System." Uploadcare | Sciencx - Thursday April 29, 2021, https://www.scien.cx/2021/04/29/cheat-sheet-mapping-out-a-powerful-file-handling-system/
HARVARD
Uploadcare | Sciencx Thursday April 29, 2021 » Cheat Sheet: Mapping out a Powerful File Handling System., viewed 2024-03-28T20:45:45+00:00,<https://www.scien.cx/2021/04/29/cheat-sheet-mapping-out-a-powerful-file-handling-system/>
VANCOUVER
Uploadcare | Sciencx - » Cheat Sheet: Mapping out a Powerful File Handling System. [Internet]. [Accessed 2024-03-28T20:45:45+00:00]. Available from: https://www.scien.cx/2021/04/29/cheat-sheet-mapping-out-a-powerful-file-handling-system/
CHICAGO
" » Cheat Sheet: Mapping out a Powerful File Handling System." Uploadcare | Sciencx - Accessed 2024-03-28T20:45:45+00:00. https://www.scien.cx/2021/04/29/cheat-sheet-mapping-out-a-powerful-file-handling-system/
IEEE
" » Cheat Sheet: Mapping out a Powerful File Handling System." Uploadcare | Sciencx [Online]. Available: https://www.scien.cx/2021/04/29/cheat-sheet-mapping-out-a-powerful-file-handling-system/. [Accessed: 2024-03-28T20:45:45+00:00]
rf:citation
» Cheat Sheet: Mapping out a Powerful File Handling System | Uploadcare | Sciencx | https://www.scien.cx/2021/04/29/cheat-sheet-mapping-out-a-powerful-file-handling-system/ | 2024-03-28T20:45:45+00:00
https://github.com/addpipe/simple-recorderjs-demo