Project: Simple filesystem

Introduction

Music:
Repository:


Resource management is all around us.
On macro scale, the sand is just a resource. Resource we, people are constantly using in construction industry. Food is a resource we are dealing with every day. Amount of wheat in given country is also a resource. And resources usually needs to be managed. As a programmer, I tend to meet resource management also, on a smaller, micro scale. My Ubuntu operating system with its shell allows me to organize my files. My applications are constantly using some resources of my computer's or servers' they are working on, which I try to optimize. Databases are working on huge sets of data and it's my job to predict and handle situation, when requested resource is available or does not exist.

Managing assets is a common thing in projects. I have worked in the past on some legacy projects, which handled this topic, but it always felt... inefficient? Not clean enough? Too complicated?

Mostly because of those reasons I decided I would like to create concept of a file system, that is system for managing files. It should work just like any operating system would, something almost everyone is familiar. As such, I created some basic "specification" for project.

Visualization of Ubuntu's filesystem

Specification

It should:
  • store files in tree-like structures of directories (or any other "grouping" abstraction) and nodes (any file not consisting of number of files)
  • let it be extensible - it should accept some basic structures at first, like text files and directories, but it should be a breeze to add some more
  • have simple yet quite generic API for management
    • adding/removing/updating elements
    • adding/removing elements to/from directories
    • nodes should be interactive in different manner, since opening a text file is a different story then opening an image file
    • handle more complicated operations, like copying, moving, deleting recursively
  • I do not care about actually storing files, it is less important. I would like to store only paths to files and structure them in a tree
  • although it is a simple project, I would like to use actual database as a backbone
  • REST API might be nice addition, also some way to visualize the file system.
I am not a frontend developer by any mean, yet it should not be complicated to create small React project and represent in some way my backend.

Building blocks: technology stack and architecture

I decided to create standard layered web application, back it with:
  • Java 11
  • Spring Boot 2.1.6
  • PostgreSQL
which allows quick development, as Spring is almost fully configurated out of the box.

Application consists of following layers:
  • Rest Controller - responsible for communication, external API definition
  • Facade - simplifies API and delegates to repositories/services
  • Service - business logic
  • Repository - database connection
Overkill for something so small? Maybe. But it suits me. I tried to work without Service layer, with business logic cluttered in entities, but it was a mess.

Building blocks: Entities

In my filesystem there are three entities with following responsibilities:
  • File - stores the "view" part of a file: 
    • the updatable (on this word) name of the file,
    • its place in tree in form of reference to parent file, 
    • flag deleted for soft deleting.
  • File Details - stores the actual data of an asset file: 
    • its size, 
    • mime, 
    • path to actual file,
    • flag deleted. 
  • Type - stores different types of files filesystem can store. Consists of: 
    • name 
    • handy flag isContainer.
This storage allows for good boundary of responsibility and also simplifies more complicated operation.

Features

From business logic point of view, there are just 5 crucial operations, which with above model are very easy to implement:
  • create - can create any File. If File Details is provided, will use it, otherwise will create new one based on passed data
  • update - since I do not want to change any uploaded File Details, it allows only for changing name
  • move - since all tree-structure is managed by File, it requires only changing parent file and recreate paths
  • copy - if file is not container, creates new File with the same File Details referenced. Otherwise creates new File as container and copies recursively all inner Files. Cannot use approach from deleting, since containers' structure should be preserved.
  • deleteRecursively - most complicated. First fetch all files containing deleted File path, then if File is container - soft-delete it, otherwise check how many copies File has, if more than one, soft-delete just the File entity, otherwise soft-delete referenced File Details.
Because of plans of creating some additional client, I created Dto, which consists of few important data when displaying Files. The big advantage here is that every File can be represented in exact same way. 

Also some helper method providing root of the tree (usually things like those are kept in user data) and streaming File Details content, so files actually can be opened.


Package hierarchy


Scaling & Extending


  • Creating new type of File is easy, just add entry to Type table.
  • Also Types can be customized to be sent via REST to client. This way client can be stateless, making it fully supplied by backend. Ant all changes with Types/Files can be introduced without downtime.
  • Copying is cheap - only original file is stored, that safes a lot of space.
  • Dockerizing the project should be easy - some Open Jdk container with PostgreSQL
  • Startup class always fill default values



Lokalizacja: Polska

Komentarze

Prześlij komentarz

Popularne posty z tego bloga

Java EE 8 & Wildfly 17 & RestEasy setup

Introduction Java SE/EE for quite a long time was slow. Not in performace, but in releasing new features. Since that was not a good thing in todays fast and modern world, when almost every week new hot thing comes up, it has had to change. And fortunatelly - it does. Now releases are much, much more frequent. And with Eclipse Foundation taking care of Java (or should I say Jakarta) the future looks bright. But Java SE/EE is nothing more than just a set of specifications. You cannot do a thing with it. You need application server. And this post is all about setting up a basic project with Wildfly, the Redhat's free, widely used and highly customizable application server. Since it is easy now to miss out on new stuff, I am going to user newest Wildfly, which is 17 (Java EE 8 support) with JDK 12 to create simple service. Whole code can be found right here:  https://github.com/KKucharczyk/javaee8-bootstrap Requirements To setup simple application, couple of things are requ
Czytaj więcej

Testing: jUnit's TemporaryFolder

Testing: jUnit's TemporaryFolder with Simple CSV generator The problem Lately I had a question in my head: how to test a method, which output is a file?  Should I divide creation of file from creation of content and test only the latter part? Well, usually I would like to call a method "generateReport" or retrieveFileByHash(), which encapsulate all file-connected operations and return simple java.io.File. In other words, my service classes looks like this: Interface (like ReportGenerator) one method returns: File Concrete class (like DefaultReportGenerator) one public implementation of interface method, (like generateReport()) returns: File set of private methods returns: whatever is needed to create File With such services, file's content creation is separated from File creation: Interface - ReportGenerator method: generateReport() returns File Concrete Class - DefaultReportGenerator method: implementation of generateRepor
Czytaj więcej

Two entities coexisting in one table

Obraz
I have stumbled across small problem. I had an abstract class, which aggregated all mutual properties of an object and two concrete classes. One did not need any additional fields, while the other needed another relation. Idea was to create one table and gather both those concrete objects from it. I have never done it before, so it was interesting thing to discover. I was working on Team abstraction. Team is a collection of heroes. But teams can be lead by someone or self-organized, hence I have three classes, like so: Notes: BaseTeam is an abstract class annotated with @MappedSuperclass . Since it is abstract I cannot instantiate it and with annotation all its properties will be passed to class extending it. Team class is a concrete class extending BaseTeam. It inherits all the properties from its superclass and does not define any new. It is annotated with @Entity and @Table with filled name property.  It is important to fill those properties, so concrete class can recogn
Czytaj więcej