I/O, Memory, and File Systems, Part II

tmpfs

tmpfs is a memory-backed file system that uses virtual memory. It can use both RAM and swap. It has two main advantages: it is fast, and its virtual space can be allocated dynamically as demand changes.

Linux kernel configurations normally enable tmpfs and mount it at /dev/shm. Use df to inspect it, mount to change its size, and ls /dev/shm to see whether files are using it. By default, /dev/shm is sized to 50% of physical memory.

Uses

• Store caches, such as a Squid cache.
• Store temporary files, such as browser caches or Unix sockets.

Caution

• Data is lost after a reboot. Persistence must be implemented separately if required.

Reading and writing large files

For a small file, reading the whole file into memory and operating on it is naturally the fastest approach. Large files require other techniques.

In Python, calling read after open loads the whole file into memory. If the file approaches or exceeds the available memory, an IOError will occur. A common alternative is to iterate over the file line by line, letting the Python runtime manage the buffer internally:

with open("s1.txt", "r") as f:
    for line in f:
        print line

In Java, memory-mapped files and RandomAccessFile map a file into the process’s virtual address space so it can be accessed through memory operations.

RandomAccessFile essentially combines DataInputStream and DataOutputStream with methods such as seek, length, and skipBytes. Through mmap, file pages are mapped into a contiguous range of the process’s virtual address space. Avoiding copies between user space and kernel space makes it fast.

Huge pages

Huge pages are mainly used for databases.

To be continued.

Originally published on SegmentFault.