Go reading files

Reading files is one of the most common operations in Go. In this article, we will learn about the different possibilities that Go offers for doing so.

The article is divided into several sections:

Load entire file into memory.
Read file in parts.
Read file line by line.
Embed file in the Go binary.

If you are new to the world of Go, I recommend the following previous articles:

Load entire file into memory:

The simplest way to read files is to load them directly into RAM using the os.ReadFile() function. This allows you to specify the file path as either relative or absolute.

package main

import (
    "fmt"
    "os"
)

func main() {
    contents, err := os.ReadFile("test.txt")
    if err != nil {
        fmt.Println("File reading error", err)
        return
    }
    fmt.Println("Contents of file:", string(contents))

    contents, err = os.ReadFile("/home/kr0m/test/test.txt")
    if err != nil {
        fmt.Println("File reading error", err)
        return
    }
    fmt.Println("Contents of file:", string(contents))
}

Contents of file: This is a test file content

Contents of file: This is a test file content

Read file in parts:

If your system has very limited resources or if the file to be read is extremely large, you might want to load the file content in parts to avoid exhausting the available RAM.

Basically, the process is:

Open the file.
If there are no errors, defer closing the file using defer.
Create a buffered reader from which you will read in chunks of len(b) bytes. The variable n will indicate the number of bytes read from the buffer.
Stop reading if the end of the file is reached.
If an error occurs and it’s not the end of the file, report the error.
If no errors occur, display the n bytes from the slice b.

package main

import (
    "bufio"
    "fmt"
    "io"
    "log"
    "os"
)

func main() {
    f, err := os.Open("test.txt")
    if err != nil {
        log.Fatal(err)
    }
    defer func() {
        if err = f.Close(); err != nil {
            log.Fatal(err)
        }
    }()

    r := bufio.NewReader(f)
    b := make([]byte, 3)
    for {
        n, err := r.Read(b)
        if err == io.EOF {
            fmt.Println("Finished reading file")
            break
        }
        if err != nil {
            fmt.Printf("Error %s reading file", err)
            break
        }
        fmt.Println(string(b[0:n]))
    }
}

lin
e1

lin
e2

lin
e3

lin
e4

lin
e5

Finished reading file

Read file line by line:

One of the most common ways to access the content of a file is by reading it line by line. The steps are very similar to the previous case, but this time we use a buffer scanner instead of a buffer reader. The scanner reads the lines from the file, and we finally check if any errors occurred while reading.

package main

import (
    "bufio"
    "fmt"
    "log"
    "os"
)

func main() {
    f, err := os.Open("test.txt")
    if err != nil {
        log.Fatal(err)
    }
    defer func() {
        if err = f.Close(); err != nil {
            log.Fatal(err)
        }
    }()

    s := bufio.NewScanner(f)
    for s.Scan() {
        fmt.Println(s.Text())
    }

    err = s.Err()
    if err != nil {
        log.Fatal(err)
    }
}

line1
line2
line3
line4
line5

Embed file in the Go binary:

In Go, it is possible to embed files directly into the binary itself. This is not limited to text files; it can even include other binaries. You simply need to indicate it using //go:embed just before the variable you will use to store the content. However, keep in mind that this is not a secure way to hide any type of information; it is merely a convenient way to ensure that your software has everything it needs to run without relying on external files.

package main

import (
    _ "embed"
    "fmt"
)

//go:embed test.txt
var contents []byte

func main() {
    fmt.Println("Contents of file:", string(contents))
}

echo SUPERSECRET_PASSWORD > test.txt
go build test.go
rm test.txt

When you run the binary, you can see that the file content is still within the binary.

./test
Contents of file: SUPERSECRET_PASSWORD

You can view the content in the binary externally using the strings command.

strings test|grep SUPERSECRET
SUPERSECRET_PASSWORD