The course was really interesting, it gave me a panoramic view of the OpenTelemetry internals, and how they work. In the course there was an hands-on part, where I could see OTel in action, seeing how OTel can be integrated with some Java and Python applications. No rocket-science tho, this course is a basic course but in my opinion it’s really worth it, if you want to start working with OTel and c’mon, it’s free!

The Physics of Software Development

Lots of business people don’t even know that behind a software project, there are actual physical laws that need to be respected to avoid ending up in situations far from reality, introducing wrong expectations, and catastrophic results.

The Software Project Triangle

There is a very simple rule for interpreting this triangle:

Important If one of these components changes, at least one of the others needs to change.

People decrease

If, for whatever reason, the number of people working on a project decreases, the scope needs to be reduced and/or the timeline increased.

Some people may push the team to work extra hours, until they realize that a tired developer is likely to introduce more bugs than a fresh one, and I don’t even want to talk about burning those people out and eventually end up with fewer people than when you started.

Scope increases

Too often I see businesses wanting to increase the scope in the middle of the project because “our priorities changed”, not respecting the physics of software development.

I’m sorry, Mr. Businessman, but if you increase the scope, then the timeline and/or the number of people must also increase to keep things working fine.

Timeline decrease

If the timeline deadline is moved closer, then the scope needs to be decreased. What we often see in this situation is that people will work more hours, increasing the “people” variable.

But be careful with that, because the adding people parenthesis is still valid, making people work more hours can introduce bugs and make them less efficient and productive.

Adding people

I want to add a side note on the “adding more people” concept since this is not very clear for many. When we add a new person to an ongoing project, that person needs to be onboarded, and it requires time, time that will reduce the original planned timeline, time that hasn’t been taken into consideration at the beginning of the project.

True, some projects can have tasks that can be parallelized or require very little onboarding, and in those cases it can definitely improve the timeline, but this always involves hard tradeoffs in onboarding time.

So no, doubling the number of engineers won’t cut the time to completion in half, but if they have domain knowledge and the work can be split up enough, then it can reduce the timeline if done carefully.

So next time the business wants to change any of these variables without adjusting the others, remembering this principle will help you avoid paying the price in delays, bugs, frustration, or burnout.

With more than 10 years of experience as a Software Engineer, I’ve had the opportunity to attend hundreds of team retrospectives, and let me tell you: it’s the most valuable meeting a team can have, if you know how to do it properly. 😏

Why is it important? 🎩

A retrospective is when a team can learn from its own mistakes, share gratitude, ask for help, celebrate and talk about new ideas. Do you already see the potential? 📈

Self-improvement 🔁

Do you see something in the team that could be improved or is slowing you down? Raise your hand. 🙋‍♂️

“What if?” 🤔

Do you have a new idea? Do you want to try something new, and want to get the team’s opinion on it? Go for it!

Celebration!! 🥳

Did your team achieve something, or do you want to say thank you to a specific person that helped you out? (making it in public is better 😉)

Reflection 🙅

The power of this ceremony lies in encouraging the team to reflect on past events like what went wrong, or what could have been better.

Important It is about continuous improvement, not blaming. 🙅🏻

In a safe environment we should aim to point our fingers at processes that didn’t work, not at people.

Team Psychological Safety

Important Psychological safety is a very important thing to have in your team, the aim is to make the people feeling safe. This should be your top priority, the sooner you have it in your team, the better.

Having people in the team who feel safe is one of the most important aspects of healthy team dynamics. A team that feels safe, will be transparent, honest, and they will take risks. During a retro, you want people to have the courage to speak up and address issues by discussing solutions and ideas. You want your team not to be afraid of saying “we should stop this” or “this is not working for us”. Without psychological safety, we can’t have all the benefits we mentioned earlier. If people don’t feel safe, your retrospective simply won’t work. I’m not going to cover this topic here, but you can definitely have a look at some articles online like:

• https://www.thoughtworks.com/insights/blog/product-innovation/psychological-safety-product-innovation
• https://www.ccl.org/articles/leading-effectively-articles/what-is-psychological-safety-at-work/
• https://psychsafety.com/about-psychological-safety/

Format 👾

Many people I’ve talked with believe there’s only one effective retrospective format

Which is completely false, changing retrospective format depending on your team situation is crucial to get the most out of this ceremony.

Retros should be fun and effective. Not all sprints or quarters look the same; so why should your retro?

Different formats?

There are lots of templates online that can be used, my favorites can be found here:

• Fun Retrospectives.
• Neatro Retro templates

Setup

Let’s pick as an example “The good, the bad and the ugly” retro format.

Set up three columns respectively with

• The good 👍🏻
• The bad 🙅🏻
• The ugly 👹

Old action items 📜

Before starting the retro, spend some time reviewing the action items created in the last retro. Get updates on them, always remember to have them assigned to someone and add them to the next sprint.

Let’s start ⏲️

In this phase the team can start creating stickies on the board.

Each sticky should be in the appropriate column.

• The good: things that went well, and we should repeat, and do more of it.
• The bad: things that should have never happened, and we must get rid of it.
• The ugly: things that did not go so well, and we should look for improvements, turning them into something better.

Write each one so the team can easily understand its context at a glance:

• ❌: “Our process doesn’t work for me”
• ✅: “The code review process is complicated and difficult to follow”
• ❌: “Lack of privileges”
• ✅: “Couldn’t do the migration due to lack of privileges.”

This activity should last around 3–5 minutes. If the team isn’t very active, wrap it up earlier; if they’re engaged, give them an extra minute or two.

Clarification and grouping 🏘️

In this phase the whole team reviews the stickies and clarify them if necessary, and start grouping similar stickies together. Something that works well for me is drawing a large square to summarize each group of related stickies, like in the image

Voting 🗳️

Now it’s time to vote, each member of the team should have 2 or 3 votes, and votes should be anonymous.

If there is a group of stickies, votes go on the main group label, otherwise they go on the single sticky.

This activity should last between 3 and 5 minutes, nowadays most tools show when everyone has voted, or you can say to your team members to raise their hand on your call to mark that they have voted.

After that you can show the voting results.

Discussion 🎙️

Pick the sticky (or group) with the highest number of votes, and set 10 minutes for the discussion.

The aim of the conversation is to explore what happened, and address it through the creation of action items.

After timing out, the team can decide if they want to continue for 5 more minutes or stop. Repeat until the team is satisfied and there are action items for that specific topic.

The conversation should be started by the author, and the team should follow up the conversation, adding insights and thoughts.

Important If there are 5 votes, at least two or three people should be talking and adding something to the conversation; having just one person talking it’s not a conversation, it’s a monologue. 🙅🏻

Action Items 🪧

As introduced above, action items are produced by the team to address their issues, they should be S.M.A.R.T.:

• Specific
• Measurable
• Achievable
• Relevant
• Time-Bound

Assign each action item to an owner that will be accountable for it to be executed during the following sprint.

Important Adding an action item to the backlog and leaving it there won’t solve the issue, solving the issue will solve the issue. 🤯

Round of Kudos and positive things 👏🏻 🎉

Before closing the meeting, spend some time reviewing kudos and positive stickies, to end the retro on a good note. 🧘🏻

Common Pitfalls 👎🏻

Team members become spectators 🥱❌

When retrospectives are always facilitated by the same person (usually the tech lead or manager), the outcome tends to be the same every time: the team tends to get used to it, and the retro becomes “another meeting that could have been an email”. Team members become spectators. Instead of feeling that this ceremony is for them, they start seeing it as someone else’s responsibility.

Problem: Always the same person facilitating the retro 🙅🏼‍♂️ Usually the tech lead or the manager facilitates the retro, leaving the team out of this duty.

Solution: Rotate 🔁 Rotate the facilitator every 1/2/4 weeks, make the people responsible for facilitating the retro correctly and efficiently. Give them feedback about how to improve it.

Tip: Automatic rotation 🤖 Add an automatic bot to rotate the person who takes the facilitator role for a specific time-window.

Hearing always the same voice 🎙️❌

Common scenario, have a sticky with lots of votes, but we hear just one or two people talking for the whole (time-boxed) time. Generally, if a sticky has a lot of votes, it means several people in the team care about the topic and have something to say. My advice is to let the author start, and then call out someone else who added a vote to it.

Question: Anybody else? Does anybody want to add anything? Can we hear another opinion? Thoughts?

Different people might have different reasons to vote a specific sticky and some people are more silent than others, the benefit of doing this might be:

1. We hear what the owner has to say
2. Hear others’ opinions (they vote it, they have at least one opinion about that topic)
3. Keep people engaged.

Problem: Cards with lots of votes being tackled by a single person 🙅🏼‍♂️ We have only one voice in the room.

Solution: Let other people speak 🗣️ Don’t let the author of the sticky be the only one who gives an opinion, ask the rest of the team who voted the card to speak up and add more about the topic the team wants to talk about.

Tip: Involve silent people Try to involve people who didn’t talk during the whole meeting, but don’t force them to do so.

Not time-boxing 🍅

As a facilitator the timer is your best friend, it is your duty to keep the meeting rolling and on-time, making sure that the important topics are discussed and that some action items have been created. To do so, you need to have time-boxed discussions. Whenever a discussion starts let’s set 10 minutes to talk through the topic, if at the end of those 10 minutes the team is satisfied you can move on, otherwise just ask about adding up 5 minutes more.

Question: What now? Shall we spend 5 more minutes on this topic or move on?

Do it just once, and focus on finding an action item, then move to the following topic, it brings lots of overlooked benefits:

1. Force the team to focus on what really matters
2. Keep the timing under control
3. It allows tackling more important topics

Problem: Spending too much time on a single topic Spending most of our precious time talking just about one or two topics for the whole meeting

Solution: Time-box discussions 🍅 Set 10 minutes for each topic conversation, the aim is to find an action item. If 10 minutes are not enough, ask the team if they want to talk about the topic for 5 more minutes or if they are okay with the outcome.

Tip: Shared timers Try to use a timer that is visually shared with the team, (or just share your screen) so everybody is aware of the timing.

Public voting 🃏

Some people in the team have more influence than others, and this can easily harm a retro meeting. Within a team there are multiple dynamics, and we have always to remember that those dynamics influence how we interact and work with each other. So, to avoid having this hidden influence within our meeting, voting should be anonymous.

Only one person doing the grouping 🏘️❌

Grouping helps the team to clarify the stickies and to come up with something that everybody understand and can vote, don’t let one person doing the dirty job when the rest of the team just stay still, involve them into the process, ask questions and how to name things. Make them part of the process.

Talk about every single sticky on the board 😴❌

Voting helps to prioritize what really matters for the team, and those stickies should be discussed before the end of the meeting. If more stuff can be discussed go for it, but it’s not something you should tackle, especially if the retro runs over time. Take those stickies into consideration and move on, closing the retro.

Problem: We ran out of time We talked about less important stuff. The team lost interest.

Solution: Pick the top 3 stickies Pick the first 3 most voted topics and discuss them, it should be enough to tackle the important stuff. Close the meeting whenever you are done with them.

Postponing the retro because “we are too busy” 📉❌

Seriously, don’t. The retro it’s one of those meetings that if run well can help your team to move faster. ⚡️ A busy team can overlook important things that in that specific moment are making the team itself slower or less efficient. Reflecting over the sprint the team just had can make more evident what is not working, especially if the team needs to be fast.

Example 🏎️ During a car race, if you have a flat tire, don’t you change it because you don’t have the time to address it?

Waiting for the team/tech lead to have a retro 🪫❌

The retro is for the whole team, not for single individuals and the team shouldn’t wait for anybody to run it. Action items are assigned to team members, making them accountable for resolving them.

Problem: Only the tech/team lead can solve the issue ✋🏻 Can be that some issues can’t be solved by individual contributors, so what’s the point to have a retro if we can’t do anything about it?

Solution: Have action items 📝 The team can always discuss the issue, create action items about it and assign the tech/team lead to follow up offline or whenever that person is back.

No product people in your retro ☎️ ❌

In some companies product and engineering teams are two separate entitites which collaborate here and there, but never share a strong communication funnel. So what I’ve been seeing is engineering teams complaining about product people that never receive that feedback promptly or with the right context.

Problem: Broken feedback loop ⛓️‍💥 Product people are not participating to the retro. Most of the engineers think that the retro would be too technical for product people, and product people think that they can’t provide any value to the engineering team conversations.

What worked very well in my experience, is involving them to the team retros, starting with just once per month and then increase the attendances according to the feedback.

Solution: Retros all together 🔗 Start involving product people in the engineering team retros, start small and then increase the frequency if it is needed.

I’ve seen product people become more aware of team issues and their surroundings. Engineering teams felt more heard by the people they were supposed to be collaborating with. Thanks to the faster feedback loop, everyone felt that collaboration improved by an order of magnitude, understanding better the reasons behind certain decisions and processes that were affecting the team.

Conclusion

I hope that these tips help you understand how you can make the most out of the retro and why you should care about having it in your team as well.

I’m just an engineer: I’m not a good faciliator! 🙋‍♂️ Facilitation is like a muscle, the more you practice it, the stronger it gets. And think about it, AI can’t do it. 😏

Next time you have your retro, apply one of these ideas, and you’ll see how fast a good conversation can transform your team. 🤞🏻

Until next time! 👋🏻

Initial situation

Implementing external interfaces is quite common in Go projects, but sometimes we have the necessity to implement those interfaces multiple times with different struct, ending up having lots of structs implementing the same methods without really having a different behavior in place.

For example, we can have something like this into an external dependency, let’s say something that helps us to return a particular response having a particular format:

1
2
3
4
5
//External dependency, living in an external project
type JSONResponse interface {
    Name() string
    GetSomething() SomeThing
}

And then, we have in our code:

1
2
3
4
5
6
7
type TypeOneResponse struct {
    ...
}


func (r *TypeOneResponse) Name() string { return r.Name }
func (r *TypeOneResponse) GetSomething() SomeThing { ..logic to return something.. }

Now let’s imagine that we have to implement it for all our responses, we are going to have:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
type TypeOneResponse struct {
    ...
}


func (r *TypeOneResponse) Name() string { return r.Name }
func (r *TypeOneResponse) GetSomething() SomeThing { ..logic to return something.. }


type TypeTwoResponse struct {
    ...
}
func (r *TypeTwoResponse) Name() string { return r.Name }
func (r *TypeTwoResponse) GetSomething() SomeThing { ..logic to return something.. }

...

In no time we are going to have hundreds of lines of code that are just duplicated. Not nice, right?

Generics to the rescue

Luckily Go introduced Generics, which helps us to solve the problem.

Let’s define a new common and generic struct:

1
2
3
4
type Response[T] struct {
    Name string
    Payload T
}

and let’s implement the methods:

1
2
3
4
func (r *Response[T]) GetName() string { return r.Name }
func (r *Response[T]) GetSomething() SomeThing { ..logic to return something.. }

...

We can then easily reuse this generic struct to instantiate our response types without re-implementing all methods again and again.

1
2
3
4
5
6
7
8
9
responseOne := Response[TypeOneResponse]{
    Payload: TypeOneResponse{ ... },   
}

responseTwo := Response[TypeTwoResponse]{
    Payload: TypeTwoResponse{ ... },
}

...

Real scenario

Let’s say we have to use JSON:API format and we want to make sure our custom responses use it. As I wrote above, it requires our structs to implement the dedicated interfaces every time leading to a very messy status.

Let’s say I have a response like this:

1
2
3
type TypeOneResponse struct {
    Field: value,
}

and I want to return it using a JSON:API format, it should result something like this:

1
2
3
4
5
6
7
{
  "type": "something",
  "id": "1",
  "attributes": {
    "field": "value"
  },
}

How to achieve it for all our responses? Implementing implementing implementing. Let’s avoid it using generics.

Let’s create a common Response:

1
2
3
4
type Common[T any] struct {
   ID string `json:"-"`
   Payload T
}

Then we need to implement all methods as we have seen above:

1
func (c *Common[T]) GetID() string { return c.ID }

The result will be something like this:

1
2
3
4
5
6
7
8
9
{
  "type": "something",
  "id": "1",
  "attributes": {
    "Payload": {
        "field": "value",
    }
  },
}

A bit different from what we expected right? We have to get rid of the Payload attribute and have only what it has inside. Let’s fix it implementing our custom MarshalJSON and UnmarshalJSON methods!

1
2
3
4
5
6
7
func (c *Common[T]) MarshalJSON() ([]byte, error) {
 return json.Marshal(&c.Payload)
}

func (c *Common[T]) UnmarshalJSON(v []byte) error {
 return json.Unmarshal(v, &c.Payload)
}

Obtaining as a result:

1
2
3
4
5
6
7
8
9
10
11
{
  "data": [
    {
      "type": "something",
      "id": "1",
      "attributes": {
        "field": "value",
      },
    }
  ]  
}

Now, every time we want to return our responses, we just need to instantiate our responses in this way:

1
2
3
4
5
6
7
8
9
resp1 := Common[TypeOneResponse]{
    Payload: TypeOneResponse{
        Field: value,
    }}

resp2 := Common[TypeTwoResponse]{
    Payload: TypeTwoResponse{
        AnotherFieldName: value,
    }}

Of course, we can further improve readability creating a factory function like:

1
func NewCommon(payload T, value valueType) Common[T]

and create our custom types:

1
2
type CommonOne Common[TypeOneResponse]
type CommonTwo Common[TypeTwoResponse]

and use them accordingly:

1
2
3
4
5
6
7
8
9
resp1 := CommonOne{
    Payload: TypeOneResponse{
        Field: value,
    }}

resp2 := CommonTwo{
    Payload: TypeTwoResponse{
        AnotherFieldName: value,
    }}

This is just a small but very effective example about how to use generics to improve your codebase. Happy coding! :rocket:

DNS Resolver what?

A DNS Resolver is a crucial component that allows you to resolve an IP address from a certain domain.
For instance, it allows your browser to know where to find the server associated with a specific domain.
(i.e. domenicoluciani.com → 172.67.144.42)

The Coding Challenge

The coding challenge consists of building a simple DNS Resolver that is capable of resolving an IP address from a certain domain. I’d like to highlight about the simple part.
You can find the challenge here: https://codingchallenges.fyi/challenges/challenge-dns-resolver/

Preface

As I did in the previous posts, I took this challenge just for fun and dive deeper into how a DNS resolver works. It’s a weekend project that obviously can contain errors, so if you find one -or more-, please let me know, never stop learning, right? 📚

Things I learned with this challenge

• Obviously, I learned A LOT about how a DNS Resolver works.
• How the name resolution works
• What encoding is used
• Went deeper into binary protocols and how they work
  • And how to fill a structure with binary data in Go.
• The DNS RFC is super clear (well done authors!)
• Testing and ChatGPT saved me from a lot of debugging time

Are you interested in one of these things? Then keep reading! 🕵🏻‍♂️

Step 0

For this challenge, I decided to use Go and I tried to use a Test Driven Development approach as usual, even tho not completely since my goal wasn’t to apply it perfectly but to have a good simple design. 🙏🏻

Step 1

This step is about creating a query message that we have to send to the name server, composed of these fields:

1. A header.
2. A question section.
3. An answer section.
4. An authority section.
5. An additional section.

Header

The header is always present, and it is composed in this way

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
                                    1  1  1  1  1  1
      0  1  2  3  4  5  6  7  8  9  0  1  2  3  4  5
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                      ID                       |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |QR|   Opcode  |AA|TC|RD|RA|   Z    |   RCODE   |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                    QDCOUNT                    |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                    ANCOUNT                    |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                    NSCOUNT                    |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                    ARCOUNT                    |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+

• Query ID
• Some flags (at the beginning for the challenge we set this flags to 1 and then to 0 because at the beginning we contact a dns resolver to then switch to an authoritative nameserver)
• QDCOUNT = Number of questions
• ANCOUNT = Number of answers
• NSCOUNT =  Number of authorities
• ARCOUNT = Number of additional

You can see the detail in the RFC, section 4.1.1.

Question

The question section is composed in this way:

1
2
3
4
5
6
7
8
9
10
11
12
                                    1  1  1  1  1  1
      0  1  2  3  4  5  6  7  8  9  0  1  2  3  4  5
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                                               |
    /                     QNAME                     /
    /                                               /
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                     QTYPE                     |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                     QCLASS                    |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+

• QNAME = encoded name of the domain (i.e. 3dns6google3com)
• QTYPE = the type of the query (i.e. A, MX, etc.)
• QCLASS = class type (i.e. internet)

The details are defined in these sections of the RFC:

• Question: https://datatracker.ietf.org/doc/html/rfc1035#section-4.1.2
• Type: https://datatracker.ietf.org/doc/html/rfc1035#section-3.2.2
• Class: https://datatracker.ietf.org/doc/html/rfc1035#section-3.2.4

Query

Both sections need to be encoded in bytes and put together in order to form the final query.

When we send the request we don’t don’t need to compose authorities and additionals, they are going to be filled out in the response.

Let’s see the code 👀

Let’s take a look at how I created and converted these two structures into bytes.

The test is quite minimal and simple:

1
2
3
4
5
6
7
8
9
    t.Run("Should encode an header into bytes", func(t *testing.T) {
        header := NewHeader(22, RECURSION_FLAG, 1, 0, 0, 0)

        encodedHeader := header.ToBytes()

        expected, err := hex.DecodeString("0016010000010000000000000")
        assert.NotNil(t, err)
        assert.Equal(t, expected, encodedHeader)
    })

1
2
3
4
5
6
7
8
9
10
const RECURSION_FLAG uint16 = 1 << 8

type Header struct {
    Id      uint16
    Flags   uint16
    QdCount uint16
    AnCount uint16
    NsCount uint16
    ArCount uint16
}

and then I implemented the function for the conversion:

1
2
3
4
5
6
7
8
9
10
11
func (h *Header) ToBytes() []byte {
    encodedHeader := new(bytes.Buffer)
    binary.Write(encodedHeader, binary.BigEndian, h.Id)
    binary.Write(encodedHeader, binary.BigEndian, h.Flags)
    binary.Write(encodedHeader, binary.BigEndian, h.QdCount)
    binary.Write(encodedHeader, binary.BigEndian, h.AnCount)
    binary.Write(encodedHeader, binary.BigEndian, h.NsCount)
    binary.Write(encodedHeader, binary.BigEndian, h.ArCount)

    return encodedHeader.Bytes()
}

The go encoding/binary package was crucial to working with anything related to the binary encoding.
Basically, I just append into encodedHeader bytes.Buffer whatever I have on each field of the struct using a Big-endian order.

I have done the same thing for the question part:

1
2
3
4
5
6
7
8
9
    t.Run("Should encode a question into bytes", func(t *testing.T) {
        question := NewQuestion("dns.google.com", TYPE_A, CLASS_IN)

        encodedQuestion := question.ToBytes()

        expected, _ := hex.DecodeString("03646e7306676f6f676c6503636f6d0000010001")
        assert.NotNil(t, expected)
        assert.Equal(t, expected, encodedQuestion)
    })

and the implementation

1
2
3
4
5
6
7
8
func (q *Question) ToBytes() []byte {
    encodedQuestion := new(bytes.Buffer)
    binary.Write(encodedQuestion, binary.BigEndian, q.QName)
    binary.Write(encodedQuestion, binary.BigEndian, q.QType)
    binary.Write(encodedQuestion, binary.BigEndian, q.QClass)

    return encodedQuestion.Bytes()
}

One thing here is that we have to encode the domain name using a simple encoding algorithm, here the test:

1
2
3
4
    t.Run("Should encode the dns name", func(t *testing.T) {
        encodedDnsName := encodeDnsName([]byte("dns.google.com"))
        assert.Equal(t, []byte("\x03dns\x06google\x03com\x00"), encodedDnsName)
    })

Basically we replace each dot with a number of characters we have right after it, so dns.google.com becomes 3dns6google3com.

The implementation:

1
2
3
4
5
6
7
8
9
func encodeDnsName(qname []byte) []byte {
    var encoded []byte
    parts := bytes.Split([]byte(qname), []byte{'.'})
    for _, part := range parts {
        encoded = append(encoded, byte(len(part)))
        encoded = append(encoded, part...)
    }
    return append(encoded, 0x00)
}

and now let’s join both header and question together, the test:

1
2
3
4
5
6
7
8
9
10
11
    t.Run("Should create a query", func(t *testing.T) {
        header := NewHeader(22, RECURSION_FLAG, 1, 0, 0, 0)
        question := NewQuestion("dns.google.com", TYPE_A, CLASS_IN)

        query := NewQuery(header, question)

        expected, err := hex.DecodeString("00160100000100000000000003646e7306676f6f676c6503636f6d0000010001")
        assert.Nil(t, err)
        assert.Equal(t, expected, query)
    })

and the resulting code:

1
2
3
4
5
6
7
8
func NewQuery(header *Header, question *Question) []byte {
    var query []byte

    query = append(query, header.ToBytes()...)
    query = append(query, question.ToBytes()...)

    return query
}

Step 2

Now we need to send our query over the network using the UDP protocol and get back the response from the name server.
The code is quite simple, I didn’t want to spend much time on error checking tho, here is the implementation:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
type Client struct {
    serverAddress string
    port          int
}

func NewClient(address string, port int) *Client {
    return &Client{serverAddress: address, port: port}
}

func (c *Client) SendQuery(query []byte) []byte {
    conn, err := net.Dial("udp", fmt.Sprintf("%s:%d", c.serverAddress, c.port))
    if err != nil {
        fmt.Printf("Dial err %v\n", err)
        os.Exit(-1)
    }
    defer conn.Close()

    if _, err = conn.Write(query); err != nil {
        fmt.Printf("Write err %v\n", err)
        os.Exit(-1)
    }

    response := make([]byte, 1024)
    lengthOfTheResponse, err := conn.Read(response)
    if err != nil {
        fmt.Printf("Read err %v\n", err)
        os.Exit(-1)
    }

    if !hasTheSameID(query, response) {
        fmt.Printf("Response doesn't have the same ID of the query q:%v, r:%v\n", query, response)
        os.Exit(-1)
    }

    return response[:lengthOfTheResponse]
}

One check I had to implement due of requirements is about the query ID (it can be whatever), the one we send should be the same as the one that we receive from the server, here the test:

1
2
3
4
5
6
    t.Run("Should check if the response starts with the same ID as the query", func(t *testing.T) {
        query, _ := hex.DecodeString("00160100000100000000000003646e7306676f6f676c6503636f6d0000010001")
        response, _ := hex.DecodeString("00168080000100020000000003646e7306676f6f676c6503636f6d0000010001c00c0001000100000214000408080808c00c0001000100000214000408080404")

        assert.True(t, hasTheSameID(query, response))
    })

and the implementation:

1
2
3
func hasTheSameID(query, response []byte) bool {
    return slices.Equal(query[:2], response[:2])
}

Step 3

Now our goal is to parse the response, the message luckily has the same structure as the one we sent, let’s see how to parse the header:

1
2
3
4
5
6
7
8
9
10
11
12
13
    t.Run("Should create an header from a response", func(t *testing.T) {
        response, _ := hex.DecodeString("001680800001000200000000")
        header, _ := ParseHeader(bytes.NewReader(response))

        assert.Equal(t, &Header{
            Id:      0x16,
            Flags:   1<<15 | 1<<7, // QR (Response) bit = 1, OPCODE = 0 (standard query), AA = 1, TC = 0, RD (Recursion Desired) bit = 1, RA = 1, Z = 0, RCODE = 0
            QdCount: 0x1,
            AnCount: 0x2,
            NsCount: 0x0,
            ArCount: 0x0,
        }, header)
    })

and the implementation:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
func ParseHeader(reader *bytes.Reader) (*Header, error) {
    var header Header

    binary.Read(reader, binary.BigEndian, &header.Id)
    binary.Read(reader, binary.BigEndian, &header.Flags)
    switch header.Flags & 0b1111 {
    case 1:
        return nil, errors.New("error with the query")
    case 2:
        return nil, errors.New("error with the server")
    case 3:
        return nil, errors.New("the domain doesn't exist")
    }
    binary.Read(reader, binary.BigEndian, &header.QdCount)
    binary.Read(reader, binary.BigEndian, &header.AnCount)
    binary.Read(reader, binary.BigEndian, &header.NsCount)
    binary.Read(reader, binary.BigEndian, &header.ArCount)

    return &header, nil
}

Thanks to the encoding/binary package we can easily get the information we need and store it into our structure.
Here I also implemented a check to verify that the response didn’t have errors. Thanks to ChatGPT I could easily generate the binary response for each use-case otherwise I would have had to do it by myself 😫

The tests are more or less like this:

1
2
3
4
5
6
7
8
9
10
    t.Run("Should return an error if the header flags contains a query error", func(t *testing.T) {
        response, _ := hex.DecodeString("001680810001000200000000")

        header, err := ParseHeader(bytes.NewReader(response))

        assert.Nil(t, header)
        assert.NotNil(t, err)
        assert.EqualError(t, err, "error with the query")
    })

Now let’s parse the rest of the message:

1
2
3
4
5
6
7
8
9
func ParseQuestion(reader *bytes.Reader) *Question {
    var question Question

    question.QName = []byte(DecodeName(reader))
    binary.Read(reader, binary.BigEndian, &question.QType)
    binary.Read(reader, binary.BigEndian, &question.QClass)

    return &question
}

Here we see a DecodeName function, which is the most difficult part in terms of implementation for the DNS Resolver:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
func DecodeName(reader *bytes.Reader) string {
    var name bytes.Buffer

    for {
        lengthByte, _ := reader.ReadByte()

        if (lengthByte & 0xC0) == 0xC0 {
            name.WriteString(getBackTheDomainFromTheHeader(reader, lengthByte))
            break
        }

        if lengthByte == 0 {
            break
        }

        label := make([]byte, lengthByte)
        io.ReadFull(reader, label)
        name.Write(label)
        name.WriteByte('.')

    }

    result, _ := strings.CutSuffix(name.String(), ".")
    return result
}

func getBackTheDomainFromTheHeader(reader *bytes.Reader, lengthByte byte) string {
    nextByte, _ := reader.ReadByte()
    pointer := uint16((uint16(lengthByte) & 0x3F) | uint16(nextByte))

    currentPos, _ := reader.Seek(0, io.SeekCurrent)

    reader.Seek(int64(pointer), io.SeekStart)

    decodedName := DecodeName(reader)

    reader.Seek(currentPos, io.SeekStart)

    return decodedName
}

I created a recursive function for simplicity; basically if the buffer starts with 0xC0 it means we are in front of a “DNS compression algorithm”.

The algorithm consists of a pointer towards the domain name we previously got in the buffer in order to not being repeated and save space. So, we calculate the offset, move there, read the domain name, and then get back to the original position, continuing with the parsing.

Of course this is a very basic algorithm and it can lead to multiple problems (like for example a malicious server can create a pointer to itself creating an infinite loop but you know, it was out of the scope of this challenge 😇)

And last but not the least, let’s parse the records we got:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
func TestResponse(t *testing.T) {
    t.Run("Should create a record from a response", func(t *testing.T) {
        response, _ := hex.DecodeString("00168080000100020000000003646e7306676f6f676c6503636f6d0000010001c00c0001000100000214000408080808c00c0001000100000214000408080404")
        reader := bytes.NewReader(response)
        const RECORD_STARTING_POINT = 32
        skipResponseTill(t, reader, response, RECORD_STARTING_POINT)

        record := ParseRecord(reader)

        assert.NotEmpty(t, record)
        assert.Equal(t, TYPE_A, record.Type)
        assert.Equal(t, CLASS_IN, record.Class)
        assert.Greater(t, record.TTL, uint32(0))
        assert.Greater(t, record.RdLength, uint16(0))
        assert.Equal(t, "8.8.8.8", record.Rdata)

        record = ParseRecord(reader)

        assert.NotEmpty(t, record)
        assert.Equal(t, TYPE_A, record.Type)
        assert.Equal(t, CLASS_IN, record.Class)
        assert.Greater(t, record.TTL, uint32(0))
        assert.Greater(t, record.RdLength, uint16(0))
        assert.Equal(t, "8.8.4.4", record.Rdata)
    })
}

func skipResponseTill(t *testing.T, reader *bytes.Reader, response []byte, startingPoint int64) {
    t.Helper()
    reader.ReadAt(response, startingPoint)
}

Here I could have added more tests and use-cases I know, I’ll leave it to you as a homework. 😎

The implementation:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
type Record struct {
    Name     []byte
    Type     uint16
    Class    uint16
    TTL      uint32
    RdLength uint16
    Rdata    string
}

func ParseRecord(reader *bytes.Reader) *Record {
    var record Record
    record.Name = []byte(DecodeName(reader))
    binary.Read(reader, binary.BigEndian, &record.Type)
    binary.Read(reader, binary.BigEndian, &record.Class)
    binary.Read(reader, binary.BigEndian, &record.TTL)
    binary.Read(reader, binary.BigEndian, &record.RdLength)
    switch record.Type {
    case TYPE_A:
        record.Rdata = readIP(reader, record.RdLength)
    case TYPE_NS:
        record.Rdata = DecodeName(reader)
    default:
        record.Rdata = string(readData(reader, record.RdLength))
    }
    return &record
}

func readIP(reader *bytes.Reader, length uint16) string {
    dataBytes := readData(reader, length)
    return fmt.Sprintf("%d.%d.%d.%d", dataBytes[0], dataBytes[1], dataBytes[2], dataBytes[3])
}

func readData(reader *bytes.Reader, length uint16) []byte {
    dataBytes := make([]byte, length)
    binary.Read(reader, binary.BigEndian, &dataBytes)
    return dataBytes
}

Here we can see how we differentiate between TYPE_A and TYPE_NS in order to be able to decode the domain correctly. This is important because with the first type we get an IP, with the second a domain name.

The record part is the most important one because it might be:

• ANSWER: A list of IP addresses, basically what we are looking for
• AUTHORITIES: A list of NS servers that potentially can have what we are looking for
• ADDITIONALS: A list of IP addresses of the NS servers we got from the AUTHORITITES section.

Let’s put everything together

Now it’s time to use all these function together:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
func resolve(domainName string, questionType uint16) string {
	nameServer := "198.41.0.4"
	for {
		fmt.Printf("Querying %s for %s\n", nameServer, domainName)
		dnsResponse := sendQuery(nameServer, domainName, questionType)
		dnsPacket := getDnsPacketFromResponse(dnsResponse)

		if ip := getAnswer(dnsPacket.answers); ip != "" {
			return ip
		}

		if nsIp := getNameServerIp(dnsPacket.additionals); nsIp != "" {
			nameServer = nsIp
			continue
		}

		if nsDomain := getNameServer(dnsPacket.authorities); nsDomain != "" {
			nameServer = resolve(nsDomain, packet.TYPE_A)
		}
	}
}

Where SendQuery:

1
2
3
4
5
6
7
8
9
func sendQuery(nameServer, domainName string, questionType uint16) []byte {
	query := packet.NewQuery(
		packet.NewHeader(22, 0, 1, 0, 0, 0),
		packet.NewQuestion(domainName, questionType, packet.CLASS_IN),
	)

	client := network.NewClient(nameServer, 53)
	return client.SendQuery(query)
}

Creates the query from the header and the question and then send the query to the nameserver.

Then we get the DNSPacket from the response parsing it:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
func getDnsPacketFromResponse(dnsResponse []byte) *DNSPacket {
	var (
		header      *packet.Header
		questions   []*packet.Question
		answers     []*packet.Record
		authorities []*packet.Record
		additionals []*packet.Record
	)

	reader := bytes.NewReader(dnsResponse)
	header, err := packet.ParseHeader(reader)
	if err != nil {
		fmt.Printf("Can't parse the response header: %v\n", err)
		os.Exit(-1)
	}
	for range header.QdCount {
		questions = append(questions, packet.ParseQuestion(reader))
	}

	for range header.AnCount {
		answers = append(answers, packet.ParseRecord(reader))
	}

	for range header.NsCount {
		authorities = append(authorities, packet.ParseRecord(reader))
	}

	for range header.ArCount {
		additionals = append(additionals, packet.ParseRecord(reader))
	}

	return &DNSPacket{
		header:      header,
		questions:   questions,
		answers:     answers,
		authorities: authorities,
		additionals: additionals,
	}
}

and at the end we check what results we get from the other sections:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
func getAnswer(answers []*packet.Record) string {
	return getRecord(answers)
}

func getNameServerIp(additionals []*packet.Record) string {
	return getRecord(additionals)
}

func getNameServer(authorities []*packet.Record) string {
	return getRecord(authorities)
}

func getRecord(records []*packet.Record) string {
	for _, record := range records {
		if record.Type == packet.TYPE_A || record.Type == packet.TYPE_NS {
			return record.Rdata
		}
	}
	return ""
}

The code and the output?

As always you can find the code on my Github, at this url: https://github.com/dlion/unnije.

And the output looks like this:

1
2
3
4
5
dlion@darkness> unnije % ./unnije domenicoluciani.com
Querying 198.41.0.4 for domenicoluciani.com
Querying 192.41.162.30 for domenicoluciani.com
Querying 108.162.192.65 for domenicoluciani.com
104.21.47.30

or

1
2
3
4
5
6
7
8
9
10
11
12
dlion@darkness unnije % ./unnije domenicoluciani.com twitter.com
Querying 198.41.0.4 for domenicoluciani.com
Querying 192.41.162.30 for domenicoluciani.com
Querying 108.162.192.65 for domenicoluciani.com
172.67.144.42
Querying 198.41.0.4 for twitter.com
Querying 192.41.162.30 for twitter.com
Querying 198.41.0.4 for a.r06.twtrdns.net
Querying 192.55.83.30 for a.r06.twtrdns.net
Querying 205.251.195.207 for a.r06.twtrdns.net
Querying 205.251.192.179 for twitter.com
104.244.42.129

If we want to try providing more than one domain.

Final thoughts and thank yous

I had lot of fun doing this challenge, I studied how the DNS works in the past but I’ve been so close to the actual implementation and I will never stop saying that theory is nothing without a good practice.

Some articles I found helpful to understand better how to overcome this challenge:

• DNS RFC
• Julia Evans blog:
  • https://jvns.ca/blog/2023/07/28/why-is-dns-still-hard-to-learn/
  • https://jvns.ca/blog/2022/11/06/making-a-dns-query-in-ruby-from-scratch/
  • https://jvns.ca/blog/2022/02/14/some-dns-terminology/
• ChatGPT

During this challenge I found extremely helpful pairing with ChatGPT, when using binary protocols having a machine that talk that language is key to dealing with problems and weird behaviors, but be careful using it if you are not sure about what you are doing, sometimes it allucinates and generates funny things. 🥸

Application Layer Load Balancer

Let’s start by describing what is an Application Layer Load Balancer which is important to understand what it does and where are the complexities behind implementing it.

Usually a Load Balancer sits in front of a group of servers and routes client requests across all of the servers that are capable of fulfilling those requests.

Load Balancers ensure that the traffic is equally distributed between our healthy servers minimising the response time.

There are different types of load balancers, they can work at different levels of the OSI. In this case, I’m gonna be focusing on layer seven of the stack, which will route HTTP requests from clients to a pool of HTTP servers.

Functionalities

• Distributes client requests efficiently across multiple servers
• Ensures high availability and reliability by sending requests only to servers that are online

flowchart TD
subgraph front
Client <--> LoadBalancer
end
subgraph back
LoadBalancer <-.-> Server1 & Server2 & Server3
end

Goals of this challenge

• Build a load balancer that can forward traffic to two or more servers.
• Health check the servers
• Handle a server going offline
• Handle a server coming back online

Preface

Just a friendly reminder that the process I took can be avoided, improved, and of course, wrong.
I’m just telling my story and my Golang-improving journey through this post.
Feel free to give me feedback about it and, if it makes you learn something new or reflect on a topic you never thought about, just let me know ☀️

First Requirement: Creating a simple request forwarder

First of all, I needed to create a simple server, Golang is very powerful and it allows you to do so in a few steps.

In the main for example, we can have:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
http.HandleFunc("/", func(writer http.ResponseWriter, request *http.Request) {  
    fmt.Printf("Received request from %s\n", request.RemoteAddr)  
    fmt.Printf("%s / %s\n", request.Method, request.Proto)  
    fmt.Printf("%s / %s\n", request.Method, request.Proto)  
    fmt.Printf("Host: %s\n", request.Host)  
    fmt.Printf("User-Agent: %s\n", request.Header.Get("User-Agent"))  
    fmt.Printf("Accept: %+v\n\n", request.Header.Get("Accept"))  
    fmt.Printf("Replied with a hello message\n")  
    fmt.Fprintf(writer, "Hello From Backend Server")  
})  
err := http.ListenAndServe(":80", nil)  
if err != nil {  
    log.Fatal("Error listening and serve")  
}

And once run we will have our small server listening at the 80 port, logging each request that has been received at the / endpoint.

To verify it you can just call curl http://localhost/ --output - having as a result: Hello From Backend Server.

Of course, it’s not enough to have a service that forwards our requests to specified servers but it’s a start to understand how Golang works.

So to get back to our problem, I started with a unit test:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
t.Run("should call the client to forward the request", func(t *testing.T) {  
    req, _ := http.NewRequest(http.MethodGet, "/", nil)  
    resp := httptest.NewRecorder()  
    mockClient := newMockClient()  
    spartimillu := NewSpartimilluServer(mockClient)  
    mockClient.On("ForwardRequest", mock.Anything).Return(&http.Response{  
       Status:     "200 OK",  
       StatusCode: 200,  
       Proto:      "HTTP/1.0",  
       Body:       io.NopCloser(bytes.NewBufferString("dummy body")),  
       Request:    req,  
    })  
    spartimillu.ServeHTTP(resp, req)  
  
    mockClient.AssertExpectations(t)  
    assert.Equal(t, "dummy body", resp.Body.String(), "got %q, want %q", resp.Body.String(), "dummy body")  
})

With this test:

• I create a new mockedRequest specifying the method and the endpoint.
• Create some mocked response thanks to the httptest package.
• Create a mocked client.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
type MockClient struct {  
    mock.Mock  
}  
  
func newMockClient() *MockClient { return &MockClient{} }  
  
func (m *MockClient) ForwardRequest(req http.Request) *http.Response {  
    args := m.Called(req)  
    return args.Get(0).(*http.Response)  
}  
  
func (m *MockClient) HealthCheck() {  
    m.Called()  
}

• I provide my mocked client as a parameter for my load balancer server that I called SpartimilluServer.

1
2
3
4
5
6
7
type SpartimilluServer struct {  
    client client.Client  
}  
  
func NewSpartimilluServer(client client.Client) *SpartimilluServer {  
    return &SpartimilluServer{client: client}  
}

• I mock the ForwardRequest function to return a custom response with dummy body as a body.
• And then call the function under test: ServeHttp.
• Checking that the body of the response that we get back is equal to what we are expecting.

The implementation was quite straightforward:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
func (s *SpartimilluServer) ServeHTTP(w http.ResponseWriter, r *http.Request) {  
    fmt.Printf("Received request from %s\n", r.RemoteAddr)  
    fmt.Printf("%s / %s\n", r.Method, r.Proto)  
    fmt.Printf("%s / %s\n", r.Method, r.Proto)  
    fmt.Printf("Host: %s\n", r.Host)  
    fmt.Printf("User-Agent: %s\n", r.Header.Get("User-Agent"))  
    fmt.Printf("Accept: %+v\n", r.Header.Get("Accept"))  
  
    resp := s.client.ForwardRequest(*r)  
  
    fmt.Printf("Response from server: %s %s\n\n", resp.Proto, resp.Status)  
  
    body, err := io.ReadAll(resp.Body)  
    if err != nil {  
       http.Error(w, "Error reading the response body", http.StatusInternalServerError)  
    }  
    stringBody := string(body)  
    fmt.Fprint(w, stringBody)  
    fmt.Println(stringBody)  
}

Forward Request

Now we need to implement our client and its functionalities.

Let’s start with an integration test!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
t.Run("should forward a GET request to a specific server", func(t *testing.T) {  
    server, address := startTestServer(t, "ok")  
    defer server.Close()  
    conf := NewSpartimilluClientConf([]string{address})  
    client := NewSpartimilluClient(conf)  
    req := httptest.NewRequest(http.MethodGet, "/", nil)  
  
    resp := client.ForwardRequest(*req)  
  
    body := getBodyFromResp(t, resp)  
    assert.Equal(t, http.MethodGet, resp.Request.Method, "got %v, wanted %v", resp.Request.Method, http.MethodGet)  
    assert.Equal(t, http.StatusOK, resp.StatusCode, "got %v, wanted %v", resp.StatusCode, http.StatusOK)  
    assert.Equal(t, "ok", body, "got %v, wanted %v", body, "ok")  
})

So let’s see what is happening here:

• Golang gives us the fantastic opportunity to spawn up a local stub server with just a few lines of code using the httptest package and defer it with the Close method:

1
2
3
4
5
6
7
8
9
func startTestServer(t *testing.T, bodyResponse string) (*httptest.Server, string) {  
    t.Helper()  
  
    server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) {  
       fmt.Printf("%s has been called\n", bodyResponse)  
       fmt.Fprint(writer, bodyResponse)  
    }))    
    return server, server.URL  
}

As you can see we are using the NewServer method to spawn up a new server and then set a handler function to get back some info.

• I wanted to have flexibility in terms of configuration so I created a SpartimilluClientConf struct:

1
2
3
4
5
6
7
type SpartimilluClientConf struct {  
    addresses           []string
}  
  
func NewSpartimilluClientConf(addresses []string) SpartimilluClientConf {  
    return SpartimilluClientConf{addresses: addresses}  
}

As you can see it just contains some info like addresses and the health-check endpoint (we will get there later).

• I created the client passing the conf as a dependency
• Then call the ForwardRequest method passing the request I created.
• Some assertions, to verify that everything is as expected. For example to read the body:

  1
  2
  3
  4
  5
  6
  7
  func getBodyFromResp(t *testing.T, resp *http.Response) string {  
    t.Helper()  
    
    bodyBytes, err := io.ReadAll(resp.Body)  
    assert.Nil(t, err)  
    return string(bodyBytes)  
  }
  

To then jump into the implementation:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
type Client interface {  
    ForwardRequest(req http.Request) *http.Response  
}  
  
type SpartimilluClient struct {  
    conf SpartimilluClientConf  
}  
  
func NewSpartimilluClient(conf SpartimilluClientConf) *SpartimilluClient {  
    return &SpartimilluClient{conf: conf}  
}  
  
func (s *SpartimilluClient) ForwardRequest(req http.Request) *http.Response {  
    switch req.Method {  
    case http.MethodGet:  
       return sendGetRequestToAnotherServer(s.conf.address + req.RequestURI)
    }    
    return nil  
}  
  
func sendGetRequestToAnotherServer(url string) *http.Response {  
    body, err := http.Get(url)  
    if err != nil {  
       log.Fatal("Can't read the response body from the GET request")  
    }
    return body  
}

As you can see I just called the http.Get(url) method emulating the GET request we got.

Try it out

To see everything in action we can just call our main:

1
2
3
4
5
func main() {  
    spartimilluClient := client.NewSpartimilluClient(client.NewSpartimilluClientConf("http://localhost:8080"))  
    spartimilluServer := server.NewSpartimilluServer(spartimilluClient)  
    log.Fatal(http.ListenAndServe(":80", spartimilluServer))  
}

• In this case I spawned up a separate server listening at the port 8080 and provided it into the configuration.
• Then created my SpartimilluServer function handler and used it for my ListenAndServe function.

To spawn up my little server you can use a different main with the code you saw before or just create a directory called for example server8080 containing an index.html file with this content:

1
2
3
4
5
6
7
8
9
10
<!DOCTYPE html>  
<html lang="en">  
    <head>  
       <meta charset="utf-8">  
       <title>Index Page</title>  
    </head>  
    <body>  
       Hello from the web server running on port 8080.  
    </body>  
</html>

and run in your terminal: python -m http.server 8080 --directory server8080, it will spawn up a python server serving the content of the directory directory8080.

So then you can just call your young load balancer: curl http://localhost/ --output -.

The result will be:

1
2
3
4
5
6
7
8
9
10
11
❯ curl http://localhost/ --output -
<!DOCTYPE html>
<html lang="en">
	<head>
		<meta charset="utf-8">
		<title>Index Page</title>
	</head>
	<body>
		Hello from the web server running on port 8080.
	</body>
</html>

flowchart TD
subgraph front
CURL --"1. GET index.html"--> SpartimilluServer
SpartimilluServer --"4. index.html"--> CURL
end
subgraph back
SpartimilluServer-. "2. GET index.html" .-> PythonServer8080
PythonServer8080-. "3. index.html" .-> SpartimilluServer
end

Second Requirement: Distribute traffic with Round Robin

Now that we have our “forwarder” in place, we have to distribute the incoming requests using a scheduling algorithm called “Round Robin”.
It’s quite simple, we just need to distribute the traffic to each server in the list, one after the other and once forwarded to all of them we start back at the beginning of the list.

For example:

Let’s start with another integration test:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
t.Run("should forward a GET request to any server using a round robin algorithm", func(t *testing.T) {  
    server1, address1 := startTestServer(t, "server1")  
    defer server1.Close()  
    server2, address2 := startTestServer(t, "server2")  
    defer server2.Close()  
    server3, address3 := startTestServer(t, "server3")  
    defer server3.Close()  
    conf := NewSpartimilluClientConf([]string{address1, address2, address3})  
    client := NewSpartimilluClient(conf)  
    req := httptest.NewRequest(http.MethodGet, "/", nil)  
  
    resp := client.ForwardRequest(*req)  
  
    body := getBodyFromResp(t, resp)  
    assert.Equal(t, http.MethodGet, resp.Request.Method, "got %v, wanted %v", resp.Request.Method, http.MethodGet)  
    assert.Equal(t, http.StatusOK, resp.StatusCode, "got %v, wanted %v", resp.StatusCode, http.StatusOK)  
    assert.Equal(t, "server1", body, "got %v, wanted %v", body, "server1")  
  
    resp = client.ForwardRequest(*req)  
  
    body = getBodyFromResp(t, resp)  
    assert.Equal(t, http.MethodGet, resp.Request.Method, "got %v, wanted %v", resp.Request.Method, http.MethodGet)  
    assert.Equal(t, http.StatusOK, resp.StatusCode, "got %v, wanted %v", resp.StatusCode, http.StatusOK)  
    assert.Equal(t, "server2", body, "got %v, wanted %v", body, "server2")  
  
    resp = client.ForwardRequest(*req)  
  
    body = getBodyFromResp(t, resp)  
    assert.Equal(t, http.MethodGet, resp.Request.Method, "got %v, wanted %v", resp.Request.Method, http.MethodGet)  
    assert.Equal(t, http.StatusOK, resp.StatusCode, "got %v, wanted %v", resp.StatusCode, http.StatusOK)  
    assert.Equal(t, "server3", body, "got %v, wanted %v", body, "server3")  
})

We spawn up 3 servers and set their address in our configuration.

1. We call ForwardRequest
2. We expect to contact and then receive an answer from the sever1
3. We call again ForwardRequest
4. We expect to contact and then receive an answer from the server2.
5. We call again ForwardRequest
6. We expect to contact and then receive an answer from the server3.

Let’s jump into the implementation:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
type SpartimilluClient struct {  
    conf    SpartimilluClientConf  
    counter int  
}

func (s *SpartimilluClient) ForwardRequest(req http.Request) *http.Response {  
    var resp *http.Response  
  
    serverIndex := s.counter % len(s.conf.addresses)  
  
    switch req.Method {  
    case http.MethodGet:  
       resp = sendGetRequestToAnotherServer(s.conf.addresses[serverIndex] + req.RequestURI)   
    }  
    s.counter++  
  
    return resp  
}

Using a counter counter and the module operator I implemented a simple round-robin algorithm.

Let’s see how it works:

And so on. 🤯

Try it out

Our main should look like this:

1
2
3
4
5
6
7
8
func main() {  
    spartimilluClient := client.NewSpartimilluClient(client.NewSpartimilluClientConf([]string{  
       "http://localhost:8080",  
       "http://localhost:8081",  
    }))
    spartimilluServer := server.NewSpartimilluServer(spartimilluClient)  
    log.Fatal(http.ListenAndServe(":80", spartimilluServer))  
}

As you can see we specified 2 addresses in our configuration, a server listening at 8080 and one listening at 8081.

Of course, before starting our load balancer we should spawn up our servers.

Let’s create a directory (as we did before for the server8080) but this time called server8081 inside an index.html containing something similar:

1
2
3
4
5
6
7
8
9
10
<!DOCTYPE html>  
<html lang="en">  
    <head>  
       <meta charset="utf-8">  
       <title>Index Page</title>  
    </head>  
    <body>  
       Hello from the web server running on port 8081.  
    </body>  
</html>

Then we can just run these commands in 2 different shells:

1. server8080: python -m http.server 8080 --directory server8080
2. server8081: python -m http.server 8081 --directory server8081

Once both servers are up we can test it out just executing our main and calling our load balancer three times to see how it works:

1
2
3
4
5
6
7
8
9
10
11
❯ curl http://localhost/ --output -
<!DOCTYPE html>
<html lang="en">
	<head>
		<meta charset="utf-8">
		<title>Index Page</title>
	</head>
	<body>
		Hello from the web server running on port 8080.
	</body>
</html>

and

1
2
3
4
5
6
7
8
9
10
11
❯ curl http://localhost/ --output -
<!DOCTYPE html>
<html lang="en">
	<head>
		<meta charset="utf-8">
		<title>Index Page</title>
	</head>
	<body>
		Hello from the web server running on port 8081.
	</body>
</html>

and

1
2
3
4
5
6
7
8
9
10
11
❯ curl http://localhost/ --output -
<!DOCTYPE html>
<html lang="en">
	<head>
		<meta charset="utf-8">
		<title>Index Page</title>
	</head>
	<body>
		Hello from the web server running on port 8080.
	</body>
</html>

flowchart TD
subgraph front
CURL --"1. GET index.html"--> SpartimilluServer
SpartimilluServer --"4/6. index.html"--> CURL
end
subgraph back
SpartimilluServer-. "2. GET index.html" .-> PythonServer8080
PythonServer8080-. "3. index.html" .-> SpartimilluServer
SpartimilluServer-. "4. GET index.html" .-> PythonServer8081
PythonServer8081-. "5. index.html" .-> SpartimilluServer
end

Third Requirement: Implement a Health Check

Now that we implemented the main functionality we have to implement the health check that helps us to always forward the request to a live server.

Let’s start with a unit test:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
t.Run("should call the client to do an health check", func(t *testing.T) {  
    req, _ := http.NewRequest(http.MethodGet, "/healthcheck", nil)  
    resp := httptest.NewRecorder()  
    mockClient := newMockClient()  
    spartimillu := NewSpartimilluServer(mockClient)  
    mockClient.On("HealthCheck").Return(&http.Response{  
       Status:     "200 OK",  
       StatusCode: 200,  
       Proto:      "HTTP/1.0",  
       Request:    req,  
    })  
    spartimillu.HealthCheck()  
  
    mockClient.AssertExpectations(t)  
    assert.Equal(t, http.StatusOK, resp.Code, "got %q, want %q", resp.Code, http.StatusOK)  
})

1
2
3
4
5
6
7
8
9
10
11
12
13
14
type MockClient struct {  
    mock.Mock  
}  
  
func newMockClient() *MockClient { return &MockClient{} }  
  
func (m *MockClient) ForwardRequest(req http.Request) *http.Response {  
    args := m.Called(req)  
    return args.Get(0).(*http.Response)  
}  
  
func (m *MockClient) HealthCheck() {  
    m.Called()  
}

As we have done before we are checking that the method HealthCheck has been implemented correctly in our SpartimilluServer.

With a very simple implementation:

1
2
3
4
5
func (s *SpartimilluServer) HealthCheck() {  
    fmt.Printf("Performing Health Check\n")  
  
    s.client.HealthCheck()  
}

This method would be called every N seconds to check if our servers are still alive.

As usual, let’s continue with an integration test for our client:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
t.Run("should perform a health check towards a server", func(t *testing.T) {  
    server1, address1 := startTestServer("server1")  
    defer server1.Close()  
    server2, address2 := startTestServer("server2")  
    defer server2.Close()  
    server3, address3 := startTestServer("server3")  
    defer server3.Close()  
  
    conf := NewSpartimilluClientConf([]string{address1, address2, address3}, "/healthcheck")  
    client := NewSpartimilluClient(conf)  
    req := httptest.NewRequest(http.MethodGet, "/", nil)  
  
    client.HealthCheck()  
    resp := client.ForwardRequest(*req)  
  
    assert.Equal(t, http.StatusOK, resp.StatusCode, "got %v, wanted %v", resp.StatusCode, http.StatusOK)  
    assert.Equal(t, server1.URL, resp.Request.URL.Scheme+"://"+resp.Request.Host, "got %v, wanted %v", resp.Request.URL.Scheme+"://"+resp.Request.Host, server1.URL)  
  
    resp = client.ForwardRequest(*req)  
  
    assert.Equal(t, http.StatusOK, resp.StatusCode, "got %v, wanted %v", resp.StatusCode, http.StatusOK)  
    assert.Equal(t, server2.URL, resp.Request.URL.Scheme+"://"+resp.Request.Host, "got %v, wanted %v", resp.Request.URL.Scheme+"://"+resp.Request.Host, server2.URL)  
  
    resp = client.ForwardRequest(*req)  
  
    assert.Equal(t, http.StatusOK, resp.StatusCode, "got %v, wanted %v", resp.StatusCode, http.StatusOK)  
    assert.Equal(t, server3.URL, resp.Request.URL.Scheme+"://"+resp.Request.Host, "got %v, wanted %v", resp.Request.URL.Scheme+"://"+resp.Request.Host, server3.URL)  
  
    server1.Close()  
  
    client.HealthCheck()  
    resp = client.ForwardRequest(*req)  
  
    assert.Equal(t, http.StatusOK, resp.StatusCode, "got %v, wanted %v", resp.StatusCode, http.StatusOK)  
    assert.Equal(t, server2.URL, resp.Request.URL.Scheme+"://"+resp.Request.Host, "got %v, wanted %v", resp.Request.URL.Scheme+"://"+resp.Request.Host, server2.URL)  
  
    server2.Close()  
    client.HealthCheck()  
    resp = client.ForwardRequest(*req)  
  
    assert.Equal(t, http.StatusOK, resp.StatusCode, "got %v, wanted %v", resp.StatusCode, http.StatusOK)  
    assert.Equal(t, server3.URL, resp.Request.URL.Scheme+"://"+resp.Request.Host, "got %v, wanted %v", resp.Request.URL.Scheme+"://"+resp.Request.Host, server3.URL)  
  
    resp = client.ForwardRequest(*req)  
    assert.Equal(t, http.StatusOK, resp.StatusCode, "got %v, wanted %v", resp.StatusCode, http.StatusOK)  
    assert.Equal(t, server3.URL, resp.Request.URL.Scheme+"://"+resp.Request.Host, "got %v, wanted %v", resp.Request.URL.Scheme+"://"+resp.Request.Host, server3.URL)  
})

Let’s see what we did:

• As usual, we spawned up 3 servers adding them to our servers list
• We call our HealthCheck method that supposed to update our list of available servers
• We call the ForwardRequest method checking then if we contact the right server.
• The first 3 times we checked if we had contacted iteratively each server.
• We shut down the server1
• We re-do the health-check
• We re-call our ForwardRequest method but this time our method should call the server2 since the server1 has been shut down.
• We do the same thing then for the server2, expecting then to call the only one left server3.

Here is the implementation:

1
2
3
4
type Client interface {  
    ForwardRequest(req http.Request) *http.Response  
    HealthCheck()  
}

1
2
3
4
5
type SpartimilluClient struct {  
    conf           SpartimilluClientConf  
    counter        int  
    healthyServers map[string]bool  
}

We added to our struct a map of healthyServers that will be updated by our HealthCheck method.

1
2
3
func NewSpartimilluClient(conf SpartimilluClientConf) *SpartimilluClient {  
    return &SpartimilluClient{conf: conf, healthyServers: make(map[string]bool)}  
}

1
2
3
4
5
6
7
8
9
10
func (s *SpartimilluClient) HealthCheck() {  
    for _, address := range s.conf.addresses {  
       resp, err := http.Get(address)  
       if err == nil && resp.StatusCode == http.StatusOK {  
          s.healthyServers[address] = true  
       } else {  
          s.healthyServers[address] = false  
       }  
    }
}

We iterate over the list of addresses and for each server we do a GET request to check if we got an OK, if so we update our healthyServers map having as a key the address and a boolean value as a value.

And this is the implementation of our ForwardRequest method:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
func (s *SpartimilluClient) ForwardRequest(req http.Request) *http.Response {  
    if len(s.healthyServers) == 0 {  
       s.HealthCheck()  
    }  
    index := s.counter % len(s.conf.addresses)  
    address := s.conf.addresses[index]  
    s.counter++  
  
    if s.healthyServers[address]{  
       switch req.Method {  
       case http.MethodGet:  
          return sendGetRequestToAnotherServer(address + req.RequestURI)
      }    
   }  
   return s.ForwardRequest(req)  
}

Here we make sure that we perform a health check at least once before we start forwarding requests around.
Then we check if the server we want to contact is healthy then we contact it otherwise we do a recursive call to call the next one in the list.
The recursion is not very efficient but it works nicely for now, we will improve it in the next step.

Try it out

Our main now looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
func main() {
  spartimilluClient := client.NewSpartimilluClient(client.NewSpartimilluClientConf([]string{  
       "http://localhost:8080",  
       "http://localhost:8081",  
    }, "/healthcheck"))  
   spartimilluServer := server.NewSpartimilluServer(spartimilluClient)  
  
  ticker := time.NewTicker(5 * time.Second)
  go func() {
      for {  
          select {  
          case <-ticker.C:  
             spartimilluServer.HealthCheck()  
          }       
      }
  }()
  log.Fatal(http.ListenAndServe(":80", spartimilluServer))  
}

This is the first raw version of our async HealtCheck.
We call asynchronously our HealthCheck every 5 seconds using a ticker.

To try it out we can reproduce the steps above spawning our 2 stubbed servers and once we run our load-balancer we can just kill one of the two to check if our load-balancer every 5 seconds decides to contact the only one that is still alive.

flowchart TD
subgraph front
CURL --"1. GET index.html"--> SpartimilluServer
SpartimilluServer --"4. index.html"--> CURL
end
subgraph back
PythonServer8080
SpartimilluServer-. "2. GET index.html" .-> PythonServer8081
PythonServer8081-. "3. index.html" .-> SpartimilluServer
end

Fighting concurrency

Another thing we have to make sure to handle is the concurrency.
As you could have seen our HealthCheck function is modifying a shared map with ForwardRequest, and it can cause concurrency issues since it can be accessed by both functions at the same time. We can do that using a mutex.

1
2
3
4
5
6
type SpartimilluClient struct {  
    conf           SpartimilluClientConf  
    counter        int  
    healthyServers map[string]bool  
    mu             sync.Mutex  
}

Here is our HealthCheck implementation:

1
2
3
4
5
6
7
8
9
10
11
12
13
func (s *SpartimilluClient) HealthCheck() {  
    for _, address := range s.conf.addresses {  
       resp, err := http.Get(address)  
  
       s.mu.Lock()  
       if err == nil && resp.StatusCode == http.StatusOK {  
          s.healthyServers[address] = true  
       } else {  
          s.healthyServers[address] = false  
       }  
       s.mu.Unlock()  
    }
}

Every time we want to access our healthyServers map we lock it down to be sure that anybody else can do it before releasing it back.

And the ForwardRequestone:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
func (s *SpartimilluClient) ForwardRequest(req http.Request) *http.Response {  
    for {  
       if len(s.healthyServers) == 0 {  
          s.HealthCheck()  
       }  
       s.mu.Lock()  
       index := s.counter % len(s.conf.addresses)  
       address := s.conf.addresses[index]  
       s.counter++  
  
       if s.healthyServers[address] == true {  
          s.mu.Unlock()  
          switch req.Method {  
          case http.MethodGet:  
             return sendGetRequestToAnotherServer(address + req.RequestURI)  
          }       
       }       
      s.mu.Unlock()  
      time.Sleep(100 * time.Millisecond)  
    }
}

Here I did a bit of refactoring and of course, used the Lock to handle concurrency issues. I also removed the recurse for a for/infinite loop with a sleep time to re-try contacting our servers.

Try it out

After a refactoring our main looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
func main() {  
    const seconds = 1 * time.Second  
  
    spartimilluClient := client.NewSpartimilluClient(client.NewSpartimilluClientConf([]string{  
       "http://localhost:8080",  
       "http://localhost:8081",  
    }, "/healthcheck"))  
    spartimilluServer := server.NewSpartimilluServer(spartimilluClient)  
  
    go doEvery(seconds, spartimilluServer.HealthCheck)  
  
    log.Fatal(http.ListenAndServe(":80", spartimilluServer))  
}  
  
func doEvery(d time.Duration, f func()) {  
    ticker := time.Tick(d)  
    for range ticker {  
       go f()  
    }
}

A this time you should have your load balancer switching from one server to another and performing health checks correctly.

Final Thoughts

It has been a quite fun challenge, iteratively I built up my application load balancer starting from a small forward to then adding a more complex logic facing up some nice challenges like evolving my code to embrace the change and how to do integration tests spawning-up stub servers.

Of course, this is a very basic load balancer, it can be improved and extended but I’m satisfied with it for now.

You can find the repository with the code in my Github profile, https://github.com/dlion/spartimillu.

What do you think about my solution? Any feedback would be appreciated and of course, if you make your solution don’t be shy and share it with me too!

Happy Coding!

Intro and Coding Challenge

Golang has always been one of my favorite languages and I’ve been using it for a few months so far, after my Take 3 experience I decided to keep studying it during my free time, and since I like hands-on projects, I used it to create a side-pet-challenge/project.

Coding Challenges

I recently discovered Coding Challenges, a website full of hands-on coding challenges that’s possible to take in different languages, I chose mine: Go.
Sometimes I struggle to find new ideas and this website helped me a lot with that.

WC

To start with something simple I decided to implement WC, the famous Word Count unix tool.
To learn more about it, run man wc in your terminal but what it essentially does is count words, lines, characters, and bytes of a specific file or pipe stream.
From this very high-level point of view, it seems quite simple but digging deeper you will see that it’s not as simple as you could have thought at the beginning.

Preface

Just a friendly reminder that the process I took can be avoided, improved, and of course, wrong.
I’m just telling my story through this project and my Golang improving journey.
Feel free to give me feedback about it and, if it makes you learn something new or reflect on a topic you never thought about, just let me know ☀️

First Requirement: Count bytes

Starting from scratch in Go is quite simple, so I just created my repo, opened my IntelliJ Goland IDE, and, created a simple hello world ready to jump into my first requirement implementation.
The first requirement is to have a small functionality, just counting the number of bytes from a specific file.
Using the file that has been provided the result of this command should be:

1
2
>./gowc -c test.txt
  342190 test.txt

Let’s see what we got from that:

Input:

• We have a -c parameter which is the way we activate the count bytes functionality
• We pass a test.txt which is the file we want to count from Output:
• Space
• Number of bytes
• Name of the file that has been read

Through my repo’s commit you can see the history of my changes, I started with something completely different (like mocking a filesystem using testify/mock) ending up with a bunch of simple unit tests:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
func TestWcBytesReader(t *testing.T) {  
    t.Run("Count reads 0 bytes", func(t *testing.T) {  
       dummyContent := make([]byte, 0)  
       r := NewWcBytesReader()  
  
       currentBytes := r.Count(dummyContent)  
  
       expected := int64(0)  
       assert.Equal(t, expected, currentBytes, "Got %d, wanted %d", currentBytes, expected)  
    })
      
    t.Run("Count reads 1 byte", func(t *testing.T) {  
       dummyContent := make([]byte, 1)  
       r := NewWcBytesReader()  
  
       currentBytes := r.Count(dummyContent)  
  
       expected := int64(1)  
       assert.Equal(t, expected, currentBytes, "Got %d, wanted %d", currentBytes, expected)  
    })
      
    t.Run("Count reads multiple bytes", func(t *testing.T) {  
       dummyContent := make([]byte, 100)  
       r := NewWcBytesReader()  
  
       currentBytes := r.Count(dummyContent)  
  
       expected := int64(100)  
       assert.Equal(t, expected, currentBytes, "Got %d, wanted %d", currentBytes, expected)  
    })
 }

The implementation as you can imagine wasn’t a big deal for this feature:

1
2
3
func (w WcBytesReader) Count(content []byte) int64 {  
    return int64(len(content))  
}

We will skip for now how I used it in the main, if you want to try out the parameter part you can just use flag and parse that calling the Count function from there once NewWcBytesReader has been called.

Second Requirement: Count Lines

The second requirement was to support the command line option -l that outputs the number of lines in a file.
The CLI input/output should be:

1
2
>./gocw -l test.txt
    7145 test.txt

Let’s see what we got from that:

Input:

• We have a -l parameter which is the way we activate the count bytes functionality
• We pass a test.txt which is the file we want to count from Output:
• Space
• Number of lines
• Name of the file that has been read

Here are some unit tests I wrote: