Writing a link checker in Rust in a weekend (plus two years)

Matthias Endler

Playlists: 'froscon2023' videos starting here / audio

Many developers suffer from the "curl" syndrome: they think they could
write curl on a weekend. After all, how hard could it be to send a network request and print the result?
I thought the same when I started writing a link checker called lychee in async Rust. After just one weekend (and two more short years) I got the basics to work. Follow me on my journey through the weirdness of the web and how seemingly simple things can get complex pretty quickly.

How hard could it be?

That's a question a lot of developers routinely ask when they look at a seemingly simple task like writing a "curl clone" or a word count tool. As developers, we often underestimate the amount of work it takes to build a fast, robust and correct tool from scratch, which handles all the real-world edge cases.

By sharing the trials and tribulations of my development process, I aim to shed light on the reality of software development, the importance of appreciating the work done by countless open-source developers, and the skills needed to successfully navigate the path from concept to functional tool. I will also briefly touch on the difficulties of funding such open source projects.