The document discusses the challenges of parsing URLs, particularly in shell scripts using curl, and proposes a dedicated tool called 'trurl' for URL parsing and manipulation. Trurl offers various features such as extracting and updating URL components, handling relative URLs, and managing query strings efficiently. Currently in development with over 140 tests, trurl is managed by the curl organization and aims for a stable one-dot-zero release in the future.

1. 2024

2. Problems
Parsing URLs is difficult - especially in shell scripts
Scripts using curl often need a certain amount of URL “fiddling”
Mixing URL parsers is a known recipe for security problems
@bagder

3. Solution
A dedicated tool for URL parsing and manipulation
A ‘tr’ for URLs
Using curl’s URL parser for consistency
A curl companion tool
@bagder

4. Features
- extract any URL part
- update any part of a URL
- create a URL from components
- apply a relative URL on an absolute: “do a redirect”
- extract specific pieces of the query
- trim off specific pieces of the query
- verify that the URL is valid
- output URL components as JSON
- all of this on any number of provided URL
@bagder

5. Examples: set
Replace the hostname of a URL
$ trurl --url https://curl.se --set host=example.com
https://example.com/
Create a URL by setting components
$ trurl --set host=example.com --set scheme=ftp
ftp://example.com/
@bagder

6. Examples: get
Extract the path from a URL
$ trurl --url https://curl.se/we/are.html --get '{path}'
/we/are.html
Extract the port from a URL
$ trurl --url https://curl.se/h.html --get '{default:port}'
443
@bagder

7. Examples: query
Remove tracking tuples from query
$ trurl "https://curl.se?search=hey&utm_source=tracker"
--trim query="utm_*"
https://curl.se/?search=hey
Show a specific query key value
$ trurl "https://example.com?a=home&here=now&thisthen" -g
'{query:a}'
home
@bagder

8. Development
https://github.com/curl/trurl
Managed by the curl organization
still pre 1.0 release
140+ tests
(has helped us exercise and polish the URL parser and its API)
@bagder

9. Status
Works
Solid
Shipped by several distros
@bagder

10. Future
A one-dot-zero release
What else?
@bagder